Same training time for different values of sliding window in Longformer
Author: allohvk
Pull request: no
Labels: name: bug

### System Info

Transformers: 4.20.1
Python: 3.8.12
Pretrained models & tokenizer from HF: "allenai/longformer-base-4096"

The training time does not change for any value of sliding window. For e.g. a sliding window of 2 or 512 (which is the default) or 1024 takes the same training time. This seems to be a bug to me. I need a very small local window span (sliding window max 64 across 4096 tokens) and the model is simply unusable in this scenario due to excessive training time

### Who can help?

@ydshieh

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

A simple: model.config.attention_window = [SLIDE_WIN_ATTN]*12


### Expected behavior

I would expect training time to fall somewhat quadratically for lower values of SLIDE_WIN_ATTN (say for 64) as compared to the default which is 512. However the training time for both cases is the same (around 24 hours per epoch). In fact SLIDE_WIN_ATTN values from 2 to 1024 roughly take the same training time which should not be the case

Comments:
@allohvk Could you provide a (minimal) training script that demonstrates this issue, probably using a dataset from HF Hub if necessary?

- I will try to do that. I need to look for a large enough dataset such that the training times show a tangible difference between different scenarios
- I was exploring alternative architectures (like Big Bird) and came across a disclaimer there stating that benefits of sparse attention become visible for only 1024 max-seq-length and beyond. Perhaps Longformer too has this limitation and if so, this becomes just a documentation issue. Maybe Longformer is just not optimized to handle sliding windows of length < 512 and hence shows no tangible difference in execution time for sliding window size=2 or sliding window size = 512.

Thanks for the info. @allohvk . BTW, on what task you trained this model? It's also a good idea to double check the way you prepare `global_attention_mask` (if you ever use it).

Taking some time to measure model `forward` timing with window size ` [4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]`.
Here is the result. ~~(Will do more measurement when I have time)~~

### Results (in seconds, for `32` forwards per window size)
```
[47.823847, 46.043184, 46.290201, 46.691181, 48.692595, 53.176747, 62.156357, 81.265798, 149.261874, 273.367502]
```

It indeed looks like the advantage appears for longer enough length.

### Code
```python
import torch
from transformers import LongformerModel, LongformerTokenizer, LongformerConfig


def measure(w_size=512):

    config = LongformerConfig.from_pretrained("allenai/longformer-base-4096")
    config.attention_window = w_size
    model = LongformerModel.from_pretrained("allenai/longformer-base-4096", config=config)
    tokenizer = LongformerTokenizer.from_pretrained("allenai/longformer-base-4096")

    print(model.config.attention_window)

    SAMPLE_TEXT = " ".join(["Hello world! "] * 1000)  # long input document
    input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1

    attention_mask = torch.ones(
        input_ids.shape, dtype=torch.long, device=input_ids.device
    )  # initialize to local attention
    global_attention_mask = None

    import datetime

    s = datetime.datetime.now()
    for i in range(32):
        outputs = model(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask)
    e = datetime.datetime.now()
    l = (e-s).total_seconds()
    print(l)

    sequence_output = outputs.last_hidden_state
    pooled_output = outputs.pooler_output

    print(sequence_output.shape)

    return l

ls = [measure(w) for w in [4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]]
print(ls)
```

Thanks @ydshieh for taking time out to test this and I must apologize if I wasn't clear. I was actually referring to the training time taken, by which I mean the time to fine-tune a pre-trained model with additional training data before actually inferring. 
I would assume that just like the inference time, the training time too should change based on the length of the sliding window. It should be shorter for a window of (say)128 compared to a window of 512 but the training hours don't change. I will share a small but complete working code with you in a couple of days. 

To answer your question - I am training a simple classifier using the pertained weights of the base model. I just pass the last state output (768 dim) to a linear regression head. The dataset is actually composed of short NL statements appended with an associated context which are long programming code snippets (something on the lines of what CodeBert does). 

Just as an FYI, I tried BigBird today and had the same issue, the training time taken taken for "sparse_attention" is the same as the training time taken for a "full_attention" for a 2048 seq_len. "sparse_attention" option actually just attends to 2 x 64 + 3 x 64 + 2 x 64 = 448 tokens which is far less than 2048 and should be much much faster.

You can choose to close this ticket if you so wish. I will change my dataset to IMDB and share a simulatable code in couple of days.

Hi @allohvk , I know you are talking about the training time. However, even with just the `forward` method of the model, we already see that the effect of `window_size` (used for local attentions), i.e. to have linear time instead of quadratic time, will appear **only for large** enough `window_size` (and therefore with long enough sequences).

For small `window_size`, some overhead will prevent this much desired effect. From this observation, I am afraid that this holds for training too.

If you try to measure this line directly https://github.com/huggingface/transformers/blob/8a61fe023430115bb61ec328a29d35571f4fc2c4/src/transformers/models/longformer/modeling_longformer.py#L820

(without any other parts, and therefore no other overhead), you will see this linear/quadratic running time.

- Got it. I suppose this is very reasonable ramification of using a specialised attention model which handles long sequences. There is no visible benefit in having sliding window size < 128. Possibly it can just be documented somewhere. I will close this as "not a bug" for now.
- I may still have a problem with the model taking quadratic time for longer sequences even with default values of sliding window. However will recheck if it is a bug in my training code. If not, will share a simulatable code by which the problem can be replicated. I will open a new ticket for that.

Hi there, I have a little question about Longformer's attention_window: 

Since the attentions are only calculated in a window, is it right or not to use the cls patch(first patch of the sequence) to do downstream tasks? I doubt whether the cls patch has relations with patches in other chunks!

I don't look into the detail, but `Longformer` has `global attentions` - some tokens attend to all tokens. If cls token is one such token (that attend to all tokens), then it has global information.

[HF Forum](https://discuss.huggingface.co/) is a better place for this question.