Why the lengthy max_length in Dataset effect the performance of T5 ?
Author: SinclairCoder
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.5
- Platform: Linux-4.15.0-172-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help

@patrickvonplaten   @Narsil 


## Information

I use t5-base to perform text generation tasks. When I conducted experiments on the benchmark, I found that the lengthy max_length (the max length of original dataset may be 128, I set the max_length to 150,200) will effect the text generation performance a lot. Theoretically, too long max length will not affect the calculation of loss through padding and mask, so I'm curious as to why it would affect performance.

The code  is available at https://github.com/IsakZhang/ABSA-QUAD

## Expected behavior

This mistake can be as retarded as it is vital. Although I have been debugging for several days, I still haven't found the problem and look forward to your replies.

Comments:
@SinclairCoder ,

Hi, `max_length` affects how many tokens are generated by the model, and even if padding an so on shouldn't apply (I haven't even looked at but it shouldn't).

However, compute is `O(n²)` in `seq_length`, so if your generation is generating much larger sequences, then it's normal that it takes much longer.

In order to confirm, could you provide a *short* example please that we could try on ? Thanks the code you shared, but it's too long to properly look over to run and identify issues.

Could you maybe also print somewhere your sequences and their lengths to confirm my hypothesis ?

Cheers.

@Narsil  @patrickvonplaten 

Thanks for your quick reply. I don't think you get my point. See the demo below. When specified different `max_input_len` of `MyDataset` (all of them are larger than the length of input sentences), the loss should be the same. But, it wasn't. 

You can compare the result by uncommenting `L86`.



```py
from pytorch_lightning import seed_everything
from transformers import AdamW, T5ForConditionalGeneration, T5Tokenizer, AutoConfig
from torch.utils.data import Dataset
from torch.utils.data import DataLoader

model = T5ForConditionalGeneration.from_pretrained('t5-base')
model = model.cuda()
model.train()
tokenizer = T5Tokenizer.from_pretrained('t5-base')
seed_everything(42)

class MyDataset(Dataset):
    def __init__(self, tokenizer, raw_inputs, raw_targets, max_input_len=128, max_output_len=128):
        self.max_input_len = max_input_len
        self.max_output_len = max_output_len
        self.tokenizer = tokenizer

        self.inputs = []
        self.targets = []

        self._build_examples(raw_inputs, raw_targets)

    def __len__(self):
        return len(self.inputs)

    def __getitem__(self, index):
        source_ids = self.inputs[index]["input_ids"].squeeze()
        target_ids = self.targets[index]["input_ids"].squeeze()

        src_mask = self.inputs[index]["attention_mask"].squeeze()  # might need to squeeze
        target_mask = self.targets[index]["attention_mask"].squeeze()  # might need to squeeze

        return {"source_ids": source_ids, "source_mask": src_mask, 
                "target_ids": target_ids, "target_mask": target_mask}

    def _build_examples(self, raw_inputs, raw_targets):


        for i in range(len(raw_inputs)):
            # change input and target to two strings
            input = ' '.join(raw_inputs[i])
            target = raw_targets[i]

            tokenized_input = self.tokenizer.batch_encode_plus(
              [input], max_length=self.max_input_len, padding="max_length",
              truncation=True, return_tensors="pt"
            )
            tokenized_target = self.tokenizer.batch_encode_plus(
              [target], max_length=self.max_output_len, padding="max_length",
              truncation=True, return_tensors="pt"
            )

            self.inputs.append(tokenized_input)
            self.targets.append(tokenized_target)


no_decay = ["bias", "LayerNorm.weight"]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
        "weight_decay": 0,
    },
    {
        "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
        "weight_decay": 0.0,
    },
]
optimizer = AdamW(optimizer_grouped_parameters, lr=3e-4, eps=1e-8)


# toy dataset
raw_inputs = ["can't wait wait for my next visit.",
"their sake list was extensive, but we were looking for purple haze, which wasn't listed but made for us upon request!",
"the spicy tuna roll was unusually good and the rock shrimp tempura was awesome, great appetizer to share!",
"we love th pink pony."
]

raw_targets = ['restaurant general is great because it is NULL',
"drinks style options is great because sake list is extensive [SSEP] service general is great because it is NULL",
"food quality is great because spicy tuna roll is good [SSEP] food quality is great because rock shrimp tempura is awesome",
"restaurant general is great because pink pony is love"
]

# max_input_len is max length of input text.
train_dataset = MyDataset(tokenizer, raw_inputs, raw_targets, max_input_len=180, max_output_len=128)
# train_dataset = MyDataset(tokenizer, raw_inputs, raw_targets, max_input_len=200, max_output_len=128)
dataloader = DataLoader(train_dataset, batch_size=4,
                                drop_last=True, shuffle=True, num_workers=4)
for idx, batch in enumerate(dataloader):
    batch = {k:v.cuda() for k,v in batch.items()}
    lm_labels = batch["target_ids"]
    lm_labels[lm_labels[:, :] == tokenizer.pad_token_id]  = -100
    optimizer.zero_grad()
    outputs = model(
        input_ids=batch["source_ids"],
        attention_mask=batch["source_mask"],
        labels=lm_labels,
        # decoder_attention_mask=batch['target_mask']
    )
    print(f'{idx}: {outputs[0]}')
    if idx>1:
        break
    # loss = outputs[0]
    # loss.backward()
    # optimizer.step()
    # print(f'{idx}:{loss}')
```




>  I don't think you get my point

I didn't sorry, I work mostly on inference so I assumed performance was speed performance, not model performance (loss).

Thanks very much for the simpler script !

Unfortunately I am not really versed in that part of the library.
Thanks to your script I could test a little, and it seems the loss will change even without modifying the script when I remove the seeding, couldn't it be something like the random dropouts are different if you change the `max_length` ?
I tried moving the model to `.eval()` but the difference is still there.

Edit: I looked at the actual `input_ids` and if you look there are some non padded values after `180` for the last input in the batch. I you use `batch_size=3` and `.eval()` then I guess the same loss (within float approximation).
Can you confirm this works for you ?

Thanks for your reply. I am so sorry that I made a mistake in `_build_examples`. 

Now, I create a **new** [gist](https://gist.github.com/SinclairCoder/3eee9d1cd78e81745de515ec594e6e2c) to test this problem. 

I found that when using `T5` different `max_input_len`, the loss is the same. However, when using `BART` under the same condition, the loss is different when different `max_input_len` specified. I think maybe they should be the same.

I'm totally confused. Wish your reply soon.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


@patrickvonplaten  @Narsil   Looking forward to your reply.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @SinclairCoder,

We sadly don't have the time to analyse costume code. We try to keep Transformers issues for bugs and questions related only to existing features. Could you try to use the forum instead: https://discuss.huggingface.co/ ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.