Make tokenizer.pad() compatible with labels
Author: BramVanroy
Pull request: no

### Feature request

In my project, I need to do some custom sorting and filtering in my sampler. Therefore, each item in my Dataset is already tokenized/truncated (and not padded). So the padding occurs in the `collate_fn` of the dataloader. Here I would need to pad both inputs and labels. While this is straightforward to do with `tokenizer.pad()` for the inputs (input_ids, attention_mask) I cannot get this to work for labels.

This first example just tries to put two samples (dixtionaries with keys input_ids, attention_mask, and labels) through tokenizer.pad. I would not expect this to work because in seq2seq, inputs and ouputs can have different lengths so the total max. sequence length will differ.

```
import torch
from transformers import MBartTokenizer

samples = [{'input_ids': torch.LongTensor([8622, 621, 5941, 2750, 765, 10, 10422, 111, 104687,
                                           27771, 4, 92136, 538, 100244, 3642, 8966, 85493, 4,
                                           53927, 621, 24911, 5245, 552, 49725, 4, 398, 621,
                                           1053, 26255, 13081, 34, 59207, 4, 87, 73275, 13,
                                           110, 2661, 9, 36904, 297, 65842, 7, 5, 2,
                                           250004]),
            'attention_mask': torch.LongTensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
                                                1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]),
            'labels': torch.LongTensor([250132, 250147, 250122, 250132, 250092, 5941, 250030, 250148, 250132,
                                        10422, 250104, 250031, 250132, 104687, 27771, 250057, 250135, 250132,
                                        250057, 250057, 250057, 250057, 2, 250004])},
           {'input_ids': torch.LongTensor([20625, 32692, 34475, 1821, 5792, 5941, 182417, 32, 4,
                                           136201, 4, 201, 1530, 4, 7911, 2, 250004]),
            'attention_mask': torch.LongTensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]),
            'labels': torch.LongTensor([250132, 250147, 250122, 250132, 8337, 250104, 250030, 250132, 32692,
                                        250057, 250031, 250132, 199, 1681, 250031, 250148, 250132, 765,
                                        250132, 250145, 250057, 250057, 250123, 250132, 136, 250072, 136201,
                                        250073, 201, 1530, 250074, 7911, 250057, 250057, 2, 250004])}]

tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-cc25")
padded = tokenizer.pad(samples,
                              padding=True,
                              return_tensors="pt")
```

Error:

> Traceback (most recent call last):
>   File "transformers\tokenization_utils_base.py", line 715, in convert_to_tensors
>     tensor = as_tensor(value)
> ValueError: expected sequence of length 24 at dim 1 (got 36)
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "scratch_2.py", line 43, in <module>
>     padded = tokenizer.pad(samples,
>   File "transformers\tokenization_utils_base.py", line 2985, in pad
>     return BatchEncoding(batch_outputs, tensor_type=return_tensors)
>   File "transformers\tokenization_utils_base.py", line 210, in __init__
>     self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
>   File "transformers\tokenization_utils_base.py", line 731, in convert_to_tensors
>     raise ValueError(
> ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

So I figured I would just split up my data into the inputs and the labels, but that also does not work:

```python
import torch
from transformers import MBartTokenizer

samples = [{'input_ids': torch.LongTensor([8622, 621, 5941, 2750, 765, 10, 10422, 111, 104687,
                                           27771, 4, 92136, 538, 100244, 3642, 8966, 85493, 4,
                                           53927, 621, 24911, 5245, 552, 49725, 4, 398, 621,
                                           1053, 26255, 13081, 34, 59207, 4, 87, 73275, 13,
                                           110, 2661, 9, 36904, 297, 65842, 7, 5, 2,
                                           250004]),
            'attention_mask': torch.LongTensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
                                                1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]),
            'labels': torch.LongTensor([250132, 250147, 250122, 250132, 250092, 5941, 250030, 250148, 250132,
                                        10422, 250104, 250031, 250132, 104687, 27771, 250057, 250135, 250132,
                                        250057, 250057, 250057, 250057, 2, 250004])},
           {'input_ids': torch.LongTensor([20625, 32692, 34475, 1821, 5792, 5941, 182417, 32, 4,
                                           136201, 4, 201, 1530, 4, 7911, 2, 250004]),
            'attention_mask': torch.LongTensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]),
            'labels': torch.LongTensor([250132, 250147, 250122, 250132, 8337, 250104, 250030, 250132, 32692,
                                        250057, 250031, 250132, 199, 1681, 250031, 250148, 250132, 765,
                                        250132, 250145, 250057, 250057, 250123, 250132, 136, 250072, 136201,
                                        250073, 201, 1530, 250074, 7911, 250057, 250057, 2, 250004])}]

tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-cc25")
inputs = [{k: v for k, v in sample.items() if k in ["attention_mask", "input_ids"]} for sample in samples]
padded_inputs = tokenizer.pad(inputs,
                              padding=True,
                              return_tensors="pt")

labels = [{"labels": sample["labels"]} for sample in samples]
padded_labels = tokenizer.pad(labels,
                              padding=True,
                              return_tensors="pt")
```

Error:

> Traceback (most recent call last):
>   File "scratch_2.py", line 49, in <module>
>     padded_labels = tokenizer.pad(labels,
>   File "transformers\tokenization_utils_base.py", line 2904, in pad
>     raise ValueError(
> ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['labels']


My assumption is that the intention for `pad` has always been to only pad inputs and not labels but I might just be missing something. 

### Motivation

Unless I am missing something, it is currently not straightforward to use tokenizer.pad() on labels.

### Your contribution

I'd need some guidance on what needs to change to make a tokenizer's `.pad` work with labels. If I know what to change, I can make a PR.

Comments:
There is no canonical way to pad labels directly using the tokenizer, since what we want for padding depends on the task (we don't want any padding in sentence classification, but a different one in token classification, summarization or translation) and the tokenizer is not aware of the task.

In your code sample, the easiest way is just to replace the name `"labels"` by `"input_ids"` in your call to pad. As shown in all examples, the canonical way is to just do everything in `__call__`, or you can use the data collators to help as well, since they contain the code to pad adapted to the task at hand.

That makes a lot of sense! I've been spending too much time on translation-related topics that I forgot that this is not as straightforward as it seems. Perhaps in a perfect world, the tokenizer (+ model) would be task-aware. Just like some tokenizers can switch between a source/target language, they could then switch between tasks and therefore appropriate padding techniques. Maybe that would make automatic integrations with pipelines easy, too! This is just me dreaming - I am aware that there are probably billions of reasons not to implement it like that. :-)