Decoding error while using DataCollatorForSeq2Seq
Author: Pavloveuge
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no>

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I decode data from DataCollatorForSeq2Seq, I get OverflowError with fast tokenizer and TypeError with default tokenizer.

Code example:
```
model_name = "facebook/bart-base"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
data_collator = DataCollatorForSeq2Seq(tokenizer, model=None)

texts = ["text" * 5, "text" * 10]
labels = ["label" * 5, "label" * 10]
features = [
    {
        "input_ids": tokenizer(i)['input_ids'],
        "labels": tokenizer(j)['input_ids']
    }
    for i,j in zip(texts, labels) 
]
result = data_collator(features)
print(result["labels"][0])
print(tokenizer.decode(result["labels"][0], skip_special_tokens=True))
```

Stack trace in case `use_fast=False`:
```
TypeError                                 Traceback (most recent call last)
[<ipython-input-7-14a1328c21fe>](https://localhost:8080/#) in <cell line: 18>()
     16 result = data_collator(features)
     17 print(result["labels"][0])
---> 18 print(tokenizer.decode(result["labels"][0], skip_special_tokens=True))

2 frames
[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py](https://localhost:8080/#) in decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
   3507         token_ids = to_py_obj(token_ids)
   3508 
-> 3509         return self._decode(
   3510             token_ids=token_ids,
   3511             skip_special_tokens=skip_special_tokens,

[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py](https://localhost:8080/#) in _decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, spaces_between_special_tokens, **kwargs)
    947                 current_sub_text.append(token)
    948         if current_sub_text:
--> 949             sub_texts.append(self.convert_tokens_to_string(current_sub_text))
    950 
    951         if spaces_between_special_tokens:

[/usr/local/lib/python3.10/dist-packages/transformers/models/bart/tokenization_bart.py](https://localhost:8080/#) in convert_tokens_to_string(self, tokens)
    305     def convert_tokens_to_string(self, tokens):
    306         """Converts a sequence of tokens (string) in a single string."""
--> 307         text = "".join(tokens)
    308         text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
    309         return text

TypeError: sequence item 9: expected str instance, NoneType found
```

Stack trace in case `use_fast=True`:
```
OverflowError                             Traceback (most recent call last)
[<ipython-input-8-d0724246272d>](https://localhost:8080/#) in <cell line: 18>()
     16 result = data_collator(features)
     17 print(result["labels"][0])
---> 18 print(tokenizer.decode(result["labels"][0], skip_special_tokens=True))

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py](https://localhost:8080/#) in decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
   3507         token_ids = to_py_obj(token_ids)
   3508 
-> 3509         return self._decode(
   3510             token_ids=token_ids,
   3511             skip_special_tokens=skip_special_tokens,

[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py](https://localhost:8080/#) in _decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
    544         if isinstance(token_ids, int):
    545             token_ids = [token_ids]
--> 546         text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
    547 
    548         clean_up_tokenization_spaces = (

OverflowError: out of range integral type conversion attempted
```


Also, if I use facebook/m2m100_418M there are no errors, but the result of decoding looks like (though I use `skip_special_tokens=True`) this:
```
labellabellabellabellabel<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
```

### Expected behavior

Hello! I have expected no errors and that skip_special_tokens would work normally. Seems like the label padding in DataCollatorForSeq2Seq with using -100 is leading to error.

I think these issue are related to this:
#22634
[this](https://github.com/huggingface/transformers/issues/3853#issuecomment-770417239)

Comments:
Yes, you need to replace the labels indices that are at -100 to be able to decode them. The -100 indicates to PyTorch the corresponding token should be ignored during the loss computation, this is not a bug.

Also please use the [forums](https://discuss.huggingface.co/) for questions like this.

OK, thank you so much.

Hey @Pavloveuge , I had the same problem  "OverflowError: out of range integral type conversion attempted", How do I solve it?


Hi, look at this [comment](https://github.com/huggingface/transformers/issues/22634#issuecomment-1595885334)

I saw that. I'm finetuning the araT5 model, and I can't understand how to replace -100 in my code. Could help me please? @Pavloveuge 


Can you share minimal code snippet?
I think, you can try:
```
collator = DataCollatorForSeq2Seq(tokenizer, model=model, label_pad_token_id=tokenizer.pad_token_id)
```

I tried it but the error was not solved. 
this screenshot of the error,
![image](https://github.com/huggingface/transformers/assets/53126908/fbb222a6-7403-45fe-819e-41b1d5f1438a)

in this line, the error occurs
![image](https://github.com/huggingface/transformers/assets/53126908/b761417b-0c42-4df1-92a6-798e257cf7a8)
![image](https://github.com/huggingface/transformers/assets/53126908/a56ae390-0aa6-40ff-89ad-444dd19f4b73)
specifically, this line is causing the error, where the preds containing -100, as you can see in the first image
![image](https://github.com/huggingface/transformers/assets/53126908/4df01146-b70c-4048-9cdf-52f2afb61d19)




@Pavloveuge , Also the error occurs in the evaluation and prediction stages

In initialize of your Seq2SeqTrainer you pass `data_collator`, try to initialize him something like this:
```
data_collator = DataCollatorForSeq2Seq(tokenizer, model=model, label_pad_token_id=tokenizer.pad_token_id)
```


Thank you @Pavloveuge, the error is solved by adding ` 
if data_args.ignore_pad_token_for_loss:
            # Replace -100 in the labels as we can't decode them.
            preds = np.where(preds != -100, preds, tokenizer.pad_token_id) ` 
in **_compute_mertic_** function before tokenizer.batch_decode for perds:
![image](https://github.com/huggingface/transformers/assets/53126908/ebe3b40f-2aac-45b8-a660-021c476ffd0c)