Range Error for BERT Masked Language Modeling on IMDB
Author: Jadiker
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1ZpYRkJVMF5r3MukUheEFtgDvqax4YCxM?usp=sharing

### Expected behavior

```shell
Evaluation to complete and give me a perplexity score, as it does [here](https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/chapter7/section3_tf.ipynb)
```


Comments:
Hi @Jadiker 👋  In your notebook, after the `tokenize_and_chunk` cell (the 2nd one, there are 2) we can see a warning that explains the error: `Token indices sequence length is longer than the specified maximum sequence length for this model (521 > 512). Running this sequence through the model will result in indexing errors`.

If you add `truncation=True` in the tokenizer call in that cell you should be able to solve the problem. Let me know if it worked :)

Nope, same error: `indices[15,32] = -9223372036854775808 is not in [0, 28996)`. (I've edited the notebook with the change.)

@Jadiker Thank you for the update :) The problem seems to raise from your custom tokenization function, which is likely not returning the correct data format. See [this](https://colab.research.google.com/drive/1oZ3CMFXOaBZYRVaNlkqEMnQrcwqr7hS_?usp=sharing) notebook, which successfully runs your code if we skip `tokenize_and_chunk`. Inside `tokenize_and_chunk`, you append `chunks.append(all_input_ids[idx: idx + context_length])`, which would explain the indexing errors.

We also reserve these GitHub issues for bugs in the repository and/or feature requests. For any other requests, like issues in your custom code, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗 I'm closing this issue, but feel free to reopen with queries that fit the criteria I described.



@gante Thanks for your time and for the information! I really appreciate it.

Two comments:
1. **All the code (including the `tokenize_and_chunk` function) in the notebook is directly from Hugging Face.** It comes from [this notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/videos/mlm_processing.ipynb) which is linked in [this tutorial](https://www.youtube.com/watch?v=8PmhEIXhBvI) on data processing. The _only_ thing I have done is added code _after_ the data processing in order to actually train a model on the processed data. (And the code for training the model comes from [this Hugging Face tutorial](https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/chapter7/section3_tf.ipynb).)

Given that, should I still have posted on the forum first? If the tutorials for data processing and model training can't be combined, how is one supposed to train a model on the processed data? It seemed like something that should be fixed in the code, rather than just discussed on the forum.

2. I don't believe the notebook you linked to is shared with me.

Thanks again for engaging with this!

> I don't believe the notebook you linked to is shared with me.

Oops, forgot to change the permissions. Should be okay now

After looking at the notebook you linked, it seems like the issue is that the tutorial notebook gives two different options for tokenizing text - by using both of them, rather than just using the first one, I introduced a bug into the code.

Does that sounds accurate?

@Jadiker Yeah, the problem seems to be at the dataset preparation stage. To be candid, I also can't find the issue from a quick glance -- I've double checked the `input_ids`, they are all within `vocabulary_size`, so `gather` shouldn't complain 🤔  Can you have a look at the script example [here](https://github.com/huggingface/transformers/blob/main/examples/tensorflow/language-modeling/run_mlm.py), which was working as of a few weeks ago (and should be working), and see if you can find the issue? My number 1 suspect is the lack of a labels column, but the thrown error does not point at that.

As I mentioned above, we don't have the resources to do proper support in situations like this, but I'd be curious to find the root cause. Perhaps we could improve documentation with the findings :) If you get stuck, I might have capacity to pick it up in a few weeks.