BigBird tokenizer - special tokens not being masked during MLM
Author: prajwal967
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.17.0
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- PyTorch version (GPU?): 1.10.0 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@ydshieh

The BigBird tokenizer doesn't seem to have the special token mask operation working correctly. Only the [CLS] and [SEP] tokens are being treated as special tokens by the get_special_tokens_mask function


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. `tokenizer = AutoTokenizer.from_pretrained('google/bigbird-roberta-base')`
2. Print the special token ids: `tokenizer.all_special_ids` - This prints out: `[1, 2, 100, 66, 0, 65, 67]`
3. Get the special tokens mask: `tokenizer.get_special_tokens_mask([1, 2, 100, 66, 0, 65, 67], already_has_special_tokens=True)` - There seems to be a bug in this step. The returned mask: `[0, 0, 0, 1, 0, 1, 0]`

### Expected behavior

```shell
At step 3 the returned special tokens mask should have the value 1 at every position,
but it seems to have the value 1 only for the ids 65 & 66.
I followed similar steps for the Roberta base tokenizer and it worked as expected.
Is there something wrong that I'm doing?
```


Comments:
Thank you reporting this, @prajwal967 

@SaulLu @Narsil 

I tried with the following example (adapted from @prajwal967 ), and it looks like this issue happens for `BigBirdTokenizerFast`, but not `BigBirdTokenizer`. 

Could you take a look on this issue, please? Are you aware of this issue (potentially for other tokenizers)?

```python
from transformers import AutoTokenizer, BigBirdTokenizer, BigBirdTokenizerFast

print("tokenizer from auto")
auto_tokenizer = AutoTokenizer.from_pretrained('google/bigbird-roberta-base')
print(f"special token ids: {auto_tokenizer.all_special_ids}")
mask = auto_tokenizer.get_special_tokens_mask([1, 2, 100, 66, 0, 65, 67], already_has_special_tokens=True)
print(f"special_tokens_mask: {mask}")
print("=" * 40)

print("bigbird_tokenizer_slow")
bigbird_tokenizer_slow = BigBirdTokenizer.from_pretrained('google/bigbird-roberta-base')
print(f"special token ids: {bigbird_tokenizer_slow.all_special_ids}")
mask = bigbird_tokenizer_slow.get_special_tokens_mask([1, 2, 100, 66, 0, 65, 67], already_has_special_tokens=True)
print(f"special_tokens_mask: {mask}")
print("=" * 40)

print("bigbird_tokenizer_fast")
bigbird_tokenizer_fast = BigBirdTokenizerFast.from_pretrained('google/bigbird-roberta-base')
print(f"special token ids: {bigbird_tokenizer_fast.all_special_ids}")
mask = bigbird_tokenizer_fast.get_special_tokens_mask([1, 2, 100, 66, 0, 65, 67], already_has_special_tokens=True)
print(f"special_tokens_mask: {mask}")
print("=" * 40)
```

Good for `RobertaTokenizerFast`

```python
from transformers import AutoTokenizer, RobertaTokenizer, RobertaTokenizerFast

print("tokenizer from auto")
auto_tokenizer = AutoTokenizer.from_pretrained('roberta-base')
print(f"special token ids: {auto_tokenizer.all_special_ids}")
mask = auto_tokenizer.get_special_tokens_mask([0, 2, 3, 1, 50264], already_has_special_tokens=True)
print(f"special_tokens_mask: {mask}")
print("=" * 40)

print("bigbird_tokenizer_slow")
bigbird_tokenizer_slow = RobertaTokenizer.from_pretrained('roberta-base')
print(f"special token ids: {bigbird_tokenizer_slow.all_special_ids}")
mask = bigbird_tokenizer_slow.get_special_tokens_mask([0, 2, 3, 1, 50264], already_has_special_tokens=True)
print(f"special_tokens_mask: {mask}")
print("=" * 40)

print("bigbird_tokenizer_fast")
bigbird_tokenizer_fast = RobertaTokenizerFast.from_pretrained('roberta-base')
print(f"special token ids: {bigbird_tokenizer_fast.all_special_ids}")
mask = bigbird_tokenizer_fast.get_special_tokens_mask([0, 2, 3, 1, 50264], already_has_special_tokens=True)
print(f"special_tokens_mask: {mask}")
print("=" * 40)
```

I wasn't aware of this one! Thanks for pinging me.

However, I am not sure what would be the right behaviour here. To get a signal on how he would like to better standardize these behaviours would it be possible to share with us for which use case you use `get_special_tokens_mask` @prajwal967 ?

The use case I was looking at was to train a masked language model. There seem to be two approaches to ensure special tokens are **not masked**

Approach 1:
- Set the return_special_tokens_mask = True in the tokenization step: https://github.com/huggingface/transformers/blob/66f893320c6a6935668d7de8bff26bd5a35ae042/examples/pytorch/language-modeling/run_mlm.py#L433
- The data collator checks if we have the tensor mask in the input - and it ensures the special tokens are not masked: https://github.com/huggingface/transformers/blob/dcb08b99f44919425f8ba9be9ddcc041af8ec25e/src/transformers/data/data_collator.py#L763

Approach 2:
- Set return_special_tokens_mask = False in the tokenization step. I would assume the only benefit for doing this would be you save some memory at the cost of the next step being slower.
- The data collator now creates the special tokens mask on the fly and it ensures the special tokens are not masked: https://github.com/huggingface/transformers/blob/dcb08b99f44919425f8ba9be9ddcc041af8ec25e/src/transformers/data/data_collator.py#L757-L761

I haven't found any reason to favor *Approach 2* over *Approach 1*. I just happened to come across this issue when I was playing around with the code.

Although, there was a time when I was creating a custom tokenizer and approach 1 wasn't returning the correct mask. It wouldn't return the mask correctly for special tokens within the sequence.
- For example: Input: [CLS, id1, id2, [SEP], id3, SEP], mask: [1, 0, 0, 0, 0, 1]. The value would be 0 for the SEP token within the sequence. 
- But for some reason *Approach 2* worked and would give the mask: [1, 0, 0, 1, 0, 1]. 
- This could just be me not creating the custom tokenizer correctly, I haven't looked further into this!

I haven't found or come across any use case where I know for certain that *Approach 2* would be definitely better. I just happened to come across this issue when I was playing around with the code.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.