RoBERTa config defaults are inconsistent with fairseq implementation
Author: PheelaV
Pull request: no
Labels: name: bug

### System Info

 python 3.12, transformers 4.14, latest mac os

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import RobertaConfig
my_config = RobertaConfig()
roberta_config = RobertaConfig.from_pretrained("roberta-base")

assert (
  my_config.max_position_embeddings == roberta_config.max_position_embeddings
), "%d %d" % (my_config.max_position_embeddings, roberta_config.max_position_embeddings)

### Expected behavior

The config defaults should correspond the the base model?

This is an implementation detail, but it did send me on a debugging spree as it hid as a sticky CUDA assertion error.
```Assertion `srcIndex < srcSelectDimSize` failed```

The problem is that by default if you create the position_ids yourself or if you let transformers roberta_modelling take care of it (it also does it the way fairseq implemented it), it will create indeces that are out of bounds with the default configuration as everything is shifted by pad_token_id.

This is more of a heads up. Do transformers generally provide defaults aligned with the original models, or are the defaults here meant to be agnostic of that?

Comments:
Hi @PheelaV, thanks for opening this issue! 

In general, our config defaults should match a corresponding implementation, [normally indicated in the docstring](https://github.com/huggingface/transformers/blob/5bcbdff15922b1d0eeb035879630ca61c292122a/src/transformers/models/roberta/configuration_roberta.py#L33), and a config's values should match its corresponding checkpoint when loading using `from_pretrained`. Once a model has been released, especially if it's popular + released a while ago, even if there's differences in the values we might not change the config as this would introduce a change in default behaviour for users. 

> The problem is that by default if you create the position_ids yourself or if you let transformers roberta_modelling take care of it (it also does it the way fairseq implemented it), it will create indeces that are out of bounds with the default configuration as everything is shifted by pad_token_id.

Could you expand on this a little bit - ideally with a code snippet triggering the error? What do you mean by "if you let transformers roberta_modelling take care of it"` here - is this the creation of the embedding layer in the modeling code? 

Hi @amyeroberts ,

Sure, it is simple actually, the default should be 514 and not 512, which gets inherited from BERT (which raises a flag on its own thanks to the difference of wordpiece vs GPT-2 style BPE). 

How I found it: I am doing pre-tokenization and taking care of MLM myself, as well as pre-training my own model. Because of this my first attempt was to create my own config as I saw most of the defaults match the RoBERTa_base, being only a human I overlooked the above.


```python
import torch
from torch.utils.data import DataLoader, Dataset
from transformers import (
  DataCollatorForLanguageModeling,
  RobertaConfig,
  RobertaForMaskedLM,
  RobertaTokenizer,
)

# Initialize tokenizer and model
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

correct_config = RobertaConfig(max_position_embeddings=514)
passing_model = RobertaForMaskedLM(correct_config)
wrong_config = RobertaConfig()  # max_position_embeddings = 512 is the default
failing_model = RobertaForMaskedLM(wrong_config)


# Sample input texts
texts = [
  # Something that will fill the context so that the position_id will cause
  # out of bounds access thanks to create_position_ids_from_input_ids
  "The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>. The cat sits on the <mask>.",
  "I love to eat <mask> for breakfast.",
  "The capital of France is <mask>.",
]


# Create a simple dataset
class SimpleDataset(Dataset):
  def __init__(self, texts, tokenizer):
    self.encodings = tokenizer(texts, truncation=True, padding=True)

  def __len__(self):
    return len(self.encodings["input_ids"])

  def __getitem__(self, idx):
    return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}


# Initialize dataset
dataset = SimpleDataset(texts, tokenizer)

# Initialize data collator
data_collator = DataCollatorForLanguageModeling(
  tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

# Create DataLoader
batch_size = 2
dataloader = DataLoader(
  dataset, batch_size=batch_size, collate_fn=data_collator
)

passing_model.eval()
for batch in dataloader:
  inputs = {k: v.to(failing_model.device) for k, v in batch.items()}
  passing_model(**inputs)

failing_model.eval()
for batch in dataloader:
  inputs = {k: v.to(failing_model.device) for k, v in batch.items()}
  failing_model(**inputs)
```

the [source](https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/models/roberta/modeling_roberta.py#L92)

this is the [culprit](https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/models/roberta/modeling_roberta.py#L1546), which is corresponding to fairseq implementation

and this is the [causation](https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/configuration_roberta.py#L36)

Suspicious, I see the config is supposed to be pulled from [here](https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-config.json), which contains the correct configuration. I guess that is only for Auto classes and `RobertaConfig.from_pretrained("roberta-base")`

> and this is the [causation](https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/configuration_roberta.py#L36)

Thanks for sharing more info! 

I see - I hadn't noticed that you were working from v4.14 - is there a reason for using this version rather than a more recent release? 

There have been two significant updates with using this model: 
* The [config definition ](https://github.com/huggingface/transformers/blob/v4.44.0/src/transformers/models/roberta/configuration_roberta.py) is such that there's a default set of values if someone does `RobertaConfig()`, and using `from_pretrained` for the model, config or autoclasses will load based [on the files on the hub](https://huggingface.co/FacebookAI/roberta-base/blob/e2da8e2f811d1448a5b465c236feacd80ffbac7b/config.json#L14). 
* We updated certain model checkpoints e.g. "gpt2" to match the `organization/checkpoint` pattern for checkpoints on the hub. You can still load with `"roberta-base"` for backwards compatibility, but the updated path to be used should be FacebookAI/roberta-base 

Hi, I just tested on latest transformers and it still fails.

In the config you posted, the correct `"max_position_embeddings": 514` is present.

```python
rob_conf = RobertaConfig()
print(rob_conf.max_position_embeddings)
```

Yields 512, which is wrong.

@PheelaV I wouldn't say it fails - but rather the defaults aren't perfectly aligned with the roberta-base checkpoint. 

The default for the `RobertaConfig` [class is documented as 512](https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/models/roberta/configuration_roberta.py#L59). 

If you load from the specific checkpoint then the default is correct: 

As mentioned above, for older, popular models we won't update the defaults for backwards compatibility reasons. However, you will get aligned values if you load both the tokenizer and config from the same checkpoint. 

```py
In [12]: from transformers import RobertaConfig, RobertaTokenizer
    ...:
    ...: checkpoint = "roberta-base"
    ...: tokenizer = RobertaTokenizer.from_pretrained(checkpoint)
    ...: config = RobertaConfig.from_pretrained(checkpoint)
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████| 25.0/25.0 [00:00<00:00, 78.9kB/s]
vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 899k/899k [00:00<00:00, 3.94MB/s]
merges.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 3.01MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████| 1.36M/1.36M [00:00<00:00, 3.53MB/s]
/Users/amyroberts/code/transformers/src/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(

In [13]: config.max_position_embeddings
Out[13]: 514
```

Ok, this would itch me, but I see why you are hesitant in changing it. To summarize:

1) Keep the current default config, that is only used when someone does not use `.from_pretrained` and requires additional modification to make it work with other default values. So someone who used in this fashion, has had to change it either way unless their context is an exotic 510 tokens.
2) Correct it to 514, which will prevent anyone adventurous from stumbling into this in the future and likely won't break existing users with the typical 512 context as they have had to changed it in the first place or load `.from_pretrained` configuration so it does not affect them.

IMO 1) < 2), either way, I have done my duty here. Feel free to close.