RoBERTa: Truncation error: Sequence to truncate too short to respect the provided max_length
Author: PremalMatalia
Pull request: no

## Environment info

- `transformers` version: 4.9.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): TPU
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Error is coming with both GPU and TPU
- Using distributed or parallel set-up in script?: No

### Who can help
Models:
RoBERTa - @LysandreJik, @patrickvonplaten, @patil-suraj, 

Library:
- tokenizers: @LysandreJik

## Information

Model I am using RoBERTa model for SQuAD 2.0 and getting below error when trying to tokenize the Question and context pair:

The problem arises when using:
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SQuAD 2.0

## To reproduce

Steps to reproduce the behavior:
I am trying to tokenize SQuAD 2.0 dataset using roberta-base tokenizer and model but it has started giving me below error.
This code snippet was working till few days before and now it is giving below error without changing anything.

```Python
  model_args = ModelArguments(
    model_checkpoint=model_checkpoint,
    token_checkpoint=token_checkpoint,
    squad_v2=True,
    max_length=384,
    doc_stride=128,
    batch_size=8,
    n_best_size=25,
    max_answer_length=30,
    min_null_score=7.0, ##FOR ROBERTa
    NA_threshold=-3,
    pad_side="right")

    token_checkpoint = "roberta-base"
    model_checkpoint= "roberta-base"

   tokenizer = AutoTokenizer.from_pretrained(token_checkpoint)
   model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint,        
                                                        attention_probs_dropout_prob=0.2,
                                                        hidden_dropout_prob=0.2)

    datasets = load_dataset("squad_v2" if model_args.squad_v2 else "squad")

    tokenized_examples = tokenizer(
        datasets["question" if model_args.pad_side else "context"],
        datasets["context" if model_args.pad_side else "question"],
        truncation="only_second" if model_args.pad_side else "only_first",
        max_length=model_args.max_length,
        stride=model_args.doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )


```

**_ERROR messages:_**
Truncation error: Sequence to truncate too short to respect the provided max_length
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "<ipython-input-14-3842fd6863c2>", line 75, in pipeline
    tokenized_datasets = datasets.map(prepare_train_features, batched=True, batch_size=1000,remove_columns=datasets["train"].column_names)
  File "/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py", line 489, in map
    for k, dataset in self.items()
  File "/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py", line 489, in <dictcomp>
    for k, dataset in self.items()
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 1679, in map
    desc=desc,
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 185, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 2014, in _map_single
    offset=offset,
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 1900, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "<ipython-input-6-54e98dcfc55e>", line 14, in prepare_train_features
    padding="max_length",
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 2385, in __call__
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 2570, in batch_encode_plus
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/tokenization_gpt2_fast.py", line 163, in _batch_encode_plus
    return super()._batch_encode_plus(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_fast.py", line 408, in _batch_encode_plus
    is_pretokenized=is_split_into_words,
Exception: Truncation error: Sequence to truncate too short to respect the provided max_length

## Expected behavior

SQuAD 2.0 dataset should be tokenized without any issue.


Comments:
After further analysis, I could see that RoBERTa tokenizer is not able to handle a question in SQuAD 2.0 dataset at index "107709" due to lot of blank spaces at the start of the question and it's length is 25651 character. 
While other tokenizers are able to handle this.


```python
print("question length | 107709:",len(dataset[107709]['question']))
print("context | 107709:",dataset[107709]['question'])
```

### Output
question length | 107709: 25651
context | 107709:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  What radiates two lobes perpendicular to the antennas axis?                                                                                                                                       


I just started running into this late last week in an internal test.
Is this new? Has something changed ? 

just happened to me as well on SQuAD1.1


The change is due to https://github.com/huggingface/datasets/pull/2586 which changed the SQUAD dataset. The failure is normal in the sense that the tokenizer is asked to truncate tokens from the second sentence (context) when it's actually the first one (question) that is too long. Removing the whitespace at the beginning of the question fixes this (this is why it doesn't happen with a BERT tokenizer, because the BERT tokenizer does it, the roberta tokenizer leaves all individual spaces however).

I have fixed the example notebook and the PR mentioned above shows how to fix it in the example scripts.

Thanks for fixing this issue. 

I very much appreciate this thread for helping me resolve this problem when it happened to me, too. I just wanted to make others aware that there is still an example notebook that will result in this error if it is used with roBERTa.

[](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb)

The correct code can be found here: [](https://huggingface.co/course/chapter7/7)
preprocess_training_examples
and
preprocess_validation_examples
which include a line to strip the leading whitespace from the question before tokenization.


Hi,
Thank you for the information, I used **BERT** model, and my questions are a lot longer than answers, even after the removing white space that the code will do, I got the same error, do you know how to fix it?

Yes, the issue still remains. I tried processing another dataset and it gave this issue with/without batching,.. I don't know what the workaround is either.