Tensor size mismatch in RoBERTa
Author: d-miketa
Pull request: no

The following error pops up while running a `TranslationPipeline` using a PyTorch `EncoderDecoderModel` (@patrickvonplaten) consisting of two `RoBERTas`. (@LysandreJik) Curiously it happens on a very specific datapoint in a large-ish dataset, but I'm having trouble digging it out. (It does well on tens of thousands of examples prior to that though.) I think it's the same issue as https://github.com/microsoft/CodeBERT/issues/73, but I don't know how to go about fixing it. Many thanks for any pointers! I'm using `transformers==4.18.0` and the same issue was present on `4.17`, too.

```python
File ~/my-repo/.venv/lib/python3.9/site-packages/transformers/pipelines/text2text_generation.py:159, in Text2TextGenerationPipeline._forward(self, model_inputs, **generate_kwargs)
    157 generate_kwargs["max_length"] = generate_kwargs.get("max_length", self.model.config.max_length)
    158 self.check_inputs(input_length, generate_kwargs["min_length"], generate_kwargs["max_length"])
--> 159 output_ids = self.model.generate(**model_inputs, **generate_kwargs)
    160 out_b = output_ids.shape[0]
    161 if self.framework == "pt":

File ~/my-repo/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py:27, in _DecoratorContextManager.__call__.<locals>.decorate_context(*args, **kwargs)
     24 @functools.wraps(func)
     25 def decorate_context(*args, **kwargs):
     26     with self.clone():
---> 27         return func(*args, **kwargs)

File ~/my-repo/.venv/lib/python3.9/site-packages/transformers/generation_utils.py:1156, in GenerationMixin.generate(self, inputs, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, typical_p, repetition_penalty, bad_words_ids, force_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, max_new_tokens, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, logits_processor, stopping_criteria, constraints, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, synced_gpus, exponential_decay_length_penalty, **model_kwargs)
   1149     model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
   1150         inputs_tensor, pad_token_id, eos_token_id
   1151     )
   1153 if self.config.is_encoder_decoder and "encoder_outputs" not in model_kwargs:
   1154     # if model is encoder decoder encoder_outputs are created
   1155     # and added to `model_kwargs`
-> 1156     model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(
   1157         inputs_tensor, model_kwargs, model_input_name
   1158     )
   1160 # 4. Prepare `input_ids` which will be used for auto-regressive generation
   1161 if self.config.is_encoder_decoder:

File ~/my-repo/.venv/lib/python3.9/site-packages/transformers/generation_utils.py:524, in GenerationMixin._prepare_encoder_decoder_kwargs_for_generation(self, inputs_tensor, model_kwargs, model_input_name)
    522 encoder_kwargs["return_dict"] = True
    523 encoder_kwargs[model_input_name] = inputs_tensor
--> 524 model_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs)
    526 return model_kwargs

File ~/my-repo/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/my-repo/.venv/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py:817, in RobertaModel.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    815 if hasattr(self.embeddings, "token_type_ids"):
    816     buffered_token_type_ids = self.embeddings.token_type_ids[:, :seq_length]
--> 817     buffered_token_type_ids_expanded = buffered_token_type_ids.expand(batch_size, seq_length)
    818     token_type_ids = buffered_token_type_ids_expanded
    819 else:

RuntimeError: The expanded size of the tensor (746) must match the existing size (514) at non-singleton dimension 1.  Target sizes: [8, 746].  Tensor sizes: [1, 514]
``` 


Comments:
I found out what it was:
```
Token indices sequence length is longer than the specified maximum sequence length for this model (746 > 512). Running this sequence through the model will result in indexing errors
```
It would've been easier to diagnose if whatever triggered this message had also emitted an explicit error, I think.

Wrapping my loop in `try:` and `except RuntimeError:` allowed me to skip this problematic datapoint even without filtering the dataset based on input sequence length.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Getting the same error here. I think that the tokenizer is not truncating correctly, that would be a bug, no?

Getting the same error here.Have you solved this problem?