LEDForSequenceClassification fine-tuning model gives: IndexError: index out of range in self
Author: darshan2203
Pull request: no
Labels: name: bug

### System Info

transformers - 4.21.1
Python - 3.8.13
torch - 1.12.0+cu113


### Who can help?

@patrickvonplaten

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm trying to fine-tune the LED model for the SequenceClassification task.

It works when I use max_sequence_length = 1024 but it doesn't when it goes beyond these numbers. D

After debugging, I found a similar issue #14312, and tried applying the proposed solution by adding decoder_input_ids even though it doesn't make sense to me that we really need this as an input.

To reproduce:
```
from transformers import LEDTokenizer, LEDForSequenceClassification
  
tokenizer = LEDTokenizer.from_pretrained("allenai/led-base-16384")
model = LEDForSequenceClassification.from_pretrained("allenai/led-base-16384")

# this works (tokens < 1024)
inputs = tokenizer("HuggingFace"*1000, return_tensors="pt")
with torch.no_grad():
      model(**input)

# this does not work! (tokens > 1024)
inputs = tokenizer("HuggingFace"*2048, return_tensors="pt")
with torch.no_grad():
      model(**input)

# this does not work as well! (tokens > 1024)
inputs = tokenizer("HuggingFace"*2048, return_tensors="pt")
inputs['decoder_input_ids'] = inputs['input_ids'][:512]
with torch.no_grad():
      model(**input)
```

_Without adding decoder_input_ids to tokenized inputs,_
It simply complains of Index out of range while calling torch.embedding.

_with decoder_input_ids as suggested in the reference issue above,_
It throws an **IndexError: The Shape of the mask [1, 1026] at index 1 does not match the shape of the indexed tensor  [1, 512, 768] at index 1**


### Expected behavior

As LEDForSequenceClassification should work for a sequence length of more than 1024 tokens.

Comments:
Hey @darshan2203,

Sorry I won't be free anytime soon to look into this issue - @ArthurZucker do you want to give it a try?

Hey! Thanks for the issue 😄 
The proposed solution, `inputs['decoder_input_ids'] = inputs['input_ids'][:512]` is bound to fail as : 
```python 
inputs['input_ids'][:512].shape
torch.Size([1, 6146])
``` 
It also seems that "HuggingFace" is not parsed into a single token (at least when I used `tokenizer = LEDTokenizer.from_pretrained("allenai/led-base-16384")` but rather 3 : `[0, 40710, 3923, 34892, 2]`. 

When I try : 
```python 
inputs = tokenizer("HuggingFace"*(1000//3), return_tensors="pt")
with torch.no_grad():
      model(**inputs)
```
It works as expected, and in that case the shape of the input is : `1001`. 
Hope this will help you! 

> Hey! Thanks for the issue 😄 The proposed solution, `inputs['decoder_input_ids'] = inputs['input_ids'][:512]` is bound to fail as :
> 
> ```python
> inputs['input_ids'][:512].shape
> torch.Size([1, 6146])
> ```
> 
> It also seems that "HuggingFace" is not parsed into a single token (at least when I used `tokenizer = LEDTokenizer.from_pretrained("allenai/led-base-16384")` but rather 3 : `[0, 40710, 3923, 34892, 2]`.
> 
> When I try :
> 
> ```python
> inputs = tokenizer("HuggingFace"*(1000//3), return_tensors="pt")
> with torch.no_grad():
>       model(**inputs)
> ```
> 
> It works as expected, and in that case the shape of the input is : `1001`. Hope this will help you!

Thanks, Arthur for getting back at this. 

I think there is a gap in our understanding. What I reported is that when we have an input tensor of size more than 1024 tokens, it doesn't work. As per the documentation of a LED-base-16384 model, it can take input sequence up to 16384 tokens.

Try this and it won't work:
 
 ```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384")
model = AutoModelForSequenceClassification.from_pretrained("allenai/led-base-16384")

# Basically use any piece of text long enough such that the LED tokenizer tokenizes it such that it yields more than 1024 tokens. 
inputs = tokenizer("Hello"*1500, return_tensors="pt")

print(inputs['input_ids'].shape)
# torch.Size([1, 1502])

with torch.no_grad():
      model(**inputs)
 ```

Error Stack trace - 
```python
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In [11], line 2
      1 with torch.no_grad():
----> 2     model(**inputs)

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/transformers/models/led/modeling_led.py:2543, in LEDForSequenceClassification.forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, head_mask, decoder_head_mask, cross_attn_head_mask, encoder_outputs, global_attention_mask, inputs_embeds, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
   2538 if input_ids is None and inputs_embeds is not None:
   2539     raise NotImplementedError(
   2540         f"Passing input embeddings is currently not supported for {self.__class__.__name__}"
   2541     )
-> 2543 outputs = self.led(
   2544     input_ids,
   2545     attention_mask=attention_mask,
   2546     decoder_input_ids=decoder_input_ids,
   2547     decoder_attention_mask=decoder_attention_mask,
   2548     global_attention_mask=global_attention_mask,
   2549     head_mask=head_mask,
   2550     decoder_head_mask=decoder_head_mask,
   2551     cross_attn_head_mask=cross_attn_head_mask,
   2552     encoder_outputs=encoder_outputs,
   2553     inputs_embeds=inputs_embeds,
   2554     decoder_inputs_embeds=decoder_inputs_embeds,
   2555     use_cache=use_cache,
   2556     output_attentions=output_attentions,
   2557     output_hidden_states=output_hidden_states,
   2558     return_dict=return_dict,
   2559 )
   2560 hidden_states = outputs[0]  # last hidden state
   2562 eos_mask = input_ids.eq(self.config.eos_token_id)

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/transformers/models/led/modeling_led.py:2263, in LEDModel.forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, head_mask, decoder_head_mask, cross_attn_head_mask, encoder_outputs, global_attention_mask, past_key_values, inputs_embeds, decoder_inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   2255     encoder_outputs = LEDEncoderBaseModelOutput(
   2256         last_hidden_state=encoder_outputs[0],
   2257         hidden_states=encoder_outputs[1] if len(encoder_outputs) > 1 else None,
   2258         attentions=encoder_outputs[2] if len(encoder_outputs) > 2 else None,
   2259         global_attentions=encoder_outputs[3] if len(encoder_outputs) > 3 else None,
   2260     )
   2262 # decoder outputs consists of (dec_features, past_key_value, dec_hidden, dec_attn)
-> 2263 decoder_outputs = self.decoder(
   2264     input_ids=decoder_input_ids,
   2265     attention_mask=decoder_attention_mask,
   2266     encoder_hidden_states=encoder_outputs[0],
   2267     encoder_attention_mask=attention_mask,
   2268     head_mask=decoder_head_mask,
   2269     cross_attn_head_mask=cross_attn_head_mask,
   2270     past_key_values=past_key_values,
   2271     inputs_embeds=decoder_inputs_embeds,
   2272     use_cache=use_cache,
   2273     output_attentions=output_attentions,
   2274     output_hidden_states=output_hidden_states,
   2275     return_dict=return_dict,
   2276 )
   2278 if not return_dict:
   2279     return decoder_outputs + encoder_outputs

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/transformers/models/led/modeling_led.py:2070, in LEDDecoder.forward(self, input_ids, attention_mask, global_attention_mask, encoder_hidden_states, encoder_attention_mask, head_mask, cross_attn_head_mask, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   2067     encoder_attention_mask = _expand_mask(encoder_attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1])
   2069 # embed positions
-> 2070 positions = self.embed_positions(input_shape, past_key_values_length)
   2072 hidden_states = inputs_embeds + positions
   2073 hidden_states = self.layernorm_embedding(hidden_states)

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/transformers/models/led/modeling_led.py:125, in LEDLearnedPositionalEmbedding.forward(self, input_ids_shape, past_key_values_length)
    121 bsz, seq_len = input_ids_shape[:2]
    122 positions = torch.arange(
    123     past_key_values_length, past_key_values_length + seq_len, dtype=torch.long, device=self.weight.device
    124 )
--> 125 return super().forward(positions)

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/torch/nn/modules/sparse.py:158, in Embedding.forward(self, input)
    157 def forward(self, input: Tensor) -> Tensor:
--> 158     return F.embedding(
    159         input, self.weight, self.padding_idx, self.max_norm,
    160         self.norm_type, self.scale_grad_by_freq, self.sparse)

File ~/opt/anaconda3/envs/hf_env/lib/python3.8/site-packages/torch/nn/functional.py:2199, in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2193     # Note [embedding_renorm set_grad_enabled]
   2194     # XXX: equivalent to
   2195     # with torch.no_grad():
   2196     #   torch.embedding_renorm_
   2197     # remove once script supports set_grad_enabled
   2198     _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2199 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

IndexError: index out of range in self

```

Oh right! Sorry will have a look asap 🤗

Hey! So as mentioned in the [issue](https://github.com/huggingface/transformers/issues/14312#issuecomment-968768138) you linked, the decoder's max input length is `1024`, and if the decoder input_ids are not provided, the model uses by default a shifted version of the `input_ids` which in this case are too long. Even if you provide the `decoder_input_ids`, the sentence representation used the `eos_tokens` from the `input_ids` as it was copy pasted from BART. We can just switch that to using the `decoder_input_ids` but there are no checkpoints and it is just a hack because the model introduced in LongFormer for text classification is a decoder only model. 

I dived a bit too deep and I am now wondering why are you using this `seq2seq` model for a task that is rather suited for the `Encoder` only model. It seems that the original implementation uses `Longformer` encoder for text classification. You can find  text classification pre-trained model [here](https://huggingface.co/models?other=longformer&pipeline_tag=text-classification&sort=downloads)

Since there are no checkpoints for `LEDForSequenceClassification`, we should probably deprecate its usage or remove it? 
Otherwise, we have to use the shifted decoder inputs, which should not include breaking changes.

WDYT @patrickvonplaten 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.