`eos_mask` is possibly supposed to be taken from `decoder_input_ids`
Author: JamesDeAntonis
Pull request: no

I'm trying to use `BartForSequenceClassification`, and I keep getting a shape mismatch error:
<img width="1020" alt="Screen Shot 2021-11-11 at 3 19 39 PM" src="https://user-images.githubusercontent.com/33379057/141363523-0a919bb1-3935-45b5-8f45-5ab420751ba4.png">
or

I think the reason for the error is that the `eos_mask` is [errantly constructed](https://github.com/huggingface/transformers/blob/v4.10.0/src/transformers/models/bart/modeling_bart.py#L1452) using the `input_ids`, while it should have been constructed with the `decoder_input_ids` whose shape matches the hidden states from the decoder.

I think the solution would simply be to change that line to `eos_mask = decoder_input_ids.eq(self.config.eos_token_id)`

Comments:
cc @patrickvonplaten @patil-suraj 

I think another problem if using finding eos in decoder_input_ids is like this:
When eos_token_id is 2:
decoder_input_ids = tensor([ 0,  7,  6, 13,  4,  7, 13, 14, 21,  7, 13,  5, 14, 20])
labels = tensor([ 7,  6, 13,  4,  7, 13, 14, 21,  7, 13,  5, 14, 20,  2])

Eos token could be removed after shifting, ans could not be found in decoder_input_ids.

Hey @JamesDeAntonis,

Could you provide a code snippet that reproduces the error? Thank you! :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi, revisiting this.  Here's a snippet:

```python
from transformers import AutoTokenizer, BartForSequenceClassification

model_name = "facebook/bart-large"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = BartForSequenceClassification.from_pretrained(model_name)

inputs = tokenizer(["These are the encoder inputs"], return_tensors="pt")
inputs["decoder_input_ids"] = tokenizer(
    ["These are the decoder inputs which have a difference shape from the encoder inputs"], return_tensors="pt"
).input_ids

print(f"Notice how {inputs['input_ids'].shape=} and {inputs['decoder_input_ids'].shape=}")

model(**inputs)
```

This gives
```
Notice how inputs['input_ids'].shape=torch.Size([1, 8]) and inputs['decoder_input_ids'].shape=torch.Size([1, 18])
Traceback (most recent call last):
  File "/workspace/cortx-models/src/cortxmodels/metrics/fluency_metric/sentence/scripts/tmp.py", line 16, in <module>
    model(**inputs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jamie/.local/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1518, in forward
    sentence_representation = hidden_states[eos_mask, :].view(hidden_states.size(0), -1, hidden_states.size(-1))[
IndexError: The shape of the mask [1, 8] at index 1 does not match the shape of the indexed tensor [1, 18, 1024] at index 1
```

Both intuitively and based on this error, doesn't it make sense that [this line](https://github.com/huggingface/transformers/blob/v4.21-release/src/transformers/models/bart/modeling_bart.py#L1514) instead be
```python
        eos_mask = decoder_input_ids.eq(self.config.eos_token_id)
```

Thanks!

cc @ArthurZucker 

Hey! Thanks for providing a reproducing script 😉 
Is there a reason why you are feeding the model with `decoder_input_ids`? My question is related to the fact that for sequence classification, I don't really know why you should need these? 

Thanks for the reply!

The classifier is supposed to measure generation quality of a separate encoder-decoder generator.  I was planning to leverage the pretraining of the generator by making the inputs of each model similar and then fine-tune off the generator model.  Hence, decoder input is the generation to evaluate, while encoder input is the context.

Do you recommend a different strategy?

I saw [this](https://github.com/osainz59/t5-encoder) repo, inspired by [this paper](https://arxiv.org/abs/2110.08426), and have been experimenting with it as well.  On the one hand, it seems to give good results without the decoder.  On the other, the formulation wouldn't be the same so the pre-training idea wouldn't apply as elegantly.