BART decoder output length changes
Author: elangovana
Pull request: no
Labels: name: bug

### System Info

Hi,  BART for conditional generation, the output sequence length changes?



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

m = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-base")
t = AutoTokenizer.from_pretrained("facebook/bart-base")

b = t(["This is test"] ,
           padding='longest',
            truncation=True,
            is_split_into_words=False,
            max_length=512,
          return_tensors='pt')

print("input data: ", b)
print("model logit output shape: ", m(**b.data).logits.shape)
```

```text
input data:  {'input_ids': tensor([[   0,  713,   16, 1296,    2]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])}

model logit output shape: torch.Size([1, 5, 50265])
```

The sequence length 5 seems to vary based on the input size. Doesn't this mean that the output can never be longer than the output?

In addition is affects how the `y` target sequence length is determined.  The y target tokenise cannot be set to `max_length`, neither to longest cos then when the loss  function is computed the predicted seq length doesn't match target.

```
  batch_y = t(
            text=label_texts,
            padding='max_length',
            truncation=True,
            is_split_into_words=False,
            max_length=self._max_length,
            return_tensors='pt',
        )
```

### Expected behavior

I am not sure how varying output length works with having to compute the loss function

Comments:
Hi @elangovana ,   

> The sequence length 5 seems to vary based on the input size. Doesn't this mean that the output can never be longer than the output?

The output can be longer than the input because Bart is an encoder-decoder model so the decoder can output variable length of tokens independent of the len of input tokens. This is because the decoder works in an auto-regressive manner.

Bart is an [encoder-decoder model](https://huggingface.co/blog/encoder-decoder) so the length of 5 for your example is something which will be consumed by the encoder [art of Bart. The length of target is something which the decoder part of Bart will be concerned about. Bart like any other model accepts `input_ids` and `labels`. The `input_ids` are input to the encoder whereas the `labels` are the expected output for the decoder. 

Now how do we get the `input_ids` for the decoder of Bart?
* During training: you would provide the model with labels which will be shifted internally by HF and will be fed as input to the decoder (for teacher forcing style of decoding)
* During Inference/test: you would generate the decoder output **auto-regressively** wherein the output of the decoder in previous time step becomes the decoder input to the current time step 

How long can the decoder output be? This is decided by your choice of stopping criteria of the auto-regressive decoding, e.g., :
* stop when the number of tokens in decoded sequence reach a max limit
* stop when all the decoded sequences in a batch have outputted a terminating token like EOS

[Check this out ](https://huggingface.co/blog/how-to-generate) for more info on `generate` function in HF and auto-regressive decoding.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.