Unable to load bart-large-cnn model correctly
Author: kao73
Pull request: no

### System Info

- `transformers` version: 4.33.3
- Platform: Linux-5.15.136-90.144.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.20.3
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: No


### Who can help?

@ArthurZucker @younesbelkada @SunMarc

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Download the `facebook/bart-large-cnn` model to local FS (/tmp/model)
2. Start TGI with parameters:
     text-generation-launcher --hostname 127.0.0.1 --port 9000 --sharded false --num-shard 1 --dtype bfloat16 --max-input-length 1024 --max-total-tokens 2048 --max-batch-total-tokens 4096 --model-id /tmp/model
3. Make request to the server by any available client. The `inputs` text can be any string
4. The model produces meaningless text. Example: 
    Input: `The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris.` 
    Output: `had had had had had had had had had had had only had`

### Expected behavior

The model should produce summarization generated text for the input request

Comments:
Probably I found the issue in the source code, but not sure how to fix it correctly. Thus, no PR from me for now.
The `facebook/bart-large-cnn` has `model_type`=`bart` in the configuration file.

Looking at the piece of code: https://github.com/huggingface/text-generation-inference/blob/v1.1.0/server/text_generation_server/models/__init__.py#L300-L316
```
    if model_type in modeling_auto.MODEL_FOR_CAUSAL_LM_MAPPING_NAMES:
        return CausalLM(
            model_id,
            revision,
            quantize=quantize,
            dtype=dtype,
            trust_remote_code=trust_remote_code,
        )
    if model_type in modeling_auto.MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES:
        return Seq2SeqLM(
            model_id,
            revision,
            quantize=quantize,
            dtype=dtype,
            trust_remote_code=trust_remote_code,
        )
```
The `bart` model type mentioned in the `modeling_auto.MODEL_FOR_CAUSAL_LM_MAPPING_NAMES` and `modeling_auto.MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES` lists.
Thus, the first condition returns `true`, and the model loaded by `CausalLM` class instead of `Seq2SeqLM`.

According to the model card: https://huggingface.co/facebook/bart-large-cnn
the model should be deployed by `AutoModelForSeq2SeqLM` (`BartForConditionalGeneration`) class:
```
# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-large-cnn")
```

Hi @kao73 thanks for the issue,
Bart architecture contains both a[ decoder-only](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/modeling_bart.py#L1949) (i.e. CausalLM model) and [enc-decoder model](https://github.com/huggingface/transformers/blob/e107ae364e5b9564c8f8a14dcc185efa506c7b6e/src/transformers/models/bart/modeling_bart.py#L1496 ). 
For the checkpoint `facebook/bart-large-cnn` it is indeed an encoder-decoder: https://huggingface.co/facebook/bart-large-cnn/blob/main/config.json#L7
Perhaps the fix should go on TGI side and in the case the model architecture is both present in `MODEL_FOR_CAUSAL_LM_MAPPING_NAMES` and `MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES`, the choice of the class to load should be determined by `model.config.is_encoder_decoder`. cc @OlivierDehaene 

@OlivierDehaene @younesbelkada Was this resolved? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.