Unrecognized configuration class ChameleonConfig
Author: sriniiyer
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.0.dev0
- Platform: Linux-5.19.0-0_fbk10_hardened_11465_g45ce90b62da2-x86_64-with-glibc2.34
- Python version: 3.10.12
- Huggingface_hub version: 0.24.0
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1+cu121 (False)
- Tensorflow version (GPU?): 2.13.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: NO

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following code:

```
from transformers import AutoProcessor, AutoModelForCausalLM
processor = AutoProcessor.from_pretrained("facebook/chameleon-7b", token="blah")
model = AutoModelForCausalLM.from_pretrained("facebook/chameleon-7b", token="blah")

```
### Expected behavior

Should run without errors, instead, we get this:

```
2024-07-18 17:28:52.471308: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-18 17:28:52.536932: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-18 17:28:53.558868: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
preprocessor_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 460/460 [00:00<00:00, 3.03MB/s]
Some kwargs in processor config are unused and will not have any effect: image_token, image_seq_length.
Traceback (most recent call last):
  File "/home/sviyer/test_download.py", line 4, in <module>
    model = AutoModelForCausalLM.from_pretrained("facebook/chameleon-7b", token="blah")
  File "/home/sviyer/anaconda3/envs/p310/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 567, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.chameleon.configuration_chameleon.ChameleonConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, ......
```

Comments:
Hi @sriniiyer, thanks for opening an issue! 

Chameleon was originally added to be loaded under AutoModelForCausalLM, but should have been under AutoModelForConditionalGeneration, which has been updated. The following should work: 

```py
from transformers import AutoProcessor, AutoModelForConditionalGeneration
processor = AutoProcessor.from_pretrained("facebook/chameleon-7b", token="blah")
model = AutoModelForConditionalGeneration.from_pretrained("facebook/chameleon-7b", token="blah")
```

If this snippet was copied from somewhere, could you share a link? This way we can make sure our docs are up-to-date 

cc @zucchini-nlp 


@amyeroberts I think the model repo needs to be updated since the [config file](https://huggingface.co/facebook/chameleon-7b/raw/main/config.json) still says

```
  "architectures": [
    "ChameleonForCausalLM"
  ]
```

@ywang96 hey! Thanks for pointing, config has to changes as well, yes. Will open a PR, so the changes probably will be merged by Monday-Tuesday

Also, to add on @amyeroberts reply, seems like there was a typo. If you want to load with auto model, you can use `AutoModelForVision2Seq` which is used for all VLMs currently

```python
from transformers import AutoProcessor, AutoModelForVision2Seq

processor = AutoProcessor.from_pretrained("facebook/chameleon-7b")
model = AutoModelForVision2Seq.from_pretrained("facebook/chameleon-7b", torch_dtype=torch.bfloat16, device_map="cuda:0")
```

@zucchini-nlp Thank you for the response! May I also ask if there's an ETA on the new release of transformers with Chameleon supported? Thanks!

@ywang96 Very soon! This week, possibly today 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.