noamaanMulla-03/transformers-issues huggingface dataset:
Adds support for the Falcon2-11B VLM
Author: YasserdahouML
Pull request: yes

This PR adds support for the Falcon2-11B Vision Language Model. The Falcon2-11B VLM is a vision-language model (VLM) for additionally handling image inputs and answering the queries corresponding to the images. To achieve this, we integrate the pretrained CLIP ViT-L/14 vision encoder with our Falcon2-11B chat-finetuned model and train with image-text data.
For enhancing the VLM's perception of fine-grained details w.r.t small objects in images, we employ a dynamic encoding mechanism at high-resolution for image inputs, similar to [LLaVA-Next](https://llava-vl.github.io/blog/2024-01-30-llava-next/).

Comments:
Fix VideoLlava imports
Author: amyeroberts
Pull request: yes

# What does this PR do?

Currently, it's not possible to run `from transformers import *` when Pillow is not installed in the environment. 

This is because [in the model init](https://github.com/huggingface/transformers/blob/4e17e7dcf8ac7c72268151823e82ff8db8c8fcba/src/transformers/models/video_llava/__init__.py#L25), the image processor import is protected by `is_vision_available`, but the [same pattern isn't reflected in the module init](https://github.com/huggingface/transformers/blob/4e17e7dcf8ac7c72268151823e82ff8db8c8fcba/src/transformers/__init__.py#L3246). 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30867). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Skip failing Object Detection example for now
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR skips the failing object detection example when running on multiple GPUs until it's fixed.

cc @qubvel 

Fixes main CI for Accelerate


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
Ah nevermind, just now saw https://github.com/huggingface/transformers/pull/30742 :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30869). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Whisper] Add model for audio classification
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Adds `WhisperForAudioClassification`: the Whisper encoder model with a sequence classification head on top for audio classification tasks.

With the changes implemented in #21756, Whisper can be fine-tuned for any audio classification task.

Results of fine-tuning for suggest that this is an extremely promising approach for audio classification. On the FLEURS language identification task, fine-tuning Whisper medium achieves an accuracy of **88%**, beating previous SoTA by 10% and the zero-shot Whisper model by 24% absolute:

<img width="614" alt="Screenshot 2023-02-28 at 09 37 27" src="https://user-images.githubusercontent.com/93869735/223370179-338e1f12-7793-48e5-9715-66d7f9da4af3.png">

See logs at [sanchit-gandhi/whisper-medium-fleurs-lang-id](https://huggingface.co/sanchit-gandhi/whisper-medium-fleurs-lang-id) for details.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

(Failing test is unrelated)



**Whisper based Audio classification**

The following code shows error message

from transformers import AutoModelForAudioClassification

num_labels = len(id2label)

model = AutoModelForAudioClassification.from_pretrained(
    model_id,
    num_labels=num_labels,
    label2id=label2id,
    id2label=id2label,
)

**ValueError: Unrecognized configuration class <class 'transformers.models.whisper.configuration_whisper.WhisperConfig'> for this kind of AutoModel: AutoModelForAudioClassification.**
Model type should be one of ASTConfig, Data2VecAudioConfig, HubertConfig, SEWConfig, SEWDConfig, UniSpeechConfig, UniSpeechSatConfig, Wav2Vec2Config, Wav2Vec2ConformerConfig, WavLMConfig.


Hey @manjualoshious - could you please open a new issue for this error? It would help tremendously in keeping track of the problem! Thanks

Ok sir

On Thu, 16 May, 2024, 7:49 pm Sanchit Gandhi, ***@***.***>
wrote:

> Hey @manjualoshious <https://github.com/manjualoshious> - could you
> please open a new issue for this error? It would help tremendously in
> keeping track of the problem! Thanks
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/21754#issuecomment-2115378320>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AN6MF7AW4DN672MEJQOIJT3ZCS56TAVCNFSM6AAAAAAVFN7PP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJVGM3TQMZSGA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
TEST: Add llama logits tests
Author: younesbelkada
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Adds the llama logits tests that have been always skipped 😢 

Tested on our A10 & T4 Runners !

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove deprecated logic and warnings
Author: amyeroberts
Pull request: yes

# What does this PR do?

Kills a bunch of deprecated code 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30743). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker Thanks for the review! I've removed all of the kwargs being passed now, and the `_cos_cached`, `_sin_cached` buffers
fix a typo in flax T5 attention - attention_mask variable is misnamed
Author: giganttheo
Pull request: yes

# What does this PR do?

Fixes a typo in the Flax code for T5 model.

There is a typo in the Attention module of the Flax version of T5, where the attention_mask updated by the `_concatenate_to_cache` method should override the previous attention_mask but does not because of a misnamed variable.

Fixes #26564 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26663). All of your documentation changes will be reflected on that endpoint.

> Very nice @giganttheo! Thanks for identifying the bug and proposing the fix 🤗 Confirming that the slow tests pass following the fix? As per [#26564 (comment)](https://github.com/huggingface/transformers/issues/26564#issuecomment-1751779825) If so, then this all LGTM!

The slow tests are passing for t5 and longt5:

`RUN_SLOW=1 pytest -sv tests/models/t5/test_modeling_flax_t5.py::FlaxT5ModelIntegrationTests`

outputs: `================== 6 passed, 4 warnings in 331.38s (0:05:31) ===================`

and for the longT5 version:

`RUN_SLOW=1 pytest -sv tests/models/longt5/test_modeling_flax_longt5.py::FlaxLongT5ModelIntegrationTests`

outputs: `=================== 1 passed, 1 warning in 401.61s (0:06:41) ===================`

Awesome - thanks for confirming! Requesting a final review from @ArthurZucker
[`warnings`] Remove-warnings / update code
Author: ArthurZucker
Pull request: yes

# What does this PR do?

Clean the code, similar to #29492

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Enabling device_map="auto" for Video-LLaVA
Author: darshana1406
Pull request: no
Labels: name: Feature request

name: Vision

### Feature request

To load model across multiple devices, _no_split_modules has to be defined in the model's pretrained model class e.g. like [here](https://github.com/huggingface/transformers/blob/v4.40.2/src/transformers/models/llava/modeling_llava.py#L235) for LLaVa.

### Motivation

To enable inference of multiple lower end devices and avoid OOM.

### Your contribution

@zucchini-nlp
I could try to work on this but I'm not sure about what modules should be included here.

Comments:
Thank for opening an issue! Sure, adding a `_no_split_modules` in the same way as it is done for other LLaVa models will do the work. See [here](https://github.com/huggingface/transformers/blob/ba56ed0869eb4bbeb1c04af7f62a04350150e8d4/src/transformers/models/llava_next/modeling_llava_next.py#L190) for an example

Let me know if you need any guidance 😄
Zero-Shot Image Classification Guide Missing Dependency and Addition for Local Inputs
Author: Tanman2001
Pull request: no
Labels: name: Good First Documentation Issue

### System Info

- transformers version: 4.39.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.1
- Huggingface_hub version: 0.22.1
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Tested using PyTorch only.

- Missing pip install pillow

1. Follow Zero-Shot Image Classification guide here: https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification
2. Error on `detector = pipeline(model=checkpoint, task="zero-shot-image-classification")` message: CLIPImageProcessor requires the PIL library but it was not found in your environment.

Additional recommendation:

Users should be able to (and likely will attempt to) use locally saved images with their model. Guide only states how to use an image from the internet using a URL.

Add to guide in both sections:
```
path = “<path of image>”
image = Image.open(path)
```
as an option to use a locally stored images.

### Expected behavior

Following guide as written should not result in crash.

Additionally, users should know how to use the model with their own, locally-saved images.

Comments:
would you like to open a PR for a fix?
Mamba: which tokenizer has been saved and how to use it?
Author: javiermcebrian
Pull request: no

### System Info

Hardware independent.

### Who can help?

@ArthurZucker 

I described the doubts in the link below around 1 month ago, but maybe model-hub discussions are not so active. Then I post it here as repo issue. Please, let me know where to discuss it :)

https://huggingface.co/state-spaces/mamba-2.8b-hf/discussions/1

Thanks!

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

.

### Expected behavior

.

Comments:
Hi @javiermcebrian, you can see which tokenizers are saved for a checkpoint from the files in the model repo e.g. [here for state-spaces/mamba-2.8B-hf](https://huggingface.co/state-spaces/mamba-2.8b-hf/blob/96c48e0292b63f5346b6d30061af2551f7101e26/tokenizer_config.json#L210)

You can use any checkpoints' tokenizer by doing:

```py
from transformers import AutoTokenizer

# You can replace state-spaces/mamba-2.8b-hf with any pretrained checkpoint with a tokenizer
tokenizer = AutoTokenizer.from_pretrained(state-spaces/mamba-2.8b-hf)
```

I suggest looking at the [Hugging Face course](https://huggingface.co/learn/nlp-course/chapter1/1) to find out about how to use all of our NLP models and integrations 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Implement Half-Quadratic Quantization (HQQ)
Author: michaelfeil
Pull request: no
Labels: name: Feature request

### Feature request

I would be curious if https://github.com/mobiusml/hqq can be supported in similar fashion to `autogptq` or `autoawq`. hqq is most similar to `bitsandbytes` `nf4/fp4` datatypes, but offers 2,3,4,8 bit quantization.
CC: @mobicham

### Motivation

HQQ performs 2/3/4 bit quantization and can do drop-in replacement. Its fast for in-place quantization / non-pre-quantized weights and performs similar to bnb a expansion to fp16 at runtime (or similar).

 Would be cool to support for models like mixtral to cut down the vram requirement.

### Your contribution

Currently have no capacity for submitting an integration, but happy to review or assist.

Comments:
cc @younesbelkada 

This is very cool ! We are definitely interested in adding HQQ inference support in transformers. The cool thing is that indeed it seems you don't need to pre-quantize the weights in order to quantize the models. We'll explore a bit on our side and let you know how it goes
cc @SunMarc @Titus-von-Koeller 

Hi! I am the maintainer of the HQQ project, happy to assist with anything needed !

Very glad to e-meet you @mobicham ! do you have an email I can use so that we can contact you through Slack to iterate quickly?

Glad to e-meet you @younesbelkada as well! Sure: hicham@mobiuslabs.com

thanks @mobicham you should have received an invite by now! 

Closing as HQQ is now part of the release!
[Trainer] Rename tokenizer to processor, add deprecation
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR updates the `tokenizer` argument which people can pass to the Trainer and TrainerCallback classes to `processor` instead. This allows for people to pass tokenizers, image processors, feature extractors or multimodal, which it will then automatically save along the model when training.

Follow-up of #29896

To do:

- [ ] perhaps use `preprocessor` instead of `processor` to avoid confusion with multimodal processors like `CLIPProcessor`?
- [x] add deprecation message for tokenizer attribute

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30102). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks, I opened #30129 as a better quick fix cause there are a huge amount of files having `tokenizer=tokenizer` and I'm not sure it's worth the effort (would require some offline discussion).

@NielsRogge Although #30129 proposes a solution for the default data collator, we should still add this update.  `tokenizer=image_processor` is a confusing API (as highlighted by a few issues from users) and considering we have more and more multimodal, audio and vision models increasingly out-of-date

Replacing `tokenizer=tokenizer` with `processor=tokenizer` might touch a lot of files but I'd expect to be fairly straightforward with some grep and find/replace. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Thanks for pointing me in the direction of this PR @amyeroberts! I personally don't think replacing `tokenizer` by `processor` is a good idea in the context of NLP users: many NLP users won't know the notion of a `processor`, since they only use the `tokenizer` to pre- and post-process their data. To me, having them set `processor=tokenizer` is just as confusing as having multimodal users set `tokenizer=processor`.

I would be more in favour of a design where allow NLP users to pass `tokenizer=tokenizer`, and multimodal users to pass `processor=processor`, as suggested in https://github.com/huggingface/transformers/pull/30864#discussion_r1603659097 -> to me, this is the cleanest and most intuitive API. Yes, it adds some complexity under-the-hood in the Trainer, since we now have to handle both object, but this is a valid burden to make the API as intuitive as possible for the user.

@sanchit-gandhi I think what's being suggested is actually a flag more like `preprocessor` or `processing_class` which would replace `tokenizer`. As Trainer is such a commonly used object, in reality we're unlikely to fully remove or deprecate `tokenizer` anytime soon, but I don't think this means we have to add a whole new argument for every processing class.
Integrate IndicTrans2 models and tokenizer into HF Transformers
Author: VarunGumma
Pull request: no
Labels: name: New model

### Model description

[IndicTrans2 ](https://openreview.net/forum?id=vfT4YuzAYA)is a multilingual transformer model developed by AI4Bharat, and is available in 3 flavors: `indic-en`, `en-indic` and `indic-indic`. Each flavor has 2 versions, a large 1B model, and a distilled 200M model. The architecture is a standard transformer, very similar to NLLB and M2M models. However, the major difference is the vocabularies of the encoder and decoder and not shared, as they require different languages.

Unlike, NLLB and M2M models, IndicTrans2 required specific preprocessing for the inputs. Hence a [custom processor class has been developed](https://github.com/VarunGumma/IndicTransTokenizer), and is required for training/inference. More examples can be found in the aforementioned repository.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Authors: @AI4Bharat @jaygala24 @PranjalChitale @oneraghavan @VarunGumma @sumanthd17 @prajdabre @anoopkunchukuttan

Official GitHub Repository: [AI4Bharat/IndicTrans2](https://github.com/ai4bharat/IndicTrans2)

The HF compatible models and tokenizer are available here as of now:
- [indictrans2-en-indic-1B](https://huggingface.co/ai4bharat/indictrans2-en-indic-1B)
- [indictrans2-en-indic-dist-200M](https://huggingface.co/ai4bharat/indictrans2-en-indic-dist-200M)
- [indictrans2-indic-en-dist-200M](https://huggingface.co/ai4bharat/indictrans2-indic-en-dist-200M)
- [indictrans2-indic-en-1B](https://huggingface.co/ai4bharat/indictrans2-indic-en-1B)
- [indictrans2-indic-indic-1B](https://huggingface.co/ai4bharat/indictrans2-indic-indic-1B)
- [indictrans2-indic-indic-dist-320M](https://huggingface.co/ai4bharat/indictrans2-indic-indic-dist-320M)

Comments:
Hi @VarunGumma, thanks for opening this model request!

This looks like a great candidate for adding the model on the hub. This is the easiest and recommended way to make a model available in `transformers` and means, once working, the model can be found and used immediately without having to go through the PR process. We find this is a lot quicker as the bar for adding code into the library is high due to the maintenance cost of every new model, and so reviews take quite a while.

We have as much support as we can for this - let us know know if there's any issues in implementation. Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you!

Hu @amyeroberts, 

Thank you for your reply. We also need some help to add `flash_attention_2` to our model. We were able to modify the modeling script for it, but it throws us an error that our model class `IndicTransForConditionalGeneration` itself is now supported. How can we proceed in this case?

@VarunGumma Could you share the error message and full traceback? 

@amyeroberts , thank you. We were able to resolve it on our end.
Add IRIS
Author: RUFFY-369
Pull request: no
Labels: name: New model

### Model description

IRIS (Imagination with auto-Regression over an Inner Speech) is a Reinforcement learning agent trained in the imagination of a world model composed of a discrete autoencoder and an autoregressive Transformer. IRIS learns behaviors by accurately simulating millions of trajectories.

The approach presented in the paper casts dynamics learning as a sequence modeling problem, where an autoencoder builds a language of image tokens and a Transformer composes that language over time.

The agent is introduced in the paper titled [TRANSFORMERS ARE SAMPLE-EFFICIENT WORLD MODELS](https://arxiv.org/abs/2209.00588).

There is also a [medium blog post](https://medium.com/@cedric.vandelaer/paper-review-transformers-are-sample-efficient-world-models-d0f9144f9c09) to understand how the algorithm works.



### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The model is adapted from the [official code repository](https://github.com/eloialonso/iris) of the paper.

The officially released weights can be found on this [Github repository](https://github.com/eloialonso/iris_pretrained_models)


Comments: