noamaanMulla-03/transformers-issues huggingface dataset:
Paligemma support for multi-image
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

> We also transfer PaliGemma to tasks which take multiple images as input. NLVR2 is one such task, which asks one question about two images, and requires looking at both to give the correct answer. Other such tasks are standard short-video understanding tasks subsampled to 16 frames. In all these cases, we follow PaLI-3 and encode each image separately, then concatenate the image tokens without any special separator or embedding tokens. Thus, 16 frames at 224px resolution result in 


Fixes #33113. Paligemma certain checkpoints are expected to support multiple images according to the arxiv paper. This PR adds support for that in our code. We expect users to pass in images in a batch if there are more than one images. For ex:

`processor(text=[text1, text2], images=[[im1, im2], [im3]]], padding=True, return_tensros="pt")`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33447). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: add docstring for `image_size` in Convnextv2 config
Author: lucianosrp
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adds missing docstring attribute `image_size`


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone! (@LysandreJik  , @stevhliu ?)




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33734). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Flash Attention Support to Mllama (Llama 3.2)
Author: shermansiu
Pull request: no
Labels: name: Feature request

### Feature request

Please add flash attention support to Llama 3.2!

### Motivation

The code will run faster with flash attention.

### Your contribution

When I have time, I can try to submit a PR, but I've been really busy lately.

Comments:
There are already SDPA implementations included which can leverage flash attention if it is installed. You also can control which backends can / will be used.

See the torch docs over here for SDPA: https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html

You can also refer to the huggingface docs over here: https://huggingface.co/docs/transformers/v4.45.1/perf_infer_gpu_one (under "PyTorch scaled dot product attention")

As @vasqu mentions, Llama already supports FA2 + SDPA, and mllama supports SDPA.

I'm closing the issue, but feel free to reopen if something is unclear
[`MllamaImageProcessing`] Update doc
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Adds back a comment that was removed! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33747). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fast vs. slow tokenizer mismatch for special tokens with lstrip/rstrip set to true
Author: xenova
Pull request: no
Labels: name: Fast Tokenizers

name: bug

### System Info

- `transformers` version: 4.45.0.dev0
- Platform: Linux-6.5.0-1025-azure-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code to reproduce:
```py
from transformers import AutoTokenizer
slow = AutoTokenizer.from_pretrained("distil-whisper/distil-small.en", use_fast=False)
fast = AutoTokenizer.from_pretrained("distil-whisper/distil-small.en", use_fast=True)
text = "   
 
   "

print(slow.tokenize(text))  # ['
', '
']
print(fast.tokenize(text))  # ['   
 ', '
   ']
print(slow.encode(text))    # [50257, 50362, 50257, 50258, 50256]
print(fast.encode(text))    # [50257, 50362, 50257, 50258, 50256]
```

Although the token ids match (`.encode`), the tokenized strings (`.tokenize`) are different.

### Expected behavior

Intuitively, I would expect `.tokenize` to have the lstrip and rstrip applied (i.e., the same as the slow tokenizer), so this may be an issue with `tokenizers` (cc @Narsil) 

Comments:
I can reproduce, this is completely expected and inherent to the way `tokenizers` works. `tokenizers` removes the spaces kind of afterwards. Closing as tokens are not the absolute source of truth
Add multi image prompts to multimodal LLMs that support it (PaliGemma)
Author: Stani-s
Pull request: no
Labels: name: Feature request

### Feature request

Adding the ability to pass many images per prompt to PaliGemma. This would mean, among other changes, to change the argument type of `images` on PaliGemmaProcessor to allow array[array[torch.Tensor]] for batch processing.

### Motivation

The model was trained for multi image / short video tasks so it should be able to take such inputs.

### Your contribution

I could document this if this is supported and I missed it.

Comments:
This is already feasible with models that support it. Could you point to where the authors mention PaliGemma being trained on multi-image domains?

At the time of writing, supported models include:

* Idefics2 and Idefics3: https://huggingface.co/docs/transformers/main/en/model_doc/idefics2#usage-tips
* [LLaVa-Interleave](https://huggingface.co/collections/llava-hf/llava-interleave-668e19a97da0036aad4a2f19)
* [LLaVa-OneVision](https://huggingface.co/collections/llava-hf/llava-onevision-66bb1e9ce8856e210a7ed1fe), currently being added at #32673 
* [Qwen2-VL](https://huggingface.co/collections/Qwen/qwen2-vl-66cee7455501d7126940800d)

cc @zucchini-nlp 

Hi,

In section 3.2.4 of the paper https://arxiv.org/html/2407.07726v1#S3 the authors mention multi image inputs as a possible target task to finetune towards.

There is also a checkpoint of the model https://huggingface.co/google/paligemma-3b-ft-nlvr2-448 that was finetuned for such tasks.

Thank you for linking those models, I'll check them out.

Good point! Indeed seems that Paligemma can take more than one image as input. 

AFAIK most if not all VLMs can theoretically support multiple images in one prompt, even though they aren't tuned explicitly and can have poor generation quality. So, imo we can accept any number of images as input to a VLM by default, if there isn't any strict requirement in modeling that prohibits multiple images. WDYT @NielsRogge?

For Paligemma @Stani-s you can give it a try, or I'll make a PR some time next week

Unfortunately I won't be able to for the next 2 weeks.

Thanks! 🤗
remove warning v2
Author: itazap
Pull request: yes

I missed the second warning in the original PR https://github.com/huggingface/transformers/pull/31938

issue: https://github.com/huggingface/transformers/issues/33719

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33761). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cleanup return_text and return_full_text options in TextGenerationPipeline
Author: Rocketknight1
Pull request: yes

The extra options in `TextGenerationPipeline` right now are confusing and poorly documented. This PR refactors things so that the documentation is clearer. It also creates a proper hierarchy of options (`return_full_text` overrides and ignores `return_text`, and `return_tensors` is incompatible with both)

Before this PR, the documentation was incorrect, as it claimed that `return_full_text` required `return_text`, whereas actually these options were not compatible with each other!

Behaviour after the change should be mostly unchanged, with the exception that `return_full_text` overrides `return_text` if both are set, rather than raising an error. The test that checked for those two options being incompatible has been removed.

Fixes #33535

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33542). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @LysandreJik for review!

Maybe cc @amyeroberts actually, since Lysandre is still away (also I know this is 6 on a Friday, don't worry about reviewing it until Monday!)

@amyeroberts @LysandreJik I reworked the PR in response to Amy's comment - I think she was right about the new behaviour not being any better. I restored the tests and behaviour to how they were before, and instead updated the docs so they correctly reflect that behaviour. As a result, this PR only touches docstrings now!
Model addition timeline
Author: LysandreJik
Pull request: yes

Adds a subsection in the model addition doc file to mention what can be expected in terms of timelines for model additions to transformers.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks all for the reviews!
Add method to get loaded adapter names.
Author: DogitoErgoSum
Pull request: no

https://github.com/huggingface/transformers/blob/3bb7b05229466ce820f76dadb250a848e7eb22e7/src/transformers/integrations/peft.py#L145C60-L145C76

The raise statement on line 146 feels off because there isn't a method that checks which adapters are already loaded. Implementing this is straightforward, as a similar solution is already in place on line 145 using list(self.peft_config).

Comments:
Hey, would you like to open a PR for this? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Maybe of interest to @BenjaminBossan 

Sorry, I don't understand what the issue is. Could you please clarify @DogitoErgoSum? 

@BenjaminBossan There is no method to check which adapters are loaded in the model. The active_adapters() method only returns the adapters that are set, not the ones that are loaded, as far as I understand. When I try to load an adapter that is already present, an error is thrown, and there is no method to check if the adapter exists. Currently, I am using this workaround to find out which adapters are loaded: 
`list(model.peft_config)`

Oh I see, so what you would like to have is a method/attribute that returns all existing adapters. The problem was that you wanted to load adapters and got an error because an adapter of that name already existed, but you didn't have an easy way to check which adapters exist. Did I get that right?

Besides the method you mentioned, `list(model.peft_config)`, you can also use this PEFT function:

```python
from peft import get_model_status

get_model_status(model).available_adapters
```

(`get_model_status` also returns a bunch of other potential useful bits about the model)

Between these two options, do you still think we need another method on transformers?



Exactly.

I don't think it's necessary because of these two options, although it would make the library more complete. What I believe is more necessary is a method to unload (delete) the adapters from the model.

Indeed, that would be a good method to add. For the time being, I think this should do the job:

```python
from peft.tuners.tuners_utils import BaseTunerLayer

for module in model.modules():
    if isinstance(module, BaseTunerLayer):
        module.delete_adapter(adapter_name)
```

Thanks!

Ah yes, and don't forget to delete the entry from the `model.peft_config` dict. I added the task to implement this method to my backlog.
Updated few changes to the code and updated the files which had few typos in it.
Author: vignesh1507
Pull request: yes

I added few lines of code in parse_test_output.py as it had few issues, i made sure not to make changes to the original file that's why i made a new file for the updated code. I have also updated few other files which had typos in it.

Comments:
Thanks! But

> i made sure not to make changes to the original file that's why i made a new file for the updated code

why not directly change the file? A new file like this is difficult to review as we don't know what exactly are the issues and how the fixes are relevant.

Ok then, i'll add my changes to the original file.

Thanks for updating! A few comments around the changes to explain what the issues are and explain a bit how the fix is done is also essential 🙏
it works for me  when adding the following two codes in the model requests code modeule
Author: ruirui-w
Pull request: no

              import os 
os.environ['HF_ENDPOINT']= 'https://hf-mirror.com'

_Originally posted by @xiaozhen228 in https://github.com/huggingface/transformers/issues/17611#issuecomment-2029996244_
            

Comments:
Mmmm would you mind closing if this is not an issue? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Make audio classification pipeline spec-compliant and add test
Author: Rocketknight1
Pull request: yes

cc @LysandreJik, this test splits the large PR at #33452 into manageable pieces.

This first PR will be the largest, since it includes the testing utils, as well as testing compliance for `AudioClassificationPipeline`. The other single-pipeline PRs will be much smaller, since they can just import the same testing utils.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33730). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@LysandreJik added output testing as well! Also yes, the only change in the actual pipeline is that I added the `function_to_apply` argument, with a default argument that matches the original behaviour.

Awesome! Very excited about this work, thanks @Rocketknight1! Good to know that starting from now on, the specs datatypes in `huggingface_hub` are used so we can't introduce breaking changes to them anymore (not complaining here^^). I'll review the changes later today or beginning of next week :)

cc @hanouticelina as well

that's awesome! thanks @Rocketknight1
Fix typing in `load_balancing_loss_func` function of `modeling_mixtral.py`.
Author: PhilipMay
Pull request: yes

fixes #33639

Not sure why this `if` is needed:

https://github.com/huggingface/transformers/blob/78b2929c0554b79e0489b451ce4ece14d265ead2/src/transformers/models/mixtral/modeling_mixtral.py#L148-L149

But this this the reason why the return type is `Union[torch.Tensor, int]` and not just `torch.Tensor`.

Comments:
Tagging @khaimt and @ArthurZucker as the original authors.

> Sounds good, you seem to have issues with `make quality`, should be fix if you run `pip install -e .[quality]`

I have done `pip install -e .[quality]` and a `make quality`. Everything seems to be fine. See here:

```bash
$ make quality
/Users/mike/code/fork/transformers/src/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
ruff check examples tests src utils setup.py conftest.py
All checks passed!
ruff format --check examples tests src utils setup.py conftest.py
3132 files already formatted
python utils/sort_auto_mappings.py --check_only
python utils/check_doc_toc.py
python utils/check_docstrings.py --check_all
/Users/mike/code/fork/transformers/src/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
```

Nothing was changed, fixed or reformatted:

```bash
$ git status
On branch fix-return-type-in-load_balancing_loss_func
Your branch is up to date with 'origin/fix-return-type-in-load_balancing_loss_func'.

nothing to commit, working tree clean
```

@ArthurZucker can you help me to fix this strange "fetch_tests - Failed" issue in the "setup_and_quality" action?

PS: I can not even see the details of the failure in CircleCI because it tells me:

> Something Unexpected Happened
> This is certainly annoying, our apologies. Maybe a refresh would help.

CI is green now. @ArthurZucker 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33641). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Uniformize kwargs for Layoutlm (2, 3, X) processors
Author: leloykun
Pull request: yes

# What does this PR do?

- Uniformizes kwargs for LayoutLM (2, 3, X) processors as discussed in https://github.com/huggingface/transformers/issues/31911
- Also fixes the bug in `tests_exotic_models` which totally prevents any test from being run

Fixes # (issue)

- https://github.com/huggingface/transformers/issues/31911

## Who can review?

@zucchini-nlp @molbap @NielsRogge 

Comments:
Note for reviewer(s): the failing tests are caused by the Jamba model

@zucchini-nlp @yonigozlan I've just rebased this to main

the failing tests are due to Llava-Next:
![image](https://github.com/user-attachments/assets/cb6c3ffc-ee13-4096-b487-95915a0791d2)

but yah, this PR should now be ready for review
Fix the bug with loading DeepSpeed MoE models
Author: foggy-frost-forest
Pull request: yes

# What does this PR do?

Fix the bug with loading DeepSpeed MoE models with `from_pretrained`


# issue

When specifying the `torch_dtype` in `from_pretrained`, the weights of the MoE layers get overwritten, which causes an exception when using DeepSpeed MoE.

By adding a MoE check, we can determine whether the model contains DeepSpeed MoE layers, and if detected, set `param_buffer_assignment` to `False`.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr





Comments:
@SunMarc could you give this PR a quick look? :)
Enhanced Parameter Freezing Capabilities in Trainer Class
Author: Hiusam
Pull request: no

### Feature request

The feature proposal aims to introduce a more streamlined and intuitive approach to freezing and unfreezing specific model components directly in the Trainer class of the Hugging Face transformers library.

### Motivation

I've found that when I need to freeze certain parameters or components of my models, the process can be a bit complicated.

Currently, I need to set requires_grad to False for the parameters I want to freeze before calling Trainer.train(). But since Trainer.train() calls model.train() before the training loop, some parameters (e.g., running mean and running var of BatchNorm layers) will still change during training. To get around this, I have to implement additional flags in my model and manually call model.eval() in the forward function for the parts of the model I want to freeze.

It would be great if there was a more streamlined way to accomplish this directly in the Trainer class. Maybe an additional argument in the Trainer.train() method, or a method in the Trainer class to freeze/unfreeze specified layers or parameters. This could make fine-tuning models easier and more intuitive, particularly for new users or those with less experience in PyTorch.

### Your contribution

I am glad to offer help.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

+1 on this