noamaanMulla-03/transformers-issues huggingface dataset:
Sync DQA pipeline with Hub spec
Author: Rocketknight1
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34752). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Note - I think the output tests aren't being run here, so don't merge even though the CI is green! 

Update: This should now be ready to merge, but depends on the matching [huggingface.js PR](https://github.com/huggingface/huggingface.js/pull/1040) to be merged. Note that **:rotating_light: the pipeline tests are not run in the CI, so make sure to run them locally before merging this or we'll get failures in the nightlies!** :rotating_light: 

In this PR, and in the matching `huggingface.js` PR, we remove the `words` key. The reason is that the pipeline never actually outputs it, and it shouldn't have been included in the docstrings in the first place! Specifically, the pipeline has two code paths in several functions, one for `VisionEncoderDecoder` models and one for everything else.

In `preprocess()`, the VisionEncoderDecoder path always sets [words to None](https://github.com/huggingface/transformers/blob/1c471fc3070a9179f32d147df40b760804f2ef4e/src/transformers/pipelines/document_question_answering.py#L348) which means it cannot be passed through to the output. In `postprocess()` , the non-VisionEncoderDecoder path [calls](https://github.com/huggingface/transformers/blob/1c471fc3070a9179f32d147df40b760804f2ef4e/src/transformers/pipelines/document_question_answering.py#L452) `postprocess_extractive_qa`. However, this function [rewrites](https://github.com/huggingface/transformers/blob/1c471fc3070a9179f32d147df40b760804f2ef4e/src/transformers/pipelines/document_question_answering.py#L497) the `answers` dict without a `words` key


In other words, the `preprocess()` method deletes words for `VisionEncoderDecoder` models and the `postprocess()` method deletes it for everything else, so it always gets deleted! The right solution is just to remove it from the docstring and the JS spec.
Fix callback key name
Author: jung-hunsoo
Pull request: yes
Labels: name: Documentation

name: trainer

Fixes typo.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
fix: Wrong task mentioned in docs
Author: ecyht2
Pull request: yes
Labels: name: Documentation

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/hub-docs/issues/1423.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Allow handling files as args for a tool created with Tool.from_space
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Allow sending files as args for a tool created with `Tool.from_space()`.

Leverages `gradio_client`'s `handle_file` under the hood to prepare files before sending them to the API.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34687). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @LysandreJik !
DistillBERT is ExecuTorch compatible
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

### Feature request

Enable DistillBERT to ["Export to ExecuTorch"](https://github.com/huggingface/transformers/issues/32253) workflow

### Motivation

See details in #32253

### Your contribution

Enable DistillBERT model

Comments:
Export to ExecuTorch with Quantization
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

### Feature request

This task is to experiment running quantized HuggingFace models with ExecuTorch out-of-the-box.

The heavy-lifting quantization work will be done through [`quantize_`](https://github.com/pytorch/ao/blob/main/torchao/quantization/quant_api.py#L94) API by [`torchao`](https://github.com/pytorch/ao), for example `quantize_(model, int4_weight_only())`.

The quantization API can be integrated with the integration points to executorch `transformers.integrations.executorch`, expanding the export workflow with a new option of "exporting with quantization". In eager, users can verify the numerics accuracy of the quantized exported artifact, e.g. the script for eval llama ([here](https://github.com/pytorch/ao/blob/main/torchao/_models/llama/eval.py)). In ExecuTorch, users can just load the quantized `.pte` files to ExecuTorch runner for inference.



### Motivation

Experiment quantization workflow w/ `transforms` + `torchao` + `executorch`

### Your contribution

Direct contribution, or provide guidance to anyone who is interested in this work

Comments:
Export-to-ExecuTorch via transformers.js integration
Author: guangy10
Pull request: no
Labels: name: Feature request

name: Deployment

### Feature request

### Feature request

Integrate `ExecuTorch` as a new backend to transformers.js, enabling an new ["Export to ExecuTorch"](https://github.com/huggingface/transformers/issues/32253) workflow through web browser

This feature will require ExecuTorch runtime w/ backend/kernels can be compiled for WebAssembly first. This task is tracked in ExecuTorch repo: https://github.com/pytorch/executorch/issues/3497

### Motivation

Enable a new e2e workflow for on-device ML use-cases via [ExecuTorch](https://github.com/pytorch/executorch)

### Your contribution

TBD

Comments:
Should this be moved to the transformers.js repo? :slightly_smiling_face: 

cc @xenova 

> Should this be moved to the transformers.js repo? 🙂
> 
> cc @xenova

I agree :) @guangy10 feel free to re-open [here](https://github.com/xenova/transformers.js/issues/new/choose).

Opened https://github.com/huggingface/transformers.js/issues/1039 in transformers.js, and close this one
SlidingWindowCache issue
Author: Cyrilvallez
Pull request: no
Labels: name: bug

### System Info

On main

### Who can help?

@zucchini-nlp  @gante 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

We currently cannot use `cache_implementation='sliding_window'` with FA2. The following snippet

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache, StaticCache
import torch

device = 3


model_name = 'mistralai/Mistral-7B-v0.1'
dtype = torch.bfloat16
attn = 'flash_attention_2'

model = AutoModelForCausalLM.from_pretrained(model_name, attn_implementation=attn,
                                            torch_dtype=dtype, low_cpu_mem_usage=True).cuda(device)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token



generation_kwargs = {
    "max_new_tokens": 50,
    "eos_token_id": None,
    "do_sample": False,
    "return_dict_in_generate": True,
}


inputs = tokenizer('Hello who are', padding=True, return_tensors='pt')

input_ids = inputs['input_ids'].to(device)
attention_mask = inputs['attention_mask'].to(device)
attention_mask

outputs = model.generate(input_ids, attention_mask=attention_mask, **generation_kwargs, cache_implementation='sliding_window')
```

fails with

```python
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the input.
```

### Expected behavior

This comes from the fact that `prepare_input_for_generation` creates a 4d mask if `if isinstance(past_key_values, StaticCache)`, but FA2 does not support 4d masks. I believe we need a check and correct expansion of the 2d mask as well with `SlidingWindowCache`. Found about it when opening https://github.com/huggingface/transformers/pull/34093

Comments:
Oh, you're right! Forgot we had issues with FA2 earlier. Prob we should check for attn implementation in `prepare_inputs` in that case, I think it is the easier fix. 

Maybe we can simply start calling `update_causal_mask` for that @gante ? So we let the model's `update_mask` method handle everything and prepare us a 4D mask instead of bloating the code every time a new exception is found? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Errors when using transformers with torch<2.5
Author: loadams
Pull request: no
Labels: name: bug

### System Info

Transformers latest from `main`, pytorch 2.3, cpu or gpu.

Needs to have #34184

### Who can help?

@ArthurZucker @muellerzr 

### Information

- [x] My own modified scripts

### Reproduction

1. `pip install torch==2.3`
2. `pip install git+https://github.com/huggingface/transformers.git`
3. `python -c "import transformers.models.vision_encoder_decoder.modeling_vision_encoder_decoder"`


### Expected behavior

This does not error out with torch 2.5, but does with torch 2.3 or 2.4, if these are not supported, we should get an error message when installing transformers.

Comments:
Yep, will open a PR for that !
Trainer hyperparameter search kwargs docs update
Author: GuillemGSubies
Pull request: yes

# What does this PR do?

superseding of #34311 

Fixes #34239 

Comments:
@stevhliu

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34459). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for your patience and for trying out all these potential fixes! Unfortunately, it looks like it didn't work :( I'm not sure why the other CI tests aren't running for you. @ydshieh do you happen to know why the other tests aren't running?

@LysandreJik @ArthurZucker, not sure why the two tests just hang without ever starting. This happened in https://github.com/huggingface/transformers/pull/34311 as well, and opening a new PR didn't help. Do you think its ok to just merge this?

Trying to push this branch on another to see if it triggers it

I pushed the branch to another and it now works. Unfortunately couldn't get to the root of the issue as the last commit was too old; @stevhliu, let me know if you see this issue happening again. In the meantime, yes, let's merge it as soon as it's green.
`Trainer.hyperparameter_search` `kwargs` parameter has an inexact definition if using Optuna
Author: GuillemGSubies
Pull request: no
Labels: name: Documentation

According to the [docs](https://huggingface.co/docs/transformers/v4.45.2/en/main_classes/trainer#transformers.Trainer.hyperparameter_search.kwargs), those `kwargs` will only be used in the `create_study` method in Optuna.

However that is not exactly true, as we can see in `run_hp_search_optuna`. Some parameters from those `kwargs` will end up being used in the `optimize` method (which is great because it is what I wanted to do but the docs said I could not do it).

https://github.com/huggingface/transformers/blob/b54109c7466f6e680156fbd30fa929e2e222d730/src/transformers/integrations/integration_utils.py#L256

https://github.com/huggingface/transformers/blob/b54109c7466f6e680156fbd30fa929e2e222d730/src/transformers/integrations/integration_utils.py#L261

I didn't check ray or sigopt but I guess that something similar can be happening.

I don't really know what would be a good solution given that not every parameter in `optimize` is used and creating a `kwargs_optimize` would not make sense given that you support 2 other backends. Maybe a simple list of accepted paramters in the docstring?

If you tell me what solution would be the best, I could send a PR.

@stevhliu

Comments:
Hey, thanks for pointing out the discrepancy! I think your proposed solution of a list of accepted parameters in the docstring would be the simplest fix 🙂 

I sent a PR, I tried not to make it too verbose.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
feat: allow to use hf-models
Author: cgebbe
Pull request: yes
Labels: name: Vision

# What does this PR do?

Problem: This doesn't currently work, but should in my opinion? 

```python
        import transformers
        model = transformers.AutoBackbone.from_pretrained(
            "hf-hub:bioptimus/H-optimus-0",
            use_timm_backbone=True,
            use_pretrained_backbone=True,
        )
```

Solution approach: 

Just remove the two lines which explicitly check whether the string is in `timm.list_models()`. Alternatively, we could introduce more e.g. `backbone_kwargs` arguments.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Models:

- vision models: @amyeroberts, @qubvel


Comments:
In addition to pinging @qubvel, pinging @rwightman for knowledge

@LysandreJik @cgebbe I assume author of PR has verified that this fix allows the desired use case to work so I'm not looking at that. A quick look I believe it should allow the use case for hf-hub timm models which definitely should be supported.

I also believe the removal of that check is appropriate in any case. If the requested model isn't in timm it will error anyways.  e.g. `timm.create_model('crumpet')` -->  `RuntimeError: Unknown model (crumpet)`

If ValueError is preferred for uniform exception handling I can change RuntimeError to slightly more specific ValueError.
[DO NOT MERGE] Add pytest-subtests to the dependencies
Author: dvrogozh
Pull request: yes

Pushing a draft PR to check that CI will start catching #34722. If it will, I suggest to update PR #34723 with this change.

As discussed in [1], pytest-subtests changes behavior of .skipTest() having effect to really skip individual subtests or skip the entire test if module is not installed. Huggingface Accelerate has module in its dependencies. It makes sense to add it for Transformers as well to avoid divergent environment between users and ci.

See[1]: https://github.com/huggingface/transformers/pull/34723#issuecomment-2479191402

CC: @ydshieh

Comments:
@ydshieh : I don't see that pytest-subtests propagated to circleci environment after I updated setup.py (it is not in the installed package list which circleci prints). Does circleci has different list of packages to install somewhere? or I did some mistake updating setup.py.

Hi @dvrogozh .

I think `skipTest()` might have some issues with `pytest-subtests`, see

https://github.com/huggingface/transformers/pull/34723#issuecomment-2479386339

And I think it would be better to (for us) to figure out what happens before merging this PR.



Let's wait 

https://github.com/pytest-dev/pytest-subtests/pull/169

(without it, using pytest-subtests is not reliable in some cases)
Uniformize model processors (models w/o special arg names)
Author: leloykun
Pull request: yes

# What does this PR do?

- Uniformizes kwargs for processors of AltCLIP, Flava, Git, InstructBlipVideo, LLaVa-NeXT-Video, MGP, Siglip, TVP, VideoLLaVa, VILT, X-CLIP as discussed in https://github.com/huggingface/transformers/issues/31911

TODO:
- [x] add tests
  - [x] add tests for AltCLIP
  - [x] add tests for Flava
  - [x] add tests for Git
  - [x] add tests for InstructBlipVideo
  - [x] add tests for Llava-NeXT-Video
  - [x] add tests for MGP
  - [x] add tests for Siglip
  - [x] add tests for TVP
  - [x] add tests for VideoLLava
  - [x] add tests for VILT
  - [x] add tests for X-CLIP

Fixes # (issue)

- https://github.com/huggingface/transformers/issues/31911

## Who can review?

@zucchini-nlp @molbap @NielsRogge 

Comments:
@zucchini-nlp I just added 4 more models:
- Git
- MGP
- TVP
- X-CLIP
Uniformize model processors (models *with* special arg names) 
Author: leloykun
Pull request: yes

# What does this PR do?

- Uniformizes kwargs for processors of ClipSeq, Nougat, Owlv2, OwlVIT as discussed in https://github.com/huggingface/transformers/issues/31911
- Adds backward compatibility for special call arguments passed as positional arguments. Special call args are arguments that carry data (e.g. negative prompt, segmentation images, etc.), but aren't `[text, images, audio, videos]` and not config values for the tokenizer, image processor, etc.

 (arguments that carry data 

TODO:

- [x] add tests
  - [x] ClipSeg
  - [x] Nougat
  - [x] OwlV2
  - [x] OwlVIT

Fixes # (issue)

- https://github.com/huggingface/transformers/issues/31911

## Who can review?

@zucchini-nlp @molbap @NielsRogge 

Comments:
I'll also swap the `text` & `images` args in a separate PR

This PR should now be ready for review

Hi @leloykun ! Pinging this as blocking requirements for this PR should all be completed soon! Notably the function to check that the images and text inputs are in the correct order is merged. [Here's how it is used in LlaVa for example](https://github.com/huggingface/transformers/pull/32858/files#).

Also I think your `test_processing_common` in this PR is great, and generalizes better to most models than the current one. Would you mind opening a separate PR just for that to get it merged quickly so that other PRs can use it? 
I'd say the same for `processing_utils`.
Thanks a lot!

Hi again @leloykun , I opened a PR with some of your changes to processing_utils.py and test_processing_common.py here https://github.com/huggingface/transformers/pull/33479 .
One notable change is in test_processing_common where I replaced the testing of images kwargs with size and crop_size to do_rescale and rescale_factor, as the fact that not all image_processors accept size or crop_size.

If you think you will have the bandwidth to work on this, feel free to open a PR and I'll close mine. In any case, thanks a lot for your contributions on this!