noamaanMulla-03/transformers-issues huggingface dataset:
Fix passing str dtype to static cache
Author: guangy10
Pull request: yes

# What does this PR do?

Notice a small bug where the StaticCache is not configured correctly when passing the str type dtype throggh `integration.executorch`. The root cause is because it's using the dtype from config, which can be `str` or `torch.dtype` depending on what is passed from user code. This PR fixed the issue by switching to always use `model.dtype` (alway `torch.dtype`) instead.  With the fix, both passing `str` and `torch.dype` will work.


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker
@gante
@amyeroberts

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33741). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's get at least one more review from @ArthurZucker or @gante as they know the cache better than me

can you run `make fixup` 🤗 

> can you run `make fixup` 🤗

That works locally, the failure is `fetch_tests - Unauthorized` same as what I see in #33707

I can finally get back to the original issue where `ci/circleci: check_*` passes but `setup_and_quality` fails due to unauthorize to access `TRANSFORMERS_CONTEXT` in the `.circleci/config.yml` which was introduced in #33594. See the message from circleci support [here](https://github.com/huggingface/transformers/pull/33707#issuecomment-2383924077). @ydshieh I think we can focus on debugging and resolving this issue 

Okay force merging
Make ASR pipeline compliant with Hub spec + add tests
Author: Rocketknight1
Pull request: yes

As well as making the ASR pipeline compliant, this PR also refactors out the output test, so we can just import that in future PRs as well.

Comments:
cc @LysandreJik @osanseviero @Wauplin - there will be quite a lot of these if I do pipelines individually, maybe I should batch them into PRs with a few pipelines each?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33769). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@vignesh1507 you've been posting this message on a few PRs; please explain exactly what you mean as it feels like spam. Thank you.

I don't feel strongly about batching PRs, if it makes things simpler then let's do it yeah

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

I'll batch simpler/related ones, in that case, which should save time without making it overwhelming to review.
Avoid using context that is not accessable from external contributors
Author: ydshieh
Pull request: yes

# What does this PR do?



Comments:
Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

> Cool, so for private fork, we go though the non-context, will it still work? 🤗

public (i.e. hf/transformers): go without context --> works as it doesn't require the token to get the artifacts
private: go with context --> works because it has the required token and it's definitely our team members on it

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33866). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ydshieh Thanks for the fix!
Move weight initilization deformabledetr
Author: g-prz
Pull request: yes
Labels: name: Core: Modeling

name: Vision

name: run-slow

# What does this PR do?

Moves the weight initialization of DeformableDetr, RtDetr and GroundingDino. 

Fixes #29818 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts 🤗 

Comments:
Hey @amyeroberts 😁 
Is it all ok on your side? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33339). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @g-prz, apologies for the delay. It looks like there's one test which is failing for grounding dino. Do you have access to a GPU to be able to run, replicate and debug: 

```
tests/models/grounding_dino/test_modeling_grounding_dino.py::GroundingDinoModelIntegrationTests::test_inference_object_detection_head_equivalence_cpu_gpu
```

Hey @amyeroberts, no problem 😊 
I have a Mac and ran locally with `mps` backend. I effectively find some differences in the output tensors. I don't have many leads on what to investigate 😅  I would be pleased for some support or any idea to start with? 

Hey @amyeroberts 🙋‍♂️ 
I spent a bit more time investigating trying to understand what is happening. Apparently the test also fails on main. To what I see, the error (1.6e-3) starts to happen in one of the `Swin` layers and propagates down the net. 
In the meantime, the weight tensors seem to be identical regardless of the cpu/gpu

@g-prz Thanks for investigating and for the update! OK, if it's also failing on main then we're good to merge :) I'm going to wait until Monday to save us a late-ish weekend merge so we can catch anything quickly
Move weight initialization for DeformableDetr
Author: EduardoPach
Pull request: no
Labels: name: Should Fix

name: Good First Issue

name: Vision

### System Info

Not relevant

### Reproduction

See [Deformable Detr Modeling](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deformable_detr/modeling_deformable_detr.py#L653). 

### Expected behavior

All weight initializations should be done in `_init_weights` of the `xxxPretrainedModel` class

Comments:
c.c. @amyeroberts 

A note for anyone coming to this: this is for all DETR related models

cc @NielsRogge 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Will work on this this week

Any update @EduardoPach?

> Any update @EduardoPach?

Busy with other things lately and prioritizing my free time to more impactful contributions. I believe this issue could be tagged with `good-first-issue` though

Hello @amyerobert ! I am interested in working on that issue (as my first issue). 

I dove a little bit into the issue, if I reformulate it: 
- mainly moving the `_reset_parameters()` function into the `_init_weights()`
- there is the same issue to tackle in `GroundingDino` model

@g-prz Yep - that's right! When you're ready feel free to open a PR and ping me for review 🤗
[run_slow test fail] ViViT test_inference_interpolate_pos_encoding
Author: RUFFY-369
Pull request: no
Labels: name: bug

name: Vision

`test_inference_interpolate_pos_encoding` slow test fails for ViViT model. 

The `interpolate_pos_encoding` method has a bug so the test fails for it.

#33815  fixes this issue



Comments:
Thanks for providing a fix :star_struck: 

> Thanks for providing a fix 🤩

@LysandreJik All my pleasure :smile:
Uniformize model processors
Author: molbap
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Adds uniformized processors following #30511, #31197 in particular, and #31198 .

Adds support for:

- [x] Blip
- [x] Blip2
- [x] Bridgetower
- [x] Donut
- [x] Wav2vec2



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31368). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

https://github.com/huggingface/transformers/blob/2c8180acd9d7fc9b91f56425e281ac7237ce3305/src/transformers/processing_utils.py#L126

Also I have question in the `ImagesKwargs`. It seems that there are a lot more variables included in various model `(blip, blip-2, bridge tower, etc..)` such as 

do_convert_rgb
https://github.com/huggingface/transformers/blob/c85510f958e6955d88ea1bafb4f320074bfbd0c1/src/transformers/models/blip/image_processing_blip.py#L175
do_thumbnail
https://github.com/huggingface/transformers/blob/c85510f958e6955d88ea1bafb4f320074bfbd0c1/src/transformers/models/donut/image_processing_donut.py#L317

Does this `ImagesKwargs` accept only the intersection of all the common variables in the whole models?

@SangbumChoi the base ImagesKwargs contains as many keyword arguments as possible, but per model, it can accept anything: that's why I opted for `total=False`. 

@amyeroberts I finished digging through feature extractors for audio, not trivial, but seems it was missing something testwise. Now all seems to work :broom: :broom: 

One slow test model fails because of #33678

@molbap Re [this comment](https://github.com/huggingface/transformers/pull/31368#issuecomment-2370858557) - is this PR ready for review now?

It should! failing tests are unrelated and should be fixed in https://github.com/huggingface/transformers/pull/33678 

I fixed a concat bug I noticed in another PR, #33678 , where I fixed it for `instructblip` but it remained for `blip_2` :face_with_peeking_eye: 

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).
Don't run reminder bot for now
Author: ydshieh
Pull request: yes

# What does this PR do?

Don't run reminder bot for now as the instructions are not correct (for example, external contributors can't add label on PRs)

cc @amyeroberts @gante : we will have to think a new process



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33883). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add SAM-HQ model
Author: IskanderNakipov
Pull request: yes
Labels: name: New model

name: Vision

Adding SAH-HQ model to the repo

@amyeroberts

Comments:
Hi @IskanderNakipov, thanks for opening this PR! 

Let us know when it's ready for review 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Hiera model
Author: p1atdev
Pull request: no
Labels: name: New model

name: Vision

### Model description

Hiera is a hierarchical vision transformer that is fast, powerful, and, above all, simple. It outperforms the state-of-the-art across a wide array of image and video tasks while being much faster.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

GitHub Repo: https://github.com/facebookresearch/hiera/ (but licensed under CC BY-NC 4.0)
arXiv: https://arxiv.org/abs/2306.00989


Comments:
Can I work on this ?

@Namangarg110 Certainly! Feel free to open a PR when you're ready and ping us for review 🤗. To avoid issues from becoming too stale, we will prioritise the first open PR when reviewing over the first comment on issues. 

Thanks @amyeroberts. This is my first open-source issue. Would it be possible for you to please share any helpful resources or similar PR to understand the code structure?


Sure! 

* Docs page: https://huggingface.co/docs/transformers/en/add_new_model
* Example model PR: https://github.com/huggingface/transformers/pull/26668

Adding models is quite a big project. If you want to tackle something smaller for your first issue to get used to the workflow of contributing to transformers, resolving a [good first issue](https://github.com/huggingface/transformers/labels/Good%20First%20Issue) is a great place to start. 

Thank you for the resources, @amyeroberts.

I've begun the task and have completed 50% of the work.

I recognize that contributing a new model can be exceptionally challenging, but I am eager to give it a try. :)

Added in 
 - https://github.com/huggingface/transformers/pull/30356
move position_ids to position_embedding's device
Author: noskill
Pull request: yes



self.position_ids.device is on cpu by default and it is used when **position_ids** is None or not passed to CLIPTextEmbeddings.forward

Comments:
Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

The issue was due to position_ids being non-persisten buffer, and not loaded with load_state_dict
Supprot for qwen2moe gguf models 
Author: BrenchCC
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.0.dev0
- Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: DEEPSPEED
        - use_cpu: False
        - debug: True
        - num_processes: 24
        - machine_rank: 2
        - num_machines: 3
        - main_process_ip: gpu007
        - main_process_port: 9901
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - deepspeed_config: {'deepspeed_config_file': '/data/vayu/train/config/deepspeed/zero2.json', 'deepspeed_hostfile': '/data/vayu/train/config/hostfile', 'deepspeed_multinode_launcher': 'pdsh', 'zero3_init_flag': False}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A800-SXM4-80GB

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoModelForCausalLM

model_id = "./qwen2moe_4x1.5b/"
file_name = "Qwen2-4x1.5B-reasoning-pro-Q4_K_M.gguf" # local file, base on Qwen/Qwen2-1.5B-Instruct


model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

### Expected behavior

miniconda3/envs/vllm_cu12/lib/python3.10/site-packages/transformers/modeling_gguf_pytorch_utils.py", line 100, in load_gguf_checkpoint    
    raise ValueError(f"Architecture {architecture} not supported")
ValueError: Architecture qwen2moe not supported

Comments:
cc @SunMarc :)

or @julien-c :eyes:

Hey @BrenchCC ! Thanks for raising this issue ! I will create a community contribution issue, so we get faster support for gguf models such as qwen2moe. 

Opened this https://github.com/huggingface/transformers/issues/33260 ! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as qwen2moe is now supported
Stable Diffusion is ExecuTorch compatible
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

### Feature request

Enable Stable Diffusion to ["Export to ExecuTorch"](https://github.com/huggingface/transformers/issues/32253) workflow

### Motivation

See details in #32253

### Your contribution

Stable Diffusion model enablement

Comments:
populate quantization_config for kv-cache-scheme only configs
Author: xkazuka
Pull request: yes

# What does this PR do?
Follow up to https://github.com/huggingface/transformers/pull/31704/files.
Previously if quantization was done for the kv-cache pathway only from compressed-tensors, then quantization_config would not populate. 
With this pr, it will now populate

Fixes # (issue)
Same as above


## Who can review?

@SunMarc @younesbelkada @dsikka @mgoin 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33874). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[BUG] ModelCard do not write true optimizer
Author: relic-yuexi
Pull request: no
Labels: name: Good First Issue

This code seems to be outdated and does not consider the `optim` parameter. I might submit a PR to fix it.

https://github.com/huggingface/transformers/blob/e4522fe399641add9b1f3207923752a35ea7fdbb/src/transformers/modelcard.py#L876-L882

Comments:
Hi @relic-yuexi, thanks for reporting the issue!

We would appreciate the fix!

Hi @relic-yuexi, may i do the fix
Thank you 

Hi @relic-yuexi may i do the fix
Thankyou

Hi @relic-yuexi  , Thanks For repoting this issue , I can fix it

@nathsujal @hritikranjan1 Thanks for your interest in contribution! The issue has been already fixed, so I'm closing it!
 - https://github.com/huggingface/transformers/pull/33581
 
 Feel free to take another one, we would greatly appreciate your help!
Add GLM4 model
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Adds GLM model.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33729). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Something went wrong with the rebasing / merging as you have unrelated changes!

Yes, currently looking at it

Superseed by #33823
Handle Trainer `tokenizer` kwarg deprecation with decorator
Author: qubvel
Pull request: yes

# What does this PR do?

Handle Trainer `tokenizer` kwarg deprecation with decorator instead of manual deprecation

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Follow-up PR for 
 - https://github.com/huggingface/transformers/pull/32385


## Who can review?
@amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.