noamaanMulla-03/transformers-issues huggingface dataset:
Change deprecated PT functions
Author: cyyever
Pull request: yes

Deprecated PT functions are replaced with counterparts.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Improve more loss computations
Author: cyyever
Pull request: yes

Combine continuous with to.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Sorry, the operations are invalid.

@cyyever I'm not sure I understand - can't `to()` be used like that?
Update w/ new account
Author: muellerzr
Pull request: yes

# What does this PR do?

Making a new account to better handle notifications

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Fix AttentionInterface following feedback
Author: Cyrilvallez
Pull request: yes

# What does this PR do?



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37010). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[generate] beam search -- fix output cropping
Author: gante
Pull request: yes
Labels: name: for patch

# What does this PR do?

VLLM is seeing some output differences in their CI when beam search is being used. The difference can be tracked to the beam search refactor (#35802).

Inspecting the outputs, we can see that there are a few additional pad tokens on the right. This is because the output was not being cropped correctly when the selected beam is shorter than the generation length (i.e. when the highest-scoring beam is NOT from the latest decoding iteration, but rather some previously completed beam). 

After #35802: output length = input length + number of decoding iterations
Before #35802 and in this PR: output length = length of the longest selected beam

This PR also changed a few beam search tests to check their special tokens, which would have prevented this bug.



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Kenlm
Author: ydshieh
Pull request: yes

# What does this PR do?

kenlm has issue, an PR is not merged yet

https://github.com/kpu/kenlm/pull/464


Let's use kenlm@git+https://github.com/ydshieh/kenlm@master

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37091). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merged as approved internally on slack
Add PlainDETR
Author: sushmanthreddy
Pull request: yes
Labels: name: New model

name: Vision

 close #27496 
 
 Still in draft ,started initial work

Comments:
Hey @sushmanthreddy, thanks for starting working on it! I hope you don't mind I renamed a PR to make it more standard and discoverable 🤗

Yes please do it

got busy with other work ,will start work within a week
Add NGen3 
Author: Thishyaketh
Pull request: yes

## This PR Fixes Issues in uploading NGen 3 to Transformers

Comments:
@ArthurZucker and @Rocketknight1 
Hope it works this time!

@Thishyaketh we need a proper description of the PR with:
- original checkpiont release link
- original checkpoint paper implementing this
and something like this to introduce the model. 
Make sure you use `transformers-cli add-new-model-like` + try to explain what is different in your model compared to llama? 🤗 

Thank you @ArthurZucker sir for the feedback I will make all the changes required and the paper + model check point link with difference between Llama and NGen 3
Add Audio inputs available in apply_chat_template
Author: junnei
Pull request: no
Labels: name: Feature request

### Feature request

Hello, I would like to request support for audio processing in the apply_chat_template function.

### Motivation

With the rapid advancement of multimodal models, audio processing has become increasingly crucial alongside image and text inputs. Models like Qwen2-Audio, Phi-4-multimodal, and various models now support audio understanding, making this feature essential for modern AI applications.

Supporting audio inputs would enable:
```python
messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "audio", "audio": "https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/examples/what_is_shown_in_this_image.wav"},
            {"type": "text", "text": "Follow the instruction in the audio with this image."}
        ]
    }
]
```
This enhancement would significantly expand the capabilities of the library to handle the full spectrum of multimodal inputs that state-of-the-art models now support, keeping the transformers library at the forefront of multimodal AI development.

### Your contribution

I've tested this implementation with several multimodal models and it works well for processing audio inputs alongside images and text. I'd be happy to contribute this code to the repository if there's interest.

Comments:
Hey! Yes, the PR is in progress and can be tracked here https://github.com/huggingface/transformers/pull/34601
Add Sdpa Support for `Electra`
Author: nnilayy
Pull request: no
Labels: name: Feature request

### Feature request

To bake Sdpa support into `transformers` for Google's Electra.



### Motivation

Towards [#28005](https://github.com/huggingface/transformers/issues/28005),
To extend support for PyTorch's Scaled Dot Product Attention across model architectures listed in transformers, including Electra, enabling faster inference and improved training performance. Helping unify attention implementations and align older models with newer performance standards.

### Your contribution

Added Sdpa Support in the linked PR below 🤗.

Comments:
Adding tokens to llama2 results in unexpected behaviour 
Author: wendlerc
Pull request: no

### System Info

transformers==4.35.2

### Who can help?

_No response_

### Reproduction

I am on transformers version 4.35.2  and get `ValueError: Trying to set a tensor of shape torch.Size([32000, 4096]) in "weight" (which has shape torch.Size([32001, 4096])), this look incorrect.
`

When I try to run:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name_or_path = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)

tokenizer.add_tokens(["<bla>"])
model.resize_token_embeddings(len(tokenizer))
out = model.generate(tokenizer.encode('The <bla> quick brown fox jumps over the lazy dog', return_tensors='pt'), max_length=50, num_return_sequences=1, pad_token_id=tokenizer.eos_token_id)
```


### Expected behavior

same as on cpu

Comments:
`model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map='auto', torch_dtype=torch.float16)
`

fixed the problem for me... weird. running on a machine with 2 small gpus.

Hey, could you share the checkpoint you were using as well? 

meta-llama/Llama-2-7b-hf


On Fri, 5 Apr 2024 at 08:29, Arthur ***@***.***> wrote:

> Hey, could you share the checkpoint you were using as well?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/30048#issuecomment-2039041974>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5YNAVVISJMND3XZB3VMSLY3ZAFRAVCNFSM6AAAAABFXLTEIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZZGA2DCOJXGQ>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


THis seems to work on my end:
```python
In [12]: model.resize_token_embeddings(len(tokenizer))
Out[12]: Embedding(32001, 4096)
```
could you use the latest version of transformers? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as it is fixed

Same issue. not fixed
rework `test_multi_gpu_data_parallel_forward`
Author: ydshieh
Pull request: no

# Description

Currently `test_multi_gpu_data_parallel_forward` is problematic and are skipped for several model testing. Many times, it looks like some cuda issue (`CUDA error: misaligned address` etc.). With other `nvidia` related stuffs (software or hardware) and/or torch versions, they might pass, fail, or pass but fail many subsequent tests.

It currently uses `nn.DataParallel` which is no longer recommended. In the long term, we should try `DistributedDataParallel` and see how this test goes.

See  #31086 for example

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Generate: support for left-padding on GPTNeoX and Llama
Author: gante
Pull request: yes

# What does this PR do?

As the title indicates, adds left-padding support for GPTNeoX and Llama.

It adds the `position_ids` input, propagates all the way to the position embedding, and gathers the position embeddings given the value in `position_ids`. All slow tests are now passing in both models, including the newly added left-padding support test and the GPTNeoX integration test.

Also makes a few changes on Llama to make it more similar to other models 🤗 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The failing CI is fixed by #22383 :)

@ArthurZucker @sgugger woopsie, I forgot that it affected the weight loading code -- I come from a place where weight names have to be specified 👼 Reverted (`self.llama` is `self.model` again)!

It appears as if this may have broken FSDP. For example, as specified in the Alpaca repo, finetuning with `--fsdp "full_sh
ard auto_wrap" --fsdp_transformer_layer_cls_to_wrap LlamaDecoderLayer` worked before this commit, but after it gives the error such as:

```python
File "/home/fsuser/.local/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 313, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/fsuser/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'position_ids'
```

Reverting the commit fixes it, although perhaps the problem is with `accelerate` not supporting `position_ids`? cc: @ArthurZucker 

@jquesnelle can you paste the full stack trace? It would allow us to find the root cause :D (maybe, as you mention, the problem is in accelerate... or maybe it comes from the Alpaca repo!)

I'm seeing a pretty significant performance hit on RedPajama-7b-chat that I think is due to this change. I ran the PyTorch profiler and all of the `repeat` operators in `apply_rotary_pos_emb` are expensive and run mostly on CPU. Reverting to transformers 4.27.x resolves the performance issue.

You should try the `main` branch, #22785 removed the repeat solving this

@gante 

```
position_ids = attention_mask.long().cumsum(-1) - 1
position_ids.masked_fill_(attention_mask == 0, 1)
```
Could you explain why you fill `position_ids` to value 1 when `attention_mask == 0`? Why the value of 1?
🌐 [i18n-KO] Translated `qwen2_vl.md` to Korean
Author: MinJu-Ha
Pull request: yes

# What does this PR do?

Translated the `qwen2_vl.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review?

May you please review this PR?
 @cjfghk5697 @garongkim @maximizemaxwell

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36750). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix pegasus init weights and other copied models
Author: jiqing-feng
Pull request: yes

The [PegasusSinusoidalPositionalEmbedding._init_weight](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pegasus/modeling_pegasus.py#L80) received a meta tensor when passing the device map, so we cannot assign values into it. But the [PegasusPreTrainedModel._init_weights](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pegasus/modeling_pegasus.py#L465) recived on-device tensor. So we can move PegasusSinusoidalPositionalEmbedding weight init to here so we can actually initialize the weight. 

The [google/pegasus-xsum](https://huggingface.co/google/pegasus-xsum) use init weight method to initialized position embedding weight so the output can be corrected.

To reproduce it:
```python
from transformers import pipeline

pipe_no_map = pipeline("summarization", model="google/pegasus-xsum")
pipe_with_map = pipeline("summarization", model="google/pegasus-xsum", device_map="auto")

print(pipe_no_map("It is done, and submitted. You can play 'Survival of the Tastiest' on Android, and on the web. Playing on the web works, but you have to simulate multiple touch for table moving and that can be a bit confusing. There is a lot I'd like to talk about. I will go through every topic, insted of making the typical what went right/wrong list. Concept Working over the theme was probably one of the hardest tasks which I had to face. Originally, I had an idea of what kind of game I wanted to develop, gameplay wise - something with a lot of enemies/actors"))

print(pipe_with_map("It is done, and submitted. You can play 'Survival of the Tastiest' on Android, and on the web. Playing on the web works, but you have to simulate multiple touch for table moving and that can be a bit confusing. There is a lot I'd like to talk about. I will go through every topic, insted of making the typical what went right/wrong list. Concept Working over the theme was probably one of the hardest tasks which I had to face. Originally, I had an idea of what kind of game I wanted to develop, gameplay wise - something with a lot of enemies/actors"))
```

Comments:
Hi @SunMarc . Please review this PR. Thanks!

> In general weight initialization should all go into the _init_weights rather than in specifc layers! If we can remove the ones from layers (breaking but more aligned) would be better!

Agree, but some tests failed after I removed `self.weight = self._init_weight(self.weight)` in `PositionalEmbedding.__init__`. I want to change some tests if it is okay.

Hi @ArthurZucker . I have fixed all relevant tests, please review it again. Thanks!

Confirmed that the provided code snippet give wrong results on main and correct results on this PR ❤️ !

Hi @ArthurZucker . Please check if I need any more tests. Thanks!
Add Mistral3
Author: Cyrilvallez
Pull request: yes
Labels: name: New model

# What does this PR do?



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36790). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Is there a reason that Mistral3 is not set to work with AutoModelForCausalLM?

Hey! Answered that `AutoModelForImageTextToText` is what we use for the multi modal model, Mistral3TextModel works with `AutoModelForCausalLM` 🤗
Fix 4090/ada not detected as having FP8 support
Author: Qubitium
Pull request: yes

# What does this PR do?

FIX 4090/Ada not detected as having valid FP8 hw support. 

Did a simple layer/module inference test with DeepSeek V3 0324 using GPTQModel and there is no issue with Transformer inference on 4090 at the layer/module level. 4090 does not have the memory to run a full all-layer generation. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @SunMarc @MekkCyber 




Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Failing test was unrelated thanks for the PR! 🤗
Paligemma connector details
Author: ytwang13
Pull request: no

https://github.com/huggingface/transformers/blob/348f3285c5114159d2ff4933b4b8ae36866d01a7/src/transformers/models/paligemma/modeling_paligemma.py#L421

Here I noticed a normalization over sqrt(text_hidden_dim), why is this?
And I do not see this line in official big_vision implementation.

Comments:
cc @molbap who added the model 

Hi @ytwang13 , this is because we want to use the Gemma implementation already existing in `transformers`, which does normalize its inputs: https://github.com/huggingface/transformers/blob/286393fbb11e3c95439ed94b01781cba632e2dfb/src/transformers/models/gemma/modeling_gemma.py#L572 

Hence we need to cancel out this normalization when using the vision features from siglip.
[MLU] Fix FA2 check error, remove deepspeed-mlu deps.
Author: huismiling
Pull request: yes

# What does this PR do?

Fixes # MLU device error in function _check_and_enable_flash_attn_2 .
Remove deepspeed-mlu dependency. # (https://github.com/deepspeedai/DeepSpeed/pull/6472) DeepSpeed 0.15.2 has supported MLU devices.
Fixes # some error tests with MLU. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
@ArthurZucker 
Hi, good day! Could you help to review this PR? All checks and tests have passed. 

@ArthurZucker @muellerzr 
Hi, this PR has been so long time .  If there is anything I can help, please let me know. thx~

@SunMarc 
Hi, Sorry to disturb. Could you help to review this PR ? Thx. 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36159). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc 
Hi, the tests decorator of tests/generation/test_fsdp.py has been switched to @require_torch_multi_accelerator .