noamaanMulla-03/transformers-issues huggingface dataset:
with pytorch cpu only version. without --no_cuda, using --bf16 will t…
Author: sywangyi
Pull request: yes

…rigger error like "Your setup doesn't support bf16/gpu. You need torch>=1.10, using Ampere GPU with cuda>=11.0"


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Library:

- trainer: @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._
fix ipex+fp32 jit trace model inference error in ipex 1.13
Author: sywangyi
Pull request: yes

error show like: “Currently the auto_kernel_selection does not support the grad mode! Please add torch.no_grad() before the inference runtime..” since jit mode only work in inference mode, it's safe to add such logic.


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
- trainer: @sgugger



Comments:
@jianan-gu 

_The documentation is not available anymore as the PR was closed or merged._
fix the issue that the output dict of jit model could not get [0]
Author: sywangyi
Pull request: yes

# What does this PR do?


Fixes # (issue)
when the model is optimized by jit.trace. and then used in pipeline inference for token classification. there is error like KeyError: 0,

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

- pipelines: @Narsil



Comments:
@sgugger @yao-matrix please help to review

_The documentation is not available anymore as the PR was closed or merged._
add low_cpu_mem_usage option in run_clm.py example which will benefit…
Author: sywangyi
Pull request: yes

… LLM loading

add low_cpu_mem_usage option in run clm example, set the option True will help reduce peak memory and loading time in LLM finetune and inference.


Comments:
@sgugger please help review

_The documentation is not available anymore as the PR was closed or merged._
add GPTJ/bloom/llama/opt into model list and enhance the jit support
Author: sywangyi
Pull request: yes

# What does this PR do?

extend the text generation to more model

- generate: @gante



Comments:
@amyeroberts please help review

@jiqing-feng @yao-matrix

_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts for context, this PR builds upon https://github.com/huggingface/transformers/pull/22265 -- an example of how to JIT trace text generation

> Thanks for iterating!

Thanks @amyeroberts  could you help to merge the PR? I check the failed case in ci. it has nothing to do with my code. 

@sywangyi Looking at the CI output, it seems that the examples test run failed before the generation tests were run. Could you rebase from main to include any recent updates? I believe the accelerate errors should now be resolved.
【Bug Report】Qwen2_5_VLVisionAttention attention_mask is None
Author: JamesCao2048
Pull request: no

https://github.com/huggingface/transformers/blob/a5c903f877fda21e739027eed133e03162eb7712/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#L250C21-L250C39

The attention_mask is None. When I use SDPA attention, the sdpa_attention_forward in transformers/integrations/sdpa_attention.py also not set.attention mask.

Comments:
Qwen-VL vision module does not need attention mask because the inputs are not padded. Internally the model packs variable length inputs, and in attention blocks we iterate over each input when SDPA attention

Not a bug, closing as resolved
Audio Flamingo3 - fix attention masking
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, it was hardcoded for "eager" attention previously prob because the contributor wanted to create mask of different length than input features. This PR uses `create_bidirectional_mask` instead and moves attention preparation to where it should be called, just before transformer blocks

Fixes https://github.com/huggingface/transformers/issues/42259 and enables FA2. Tested that FA2 generates the same thing as SDPA with 

```python
from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor
import torch

model_id = "nvidia/audio-flamingo-3-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="cuda", dtype=torch.bfloat16, attn_implementation="kernels-community/flash-attn")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What do you hear in this audio?"},
            {"type": "audio", "path": "audio.wav"},
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(dtype=torch.bfloat16, device="cuda")

outputs = model.generate(**inputs, max_new_tokens=50)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(decoded_outputs)
>> "A drum beat"
```

Comments:
run-slow: audioflamingo3

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3

This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42278). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19498440261)

### Model CI Report

#### ❌ Failed tests

- [audioflamingo3](https://github.com/huggingface/transformers/actions/runs/19498440261/job/55806951151):
    tests/models/audioflamingo3/test_modeling_audioflamingo3.py::AudioFlamingo3ForConditionalGenerationIntegrationTest::test_fixture_batched_matches
    tests/models/audioflamingo3/test_modeling_audioflamingo3.py::AudioFlamingo3ForConditionalGenerationIntegrationTest::test_fixture_single_matches



run-slow: audioflamingo3



This comment contains `run-slow`, running the specified jobs: 

models: ["models/audioflamingo3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19499447344)

✅ No failing test specific to this PR 🎉 !


thanks @zucchini-nlp and @vasqu!

Yep, slow tests are ✅  now after fixing padding. Merging!
Add support for torch device objects in device validator
Author: yonigozlan
Pull request: yes

# What does this PR do?

Cc @zucchini-nlp 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42267). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Enhance documentation for vision-language logits
Author: Nomzy-kush
Pull request: yes
Labels: name: Code agent slop

Clarified the return values and computation of logits for vision-language models in the documentation.

# What does this PR do?

This PR updates the documentation and comments for GitForCausalLM. It now accurately shows the joint vision-language logits returned by the model.

Before, the docstring and comments suggested that the logits were text-only. This was wrong for vision-language models. The logits actually include both image-token and text-token predictions from the unified sequence embedding.

### **Changes included**:

-   Updated `forward()` docstring to explain that logits include **both text and image tokens**.

-   Updated inline comment above `logits = self.output(hidden_states[:, slice_indices, :])` to clarify joint vision-language computation.

-   Ensured descriptions in `CausalLMOutputWithPast` match the new explanation.

This makes it clearer for contributors and users. It also helps avoid confusion when using the model for tasks like image captioning or VQA.

* * * * *

Fixes #1789 
from hub-docs


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
Remove doc files of other langs for deleted models
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove doc files of other langs for deleted models

Comments:
build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/19495174689)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42276). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/19500049367)
[testing] fix `cwm`
Author: ydshieh
Pull request: yes

# What does this PR do?

Missing tokens

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42261). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: cwm

This comment contains `run-slow`, running the specified jobs: 

models: ["models/cwm"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19474816067)

✅ No failing test specific to this PR 🎉 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cwm

Will merge. As discussed internally, the weight loading / offloading at this moment is still under some work and the fixes in this PR still fails with the current main.

However, it works with the commit c4cfc2e0
Enable glm46v UTs on XPU
Author: YangKai0616
Pull request: yes

# What does this PR do?

This PR enabled testing of `glm46v` on XPU.

**Notes:**

1. [General] Directly running `Glm46VIntegrationTest` currently fails with error: 
```
FAILED tests/models/glm46v/test_modeling_glm46v.py::Glm46VIntegrationTest::test_small_model_integration_test_batch_flashatt2 - RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got XPUBFloat16Type instead (while checking argumen...
```
Need to wait for `zai-org/GLM-4.1V-9B-Thinking` PR [21](https://huggingface.co/zai-org/GLM-4.1V-9B-Thinking/discussions/21) to be merged, or use the workaround: change [here](https://github.com/huggingface/transformers/blob/a5c903f877fda21e739027eed133e03162eb7712/src/transformers/models/glm46v/modeling_glm46v.py#L90) to `self.visual = AutoModel.from_config(config.vision_config.vision_config)`.

2. [XPU] Tests related to `flash_attention_2` need to wait for PR [41956](https://github.com/huggingface/transformers/pull/41956) to be merged.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm46v

cc @ydshieh

thank you 🤗
fix a typo: pbd -> pdb
Author: jaeminoh
Pull request: yes

# What does this PR do?

Fix a typo in the docstring of `output_to_pdb` method of `EsmforProteinFolding` class.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: esm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42268). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove random flag
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. No idea why this was here

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bigbird_pegasus, chameleon, deepseek_vl, deepseek_vl_hybrid, emu3, encoder_decoder, gpt2, gptj, janus, lxmert, speech_encoder_decoder, tapas, vision_encoder_decoder

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42282). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AddedToken should check content on `_update`
Author: phiott
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.54.0
- Platform: macOS-14.4-arm64-arm-64bit
- Python version: 3.11.12
- Huggingface_hub version: 0.33.4
- Safetensors version: 0.5.3
- Accelerate version: 1.9.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker, @itazap

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Hey! While doing some benchmarking, I noticed that `_update_trie` will unnecessarily call `trie.add` for each token added to the tokenizer. When adding special tokens to large vocabularies, every `AddedToken` will be re-traversed in the trie despite just adding a few special tokens.

The simplified script below shows ~vocab_size additional calls when compared to removing the last `add_special_tokens` call.

```python
import cProfile
import pstats
from transformers import BertTokenizer

tokenizer1 = BertTokenizer.from_pretrained("bert-base-multilingual-cased")
tokenizer2 = BertTokenizer.from_pretrained("bert-base-cased")

tokens_to_add = set(tokenizer2.vocab).difference(tokenizer1.vocab)

def run():
    tokenizer1.add_tokens(list(tokens_to_add)) #len(tokenizer1.get_vocab()) ~= 130k
    tokenizer1.add_special_tokens({'additional_special_tokens': ['
', '
']})

profiler = cProfile.Profile()
profiler.enable()
run()
profiler.disable()

stats = pstats.Stats(profiler)
stats.sort_stats('cumtime')  # Sort by cumulative time
stats.print_stats('Trie.add')
```

```
335064 function calls in 0.101 seconds #baseline
446404 function calls in 0.140 seconds #with add_special_tokens
```




### Expected behavior

I'd expect to see less `add` calls when adding such a small number of special tokens

Comments:
Opened https://github.com/huggingface/transformers/pull/39587 - 

```
354643 function calls in 0.128 seconds #with add_special_tokens, reduced by ~len(vocab) calls
```

compared to the above.
Fix accelerate integration
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

The `device_map` computation is currently broken, after https://github.com/huggingface/transformers/pull/42043 switched to using an integration of the accelerate functions to simplify them and use the already available `all_tied_weights_keys` instead of computing them again and again. But it does not compute the double list of `tied_parameters` correctly, in the format that the `accelerate` functions are used to, i.e. groups of all similar parameters.
Also, now that we don't tie weights before device_map computation, we need to exclude them explicitly from the loop in `compute_model_sizes`, as they will be present in the iterated params.

This PR fixes it. This is needed in https://github.com/huggingface/transformers/pull/42242, which is where I noticed it.
Also fixes a more long-standing issue, where `compute_model_sizes` was not taking non-persistent buffers into account, but they actually take some space! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42264). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: kv cache retention across conversations
Author: McPatate
Pull request: yes

Adding a conversation id to the Cookies so that we can track conversations across http requests. This enables efficient KV Cache reuse.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40801). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Can you help by documenting a tad but the class (Iike when we should call touch)

Yes sry, this was just the functioning code, will make pretty today (docs & tests)

> LGTM, I am not seeing a way to opt out (de activate TTL) probably setting it to 0 (but you still "manage" the cache.
Thanks for the clean addition!

Functionally setting the `--cache-timeout` arg to 0 will "deactivate" TTL caching. Although a `ConversationCacheManager` will still be created and there will be a little overhead wrt to having an `if cache_enabled: # only then do cache mgmt`.
I think we can leave it like so for now.

closing as we now have prefix caching #42094
Mllama fixes
Author: remi-or
Pull request: yes

This PR adds the `is_causal` attribute to some Attention modules in mllama and disables FA2 for the `MllamaVisionModel` . 
Some tests used to fail when `is_causal` was missing: `MllamaForCausalLMModelTest::test_flash_attn_2_fp32_ln`,  `MllamaForConditionalGenerationModelTest::test_eager_matches_fa2_generate`, ...and when it was added FA2 failed for these tests, on both Mi355 and A100. This is because the vision model uses a 4D attn mask which is not supported by FA2. 
Not an expert in VLM so can you chek `is_causal` is right @zucchini-nlp please? 


I also added Expectations for AMD MI355. After these changes, we go from `15 failed, 185 passed, 94 skipped` to `196 passed, 98 skipped` on AMD MI355. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39182). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mllama

Closing this as this is too old :/