noamaanMulla-03/transformers-issues huggingface dataset:
Incorrect tokenizer created for gemma gguf files
Author: amychen85
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.0
- Platform: Linux-5.15.0-144-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.5.3
- Accelerate version: 0.34.2
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.3.1+cu121 (NA)
- Tensorflow version (GPU?): 2.17.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: NA


### Who can help?

@yijun-lee 
@Isotr0py 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer                                                              
                                                                                                 
t1 = AutoTokenizer.from_pretrained("unsloth/gemma-3-4b-it-GGUF", gguf_file="gemma-3-4b-it-Q8_0.gguf")
x1 = t1.tokenize("<bos>What is eunoia?")
print(f"{x1=}")

t2 = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")
x2 = t2.tokenize("<bos>What is eunoia?")
print(f"{x2=}")
``` 

### Expected behavior

The print out of the x1 and x2 should be the same. However,

```
x1=['<bos>', 'Wh', 'at', '▁is', '▁eu', 'no', 'ia', '?']
x2=['<bos>', 'What', '▁is', '▁e', 'uno', 'ia', '?']
```
Looking more into it, the tokenizer created for HF model (t2) is BPE while the tokenizer created for the GGUF model (t1) is Unigram.

Comments:
cc @sunmarc @mekkcyber for GGUF

Hey `@amychen85`, This is a bug on our end, likely because our GGUF metadata parser is failing to read the `tokenizer.ggml.model` key from the GGUF header, which should specify `"bpe"`. The proper fix is to correct this parsing logic in the `transformers` library to ensure the correct `GemmaTokenizer` class is used. As an immediate workaround until this is patched, you should load the tokenizer directly from the original HF repository (`"google/gemma-3-4b-it"`) and use that with your GGUF-loaded model. Thanks again for the high-quality report!

Thanks for the response.

Let me know if I should create another bug report. The eos_token and bos_token is swapped in the tokenizer created for llama gguf files.


https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/ggml.py (Line 442)
```
        self.additional_kwargs["unk_token"] = unk_token
        self.additional_kwargs["eos_token"] = bos_token
        self.additional_kwargs["bos_token"] = eos_token
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Broken documentation link in Transformers website
Author: ruheena-shaik
Pull request: no

A documentation link on the Transformers site redirects to a 404 page. It should point to the latest version of the model guide. Please verify and update the correct URL in the docs section.

Comments:
 Can you specify the exact link? We can look into it once #41459 is merged 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
'Automatic Speech Recognition' pipeline not working for Phi4 Multimodal Instruct
Author: palladium123
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.53.0
- Platform: Windows-11
- Python version: 3.10.4
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.5.3
- Accelerate version: 1.6.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.6.0+cu124 (CUDA)
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA GeForce RTX 3060 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


```
import torch
import soundfile as sf
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig, pipeline

model_path = './phi4-multimodal-instruct  #directory to local model
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, 
    torch_dtype=torch.bfloat16, 
    trust_remote_code=True,
    device_map = "cuda",
    _attn_implementation="flash_attention_2",
)
generation_config = GenerationConfig.from_pretrained(model_path, "generation_config.json")
generation_config.num_beams = 1

user_prompt = "
"
assistant_prompt = "
"
prompt_suffix = "
"

audio_path = <path_to_audio>
speech_prompt = "Transcribe the audio to text verbatim."
prompt = f'{user_prompt}
{speech_prompt}{prompt_suffix}{assistant_prompt}'

audio, samplerate = sf.read(audio_path)
inputs = processor(text=prompt, audios=[(audio, samplerate)], return_tensors="pt").to("cuda:0")
asr_pipe = pipeline(
     "automatic-speech-recognition",
     model=model,
     processor=processor,
     chunk_length_s=30  #**this is why I want to use pipeline rather than model.generate() - which works**
)

generate = asr_pipe(**inputs, max_new_tokens = 100)  #I am not sure if this would work, irrespective of the original issue. Some advice would be appreciated here, also. 
```

Traceback:

```
asr_pipe = pipeline(
AttributeError: 'NoneType' object has no attribute '_processor_class'
```


### Expected behavior

ASR pipeline to work. 
I tried using `'text-generation'` as task instead of` 'automatic-speech-recognition'`, but there does not appear a way to include the audio tokens nor` chunk_length_s` in that pipeline.....

Comments:
The behavior you are seeing is expected. The ASR pipeline only supports dedicated speech-recognition models, not models like Phi-4 Multimodal that simply allow audio as an input modality. You’ll need to manually chunk the audio and pass each chunk to the processor and model.generate().
Add EVA-02 / EVA-02-CLIP vision encoder to Transformers
Author: Aaraviitkgp
Pull request: no
Labels: name: New model

### Model description

EVA-02 and EVA-02-CLIP are state-of-the-art vision encoders released by the BAIDU / IDEA Research team.
They achieve extremely strong performance across image classification, retrieval, and multimodal benchmarks, and serve as the visual backbone in several high-performing VLMs (e.g., LLaVA-Next, Qwen-VL).


### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Code (MIT License):
	•	EVA GitHub: https://github.com/baaivision/EVA

Model weights:
	•	EVA-02-CLIP models on HuggingFace Hub:
	•	https://huggingface.co/QuanSun/EVA-CLIP


Papers:
	•	EVA-02: A Visual Representation Model for the Era of Foundation Models
                 https://arxiv.org/abs/2308.02201

Authors:
	•	BAAI / IDEA Research team

cc @molbap @yonigozlan @zucchini-nlp 

Comments:
Hmn, if they're the vision backbone for existing models, that means their code is in Transformers already!

@Rocketknight1 It is partially implemented in Qwen2VL as Qwen2VLVisionTransformer but no standalone implementation is there.

Hmmn - cc @molbap @yonigozlan @zucchini-nlp in that case, do we want support for the standalone vision encoder?

Seems to be supported in timm already, I can see Eva and Eva-2 in https://github.com/huggingface/pytorch-image-models?tab=readme-ov-file#models 👀 



@zucchini-nlp timm supports eva and eva-2 but having a native Eva02Config and Eva02Model would help, native support would allow nesting vision configuration directly inside vlm configuration which will allow the entire multimodal to be initialized,trained and saved as single unit which will allow users to use feature like gradient checkpointing and device_map="auto" usage which is complicated via timm.

Transformers already supports all the models from timm as standalone or as part of VLMs thanks to [`TimmWrapper`](https://huggingface.co/docs/transformers/en/model_doc/timm_wrapper). For ex gemma3 models' vision encoder is not implemented in transformers, we import and re-use from timm
Fix `pr_build_doc_with_comment.yml` not reporting correctly
Author: ydshieh
Pull request: yes

# What does this PR do?

The condition that determines if the run is ok or not uses the wrong job, so the reported comment was wrong.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42505). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[don't merge] check  build on other langs"
Author: ydshieh
Pull request: yes

[don't merge] 

just a check

[don't merge] 

Comments:
build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/19817713794)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42506). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
remove tokenizer warning
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes a warning that shouldn't appear in a specific case when calling AutoTokenizer.from_pretrained.  `config_tokenizer_class` and `cls.__name__` might not the same due to the alias we created.

```
# Backward-compatible alias: allow referring to TokenizersBackend as PreTrainedTokenizerFast
PreTrainedTokenizerFast = TokenizersBackend
```

So `config_tokenizer_class` ends up being `PreTrainedTokenizerFast` and when checking the class name `cls.__name__` we get `TokenizersBackend` . 

The warning is not needed anymore, so let's remove it entirely.

## Reproduction

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
```

Warning: 

```
The tokenizer class you load from this checkpoint is 'PreTrainedTokenizerFast'.
The class this function is called from is 'TokenizersBackend'.
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42483). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Handle decorator with optional arguments better
Author: hmellor
Pull request: yes
Labels: name: for patch

In https://github.com/huggingface/transformers/pull/40994 this decorator was changed so that, without arguments, it had to be used like

```py
@check_model_inputs()
def ...
```

instead of 

```py
@check_model_inputs
def ...
```

This PR fixes that so that custom models which use this decorator can continue to work.


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: afmoe, albert, apertus, arcee, aria, audio_spectrogram_transformer, aya_vision, bert, bert_generation, bitnet, blip_2, blt, camembert, cohere, cohere2, cohere2_vision

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42512). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Patch Mistral Common Tokenizer 2
Author: juliendenize
Pull request: yes

# What does this PR do?

1. Fixes the behavior of add_special_tokens to match PretrainedTokenizer goals:
- if mode is finetuning: add both bos and eos tokens
- if mode is test: add only bos tokens so that the model can generate freely.

2. Add `ValidationMode` as a string support to avoid import from mistral-common for the user.

3. Fix decode when user passes an int instead of a sequence of ints

4. Add support to batch for `decode`.

5. Add special tokens properties to list ids and tokens string representation.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

cc @patrickvonplaten for viz

@ArthurZucker and @itazap


Comments:
Fix a bug where compile_config gets deleted
Author: remi-or
Pull request: yes

Currently,  `to_dict` method of `GenerationConfig` always deletes the `compile_config` attribute. This is because when we call `save_pretrained` on a `GenerationConfig` object, we don't want to save the `CompileConfig` object.
But this create an issue where `compile_config` is deleted even when we load the model. For instance, this snippet:

```python
import torch
from transformers import AutoModelForCausalLM, CompileConfig, GenerationConfig

compile_config = CompileConfig(fullgraph=True)
generation_config = GenerationConfig(compile_config=compile_config)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", dtype=torch.bfloat16, generation_config=generation_config)

print(f"After model loading:\n{model.generation_config = }")
model.generation_config.save_pretrained("test.json")

reloaded_generation_config = GenerationConfig.from_pretrained("test.json")
print(f"After reloading:\n{reloaded_generation_config = }")

```

Running on `main` produces:
```
After model loading: model.generation_config = GenerationConfig {}
After reloading from file: reloaded_generation_config = GenerationConfig {}
```

Running after this PR:
```
After model loading: model.generation_config = GenerationConfig {
  "compile_config": {
    "backend": "inductor",
    "dynamic": null,
    "fullgraph": true,
    "mode": "reduce-overhead",
    "options": null
  }
}
After reloading from file: reloaded_generation_config = GenerationConfig {}
```

Thus we get the desired behavior of not saving the `compile_config` without deleting it as soon as we load the model.

Comments:
The fails in `tests_tokenization` and `tests_torch` seem unrelated. `test_processors` fails with the same error as on main.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix broken link to dataset food101
Author: JuanFKurucz
Pull request: yes

# What does this PR do?

The old link is returning 404, https://huggingface.co/datasets/food101 it should be https://huggingface.co/datasets/ethz/food101 I also took the liberty of changing the load datasets methods which do work to the actual user/dataset name path.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

Documentation: @stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42495). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: allow CP with trainer
Author: kmehant
Pull request: yes

# What does this PR do?

1. does batch padding to cp_size * 2 with shift_label calculation (we can omit this if you think this has to be outside the train loop)
2. adds cp accelerate context
3. loss calculation for cp and cp + dp cases


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- trainer: @SunMarc

Comments:
[loading] `tie_weights` is finally symmetric
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. This makes `tie_weights` work fully symmetrically, even in cases where one weight is tied between more than 2 keys.
This will hopefully end the pain of all models checkpoints having the saved weight in the "wrong" key.

This fixes several opened issues: https://github.com/huggingface/transformers/issues/42313 (though we still have an issue with the tokenizer here apparently), https://github.com/huggingface/transformers/issues/42460 and I think others that I lost track of.

Note: for `facebook/bart-large` bart checkpoint, all 3 tied weight keys are explicitly present in the safetensors checkpoint. So according to this PR, they are not tied anymore (with a warning), whereas they would be anyway before. IMO this is fine as if they are in the checkpoints, they were not tied when saved. We could introduce a value check if both source and targets are present and still tie if everything is equal, but I think this is too expensive and too niche to add for now. We may consider doing it if we notice this is more frequent so those `bart` checkpoints

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42507). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bart, openai
Delete irrelevant test that sometimes fails
Author: Rocketknight1
Pull request: yes

`test_tokenizer_mismatch_warning` is causing errors in `BertJapanese`. This test has been removed in the main tokenizer classes now, and I don't think it tests anything useful anymore. I also remove the skip for it in `test_tokenizer_fast` because there's nothing to skip anymore!

Comments:
cc @ydshieh

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bert_japanese

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42515). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ah, I'm sorry! I'll do it next time.
After the weight refactor, some tie weight might not have gone as expected. If the lm head was removed, then we are kinda fucked when using `from_pretrained`. 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Tie weights
```python
from transformers import AutoModel, MvpModel, MvpForConditionalGeneration
from transformers import AutoformerModel

MvpForConditionalGeneration._tied_weights_keys = {"model.shared.weight":"lm_head.weight", "model.decoder.embed_tokens.weight": "model.shared.weight", "model.encoder.embed_tokens.weight":"lm_head.weight"}
model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")

MvpModel._tied_weights_keys = {"model.shared.weight":"lm_head.weight", "model.decoder.embed_tokens.weight": "model.shared.weight", "model.encoder.embed_tokens.weight":"lm_head.weight"}
model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")
```

There are a few issues, I think we are gonna put the tie weight keys into a weight converter -> properly copies them even if the source key is unexpected?

```python
model = UMT5EncoderModel.from_pretrained('google/umt5-xxl')
```

Comments:
Refactor bert-based models to use global attention function
Author: Marcel256
Pull request: no
Labels: name: Feature request

### Feature request

Refactoring of the attention modules in bert-based models to use global attention function

### Motivation

Enabling easier support of SDPA and flash attention while minimizing code duplication in Bert copies

### Your contribution

I already created a draft PR #37494 to outline the changes required. Would love to get feedback and would continue working on this PR if needed

Comments:
is `flex_attention` planned?
https://github.com/huggingface/transformers/blob/40cba20e8781e2ac5936fca081a88493c3ce8b43/src/transformers/modeling_utils.py#L5928-L5932

I am currently considering it, but I would probably do it in a followup PR

@ArthurZucker should I continue working on this?

I think it is coming in #38108 

I dont think #38108 covers encoder only models yet

Hey there 👋 I'll probably cover encoder models in a follow-up PR to #38108 @ccdv-ai 

The necessary attention types are already covered by the refactor (bart encoder for example) - will hopefully be smoother to go through :)


For those that are interested: #38301

Yep sorry it's the follow up! 🤗 

Done in  #38301