noamaanMulla-03/transformers-issues huggingface dataset:
Any plans to integrate GTE model natively into transformers
Author: yaswanth19
Pull request: no
Labels: name: New model

### Model description

Any plans to integrate `gte` model natively into transformers as right now we are using this model with `trust_remote_code=True` argument

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation
Model Implementation: https://huggingface.co/Alibaba-NLP/new-impl/blob/main/modeling.py
Model Weights: https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5

Comments:
@ArthurZucker @Rocketknight1 If we do intend to integrate this model then I can work on creating a draft PR.

Is there a place, I can help you to add the model @yaswanth19 ?

@ArthurZucker A gentle ping. 
@tomaarsen Can this also be integrated into `sentence-transformers` 

This seems popular enough to justify an integration, yes. WDYT @tomaarsen?

@Rocketknight1  
I suspect there are 3 popular and promising models built on this architecture:
* https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5
* https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5
* https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0

Beyond that, the authors are now using another implementation on top of Qwen:
* https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct
* https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct

Some of the mechanisms are similar to ModernBERT (I see unpadding), but some differ as well (xformers). It might require a good bit of effort to get everything to line up with `transformers`, and I think there's a chance that there will be no more big models based on this architecture.

- Tom Aarsen

@Rocketknight1 Should I start implementing support for this model, or do you think the effort outweighs the potential benefit and keep using these models with `trust_remote_code`?

Hi @yaswanth19, given @tomaarsen's comment above, I think it's okay to leave them as `trust_remote_code` models, especially if future versions of `gte` already exist on a different architecture.

On this topic, the Alibaba team actually just released superior modes based on the new ModernBERT architecture today:
* https://huggingface.co/Alibaba-NLP/gte-modernbert-base
* https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base

I imagine that they might not move forward with their previous architecture, especially considering they mention that the only parameter they changed for these compared to their previous models was the base model. 

- Tom Aarsen
[Backend support] Allow `num_logits_to_keep` as Tensor and change it to `logits_to_keep` + add flag
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Allowing `num_logits_to_keep` as a Tensor allow efficient slicing when using packed tensor format. It will be useful for us in the future as well as we integrate packed format for FA2 path.


Comments:
Fix more CI tests
Author: ArthurZucker
Pull request: yes

# What does this PR do?
For now only gemma2 but having a look at recurrent Gemma, gemma and others!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35661). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typo
Author: SilverSoldier
Pull request: yes

# What does this PR do?

Fixes typo

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
Yep, that's definitely a typo. cc @muellerzr @sunmarc to confirm that we won't break anything downstream by fixing it!
Fix set_initialized_submodules bugs + improve asymptotic runtime
Author: Rocketknight1
Pull request: yes

Our old `set_initialized_submodules` function had this line:

```python
loaded_keys = {k.replace(f"{module_name}.", "") for k in state_dict_keys if k.startswith(f"{module_name}.")}
```

This line has two problems. Firstly, it's buggy - it can replace the substring `"{module_name}."` **even if it does not occur at the start of the string.** In testing, this was mangling keys in several models, particular models which had a key on the base model object with a name like `"embeddings"`.

The second problem is performance: this line iterates over every key in the state dict. That iteration is O(N), where N is the number of weights in the state dict, and is executed once per `module` in the model. Since there are O(N) modules as well, the overall function is O(N^2). As is characteristic of O(N^2) functions, they're fine until inputs get sufficiently large and then their runtime totally blows up, which [happened recently with DeepSeek-v3](https://github.com/huggingface/transformers/issues/35635)

This PR replaces the full list scan with a sort, O(NlogN), followed by binary search which is O(log N), and then iterates forward until the prefix no longer matches. Therefore, the scan is now O(log(N) + k), where k is the average number of weights in a module, and the overall runtime is O(NlogN + Nk), which is much faster than O(N^2). It also fixes the incorrect replacements bug by using `removeprefix()` instead of `replace()`.

Fixes #35635

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35698). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @arthurzucker for review!

Closing because #35493 is a cleaner (and probably faster!) fix for this
Exploring use of kwargs for timm model and transforms creation
Author: rwightman
Pull request: yes

# What does this PR do?

Allow kwargs to be passed through to timm create_model and create_transforms fns via the TimmWrapperModel and TimmWrapperImageProcessor classes respectively. 

Current exploring use and not ready to finalize. Would like to allow flexibility to configure model and transforms at runtime for added flexibility.

Examples:

Allow dynamic image size for vits
`model = AutoModelForImageClassification.from_pretrained("timm/vit_base_patch16_224.augreg2_in21k_ft_in1k", dynamic_img_size=True)`

Change image size of pre-processor
```
model = AutoModel.from_pretrained("timm/vit_large_patch14_dinov2.lvd142m", dynamic_img_size=True)
proc = transformers.AutoImageProcessor.from_pretrained("timm/vit_large_patch14_dinov2.lvd142m", img_size=448)
```

Customize image transforms, this will enable auto_augment and rand-erasing in train transforms and use the train resolution instead of test target size for eval transform
```
proc = transformers.AutoImageProcessor.from_pretrained("timm/resnet50.a1_in1k", use_train_size=True, auto_augment='rand-m9-inc1-mstd101', re_prob=0.3)
```

Comments:
I though this could also be neat to wire through in the image classification example script, though not quite sure how to add the args to HfArgumentParser, would be least obtrusive if a --model-kwargs and --preprocessor-kwargs could be added that'd pass through lists of key-value pairs as kwargs to the AutoModel and AutoImageProcessor...

Not sure who else would have opinions re the example scripts, this could allow all the advanced timm augmentations, and also enabling of some model regularization like stochastic depth (e.g. `drop_path=0.2`) for training through transformers.  @merveenoyan @NielsRogge 

I added properties to the image processor so use cases where people were read image_processor.size, image_processor.image_mean, etc will work. autotrain relied on this and now it works with timm models.

Great! Can you please add some test cases to `tests/models/timm_wrapper/`? And would be great to add some code snippets to the docs `timm_wrapper.md` to make sure it's clear how to modify image processor/model parameters.

> Great! Can you please add some test cases to `tests/models/timm_wrapper/`? And would be great to add some code snippets to the docs `timm_wrapper.md` to make sure it's clear how to modify image processor/model parameters.

I'll get to this eventually, I wanted to explore the use cases a bit more with some notebooks, train runs and see if any other changes are worthwhile.
set_initialized_submodules too slow when loading big model like DeepSeekV3
Author: hongchuan666
Pull request: no
Labels: name: bug

### System Info

transformers==4.44.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Load a model that has 40000+ nn.Modules （etc. DeepSeekV3），then you will find it cost too much time on set_initialized_submodules.

### Expected behavior

This function is O(N^2) and should be optimized.

Comments:
Yep, it's definitely O(n^2). Let me see if I can whip something up!

Fixed by #35493
fix apply_chat_template() padding choice
Author: baoyf4244
Pull request: yes

# What does this PR do?

fix apply_chat_template() padding choice to bool, str, PaddingStrategy and the docstring of pad()


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
I find in the encode() method, it will pass padding to encode_plus() method,  and in this method, it will process padding through _get_padding_truncation_strategies() method. This method do the same with the docstring. And this arg will be finally used in pad() method, which has the same docstring with this arg.
[Question] Why doesn't `trainer.state.epoch` fall round after training?
Author: qgallouedec
Pull request: no
Labels: name: trainer

```python
# run.py
from datasets import Dataset
from transformers import TrainingArguments, Trainer, AutoModelForCausalLM


def main():
    model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B")

    dataset = Dataset.from_dict(
        {
            "input_ids": [[1, 2, 3] for _ in range(260)],
            "labels": [[4, 5, 6] for _ in range(260)],
        }
    )

    trainer = Trainer(
        model=model,
        args=TrainingArguments(
            output_dir="my_output_dir",
            per_device_train_batch_size=16,
            gradient_accumulation_steps=2,
            num_train_epochs=1,
            report_to="none",
        ),
        train_dataset=dataset,
    )

    trainer.train()
    print(trainer.state.epoch)  # 0.9411


if __name__ == "__main__":
    main()
```


```
python run.py
```

In this case, I would expect `trainer.state.epoch` to be 1 after the training, but I end up with 0.9411 (=16/17). How to explain this?

@muellerzr 

## System info

- `transformers` version: 4.47.0.dev0 22834eeba1c2bf8d632e22fca238ab7c15d1b904
- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.11.10
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.2.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3

Comments:
Try adjusting your data set to ensure that the epoch reach 1

```
dataset = Dataset.from_dict(
    {
        "input_ids": [[1, 2, 3] for _ in range(288)],  # changes from 260 to 288 
        "labels": [[4, 5, 6] for _ in range(288)],
    }
)
```


This won't give any leftover, give it a shot 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Handle num_items_in_batch in Mistral's forward
Author: gheinrich
Pull request: yes
Labels: name: bug

# What does this PR do?

This PR enables handling loss keyword arguments in the Mistral model's `forward()` method.
Specifically, if `num_items_in_batch` is passed, the value is used to properly normalize the loss value.
    
This relates to the Gradient Accumulation fix (#34191)
    
Fixes #34575

cc @ArthurZucker as it relates to text models.


Comments:
cc @muellerzr for the GA fix as well!

Hello, other models have the loss function defined in a parent class. Mistral models have it defined in the `forward` method. If I don't want to change this behavior, how do you suggest I proceed?

You need to pretty much copy-paste the code in Llama / other models 🤗 The parent class is the same for Mistral as well

Will be superseded/fulfilled with https://github.com/huggingface/transformers/pull/35875
Support Shared Cache
Author: LoserCheems
Pull request: no
Labels: name: Feature request

name: Cache

### Feature request

A new cache class that supports sharing the same or part of the KV cache between different layers to improve cache efficiency.

### Motivation

Many studies have shown that attention weights between different attention layers are always similar, and `KV cache sharing` only causes a small quality degradation, while improving **2~3 times token/sec**.

### Your contribution

I would try to submit a PR.

Comments:
cc @gante for vis
use torch.testing.assertclose instead to get more details about error in cis
Author: ArthurZucker
Pull request: yes

# What does this PR do?
This should help when we have red CIs + is good practice

Comments:
Good! But wondering if we really need to have the new `rtol` in some place. (I personally would probably just set it to 0.0 so the comparison is just the absolute difference)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35659). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

we juste have to put rtol , its required. We can put it to 0 if it causes issues!
add xpu device check in device_placement
Author: faaany
Pull request: yes

## What does this PR do?
To align with other hardware accelerators, we need to add XPU device check for the following code:
```python
from transformers import pipeline
pipe = pipeline(model="facebook/opt-1.3b", device=2)
with pipe.device_placement():
    output = pipe("This is a cool example!", do_sample=True, top_p=0.95)
print(output)
```

@ydshieh 

Comments:
@Rocketknight1 Since it's pipeline and the change is tiny and straightforward, so I ping you for a second (and hopefully the final) eye 🙏
Add `Rocketknight1` to `self-comment-ci.yml`
Author: ydshieh
Pull request: yes

# What does this PR do?

Sorry @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35881). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
if split_special_tokens==True，fast_tokenizer is slower than slow_tokenizer
Author: gongel
Pull request: no
Labels: name: bug

### System Info

if split_special_tokens==True，fast_tokenizer is slower than slow_tokenizer

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import LlamaTokenizer, LlamaTokenizerFast
import time
tokenizer1 = LlamaTokenizer.from_pretrained("./Llama-2-7b-chat-hf", split_special_tokens=True) # LlamaTokenizer
tokenizer2 = LlamaTokenizerFast.from_pretrained("./Llama-2-7b-chat-hf", split_special_tokens=True) # LlamaTokenizer
print(tokenizer1, tokenizer2)

s_time = time.time()
for i in range(1000):
    tokenizer1.tokenize("你好，where are you?"*100)
print(f"slow: {time.time() - s_time}")

s_time = time.time()
for i in range(1000):
    tokenizer2.tokenize("你好，where are you?"*100)
print(f"fast: {time.time() - s_time}")
```
output: 
        slow: 0.6021890640258789
        fast: 0.7353882789611816

### Expected behavior

-

Comments:
cc @arthurzucker @itazap 

Hey! This is not related to `split_special_tokens`! 
See #25873 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

thx~
ZeroShotClassificationArgumentHandler should be explicit it has a somewhat unsafe internal behaviour.
Author: nicolasdalsass
Pull request: no
Labels: name: Feature request

### Feature request

Currently, `ZeroShotClassificationArgumentHandler::__call__` will execute https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/zero_shot_classification.py#L41 , that is, it will call python `.format()` on the hypothesis provided to format the label in it, while allowing the full extent of .format() placeholders syntax, which is quite large.

For example, passing `hypothesis_template = "{:>9999999999}"` and any label will happily eat 100Go of RAM because the whole scope of python formatting is allowed.

This is not made clear annywhere, but library users need to know they have to sanitize those inputs very carefully.

I think that at least the docstring of the class, and ideally the reference doc for "hypothesis_template" on https://huggingface.co/docs/huggingface_hub/package_reference/inference_client#huggingface_hub.InferenceClient.zero_shot_classification should be updated to mention this, it's quite important for users of the lib (in particular for parameters that will naturally tend to be user facing in the end).

Alternatively, this call could accept {} only as a placeholder, it's hard to see a legitimate use case for exotic formatting of labels in the hypothesis template.

Thanks :-) 

### Motivation

I think it's good to help the internet be a safer place in general :-) 

### Your contribution

It's unclear to me whether I can contribute to the documentation on hugginface.com.

I could contribute a fix to be stricter on allowed hypothesis_template in transformers though if you want to take this route (I'm pretty sure even an AI model could contribute the two lines needed though...)

Comments:
Hi @nicolasdalsass, this is a good point! I don't think the intention here was to allow the full range of Python formatting behaviour, it was just intended as a simple string insertion. If you'd like to make the PR to tighten up security here, I think we'd be happy to review/accept it. It's up to you whether you use an AI or write it yourself 😅 

@Rocketknight1 saw this issue and raised a fix in the PR #35886  . Hope you can review this and merge if issue is resolved. I'll acknowledge any comments you have on it as well if required . 

Thank you @nicolasdalsass for raising the issue . 🤗

@sambhavnoobcoder Thanks for the very quick PR on the issue :-) I had a look, only allowing {} seems good and safe to me :-) 

It was my pleasure . I've been actively looking through open issues in the transformers library and contributing PRs where I can help. Looking forward to seeing this merged :-)
Links in release note are broken
Author: oraluben
Pull request: no
Labels: name: bug

This should be a github-config issue

@ArthurZucker 

### Reproduction

<img width="1034" alt="image" src="https://github.com/user-attachments/assets/0c27855f-bc4f-469d-b94c-62e0b84cbdc2" />


### Expected behavior

links point to correct PR

Comments:
Sorry just fixed it!

@ArthurZucker the links seems still wrong, it should be github/huggingface/transformers, not github/transformers, and there's also some left unchanged :)