noamaanMulla-03/transformers-issues huggingface dataset:
Add embedding scaling
Author: literid
Pull request: yes

# What does this PR do?

This PR introduces support for **embedding scaling** in the `MistralModel`. The feature is inspired by the [Scaled Embed method](https://arxiv.org/pdf/2312.16903), which demonstrates that applying a scaling factor to the embeddings significantly improves the stability of large language model (LLM) training, effectively mitigating gradient spikes.

### Key Changes:
- Adds a new configuration parameter:  
  **`embedding_scale`** (`float`, *optional*, defaults to `1.0`): A scaling factor applied to the model's embeddings.
- Updates the `MistralModel` implementation to apply the scaling factor to the embeddings during training and inference.

This implementation currently supports the **PyTorch backend**. Support for TensorFlow and Flax backends can be added in the future.

---

# Motivation

The Scaled Embed method improves training stability and helps mitigate gradient spikes, as shown in the referenced paper. By implementing this feature, we aim to bring these benefits to the `MistralModel` while maintaining backward compatibility.

---

# Open Questions for Discussion

1. **Relevance:**  
   Do you see this feature as relevant for integration into the library? Would it make sense to extend this functionality to other models, such as `LlamaModel`?

2. **Implementation Scope:**  
   Should embedding scaling be integrated more broadly across models in the library or should it remain model-specific ?  
---

Let me know if further adjustments or refinements are needed!

Comments:
Hi @literid, this is an interesting paper! However, I'm not sure it makes sense to add it to an existing model class that wasn't trained with it - it's very likely that everyone will just leave it at `1.0`.

I think we should only add this feature when a model was trained with it, to ensure it'll actually get used. This might change in future if this becomes a very widespread and popular method - then we might go back and enable it for every model.
Offloaded cache: fix generate
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/34718 and adds tests. Also had to fix some inconsistencies in the code so that the test passes for all models, for ex using `max_batch_size` instead of `batch_size` in all cache classes

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34921). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ValueError: Invalid `cache_implementation` (offloaded). 
Author: leigao97
Pull request: no
Labels: name: bug

name: Cache

### System Info

- `transformers` version: 4.46.2
- Platform: Linux-5.15.0-1049-oracle-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.25.1
- Safetensors version: 0.4.5
- Accelerate version: 1.0.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-40GB

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am following the official example to enable KV cache offloading. https://huggingface.co/docs/transformers/en/kv_cache#offloaded-cache

And I got the error message:
```
  File "/transformers/generation/configuration_utils.py", line 726, in validate
    raise ValueError(
ValueError: Invalid `cache_implementation` (offloaded). Choose one of: ['static', 'offloaded_static', 'sliding_window', 'hybrid', 'mamba', 'quantized', 'static']
```


### Expected behavior

I expected that `cache_implementation="offloaded"` is a valid option taken by `model.generate()`. After enabling KV cache offloading, the peak memory usage should go down and inference time should go up.

Comments:
hmm, indeed the latest changes to verify cache implementation in the config broke offloaded cache. You can try to pass in the cache object directly to `generate()` until it is fixed. 
```
from transformers import OffloadedCache

model.generate(**inputs, past_key_values=OffloadedCache()) 
```

cc @gante if you have this already on your radar or I can fix it next week since you'll be off for until December 

cc @ArthurZucker as well

Since @gante is not here, @zucchini-nlp can you have a look ?
Fix `utils/check_bad_commit.py` (for auto ping in CI)
Author: ydshieh
Pull request: yes

# What does this PR do?

We check a test if failing between a range commits.
If it is a new test that is not in previous commits, the status from those previous commits should be marked as `passing` (as that test doesn't exist in those commits)

Otherwise `git bisect` would fail and we don't receive any report.

This is the case in today CI with the 

> tests/models/olmo2/test_modeling_olmo2.py::Olmo2ModelTest::test_generate_compile_1_end_to_end

(see PR #34864)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34943). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Replace all torch.FloatTensor by torch.Tensor
Author: julien-blanchon
Pull request: no
Labels: name: Feature request

### Feature request

I think `torch.FloatTensor` dtype should be replace by `torch.Tensor` now. torch.FloatTensor is quite anoying as it trigger warning most of the time, and force users to manually cast type to avoid such warning.

It seems that FloatTensor is not really use anymore in Torch. torch.FloatTensor and torch.cuda.FloatTensor are still available to ensure backward compatibility

### Motivation

Fix typing warnings

### Your contribution

Replace every occurence of torch.FloatTensor by torch.Tensor. Including docstrings

Comments:
Fix docker CI : install autogptq from source
Author: MekkCyber
Pull request: yes

# What does this PR do?
Install `autogptq` from source for `pytorch==2.5.1` compatibility. I added the `TORCH_CUDA_ARCH_LIST="7.5+PTX"` to make it compile for Tesla T4 gpus available for the CI.
## Who can review ?
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35000). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Let server decide default repo visibility
Author: Wauplin
Pull request: yes

Until now, all repos were public by default when created without passing the `private` argument. This meant that passing `private=False` or `private=None` was strictly the same. This is not the case anymore. Enterprise Hub offers organizations to set a default visibility setting for new repos. This is useful for organizations forbidding public repos for security matters. This PR mostly updates some docstrings + the default value for `hub_private_repo` in Trainer's args.

This PR doesn't create any breaking change. The real update has been done server-side when introducing the new Enterprise Hub feature. Related to https://github.com/huggingface/huggingface_hub/pull/2679.

Comments:
Note: CI seems to be failing but I don't think it's related to this PR
```py
> python3 .circleci/parse_test_outputs.py --file tests_output.txt --fail
   1 failed because `AssertionError: 0.0002062409 not less than or equal to 0.0002 : outputs.logits` -> Difference between PyTorch and TF is 0.00020624090393539518 (>= 0.0002) for Data2VecVisionForSemanticSegmentation
   1 failed because `AssertionError: 1.9396484e-05 not less than or equal to 1e-05 : outputs.hidden_states_1` -> Difference between PyTorch and TF is 1.9396484276512638e-05 (>= 1e-05) for SwiftFormerModel
Number of failed tests: 2
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34999). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(can I let you merge it? I don't have permissions myself since failing CI)

Of course!
Incompatibility between transformers 4.45.0 and torch 1.9.1
Author: realjoshqsun
Pull request: no
Labels: name: PyTorch

name: dependencies

name: bug

### System Info

Hello, I was using depth estimation model,
`pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf")`
But I got this error:
```
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf")
  File "/home/q84sun/miniconda3/envs/vlnce_py3.8/lib/python3.8/site-packages/transformers/pipelines/__init__.py", line 896, in pipeline
    framework, model = infer_framework_load_model(
  File "/home/q84sun/miniconda3/envs/vlnce_py3.8/lib/python3.8/site-packages/transformers/pipelines/base.py", line 288, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "/home/q84sun/miniconda3/envs/vlnce_py3.8/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/q84sun/miniconda3/envs/vlnce_py3.8/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3808, in from_pretrained
    state_dict = load_state_dict(resolved_archive_file)
  File "/home/q84sun/miniconda3/envs/vlnce_py3.8/lib/python3.8/site-packages/transformers/modeling_utils.py", line 556, in load_state_dict
    return safe_load_file(checkpoint_file)
  File "/home/q84sun/miniconda3/envs/vlnce_py3.8/lib/python3.8/site-packages/safetensors/torch.py", line 315, in load_file
    result[k] = f.get_tensor(k)
AttributeError: module 'torch' has no attribute 'frombuffer'
```

It seemed like a compatible issue between transformers and torch. What is the right torch version to match transformers 4.45.0?

My environment:
ubuntu: 22.04
Python: 3.8.20
torch: 1.9.1+cu111
transformers: 4.45.0
nvcc: cuda_11.7

### Who can help?

@amyeroberts @qubvel @Rocketknight1 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I was using the official example scripts from https://huggingface.co/depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf

```
from transformers import pipeline

pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf")
```

When running this test script with the my own environment, it raised up an error.

### Expected behavior

Should be working.

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/25847

The pytorch version 1.9.1 was released >2 years ago and therefore isn't supported by newer libraries. The problem here doesn't stem with `transformers` but with the `safetensors` library; the `safetensors` maintainer posts in the thread mentioned above that, unfortunately, supporting 1.9 would be a significant chunk of work.

Thanks. I tried torch 1.11 and it worked.
[i18n-ar] Translated file : `docs/source/ar/torchscript.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/torchscript.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs 

The following pull requests have been added to the current pull request to be merged once:

- [x]  Troubleshoot #33017

@abodacs  @stevhliu 

Thank you

أخي عبدالله @abodacs  هذا ما تم إنجازه في هذا القسم 
إلى الامام إن شاء الله



# DEVELOPER GUIDES

- [x]  Use fast tokenizers from 🤗 Tokenizers #33034
- [x]  Run inference with multilingual models #33048
- [x]  Use model-specific APIs #33030
- [x]  Share a custom model #33031
- [x]  Templates for chat models #33026
- [x]  Trainer #33080
- [x]  Run training on Amazon SageMaker #33071
- [x]  Export to ONNX #33072
- [x]  Export to TFLite #33077
- [x]  Export to TorchScript #33079
- [ ]  Benchmarks #33023
- [ ]  Notebooks with examples #33049
- [ ]  Community resources #33027
- [x]  Troubleshoot #33017
- [x]  Interoperability with GGUF files #33037


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33079). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

All Done!  @stevhliu 

Thank you
[i18n-ar] Translated file : `docs/source/ar/benchmarks.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/benchmarks.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33023). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @stevhliu
change inputs_embeds to input_embeds
Author: yiqiao-BUAA
Pull request: no
Labels: name: Feature request

### Feature request

Change the inputs_embeds in the LlamaModel(Maybe others also) input parameters to input_embeds

### Motivation

Our code work should be based on one of these two, and we will also switch from A to B or vice versa, and the current difference in the prefixes of the two makes the experience of this process not so good. This process can be made more convenient with simple modifications.

### Your contribution

I can help to submit a PR for it

Comments:
Hi @Luohongzhige, although I agree "input_embeds" is a little more grammatically correct, "inputs_embeds" is the standard in the codebase and used by all of our models. You can standardize on "inputs_embeds" and it should work with any Huggingface model.

Thank you for your reply，but I'm still pity that we won't push for such a change that make it more comfortable to use😭
🚨🚨🚨 Uniformize kwargs for TrOCR Processor
Author: tibor-reiss
Pull request: yes
Labels: name: Vision

name: Multimodal

name: Processing

Adds uniformized processors for TrOCR following #31911 

Added tests (which also pass in main branch).

@qubvel @molbap

Comments:
Thanks for working on this @tibor-reiss!
[DO NOT MERGE] Update timm version
Author: qubvel
Pull request: yes

# What does this PR do?

Bumps timm version


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34996). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update timm version
Author: qubvel
Pull request: yes

# What does this PR do?

Bump timm version to use `forward_intermediate` in `TimmWrapper`
 - https://github.com/huggingface/transformers/pull/34564
 - https://github.com/huggingface/transformers/pull/34996 (CI images build here)

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc @ydshieh 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@ArthurZucker please have a look at the docker image update

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35005). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.