noamaanMulla-03/transformers-issues huggingface dataset:
Integrate Liger (Linkedin GPU Efficient Runtime) Kernel to HuggingFace
Author: JasonZhu1313
Pull request: no
Labels: name: trainer

name: Feature request

### Feature request

Integrate Liger (Linkedin GPU Efficient Runtime) Kernel to HuggingFace Trainer, user could decide whether to enable kernel with a simple flag

### Motivation

Liger (Linkedin GPU Efficient Runtime) Kernel is a collection of Triton kernels designed specifically for LLM training. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%. The kernel works out of the box with [flash attention](https://github.com/Dao-AILab/flash-attention), PyTorch FSDP, and Microsoft DeepSpeed. We welcome contributions from the community to gather the best kernels for LLM training.

### Your contribution

We (LinkedIn) will take care of work for a smooth integration and would need HF review and feedback for changes.


### Benchmark

Benchmark conditions: LLaMA 3-8B, Alpaca Dataset, Max seq len = 512, Data Type = bf16, Optimizer = AdamW, Gradient Checkpointing = True, Distributed Strategy = FSDP1 on 4 A100s.

The throughput increases by approximately 20% with more data, but the GPU memory is reduced by 40%. This means you can train the model on smaller GPUs, with larger batch sizes, or with longer sequence lengths at no additional cost.


![image (3)](https://github.com/user-attachments/assets/779240ce-531a-48ad-84a2-570421d2a43c)
![image (4)](https://github.com/user-attachments/assets/3ac951d9-ffc0-4b61-b98a-b27ec3bfc1dc)


For more detailed benchmark setup and more exciting efficiency for multi-head training (Medusa), please refer to original repo: https://github.com/linkedin/Liger-Kernel (Repo will be public soon)

Comments:
cc @ArthurZucker @muellerzr

Sounds great! Awesome work from your team 🥳 

https://github.com/linkedin/Liger-Kernel/issues/70 Would love to have discussion on the better UX. cc @ArthurZucker @philschmid et al 

It looks like there is still an issue if using `use_liger_kernel=True` and `torch_compile=True` in Trainer with Llama: https://github.com/linkedin/Liger-Kernel/issues/174
chat_template.json is not saved when using LlavaProcessor.save_pretrained()
Author: jp1924
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.0.dev0
- Platform: Linux-5.15.0-119-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100 80GB PCIe

### Who can help?

@zucchini-nlp

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoImageProcessor, AutoTokenizer


vision_model_name_or_path = "Bingsu/clip-vit-large-patch14-ko"
language_model_name_or_path = "beomi/llama-2-ko-7b"
output_dir = "test"

tokenizer = AutoTokenizer.from_pretrained(language_model_name_or_path, chat_template="test")

processor = LlavaProcessor(
    tokenizer=tokenizer,
    image_processor=AutoImageProcessor.from_pretrained(vision_model_name_or_path),
    chat_template=tokenizer.chat_template,
)

processor.save_pretrained(output_dir)
print(processor.from_pretrained(output_dir).chat_template)
```

### Expected behavior

Hi huggingface!
As the title says, chat_template.json is not generated properly when I do LlavaProcessor.save_pretrained().

So when I do from_pretrained, the chat_template is not loaded properly.

Although there is a [section](https://github.com/huggingface/transformers/blob/main/src/transformers/processing_utils.py#L504-L510) in ProcessorMixin.save_pretrained() to save chat_template,

[ProcessorMixin.to_dict()](https://github.com/huggingface/transformers/blob/main/src/transformers/processing_utils.py#L504-L510) method is removing the chat_template.

Looking at the history, I see that this syntax was added when I added LLava-OneVision.

My guess is that chat_template.json should be saved.

Comments:
cc @Rocketknight1 as this is about chat templates

Indeed, this is caused by my latest changed. The issue is that chat template should be saved in its own file, and not saved in `processor_config.json`. But seems like removing it from `to_dict` removed chat template even before saving in `chat_template.json`

I can make a fix soon
Fix: MPS device is not being set on current implementation 
Author: mattusi
Pull request: yes

# What does this PR do?
This PR changes the behavior of the function `_setup_devices(self)` to actually return an mps device when mps is available on the current machine. 

(Also this is my first PR on this repo so if I am doing something wrong sorry! And please help me making things right

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [✅] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@muellerz and @SunMarc 


Comments:
I've also added the changes that @katherine-atwell made in her PR: #31895 , so that this PR fixes the full issue of training on a Mac with Apple Silicon

cc @muellerzr if you can review! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32355). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[i18n-ur] Added README_ur.md file
Author: akkefa
Pull request: yes

# What does this PR do?

Translate docs into Urdu language issue #33139

Documentation:
 @stevhliu @LysandreJik


Comments:
> Thanks for your translation! There are a couple of Unicode characters `&#8207;` scattered throughout the docs. Can these be removed?
> 
> Lastly, do you know anyone who would be interested in reviewing your translation? I can't read Urdu so unfortunately I can't review the content. The Discord or forums could be good places to look for a collaborator!

I will find a Urdu language collaborator. Which will review this PR.

@syedshahiq is reviewing the my pull request. He will verify the Urdu language translation. 

@stevhliu Please add @syedshahiq in the reviewer. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33199). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> LGTM, thanks for your contribution!
> 
> I think the errored CircleCI test can be resolved by following these steps [here](https://support.circleci.com/hc/en-us/articles/12504745606171-Why-am-I-seeing-the-Could-not-find-a-usable-config-yml-you-may-have-revoked-the-CircleCI-OAuth-app-message). Can you give that a try?


@stevhliu I tried the mentioned steps but its not working for me.

<img width="1204" alt="image" src="https://github.com/user-attachments/assets/77a86ba2-66d3-493a-b6d1-7d8093983b53">


cc @LysandreJik, do you know how else we can resolve this error?

> cc @LysandreJik, do you know how else we can resolve this error?

@stevhliu Any update on this PR?

Hmm, we haven't seen this error before and if the recommended steps aren't working, can you try closing this PR and opening a new one please?
Pass input_ids to _update_model_kwargs_for_generation
Author: alex-hh
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

For flexibility in adding additional kwargs to models and handling them consistently with generation utils, it might be useful if input_ids was available in _update_model_kwargs_for_generation. 

### Motivation

My current use-case involves passing a user-defined position index, separate from position_ids, and incrementing it during generation depending on which tokens are generated (e.g. if a sep token is generated, the position index resets, whereas if a standard token is generated, it increments by 1).

The most natural way for me to handle generation in such a model would be to overwrite _update_model_kwargs_for_generation and handle the incrementation index there.

However, this would require access to input_ids. Although the logic can be handled by prepare_inputs_for_generation, because this cannot update model_kwargs, it cannot be computed incrementally, making the computation more complicated.

I expect other custom use cases would also benefit from this change.

### Your contribution

If this sounds reasonable it is a simple change which I'd happily help with

Comments:
cc @gante
Decorator for easier tool building
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

We need an easier way to create tools than the heavy Tool class : but at the same time this class is useful since it's flexible. Se I just add a `@tool` decorator to make it easy to create a tool from a function, provided that it has the proper type hints and description.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33439). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @Rocketknight1 
@LysandreJik I replace all input types "text" with "string" to conform with the common JSON schema type that we also use in `apply_chat_template`.
Fix tests in ASR pipeline
Author: ylacombe
Pull request: yes

# What does this PR do?

#33509 highlighted that some slow ASR tests were never passing, due to how data was loaded. This PR mainly fixes this + fixes some slow Whisper tests that are easy to fix.

There's still 3 slow failing tests in the ASR pipeline related to Whisper, but they'll have to wait for other Whisper fixes. 
<details><summary>Remaining failing tests</summary>
<p>

```sh
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_simple_whisper_asr - AssertionError: {'tex[827 chars]4.56, 4.92)}, {'text': ' gospel.', 'timestamp': (4.92, 5.82)}]} != {'tex[827 chars]4.56, 4.92)}, {'text': ' gospel.', 'timestamp': (4.92, 5.84)}]}
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_simple_whisper_translation - AssertionError: {'text': ' A man said to the universe, Sir, I exist.'} != {'text': ' Mr. Quilter is the apostle of the middle [43 chars]el.'}
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_whisper_word_timestamps_batched - AssertionError: {'tex[81 chars] his to welcome his gospel.', 'chunks': [{'tex[898 chars]2)}]} != {'tex[81 chars] his gospel.', 'chunks': [{'text': ' Mr.', 'ti[747 chars]2)}]}

```

</p>
</details> 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33545). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added support for bfloat16 to zero-shot classification pipeline
Author: umarbutler
Pull request: yes

This PR fixes #33386 and adds support for models that output logits in `torch.bfloat16` in the zero-shot classification pipeline by casting logits to `torch.float32` before they are converted to `numpy` floats (instead of attempting a direct conversion, which is currently not possible and raises `TypeError: Got unsupported ScalarType BFloat16`).

@Rocketknight1

Comments:
@Rocketknight1 Makes sense, thanks for the improvements, I've committed them all.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33554). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix bnb dequantization 
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR fixes a small error on how dequantization is handled in bnb. We need to set `has_been_replaced` to `True` if the dequantization happens. 

Fixes an issue raised by @sayakpaul 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33546). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We do have a test to check the quality of the dequantized model. In this case, it was just the flag that was not set, hence triggering a false alarm.
Codec integration
Author: ylacombe
Pull request: yes

# What does this PR do?
cc @amyeroberts @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33565). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[i18n-ur] Added README_ur.md file
Author: akkefa
Pull request: yes

# What does this PR do?

Translate docs into Urdu language issue #33139

Documentation:
 @stevhliu @LysandreJik

Comments:
@stevhliu I’ve created a new pull request and hope to avoid any CircleCI errors this time.

@LysandreJik or @ydshieh, would you happen to know why the other two tests aren't being run?

Given the changes only affect docs, I believe we can safely ignore the circle ci docs @stevhliu 

@stevhliu Who will merge the pull request ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33461). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Given the changes only affect docs, I believe we can safely ignore the circle ci docs

Unfortunately, I can't merge it because the required tests aren't passing @LysandreJik  :(



Merged :rocket: 

For the recorder

> The CI is running under akkefa personal account. Please follow https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization- to fix it.

> If you're following the fork instead of the upstream repo

A user who submits a pull request to your repository from a fork, but no pipeline is triggered with the pull request. This can happen when the user is following the project fork on their personal account rather than the project itself on CircleCI.

This will cause the jobs to trigger under the user's personal account. If the user is following a fork of the repository on CircleCI, we will only build on that fork and not the parent, so the parent’s PR will not get status updates. 

In these cases, the user unfollows their fork of the project on CircleCI. This will trigger their jobs to run under the organization when they submit pull requests. Those users can optionally follow the source project if they wish to see the pipelines.
UnicodeDecodeError:
Author: ZhaoxinRuc
Pull request: no
Labels: name: wontfix

Traceback (most recent call last):
  File "run_classifier_br.py", line 1061, in <module>
    main()
  File "run_classifier_br.py", line 772, in main
    tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
  File "/home/luwei/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 197, in from_pretrained
    tokenizer = cls(resolved_vocab_file, *inputs, **kwargs)
  File "/home/luwei/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 97, in __init__
    self.vocab = load_vocab(vocab_file)
  File "/home/luwei/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 56, in load_vocab
    token = reader.readline()
  File "/home/luwei/anaconda3/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte


Comments:
@ZhaoxinRuc 

I'm assuming that some data in your `vocab.txt` file contains bad characters which your `codecs.py` can't decode properly. When I downloaded the pre-trained weights folder, it came with `vocab.txt` which didn't had this issue. Check if you've downloaded a wrong version or some how changed contents of `vocab.txt`.

It's also likely that your operating system actually messed with the encoding of certain characters in your `vocab.txt` file. For that, try saving the text file in 'utf8' format , just to make sure.

Hope it helps.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Similar issue, but unresolved: https://github.com/fastnlp/CPT/issues/84
Load and save video-processor from separate folder
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/33484. 

One thing to note is that we are forced to save in a separate folder, not a json file with `video_` prefix. I see a workaround here by accepting `save_filename` when saving image processorr, but seems quite against the general idea of saving/loading in transformers. So I guess we can leave it as separate folder and for this model only we'll maintain folder loading for a while, even after new video processor class is addded

cc @yonigozlan 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33562). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Will update files in the hub soon, merging for now. Loading with old format works anyway

Done! Updated and tested by loading back
rag: fix CI
Author: gante
Pull request: yes

# What does this PR do?

Fixes failing CI on `rag`, since #32564. 

From diving into the error, #32564 did not cause the error, but rather the bug it fixes probably exposed this existing configuration issue.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33578). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cache: don't show warning in forward passes when `past_key_values` is None
Author: gante
Pull request: yes

# What does this PR do?

Because of the transition from tuple of tuples to `Cache` instances, we were throwing a warning when converting `past_key_values` to the new cache format in the forward passes. 

One of those situations was when `use_cache=True` and `past_key_values is None`... but there is nothing to convert there. In fact, most of the times, the user didn't even specify the argument (see test script below). Moreover, after the transition is complete, we want to keep the default `past_key_values=None` argument.

As such, this PR removes the warning when `past_key_values=None`.

Fixes #33489

____________________________________________
Test script:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B")

inputs = tokenizer(["The quick brown"], return_tensors="pt")
gen_out = model(**inputs)
```

Before:
```
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/internal/generation_utils#transformers.Cache)
```

Now: no warning :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33541). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
VLMs: enable generation tests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Part of https://github.com/huggingface/transformers/issues/33374. This PR adds GenerationTesterMixin in all VLMs (except BLIP) and modifies tests to accept all input kwargs, not only `input_ids`. That way we can test the whole logic, including merging image and text embeddings. 

I also ran a few tests for other models, and locally nothing seemed broken

cc @amyeroberts if you want to take a look :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Will try to find the root reason for flaky tests, otherwise we'll need to override and add `is_flaky` for VLMs. Maybe related to VLMs generating special image tokens because weights are not trained and technically anything can be generated

@zucchini-nlp the PR still missed a review from a core maintainer :P 

(cc @LysandreJik for a quick look at the PR)

Omg! O_o you're right, sorry, completely missed that
fix tests with main revision and read token
Author: molbap
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fixes slow tests in mamba2 not passing because read token was missing on the decorators.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33560). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

slight overlap with https://github.com/huggingface/transformers/pull/32599 where read_token is added, I also added back some tests that were skipped before but are now passing. 
cc @amyeroberts for review :) 

Only the multi-GPU test fails as reported in https://github.com/huggingface/transformers/issues/33567, merging now.