noamaanMulla-03/transformers-issues huggingface dataset:
Main
Author: vukasin
Pull request: yes

# Speed up PrefixConstrainedLogitsProcessor

The `__call__` method creates an empty tensor and then updates it repeatedly. This causes a significant slowdown when executing on a GPU since it requires updating the GPU memory over and over. This pull request creates the mask in CPU memory and then creates a tensor on GPU with one call. The tests I ran (the test case from https://github.com/worldbank/REaLTabFormer/tree/main/colab) has a speed increase of over 200% up to depending on the batch size, 300%. 


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
inconsistent generation
Author: slatter666
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.2
- Python version: 3.8.18
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.1
- Accelerate version: 0.32.1
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- GPU type: NVIDIA A10

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction


I used the same input, but changed the code logic slightly, resulting in different results

here is the context of code(mainly load model)
```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig, DynamicCache

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_path = "meta-llama/Meta-Llama-3-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_path, attn_implementation="flash_attention_2", device_map=device).eval()
tokenizer = AutoTokenizer.from_pretrained(model_path)

encoded_input = tokenizer("what is your name", return_tensors='pt').to(device)
window_size = 1
front_input = {key: value[:, :-window_size] for key, value in encoded_input.items()}
rear_input = {key: value[:, -window_size:] for key, value in encoded_input.items()}
```

and here is the first generation code
```
past_key_values = DynamicCache()
generation = model.generate(**encoded_input, past_key_values=past_key_values, max_new_tokens=32, do_sample=False)
generation = tokenizer.batch_decode(generation)[0]
print(generation)
```
the generation is as below:
```
what is your name?" and "what is your occupation?" are not necessary. The form is designed to be as simple and easy to fill out as possible, while still gathering the
```

and the seconde generation code is:
```
past_key_values = DynamicCache()
with torch.no_grad():
  _ = model(**front_input, past_key_values=past_key_values, use_cache=True)
generation = model.generate(**encoded_input, past_key_values=past_key_values, max_new_tokens=32, do_sample=False)
generation = tokenizer.batch_decode(generation)[0]
```
the generation is as below:
```
what is your name?" and "what is your occupation?" are not necessary. The form is designed to be as simple and easy to fill out as possible, so that you can
```

### Expected behavior

well, it's weird, I think these two generation process is the same since I do not use sampling, but why the results are different. Is there anything wrong with my operation?

Comments:
but when I change to use A100, the result is the same, OMG why is that

Hey @slatter666 ,

Since you are generating with precomputed cache of size `window length` in one of the examples, while in another you generate from the whole input, it might lead to tiny numerical precision errors.

See https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535 for more on why caching can accumulate numerical precision errors

Thank you so much, that solves my issue.
Improve object detection task guideline
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes some issues mentioned in #29964 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29967). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts could we already merge this PR which addresses the first 3 points of https://github.com/huggingface/transformers/issues/29964?

@NielsRogge Did you run and check re 100 epochs? https://github.com/huggingface/transformers/pull/29967#discussion_r1551806857 If not, then we should revert back until confirmed 

With 100 epochs it converges better but still has pretty low metrics. Also, predictions are bad because we use it for inference with batch size = 1.
We can align it with https://github.com/huggingface/transformers/pull/30422 when it's merged to get better metrics.

@amyeroberts people have reported to replicate it with 100 epochs here: https://github.com/huggingface/transformers/issues/30557#issuecomment-2086401493

Feel free to approve the PR in order to clarify this for now in the docs. Later on we could improve metrics more
🚨🚨🚨 [SuperPoint] Fix keypoint coordinate output and add post processing
Author: sbucaille
Pull request: yes
Labels: name: bug

name: Vision

# What does this PR do?

There is a bug highlighted by @merveenoyan where SuperPoint returns absolute keypoint coordinates at a wrong scale.
For example, an image of size (1000, 1000) will first be resized into (640, 480) by the image processor but SuperPoint will output keypoint coordinates between 0 and 640 for x and 0 and 480 for y.
This PR fixes this bug by returning keypoints in relative coordinates out of the SuperPoint forward method.
An additional ``post_process_keypoint_detection`` is added to the ``SuperPointImageProcessor`` to cast the keypoint coordinates into the original image sizes.
I also added an "unwrapping" logic where the output will be a list of ``keypoints, scores, descriptors`` for each image so that the user don't have to deal with the current mask mechanism. This is conditionned by a boolean argument in the signature.

I have not finished all the details but most of the implementation is there and I will add tests tomorrow but at least the PR is open as I promised Merve ☺️

Fixes #33825


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. 
-> On Slack
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@amyeroberts @NielsRogge @merveenoyan 

Comments:
@amyeroberts comments addressed, tests are passing, let me know if I can do a final rebase for the merge

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33200). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @sbucaille, thanks for working on this and addressing comments, I will review this week!

@qubvel thanks for the review, I've addressed your comments. I initially forgot to change the tests to reflect the changes in terms of SuperPoint's outputs, so this is also fixed. Let me know if there is anything else to do

Hello, I was following this PR and curious that is it going to merge or any change still require ? 

Hi @onuralpszr ,
Thanks for asking, there is no change required on my side !

@sbucaille thanks a lot for your contribution!
[i18n-zh]Translated perf_train_special.md into Chinese
Author: blueingman
Pull request: yes

# What does this PR do?
translated perf_train_special.md into Chinese

Part of https://github.com/huggingface/transformers/issues/26803


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


@stevhliu @Isotr0py Would you mind taking a look when you have time?Thank you so much for your help again!




Comments:
> Overall looks good, just some nits.

Thank you for your careful suggestions!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34948). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[i18n-zh]Translated tiktoken.md into chinese
Author: blueingman
Pull request: yes

# What does this PR do?

translated tiktoken.md into Chinese


Part of https://github.com/huggingface/transformers/issues/26803


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@stevhliu @Isotr0py  Would you be kind enough to review the PR when you have time? Any feedback or suggestions you have would be greatly appreciated.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34936). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
translated gguf.md into chinese
Author: blueingman
Pull request: yes

# What does this PR do?
I am new to the open-source community, but I’m very eager to contribute to the Transformers project. I have translated the gguf.md file into Chinese. If there are any areas that need improvement or if I overlooked anything, please let me know


## Before submitting
- [y] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [y] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.




Comments:
Nice! Maybe @Isotr0py would you be willing to review this PR? :)

OK, I will take a look tonight. :)

I have updated the PR accordingly.Thank you very much for detailed guidance,and I 'll pay more attention to the details next time. @Isotr0py @LysandreJik 

Thank you again for your careful suggestion!

Hi @Isotr0py ,hope you're doing well. I have addressed the changes you suggested and updated the PR accordingly. When you have a moment, could you please review the updated code? Thank you for your time!

The updated docs LGTM. :)

Also cc @LysandreJik
Aggeregate test summary files in CircleCI workflow runs
Author: ydshieh
Pull request: yes

# What does this PR do?

See the new `collection_job` outputs in the artifacts tab.

This could ease the searching if a test is run or not, run in which jobs, their status in each job etc.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34989). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's fix the test to merge!

will merge on Monday instead :-) Thanks for approval
Blip: fix offloading and MP tests 
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, after adding `no_split_modules` some tests started failing (those that were prev skipped). This PR fixes them

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35239). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix : model used to test ggml conversion of Falcon-7b is incorrect
Author: MekkCyber
Pull request: yes

# What does this PR do?
The model used to test the ggml conversion of `falcon-7b` in `fp16` format is wrong : 

<img width="1259" alt="image" src="https://github.com/user-attachments/assets/47ee7ff2-e8dd-40d5-83ba-6e056810f371">
You can see that it contains some Q4 weights which is unexpected in a `fp16` model, and its size is only 4GB but it should be around 7x2 = 14GB. 
I did my own model conversion to gguf to fix the issue : 

<img width="1261" alt="image" src="https://github.com/user-attachments/assets/5fcccb76-e8ae-4678-8ced-326b7e299d65">

## Who can review ?
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35083). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Temporarily disable amd push ci
Author: ivarflakstad
Pull request: yes

# What does this PR do?
Temporarily disable the AMD push CI to reduce noise when debugging

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35293). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Whisper] patch float type on mps
Author: eustlb
Pull request: yes

# What does this PR do?

Fixes https://github.com/Vaibhavs10/insanely-fast-whisper/issues/255

https://github.com/huggingface/transformers/pull/34537 introduced casting to float64, which is not supported on the MPS backend.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35295). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Only Fine-tune the embeddings of the added special tokens
Author: ys-zong
Pull request: no
Labels: name: Feature request

### Feature request

Hi, I added some new special tokens to the LLMs (specifically I'm using Qwen2-VL) and then I only want to fine-tune the embedding layers of these added tokens while keeping all other parameters (and the embedding layers for other tokens) frozen. I wonder if there is a built-in way to do so instead of fine-tuning the whole embedding matrix?


### Motivation

If we want to maximumly retain the original capabilities of the model while adding new tokens for certain scenarios, this might be needed, especially when we don't have much data and do not want to alter the pretrained weights.

Another question: if we have a considerable amount of data, is it recommended to fine-tune the whole embedding matrix or only the embeddings for the added tokens?

### Your contribution

If it's a reasonable feature and not implemented yet, I'm happy to submit a PR.

Comments:
Just fyi, idefics already has a similar layer for freezing some embeddings. Maybe you can change the embedding layer in Qwen2-VL with the below module and thus freeze unwanted token ids when training

https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/models/idefics/modeling_idefics.py#L201-L208

I also need this feature!

I see a way to implement it by masking out gradients for all tokens except the newly added ones. However, with Zero-deepspeed it is complicated (see this issue - https://github.com/microsoft/DeepSpeed/issues/5199). Maybe someone has found another way to do it (preferably without changing model architecture with new classes as @zucchini-nlp suggested)?
Translating "translate perf_infer_gpu_multi.md" to Chinese
Author: HMJ0628
Pull request: yes

# What does this PR do?
I have translated translate perf_infer_gpu_multi.md to Chinese. Please let me know if there are any corrections. Thank you very much!
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
@asdkfjsd Please help me review it.Thank you!

The translation is pretty good!I can understand it easily!

cc @stevhliu
Enhance documentation of Automatic speech recognition
Author: Uvi-12
Pull request: no

The ASR documentation requires improvements in grammar, phrasing, and consistency to ensure clarity and readability. Suggested updates include:

- Replacing informal language (e.g., "this'll" to "this will").
- Correcting grammatical errors (e.g., "everyday" to "every day").
- Refining phrasing to match standard technical documentation style (e.g., "focuses on" instead of "you'll focus on").

Comments:
cc @stevhliu
fix modular order
Author: ArthurZucker
Pull request: yes

# What does this PR do?
The order of modular conversion was wrong. Now we do have: 

```bash
No differences found for src/transformers/models/gemma/configuration_gemma.py.
No differences found for src/transformers/models/gemma/tokenization_gemma.py.
No differences found for src/transformers/models/gemma/modeling_gemma.py.
No differences found for src/transformers/models/llava_next_video/configuration_llava_next_video.py.
No differences found for src/transformers/models/llava_next_video/modeling_llava_next_video.py.
No differences found for src/transformers/models/ijepa/modeling_ijepa.py.
No differences found for src/transformers/models/instructblipvideo/configuration_instructblipvideo.py.
No differences found for src/transformers/models/instructblipvideo/modeling_instructblipvideo.py.
No differences found for src/transformers/models/starcoder2/modeling_starcoder2.py.
No differences found for src/transformers/models/olmo2/configuration_olmo2.py.
No differences found for src/transformers/models/olmo2/modeling_olmo2.py.
No differences found for src/transformers/models/gemma2/configuration_gemma2.py.
No differences found for src/transformers/models/gemma2/modeling_gemma2.py.
No differences found for src/transformers/models/aria/configuration_aria.py.
No differences found for src/transformers/models/aria/modeling_aria.py.
No differences found for src/transformers/models/aria/image_processing_aria.py.
No differences found for src/transformers/models/aria/processing_aria.py.
No differences found for src/transformers/models/cohere2/configuration_cohere2.py.
No differences found for src/transformers/models/cohere2/modeling_cohere2.py.
No differences found for src/transformers/models/glm/modeling_glm.py.
```
PS: removed the `examples` as it slows downs the make fix-copies for everyone
(meaning, gemma2 after gemma)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35297). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.