noamaanMulla-03/transformers-issues huggingface dataset:
Use inherit tempdir makers for tests + fix failing DS tests
Author: muellerzr
Pull request: yes

# What does this PR do?

Following https://github.com/huggingface/transformers/pull/35266, we have this capability inheritently via `self.get_auto_remove_tmp_dir()`. As a result, updates all of the tempdir makers for this to be consistent, and fixes the deepspeed tests that weren't already using this to make sure we create tempdirs properly (currently failing tests on accelerate main)


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ydshieh @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35600). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added error when sequence length is bigger than max_position_embeddings
Author: Taha1506
Pull request: yes

Fixes # 32154




Comments:
Hi, It would be nice If you give some time to reviewing this pull request. @ArthurZucker . It relates to [this](https://github.com/huggingface/transformers/issues/32154) issue.

Hi! Can you take a look at this pull request again please? Thank you. @amyeroberts 

This is gonna break remote code, as they won'strore the self.max_position embedding

Fine for me @amyeroberts feel free to merge 

@amyeroberts 
Sorry for the delay, I've made the requested changes.

LGTM and we have reviews already. Thanks for the helpful error message!
Let `EarlyStoppingCallback` not require `load_best_model_at_end`
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR tweaks `EarlyStoppingCallback` to still require a metric, but doesn't need `load_best_model_at_end`. Doing so lets users still use it, but optionally doesn't require them to load it back in at the end of training if they don't want to. 

Fixes # (issue)

Request of @BenjaminBossan 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Rocketknight1 @SunMarc 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35101). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hmm, I'm negative on warnings in general, but maybe a single warning at the end of training might be understandable here, given the potential performance damage if users want the best checkpoint but get the one after n epochs of metrics getting worse.

@muellerzr What is your opinion on giving a warning?

Let's not forget about this PR, not a lot missing for it to be merged.

I'll add in the warning now :)
Fix flaky `test_beam_search_low_memory`
Author: ydshieh
Pull request: yes

# What does this PR do?

`test_beam_search_low_memory` is very flaky for `Emu3Vision2TextModelTest`.

The 3 fixes in this PR are all necessary to have 0 failure in a suite of 100 runs.

- call `set_xxx_less_flaky` methods
- avoid generate image tokens
- compare with `_check_similar_generate_outputs`

The final version run 500 times without failure.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35611). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> tests/models/mobilenet_v1/test_modeling_mobilenet_v1.py::MobileNetV1ModelTest::test_batching_equivalence - AssertionError: tensor(False) is not true : Batched and Single row outputs are not equal in MobileNetV1ForImageClassification for key=logits. Difference=0.48479270935058594.

This is a bit nasty but irrelevant to this PR. Will merge and try to fix the above one next week.

@ydshieh Cool, feel free to close https://github.com/huggingface/transformers/issues/29516 after fixing that. We had a small tracker for flaky vision model with that test

Thanks. The tracker is more for batching tests, but I will check with #35564 next week
Skip `MobileNetV1ModelTest::test_batching_equivalence` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

skip for now as #35564 fails this test more frequently for this model

will check it later

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35614). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update codeowners with individual model owners
Author: Rocketknight1
Pull request: yes

Right now, the `CODEOWNERS` file tags me to review everything, and this will cause me to die :headstone: 

This PR adds an autogenerated list of model owners. Right now it gets categories from `_toctree.yml` and default owners from `ISSUE_TEMPLATE/bug_report.yml`, but I can probably improve on this!

TODO:
- [x] Split up text models, they shouldn't all go to @arthurzucker
- [x] Split up multimodal models, they got assigned to @zucchini-nlp but not all of them are VLMs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Changed the globs to match `/[model_name]/mod*_[model_name]` so we capture `modular` and `modeling` changes
Add Moonshine 
Author: eustlb
Pull request: yes
Labels: name: New model

name: Audio

# What does this PR do?

This PR adds support for [Moonshine](https://github.com/usefulsensors/moonshine) to the Transformers library.

Moonshine builds on top of Whisper’s architecture to overcome some of its limitations, primarily the restriction to a fixed 30-second audio window.

Key improvements in Moonshine’s architecture:
	1.	It uses SwiGLU activation instead of GELU in the decoder layers.
	2.	Most importantly, it replaces absolute position embeddings with Rotary Position Embeddings (RoPE), enabling Moonshine to process audio inputs of any length—unlike Whisper, which is limited to fixed 30-second windows.

# Who can review?
@ArthurZucker

# TODO 

- [x] update UsefulSensors model repos
- [ ] run benchmarks

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34784). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @eustlb, awesome implementation! 🔥
Although, I think you meant to link to [Moonshine](https://github.com/usefulsensors/moonshine/) instead of the current link pointing to [Moonshine](https://github.com/moonshine-software/moonshine). 

Indeed 😅 corrected, thanks !!

Amazing work @eustlb and team! 🤗 

please check https://github.com/usefulsensors/moonshine/issues/81. there is an error when using huggingface to load moonshine.
OverflowError: out of range integral type conversion attempted
Author: test3211234
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.1
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.8
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: AMD Radeon RX 6600 [ZLUDA]

### Who can help?

@amyeroberts @qubvel 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

![image](https://github.com/user-attachments/assets/9a8e53a2-bc7c-412b-80a4-9b716f3764ec)


When trying to run this official example code from the GOT-OCR2_0 HuggingFace page:

```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('ucaslcl/GOT-OCR2_0', trust_remote_code=True)
model = AutoModel.from_pretrained('ucaslcl/GOT-OCR2_0', trust_remote_code=True, low_cpu_mem_usage=True, device_map='cuda', use_safetensors=True, pad_token_id=tokenizer.eos_token_id)
model = model.eval().cuda()


# input your test image
image_file = 'xxx.jpg'

# plain texts OCR
res = model.chat(tokenizer, image_file, ocr_type='ocr')

# format texts OCR:
# res = model.chat(tokenizer, image_file, ocr_type='format')

# fine-grained OCR:
# res = model.chat(tokenizer, image_file, ocr_type='ocr', ocr_box='')
# res = model.chat(tokenizer, image_file, ocr_type='format', ocr_box='')
# res = model.chat(tokenizer, image_file, ocr_type='ocr', ocr_color='')
# res = model.chat(tokenizer, image_file, ocr_type='format', ocr_color='')

# multi-crop OCR:
# res = model.chat_crop(tokenizer, image_file, ocr_type='ocr')
# res = model.chat_crop(tokenizer, image_file, ocr_type='format')

# render the formatted OCR results:
# res = model.chat(tokenizer, image_file, ocr_type='format', render=True, save_render_file = './demo.html')

print(res)
```

### Expected behavior

Should print result.

Comments:
Hi @test3211234! You are using 3rd party implementation, it's better to open discussion directly on the hub
https://huggingface.co/stepfun-ai/GOT-OCR2_0/discussions

We also have `transformers` native implementation ongoing, you can try this out by installing it from the PR
https://github.com/huggingface/transformers/pull/34721
ImportError: cannot import name 'HfApiEngine' from 'transformers'
Author: cluiverto
Pull request: no
Labels: name: bug

### System Info

Please update tutorials cause you confusing people: https://huggingface.co/docs/transformers/agents

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Make update for documentation

### Expected behavior

Make update for documentation

Comments:
Works for me - have you tried `pip install --upgrade transformers`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
VLlama3ForCausalLM in SmolVLM
Author: BakingBrains
Pull request: no

In the SmolVLM [config](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct/blob/77f80ed86046795eab54cfb60d9442c2ad813f57/config.json)  there is **VLlama3ForCausalLM**. Any plans on adding this to the library?

Regards,


Comments:
cc @zucchini-nlp

I believe it was meant to be just LlamaForCausalLM, cc @andimarafioti who added the model 

Thank you @zucchini-nlp and @Rocketknight1 . I'm just curious if any changes have been made to **text_model** so that it fits well with the encoder and connector.

Anything on this @andimarafioti ?

Thank you

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Documentation for SWAG contradicts itself when constructing the first sentence.
Author: bauwenst
Pull request: no
Labels: name: bug

### System Info

Not relevant.

### Who can help?

@stevhliu @ArthurZucker 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The [docs for multiple choice](https://huggingface.co/docs/transformers/tasks/multiple_choice) use SWAG as an example, which is the task of selecting the next sentence given a context. Somewhat strangely, rather than being given in the format `(sentence1, [sentence2a, sentence2b, sentence2c, sentence2d])`, the dataset is given in the format `(sentence1, sentence2_start, [sentence2_endA, sentence2_endB, sentence2_endC, sentence2_endD])`.

The code given in the docs basically turns the dataset into the first format, where **sentence 1 is kept intact** and the start of sentence 2 is concatenated to each ending:
https://github.com/huggingface/transformers/blob/a06a0d12636756352494b99b5b264ac9955bc735/docs/source/en/tasks/multiple_choice.md?plain=1#L96-L100

Yet, the docs say:
https://github.com/huggingface/transformers/blob/a06a0d12636756352494b99b5b264ac9955bc735/docs/source/en/tasks/multiple_choice.md?plain=1#L85-L88

What is being described is formatting the dataset as `(sentence1 + sentence2_start, [sentence2_start + sentence2_endA, sentence2_start + sentence2_endB, sentence2_start + sentence2_endC, sentence2_start + sentence2_endD])`, where **there is overlap between the first and the second sentence** (namely `sentence2_start`).

### Expected behavior

Either the code is wrong or the description is wrong. 

If the description is wrong, it should be:

> The preprocessing function you want to create needs to:
> 1. Make four copies of the sent1 field.
> 2. Combine sent2 with each of the four possible sentence endings.

If the code is wrong, it should be:
```python
    first_sentences = [[f"{s1} {s2_start}"] * 4 for s1,s2_start in zip(examples["sent1"], examples["sent2"])]
    second_sentences = [
        [f"{s2_start} {examples[end][i]}" for end in ending_names] for i, s2_start in enumerate(examples["sent2"])
    ]
```

Comments:
cc @stevhliu 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Bug of eval loss when enabling average_tokens_across_devices
Author: Snowdar
Pull request: no

https://github.com/huggingface/transformers/blob/329f5dbf97a5cb2473914c88c05aa3dcb242e19a/src/transformers/trainer.py#L3737

For the eval set, there is no need to reduce_mean the gradient, so does the loss not need to multiple the `num_processes`?
```
if self.args.average_tokens_across_devices and self.model_accepts_loss_kwargs:
            loss *= self.accelerator.num_processes
```

In my experiments, I found the loss value of eval will be always greater `num_processes` times than the running of `average_tokens_across_devices=false`.

Comments:
cc @muellerzr @SunMarc 

I think the bug is caused by the code at https://github.com/huggingface/transformers/blob/v4.46.3/src/transformers/trainer.py#L3661.

Specifically, `num_items_in_batch` is `None` at the eval stage (https://github.com/huggingface/transformers/blob/v4.46.3/src/transformers/trainer.py#L4385) and may be `None` at the training stage (https://github.com/huggingface/transformers/blob/v4.46.3/src/transformers/trainer.py#L5054). `model_accepts_loss_kwargs` is `True` when the model has a `loss_kwargs` argument. `compute_loss_func` is not `None` when users define their own loss function. `num_items_in_batch = self.accelerator.gather(num_items_in_batch).sum().item()` is always triggered at the training stage when `average_tokens_across_devices` is set to `True`.

Therefore, we need to trigger `loss *= self.accelerator.num_processes` only when `num_items_in_batch` is actually used, which is when either `compute_loss_func` is defined or (`model_accepts_loss_kwargs` is `True` and `num_items_in_batch` is not `None`). Changing https://github.com/huggingface/transformers/blob/v4.46.3/src/transformers/trainer.py#L3648-L3651 to something like:
```
if self.args.average_tokens_across_devices and (self.compute_loss_func is not None or (self.model_accepts_loss_kwargs and num_items_in_batch is not None))
```
may temporarily fix this problem. I say "temporarily" because the eval loss will still be different between multi-GPUs and single GPU setups.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Docs: Fixed `whisper-large-v2` model link in docs
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?


Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stevhliu 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32871). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix: Fixed directory path for utils folder in `test_tokenization_utils.py`
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Fixed directory path for utils folder in `test_tokenization_utils.py` and removed some unused expressions.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker 

Comments:
Thanks for the review @amyeroberts. I think I found the issue with failing tests, the reason is that, in this line:
https://github.com/huggingface/transformers/blob/bd251e4955fee920dc770130da91b2cc802475ba/tests/utils/test_tokenization_utils.py#L39
to get the `utils/test_module` we should go one more level up in the directories. It is currently adding 
`transformers/tests/utils` to the path instead of `transformers/utils`.

Even in the main branch, if we run `pytest -v -s tests/utils/test_tokenization_utils.py` we are getting:
``` Python
ImportError while importing test module '/home/suraj/repos/transformers/tests/utils/test_tokenization_utils.py'.
```
Add keypoint-detection task
Author: vincentmin
Pull request: no
Labels: name: New model

### Feature request

Add support for keypoint detection. This includes a task, pipeline, dataset label and training pipeline.
The task is to take an image and predict the x and y locations of a set of keypoints. Which keypoints are predicted should depend on the model trained for this task. The training pipeline for keypoint detection should allow to swap components. For example, one should be able to choose the backbone to be any suitable vision transformer model that is available on the huggingface hub.

### Motivation

Keypoint detection is a use case that is prevalent in computer vision. The computer vision subset of the huggingface ecosystem would benefit from adding the popular keypoint detection task to the existing set of tasks.
At the time of writing, existing repositories for keypoint detection often focus on a single particular model, e.g.:

- yolov7: https://github.com/RizwanMunawar/yolov7-pose-estimation
- yolov8: https://docs.ultralytics.com/tasks/pose/
- vitpose: https://github.com/ViTAE-Transformer/ViTPose

The computer vision community could benefit greatly from a high quality community oriented open source hub for keypoint detection.

### Your contribution

I am happy to be part of the discussion, but probably can do little in terms of PR's at this point in time.

Comments:
ViTPose has been added in #30530.

Task: https://huggingface.co/tasks/keypoint-detection