noamaanMulla-03/transformers-issues huggingface dataset:
Donut model only works on 4.36.2 for inference, not 4.37.2
Author: VikParuchuri
Pull request: no

### System Info

I have a donut model with some slight customizations to the mbart decoder (added gqa and moe).  It works fine on 4.36.2 and 4.37.2 for training.

But inference only works on 4.36.2.  When I run inference on 4.37.2, then the output degenerates into repetition. @amyeroberts 

Here is an example (the text has been ocred with donut, then rendered back onto a page image):

This is with 4.36.2:

![image](https://github.com/huggingface/transformers/assets/913340/0382c56c-66eb-4989-a35d-92c1c6dbaba9)

And this is with 4.37.2:

![image](https://github.com/huggingface/transformers/assets/913340/01d710c3-d19e-4f00-a6bf-1cb1e530bd15)

Everything else is identical (same system, same packages).  I don't see anything obvious in the release notes that would cause this.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

- Clone https://github.com/VikParuchuri/surya/tree/rec (has to be the rec branch)
- Install (see README)
- Run `python benchmark/recognition.py --max 1 --debug`

You should see different output with different transformers versions.

### Expected behavior

I expect the output to be the same with both versions, and to not degenerate into repetition.

Comments:
fyi @NielsRogge 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi sorry for the late reply, I've checked all slow integration tests of Donut, and they all pass:
```
RUN_SLOW=yes pytest tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::DonutModelIntegrationTest
```
Which would mean that Donut works as expected on v4.38.2 dev.

This might be explained because of the customizations? Could you clarify which things are changed compared to Donut?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Documentation Inconsistency for Trainer.place_model_on_device Parameter
Author: Thytu
Pull request: no
Labels: name: Good First Issue

### System Info

Not relevant, this is only about documentation.

### Who can help?

CC @muellerzr and @pacman100

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In the documentation for the Trainer class, there is a mention of the place_model_on_device argument ([source](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L333-L335)).\
However, it appears that the Trainer class does not recognize or accept this parameter.

I'd like to confirm if this is an oversight or if there are any considerations regarding its usage that I might have missed.

### Expected behavior

I would expect to either be able to provide the `Trainer.place_model_on_device` parameter as described in the documentation or for the documentation to be updated to reflect the current implementation.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This is probably a matter of updating the documentation, feel free to open a PR

Hi @ArthurZucker, I'd be happy to look into this. 

@Thytu -- after looking into the code, it seems that `place_model_on_device` is not a parameter for the `Trainer` constructor but an attribute (as correctly stated in the docstring). Its definition [can be found here](https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/src/transformers/trainer.py#L481-L489), which highlights on which conditions it will be `True` or `False`. Does this clarify your issue?

> @Thytu -- after looking into the code, it seems that `place_model_on_device` is not a parameter for the `Trainer` constructor but an attribute (as correctly stated in the docstring). Its definition [can be found here](https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/src/transformers/trainer.py#L481-L489), which highlights on which conditions it will be `True` or `False`. Does this clarify your issue?

True, thanks for taking a look.

Closing the issue
Fix and simplify semantic-segmentation example
Author: qubvel
Pull request: yes

# What does this PR do?

It seems like the `semantic-segmentation` example is a bit outdated and doesn't run without fixing some issues. This PR does the following to improve the current state of the example:

1. Added instructions for accessing the sidewalk dataset to the semantic-segmentation example's README.md
2. Fixed missing dependencies required to run the semantic-segmentation example (accelerate, Pillow)
3. Fixed failing image transforms; replaced implemented transforms with those from `albumentations`
4. It was also observed that the "mean_iou" metric computation takes a lot of time and can be optimized. To improve this, I created a PR in the Evaluate project (huggingface/evaluate#569). As soon as the fix is merged, the time to perform training will be reduced by half.

Updated scripts have been evaluated and the fine-tuned model achieved similar metrics to the previous one:
 - previous model: https://huggingface.co/nielsr/segformer-finetuned-sidewalk-10k-steps
 - new model: https://huggingface.co/qubvel-hf/segformer-finetuned-sidewalk-10k-steps

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @NielsRogge 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30145). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[CI] Quantization workflow fix
Author: SunMarc
Pull request: yes

# What does this PR do ? 

This PR make sure that the setup happens before the quantization tests.  cc @younesbelkada 

The tests are triggered this time: 
<img width="339" alt="Screenshot 2024-04-10 at 11 22 25 AM" src="https://github.com/huggingface/transformers/assets/57196510/4fd0b1dd-6d2c-4e57-a32a-9eb397e5ab8d">


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30158). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

merging this since this is a tiny change
[UDOP] Fix tests
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes the failing slow tests for UDOP.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29573). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix probability computation in `WhisperNoSpeechDetection` when recomputing scores
Author: cifkao
Pull request: yes

# What does this PR do?

Fix #29313.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@patrickvonplaten @sanchit-gandhi @ylacombe

Comments:
@ArthurZucker I added a slow test where I set the `logprob_threshold` to 0 to make sure the no-speech detection is triggered. Without the fix, all the outputs are empty because all the no-speech probabilities are >1.

Hey @cifkao, thanks for the PR!
Could you make sure to rebase the branch to fix the failing test ?

> This scenario arises when the model is called with language set and with num_beams > 1.

Also, could you point out why this scenario applies when language is set ? I understand the case for num_beams>1 but don't see the point for the other case!
Many thanks!




The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29248). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Also, could you point out why this scenario applies when language is set ? I understand the case for num_beams>1 but don't see the point for the other case!

@ylacombe The bug manifests only when _both_ conditions (`num_beams > 1` and `language` set) are true. The latter condition causes decoding to start after the language token, which is why we need to call the model again to get the logits at that position (which are then incorrectly treated as log-probabilities). With `language` unset, the decoding starts from the beginning, so we never enter that branch.

Ready for final review from @ArthurZucker 

Thank you for taking the time to add a test 🔥 

Thanks for the contribution @cifkao!
[UDOP] Improve docs, add resources
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR improves the documentation of UDOP by explaining usage of `UdopProcessor`, adding a link to demo notebooks. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29571). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix typing annotation in hf_argparser
Author: xu-song
Pull request: yes

# What does this PR do?
Fix typing annotation in hf_argparser.



The following code raise warning in IDE: `Expected type 'str', got 'Path' instead `
https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/tests/utils/test_hf_argparser.py#L382



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?


Library:
- pipelines: @Narsil
- trainer: @muellerzr and @pacman100

Documentation: @stevhliu and @MKhalusova



Comments:
Fixing a bug when MlFlow try to log a torch.tensor
Author: etiennebonnafoux
Pull request: yes

# What does this PR do?

Add the case where a torch.tensor with one element is log on Mlflow. For now it only accept float and int.


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
Here https://github.com/huggingface/transformers/issues/29784
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
No need
- [ ] Did you write any new necessary tests?
No need (two lines of code)


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
Sorry but what should I do to pass  check_code_quality  and setup_and_quality  test ? this is my first PR

@etiennebonnafoux Running `make fixup` and pushing the changes should resolve this. There's more info in the [contributing guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)
Fix dinat in docker
Author: ydshieh
Pull request: yes

# What does this PR do?

Update `natten` installation command in docker file. Currently, all relevant tests are not run due to 
```
E   ImportError: Failed to import NATTEN's CPP backend. This could be due to an invalid/incomplete install. Please uninstall NATTEN (pip uninstall natten) and re-install with the correct torch build: shi-labs.com/natten
```
(during pytest collection)

In particular, doctest is not run at all.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30161). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX / bnb: fix torch compatiblity issue with `itemize`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/29825

Similar fix as : https://github.com/huggingface/peft/pull/1635 from @BenjaminBossan - `itemize` is only available on the latest releases of PT

cc @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30162). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update config class check in auto factory
Author: Rocketknight1
Pull request: yes

Our code for custom classes checks that `model_class.config_class` matches `config_class`, but this test is behaving oddly - it's returning a failure even though the two classes look identical to me. I changed the test to a string comparison test instead - it's possible that we're just initializing the two classes in slightly different ways, and that's creating two different objects that still represent the same class.

Update: The underlying cause is not my original PR, but PR #29262. The check is failing because we now use `importlib.machinery.SourceFileLoader()` rather than `importlib.import_module`, and this causes the two classes to not be equal (even though all their attributes are the same). I think the string comparison test should work fine as a solution for now!

Fixes #29828

(Probably) fixes #30149

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29854). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pinging @amyeroberts for a quick review here, since we now have two issues complaining about it!

Only concern is that we won't catch things if the attributes aren't the same 

I'm tentatively hopeful based on my testing, but I admit there are some mysteries in the import machinery that I don't fully understand.
Trainer is attempting to log a torch.Tensor but MLflow's only accepts float.
Author: etiennebonnafoux
Pull request: no
Labels: name: trainer

name: Integrations

### System Info

When I finetune a LLM model (Mistral-7B) I got a very explicit error when the the trainer log into MlFlow

`Trainer is attempting to log a value of "0.528405487537384" of type <class 'torch.Tensor'> for key "grad_norm" as a metric. MLflow's log_metric() only accepts float and int types so we dropped this attribute.`

I do not know how to say to the MlFlow callback to convert torch.Tensor to float before logging.

### Who can help?

@sanchit-gandhi @muellerz @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Unfortunately, I cannot share neither the model or the dataset since they do not belong to me.

### Expected behavior

Convert torch.Tensor to float before logging.

Comments:
Adding
 ```               
            elif isinstance(v, torch.Tensor) and v.numel() == 1:
                    metrics[k] = float(v)
```
to the on_log fonction of the MlFlowCallback code solve it.

Hi @etiennebonnafoux, thanks for reporting! 

Integrations, like MLFlow aren't actively maintained by us - rather the contributors who added them. We do want them to work, however! Would you like to open a PR with this fix? This way you get the github contribution for your suggestion 

This issue is closed with https://github.com/huggingface/transformers/pull/29932
Fixed typo in comments/documentation for Pipelines documentation
Author: DamonGuzman
Pull request: yes

# What does this PR do?

<!--Changed `lenth` the `length`-->



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


@stevhliu @MKhalusova

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30170). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Grounding DINO 
Author: EduardoPach
Pull request: no
Labels: name: New model

### Model description

Grounding DINO is a zero-shot object detection developed by IDEA-Research presented in the paper

[Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://arxiv.org/abs/2303.05499)

The model uses a Swin transformer and BERT as image and text backbones, respectively,  a feature enhancer and a cross-modality decoder.




### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Model implementation and weights can be found in their GitHub repo: https://github.com/IDEA-Research/GroundingDINO/tree/main#luggage-checkpoints

Comments:
Hi @EduardoPach! will you be working on this or is this an issue I can claim and work on?

@chiral-carbon Hello, I'm already working on this issue. I'm going to create a pull request and attach here as a WIP

Add Grounding DINO #25424 

Closed, PR due to an issue and I'm going to open another one.

> Closed, PR due to an issue and I'm going to open another one.

WIP in #25451
Fixed error in GPT-SW3 tokenizer
Author: haukurb
Pull request: yes

This pull request fixes an error in the  GPT-SW3 tokenizer in `transformers/models/gpt_sw3/tokenization_gpt_sw3.py`.
There is a string comparison which checks for a specific model size. The string is incorrect so for some model sizes this improperly results in additional tokens being added to the vocabulary which did not exist at training time and don't match the size of the embedding matrix of that particular model (6.7b).

It can be reproduced with:
```
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "/data/models/gpt-sw3/AI-Sweden-Models/gpt-sw3-6.7b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# this raises an IndexError
model(torch.tensor([[tokenizer.bos_token_id]]))
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.