noamaanMulla-03/transformers-issues huggingface dataset:
Add torch_empty_cache_steps to TrainingArguments
Author: aliencaocao
Pull request: yes

# What does this PR do?
Following up on https://github.com/huggingface/transformers/pull/31530, add `torch.cuda.empty_cache()` as an optional `TrainingArgument` to the training loop every N step, such that those who need it to avoid OOMs can use it.

Also added to the reduce vram usage docs and changed the table for mixed precision training a bit - let me know if you prefer this change to be isolated in another PR.

Not sure if a test is needed - we can't really 'verify' it being called besides monitoring vram usage?

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerz @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31546). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Test failures look unrelated, so we're okay to merge here I think.
Add model RTDetr
Author: rafaelpadilla
Pull request: yes
Labels: name: New model

# What does this PR do?

Adds new model RTDetr. 

Image processing:
- [x] preprocess 
- [x] post_process
- [x] post_process_object_detection

Tests:
- [x] image processing: test_image_processor_outputs
- [x] image processing: test_multiple_images_processor_outputs 
- [x] model: logits and boxes match the original model
- [ ] model: unit tests for `modeling_rt_detr.py` are passing

Backbone:
- [x] adjust backbone to be compatible with Timm
- [x] convert backbone weights to be compatible with Timm 

General:
- [x] review docstrings
- [x] check variable names
- [x] check order of classes

Fixes #26742 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@amyeroberts 


Comments:
@rafaelpadilla Nice work! Excited to get this model added to the library 💪 

I'm removing the review request for Arthur, as you only need one core maintainer's approval. @NielsRogge if you have time, could you give this a quick review once tests are all passing? It would be good to get your thoughts especially as you've handled the other DETR ports. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27247). All of your documentation changes will be reflected on that endpoint.

Doesn't more contribute?

Will this be reviewed?

@roboserg There's an active PR adding this model here: #29077 

@amyeroberts I think we can close this :)
Fix serialization for offloaded model
Author: SunMarc
Pull request: yes

# What does this PR do ? 

This PR fixes the serialization of offloaded model. With this PR, the logic behind saving offloaded model is triggered only when we use more than one device that contain either "cpu" or "disk" device. Moreover, we fix a small bug that was introduced with this [PR](https://github.com/huggingface/transformers/pull/31091). We rename the `state_dict` variable to `shard_state_dict`. I also fixed the test `test_save_offloaded_model`.

### Example: 

```python 
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", device_map={"":"cpu"})
model.save_pretrained("llama")
```

Fixes https://github.com/huggingface/transformers/issues/31685

Comments:
Could you implement a test that ensures that the previous faulty behavior has been fixed? :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31727). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Could you implement a test that ensures that the previous faulty behavior has been fixed? :)

Done ! 

Do you know where I can see the results of the test for test_modeling_utils.py file @ydshieh ? I didn't see the following test `test_save_offloaded_model` in the daily report. On my local machine, it was failing so I was expecting to see it in the report. 

Hi @SunMarc Let me check, but we might have a problem of some test files not included 😭 

> Hi @SunMarc Let me check, but we might have a problem of some test files not included 😭

Sounds good ! Maybe we can put all the utils python file into a separate folder to not have them mixed with the common test files. 

I confirmed that we have some tests not running on daily CI.
(We also have push-ci where we try to collect the impacted files, but for daily CI, tests are collected from the subfolders of `tests`)

> Maybe we can put all the utils python file into a separate folder to not have them mixed with the common test files.

Sound a quick solution. It's just we have to make sure there is not subclass of `TestCase` in other files under `tests`. But for now let's do what you suggested above.




> Sound a quick solution. It's just we have to make sure there is not subclass of TestCase in other files under tests. But for now let's do what you suggested above.

Great ! 

Here it is #31730  but I have to see how CI is going.
Tensors' device passed to a model is not correct when ACCELERATE_TORCH_DEVICE is privateuseone
Author: kiszk
Pull request: no

### System Info

- `transformers` version: 4.43.0
- Platform: Linux-5.4.0-187-generic-x86_64-with-glibc2.31
- Python version: 3.10.14
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): 2.15.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: Tesla V100-PCIE-32GB


### Who can help?

@muellerzr @SunMarc

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Execute the following script `test.py` with accelerators(`TPU`, `CUDA`, `CPU`, `privateuseone:0`) with `accelerate`.

```python
import sys
import accelerate
import torch
import transformers
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
)

model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

def forward(input_ids=None, attention_mask=None, **kwargs):
    # print devices for input tensors instead of doing inference
    print(input_ids.device, attention_mask.device)
    sys.exit(1)
model.forward = forward

trainer = Trainer(
    model = model,
    tokenizer = tokenizer,
)

class MyDataset(torch.utils.data.Dataset):
    def __init__(self):
        self.data_list = [{"input_ids":[1], "label":[1]}]

    def __len__(self):
        return len(self.data_list)

    def __getitem__(self, idx):
        return self.data_list[idx]

predict_dataset = MyDataset()
trainer.predict(predict_dataset)
```

### Expected behavior

When `ACCELERATE_TORCH_DEVICE=privateuseone:0 python test.py` is executed, I expect `privateuseone:0 privateuseone:0` will be displayed. However, I saw `cpu cpu`.

With `TPU`, I got `xla:0 xla:0`. With NVIDIA GPU, I got `cuda:0 cuda:0`. With only CPU, I got `cpu cpu`.

Comments:
Exclude torch.compile time from metrics computation
Author: zxd1997066
Pull request: yes

When using torch.compile in accelerator.prepare_model, compilation time is being included while computing statistics. And this  may result in incorrect performance metrics calculations. This PR measures time after the torch.compile called in accelerate to exclude the compilation time.

Comments:
cc @muellerzr @SunMarc 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31443). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks for the PR @zxd1997066 ! I think it would be better if we can do similar to what we did with f"{metric_key_prefix}_jit_compilation_time".
> 
> ```
>         if f"{metric_key_prefix}_jit_compilation_time" in output.metrics:
>             start_time += output.metrics[f"{metric_key_prefix}_jit_compilation_time"]
> ```

Hi @SunMarc , thanks for the review, I will take a look at it.

> This is a lot better @zxd1997066 , thanks for iterating ! Could you fix the quality issue by typing `make style` ?

Fixed the quality issue, thanks!

Hi @SunMarc , could you please help merge this PR if all good? I do not have access. Thanks!

Hey @zxd1997066, I've pinged @amyeroberts for a final review !
from transformers import Phi3VModel, Phi3VConfig. Phi3 Vision config not added to transformers.
Author: sankydesai
Pull request: no

### System Info

from transformers import Phi3VModel, Phi3VConfig.
Fails with error: 
ImportError: cannot import name 'Phi3VModel' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)

Note: Installed Transformers from source. 

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

!pip install -U git+https://github.com/huggingface/transformers
from transformers import Phi3VModel, Phi3VConfig

### Expected behavior

No Error.

Comments:
Hey @sankydesai, I don't think you can really import `Phi3VModel` or `Phi3VForCausalLM` directly from transformers 

The configuration for the Phi3 Vision model includes an auto_map field, which indicates the appropriate classes to use for loading the model.
```json
"auto_map": {
    "AutoConfig": "configuration_phi3_v.Phi3VConfig",
    "AutoModelForCausalLM": "modeling_phi3_v.Phi3VForCausalLM"
}
```
Which is uploaded on the hub itself

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Token Classification Pipeline support for Layout models 
Author: yaswanth19
Pull request: no

### Feature request

Currently, It seems the pipeline feature doesn't support layout models like LayoutLM and LILT for Token Classification task. The pipeline as of now only takes the input string which alone is not sufficient in this case and additional code modification should be made to incorporate the bounding box info.

### Motivation

I was trying out pipeline feature for LayoutLMv3 model and found out that there is no support for bbox as inputs.So, currently I have different end-to-end functions which does the job but the pipeline feature can be used which replaces all those functions.  

### Your contribution

I could contribute and raise a PR to incorporate this change in the Token Classification Pipeline. 

Comments:
WDYT @amyeroberts ?

@yaswanth19 As the input / output for the layout models is highly specific, I'm not sure a pipeline makes much sense here. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix gemma tests
Author: ydshieh
Pull request: yes
Labels: name: run-slow

# What does this PR do?

- skip 3 `gemma-7b` tests: it's just OOM (even run each single one test)
- fix wrong checkpoint `google/gemma-2-9b` used in `test_model_2b_fp16` (it also gives OOM)
- update expected output in `test_model_2b_eager` and `test_model_7b_4bit`. See comment in the change

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31794). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Code agent: allow function persistence between steps
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Allow Code agent to create functions and re-use them later on.

Comments:
There are a few failing tests that seem linked to this PR!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31769). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[CI] 2 Cohere tests are failing and skipped for now
Author: ydshieh
Pull request: no
Labels: name: Tests

name: Good First Issue

The following 2 tests are currently failing and being skipped for now.

- test_initialization
- test_fast_init_context_manager

See  #31263

Comments:
A bit more info:

tests/models/cohere/test_modeling_cohere.py::CohereModelTest::test_fast_init_context_manager
(line 551)  AssertionError: Tensor-likes are not close!

tests/models/cohere/test_modeling_cohere.py::CohereModelTest::test_initialization
(line 753)  AssertionError: inf not found in [0.0, 1.0] : Parameter layers.0.input_layernorm.weight of model <class 'transformers.models.cohere.modeling_cohere.CohereModel'> seems not properly initialized

Hello @ydshieh, I would love to try to help with this! However, I am struggling to recreate the issue as when I run these two tests in my local dev env they pass without any problem. Do I need to open a [WIP] PR to run the tests through GitHub actions?

Hi @jkmschoultz It's passing because I skipped those 2 tests in #31263

https://github.com/huggingface/transformers/pull/31263/files

So you will have to remove `@unittest.skip("Failing. Issue opened in #31351")` first :-)

Hi @ydshieh, thank you and yes I have already done so, see

https://github.com/jkmschoultz/transformers/commit/4a35d5fa95b7bedb282538dfcaab3278851edde8

This is where I cannot recreate the same test failure as you encountered, because the tests pass successfully when I run them locally.

This is likely a flaky tests. Would you like to try 

https://huggingface.co/docs/transformers/testing#repeat-tests

> pip install pytest-flakefinder
> pytest --flake-finder --flake-runs=100 tests/.....

If they are still passing on your side, I will have to check what to proceed


Not flaky on my end, 300 runs on each test and they both pass all of them :/

OK, let me check first. Thanks!

I can't reproduce it anymore 😅 . Maybe some commits in between fixed them. I will check further

Good thing it's working now at least! ... I guess? 😂 Let me know if I can help in any other way or if I should open a PR for this :)

Hi, thank you for the offering. These 2 tests are passing now but I can't figure out why. I opened #31517.

@ydshieh It sounds like this can be closed as you enabled the skipped tests again.

Yes, thanks for reminding me 🤗
Repeating an important warning in the chat template docs
Author: Rocketknight1
Pull request: yes

One important detail with chat templates is that they include all necessary special tokens, so if you apply them and then tokenize the text yourself, you should always use `add_special_tokens=False`. This is mentioned lower down in the template writing tips, but that's an advanced section, so I also added a tip about it further up to ensure people don't miss it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31796). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Reworded!
Add training support for SigLIP
Author: aliencaocao
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Add the sigmoid contrastive loss function of SigLIP from https://github.com/google-research/big_vision/blob/01edb81a4716f93a48be43b3a4af14e29cdb3a7f/big_vision/trainers/proj/image_text/siglip.py#L287

This will allow training/finetuning SigLIP models.

Already verified to work on my own dataset.

I saw the note on using `torch.distributed` for loss function and `open_clip`'s implementation, but I'm not sure why is it needed. I ran my training with both DDP and FDSP with full sharding and it seem to work just fine, also getting the expected speedup and ability to set larger BS. The only issue is https://github.com/huggingface/transformers/issues/31034 when using FDSP but I don't think its `SigLIP` specific.

Nonetheless, I updated the docs to mention the lack of usage of `torch.distributed` if that ended up important to some users.

Not sure if a training test is needed.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@amyeroberts 


Comments:
@aliencaocao Could you rebase to include the upstream changes on main? This should fix the failures on the CI runs

Added the training tests and also enabled gradient checkpointing tests. I note that CLIP had issues with GC but I have used it with siglip myself and did not find any issue on convergence/accuracy on a single RTX 3080Ti with fp16 training and grad accum=16.

Will let the tests run and see how it goes.

@amyeroberts seems to need you to enable slow tests?

@amyeroberts now that the GC tests are properly skipped, shall we move forward with this?
Fix galore lr display with schedulers
Author: vasqu
Pull request: yes

# What does this PR do?
See #31707 for a detailed rundown. Fixes #31707

Tl;dr: Galore still has issues displaying the correct lr due to the lr scheduler this time. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @SunMarc @amyeroberts @Minami-su 

Comments:
Failing tests seem unrelated to me: TF and hub issues.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31710). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@muellerzr Wouldn't it make more sense over here? https://github.com/huggingface/transformers/blob/3345ae733b6f4aeb7204a0f3e646a3cdbaad0023/tests/trainer/test_trainer.py#L1452

I would add two tests:
- A basic learning rate check without a scheduler.
- Cosine with warmup steps and check the lr logs to roughly follow the correct pattern.

Is that reasonable? I'm not sure when I'll have the time tho. 

@amyeroberts One timeout didn't make it through. Is it just my luck? :laughing: 

@vasqu Just bad luck - although we'll need to look into it on our side why these flaky failures are happening. Thankfully some re-runs worked. Thanks for your patience!
Fix Wav2Vec2 Fairseq conversion (weight norm state dict keys)
Author: gau-nernst
Pull request: yes
Labels: name: Audio

# What does this PR do?

Close #31633

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sanchit-gandhi @amyeroberts

## To test

```bash
wget https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt  # wav2vec2-base checkpoint from facebook
pip install git+https://github.com/facebookresearch/fairseq.git  # install fairseq

python src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py \
    --pytorch_dump_folder wav2vec2-base \
    --checkpoint_path wav2vec_small.pt \
    --config_path facebook/wav2vec2-base \
    --not_finetuned
```

I only checked for Wav2Vec2 base. Other checkpoints should work too. See other checkpoints from fairseq here: https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec

e.g. https://dl.fbaipublicfiles.com/fairseq/wav2vec/libri960_big.pt corresponds to https://huggingface.co/facebook/wav2vec2-large

Comments:
cc @kamilakesbi 

gentle ping @amyeroberts @kamilakesbi