noamaanMulla-03/transformers-issues huggingface dataset:
FSDP on XLA only saves the sharded checkpoint of rank0
Author: yitongh
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.44.0.dev0
- Platform: Linux-5.10.112-005.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.24.2
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: (True)
- Using GPU in script?: (True)
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

@muellerzr @SunMarc

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. `fsdp_config.json`
```
{
    "fsdp_transformer_layer_cls_to_wrap": [
        "Qwen2DecoderLayer"
    ],
    "xla": true,
    "xla_fsdp_settings": {
        "compute_dtype": "bfloat16",
        "buffer_dtype": "bfloat16",
        "pin_layout_in_collective_ops": false
    },
    "xla_fsdp_grad_ckpt": false
}
```
2. use FSDP to train Qwen2-7B and save checkpoints 
```
torchrun --nproc_per_node 8 \
    examples/pytorch/language-modeling/run_clm.py \
    --num_train_epochs 2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-103-raw-v1 \
    --use_fast_tokenizer false \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --do_train \
    --output_dir /tmp/test-clm \
    --model_name_or_path Qwen/Qwen2-7B-Instruct \
    --tokenizer_name Qwen/Qwen2-7B-Instruct \
    --trust_remote_code true \
    --block_size 4096 \
    --optim adamw_torch \
    --save_strategy steps \
    --max_steps 10 \
    --logging_strategy steps \
    --gradient_checkpointing no \
    --logging_steps 1 \
    --bf16 true \
    --fsdp "full_shard" \
    --fsdp_config fsdp_config.json
```

### Expected behavior

The saved checkpoint should contain the complete weights, but currently, FSDP on XLA only saves the sharded weights from rank0.

Comments:
Fix RT-DETR optional losses
Author: qubvel
Pull request: yes

# What does this PR do?

In the modeling file of RT-DETR there are the following losses defined 
 - labels_varifocal (used by default)
 - boxes (used by default)
 - labels_focal (not used, broken)
 - labels_cross_entorpy (not used, broken)
 - labels_binary_cross_entropy (not used, broken)
 
This PR fixes additional losses that might be defined for fine-tuning and enable their usage for fine-tuning with config option

```python
model = RTDetrForObjectDetection.from_pretrained(
    "PekingU/rtdetr_r18vd",
    losses=["labels_varifocal", "boxes"],  # default value
    
    # Changing RT-DETR labels loss:
    # losses=["labels_focal", "boxes"],
    # losses=["labels_binary_cross_entropy", "boxes"],
    # losses=["labels_cross_entropy", "boxes"],
    
    # Combining multiple losses
    # losses=["labels_focal", "labels_varifocal", "boxes"],
)
```

I made several finetuning runs to make sure the model converges fine for every loss function
<img width="1400" alt="Screenshot 2024-08-13 at 12 19 22" src="https://github.com/user-attachments/assets/fe0e31a6-0582-421a-950b-510f53f24cab">


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32658). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Mamba / FalconMamba: Fix mamba left padding
Author: younesbelkada
Pull request: yes

# What does this PR do?

As pointed out in https://github.com/huggingface/transformers/pull/32080#discussion_r1705555054 - it is important to zero-out hidden states that corresponds to the padd tokens before and after the causal convolution so that the padd token will not have an impact on the calculated hidden states. 

This can be empirically proven by generation quality before / after this fix (note by default FalconMamba uses left padding):

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tiiuae/falcon-mamba-7b"
tok = AutoTokenizer.from_pretrained(model_id)
tok.pad_token_id = tok.eos_token_id

texts = [
    "Hello today",
    "Hello my name is Younes and today"
]

inputs = tok(texts, return_tensors="pt", padding=True, return_token_type_ids=False).to(0)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map=0, torch_dtype=torch.bfloat16)

out = model.generate(**inputs, max_new_tokens=20)
print(tok.batch_decode(out, skip_special_tokens=True))
```

### Before the fix:

> Hello today I'm.\nI'm.\n Hello today.\n Hello today.\n Hello today

### After the fix:

> Hello today I'm going to show you how to make a 3D model of a house.\n

Propagated the changes in Mamba1 as well

cc @ArthurZucker @molbap 

Comments:
Can we propagate this to Jamba as well :D thx for this fix ❤️ 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32677). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
dbg
Author: sshleifer
Pull request: yes


Comments:
no sorry you can close

On Mon, Aug 19 2024 at 9:24 AM, Arthur < ***@***.*** > wrote:

> 
> 
> 
> 
> 
> 
> 
> ***@***.**** commented on this pull request.
> 
> 
> 
> Hey! Anything we can do on our side to help you ?
> 
> 
> 
> —
> Reply to this email directly, view it on GitHub (
> https://github.com/huggingface/transformers/pull/32414#pullrequestreview-2245542256
> ) , or unsubscribe (
> https://github.com/notifications/unsubscribe-auth/ABOD2YPXYUZDT54NFQGGVO3ZSHWZNAVCNFSM6AAAAABL7EDH4KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDENBVGU2DEMRVGY
> ).
> You are receiving this because you authored the thread. Message ID: <huggingface/transformers/pull/32414/review/2245542256
> @ github. com>
> 
> 
>
Add tip to clarify tool calling
Author: Rocketknight1
Pull request: yes

This small PR clarifies a potentially-confusing step during tool calling pointed out by [ztgeng](https://huggingface.co/ztgeng) - no matter what the model outputs, the message should be appended to the chat in the universal format.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32883). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: resolve bug with `use_mps_device` setting not taking effect
Author: cooljacket
Pull request: yes

# Fixes the bug with `use_mps_device` setting not taking effect.
In training_args.py,, even when `use_mps_device` is set, the current implementation overrides it with `device=cuda/cpu`, causing `use_mps_device` to not take effect.

It's a minor code typo, so I didn't open an issue for it. 

<img width="982" alt="image" src="https://github.com/user-attachments/assets/68b84763-7b67-4064-aa00-34f77bd8f453">


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.pytorch.org/)? Please add a link to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/pytorch/pytorch/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/pytorch/pytorch/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Hi @cooljacket, thanks for opening this PR! 

Swapping the `if` to `elif` means that you don't hit the `else` branch, but I don't think this is the correct fix. 

The first `if` statement isn't related to the later logic - it's there for triggering warnings if the user is using `use_mps_device` as this is due to be deprecated. I

f the `else` branch is being reached, it means either `self.use_cpu` is `True` or `is_torch_mps_available()` is evaluating to `False`. Could you check what each of these values are when you're experiencing this issue?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix attention mask creation for GPTNeo
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

It seems that #26486 broke the way the attention mask was created. It creates a causal attention mask by default, but there is already a causal attention mask in the `GPTNeoSelfAttention` modules, resulting in `NaN`s.

I am not sure the solution is perfect, so opened to suggestions.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I will work on that when I have the bandwidth!

Gentle ping @michaelbenayoun 

Any update @michaelbenayoun?

I can try to work on that next week.

While trying to write an example, I noticed that I think it works now...

OK - that's good news really! Shall we close the PR? 

Yes closing it.
🚨🚨🚨 Update min version of accelerate to 0.26.0
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR updates the minimum version required for accelerate to `0.26.0`. 

Fixes https://github.com/huggingface/transformers/pull/31449

Creating a dev docker, waiting for the tests to pass. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32627). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> we should also have a check inside the TrainingArguments for this min version and raise an error in the __post_init__

Oh that's true that we don't have any check for accelerate inside TrainingArguments. Is accelerate a required dep for using `Trainer` ? 

@SunMarc yes it is (the pytorch version). You can see some similar checks for `accelerate` in the training arguments. This should go in during `def _setup_devices`

> @SunMarc yes it is (the pytorch version). You can see some similar checks for accelerate in the training arguments. This should go in during def _setup_devices

Thanks ! This should be fixed in the latest commit. Also I removed a few checks that aren't needed anymore. 

Failing tests are not related to my PR. See the linked PR above.  Other than that, tests are passing ! I'll wait until @ydshieh approves to merge this PR ! 

Failures are indeed unrelated, you can rebase on https://github.com/huggingface/transformers/pull/32649 to skip tests that are currently failing and it should work (or wait till it's merged on main)

> Don't forget to check the images used for daily CI and the related jobs too.

The daily CI is using the most recent version of accelerate, so that should be fine !
Added docstring to the SchedulerType class
Author: Arunprakash-A
Pull request: yes

# What does this PR do?
It adds the docstring for the class `SchedulerType` . Currently, the AutoDoc only displays the `__call__ ` signature of `enum.EnumType` base class which is not helpful.  One needs to look for the `names` and the respective mapping in the source code of `trainer.py`. Therefore, added the docstring that lists out all the available names and the scheduler types they are mapped to. 

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case)

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32872). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks! Can you implement the fix suggested from the CI test [here](https://app.circleci.com/pipelines/github/huggingface/transformers/101290/workflows/6c09ee5a-751f-46ca-b50f-4451a46219c9/jobs/1349645) and then run `make style` after?

> Thanks! Can you implement the fix suggested from the CI test [here](https://app.circleci.com/pipelines/github/huggingface/transformers/101290/workflows/6c09ee5a-751f-46ca-b50f-4451a46219c9/jobs/1349645) and then run `make style` after?

I implemented the suggestions. Still, the test failed. Sorry, I have no clue about running `make style` ?  Could you help me with that?

Yeah, you can run `make style` from the command line

> Yeah, you can run `make style` from the command line

It passed all checks.

``` 
make style
ruff check examples tests src utils setup.py conftest.py --fix --exclude ""
All checks passed!
ruff format examples tests src utils setup.py conftest.py --exclude ""
3054 files left unchanged
make autogenerate_code
make[1]: Entering directory '/home/arun/transformers'
running deps_table_update
updating src/transformers/dependency_versions_table.py
make[1]: Leaving directory '/home/arun/transformers'
make extra_style_checks
make[1]: Entering directory '/home/arun/transformers'
python utils/custom_init_isort.py
python utils/sort_auto_mappings.py
python utils/check_doc_toc.py --fix_and_overwrite
make[1]: Leaving directory '/home/arun/transformers'
```
`truncate_dim` on `BertModel`
Author: kerem0comert
Pull request: no
Labels: name: Feature request

### Feature request

I have a pipeline to finetune an instance of `BertModel`, on a `text-classification` task. 
I would like to use [this new embedding model](https://huggingface.co/aari1995/German_Semantic_V3#usage) as my base embedding now.
As can be seen in the example they provide, they are able to pass different values for `matryoshka_dim` into the `SentenceTransformer` instance through the `truncate_dim` argument.
However, I was not able to do this on the `BertModel` in the following code snippet that I have in my code:

```python
 self.bert_backbone = BertModel.from_pretrained(
            pretrained_model_name_or_path=self.config.embedding_model_file.model_name,
            cache_dir=Path(self.config.embedding_model_file.cache_dir),
        ).to(self.device)
```

And I do not want to use a `SentenceTransformer` instance either as in my training loop I would like to be able to get:

```python
bert_outputs: BaseModelOutputWithPoolingAndCrossAttentions = (
                    self.bert_backbone(
                        input_ids=input_ids, attention_mask=attention_mask
                    )
                )
                bert_logits: Tensor = bert_outputs.last_hidden_state[
                    :, 0, :
                ]  # Take the [CLS] token output
```

and I am not sure if this code would work also with a simple swap to `SentenceTransformer`. In any case, I think that this is a potential parameter that `BertModel` should support, and maybe it does but I am just missing it.

Thanks in advance!

### Motivation

To be able to extend the `BertModel` further

### Your contribution

.

Comments:
Hi, thanks for raising an issue! 

As this involves custom code, this is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Alright, [did so here for reference.](https://discuss.huggingface.co/t/truncate-dim-on-bertmodel/103110) Although if this is indeed something that is not supported I would assume that it would be categorized under feature request.

Technically yes, although for it to be a feature request the exact behaviour and feature added should be clear and well specified. 

Half of the issue here is a question about how to do and approach a custom use case. If after discussion in the forum you realise there's a specific, singular feature that you think should be added (which improves the library as a whole, rather than enables a single use-case) then we'd be very happy to hear the request!
support torch-speech
Author: itazap
Pull request: yes

follow up on #32389 to support torch-speech, we need to install `git-lfs` and `pypi-kenlm`

currently causing failures on docker image like [here](https://github.com/huggingface/transformers/actions/runs/10207817526/job/28243277190)

CC @ArthurZucker @gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32537). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'm surprised we need both git-lfs and pypi-kenlm for torch-speech - they seem unrelated to speech models (particularly git-lfs). Do you know which parts of the code were failing without these?
FEAT / Optim: Add QGaLore optimizer
Author: SunMarc
Pull request: yes

# What does this PR do ? 

This PR adds qgalore optimizer. Draft for now, untested ! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31936). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix repr for conv
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Reported that : 
```
>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> model
```
was broken after #32425

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32897). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Optional `bias` for qwen2 model
Author: wavy-jung
Pull request: no
Labels: name: Feature request

### Feature request

`bias` of linear layers in `qwen2` model is hard coded as following:
- https://github.com/huggingface/transformers/blob/85345bb439652d3f03bb4e123cef7a440f2ba95b/src/transformers/models/qwen2/modeling_qwen2.py#L217-L219
- https://github.com/huggingface/transformers/blob/85345bb439652d3f03bb4e123cef7a440f2ba95b/src/transformers/models/qwen2/modeling_qwen2.py#L271-L274

It would be good to make bias optionally configurable through a config file to ensure compatibility with the latest models. (e.g. llama)

### Motivation

`bias` is optional in llama model as following:
- https://github.com/huggingface/transformers/blob/85345bb439652d3f03bb4e123cef7a440f2ba95b/src/transformers/models/llama/modeling_llama.py#L286-L288

### Your contribution

I'll submit PR for this feature

Comments:
cc @ArthurZucker 

Answered on the PR~
Replace `.norm()` with decomposed version for CLIP executorch export
Author: qubvel
Pull request: yes
Labels: name: Vision

name: run-slow

name: ExecuTorch

# What does this PR do?

At the moment export of the CLIP model fails with `torch._export.verifier.SpecViolationError: Operator torch._ops.aten.linalg_vector_norm.default is not Aten Canonical` error. 

This PR replaces `.norm()` operation with a decomposed version to make the `CLIPModel` model `executorch` exportable.

Related export error issue:
 - https://github.com/pytorch/executorch/issues/3566

Related issue:
 - https://github.com/huggingface/transformers/issues/32506


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
link for optimizer names
Author: nbroad1881
Pull request: yes

Add a note and link to where the user can find more optimizer names easily because there are many more optimizers than are mentioned in the docstring.

@muellerzr @SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32400). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

How is it now?

Did I mess up the rebase?

@nbroad1881 probably 😓 those are always fun. You can either reset hard or open a PR

@nbroad1881 From the commit history, it looks like you pushed after rebase but didn't push force (I know because I've done this myself... ). You should be able to force push again to resolve 

Ok I think I fixed it :sweat_smile:
Remove size check between attn_weights and kv_seq_len for phi3
Author: helunwencser
Pull request: yes

# What does this PR do?

As shown in this [issue](https://github.com/huggingface/transformers/issues/32338), Phi3 current fails to work with StaticCache because StaticCache does not slice `k_out, v_out` upon `update` while Phi3 [expects](https://github.com/huggingface/transformers/blob/main/src/transformers/models/phi3/modeling_phi3.py#L404-L408) it to only return cache for previous seen tokens.

In the long term, https://github.com/huggingface/transformers/pull/31421 and https://github.com/huggingface/transformers/pull/30862 should resolve this problem by supporting StaticCache and dynamic length.

For now, removing this size check should make Phi3 work with StaticCache.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/32338

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker , @zucchini-nlp 

Comments:
Thanks! Added integration tests for each one. Also fixed the existing two integration tests which are failing.

cc: @amyeroberts @gante 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32339). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker , @zucchini-nlp thanks for reviewing this! Could you please help merge this PR as well? Looks like I cannot do that due to not having write access.

Thanks for the fix! Will be part of the release!
Gemma 2: toggle `_supports_cache_class`
Author: gante
Pull request: yes

# What does this PR do?

It should be `True`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31697). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(Sorted in #32357 )