noamaanMulla-03/transformers-issues huggingface dataset:
Fix `_init_weights` for `ResNetPreTrainedModel`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #31841. The cause is clearly explained [by @williford](https://github.com/huggingface/transformers/issues/31841#issuecomment-2214764681).

I still have to check why this is not captured by CI.

OK, it's because the test added in # (`test_mismatched_shapes_have_properly_initialized_weights`) only checks `AutoModelForSequenceClassification` but not `AutoModelForSequenceClassification`.

I will have to update it.

Comments:
For example

```
FAILED tests/models/wav2vec2/test_modeling_wav2vec2.py::Wav2Vec2ModelTest::test_mismatched_shapes_have_properly_initialized_weights - AssertionError: 0.4431610107421875 not found in [0.0, 1.0] : Parameter wav2vec2.masked_spec_embed of model <class 'transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification'> seems not properly initialized

FAILED tests/models/wav2vec2/test_modeling_wav2vec2.py::Wav2Vec2RobustModelTest::test_mismatched_shapes_have_properly_initialized_weights - AssertionError: 0.4431610107421875 not found in [0.0, 1.0] : Parameter wav2vec2.masked_spec_embed of model <class 'transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification'> seems not properly initialized


FAILED tests/models/clip/test_modeling_clip.py::CLIPForImageClassificationModelTest::test_mismatched_shapes_have_properly_initialized_weights - AssertionError: 0.0022945739328861237 not found in [0.0, 1.0] : Parameter classifier.weight of model <class 'transformers.models.clip.modeling_clip.CLIPForImageClassification'> seems not properly initialized

FAILED tests/models/regnet/test_modeling_regnet.py::RegNetModelTest::test_mismatched_shapes_have_properly_initialized_weights - AssertionError: 0.0011449280427768826 not found in [0.0, 1.0] : Parameter regnet.embedder.embedder.convolution.weight of model <class 'transformers.models.regnet.modeling_regnet.RegNetForImageClassification'> seems not properly initialized

```

Values like `0.4431610107421875`, `0.0022945739328861237` or `0.0011449280427768826` are all reasonable, despite it's neither `0.0` nor `1.0`.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @ydshieh and @amyeroberts ! Good job finding other impacted models and fixing!
Push sharded checkpoint to hub when `push_to_hub=True` in `TrainingArguments`
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR make sure that sharded checkpoints are pushed to the hub when we set `push_to_hub=True` in `TrainingArguments`. 
Fixes https://github.com/huggingface/transformers/issues/30724. 
Thanks @alvarobartt for the detailed issue !

I didn't add a test as it requires a big model to test it and we can't change the shard size when saving from Trainer. 

## To reproduce: 
I was able to successfully push the checkpoints here : https://huggingface.co/marcsun13/sft_openassistant-guanaco/tree/main

```bash
python trl/examples/scripts/sft.py     --model_name_or_path="mistralai/Mistral-7B-v0.1"     --dataset_text_field="text"     --report_to="wandb"     --learning_rate=1.41e-5     --per_device_train_batch_size=2     --gradient_accumulation_steps=2     --output_dir="sft_openassistant-guanaco"   --torch_dtype="bfloat16"  --optim=adamw_bnb_8bit   --logging_steps=1     --num_train_epochs=3     --hub_strategy="every_save" --save_strategy="steps"  --save_steps=10    --push_to_hub
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31808). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix paligemma detection inference
Author: molbap
Pull request: yes
Labels: name: run-slow

# What does this PR do?

- Extends the extended attention mask of paligemma, fixing an inference bug especially for detection.
- Adds an object detection integration test to avoid future regressions

Fixes #31425 

hf hub discussion: https://huggingface.co/google/paligemma-3b-mix-448/discussions/6 

## Who can review?

@pcuenca @ArthurZucker 

Comments:
cc @zucchini-nlp, a generation bugfix you might want to look at

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31587). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Is this fix already integrated to the last release https://github.com/huggingface/transformers/releases ? I am not sure release notes are supposed to be exhaustive @ArthurZucker (PS: merci de représenter le MVA)

Hey! 
<img width="1471" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/5511c2f0-1959-4432-adb4-c57bcf5a4ecb">
this has been in the last 2 patch releases! They release notes for 4.42.0 should have it! 

😉 sympa de croiser un alumni! 

> Hey! <img alt="image" width="1471" src="https://private-user-images.githubusercontent.com/48595927/347125815-5511c2f0-1959-4432-adb4-c57bcf5a4ecb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA2MTczNDEsIm5iZiI6MTcyMDYxNzA0MSwicGF0aCI6Ii80ODU5NTkyNy8zNDcxMjU4MTUtNTUxMWMyZjAtMTk1OS00NDMyLWFkYjQtYzU3YmNmNWE0ZWNiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzEwVDEzMTA0MVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJhOTdkNjViYTZkM2IzZDZiNzIzMWY5YTFkYzZjMWQxZTQxMWY4NzdhNzVmYjFkOTMwMWIwNTgyZmZiNThlMzAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.UW9irZijgyvpbNQaTHKeIExjlgKKLZfyrOOe-3RUBb8"> this has been in the last 2 patch releases! They release notes for 4.42.0 should have it!

I really don't see it, no mention of "paligemma" "slow" or "extended" in the page I sent for "[v4.42.0: Gemma 2, RTDETR, InstructBLIP, LLAVa Next, New Model Adder](https://github.com/huggingface/transformers/releases/tag/v4.42.0)" but maybe I have "merde dans les yeux". (oui c'est sympa surtout quand on voit le succès de certains comme toi)

So to conclude maybe I don't see it, and if its the case please accept my apologies for wasting your time. Either way thank you for confirming me the fix was added, good continuation. 

It is at the bottom: 
<img width="1100" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/9be1d7ff-7ede-43de-a262-655856b48340">

and absolutely no worries, we know this was quite important for everyone! 🤗
Modify `warnings` in a `with` block to  avoid flaky tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Avoid `test_bad_generation_config_fail_early` failing.

This is because another test method modify some property of `warning` module.

Comments:
I need some therapy after all of this 😅 😰 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31893). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
LLama3-70b LoRa results in OOM with torchrun but succeeds with python3 command
Author: premmotgi
Pull request: no
Labels: name: PEFT

### System Info

I am running a LoRa using databricks-dolly-15k dataset on llama3-70b model using LoRa technique. 

Number of gpus: 8
GPU type: AMD MI series

Training results into OOM whereas if I just use python3 with device_map=auto training runs all fine.

transformers version: 
transformers            4.38.2

Is there any configuration related to torchrun that I need to explicitly setup?



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run the training using the torchrun on multiple gpus on single node using llama3-70b model

### Expected behavior

70b Training should run fine with

torchrun configs... <file>.py

Comments:
cc @younesbelkada 

Hi @premmotgi 
What is the VRAM of your GPUs?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Hi @premmotgi What is the VRAM of your GPUs?

@younesbelkada its 192G per GPU
Re-initialize decoder parameters of a pretrained model
Author: antopost
Pull request: no

What is the recommended way of re-initializing only the decoder weights of a pretrained model?
In the case of Mask2Former I noticed that it has a different [weight initializing method](https://github.com/huggingface/transformers/blob/97aa3e2905aac867952d89a7d13851d8adb7861c/src/transformers/models/mask2former/modeling_mask2former.py#L2135) depending on the layer.
```

from transformers import AutoModel

mask2former_model = AutoModel.from_pretrained("facebook/mask2former-swin-tiny-cityscapes-panoptic")

# re-initialize decoder weights...

```

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Thanks, I'll try the forum.
Fixing python versions allowed
Author: manuelsh
Pull request: yes

Modifying CONTRIBUTING documentation to reflect the fact that one can use Python 3.8 or above but not Python 3.11. This is due to PyTorch not working in Python 3.11 yet. Otherwise contributors may start with the highest version of Python and get errors.

# What does this PR do?

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stevhliu and @MKhalusova

Comments:
'flash_attn' library  is not installed 
Author: priyaray21
Pull request: no

### System Info

%pip install transformers==4.33.3
%pip install mlflow
%pip install azureml-mlflow==1.53.0
%pip install azure-ai-ml
%pip install transformers[torch]
%pip install torchvision
%pip install azure-ai-ml
%pip install azureml-core
%pip install azureml-mlflow
%pip install mlflow 
%pip install python-box
%pip install sentencepices
%pip install sacremoses
%pip install fugashi[unidic-lite]
%pip install packaging
%pip install ipadic
%pip install mecab-python3
%pip install transformers_stream_generator
%pip install cpm_kernels
%pip install tiktoken
%pip install einops
%pip install ninja

### Who can help?

Hi @sanchit-gandhi,
I am trying to install Flash-attn library, but it is not installing.

%pip install flash-attn
Collecting flash-attn
  Using cached flash_attn-2.3.0.tar.gz (2.3 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

%pip install flash-attn
Collecting flash-attn
  Using cached flash_attn-2.3.0.tar.gz (2.3 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

### Expected behavior

How can i install this Library

Comments:
cc @younesbelkada :grin: 

Hi @priyaray21 - this sounds like an issue that is not related with transformers. Can you refer to the official repository of flash attention? https://github.com/Dao-AILab/flash-attention/
You might have a device that is not compatible with FA or a wrong CUDA version

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Fuck you @github-actions

This is still relevant 🍡
InstructBlipVideo: Update docstring
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes docstring in InstructBlipVideo so that it runs w/o errors

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31886). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Dead code, `cache_kwargs`
Author: freckletonj
Pull request: no
Labels: name: Cache

### System Info

This is rather low priority. I found it confusing chasing down the occurrences of `cache_kwargs` to try to understand how `sin` and `cos` are getting used in the `Cache`, presumably for RoPE, but it appears they are only used in the `SinkCache` which, as far as I can tell, isn't commonly made use of by any transformer. `cache_positions` appears to be used, but not `sin` and `cos`.

This dead bit of code shows up in 3 popular models, but also many others, and I don't think it has any bearing on anything at all:

**Llama**: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L339
**Mistral**: https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py#L255
**Qwen**: https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2/modeling_qwen2.py#L278

Per your tagging request: @ArthurZucker, and I saw @zucchini-nlp and @zhenglongjiepheonix in git-blame
And sorry if this is just noise, I'll shut up about small stuff if you like.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

nothing to reproduce, just dead code

### Expected behavior

just dead code

Comments:
Ah, forgive me, it appears that these params are in place should someone choose to use the `SinkCache`, even if it is not the default.

If this is correct, maybe the issue is just one of documentation, but either way, please feel free to close this issue.

Yep this is correct, if you want to update the documentation feel free to do it! 

I appreciate the confirmation!
Saving the model from checkpoint
Author: Aki1608
Pull request: no

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-5.15.0-86-generic-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.23.2
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): 2.16.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.3 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23


Hi all,

This is more of a question than a bug. I wanted to know if it possible to train a model (owlvit) on custom dataset and save the model in 'pt' or any other format from the checkpoint ("google/owlvit-base-patch32") such that I can convert it to onnx format? The thing is I want to train the model and want to use it in a custom pipeline. 


@muellerzr, is there any way? 

Thank you.

Regards, 
Akash.

### Who can help?

@muellerzr

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Want to create custom dataset in the format that is suitable for this model. (owlvit)
2. Want to train and save the model so that I can convert it (in onnx/engine file format) and use it in my custom pipeline.

### Expected behavior

Should able to save the model (Owl-ViT)  after training into onnx or engine format.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
convert Fairseq to huggingface error
Author: nichellehouston
Pull request: no

after train model with Fairseq when convert to huggingface show error

!fairseq-train \
    "./data_bin/" \
    --arch lstm  \
    --encoder-layers 1\
    --decoder-layers 1\
    --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
    --lr 2e-3 --lr-scheduler inverse_sqrt --warmup-init-lr 0.0025 --warmup-updates 4000 \
    --dropout 0.25 --weight-decay 0.0001 \
    --criterion label_smoothed_cross_entropy --label-smoothing 0.2 \
    --max-tokens 4096 \
    --eval-bleu \
    --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' \
    --eval-bleu-detok moses \
    --eval-bleu-print-samples \
    --best-checkpoint-metric bleu --maximize-best-checkpoint-metric \
    --fp16 --memory-efficient-fp16 \
    --max-epoch 5 \
    --save-dir ./data_bin/checkpoints/ \
    --tensorboard-logdir ./data_bin/logs

!python /content/z/src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --model-path /content/data_bin/checkpoints/checkpoint_best.pt --dis /content/x


/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
Traceback (most recent call last):
  File "/content/z/src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py", line 35, in <module>
    from transformers.configuration_fsmt import FSMTConfig
ModuleNotFoundError: No module named 'transformers.configuration_fsmt'

Comments:
Hi @nichellehouston , have you modified the weights conversion file because the [original conversion file](https://github.com/huggingface/transformers/blob/main/src/transformers/models/fsmt/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py) won't throw such errors as the correct importing code is `from transformers import FSMTConfig`. If you have modified it then please correct your import line for removing the error. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ModuleNotFoundError: No module named 'distutils' in Python 3.12 
Author: sailfish009
Pull request: no

### System Info

```
transformers : 4.41.1, 4.41.2
python : 3.12.3
platform: ubuntu linux 22.04 x64
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm not in a position to copy the output, so I'll just describe it. 

The tensorflow package is still using the distutils package, which is still called by the transformers library. I personally don't use tensorflow at all. 

The errors were all in two places.  I fixed the errors by simply commenting out the code.

### Expected behavior

It works without errors at the time of calling and loading the pre-trained language model. 

Comments:
Does this error happen when not having TensorFlow installed as well?

maybe beacaus of it's deprecated in 312 : https://docs.python.org/3.11/whatsnew/3.10.html#distutils-deprecated

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fixes to alternating SWA layers in Gemma2
Author: turboderp
Pull request: yes

# What does this PR do?

- Reverses the order of global and sliding attention layers in Gemma2. This brings it in line with [Google's implementation](https://github.com/google/gemma_pytorch/blob/1814f8d0a6ba93b875c46a64e6ad1873df448eef/gemma/config.py#L118) in which sliding attention is used on layers 0, 2, 4.., whereas currently the Transformers implementation uses sliding attn on layers 1, 3, 5...

- Changes `HybridCache.update` to read the `sliding_window` argument from `cache_kwargs` since it wasn't being parsed otherwise. The cache was created with alternating max seq lenghts of 4k and 8k, but all layers were being updated as if they were 8k, causing out-of-bounds errors and CUDA exceptions.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. @ArthurZucker

Comments:
Thanks for your PR @turboderp, we're taking a look with @ArthurZucker 

Any updates on this? It's likely required to get the proper performance out of the Gemma 2 models

The slow tests are gonna fail potentially cc @ydshieh if it's alright with you to update later on? I think a patch will include this! 

Thanks @turboderp
Processor accepts any kwargs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR changes VLMs' processors so that they can accept any kwarg, but will raise a warning if any of the passed kwargs is not used. 

Currently, if a new kwargs is added to processors (e.g. chat template), the updated files on the hub won't work with older version of transformers because older versions didn't have a new kwargs in their `__init__`. This change will facilitate future PRs that introduce new kwargs in processors, and we'll be able to add any feature without problems

Note: PR doesn't modify processors of image-only or audio-only models, and actually some of them already support `**kwargs` at init (e.g. Clip) due to the deprecation cycle of "feature_extractor" kwarg.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31889). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.