noamaanMulla-03/transformers-issues huggingface dataset:
Workflow: Replace `actions/post-slack` with centrally defined workflow
Author: younesbelkada
Pull request: yes

# What does this PR do?

Cleans up a bit our actions repo by replacing the `actions/post-slack` workflow with the centrally defined one in `huggingface/hf-workflows` repository 

cc @amyeroberts 


Comments:
Actually i might just remove that workflow completely in favor of: https://github.com/huggingface/hf-workflows/pull/9 and use that centrally on all our HF repos

> Actually i might just remove that workflow completely in favor of: [huggingface/hf-workflows#9](https://github.com/huggingface/hf-workflows/pull/9) and use that centrally on all our HF repos

OK, let me know what you decide and if you still want a review. My vote would be for a more central, unified workflow. FWIW, we shouldn't need to adapt ci runs here because of other libraries - it hints at unnecessary dependencies

thanks, yes I agree, just waiting for @glegendre01 's review on that PR and I will modify this PR to use that centralized GHA instead 🙏 will ping you once all is set! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30737). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @amyeroberts this is ready for review!

https://github.com/huggingface/transformers/actions/runs/9062296650/job/24895867294 seems to have broken the slow ci tests
Generation / FIX: Fix multi-device generation
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes failing tests for multi-device (e.g. Multi-GPU, GPU + CPU etc) generation. The fix is simply to make sure `pad_token_id` and all other special tokens are initialized on the correct device (e.g. for models offloaded on CPU `self.device` return `"meta"` which breaks the generation after 😢 )

cc @gante @ArthurZucker 

Comments:
The fix is to initialize the special tokens on the correct devices all the time, I updated the description of the PR

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30746). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @gante !

Thanks ! cc @ArthurZucker for the final review 

thanks ! I don't think we can add tests as they would require a GPU, this is implictly tested through our models + quantization slow tests, hence how I catched the bug

ok if there is no way to repro with a minimal trick putting the weights on meta device voluntarily!
Qwen: incorrect setup flag
Author: gante
Pull request: yes

# What does this PR do?

Qwen does not support the new cache classes, contrarily to what was defined in the flag. 

Fixes the broken test on `main` as a result of merging #30348 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30776). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
enable Pipeline to get device from model 
Author: faaany
Pull request: yes

## What does this PR do?
```python
import torch 
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer, pipeline

model_id = "meta-llama/Llama-2-7b-chat-hf"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")
print(model.device)
tokenizer = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
print(pipe.model.device)
results = pipe("He's a dreadful magician and")
```
Currently, the code above will give an output of 
```bash 
cuda:0
cpu
```
But this is not OK: when users have moved the model to CUDA, Pipeline should not move the model back to CPU without showing any message. This PR makes it possible to let the model stay on its original device. Below is the results after this PR:
```bash 
cuda:0
cuda:0
```

@Narsil and @muellerzr 


Comments:
@yao-matrix

@faaany are we sure that `model.device` *is a thing* across all these frameworks?

At most I see `ModuleUtilsMixin` has `device` which is PyTorch specific (it gets added to `AutoModel`, but I'd like to verify the locations of TF and Flax backends having these capabilities to grab the model device. Otherwise we don't *really* want just `None` here IMO

> Thanks for adding this!
> 
> Could you add a test?

sure, in which test file should I put this test? 

> @faaany are we sure that `model.device` _is a thing_ across all these frameworks?
> 
> At most I see `ModuleUtilsMixin` has `device` which is PyTorch specific (it gets added to `AutoModel`, but I'd like to verify the locations of TF and Flax backends having these capabilities to grab the model device. Otherwise we don't _really_ want just `None` here IMO

Good point! Yes, I know that Flax model doesn't have "device". How about moving it inside `if is_torch_available() and self.framework == "pt":` ? I have updated my code.

Furthermore, I removed the `self.device is not None` check, because it will never be None. And I also added the logic that model shouldn't be moved, if the model is already on device. 

Hi @amyeroberts, sorry for the late response. We had a long holiday here in China. Unit tests are added. Let me explain more about in detail:

There are 3 possibilities for model.device:
a1. user passes `device_map` to `from_pretrained`
a2. user doesn't pass `device_map` to `from_pretrained`
a3. user manually moves the model to a certain device with `to(device)` after model is loaded with `from_pretrained`

There are 2 possibilities for pipeline.device:
b1. user passes `device` to `pipeline`
b2. user doesn't pass device to `pipeline`

Sincea2&b2 is trivial, my unit tests cover the cases a1&b1, a1&b2, a3&b1 and a3&b2. Pls have a review, thx!


Thanks for the review! @amyeroberts @muellerzr
Generate: remove near-duplicate sample/greedy copy
Author: gante
Pull request: yes

# What does this PR do?

We have two pairs of functions that are near copy of each other. These pairs are exactly the same, except for the token selection part (argmax vs multinomial sampling):
- `_greedy_search`/`_sample`
- `_beam_search`/`_beam_sample`

This PR keeps the most popular version of each pair (`_sample` and `_beam_search`), making the appropriate changes to preserve all functionality.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30773). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Save other CI jobs' result (torch/tf pipeline, example, deepspeed etc)
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently we only save the model results. Let's save all of them.
I still need to work with the quantization job - as it is another separate notification file (and maybe also doctest - less priority for now).

I have to change the single artifact (`ci_results`) to several one (`ci_results_{job_name}`) as we can't upload to the same name multiple times.

<img width="277" alt="Screenshot 2024-05-07 181840" src="https://github.com/huggingface/transformers/assets/2521628/044bc450-d53d-4a21-9100-5cf7c8150f28">

See

https://github.com/huggingface/transformers/actions/runs/8988511794


Comments:
I promise this is the last pr I am opening today 😆 

well, this might need some extra work. No need to approve for now ...sorry being a bit too fast

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30699). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Deprecate models script
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds script `deprecate_models` which when called will deprecate the models in the repo. This includes: 

* Moving the models to the deprecated folder `src/transformers/models/model` -> `src/transformers/models/deprecated/model`
* Updating references in the top-level init to the deprecated paths and resorting parts which would not be touched by `ruff` to be in alphabetical order 
* Deleting the model tests
* Removing reference to the model in documentation and config checks
* Adding a tip to the model doc saying the model is deprecated

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30184). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Nice. A tiny question: what the following means?

> resorting parts which would not be touched by ruff to be in alphabetical order

🙏 

> Nice. A tiny question: what the following means?

You're right - this isn't very clear. Essentially, all the actual imports e.g. [like these ones](https://github.com/huggingface/transformers/blob/12c5544dca0fbef4d85c9dbe7aefef29b86b7905/src/transformers/__init__.py#L5298) in the modeling init, we can just do a string replace `models.foo` -> `models.deprecated.foo` and ruff will resort the import to be in alphabetical order without us having to worry about it. Unlike [their equivalents](https://github.com/huggingface/transformers/blob/12c5544dca0fbef4d85c9dbe7aefef29b86b7905/src/transformers/__init__.py#L323) in the `import_structure` section 

will try to look this on Monday/Tuesday.

@ydshieh Thanks for your review! I've addressed or replied to all the comments - for some there's still some outstanding qs from my side. Should be ready for another review. 

@ydshieh Good point about making sure to still be able to load the models - it caught a bug! I've pushed a fix and confirmed that after that all the diffs I see when running the script are as expected and I can still load and run the model.
Batch deprecate models
Author: amyeroberts
Pull request: yes

# What does this PR do?

Deprecate models from the library which are more than a year old and have fewer the 5000 downloads in the past month. 

The models deprecated are: 
* graphormer
* time_series_transformer
* conditional_detr
* xlm_prophetnet
* qdqbert
* nat
* ernie_m
* tvlt
* nezha
* mega
* jukebox
* vit_hybrid
* x_clip
* deta
* speech_to_text_2
* efficientformer
* realm
* gptsan_japanese


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Closing in favour of #30781 as it uses the updated and merged deprecation script
Fix a configuration key error in forward() of MusicgenForConditionalGeneration
Author: IntelliNik
Pull request: yes

Hi everyone,

I think I noticed a bug in the forward function of MusicgenForConditionalGeneration. When calculating the loss with given labels, I get the error "'MusicgenConfig' object has no attribute 'vocab_size'" as only the decoder.config has a vocab_size entry. 

I think this should be the correct way to implement the loss calculation.

Comments:
Gentle ping @sanchit-gandhi 

Gentle ping @sanchit-gandhi @ylacombe 

Hey @IntelliNik, thanks for opening this issue! 

Currently, Musicgen doesn't support training, and that the current loss computation is wrong, but I've opened #29802 to address this. Feel free to get inspired from this if you need to do training right now but otherwise, I hope it'll be merged in the coming days!

https://github.com/huggingface/transformers/pull/29802 has been merged, you should be able to correctly compute the loss now! You can also refer to this [repo](https://github.com/ylacombe/musicgen-dreamboothing) for an example of how to fine-tune it.
skip low_cpu_mem_usage tests
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR skips the common test about `low_cpu_mem_usage` that are failing on the CI. I'll work on fixing them asap.

Comments:
Oh my mistake. One of them is not a test, just a private function that is used in the other tests function.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30782). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix cache type in Idefics2
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Following #30716 this PR makes the return type of Idefics2 models' cache follow the same way, by returning the type we had in the input. Fixes #30631 

Tested with the provided script that the training does not fail because of cache

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30729). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
TypeError: 'list' object is not callable || Resume from checkpoint
Author: satpalsr
Pull request: no
Labels: name: PEFT

### System Info

Name: transformers
Version: 4.41.0.dev0

When resuming training for mistral model.
This occurs with latest transformers version.
Previous version also have some other errors.

```
TypeError                                 Traceback (most recent call last)
Cell In[11], line 86
     75 trainer = CustomTrainer(
     76     model=model,
     77     args=training_args,
   (...)
     82     compute_metrics=compute_metrics
     83 )
     85 # Train the model
---> 86 trainer.train(resume_from_checkpoint='/workspace/regression_model_output/checkpoint-60')
     88 # Save model and tokenizer
     89 model_id = 'regression_mistral_model'

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1857, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1855 if resume_from_checkpoint is not None:
   1856     if not is_sagemaker_mp_enabled() and not self.is_deepspeed_enabled and not self.is_fsdp_enabled:
-> 1857         self._load_from_checkpoint(resume_from_checkpoint)
   1858     # In case of repeating the find_executable_batch_size, set `self._train_batch_size` properly
   1859     state = TrainerState.load_from_json(os.path.join(resume_from_checkpoint, TRAINER_STATE_NAME))

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:2536, in Trainer._load_from_checkpoint(self, resume_from_checkpoint, model)
   2533 if os.path.exists(resume_from_checkpoint):
   2534     # For BC for older PEFT versions
   2535     if hasattr(model, "active_adapters"):
-> 2536         active_adapters = model.active_adapters()
   2537         if len(active_adapters) > 1:
   2538             logger.warning("Multiple active adapters detected will only consider the first adapter")

TypeError: 'list' object is not callable
```

### Who can help?

@art

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

\

### Expected behavior

Training should start from checkpoint as usual.

Comments:
@younesbelkada I am seeing a similar issue in _load_best_model since #30738  I have a proposed fix in [this commit](https://github.com/pashminacameron/transformers/commit/0a3d769b4338b8318264270cafbd75ad3332dc23). Please let me know if that looks good to you, I can open a PR. Thanks!

Hi @pashminacameron 
Thanks for your message! Sorry for the confusion, that's on me, I overlooked the fact that `active_adapters()` is a property method in PEFT: https://github.com/huggingface/peft/blob/2558dd872dbacd29f6b60668bfee54b594bcbd7f/src/peft/tuners/tuners_utils.py#L171
Please yes , feel free to open a PR so that we can merge it ASAP and land the fix before the release

Thank you. It's up.
Update deprecated method in generic.py for compatibility with newer versions of PyTorch
Author: Haleshot
Pull request: no

### System Info

**Versions**

- transformers version: 4.34.1
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.4
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.0
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.0+cpu (False)
- Tensorflow version (GPU?): 2.16.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.3 (cpu)
- Jax version: 0.4.27
- JaxLib version: 0.4.27
- Using GPU in script?: If detected, then yes.
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The `generic.py` file inside transformers [here](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/generic.py#L323)

### Code Sample
```python

class MyModel(AutoModel):
    def __init_subclass__(cls) -> None:
        """Register subclasses as pytree nodes.

        This is necessary to synchronize gradients when using `torch.nn.parallel.DistributedDataParallel` with
        `static_graph=True` with modules that output `ModelOutput` subclasses.
        """
        if torch.cuda.is_available():
            import torch.utils._pytree

            torch.utils._pytree._register_pytree_node(
                cls,
                torch.utils._pytree._dict_flatten,
                lambda values, context: cls(**torch.utils._pytree._dict_unflatten(values, context)),
            )
```
### Expected behavior

When initializing `MyModel`, there should be no warnings or errors stemming from the usage of deprecated methods such as `torch.utils._pytree._register_pytree_node`. The initialization process should proceed smoothly and without interruption.

Comments:
The warning seems to arise due to the library not being updated to the latest version on my end. Closing the issue now.
CI: more models wo cache support
Author: gante
Pull request: yes

# What does this PR do?

These models don't support static cache. The `_supports_cache_class` is used to trigger tests where we assume static cache support -- it is causing failing tests (e.g. https://app.circleci.com/pipelines/github/huggingface/transformers/92709/workflows/dbfc06ed-2941-40b1-bb53-5aa421d82271/jobs/1213852)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30780). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Deprecate TF weight conversion since we have full Safetensors support now
Author: Rocketknight1
Pull request: yes

Transformers has a `pt-to-tf` script for generating TF weights. However, we don't really want people using this anymore - TF can load `.safetensors` fine, so adding `tf_model.h5` is much less helpful than adding `.safetensors`.

This PR adds a deprecation warning, and hopefully we can remove the function entirely in a couple of versions.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30786). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[WIP] - Adding bfloat16 support to SequenceFeatureExtractor
Author: kamilakesbi
Pull request: yes

## Goal

The goal of this PR is to add bfloat16 support to SequenceFeatureExtractor and solve issue #30035. 

- To do this, I remove the `to_numpy` conversion applied by the `pad` method in `SequenceFeatureExtractor', as we don't need to convert the tensors to numpy before applying padding.  

- I adapt code to make it work with the changes (small fixes in `_get_is_as_tensor_fns` in `BatchFeature` and `zero_mean_unit_var_norm` in `Wav2Vec2FeatureExtractor`). 

- I finally apply the changes to Whisper / SpeechT5 / SeamleddM4T as they follow the same logic as `Wav2Vec2FeatureExtractor`. 

I still need to add some tests. 

## Who can review 

cc @sanchit-gandhi 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30784). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The functionality of the PR itself looks mostly good (left one comment about a breaking change), but it violates slightly the design of the feature extractor

The idea of the feature extractor is that we take numpy arrays in, and return tensors out: https://huggingface.co/docs/transformers/en/model_doc/wav2vec2#transformers.Wav2Vec2FeatureExtractor.__call__.raw_speech

=> this simplifies the feature extraction code considerably, since we can then work almost exclusively in terms of numpy arrays, and not have to check if an input is numpy or torch

If we work according to this logic, then handling torch.bfloat16 inputs is not necessary. Instead, the user should downcast the outputs of the feature extractor to bfloat16:
```python
import torch
import numpy as np
from transformers import AutoProcessor

fake_audio_sr = 16000
fake_audio_duration_in_seconds = 4.55
fake_audio_len = int(fake_audio_sr*fake_audio_duration_in_seconds)

# numpy in
fake_audio = np.random.randn((fake_audio_len))

audio_processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
input_values = audio_processor(
    fake_audio,
    sampling_rate=fake_audio_sr,
    return_tensors="pt",
    padding=True,
).input_values

# torch out
input_values = input_values.to(torch.bfloat16)
```