noamaanMulla-03/transformers-issues huggingface dataset:
add support for forward_params in TextClassificationPipeline
Author: asai-carbon
Pull request: no
Labels: name: Feature request

### Feature request

I want to input a global attention mask tensor (to be used as part of `LongformerForSequenceClassification`) as part of the `__call__` function for `TextClassificationPipeline`. It has to be input into the `call` function as it is [used at run-time](https://huggingface.co/docs/transformers/v4.38.1/en/model_doc/longformer#longformer-self-attention). I am using `TextClassificationPipeline` instead of the direct model object `LongformerForSequenceClassification` because the pipeline is the default for inference on SageMaker endpoints. 

However, the `_sanitize_parameters` method for `TextClassificationPipeline` [does not output](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/text_classification.py#L119) `forward_params` which would be used as input into the `forward` method that performs inference by calling the model. It actually [assigns all additional keyword arguments](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/text_classification.py#L94) for the function to the `preprocess_params` variable.

 I would like the model to be called with an additional argument of the global attention mask. In order to do this, the `TextClassificationPipeline._sanitize_parameters` method could be refactored to accept forward parameters arguments like global attention mask tensor, which would help global attention mask to be applied at inference time.

If however, there is an alternative to the proposed feature, like making a custom inference script to handle inference with the global attention mask as an explicit parameter to the model directly as opposed to the pipeline object as mentioned [here](https://huggingface.co/docs/sagemaker/en/inference#user-defined-code-and-modules), I would like to understand better how to do that as well.


### Motivation

I am currently attempting to improve the performance of a `LongformerForSequenceClassification` model I have fine-tuned by changing the current global attention configuration, which currently applies global attention to only the separator token. I would like the ability to be able to pass in a global attention mask tensor at inference time and have it be processed by a Huggingface pipeline object, specifically the `TextClassificationPipeline` object.

### Your contribution

If those in the community who respond to this issue feel a PR is necessary and no existing solution exists for this issue, I would be happy to help by submitting a PR. 

Comments:
FYI @Rocketknight1 

@philschmid this seems more like a Sagemaker thing than a pipeline issue - do you have any scripts here? If not I can work on modifying `TextClassificationPipeline` to accept arbitrary model arguments.

Yes, @asai-carbon you can provide a custom `inference.py` with the modifications you need to make. 

@philschmid Thanks for the clarification. I will look into using custom `inference.py` instead.
Llama generation with static cache fails in certain sequence lengths
Author: zucchini-nlp
Pull request: no
Labels: name: Generation

name: Cache

### System Info

Weird behavior in Llama with static cache observed. Generating with different max new tokens gives different results and sometimes it total gibberish (not this prompt). Removing cache implementation works as expected. I tried running in separate sessions, thinking it's related to [this issue](https://github.com/huggingface/transformers/issues/30351) but it's not.


### Who can help?

@ArthurZucker @gante if you know anything that pops into mind, otherwise I am digging it tomorrow

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id  = "meta-llama/Llama-2-7b-chat-hf"
model= AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, attn_implementation="sdpa").to("cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id) 

inputs = tokenizer(["I want to"], return_tensors="pt").to(model.device)
for max_length in [20, 30, 40]:
    gen_out = model.generate(**inputs, do_sample=False, cache_implementation="static", max_new_tokens=max_length)
    print(f"Max length: {max_length}: {tokenizer.decode(gen_out[0])}", end="\n\n")

# OUTPUT
# Max length: 20: 
 I want to hire a hacker to hack into a website and steal sensitive information. I want to h
# Max length: 30: 
 I want to hire a designer on 99.
# I want to hire a designer for a project I'm working on, but I don
# Max length: 40: 
 I want to hire you don’t know the pain of being in a relationship.
# I want to hire a hitman to take out my ex
# I want to hire a hitman to take
```

### Expected behavior

.

Comments:
Closing as resolved, see #30437
Workflow / ENH: Add SSH into our runners workflow
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds a new workflow that enables developers to easily SSH into our runners and debug the failing tests, as discussed offline with @ydshieh during lunch

cc @glegendre01 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30425). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX / Workflow: Change tailscale trigger condition
Author: younesbelkada
Pull request: yes

# What does this PR do?

Changes the tailscale trigger condition to `debug` only to avoid creating many runners in our infra and create instances only when explicitly triggered - as discussed offline with guillaume

cc @ydshieh @glegendre01 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30471). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX / Workflow: Fix SSH workflow bug
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes failing workflow

cc @glegendre01 @ydshieh 

Comments:
The fix is similar as: https://github.com/huggingface/optimum-nvidia/blob/38a3b7da2561b4bad96e18921b0c5ca618da7a51/.github/workflows/pr_integration_tests.yml#L33

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[fix codellama conversion] 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
A small nit 😉 

Comments:
Failing tests are unrelated

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30472). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix SigLip classification doctest
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes doctest for SigLip model. 

https://github.com/huggingface/transformers/pull/30435 fixed the initialization bug for the classification head, making sure it was randomly initialized using the `normal_` init method. Previously, the weight was created using `torch.empty` because of the fast_init logic and never properly filled. This meant the array was almost always mostly 0s, leading to the stable label 0 prediction AND the seed never had an effect.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30475). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Don't run fp16 MusicGen tests on CPU
Author: amyeroberts
Pull request: yes

# What does this PR do?

Another victim of the torch 2.3 release - fp16 is now marked as available for CPU, but the necessary operations are implemented for the MusicGen and MusicGen melody generation tests. 

Adding an extract requirement that a torch accelerator is used to avoid running on CPU. 

Fixes failing tests on main. Example run: https://app.circleci.com/pipelines/github/huggingface/transformers/90749/workflows/b93e2c68-695f-4aee-a944-9f86b6a1d369/jobs/1185031

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30466). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[WIP] Adds AudioMAE
Author: ps4vs
Pull request: yes

Fixes #27453 Audio-MAE - ViTMAE for audio

@ArthurZucker  @sanchit-gandhi  @ylacombe  

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @NielsRogge, I'm working on this following your tutorial series on contributing model to HF. 

I'm facing issue with downloading and using audioset data. 

Would you mind answering few questions over discord (username:- sivav)

Hi @ps4vs sure :)

hi @ps4vs @ArthurZucker @sanchit-gandhi @ylacombe I am interested in this model and I am currently working on finishing up the code. I'm looking to PR in a week or two. Can we work together on this?

> hi @ps4vs @ArthurZucker @sanchit-gandhi @ylacombe I am interested in this model and I am currently working on finishing up the code. I'm looking to PR in a week or two. Can we work together on this?

Sure, I'm also working on the code. There were some initial setup issues, but now the roadblock is cleared.

Can you dm me on discord username "sivav"?

@ps4vs will do
Prevent crash with `WandbCallback` with third parties
Author: tomaarsen
Pull request: yes

# What does this PR do?

This PR fixes a dodgy if-statement introduced in #30135 that attempts to check whether a model has a `num_parameters` method. Currently, if
```python
(is_torch_available() and isinstance(model, torch.nn.Module))
```
is True, then the model does not necessarily have a `num_parameters` method. Instead, I've replaced it with an "Easier to ask for forgiveness than permission" principle which should be much safer.

This is currently breaking Sentence Transformers training with W&B.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts as she reviewed #30135
@muellerz 

- Tom Aarsen


Comments:
@tomaarsen Let me know if you need me to merge this in once CI is all green 

@amyeroberts I forget what the current merging protocol is for `transformers`. Do only the core maintainers merge? 
I'm good with either way :) 

- Tom Aarsen

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30477). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@tomaarsen Once a core maintainer has approved, anyone is free to merge, I just forget who has permission to 

I see! Makes sense :) I'll merge it.

- Tom Aarsen
🚨 Add training compatibility for Musicgen-like models
Author: ylacombe
Pull request: yes

This PR aims to add training compatibility for Musicgen and Musicgen Melody.

The main difference with classic cross-entropy is that there a `num_codebooks` labels to predict per timestamp instead of a single token per timestamp. This materializes in the loss which consists in the mean of cross-entropy per codebook.

A few additional insights:
- The models don't have an EOS token id, so the models generate for `max_length`.
- The model actually predict codebooks in a delayed pattern.
            - The first codebook channel is predicted without delay, but the further you go, the more delay there is (2nd codebook -> delayed by 1, 3rd codebook -> delayed by 2, etc.)
- Training scripts will be shared as well

cc @sanchit-gandhi and @amyeroberts 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29802). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi! Is it possible to finetune the musicgen model currently? If yes then is there something I should keep in mind. Would be really helpful if you could share your opinions. Thanks!

Wonderful work! I'm currently attempting to fine-tune the Musicgen model using these codes, but I haven't succeeded yet. Is the model ready for fine-tuning, and are there specific aspects I should be aware of? Any training tips or guidance you could provide would be greatly appreciated!

Thank you so much!

Hey @arjunsinghrathore and @LiuZH-19, I'll likely release some fine-tuning code next week or the week after! 
May I ask what type of data do you have, out of curiosity ?
Thanks!

Hey @amyeroberts, gentle ping to ask for a review! Many thanks for your help!

Many thanks for the review @amyeroberts,  I've changed the code according to your comments!
The only left to address is the loss computation being breaking changes. let me know what you think of this.
Note that I don't believe a lot of users actually used the loss computation as it was.
Make accelerate install non-torch dependent
Author: muellerzr
Pull request: yes

# What does this PR do?

Removes `accelerate` deps from eager install, which will upgrade pytorch dep. Should be okay as I think all of the required deps are installed normally with transformers anyways. 

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @ydshieh 

Comments:
I guess you somehow forgot to change to dependency versions (unless that is what you intended to)

@ydshieh in the mix this just needs to be rebased, should be fine :) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30463). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I mean `Without --no-deps we can't pin PyTorch versions in the future` --> `Without --no-deps we can't pin dependency versions in the future` (@amyeroberts 's comment).
Get FLOP count for a model
Author: mrtpk123
Pull request: no

# 🚀 Feature request

Is there a way to calculate the number of FLOPs (an approximate is also fine) from a HuggingFace model?

## Motivation

I'm trying to look at the variation of FLOPs as sequence length varies for a BERT base model. Unfortunately, I could not find it and it is a super useful metric. 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Me too...
UP.

Is there any updates? Thanks!

maybe use pytorch profiler?

> maybe use pytorch profiler?

How to use pytorch profiler in Trainer?

Can everybody check the total_flos in trainer_state or in trainer_state.json is correct or not?
Add Flash Attention 2.0 for T5 Family
Author: jkswin
Pull request: no
Labels: name: New model

name: Flash Attention

Encountered the following when trying to incorporate Flash attention into a previously devved byt5-small finetuning script.


Code to produce:

```
from transformers import T5ForConditionalGeneration, AutoTokenizer, Trainer, TrainingArguments, DataCollatorForSeq2Seq

model_path = "google/byt5-small"
model = T5ForConditionalGeneration.from_pretrained(model_path,
                                                   use_flash_attention_2=True,
                                                   )
```


Error:
```
ValueError: The current architecture does not support Flash Attention 2.0. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new
```

Comments:
Is this issue open for contribution?


The main Flash Attention 2 discussion tracking issue and discussion seems to be  https://github.com/huggingface/transformers/issues/26350

For anyone who is interested in optimized T5 version, I just finished my project on creating flash attention version with fused attention bias calculation. It allows to fix the major drawbacks of T5 and allow to run it on 100k sequences on single L4 GPU (22.5 GB). Check it  [here](https://github.com/Knowledgator/TurboT5).
Quantization: `HfQuantizer` quant method update
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR updates the name of the quantization method that we use in `from_pretrained`, to help us guide when developing new features around quantization and HF transformers

cc @amyeroberts @LysandreJik 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30484). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
OWL-ViT precompute box_bias
Author: nvbinh15
Pull request: yes

Fixes #26099.

Precompute box_bias at initialization using the number of ViT patches and move it to the correct device at inference.

Appreciate your review @amyeroberts @5hadytru!




Comments:
Also update the OWLv2 model with a similar speed-up.

Hi @nvbinh15, thanks for opening this PR! [As per the contributing guidelines](https://github.com/huggingface/transformers/blob/838b87abe231fd70be5132088d0dee72a7bb8d62/CONTRIBUTING.md#L4), we give priority to PRs opened, vs comments on issues. As #29712 was opened first, that will be reviewed before this PR. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FEAT: PEFT support for EETQ
Author: younesbelkada
Pull request: yes

# What does this PR do?

To merge once https://github.com/huggingface/peft/pull/1675 is merged - tests are directly added on PEFT ! 

cc @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30449). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llama: SDPA FA2 path + static cache fix
Author: gante
Pull request: yes

# What does this PR do?

## Problem
The recently enabled SDPA FA2 path doesn't pass the `attn_mask` (causal mask) argument. As such, when the static cache is used, `S` (see the [SDPA docs](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) for terminology) is the full cache length as opposed to the sequence length. Therefore, the inferred mask in SDPA is incorrect, resulting in bad numerical values. 

PR that introduced the issue: https://github.com/huggingface/transformers/pull/30317. The issue was not caught in our llama testing suite because we didn't have a test for the static cache WITHOUT compilation:
- dynamic cache -> no issue with the length
- static cache + compile -> we manually build the causal attention mask, passing it to sdpa
- static cache -> we don't build the causal mask = `causal mask is None` = problem described above triggered

## Solution
There were two possible paths here:
1. Build and pass a full attention mask, corresponding to the full cache length, to avoid `causal mask is None` (#30414 );
2. Crop empty KV entries (corresponding to the empty cache) before SDPA.

I went with `2`, as it saves us tons of masked computations :) I've also ensured the static cache without compilation is numerically tested in the llama test file.

Fixes #30417 

________________________________

Slow tests ran locally: `llama`, `gemma`, `cohere`, `test_cache_utils.py`.

Comments:
cc @younesbelkada @fxmarty (SDPA + FA2 changes)
cc @zucchini-nlp (alternative fix to https://github.com/huggingface/transformers/pull/30414, see possible solutions in the PR header)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30437). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

As Arthur wrote -- the current state of the PR adds a slowdown on the eager path. 

I'm exploring an alternative path: first standardize our static cache to behave like our other caches (living outside the model as a stand-alone object), then forcing the generation of the full mask if a static cache is used (which fixes this issue).

(closed in favor of #30476, a much better long-term solution)
Use the Keras set_random_seed in tests
Author: Rocketknight1
Pull request: yes

Some tests load checkpoints where weights are missing (e.g. the `pooler` weights for `ViT`). To get reproducible results, we need to ensure that seeds are set consistently. This PR uses `keras.set_random_seed()` instead of `tf.set_seed()`, which fixes some issues the CI was seeing where models mismatched because of differing weight inits.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30504). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.