noamaanMulla-03/transformers-issues huggingface dataset:
Add logit scaling sdpa using `FlexAttention` for Gemma2
Author: ArthurZucker
Pull request: no
Labels: name: Feature request

name: Good Difficult Issue

### Feature request

With the recent publish of `FlexAttention`, think it makes senses to support it with sdpa path of `Gemma2`.
Leaving it open to the community:

- [ ] implement the path
- [ ] make sure integration tests work 
- [ ] add some doc


Related to #32865

Comments:
For a code pointer: https://github.com/pytorch-labs/attention-gym/blob/main/attn_gym/mods/softcapping.py
Gemma capping
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Adds capping for gemma2, fixes #32877

Comments:
Okay @Cyrilvallez good point regarding cleaning! Makes more sense indeed, will update to fix 😉 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34282). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix cache_utils for optimum.quanto kvcache quantization 
Author: SunMarc
Pull request: yes

# What does this PR do?
Supersedes https://github.com/huggingface/transformers/pull/34606 and updated a bit the docs

Quanto kv cache tests are passing 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34750). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Modular fix
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. This is basically the changes already present in https://github.com/huggingface/transformers/pull/34282 + a fix for protected imports duplication.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34802). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Move Trainer's tokens per second metric into the inner training loop
Author: crclark
Pull request: no
Labels: name: Feature request

### Feature request

Right now `include_tokens_per_second=True` in `Trainer` only reports the tokens per second metric at [the end of training](https://github.com/huggingface/transformers/blob/c1753436dbb8bcbcee183cdd6eba9f08a90d602a/src/transformers/trainer.py#L2610). It would be very useful to have this metric reported continuously inside the training loop so we can monitor it during training. 

### Motivation

The current behavior is counter-intuitive, doesn't align with other convenient trainers (like torchtune), and it's undocumented, so I had to RTFC to figure out why the metric wasn't showing up.

### Your contribution

I probably don't have time to contribute it myself.

Comments:
cc @muellerzr @SunMarc
Allow to give the dataset multiprocessing_context
Author: ierezell
Pull request: no
Labels: name: Feature request

### Feature request

In Huggingface Trainer, allow to pass the multiprocessing context : https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

### Motivation

For a dataset that is loaded on multiple cpu cores, sometimes the fork method creates problems (with polars for example) and the spawn method is more adapted. 

### Your contribution

I could do a PR. A fix could be to add one more parameter to Trainer and pass it to the Dataloader down the line.

Comments:
This looks like a crossover datasets/Trainer issue, so cc @lhoestq @SunMarc @muellerzr
Run `test_medium_seamless_m4t_pt` in `subprocess` to avoid many failures
Author: ydshieh
Pull request: yes

# What does this PR do?

This (pipeline) test 

> test_medium_seamless_m4t_pt

is always failing all the time. 

Previously, the pipeline test is running on CPU, but after #34026, it's running on GPU if available.

And with the issue #34811, this failed test affect many subsequent pipeline tests because of

> RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34812). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `check_training_gradient_checkpointing`
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently inour CI runners' environment, there is no `pytest-subtests` installed. In this case, using `subTest` with `skipTest` in it will skip the whole test even if our goal is to just skip that `subTest` and continue to the next step in the for loop. In `check_training_gradient_checkpointing`, the first `model_class` is the base model and it calls `skipTest` and no other `model_class` being checked.

However, `pytest-subtests` has some issues regarding the reporting. A PR is opened https://github.com/pytest-dev/pytest-subtests/pull/169.

In the meantime, let's use `continue` for now instead of `skipTest` so we can really test gradient checkpointing.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added image-text-to-text pipeline to task guide
Author: merveenoyan
Pull request: yes

Documented the pipeline made by @yonigozlan cc @stevhliu 

Comments:
Thanks a lot for taking care of that! Looks great 🤗

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34783). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Mamba is ExecuTorch compatible
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

### Feature request

Enable Mamba to ["Export to ExecuTorch"](https://github.com/huggingface/transformers/issues/32253) workflow

### Motivation

See details in #32253

### Your contribution

Mamba model enablement

Comments:
See upstream:

- https://github.com/state-spaces/mamba/issues/626
- https://github.com/pytorch/pytorch/issues/130150
- https://pytorch.org/tutorials/intermediate/torch_export_tutorial.html#custom-ops

An interesting pointer from @vasqu https://github.com/facebookresearch/lingua/tree/main/apps/mamba/component
Keras 3 compatibility
Author: baslia
Pull request: no
Labels: name: Feature request

### Feature request

Since version 2.16, Tensorflow default keras is the version 3.
It would be great to have Transformers using keras 3 as it won't have conflict on the version used.
For now we need to set 
```
ENV TF_USE_LEGACY_KERAS=1
```
whenever we use Tensorflow and Transformers

### Motivation

It will simplify deployment, as well as prevent bugs on the long run.

### Your contribution

Not sure about the complexity of the task, but I am volunteering to help to modify the python codebase to do this upgrade.
Feel free to let me know where to start.


Comments:
Hi @baslia,

Thank you for your feature request and willingness to contribute! I’ll defer to @Rocketknight1 for a more detailed response, but here’s what I understand about the current situation:

The `transformers` library is designed to be platform-agnostic, supporting PyTorch, TensorFlow, and JAX. Keras 3, with its backend-agnostic design, introduces significant internal changes. While these changes are great for enhancing flexibility and adoption across platforms, they also introduce complexity for libraries like `transformers` that already strive for cross-platform compatibility. Supporting Keras 3 natively in this context would likely add another layer of complexity, which the Hugging Face team might prefer to avoid for now.

That said, there’s good news if you’d like to use `transformers` models with Keras 3. The Keras team provides an API to load `transformers` safetensor checkpoints into Keras 3 models. You can find a list of currently supported models and utilities in this [folder](https://github.com/keras-team/keras-hub/tree/master/keras_hub/src/utils/transformers). This approach might simplify your workflow and allow you to leverage the benefits of Keras 3 without additional configuration.

I hope this helps! Let me know if you have more questions or need further assistance.
Here is a [blog post](https://huggingface.co/blog/keras-hub-integration) that talks about the integration.

Seconding the comment from @ariG23498 - we cannot really update to Keras 3 right now without breaking a lot of backward compatibility and hugely overhauling the entire codebase. However, it is now possible to load many `transformers` checkpoints directly in Keras 3, without needing to use `transformers` at all. This is our recommended workflow for Keras 3 right now!

Hey, 
that makes sense, thank you for your detailed explanation.
Fix runtime error for devices with MPS support
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #31571 (issue)

Tested on a M1 Mac and 7900 XTX. Got the same accuracy and loss on both systems.

```
***** eval metrics *****
  eval_accuracy           =     0.4286
  eval_loss               =     1.0798
  eval_runtime            = 0:00:02.70
  eval_samples_per_second =     49.155
  eval_steps_per_second   =      6.283
```

## Who can review?

@qubvel @yeseliushi

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @jla524 , thanks for fixing it!
Can you run `make modified_only_fixup` and push changes to fix formatting issues?
Fix dependencies for image classification example
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #30060 (issue)

Tested on an M1 Mac and a 7900 XTX system

## Who can review?

@ArthurZucker @amyeroberts 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Fix OWLv2 Doc
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/pull/30686#issuecomment-2107416445

Tested on a 7900 XTX machine

```
% pytest --doctest-modules docs/source/en/model_doc/owlv2.md -sv --doctest-continue-on-failure --doctest-glob="*.md"
<log redacted>
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /home/jacky/repos/transformers/env/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/jacky/repos/transformers/.hypothesis/examples'))
rootdir: /home/jacky/repos/transformers
configfile: pyproject.toml
plugins: timeout-2.3.1, hydra-core-1.3.2, hypothesis-6.102.1, rich-0.1.1, xdist-3.6.1, dash-2.17.0, anyio-4.3.0
collected 1 item                                                                                                                           

docs/source/en/model_doc/owlv2.md::owlv2.md PASSED

<warnings redacted>
====================================================== 1 passed, 2 warnings in 4.26s =======================================================
```

## Who can review?

@ydshieh @amyeroberts 


Comments:
Enable dynamic resolution for vivit
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Addresses #30579 (issue)

## Who can review?

@amyeroberts 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The usage of the "forced_decoder_ids" parameter
Author: Alsace08
Pull request: no
Labels: name: Feature request

### Feature request

How to use the "forced_decoder_ids" parameter for decoder-only models? This parameter seems to be deprecated in the latest version.
![微信图片_20241119222811](https://github.com/user-attachments/assets/a5f2d44f-d98f-4959-85ae-e64489e3b9df)



### Motivation

This is an important function.

### Your contribution

This is an important function.



Comments:
@Alsace08 hey! If you are trying to use it with Whisper models, then it is recommended to indicate the task/language/decoder_input_ids using the keywords  task/language/prompt_ids when you are calling `generate()`. For ex:

`model.generate(input_features, task=task, language=language, prompt_ids=prompt_ids)`


> @Alsace08 hey! If you are trying to use it with Whisper models, then it is recommended to indicate the task/language/decoder_input_ids using the keywords task/language/prompt_ids when you are calling `generate()`. For ex:
> 
> `model.generate(input_features, task=task, language=language, prompt_ids=prompt_ids)`

I'm trying the Llama model, any solutions?

@Alsace08 I see. Since we had the `forced_decoder_ids` added and supported only for Whisper, it was deprecated in favor of other kwargs. 

If you want to force certain tokens in Llama you can try [constrained decoding](https://huggingface.co/blog/constrained-beam-search) or `PrefixConstrainedLogitsProcessor` depending in your use case. Otherwise you can write your own custom logits processors and pass them down to the `generate()`. Would that work for you?
Fix low memory beam search
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/34232 and bring the test back

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34746). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix the memory usage issue of logits in generate()
Author: kjohew
Pull request: yes

# What does this PR do?

As discussed in [#34796](https://github.com/huggingface/transformers/issues/34796), logits should be sliced before being copied to avoid unnecessary memory usage, especially during the first call to generate.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@zucchini-nlp


Comments:
Copy of entire logits incur large memory usage at first call of generate. (from v4.45.0)
Author: kjohew
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.2
- Platform: Linux-5.15.0-122-generic-x86_64-with-glibc2.35
- Python version: 3.10.15
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes

### Who can help?

@zucchini-nlp

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The problem is included since [this commit](https://github.com/huggingface/transformers/blob/19d58d31f19049e8280ccb62a5b098d89909bf5a/src/transformers/generation/utils.py#L3015). Before that, the logits were sliced first and then copied, which would only incur negligible memory overhead. But now, the entire logits will be copied and lead to double large memory usage during the first call of `generate()`.

```
# Previous
next_token_logits = outputs.logits[:, -1, :].clone().float()

# Current
next_token_logits = outputs.logits.clone()[:, -1, :].float()
```

### Expected behavior

hope this problem can be fixed. 🤗

Comments:
@kjohew right, seems like I made some changes and reverted them incorrectly. Would you like to open a PR to fix it?

Hi @zucchini-nlp, thanks for response. sure, I will submit a PR later today.

Lets close it as the PR was merged :)