noamaanMulla-03/transformers-issues huggingface dataset:
Update the pipeline tutorial to include `gradio.Interface.from_pipeline`
Author: abidlabs
Pull request: yes

# What does this PR do?

Adds `gradio.Interface.from_pipeline` to the pipeline docs, as [discussed internally](https://huggingface.slack.com/archives/C01N44FJDHT/p1702977938149529?thread_ts=1702943692.389539&cid=C01N44FJDHT).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29684). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @stevhliu for the fast review! Accepted your suggestions and made a couple of small tweaks

Thanks @amyeroberts and @stevhliu ❤️
storing & logging gradient norm in trainer
Author: shijie-wu
Pull request: no
Labels: name: Feature request

### Feature request

store and log gradient norm in trainer

### Motivation

Gradient norm is an important metric but currently the gradient norm is discarded during clipping. 

### Your contribution

I checked and all of the following grad_norm_clip functions return the gradient norm. We can get gradient norm without extra compute by storing the return value, and we can then log it. If this sounds good i am happy to prepare a PR.

https://github.com/huggingface/transformers/blob/32ec7345f2d752c294ddf5aff495b657c9cd9d3b/src/transformers/trainer.py#L1918-L1936

Comments:
I want disable it, how?
🚨 Fully revert atomic checkpointing 🚨
Author: muellerzr
Pull request: yes
Labels: name: bug

# What does this PR do?

As discussed offline, while on paper having atomic checkpointing was a good idea, it's unmaintainable by the core maintainers and has resulted in an influx of issues from users considering the wide variety of storage options available to them. (and untestable)

As the feature request, while reasonable and from a good mindset, was not due to an outright bug, we are reverting it entirely. 

Related PRs:

https://github.com/huggingface/transformers/pull/27820, https://github.com/huggingface/transformers/pull/28364

Did revert based on this commit: https://github.com/huggingface/transformers/commit/fe8d1302c7d5852ef96a9fb0fdd2a81eb3e1bdc6

Related issues: 
* https://github.com/huggingface/transformers/issues/27925
* https://github.com/huggingface/transformers/issues/28027
* https://github.com/huggingface/transformers/issues/29382


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @LysandreJik 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29370). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

wonderful PR! hope 4.38.3 version release soon

Will this change be included in 4.38.3? Thanks

@Chandler-Bing @shiningliang 4.38.2 -> 4.38.3 would represent a patch release, these are reserved for fixing regressions or unintended breaking changes from a minor release i.e. 4.37 -> 4.38. 

We have minor releases roughly once a month, so the next release, 4.39.0 will be in a week or two. 

 Nice PR! Upgraded transformers to the latest version (i.e., main), now the checkpoints can be saved smoothly in mult-node setting.
fix model ids for Qwen2
Author: JustinLin610
Pull request: yes

This PR fixes the names of doc and test files.

Comments:
@ydshieh @fxmarty can you take a look?

Hi @JustinLin610 Thank you for the fix.

`Qwen2-450m-beta` has 8 `occurrence` on the `main` branch (in the test files), but I only see 2 places are replaced.

Also, it would be better to pull huggingface/transformers main and create a new branch for the PR (rather than just based on your original model addition PR). It looks a bit strange this PR contains 78 commits, despite the file changes are 3 files.

I see what you mean. I'll just close this PR and start another one with a new branch


> Hi @JustinLin610 Thank you for the fix.
> 
> `Qwen2-450m-beta` has 8 `occurrence` on the `main` branch (in the test files), but I only see 2 places are replaced.
> 
> Also, it would be better to pull huggingface/transformers main and create a new branch for the PR (rather than just based on your original model addition PR). It looks a bit strange this PR contains 78 commits, despite the file changes are 3 files.

https://github.com/huggingface/transformers/pull/29723 I just committed another PR here.
Use logging.warning instead of warnings.warn in pipeline.__call__
Author: tokestermw
Pull request: yes

# What does this PR do?

Use HF logging instead of `warnings.warn` for the following warning:

```python
You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
```

When using HF `pipeline` for inference, this warning is shown a lot, and logging can get expensive. Moving to HF logging, then we can control using `transformers.logging.set_loglevel` instead of the `warnings` package which applies to all libraries using the `warnings` package.

There is a heuristic here https://github.com/huggingface/transformers/pull/26527. Not quite sure if this applies in this case.

```python
What does that mean for developers of the library? We should respect the following heuristic:
 - `warnings` should be favored for developers of the library and libraries dependent on `transformers`
 - `logging` should be used for end-users of the library using it in every-day projects
```

Maybe another solution is to warn just once.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@LysandreJik 


Comments:
Adds pretrained IDs directly in the tests
Author: LysandreJik
Pull request: yes

Stop relying on pretrained maps to get the rust tokenizer. First PR to enable #29112

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29534). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: GaLore support
Author: kristaller486
Pull request: no
Labels: name: Feature request

### Feature request

Support for the GaLore training strategy, which allows full-weights training of 7B models on consumer 24Gb cards like RTX 4090.

Arxiv: https://arxiv.org/abs/2403.03507
Implementation: https://github.com/jiaweizzhao/GaLore

### Motivation

This is another step towards democratizing LLM. And it's awesome!

### Your contribution

I am teapot

Comments:
cc @pacman100 @younesbelkada 

Stay tuned ! #29588 should be landing soon
[`GemmaConverter`] use user_defined_symbols
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #29440 by always adding the `user_defined_symbols` to the tokenizer, otherwise because no merges exist for them, they will be split.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29473). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I will add a test and we can merge

Thank you for warning me. But could you share with me the reason to merge before we resolve the failing tests?

We need to release ! 

It would be very appreciated that the expected failing tests to be skipped temporarily (before they would be fixed/updated after the release).

We can open a PR to skip / fix them tomorrow! 

Failing tests are unrelated, merging!
[`Gemma`]  final fixes to the modeling
Author: ArthurZucker
Pull request: yes

# What does this PR do?

Superseeds #29402

What this gives:
<img width="696" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/17d7fd77-0d43-4cbc-b1be-2c9b21c8e251">


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29729). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ran the slow tests and they looked good!
[i18n-<zh>] Translating trainer.md to chinese
Author: chenin-wang
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.md) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md) (waiting for initial PR to go through)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->


Comments:
Hi, thanks for your interest in translating the docs! We already have an open issue tracker for Chinese in #26803 so let's keep track of everything there instead of creating a new issue 🙂
Add Quiet-STaR support
Author: fakerybakery
Pull request: no
Labels: name: New model

### Feature request

Hi,
Please add [Quiet-STaR](https://github.com/ezelikman/quiet-star) support. The Transformers code has already been written.

### Motivation

Quiet-STaR allows LLMs to reason before thinking. [Paper](https://arxiv.org/abs/2403.09629).

Comments:
Hi @fakerybakery, thanks for opening this model request! 

If there's already transformers compatible code written, then the fastest and recommended way to make the model available is by putting the code on the hub and we have as much support as we can for enabling this. Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models)

Ah, I see, so use remote code? I’ll see if I can open a PR on the model. Thanks!
Clean-up generation tests after moving methods to private
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Folllowing #29437 , this PR removes tests that call "to be private" generation methods.  Instead we check each generation method by passing its arguments to the `generate` method. 

All tests in generation, including slow are passing.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29582). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp a more descriptive PR name would be nice :D Musicgen seems to need some retouches, looking at the failing CI.

@amyeroberts see the PR header in https://github.com/huggingface/transformers/pull/29437 for more context on why we are deprecating calls to the internal decoding methods

Fixed MusicGen. All passed except for two tests for multigpu, but those are failing in `main` also. Not related to current changes

@zucchini-nlp suggestion: revert docsting changes, as they will cause CI to fail ( https://github.com/huggingface/transformers/pull/29718 fixes the CI error) :)
Llama: partial 4d masks
Author: gante
Pull request: yes

# What does this PR do?

Reintroduces support for partial 4D masks in Llama (and other models with support for the static cache).

Fixes #29525 

Thank you @poedator for a clean description and a test case -- I was unaware our previous versions supported this use of 4D attention masks 🤗 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29731). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante,  thank you for the PR!
I tested it with more examples and it works OK. 

I have couple of suggestions:

- add a few lines to documentation on how to submit custom attention masks properly.

- add ability to pass a custom attention mask unchanged if its shape matches `self.causal_mask.shape`. Ideally do this before any line where causal_mask is re-created to save time and RAM. My usecase is to have a static mask for drafting model in spec decoding and only updating its elements. I'd like this to pass torch.compile.

- if you are open to extreme time saving (~ 200 µs for 4k attention_mask), the It may even be assumed that the custom mask has the right `dtype`, and has `min_dtype` in place of zeros and 0 in place of ones to avoid conversions like `(attention_mask.eq(0.0)).to(dtype=dtype) * min_dtype`


Hi @poedator 👋 

We have other bug fixes and more impactful features in our pipeline, so I'll not work on your suggestions (at least not for now) :) However, we're always open to PRs!
Fix `check_copies` not capturing the diff in model/paper title and link
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently, if there is/are some difference(s)  in `README.md` and one of its localized version, more precisely, model or paper's title and/or link

- if the diff is in `model (link)`:
  - `fix_and_overwrite=False`: it (the localized version) is updated automatically and no erro raised (not good!)
  - `fix_and_overwrite=True`: works as expected
- if the diff is in `paper (title/link)`
  - `fix_and_overwrite=False`: not detected and no error raised (not good!)
  - `fix_and_overwrite=True`: not detected and no update (not good!)

This PR fixes this situation.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29724). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Tests: Musicgen tests + `make fix-copies`
Author: gante
Pull request: yes

# What does this PR do?

https://github.com/huggingface/transformers/pull/29582 was stale with `main`, the repository check failed. This PR applies `make fix-copies` on `main`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29734). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(hold up, need more fixes 😭 should have triggered the tests against `main` before merging #29582 )

@amyeroberts @ydshieh ready for review 🙏
Larger runner on CircleCI
Author: ydshieh
Pull request: yes

# What does this PR do?

To avoid OOM on some job (torch/tf/flax jobs). With `-n 12`, we have ~83% RAM usage (of 32GB)

<img width="1001" alt="Screenshot 2024-03-20 094333" src="https://github.com/huggingface/transformers/assets/2521628/c26e82f8-da51-4456-ad80-6ffcb66b8efe">



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29750). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add option to not re-load the model when resuming from checkpoint.
Author: stevemadere
Pull request: no
Labels: name: trainer

name: Feature request

### Feature request


I'd like to have the option to get the trainer to resume_from_checkpoint *without* reloading the model.

I propose adding a boolean option parameter:  `without_checkpoint_model: bool = False`.

Starting from a checkpoint with this parameter set to True would require that the model was supplied to the Trainer constructor (pretty standard practice anyway) and would simply skip this line of code in the Trainer:
`self._load_from_checkpoint(resume_from_checkpoint)`



@muellerzr , @pacman100, @amyeroberts 


### Motivation

When Trainer.train(resume_from_checkpoint=my_checkpoint) is called, the Trainer will attempt to re-load the model in the my_checkpoint directory during the resumption process.

This is kind of wasteful but no big tragedy if the model in question in fully supported by Trainer.
Right now, Trainer needs to know how to reload a model before it can be used with resume_from_checkpoint.

This was (and may be still) a big problem for QLoRA model users who had to spend quite some time waiting for Trainer to be modified to specifically support re-loading QLoRA models.  One could train these models from scratch just fine by loading them first and supplying them to the Trainer constructor, it was just impossible to resume_from_checkpoint with them.

It looks like there is supposed support for resuming with QLoRA added very recently but then some folks complaining that it does not actually work in practice:

- https://github.com/huggingface/transformers/issues/29607
- https://github.com/huggingface/transformers/issues/29383

But even if QLoRA reloading does work today or will soon, there is a wider problem.

In general, whenever there is a new model type that the Trainer does not explicitly know how to load, it will be impossible to resume_from_checkpoint for that kind of model until support is added to Trainer for loading that type of model.
Lots of new kinds of quantization and LoRA initialization methods are being invented all the time (e.g LoftQ).  Do we really want to prevent their adoption because it takes time to adapt Trainer to intrinsically know how to load every kind of model under the sun?

We might do well to just provide folks a way to work around that without having to fork transformers and then use their patched fork (like I've been doing).

Really, since the Trainer can be instantiated with the fully restored checkpoint, there is no reason why the train() call should force re-loading the model from the checkpoint.


### Your contribution

I've already got a POC implementation in my fork which I've pretty thoroughly integration tested, having resumed  QLoRA training from checkpoint with it dozens of times.

https://github.com/stevemadere/transformers/commit/5c2a1c8200392abfb268fc7407121de605fc7a33

PR incoming.

Comments:
PR #29741