noamaanMulla-03/transformers-issues huggingface dataset:
[docs] Refactor optimizing inference section
Author: stevhliu
Pull request: no

Hey y’all! I’m planning on simplifying the Optimizing Inference section to be more concise. The [inference on many GPUs](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_many) is essentially the same as the [inference on one GPU](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one) doc. For example:

1. The Flash Attention section links back to the single GPU doc.
2. BetterTransformer decoder model and encoder model sections are more or less identical.
3. The Advanced usage section is also the same.

With so much of the same content being reused, I think combining the GPU inference docs under a single page makes more sense instead of adding more noise to the docs. We can add a callout at the beginning of the doc saying these optimizations work on single and multi-GPU setups, and clearly document where usage between single and multi-GPUs differ (such as using `max_memory` to allocate RAM to each GPU).

Finally, I’m also considering removing the [inference on specialized hardware](https://huggingface.co/docs/transformers/main/en/perf_infer_special) doc since this has been empty for a while now with no new updates, and this also seems to be more of an Optimum topic.

Let me know what you think! cc @LysandreJik @MKhalusova @younesbelkada 

Comments:
Sounds like a great idea @stevhliu ! thanks for proposing to take the lead on this! Would be very happy to look at your PR once ready! 🤗
Enable split_batches through TrainingArguments
Author: muellerzr
Pull request: yes
Labels: name: trainer

# What does this PR do?

This PR passes `split_batches` in the `TrainingArguments` to the `Accelerator()`, the same way that `dispatch_batches` is currently done.

In the future we should consider whether we want to make a dataclass with the args for `Accelerator` that a user can pass instead. (Or a raw Accelerator potentially with guards)

Fixes # (issue)

Solves https://github.com/huggingface/accelerate/issues/2023


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @LysandreJik @pacman100 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fuyu processing update
Author: amyeroberts
Pull request: yes

# What does this PR do?

This PR builds upon #27007 - ticking off some elements in the TODO list and bringing the processor and image processor more in-line with expected patterns in the library. 



Comments:
@pcuenca Here's the draft PR for updating the image processor. In relation to your PR with the box coordinate transformations, you'll notice that I've removed the `target_height` and `target_width` attributes and have replaced them with the dictionary `size`. This is to reflect the pattern in other image processors.

cc @molbap 

_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts Nice! I'll update accordingly.

LGTM, I'll add some tests related to model in my PR! Ok to merge to https://github.com/huggingface/transformers/pull/27007 when https://github.com/amyeroberts/transformers/pull/113 is merged, and I'll add a model tester there
[Whisper, Bart, MBart] Add Flash Attention 2
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR adds Flash Attention for Whisper, Bart & MBart.

Whisper depends on Bart and MBart quite a bit for Flash Attention like 20+ other model architectures.
As this is the first PR that adds Flash Attention 2 to a encoder-decoder model, I wanted to make sure it's done for the two template models (Bart and MBart) as well so that Whisper (and all other encoder-decoder models that follow) don't loose their "# Copied from" statements.

Note that while this PR changes 27 files, only 4 files are really relevant to review because all other files are just consequences of the "# Copied from mechanism":
The following there files fully implement Flash Attention 2:
- src/transformers/models/bart/modeling_bart.py
- src/transformers/models/mbart/modeling_mbart.py
- src/transformers/models/whisper/modeling_whisper.py
The test files is restructured so that Flash Attention 2 tests can nicely run for different kinds of models (audio & nlp as well as decoder-only and encoder-decoder).
- tests/test_modeling_common.py

I ran the following tests to make sure everything works as expected:
```
CUDA_VISIBLE_DEVICES="0" RUN_SLOW=1 pytest tests/models/whisper/test_modeling_whisper.py
CUDA_VISIBLE_DEVICES="0" RUN_SLOW=1 pytest tests/models/mbart/test_modeling_mbart.py
CUDA_VISIBLE_DEVICES="0" RUN_SLOW=1 pytest tests/models/bart/test_modeling_bart.py
```
as well as:
```
RUN_SLOW=1 pytest -m flash_attn_test tests
```

All tests pass that also pass on "main". The only failures are related to disk offloading which should be fixed in: https://github.com/huggingface/transformers/pull/27204

There are some "error not raised" failures for flash attn and mistral, but they are also present in "main" and seem to be related to this PR: https://github.com/huggingface/transformers/pull/27125 (cc @younesbelkada), I'd suggest to also fix those in another PR.



Other CI test failures are unrelated.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ok ran some more tests and it should be good now. I'm getting some flaky behavior with the flash attention tests on my RTX 4090 (especially extreme for Whisper). We should maybe think about how we can make them more robust now that we've added some more models (cc @younesbelkada)
[LlamaTokenizerFast] nit update `post_processor` on the fly
Author: ArthurZucker
Pull request: yes

# What does this PR do?
This PR adresses #23833 where it appears that being able to change the `add_eos_token` and `add_bos_token` should be made possible for an easier use of the interface. 
Fix comes in two changes 
- added `_add_bos_token` attribute, as well as setters and getters for `add_bos_token`
- added a `self.update_post_processor` that update the post processor based on the current values of `add_eos_token` and `add_bos_token`. 

- added a test to make sure this is properly working

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failing test is unrelated to the changes

this is not saving `add_eos_token` and `add_bos_token`.

@sgugger @ArthurZucker Can you guys take a look at this?
Translate task summary to chinese 
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu  

here is another pr. it may result in some merge conflict problem as i have another PR is still under review. 

You can deal with it later.

Best

@stevhliu 

hi, thansk for your review. I have fix the reviews and conflict problem. 

Best 

It seems there is still some problem with yml file. I will fix it later

@stevhliu

hi, thansk for your review.  I have fixed yml problem.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27180). All of your documentation changes will be reflected on that endpoint.
translate traning.md to chinese
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27122). All of your documentation changes will be reflected on that endpoint.

@stevhliu I have fixed the _tortree.yml.

Best
Transformers: Streamers: Output for yield generator or similar, not just stdoutput
Author: gidzr
Pull request: no

### Feature request

Currently streaming goes to stdoutput, which can't be captured and pushed as streaming function to return to another function

I've been trying this type of approach to capture what's streamed to stdout and then stream the capture with a yield, but it's not working. Even though this approach works with a timer clock (1,2,3,4...) which does stream back to the parent function, the transformer textStreamer doesn't let me do this.

  > f = io.StringIO()
  > with contextlib.redirect_stdout(f):

(https://github.com/huggingface/transformers/blob/main/src/transformers/generation/streamers.py, https://huggingface.co/docs/transformers/v4.34.1/en/internal/generation_utils#transformers.TextStreamer)

Would it be possible to return the streamed output as a generated element, which could be yielded or printed to stdout?

Much neater.

### Motivation

Help with calling streaming transformers as a callback, and more easily deal with the output for other types of functions and manipulation.

### Your contribution

I think I've found it!! https://huggingface.co/docs/transformers/internal/generation_utils#transformers.TextIteratorStreamer.example

```
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
from threading import Thread

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
streamer = TextIteratorStreamer(tok)

# Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way.
generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20)
thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()
generated_text = ""
for new_text in streamer:
    generated_text += new_text
generated_text
```

Comments:
Use github.com/apssouza22/chatflow as a conversational layer. It would enable actual API requests to be carried out from natural language inputs.
Author: GiovanniSmokes
Pull request: no

### Feature request

Adding this conversational UI would enable people to 'talk' directly with the backend and API requests to be carried out more effectively. RAG can help with some of the problems function calling by language models face at the moment.

https://youtu.be/r3cegH2kviQ



### Motivation

I'm trying to accelerate the adoption of natural language interfaces.

### Your contribution

We're developing in public in our Discord and we would love for you guys to join us. 

Comments:
Hey @GiovanniSmokes, we're happy to provide support in case you would like to integrate `transformers` within `chatflow`. 

We have a lot of requests so we prefer to centralize everything in our GitHub issues, feel free to open an issue in case you run into problems! Thank you :hugs: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add pipeline support for Ascend NPU
Author: ji-huazhong
Pull request: yes

# What does this PR do?
Currently, Transformers's Trainer library has supported Ascend NPU([see](https://github.com/huggingface/transformers/pull/24879)).

This PR makes the pipeline library available on Ascend NPU. For example, we can run a text-classification pipeline on Ascend NPU with the following command(reference the example of `TextClassificationPipeline`):

```python
>>> from transformers import pipeline

>>> classifier = pipeline(model="distilbert-base-uncased-finetuned-sst-2-english", device="npu:1")

>>> classifier("Director tried too much.")
[{'label': 'NEGATIVE', 'score': 0.9963733553886414}]
```

## Who can review?
Anyone in the community is free to review the PR once the tests have passed.



Comments:
rebase commits from the main branch.

@muellerzr and @Narsil Good day. Please take a look at this PR, it mainly registers the NPU backend in the pipeline, and then we can pass `device=npu` to the `pipeline` method without other additional modifications.

Hey! Sorry I don´t think this is the way we want to proceed to support other hardwares cc @Narsil I think you were thinking of a more general way of doing this no? 

I think if users send directly `torch.device("npu")`  it should actually work.
Since torch added support for many more hardware, I think it's gonna be hard to add such support for all of them

@Narsil could we not just make use of the `PartialState().device` from Accelerate and let us handle that for you? Even if you don't use any other aspect, it can pick up the device and just be used for that sole purpose :)
Reproducible checkpoint for npu
Author: ji-huazhong
Pull request: yes


# What does this PR do?

Save npu's RNG states when saving a checkpoint and set after all the data skip phase when resuming training.
Links to https://github.com/huggingface/transformers/pull/11582


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

cc @muellerzr


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27208). All of your documentation changes will be reflected on that endpoint.

@amyeroberts Good day, this PR is ready to be merged :-)
[`core` / `Quantization`] Fix for 8bit serialization tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

Currently on main, 8 bit serialization is broken due to the switching of safetensors serialization by default. 

Link to failing job: https://github.com/huggingface/transformers/actions/runs/6727555396/job/18285709702

For 8bit models, some other objects (such as strings) are stored in the state dict. The fix is to simply deal with non-tensor case by appending the pointer address of the non-tensor object 

Added also regression tests to make sure previous behaviour is preserved.

With this fix, all failing tests with respect to 8bit serialization now pass.

cc @amyeroberts @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
wandb watch frequency
Author: yuanenming
Pull request: no

https://github.com/huggingface/transformers/blob/af3de8d87c717c4bb090f037d0d89413c195a42f/src/transformers/integrations/integration_utils.py#L751

The minimal watch frequency is 100 by default. I think people may want to monitor the gradients more frequently. So why it has to be larger than 100?

Comments:
Hi @yuanenming, thanks for raising this issue! 

The integrations are maintained by the contributors, however in this case I can see @muellerzr modified the line. I suspect it's a pragmatic number which is reasonable and suits most cases. 

The great thing about open source code is that you can modify it as you like! If you want to log things more frequently the number can be changed in a fork or your local copy. If this is a feature you think would be useful to the wider community, then please feel welcome to open a PR to make this configurable and we'll be happy to review. 

Hi, Thank you for your prompt reply.
Yes I have modified it locally. BTW, I find it is reasonable because it takes ~20 mins to log all the gradients and weights distributions for LLaMA2-70B.
Fix safetensors failing tests
Author: LysandreJik
Pull request: yes

cc @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The tests all passed, and overall looks good despite I am not familiar with this part.

I am a bit worried tests like `test_save_load_keys_to_ignore_on_save` will now only test with `safetensors` and never torch bin format anymore. If this is the case, I believe we need to test both cases, but that could be done in a follow up PR.

Thanks for the review!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27231). All of your documentation changes will be reflected on that endpoint.
Avoiding the time consuming for downloading the pre-trained models
Author: ForawardStar
Pull request: no
Labels: name: Migration

# 📚 Migration
Very great project! I can successfully run the code:

```
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
model_Bert = AutoModel.from_pretrained("bert-base-chinese")
```

But, every time I run my code, the downloading processing are repeated, which is time consuming. So I wonder how to load the pre-trained models from my local directory instead of downloading the pre-trained models everytime?

Comments:
Hi! It should work (examples for Linux systems)
```python
# Save model to directory:
model.save_pretrained("./my_model_directory/") 

# Load model from directory:
model = AutoModel.from_pretrained("./my_model_directory/")

####################### OR ###########################

# Cache model to directory:
model = AutoModel.from_pretrained("bert-base-chinese", cache_dir="./my_model_directory/")
```

Thanks! It is useful for me. One more question: I found that the files saved in "./my_model_directory/" contain config.json, pytorch_model.bin, and other .json files, but do not contain 'vocab.txt', is it reasonable?

I think `vocab.txt` is only needed for the tokenizer. The tokenizer can also be saved in the same directory.

When you do 
```
tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
model_Bert = AutoModel.from_pretrained("bert-base-chinese")
```

this should be caching the files in your local cache, it shouldn't redownload the files every time. You shouldn't need to specify a local folder in which to save them.

How are you identifying that the files are redownloaded every time?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.


> this should be caching the files in your local cache, it shouldn't redownload the files every time. You shouldn't need to specify a local folder in which to save them.

model cards are updated frequently.   so people re-download them a lot!
fix-deprecated-exllama-arg
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR fixes the logic with disable_exlllama so that it is BC compatible. It fixes the case were the user overwrite the `GPTQConfig`.

```py
checkpoint = "marcsun13/opt-350m-gptq-4bit"
quantization_config = GPTQConfig(bits=4, disable_exllama=True )
# True
print(quantization_config.use_exllama)
# False 
print(quantization_config.disable_exllama)
-> this will lead to an error as both are defined. 
model = AutoModelForCausalLM.from_pretrained(checkpoint, quantization_config=quantization_config)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Remove redundant code from T5 encoder mask creation
Author: pietrolesci
Pull request: yes

# What does this PR do?

Removes redundant code in the creation of the encoder attention mask for T5 as discussed in https://github.com/huggingface/transformers/issues/27211.

@amyeroberts

Comments:
Done :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27216). All of your documentation changes will be reflected on that endpoint.