noamaanMulla-03/transformers-issues huggingface dataset:
Translation Italian: perf_train_cpu and perf_train_cpu_many
Author: nickprock
Pull request: yes

## What does this PR do?

Italian translation of doc related to the preprocessing of :hugs: Transformers.
* updated _toctree.yml
* added perf_train_cpu.mdx
* added perf_train_cpu_many.mdx

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).

See issue: [[#17459](https://www.linkedin.com/feed/hashtag/?keywords=%2317459)](https://github.com/huggingface/transformers/issues/17459)


@sgugger, @stevhliu and @MKhalusova @omarespejel 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[i18n-it] Translating docs to it
Author: davidegazze
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @ArthurZucker, @sgugger for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.mdx) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.mdx) (waiting for initial PR to go through)
- [ ] [installation.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.mdx).

## Tutorial section
- [ ] [pipeline_tutorial.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.mdx)
- [ ]  [autoclass_tutorial.mdx](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.mdx)
- [ ]  [preprocessing.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.mdx)
- [ ]  [training.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.mdx)
- [ ]  [accelerate.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.mdx)
- [ ]  [model_sharing.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.mdx)
- [ ]  [multilingual.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.mdx)

<!--
Keep on adding more as you go 🔥
-->

## How-to Guide

- [] [big_models](https://github.com/huggingface/transformers/blob/main/docs/source/en/big_models.mdx)































## How-to guides
- [] [big_models](https://github.com/huggingface/transformers/blob/main/docs/source/en/big_models.mdx)


Comments:
Add BART-LS
Author: KMFODA
Pull request: no
Labels: name: New model

### Model description

BART-LS (Long Bart), presented in this [paper](https://arxiv.org/pdf/2209.10052.pdf), establishes a new SOTA on a number of  NLP tasks and long form datasets. It uses pooling-augmented block-wise attention and a novel pre-training strategy to achieve this.

Given my interest in long text summarisation I'm very keen to get this into the wonderful transformers library and to start benchmarking it against other models. Therefore, happy to take this on and ping any members of the team if I face any blockers. If this fits with the library's plans let me know and I'll start working on a PR for this.

### Open source status

- [X] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Original Model Repo (which includes the model weights): https://github.com/facebookresearch/bart_ls

Comments:
Any update on this ? @KMFODA  I can help please let me know if you want to collaborate on this ?

Hey @thakursc1 I'm still waiting on someone from the HF team to confirm if this can be integrated into their codebase if we work on this as this only becomes beneficial for my use case if I can use it in the transformers master branch.

Happy to collaborate on this as soon as we hear back.

Hey @KMFODA, wondering if there are any updates on this? Thanks! 

Hey @jmzeng. I haven't heard back from anyone from the HF team yet and unfortunately a few things have changed in my workloads and I don't think I'll be able to work on this. Maybe someone else can work on this if they have the bandwidth and ping the HF team when they have a draft PR for them to review?

@KMFODA BART-LS looks like it would be a great addition to the library :) 

If you or another community member would still like to add the model, please feel free to open a PR and let us know in the meantime if there's any difficulties integrating it.
Fix big model inference for T5 models in float16
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes big model inference for large T5 models. The problem is that T5 models have some weights kept in float32, which interferes with the computation of `infer_auto_device_map`. Accelerate adds the functionality to deal with this in [this PR](https://github.com/huggingface/accelerate/pull/1179), and a patch release will be out soon with the fix in a release.

When this is done, this PR can be merged so the fix can be used. With this I can do
```py
from transformers import T5ForConditionalGeneration, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained('google/flan-ul2')
model = T5ForConditionalGeneration.from_pretrained('google/flan-ul2', device_map = 'auto', torch_dtype=torch.float16)
input_string = 'Answer the following question by reasoning step by step. I start with 10 bananas. A monkey eats three of them, and then gives me an avocado. How many bananas do I have left?'
inputs = tokenizer(input_string, return_tensors = 'pt').to('cuda:0')
outputs = model.generate(inputs['input_ids'], max_length = 200)
print(tokenizer.decode(outputs[0]))
```
whereas before this went OOM.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[i18n-it] Translating docs to it
Author: davidegazze
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @ArthurZucker, @sgugger for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.mdx) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.mdx) (waiting for initial PR to go through)
- [ ] [installation.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.mdx).

## Tutorial section
- [ ] [pipeline_tutorial.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.mdx)
- [ ]  [autoclass_tutorial.mdx](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.mdx)
- [ ]  [preprocessing.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.mdx)
- [ ]  [training.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.mdx)
- [ ]  [accelerate.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.mdx)
- [ ]  [model_sharing.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.mdx)
- [ ]  [multilingual.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.mdx)

<!--
Keep on adding more as you go 🔥
-->

## How-to guides

- [] [perf_infer_gpu_one](https://github.com/huggingface/transformers/blob/main/docs/source/en/perf_infer_gpu_one.mdx)


Comments:
Please stop opening those issues without filling the template. This is spamming every maintainer of the library.
Longformer FP16 training broken since transformers 4.21
Author: geniki
Pull request: no

### System Info

transformers 4.20 / transformers 4.21
Ubuntu 20, python 3.8

### Who can help?

@ydshieh 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Apologies, I'm using my own dataset but the problem should be easy to reproduce with any Longformer + FP16 example. Upgrading from transformers 4.20 to 4.21 causes Longformer training loss to stay stuck around its initial value. When using transformers 4.20 + FP16 and transformers >= 4.21 + FP32, training loss declines as expected.

https://github.com/huggingface/transformers/pull/17306 seems to be what caused this. You can see on that issue that it affected other models too, some of which have been fixed one by one. Longformer is still affected as of transformers 4.26.



### Expected behavior

Be able to train Longformer using fp16 precision on recent version of transformers.

Comments:
Hi @geniki Thank you for reporting the issue.

> but the problem should be easy to reproduce with any Longformer + FP16 example

It would be really nice if you can provide an example script that could reproduce the issue you reported, especially you mentioned `should be easy to reproduce` 🙏 Looking forward for it!

> some of which have been fixed one by one

Could you remind me which PRs or commits fixed this issue 🙏 That will help a lot, thank you.

Thanks for your response @ydshieh. Here are some example where this issue has been addressed for other models: 
https://github.com/huggingface/transformers/pull/20605
https://github.com/huggingface/transformers/pull/18057
https://github.com/huggingface/transformers/pull/19229
https://github.com/huggingface/transformers/pull/17437

I'll try to make an online example with Longformer work somehow. Do you have any model training tests with small dummy data?

Hi @geniki You can take any dataset on HF Hub (that are for specific task you are working on), and select a subset of it (say the first 1024 examples).

However, as you already know some fixes (in you above comment), would you like to try to experiment a fix for this model (with your own dataset, potentially a subset) and open a PR ❤️ ? If not, no worry, but in this case, as I mentioned, a script that could reproduce would be really nice 👍 



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`Gpt-neo-x`] Fix gpt neo-x multi gpu training
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR attempts to solve some issues users may encounter when training gpt-neo-x in multiple GPUs

Related: https://github.com/lvwerra/trl/pull/210

This PR might be not needed so putting it as draft for now

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes that also what I have thought after opening the PR, I'll find a workaroud to set everything on the correct device
to_pil - don't rescale if int and in range 0-255
Author: amyeroberts
Pull request: yes

# What does this PR do?

#21969 Introduced a bug where binary masks would have their values rescaled to between 0-255. 

This was because of [this part](https://github.com/huggingface/transformers/blob/4063fd9cba6b72ebfd5c663a307ab9d5ff1a153d/src/transformers/image_transforms.py#L161) of the logic check. The original assumption was that inputs with their values between 0-1 would be rescaled images with float pixels. However, binary masks aren't and shouldn't be rescaled. 

We now check first if the input is of type uint8. Then check if any precision is lost when converting to int and that the int values are in the valid range 0-255 and finally if float values are between 0-1. 


Fixes #22147


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Enforce same behavior as PyTorch 2.0 for older versions
Author: sgugger
Pull request: yes

# What does this PR do?

The default of `set_to_none` will change in PyTorch 2.0 because it is slightly better in terms of memory consumption. This PR uses it for all versions of PyTorch in the Trainer.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

hmm, that didn't work on DS side - perhaps they have an old pytorch - checking

```
2023-03-13T21:25:45.7750922Z Traceback (most recent call last):
2023-03-13T21:25:45.7751317Z   File "/tmp/actions-runner/_work/DeepSpeed/DeepSpeed/transformers/examples/pytorch/translation/run_translation.py", line 664, in <module>
2023-03-13T21:25:45.7751409Z     main()
2023-03-13T21:25:45.7751794Z   File "/tmp/actions-runner/_work/DeepSpeed/DeepSpeed/transformers/examples/pytorch/translation/run_translation.py", line 581, in main
2023-03-13T21:25:45.7751970Z     train_result = trainer.train(resume_from_checkpoint=checkpoint)
2023-03-13T21:25:45.7752299Z   File "/tmp/actions-runner/_work/DeepSpeed/DeepSpeed/transformers/src/transformers/trainer.py", line 1631, in train
2023-03-13T21:25:45.7752421Z     return inner_training_loop(
2023-03-13T21:25:45.7752790Z   File "/tmp/actions-runner/_work/DeepSpeed/DeepSpeed/transformers/src/transformers/trainer.py", line 1814, in _inner_training_loop
2023-03-13T21:25:45.7752913Z     model.zero_grad(set_to_none=True)
2023-03-13T21:25:45.7753166Z TypeError: zero_grad() got an unexpected keyword argument 'set_to_none'
```

They are using torch==1.8.2 and it fails, which means that this change will break for any user with older pytorch versions - let me find out when `set_to_none` was added.

Using this as a test:
```
python -c 'import torch; m=torch.nn.Linear(1,1); m.zero_grad(set_to_none=True)'
```
TrOCR comment change
Author: AvishekMondalQC
Pull request: yes

Thanks for the TrOCR modules @NielsRogge !

I noticed something very small in one of the comments - if you copy pasted the lines from the comment block, they wouldn't work, because the default image size in `ViTConfig` is 224, while the `processor` is expecting the image to be resized to 384x384. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi,

Thanks for your PR! We actually test that code snippet as TrOCR is [included in the doc tests](https://github.com/huggingface/transformers/blob/b29e2dcaff114762e65eaea739ba1076fc5d1c84/utils/documentation_tests.txt#L190). So the code runs fine. Of course, if you would use that model in combination with TrOCRProcessor, you would have to instantiate the image processor's size to be {"height": 224, "width": 224}, like so:

```
from transformers import RobertaTokenizer, ViTImageProcessor, TrOCRProcessor, ViTConfig, TrOCRConfig, VisionEncoderDecoderModel

tokenizer =  RobertaTokenizer.from_pretrained("roberta-base")
image_processor = ViTImageProcessor(size={"height": 224, "width": 224})
processor = TrOCRProcessor(tokenizer=tokenizer, image_processor=image_processor)

config_encoder = ViTConfig()
config_decoder = TrOCRConfig()

config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
model = VisionEncoderDecoderModel(config=config)
```

Thanks for your quick reply! 

I believe that particular bit of code in the docstring runs fine because `model` gets overwritten a couple of lines below it. Do you think it is worth it to put the lines of code you've written in your comment in the docstring as well in case someone wants to try to train a model without the weights from microsoft? 

If not, it's not a big deal, I'll just close this PR.
Add OPT-IML Checkpoints
Author: chujiezheng
Pull request: no
Labels: name: New model

### Model description

OPT-IML models are instruction-finetuned from the OPT checkpoints. Here is the [technical report](https://github.com/facebookresearch/metaseq/blob/main/projects/OPT-IML/optimal_paper_v1.pdf).

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

* **Technical report:** https://github.com/facebookresearch/metaseq/blob/main/projects/OPT-IML/optimal_paper_v1.pdf
* **Model implementation:** same as OPT
* **Model weights:** https://github.com/facebookresearch/metaseq/tree/main/projects/OPT-IML
* **Authors:** Meta AI

Comments:
🙏 

I tried to convert and also ran into this issue:

https://github.com/facebookresearch/metaseq/issues/567
https://github.com/facebookresearch/metaseq/issues/594

But it seems like meta folks are working to upload it to huggingface:

https://github.com/facebookresearch/metaseq/issues/567#issuecomment-1370415582

maybe I will tag @patrickvonplaten in case who I believe converted the last OPT checkpoints :-)

I believe it was @ArthurZucker 

Thanks for bumping on this! 

Also working on this, have run into the same issues mentioned above. Let me know if I can be an extra pair of eyes/hands for working on this.

Weights are available here thanks to (https://huggingface.co/rpasunuru) : 
- https://huggingface.co/facebook/opt-iml-30b
- https://huggingface.co/facebook/opt-iml-1.3b
Closing!
shift_tokens_right function in XforConditionalGeneration classes
Author: sajastu
Pull request: no

Hi there,

I am using BART for summarization and I noticed an issue with the `DataCollator` class when using it with multiple batches. The class uses the `prepare_decoder_input_ids_from_labels` function to create the `decoder_input_ids` for the labels (target sequence), which shifts the tokens one to the right and appends the `decoder_start_token_id` at the start of the shifted tensor. This works well when the batch size is 1, but it seems to be buggy when using multiple batches.

For example, I observed that the second example in my batch (which is padded with -100 in the labels tensor) has two `decoder_input_ids`, one to the left and one to the right. I would like to understand if this implementation is correct, and if so, what is the reasoning behind shifting the longer example while appending the `decoder_start_token_id` to the shorter example while keeping it at the end of the target sequence?

Here's an example with `batch_size` of 2:

(Examples are truncated for better visualization and with the `decoder_input_ids`s as `**2**`)

```
pdb>> decoder_input_ids

tensor([[    **2**,     0,   133,  6741, 31275,  2633,    41, 13433,     9,     5,
          3425,  6411,     9,    10,  6063,   797,  2187,     4, 50267,    ...
             0,   133,  6063,    40,    45],

        [    **2**,     0,   133, 27913, 39322, 31275,  3373,  1437,   103, 12720,
             8,  8047,  ... ,     7,    33,    10,  2200, 12628,
         50267,     0,  2264,  8089,     5,  6063,   197,   283,    11,     4,
         50267,     **2**,     1,     1,     1]], device='cuda:0')
```

P.S.: 

Given the reasoning behind shifting operation in decoding time, the second example makes more sense to me; as the first "2" denotes the start of the sequence and makes the decoder to predict the first word of the target sequence, and the last "2" is again predicted in the second example to denote the end of the sequence. However, I do not see any second "2" in the first example. Will the decoder be able to take care of both examples? 

Comments:
cc @ArthurZucker and @younesbelkada 

Hey for BART, the `eos_token_id=2`, which is why you have one at the end of a sequence, and a the beginning (for training purposes). Does that make sense?