noamaanMulla-03/transformers-issues huggingface dataset:
Add LayerScale to NAT/DiNAT
Author: alihassanijr
Pull request: yes



# What does this PR do?

This follows PR #20219 .

I completely dropped the ball on LayerScale in the original PR. 
This is just an optional argument in both models, and is only activated for larger variants in order to provide training stability.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger @NielsRogge .

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add perf_infer_gpu_one.mdx italian translation
Author: davidegazze
Pull request: no
Labels: name: WIP

See issue [https://github.com/huggingface/transformers/issues/17459]
Add Italian translation of perf_infer_gpu_one.mdx and update _toctree.yml.
It's my first pull request, so I hope it's ok

Comments:
Thanks again for the contribution and congrats on your first PR @davidegazze  🔥 ! Feel free to close the issue if all of the relevant pieces of work have been merged in.
Correct NATTEN function signatures and force new version
Author: alihassanijr
Pull request: yes

# What does this PR do?

This complements #22229. (Sorry for breaking it into two; I was traveling when I realized the issue so I only started a quick PR to fix circleci builds.)

We're releasing a new [NATTEN](https://github.com/SHI-Labs/NATTEN/pull/24) build that corrects the signature inconsistency between the function calls (see [my comment in the previous PR for more](https://github.com/huggingface/transformers/pull/22229#issuecomment-1474015157).)

Rather than wait for a future build, we decided to do it right now because we could end up forgetting to open a PR to transformers.

We're finishing up testing the new build, but I figured I'd open this PR before I push to PyPI. If circleci tries to get the latest version from PyPI it would fail the unit tests associated with models depending on this package.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ydshieh @amyeroberts @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Unsure why a Flax test is failing. I'm assuming I'll have to wait for another PR to merge and then rebase?

Hi, Thank you for the fix and PR. Regarding failing flax test, you can ignore it🤗

This is needed to fix the CI on the main branch (broken by the new release of `natten`) so merging.
Fix balanced and auto device_map
Author: sgugger
Pull request: yes

# What does this PR do?

In #22095 some of the arguments passed to `infer_auto_device_map` were grouped in kwargs. The problem is that one of those (`max_memory`) is not updated anymore after being changed (when device_map is `"auto"`, `"balanced"` or `"balanced_low_0"`). This PR fixes that.

Note: this is a regression so this will need to go in a patch.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Final update of doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix the remaining doc examples. The doc example in `feature_extraction_markuplm` is un-fixable (unless we remove the example), so not added to the list for testing.

Follow up PR of #22268 and  #22292

Comments:
There is a style issue for `src/transformers/models/bertweet/tokenization_bertweet.py`. Will check it tomorrow instead. Sorry.

_The documentation is not available anymore as the PR was closed or merged._
[deepspeed] offload + non-cpuadam optimizer exception doc
Author: stas00
Pull request: yes

part 2 of https://github.com/huggingface/transformers/pull/22043, but we can't merge it until `deepspeed==0.8.3` is released.

This PR documents the new feature and up's the min deepspeed version.

**XXX: DO NOT MERGE UNTIL `deepspeed==0.8.3` is released.**

I'm keeping it as a DRAFT so that I don't mistakenly merge it to soon. But we can pre-approve.

cc: @jeffra 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Restore fp16 support on xla gpu device
Author: ymwangg
Pull request: yes

https://github.com/huggingface/transformers/pull/20684 accidentally disabled fp16 support on xla gpu device, which leads to significant performance regression. This PR restores this feature.

cc @jeffhataws @sgugger @Lokiiiiii

Tested with 
```
GPU_NUM_DEVICES=1 python run_mlm.py \
    --model_name_or_path bert-base-uncased \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --overwrite_output_dir true \
    --output_dir /tmp/test-mlm \
    --per_gpu_train_batch_size 24 \
    --do_eval \
    --fp16 true \
    --do_train \
    --num_train_epochs 3 \
    --optim adamw_torch_xla
```

```
***** train metrics *****
  epoch                    =        3.0
  train_loss               =     1.7725
  train_runtime            = 0:04:58.00
  train_samples            =       4627
  train_samples_per_second =      46.58
  train_steps_per_second   =      1.943
INFO:__main__:*** Evaluate ***
[INFO|trainer.py:739] 2023-03-21 19:05:53,483 >> The following columns in the evaluation set don't have a corresponding argument in `BertForMaskedLM.forward` and have been ignored: special_tokens_mask. If special_tokens_mask are not expected by `BertForMaskedLM.forward`,  you can safely ignore this message.
[INFO|trainer.py:3072] 2023-03-21 19:05:53,487 >> ***** Running Evaluation *****
[INFO|trainer.py:3074] 2023-03-21 19:05:53,487 >>   Num examples = 479
[INFO|trainer.py:3077] 2023-03-21 19:05:53,487 >>   Batch size = 8
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:07<00:00,  8.38it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_loss               =     1.5811
  eval_runtime            = 0:00:29.83
  eval_samples            =        479
  eval_samples_per_second =     16.055
  eval_steps_per_second   =      2.011
  perplexity              =     4.8601
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failure is unrelated and due to the branch being old, it's fixed on main, so merging.
Add MaskedImageModelingOutput 
Author: alaradirik
Pull request: yes

# What does this PR do?
- Adds `MaskedImageModelingOutput` and `TFMaskedImageModelingOutput` classes for masked image modeling / completion / in-painting models.
- Replaces the inaccurate MaskedLMOutput used for ViT and DeiT MIM heads with the new output class
- Ensures backward compatibility by adding `logits` as a property to the new output class

## Before submitting
- [X ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Pinging @NielsRogge for the final approval

@NielsRogge could you take another look? I think all comments are addressed
Enable traced model for text-generation task
Author: jiqing-feng
Pull request: yes

@gante Hi, Gante.
Refer to: https://github.com/huggingface/transformers/pull/22072
Thanks for your advice. This PR only changed the example, would you please help me to review it? Thanks!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger @gante

@gante Thanks for your attention. Would you please help me to merge it? Thanks! I think the demand for `jit trace` will grow, and I hope we can keep on working on it so it will be adapted to all models and all tasks in the future.
run_summarization requires a dataset_name or train_file or validation_file in all cases
Author: coreyfournier
Pull request: no

### System Info

In the latest version of run_summarization on line 264 it requires dataset_name or train_file or validation_file. I was trying to perform a "do_predict" with the parameter "test_file" set, but the validation will not let me proceed. Looking at the code, do_predict only uses the test dataset anyway, so this appears to be a bug.

See the specific file
https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization.py

### Who can help?

@sgugger Unable to "do_predict" due to validation not checking for "train_file". As a workaround I set "validation_file".

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

in a Colab notebook:
%%shell
/content/transformers/examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path /content/gs/models/Clinical-T5-Large/ \
    --do_predict \
    --test_file "/content/gs/models/Clinical-T5-Large/validation.csv" \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /content/gs/models/Clinical-T5-Large \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate \
    --max_source_length=1024

You get the error "Need either a dataset name or a training/validation file."

### Expected behavior

Expected normal prediction output like below:

Running tokenizer on prediction dataset:   0% 0/5040 [00:00<?, ? examples/s]03/20/2023 15:40:20 - INFO - datasets.arrow_dataset - Caching processed dataset at /root/.cache/huggingface/datasets/csv/default-73bac40c02a24f34/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317/cache-1e88c7213311bd88.arrow
Downloading builder script: 100% 6.27k/6.27k [00:00<00:00, 3.59MB/s]
03/20/2023 15:40:52 - INFO - __main__ - *** Predict ***
[INFO|trainer.py:3066] 2023-03-20 15:40:52,611 >> ***** Running Prediction *****
[INFO|trainer.py:3068] 2023-03-20 15:40:52,611 >>   Num examples = 5040
[INFO|trainer.py:3071] 2023-03-20 15:40:52,611 >>   Batch size = 4
[WARNING|logging.py:280] 2023-03-20 15:40:52,624 >> You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
[INFO|configuration_utils.py:575] 2023-03-20 15:40:52,636 >> Generate config GenerationConfig {
  "_from_model_config": true,
  "decoder_start_token_id": 0,
  "eos_token_id": 1,
  "pad_token_id": 0,
  "transformers_version": "4.28.0.dev0"
}

Comments:
Indeed. Would you like to open a PR to fix this?
Enforce `max_memory` for device_map strategies
Author: sgugger
Pull request: yes

# What does this PR do?

In #22271, by removing the `max_memory` from the kwargs before it gets passed to `get_balanced_memory`, I effectively made the `max_memory` argument ignored when `device_map` is `"auto"`, `"balanced"` or `"balanced_low_0"` (as was caught in the multi-GPU tests). This PR fixes that.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[WHISPER] - ValueError: Malformed soundfile
Author: altryne
Pull request: no

### System Info

I had a few files with this error. 
In @openai/whisper the same files worked, so I figured this was a bug.

Digging a little bit, this has to do with some ffmpeg parameters inside : 
`File "/usr/local/lib/python3.9/site-packages/transformers/pipelines/audio_utils.py", line 41, in ffmpeg_read`


Instead of 
```
file_name = "audio.mp3"
output = pipe(file_name,
                  generate_kwargs=props,
                  return_timestamps=True,
                  chunk_length_s=30,
                  stride_length_s=[6, 0],
                  batch_size=32,
                  ignore_warning=True)
```

The workaround I used, is to load the audio with whisper.load_audio instead, and pass that into the pipeline (until this if fixed) 

```
audio = whisper.load_audio(source_file)
output = pipe(audio,
                  generate_kwargs=props,
                  return_timestamps=True,
                  chunk_length_s=30,
                  stride_length_s=[6, 0],
                  batch_size=32,
                  ignore_warning=True)
```

This does require a whisper dependency, but doesn't load it's model, just uses a more up to date load_audio than the one in transformers. 


### Who can help?

@ArthurZucker @sanchit-gandhi 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The only file I can reproduce this on is a video of my kid, so I can share via DM, don't want to paste the link here.

### Expected behavior

File is valid, and whisper runs properly

Comments:
Hey @altryne! Indeed, it should be possible to load up any audio file using transformers `pipeline` alone. Could you share the full traceback of the error? This will help in pinpointing its exact nature! It would be great if you are able to share the `.mp3` file! My email is `sanchit@huggingface.co`. Thanks!

Thanks Sanchit! 

Shared! (it's a video file, but whisper doesn't mind) 

any news on this case ?

> (it's a video file, but whisper doesn't mind)

It was an mp4 video file that needed to be converted to an mp3 or wav audio file first. HF's audio pipeline works with any audio file format, but not video.

Thanks for the clarification

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add MegatronT5
Author: eagle705
Pull request: no
Labels: name: New model

### Model description

In NeMo Megatron, the T5 model is available, but there is currently no MegatronT5 class for huggingface, such as MegatronBERT or MegatronGPT2. I have recently finished the porting work and have tested the model internally. I would like to share this model with the community.



### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

- [NeMo Megatron models](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/intro.html)
- [NeMo](https://github.com/NVIDIA/NeMo)
- [Megatron-LM T5 model](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/model/t5_model.py)

Comments:
docs: Resolve incorrect type typo in trainer methods
Author: tomaarsen
Pull request: yes

# What does this PR do?
Replace incorrect `Lst[str]` with `List[str]` in docstrings in various locations in the `Trainer`.

## Before submitting
- [x] This PR fixes a typo or improves the docs

## Who can review?

Documentation: @sgugger

* Tom Aarsen

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Hardware Auto-Setup for Tutorials and Examples
Author: dongreenberg
Pull request: yes

# What does this PR do?

Discussed with @sgugger and @LysandreJik . This PR introduces auto-setup functionality for tutorials and examples (we are sending a parallel PR to accelerate, and maybe Diffusers and Spaces shortly). This allows users to run transformers code, tutorials, and scripts on self-hosted hardware (either their own instance or a cloud instances) including on-demand allocation of the hardware itself on AWS, GCP, Azure, or Lambda Labs, and installation of dependencies. This introduces a similar level of turnkey usage and reproducibility that users only typically expect in Colab, but for any type of hardware on any cloud (we've tested on Paperspace and Coreweave as well, allocating the instance in their UI and then plugging in the IP as a static cluster). Note that Runhouse OSS is facilitating the setup (via SkyPilot) and rpc, but users don't don't need to create a Runhouse account or anything like that, this is strictly inside their own cloud accounts with their own credentials (or using their IP and ssh creds without a cloud account).

This PR is WIP and seeking feedback. A few open questions:
1. launch_auto_hardware.mdx is structured to be like a notebook, but how do I make it have the "launch in colab" etc. buttons on the top right? (I'm happy to refactor it into an ipynb if needed)
2. launch_auto_hardware.mdx shows only inference. I think it could be valuable to make it mirror the initial PyTorch parts of [the yelp fine-tuning tutorial](https://huggingface.co/docs/transformers/training), to show preprocessing, training, and inference all on remote hardware. Does that make sense?
3. We're generally using the term "remote hardware" with "auto setup". Would it be better to say "self-hosted" or something like that (to avoid confusion with hosted solutions)?
4. Should we showcase more examples? Right now we show just a few but can add more if you think that's valuable (e.g. @sgugger mentioned showing TPUs before). We can also tailor the auto-setup scripts to different examples if desired, rather than the one-size fits all approach.

Thank you for your input on this so far!!


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).

Comments:
Whoops, shouldn't be on main, I'll close and reopen on a new branch.

_The documentation is not available anymore as the PR was closed or merged._
Fix PipelineTests skip conditions
Author: ydshieh
Pull request: yes

# What does this PR do?

While I am starting to try to fix some pipeline tests that are skipped previously, I realized PR #21516 (removing the usage of meta class in pipeline testing) **accidentally skip more tests than it should**.

- previously, each combination of model/tokenizer/processor class has its own test case (being generated on the fly), so we can use `self.skipTest` to skip some cases.
- After #21516，each model + task has its test cases, but inside that test, it runs against all tokenizers/processors that are available. In particular, slow / fast tokenizer. 
  - As mentioned before, we have some slow tokenizer issues in pipeline testing (never tested before #20426), and we skip some failing cases for now.
  - but after #21516, when we use `self.skipTest`, we **might** **skip the next combination(s)**. This is **NOT good/expected**.

This PR instead uses `logger.warning` to log the skipped cases and continue to the next combination in a test. This is not very pleasant, but as we don't want to use meta class, this is the only way I can think of for now.

It's better to prepare a report file that clearly indicate which test cases and/or combination being [skipped].

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[deepspeed zero3] need `generate(synced_gpus=True, ...)`
Author: stas00
Pull request: yes

As discussed in https://github.com/huggingface/transformers/issues/22231 `generate` under deepspeed zero3 using different input streams on different gpus may hang. It's documented [here](https://huggingface.co/docs/transformers/main/main_classes/deepspeed#custom-deepspeed-zero-inference) and in the API docs, that `synced_gpus=True` is required but who reads the docs.

So this PR will automatically turn this flag on under ZeRO Stage-3, so everything works out of box and it'll warn the user once for their awareness.

Fixes: https://github.com/huggingface/transformers/issues/22231


Comments:
_The documentation is not available anymore as the PR was closed or merged._

So a potential issue I see is that deepspeed is potentially not enabled on all modele, & this would enable `synced_gpus` even for models where it's not enabled? depends on how `is_deepspeed_zero3_enabled` works, which you likely know better than me

- For Accelerate and HF Trainer everything is done automatically for you.
- If you build your own trainer and follow [the instructions](https://huggingface.co/docs/transformers/main/main_classes/deepspeed#nontrainer-deepspeed-integration) it'll work as well.



1. Are you proposing:
```
def generate(..., synced_gpus=None)
[...]
if synced_gpus == None:
    if is_deepspeed_zero3_enabled() and dist.world_size() > 1:
        synced_gpus = True
    else:
        synced_gpus = False
```
which would preserve BC wrt current `synced_gpus=False` in the function definition.

yes?

2. and no warning needed then? or still keeping it? 

3. now docs will be mismatching so will need to adapt those to say that by default with multi-gpu it'll be set to `True`, but the user can choose to set it to `False` if they want to.

Yes, your code is exactly what I'm suggesting. I think it would be a better API since the user wouldn't have to look for warnings (no need for a warning indeed in this case) and would preserve backward compatibility as you mention.

That sounds good. Thank you for proposing it, Sylvain.

So no warning needed, right? As this logic is really about dynamic default setting and it'll be documented as such.

Yup!

Thank you for suggesting a more elegant solution than my initial one, Sylvain.

thanks folks, this is great