noamaanMulla-03/transformers-issues huggingface dataset:
Fix CI tests hang forever (sometimes)
Author: ydshieh
Pull request: yes

# What does this PR do?

Set `--max-worker-restart=0` for `pytest` command in CircieCI workflow file.

Currently, the tests could hang forever if `--dist=loadfile` is specified, and when a worker is crashed, and we finally got `Too long with no output (exceeded 10m0s): context deadline exceeded`. For example, [this job run](https://app.circleci.com/pipelines/github/huggingface/transformers/41092/workflows/9d84d20f-be88-4a20-b17e-07d5437b59f9/jobs/467619).

## Remark:

 - I opened [an issue](https://github.com/pytest-dev/pytest-xdist/issues/784#issue-1252433606) in `pytest-xdist` repo. 
 - The reason that the worker crashed is still unclear yet. It might be related to [memory issue #17470](https://github.com/huggingface/transformers/pull/17470). But a general issue of memory leak in tests also exists.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

From the experimentation, with the argument `--max-worker-restart=0`:

- all tests passed before the worker crashed --> marked as passed
- the test where the worker crashes --> marked as failed
- all tests not yet run before the worker crashed -> won't be marked at all, neither as passed nor as failed (i.e. won't be sent to other workers)

> From the experimentation, with the argument `--max-worker-restart=0`:
> 
> * all tests passed before the worker crashed --> marked as passed
> * the test where the worker crashes --> marked as failed
> * all tests not yet run before the worker crashed -> won't be marked at all, neither as passed nor as failed (i.e. won't be sent to other workers)

I understand we might want to the tests to be run on other workers, but currently without  `--max-worker-restart=0`, the CI hangs forever (if a worker crashes), which is not good neither.

~~(BTW, it looks like this issue happens in docker, say `circleci/python:3.7`, but not on the GCP VM without docker)~~
fix run_clip README
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `run_clip` example's README.

The path should be the absolute path to the downloaded data directory (using relative path will make `load_dataset` look for the data directory on the Hub).

From slack discussion:
- _basically the base path used to resolve relative path corresponds to where the dataset is loaded from_
- _relative paths are transformed to the corresponding URL in the repo the script comes from_

Fix #18291

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Remove Flax OPT from `documentation_tests.txt`
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove Flax OPT from `documentation_tests.txt` for now. The test uses the docker image `huggingface/transformers-all-latest-gpu` which has no JAX/Flax installed.


[Failed job run](https://github.com/huggingface/transformers/runs/7551001967?check_suite_focus=true)
Error:

```python
src/transformers/models/opt/modeling_flax_opt.py:20: in <module>
    import flax.linen as nn
E   ModuleNotFoundError: No module named 'flax'
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker This file is used to test the docstrings in model files and some documentation files. It is something different from the usual model testing.
Fix TFSegformerForSemanticSegmentation doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `TFSegformerForSemanticSegmentation` doctest.
With @amyeroberts uploading `nvidia/mit-b0`, all doctests for TF Segformer pass now.

(I was waiting for a TF checkpoint and not fixed this one in the previous PR. This model (SemanticSegmentation) already had a TF checkpoint however)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add a check regarding the number of occurrences of ```
Author: ydshieh
Pull request: yes

# What does this PR do?

We have `TFOPTForCausalLM` doctest failed due to the wrong expected value. The file `prepare_for_doc_test.py` didn't change that model file. It comes from the existence of **``decoder_input_ids```**.

This PR adds a check, and also fixes all problematic places found with this new check.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix the hub user name in a longformer doctest checkpoint
Author: ydshieh
Pull request: yes

# What does this PR do?

`jpelhaw` not exist on hub, and test fails at this moment.

Run locally with this PR: doctest pass for this model now.

Comments:
cc @ArthurZucker 

_The documentation is not available anymore as the PR was closed or merged._

Checked with the last run (May 18th -- sorry I should check doctest status much earlier), it worked with the previous checkpoint string, which suggests that the user renamed since then.

I agree with you regarding `not the most strategic thing` (I raised the doubt before, but we decided to continue with this approach to see how things go)

@LysandreJik just reminded me that the migration to the new cache system on huggingface_hub will magically support repo renames so this won't be a problem in the future!
Change audio kwarg to images in TROCR processor
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix a bug in TROCR processor introduced in #18325

Currently, we have [failed job run](https://github.com/huggingface/transformers/runs/7603716998?check_suite_focus=true) with error

```bash
        if audio is None and text is None:
>           raise ValueError("You need to specify either an `audio` or `text` input to process.")
E           ValueError: You need to specify either an `audio` or `text` input to process.
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[VideoMAE] Improve code examples
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR simplifies the code examples of VideoMAE, and adds a seed to make sure the video classifier always predicts "eating spaghetti" on the video (as, due to the sampling of frames, it may occur the model predicts another class, like "eating ice cream"):

```
1019 
1020         >>> inputs = feature_extractor(list(video), return_tensors="pt")
1021 
1022         >>> with torch.no_grad():
1023         ...     outputs = model(**inputs)
1024         ...     logits = outputs.logits
1025 
1026         >>> # model predicts one of the 400 Kinetics-400 classes
1027         >>> predicted_label = logits.argmax(-1).item()
1028         >>> print(model.config.id2label[predicted_label])
Expected:
    eating spaghetti
Got:
    eating ice cream
```

Weirdly, this wasn't caught by the doc test CI. It could have to do with the addition of i`mport numpy as np` to the code snippet.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update TF fine-tuning docs
Author: Rocketknight1
Pull request: yes

This PR updates the fine-tuning sidebar tutorial with modern TF methods that were added in the most recent release.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@stevhliu is there a better way to format a link to the `prepare_tf_dataset` docs [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18654/en/training#loading-data-as-a-tfdatadataset) than the way I did it here? It prints the whole `TFPreTrainedModel.prepare_tf_dataset` text, which looks a bit ugly on the page!

Yes, I believe you can just squiggly it!

[`~TFPreTrainedModel.prepare_tf_dataset`]

@stevhliu Thank you for the suggestions! I'll make more edits to incorporate the other bits and ping you again for a final look.

@sgugger @stevhliu I finished incorporating your edits and did some other cleanup. I also replaced `to_tf_dataset` in the other fine-tuning pages. I didn't touch the translations, though - should I edit those too?

@sgugger I tried those edits but it looked a little odd because there was no separate header for the PyTorch section. I added a `Train` header to the whole thing with a brief intro, and then a `Train with PyTorch Trainer` header inside that block, which I think works a little better and makes it easier for people to find what they want in the sidebar. Let me know what you think!

@sgugger :facepalm: I knew I'd regret pinging you before waiting for the job to finish and checking it myself. Fixed!
TF: final bias as a layer in seq2seq models (replicate TFMarian fix)
Author: gante
Pull request: yes

# What does this PR do?

This PR replicates the exact same change as in https://github.com/huggingface/transformers/pull/18833 (applied to TFMarian) to the other seq2seq TF models. **_The change is exactly the same for all models._**

In essence, weights that are not in layers are not stored/loaded with `.save_weights()` and `.load_weights()`, the functions we use to store to/load from the hub. These changes move `final_logits_bias` to a layer. Many models do NOT use this bias, but some do.

⚠️ Prior to this change, existing TF models from `Helsinki-NLP` (`TFMarian`) were wrong (and new conversions failed the automatic checks). I will revisit the canonical models using these architectures to ensure they are okay, and open PRs with weights if not.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
facebook/wav2vec2-xls-r-300m-21-to-en  TypeError: expected str, bytes or os.PathLike object, not NoneType
Author: Shiro-LK
Pull request: no
Labels: name: bug

### System Info

transformers == 4.21.3
python == 3.9.2
ubuntu 18

### Who can help?

@patrickvonplaten @sanchit-gandhi 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. from transformers import  Speech2Text2Processor
2. processor =Speech2Text2Processor.from_pretrained("facebook/wav2vec2-xls-r-300m-21-to-en")

### Expected behavior

the processor should be loaded but got this error instead :

`TypeError: expected str, bytes or os.PathLike object, not NoneType`

Comments:
Hey @Shiro-LK!

Good catch, we're loading a Wav2Vec2 processor here so need to instantiate the corresponding class accordingly:

```python
from transformers import Wav2Vec2Processor

processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-xls-r-300m-21-to-en")
```

I've opened a PR to update the example on the model card with these steps: https://huggingface.co/facebook/wav2vec2-xls-r-300m-21-to-en/discussions/3
remvoe `_create_and_check_torch_fx_tracing` in specific test files
Author: ydshieh
Pull request: yes

# What does this PR do?

Remvoe `_create_and_check_torch_fx_tracing` in specific model test files, as the common one can handle them correctly.
The only exception is `Hubert` model, but we can also remove it, and set `fx_compatible` to `False` (just as for `Wav2Vec2`).

It might be better to add `torch_script_compatible` to handle `Hubert` and related models.

**Motivation**: Make the change in #18547 available to all tests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The common case test for TorchScript, and if I recall correctly there was an issue for those models on that aspect?

You suggest to add a flag called `torch_script_compatible`? If so, [that is what I suggested back then](https://github.com/huggingface/transformers/pull/17206#discussion_r875989932), pinging @sgugger here.

Also, I think that some of those models can actually be torchscripted with torch 1.12, but the issue was that we are (were?) testing in torch 1.11.

> The common case test for TorchScript, and if I recall correctly there was an issue for those models on that aspect?

There might be before. But as far as I can tell, the issue probably came from the input and label names preparation. As the tests pass after I remove their re-definitions from the specific model test files, I think it's fine and better to clean them up. (The only failure is from Hubert).

> You suggest to add a flag called torch_script_compatible? 

This is to allow torch trace test still run while skip the torch script test, as currently Hubert test will fail on torchscript.
But I would prefer to add this flag (if the idea is approved) in a separate PR (and where we can enable the test for Wav2Vec2 too, for example)

> Also, I think that some of those models can actually be torchscripted with torch 1.12, but the issue was that we are (were?) testing in torch 1.11.

We can re-evaluate this, but again, let's not to do changes regarding this part in this PR.

This PR is merely to avoid overwriting `_create_and_check_torch_fx_tracing` unnecessary :-)

@michaelbenayoun If this PR is OK on your side,  I am going to merge. Regarding the flag, let's see what we can do in a separate PR.
NLLB-200 is too slow
Author: Yakovtam
Pull request: no

### Feature request

When I use modle 'facebook/nllb-200-distilled-600M' to translate , it consuming 0.5 second. And it look like not async。
I want make it consuming 0.1 second ，and make it async。 Is there anyone who can help me 。Thanks!

### Motivation

When I use modle 'facebook/nllb-200-distilled-600M' to translate , it consuming 0.5 second. And it look like not async。
I want make it consuming 0.1 second ，and make it async。 Is there anyone who can help me 。Thanks!

### Your contribution

When I use modle 'facebook/nllb-200-distilled-600M' to translate , it consuming 0.5 second. And it look like not async。
I want make it consuming 0.1 second ，and make it async。 Is there anyone who can help me 。Thanks!

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Getting Torchvision Transforms of `feature_extractor`s
Author: sachinruk
Pull request: no
Labels: name: Feature extractors

### Feature request

Currently if I was to add any transforms in my training pipeline, it's not quite obvious how to do so. My usual process is to read through the source code and hope to find what I'm after.

What I'm after is something like in `timm` where you can do
```
config = resolve_data_config({}, model=model)
transform = create_transform(**config)
```
In the above I can append any torchvision transforms at will by inspecting transforms.

While I'm here it seems to be that the mean and standard deviations are 0.5 each for `VitFeatureExtractor` and same with Beit. Was this intentional as it might be incorrect if trained using imagenet data.

### Motivation

See above.

### Your contribution

Happy to contribute, but not sure where and how to start on unifying `FeatureExtractor` classes to return `torchvision.transforms`.

Comments:
cc @amyeroberts 

Thanks for adding this request @sachinruk :) 

Regarding the the mean and standard deviation values, can you raise a separate issue? 

We're currently going through an update of the feature extractor class for images. At the moment, it's not possible to compose the individual transformations we apply, like `create_transform` does. It's something we were thinking about doing down the road and it's great to hear there's support for it! 

To control what is and isn't applied by a `FeatureExtractor` you can toggle flags like e.g. `do_normalize` on the call. Note: there are some known bugs with this logic we're looking to address soon (see: [#15055](https://github.com/huggingface/transformers/issues/15055))

If you want to add transformations that aren't already applied by the `FeatureExtractor`, or work completely within `torchvision.transforms` there's a great example of a custom pipeline in our [example notebooks here](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb). 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WIP] Accelerate gpt2
Author: thomasw21
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18364). All of your documentation changes will be reflected on that endpoint.

Hope you will also check how will it convert to onnx and will be optimized by onnxruntime after these changes in Attention. 

N. B.: Attention optimizations (FusedAttention) in onnxruntime while converting to fp16 greatly help with inference time.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Disable model checkpoint sharding of large models for SageMaker Model Parallel
Author: viclzhu
Pull request: yes

Disable model checkpoint sharding of large models for SageMaker Model Parallel

 * SageMaker Model Parallel does not support loading these

# What does this PR do?
This PR disables the automatic model checkpoint sharding done in `PreTrainedModel` for SageMaker Model Parallel as SMP does not support loading these.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @philschmid 

Ok, sounds good! I opened a new PR with changes in Trainer instead. #18928
Disable model checkpoint sharding of large models for SageMaker Model Parallel
Author: viclzhu
Pull request: yes

Disable model checkpoint sharding of large models for SageMaker Model Parallel

* Using large max shard size since can't disable completely from the `save_pretrained()` call

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Oh ok, I see. That makes sense, thanks!
Fix XLA fp16 and bf16 error checking
Author: ymwangg
Pull request: yes

This PR fixed a bug introduced in https://github.com/huggingface/transformers/pull/15022 that will wrongfully throw an error when training with XLA device + fp16. `GPU_NUM_DEVICES` is unset by torch_xla in distributed training [here](https://github.com/pytorch/xla/blob/master/torch_xla/distributed/xla_multiprocessing.py#L229).

Tested using the following scripts:
```sh
GPU_NUM_DEVICES=8 python -m torch_xla.distributed.xla_spawn --num_gpus 8 language-modeling/run_mlm.py \
    --model_name_or_path bert-base-uncased \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --overwrite_output_dir true \
    --output_dir /tmp/test-mlm \
    --per_device_train_batch_size 10 \
    --do_eval \
    --fp16 true \
    --do_train \
    --num_train_epochs 3 \
    --optim adamw_torch_xla
```

Thanks to @Lokiiiiii @comaniac for reporting this issue.

cc @sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._

XLA already identifies the device type and publishes it in the environment variable for distributed training:
```
XRT_MULTI_PROCESSING_DEVICE="device:ordinal"
```

Eg: XRT_MULTI_PROCESSING_DEVICE=GPU:0
Eg: XRT_MULTI_PROCESSING_DEVICE=TPU:0

Refer to relevant device specific setup in PT-XLA: https://github.com/pytorch/xla/blob/master/torch_xla/distributed/xla_multiprocessing.py#L219-L276

Looking into single worker training now.

There might be a much easier solution:
The presence of environment variables of [TPU_NUM_DEVICES](https://github.com/pytorch/xla/blob/6e42e7cb3af01d9f8909e112a3be0148a87acad0/torch_xla/distributed/xla_multiprocessing.py#L79-L88) or [XRT_TPU_CONFIG](https://github.com/pytorch/xla/blob/e7e7fe406c7f276469d3e47ecd23e8c9423ab1b5/TROUBLESHOOTING.md) indicates a TPU environment.
The presence of environment variable GPU_NUM_DEVICES indicates a GPU environment.


The most systematic logic should be like:

```
and not (self.device.type == "xla" and is_torch_tpu_available() and xm.xla_device() == gpu)
```

Inspired by this logic, it might be better to have an API to return the current torch_xla device so that we could use it here:

```
and not(self.device_type "xla" and get_torch_xla_device() != gpu)
```

I'm fine with both solutions :-)

I just realized torch_xla already has the API to distinguish different backends.
```
torch_xla._XLAC._xla_real_devices([str(device)])
```
For GPU, it returns
```
['GPU:0']
```
For TPU, it returns
```
['TPU:0']
```
I'll try to implement it with this API.

It looks like "torch.device" as a type hint can cause CI failure if pytorch is not installed. I've removed it.
Add image height and width to ONNX dynamic axes
Author: lewtun
Pull request: yes

# What does this PR do?

This PR enables dynamic axes for image height / width of ONNX vision models. This allows users to change the height and width of their inputs at runtime with values different from those used to trace the model during the export (usually 224 x 224 pixels)

Here's an example with ResNet and `optimum`:

```python
import requests
from PIL import Image
from optimum.onnxruntime import ORTModelForImageClassification
from transformers import AutoFeatureExtractor

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
# Raw image size 480 x 640 pixels
image = Image.open(requests.get(url, stream=True).raw)
# Resize image to 40 x 40 pixels
preprocessor = AutoFeatureExtractor.from_pretrained("microsoft/resnet-50",  do_resize=True, size=40)
model = ORTModelForImageClassification.from_pretrained("onnx")
inputs = preprocessor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
logits.shape
```

I've also checked the slow tests pass:

```
RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -k "beit or clip or convnext or data2vec-vision or deit or detr or layoutlmv3 or levit or mobilevit or resnet or vit" -s
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._