noamaanMulla-03/transformers-issues huggingface dataset:
Flax LM scripts doesn't scale-up on TPU Pod.
Author: agemagician
Pull request: no

## Environment info
- `transformers` version: 4.18.0.dev0
- Platform: Linux-5.8.0-1035-gcp-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.1 (cpu)
- Jax version: 0.3.4
- JaxLib version: 0.3.2
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Models:

- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- JAX/Flax: @patil-suraj


## Information

Model I am using (Bert, XLNet ...): T5 - Flax

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details below)

The T5 MLM as well as any Masking models for flax is working fine in a single TPU node. However, when running the script on a TPU pod, it doesn't scale up correctly.
Assuming the batch size on a single tpu v4-8 node per device is 32 and the global batch size is 128 when we scale it up to a tpu v4-64, then the per device batch size of 32 doesn't work and gives OOM, and it has to be reduced to 4, which means the global batch size will be kept 128.
Furthermore, the training speed remains the same between a single node and a pod.


## To reproduce

Steps to reproduce the behavior:

1. Run any example on https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling using a single tpu node.
2. Re-run it again using a tpu pod.


## Expected behavior

We are expecting the script scale-up correctly on a tpu pod. 


Comments:
So, I think we have figured out the problem with flax team :
https://github.com/google/flax/discussions/2017

The main problem that you send to the shard function the global batch for all hosts and nodes.
In the examples, it should send only local batch size per host.

I believe something like that should work:

replace:
`model_inputs = shard(model_inputs.data)`

with:
```
loca_host_model_inputs = {}
for key, value in model_inputs.data.items():
    loca_host_model_inputs[key] = np.split(model_inputs.data[key], jax.process_count(), axis=0)[jax.process_index()]

model_inputs = shard(loca_host_model_inputs)
```


A more simplified  version:

loca_host_model_inputs = {key:np.split(model_inputs.data[key], jax.process_count(), axis=0)[jax.process_index()]  for key, value in model_inputs.data.items()}

If you think it makes sense, I will create a pull request to fix this issue.

I think @patil-suraj @borisdayma are working quite a bit with TPU Pods sharding at the moment

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Bug in T5 Tokenizer - Adding space after special tokens
Author: yuvalkirstain
Pull request: no

## Environment info

- `transformers` version: 4.16.1
- Platform: Linux-4.15.0-65-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.2+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help
@SaulLu @patrickvonplaten @PhilipMay 


## Information

When decoding with the T5 tokenizer it inserts redundant spaces after special tokens; this leads to incorrect decoding. Here is an example:

```bash
>> tokenizer.convert_ids_to_tokens(self.encode("``America''"))
['▁', '<unk>', 'America', "'", "'", '
']

>>self.convert_ids_to_tokens(self.encode(self.decode(self.encode("``America''"))))
['<unk>', '▁America', "'", "'", '
']
```

This happens in line 291 of file `tokenization_t5.py`: 
```python
out_string += self.sp_model.decode_pieces(current_sub_tokens) + token + " "
```

What is the purpose of the last space (" ")?

## Expected behavior

IMO the decoding in this example should be exactly the same as the original text. 

Thank you! :) 


Comments:
That's a good question! @SaulLu @LysandreJik @n1t0 - do you have an idea by any chance?

This seems to be about special tokens:

```python
            # make sure that special tokens are not decoded using sentencepiece model
            if token in self.all_special_tokens:
                out_string += self.sp_model.decode_pieces(current_sub_tokens) + token + " "
                current_sub_tokens = []
```

https://github.com/huggingface/transformers/blob/2ca6268394c1464eaa813ccb398c3812904e2283/src/transformers/models/t5/tokenization_t5.py#L289-L292

@PhilipMay will removing the additional `" "` lead to any negative effect?

If not, can we please remove the additional `" "`? 
Sometimes we need to decode encoded inputs, and we will prefer the outcome to be as similar as possible to the original text.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
PyTorch FSDP integration in Trainer
Author: pacman100
Pull request: yes
Labels: name: PyTorch FSDP

# What does this PR do?

PyTorch recently upstreamed the Fairscale FSDP into PyTorch Distributed with additional optimizations. This PR is aimed at integrating it into Trainer API. 
- It enables Distributed Training at Scale. It's a wrapper for sharding Module parameters across data parallel workers. This is inspired by Xu et al. as well as the ZeRO Stage 3 from DeepSpeed.
- PyTorch FSDP will focus more on production readiness and long-term support. This includes better integration with ecosystems and improvements on performance, usability, reliability, debuggability and composability.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Opened a new PR on the branch of main repo instead of fork [17136](https://github.com/huggingface/transformers/pull/17136)
split single_gpu and multi_gpu
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix the scheduled CI issue caused by the 256 limits (jobs generated from matrix).

Note that the workflow run page has a graph that has no single-gpu and multi-gpu on it. But on the left side, the job names have matrix mentioned.

<img width="271" alt="Screenshot 2022-05-04 145601" src="https://user-images.githubusercontent.com/2521628/166685617-6c008556-dd44-4369-9b6a-b7b1c22cbc9d.png">



Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for your PR! Did you launch it as a trial to see if it works? I see the following should be completed, on line 303:
> 
> [setup, run_tests_gpu, run_examples_gpu, run_pipelines_tf_gpu, run_pipelines_torch_gpu, run_all_tests_torch_cuda_extensions_gpu]

You are right, that line should be changed. I haven't launched it (just tried with a dummy example). I will launch it now.

You can launch it with only 1-2 models in each run, for example by updating this line:
```
          echo "::set-output name=matrix::$(python3 -c 'import os; tests = os.getcwd(); model_tests = os.listdir(os.path.join(tests, "models")); d1 = sorted(list(filter(os.path.isdir, os.listdir(tests)))); d2 = sorted(list(filter(os.path.isdir, [f"models/{x}" for x in model_tests]))); d1.remove("models"); d = d2 + d1; print(d)')"
```
to
```
          echo "::set-output name=matrix::$(python3 -c 'import os; tests = os.getcwd(); model_tests = os.listdir(os.path.join(tests, "models"))[:2]; d1 = sorted(list(filter(os.path.isdir, os.listdir(tests)))); d2 = sorted(list(filter(os.path.isdir, [f"models/{x}" for x in model_tests]))); d1.remove("models"); d = d2 + d1; print(d)')"
```

This way you'll test the full behavior without having 12-hour long iterations.

It took sometime, but the run looks good.

https://github.com/huggingface/transformers/actions/runs/2276209307

Looks good, thanks @ydshieh!
Add type hints for BigBirdPegasus and Data2VecText PyTorch models
Author: robsmith155
Pull request: yes

Added missing type hints for the `BigBirdPegasus` and `Data2VecText` PyTorch models as requested in https://github.com/huggingface/transformers/issues/16059.


@Rocketknight1

Comments:
_The documentation is not available anymore as the PR was closed or merged._
PyTorch FSDP integration in Trainer
Author: pacman100
Pull request: yes
Labels: name: PyTorch FSDP

# What does this PR do?

PyTorch recently upstreamed the Fairscale FSDP into PyTorch Distributed with additional optimizations. This PR is aimed at integrating it into Trainer API. 
- It enables Distributed Training at Scale. It's a wrapper for sharding Module parameters across data parallel workers. This is inspired by Xu et al. as well as the ZeRO Stage 3 from DeepSpeed.
- PyTorch FSDP will focus more on production readiness and long-term support. This includes better integration with ecosystems and improvements on performance, usability, reliability, debuggability and composability.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
'BaseModelOutput' object has no attribute '_OrderedDict__map' when using Wav2Vec 2.0
Author: joeyism
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.27
- Python version: 3.8.8
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.1+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@patrickvonplaten, @anton-l

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here is the code
```python
import soundfile as sf
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

# load pretrained model
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")


librispeech_samples_ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")

# load audio
audio_input, sample_rate = sf.read(librispeech_samples_ds[0]["file"])

# pad input values and return pt tensor
input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values

# INFERENCE

# retrieve logits & take argmax
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)

# transcribe
transcription = processor.decode(predicted_ids[0])
```
which is taken from [the official documentation](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec#use-wav2vec-20-with-transformers)

### Expected behavior

```shell
Not throw an error. However, I get the error

AttributeError: 'BaseModelOutput' object has no attribute '_OrderedDict__map'
```


Comments:
I was able to reproduce this pythonically on python 3.8.8
```python
from collections import OrderedDict
from dataclasses import dataclass

class A(OrderedDict):
    def __post_init__(self):
        self["a"] = 1

@dataclass
class B(A):
    some_val = None

b = B()
```
(A would be `ModelOutput` in this case, and B would be `BaseModelOutput`)
which throws the same error
```
AttributeError: 'B' object has no attribute '_OrderedDict__map'
```

I updated the python version to `3.8.13` and it worked
```
- `transformers` version: 4.8.2
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.27
- Python version: 3.8.13
- PyTorch version (GPU?): 1.10.1+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```
Fix MLflowCallback end_run() and add support for tags and nested runs
Author: orieg
Pull request: yes

# What does this PR do?

This PR includes:
- Add support for MLFLOW_TAGS, MLFLOW_RUN_ID, and MLFLOW_NESTED_RUN
- ensure that MLflow runs that were started with on_train_begin() are also properly terminated when on_train_end() hook is called.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix install directions for all docs using no_trainer scripts
Author: muellerzr
Pull request: yes
Labels: name: Documentation

# What does this PR do?

This PR fixes the directions in the documentation for installing Accelerate, as pointed out here: https://github.com/huggingface/accelerate/issues/349

Currently they state to install just with pip, but as they are changed based on the git version of Accelerate, users should install with it instead.

The general Accelerate docs can remain the same however. 

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
add `mobilebert` onnx configs
Author: manandey
Pull request: yes

# What does this PR do?

This PR adds MobileBert OnnxConfig to make this model available for conversion. #16308 

## Who can review?

@lewtun @LysandreJik 

Anyone in the community is free to review the PR once the tests have passed.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @manandey thanks for the PR, it looks really clean. Did you try to convert one MobileBERT model with this config?

It could be nice to upload a converted MobileBERT model of your choice to the `ONNXConfig for all` if you have time.

Hi @lewtun, I tried to address the fixes you had suggested, and the tests are passing after running `RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -s -k "mobilebert" `.  :)
Optional keys in TrainingArguments aren't always labelled as such
Author: d-miketa
Pull request: no

Inside `TrainingArguments` there are a few keys such as [tf32](https://github.com/huggingface/transformers/blob/098b0026447271a340d2d7e6bff428c82cb6d744/src/transformers/training_args.py#L594) which are optional and have a default of `None`, but aren't explicitly labelled as such. This can cause problems downstream; for example, OmegaConf will complain that 
```
omegaconf.errors.ValidationError: Non optional field cannot be assigned None
    full_key: tf32
    object_type=Seq2SeqTrainingArguments
```

The fix is simple, just make sure every key with a default of `None` has `Optional` type. I'm busy with another PR atm, but can have a look later.

Comments:
cc @sgugger
Train args defaulting None marked as Optional
Author: d-miketa
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/16701 by marking every `TrainingArguments` argument `Optional` if it currently defaults to `None`. Other applications which depend on static type hinting will no longer be confused by arguments defaulting to `None` although their declared type is `bool`.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Increase the usage of augmented assignment statements
Author: elfring
Pull request: no

:eyes: Some source code analysis tools can help to find opportunities for improving software components.
:thought_balloon: I propose to [increase the usage of augmented assignment statements](https://docs.python.org/3/reference/simple_stmts.html#augmented-assignment-statements "Augmented assignment statements") accordingly.

Would you like to integrate anything from a transformation result which can be generated by a command like the following?
(:point_right: Please check also for questionable change suggestions because of an evolving search pattern.)
```
[Markus_Elfring@fedora lokal]$ perl -p -i.orig -0777 -e 's/^(?<indentation>\s+)(?<target>\S+)\s*=\s*\k<target>\s*(?<operator>[+\-%&|^@]|\*\*?|\/\/?|<<|>>)/$+{indentation}$+{target} $+{operator}=/gm' $(find ~/Projekte/Transformers/lokal -name '*.py')
```


:crystal_ball: How will the development interests evolve further also according to update candidates in 1134 lines of this software?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

:thought_balloon: Do any factors hinder the wider application of [functionality which became generally available with Python 2](https://docs.python.org/3/whatsnew/2.0.html#augmented-assignment "Augmented assignments")?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I would find it nice if development interests can grow for the mentioned source code transformation approach.
Inconsistent output of code example in Proprocessing Chapter in Docu
Author: HallerPatrick
Pull request: no

The following [docu](https://huggingface.co/docs/transformers/preprocessing) contains some inconsistent/wrong output examples for the `Build Tensor` subsection:

```python
batch_sentences = [
    "But what about second breakfast?",
    "Don't think he knows about second breakfast, Pip.",
    "What about elevensies?",
]
encoded_input = tokenizer(batch, padding=True, truncation=True, return_tensors="pt")
print(encoded_input)
```
Output:
```txt
{'input_ids': tensor([[  101,   153,  7719, 21490,  1122,  1114,  9582,  1623,   102],
                      [  101,  5226,  1122,  9649,  1199,  2610,  1236,   102,     0]]), 
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0]]), 
 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1],
                           [1, 1, 1, 1, 1, 1, 1, 1, 0]])}
```

The output does not contain the applied truncation and padding and also misses the last sentence of `batch_sentences`. This also applies to the TensorFlow example. 


Comments:
[Deepspeed] add many more models to the model zoo test 
Author: stas00
Pull request: yes
Labels: name: DeepSpeed

name: WIP

This PR continues figuring out how to make various models work with Deepspeed (a lot of fixes happen on the Deepspeed side), most models just work out of the box - the main purpose of this PR is to test as many models as possible. so there are no fixes to add.

- [x] update coverage to albert, bart, bert, bigbird_pegasus, big_bird, blenderbot, deberta, deberta_v2, distilbert, electra, flaubert, fsmt, funnel, gpt2, gptj, gpt_neo, layoutlm, led, longformer, marian, mbart, mobilebert, mpnet, pegasus, prophetnet, roberta, squeezebert, t5, t5_v1, vit, xlm_roberta, xlnet

Thanks to @LysandreJik for creating the tiny test models for many of HF models!

Some models I couldn't cover for a variety of reasons unrelated to Deepspeed (missing tokenizers, missing tiny models, missing example scripts to exercise these). But their status is documented in the script. Over time more will be tested.

Blocking events - all resolved:

- [x] https://github.com/microsoft/DeepSpeed/pull/1227 (fixes reference counting)
- [x] https://github.com/microsoft/DeepSpeed/pull/1380 (fixes zero_to_fp32 recovery of uneven param shapes)
- [x] https://github.com/huggingface/transformers/pull/13665 (fixes positional embeddings: m2m_100 and others)
- [x] https://github.com/microsoft/DeepSpeed/pull/1916#event-6563217392 (fixes tracing)
- [x] 0.6.4 Deepspeed release that includes all the merged PRs



Comments:
_The documentation is not available anymore as the PR was closed or merged._

nice work @stas00, have you tested Perceiver with DeepSpeed.

Would be glad to do that, @sameeravithana- in order to do that I need is a Trainer-based example script that I can test with.

As you can see from this map:

https://github.com/huggingface/transformers/blob/4a419d4995111c22d6842ee1bcd2d3f500150845/tests/deepspeed/test_model_zoo.py#L231-L270

I have each model tested by one of HF Trainer examples. Is there one that can be used with perceiver?