noamaanMulla-03/transformers-issues huggingface dataset:
fix flaky m2m100 test
Author: patil-suraj
Pull request: yes

# What does this PR do?

The `test_retain_grad_hidden_states_attentions` test is sometimes failing for `M2M100`, with the error
` AttributeError: 'NoneType' object has no attribute 'retain_grad'`

 This is because of `layerdrop` sometimes a layer is skipped and the `encoder_attenion/decoder_attentions/cross_attentions` can be `None`.

This PR sets the `config.encoder_layerdrop` and `config.decoder_layerdrop` to 0  in tests to make the tests deterministic.

Comments:
[examples] add --max_train_samples  --max_val_samples  --max_test_samples cl args to all scripts
Author: stas00
Pull request: no
Labels: name: Good First Issue

As a part of an effort to make all examples have the same look and feel this issue requests to sync the support for these 3 cl args in `run_seq2seq.py`:
```
--max_train_samples 5 --max_val_samples 5 --max_test_samples 5
```
into:
1. all other `examples/*/run_*.py`
2. `templates/adding_a_new_example_script`

Part B. the metrics should be now updated to include the actual number of samples that were run. here is an example for train:
https://github.com/huggingface/transformers/blob/f52a15897b46ffa40af5c96d3726f0e18e91879b/examples/seq2seq/run_seq2seq.py#L586-L590
and the same for eval/test.

I'd say this can probable refactored too. Let me check with Sylvain.

The way it's currently used is to limit the number of dataset entries w/o needing to change the dataset, example:
```
run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir  --do_eval --do_predict --do_train \
--evaluation_strategy=steps  --predict_with_generate  --task summarization     --dataset_name xsum \
--max_train_samples 60 --max_val_samples 10 --n_test 10
```

All the code that currently takes care of it can be found inside https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py

This issue is open to anybody in the community who would like to tackle it.

Thank you!



Comments:
Hi @stas00,
Can I work on this?

Yes please! Thank you, @bhadreshpsavani 

I was just thinking about this and why do we not have this functionality in the Trainer in first place? then perhaps none of this will be needed.

I'm asking here: https://github.com/huggingface/transformers/issues/10437

Perhaps this task will become redundant then. Please wait a little bit.

Cool!

Hi @stas00,
`templates/adding_a_new_example_script` is still remaining right?

That's correct! Thank you for remembering it!

for max cl args and also the metrics please! Thank you!

Hi @stas00,
Since its just a template there no way to test the changes right?

It sounds right. Use your internal compiler.

Probably once it's written, it can be run through the cookie-cutter and then tested? But I think it shouldn't be too difficult to test it visually.

If you want to try the cookie-cutter, the doc is here: 
https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_example_script
[examples template] added max_sample args and metrics changes
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

This PR adds the same as https://github.com/huggingface/transformers/pull/10551 and https://github.com/huggingface/transformers/pull/10436 to the cookie-cutter template.

Fixes #10423



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## review:
@stas00

Comments:
Ya @stas00,
Actually I figured that and tested visually with the example tamplate not the model template
[tests] failing test only when run in a group
Author: stas00
Pull request: no
Labels: name: Good First Issue

name: Good Second Issue

If someone wants to solve a puzzle, this test:

```
RUN_SLOW=1 pytest examples/seq2seq/test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow
```
works on its own, but fails if it's run in the group with other tests:
```
RUN_SLOW=1 pytest examples/seq2seq/test_finetune_trainer.py
```
it doesn't learn anything - eval_blue remains 0.0

The only small issue is that the test is being renamed and moved to use `run_seq2seq.py`, so if you're reading this in a few days, most likely it will be the following case instead - which has the exact same problem:

```
RUN_SLOW=1 pytest examples/tests/trainer/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_slow
```
works on its own, but fails if it's run in the group with other tests:
```
RUN_SLOW=1 pytest examples/tests/trainer/test_trainer_ext.py
```
it doesn't learn anything - eval_blue remains 0.0

Thanks.

Comments:
Hi @stas00,

I could not understand how to use `RUN_SLOW` in the windows command line, When I run it I was getting
```
'RUN_SLOW' is not recognized as an internal or external command,
operable program or batch file.
```

It was mentioned in the contribution guidelines but I don't know how to enable it

Sorry, I don't know much about windows. 

Aren't you using some unixy shell on windows to run this? In which case it should support it?

Otherwise look up how to setup env vars in your windows shell.

And of course the simplest hack is for the duration of your test to simply comment out the `@slow` decorator inside the test file.


oh, but I actually just fixed it here https://github.com/huggingface/transformers/pull/10584 - thank you for unearthing this one. I can close it now.
Fairscale FSDP fix model save
Author: sgugger
Pull request: yes
Labels: name: fairscale

# What does this PR do?

This PR fixes the fact training with fairscale fully-sharded wrapper was hanging: it looks like recent changes in fairscale make a synchronization during the model state dict call, which results in the training hanging if we don't call that state_dict method on all processes. This PR addresses that.

Comments:
Implementing efficient self attention in T5
Author: JamesDeAntonis
Pull request: no
Labels: name: New model

# 🌟 New model addition

My teammates and I (including @ice-americano) would like to use efficient self attention methods such as Linformer, Performer and Nystromformer

## Model description

These new methods serve as approximations of regular attention, but reduce complexity from quadratic in the inputs to linear.  We would like to add a parameter to T5 where users can specify an efficient attention method to use instead of regular attention.  Ideally, this would be implemented across all models, but the models tend to have varying implementations of attention, rendering this generalization fairly tedious.

## Open source status

* [x] the model implementation is available: repos are https://github.com/mlpen and https://github.com/lucidrains/performer-pytorch
* [ ] the model weights are available: N/A
* [x] who are the authors: @mlpen and @lucidrains


Comments:
There are already some PRs regarding these models, I'm working on adding the Linformer (#10587), there's also a PR for the Performer (#9325, see further down the thread - people can already train T5 with Performer).
Model "deberta-v2--xxlarge-mnli" doesn't work!!!
Author: ngoquanghuy99
Pull request: no

Whenever i try to load the tokenizer by
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-v2-xxlarge-mnli')
```
it returns this issue:
```
config_class = CONFIG_MAPPING[config_dict["model_type"]]
KeyError: 'deberta-v2'
```

Is this model not available in Transformers 4.3.3 (the latest release)?

Comments:
No, DeBERTa-v2 is not available in v4.3.3, it's only available from source as of now. Version v4.4.0 should be released end of this week or early next week, and will have DeBERTa-v2.

Yes, exactly what i think! Thanks Lysandre for confirming this.
Average checkpoints
Author: mudong0419
Pull request: no

Is it possible to average weights of several checkpoints? Some thing like https://github.com/pytorch/fairseq/blob/master/scripts/average_checkpoints.py

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Add new GLUE example with no Trainer.
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a new GLUE example that does not use the `Trainer`, leveraging [accelerate](https://github.com/huggingface/accelerate) for the distributed training. The necessary instructions are added in the text-classification README.

cc @JetRunner as it should be of interest to you.

Comments:
This kind of logging is very useful for researchers. Let's add them back?

https://github.com/google-research/bert/blob/master/run_classifier.py#L871

In a nutshell, I'll burst into tears if we can just have Google's `run_classifier.py` back but with `accelerate` :)

Maybe we should tag other researchers (even external) to give some feedback. cc @VictorSanh @TevenLeScao 

Addressed most of your comments except the logging/saving steps. I do not have time to add this right now, so I suggest we merge the current version and someone from the community can finish it.
Copy tokenizer files in each of their repo
Author: sgugger
Pull request: yes

# What does this PR do?

This PR cleans the maps in the tokenizer files to make sure each checkpoint has the proper tokenization files. This will allow us to remove custom code that mapped some checkpoints to special files (like BART using RoBERTa vocab files) and take full advantage of the versioning systems for those checkpoints. All checkpoints changed have been properly copied in the corresponding model repos in parallel.

For instance, to accomodate the move on the fast BART tokenizers, the following commits have been on the model hub:
- in [facebook/bart-base](https://huggingface.co/facebook/bart-base/commit/c2469fb7e666a5c5629a161f17c9ef23c85217f7)
- in [facebook/bart-large](https://huggingface.co/facebook/bart-large/commit/22fa33834dccc11df99c4fc5fcc96c67f806dfdb)
- in [facebook/bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli/commit/6a35c499ad1087bad8d9c348a05b1fa10c5ad47d)
- in [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn/commit/18614750f248f300641757e8e44e6afce801d664)
- in [facebook/bart-large-xsum](https://huggingface.co/facebook/bart-large-xsum/commit/96ea79a741cd376cdc8a740b225330773da151f0)
- in [yjernite/bart_eli5](https://huggingface.co/yjernite/bart_eli5/commit/38797dd2ef06f5542c6f7db853518703f6b3da21)

In the PR I've also uniformized the way the maps are structured across models, to make it easier to alter (and ultimately remove) them in the future via automatic scripts.

Comments:
Love it!

Maybe a good practice to link to a sample of the related commits on hf.co: for instance here https://huggingface.co/facebook/bart-base/commit/c2469fb7e666a5c5629a161f17c9ef23c85217f7

I think I did around 50 of them in various repos to move all the tokenizers files, so a bit hard to keep track of all of them.

Yep just link one, or a small sample.

Makes it easier to see what this PR entails on hf-hub side
wav2vec2: `convert_tokens_to_string` contracts legitimately repeated characters
Author: elgeish
Pull request: no

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help

@patrickvonplaten - issue is most probably due to https://github.com/huggingface/transformers/blob/cb38ffcc5e0ae2fac653342ac36dc75c15ea178f/src/transformers/models/wav2vec2/tokenization_wav2vec2.py#L203

## Information

Model I am using (Bert, XLNet ...): wav2vec2

The problem arises when using:
* [x] the official example scripts: run_asr.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: wav2vec2
* [ ] my own task or dataset: (give details below)

## To reproduce
```python
from transformers import Wav2Vec2Processor

tokenizer = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base').tokenizer
tokenizer.decode(tokenizer('CARRY').input_ids)
# CARY
```

Decoder should have returned `'CARRY'` instead.

Comments:
Can you try this?

```python
from transformers import Wav2Vec2Processor

tokenizer = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base').tokenizer
tokenizer.decode(tokenizer('CARRY').input_ids, group_tokens=False)
# CARRY
```

Because we need to decode the predicted tokens with CTC, `"RR"` is decoded to `"R"` by default. See this blog post for more information: https://distill.pub/2017/ctc/

Yeah looks good! Thanks!

By the way, I totally get why it's needed for CTC, I mistook it for the tokenizer used to decode the final results but noticed it wasn't the case. The final results work as expected. Sorry for the false alarm!
Speech2TextTransformer
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR adds the S2T model from [fairseq](https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text) for end-to-end ASR and Speech-Translation (ST).

The model architecture is somewhat similar to the mBART model, except
- the encoder contains the convolutional subsampling module to downsample the speech features.
- no token embeddings in encoder.

This PR also adds the `Speech2TextFeatureExtractor`, and `Speech2TextProcessor` classes analogous to `Wav2Vec2` extractor and processor.

The `Speech2TextFeatureExtractor` here has an extra dependency on `torchaudio` which is required for extracting fbank features

The `generate` method works out-of-the-box for S2T!  Usage example 

```python
 import torch
 from transformers import Speech2TextForConditionalGeneration, Speech2TextProcessor
 from datasets import load_dataset
 import soundfile as sf

 def map_to_array(batch):
     speech, _ = sf.read(batch["file"])
     batch["speech"] = speech
     return batch

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
samples = ds.map(map_to_array)[5:8]

model = Speech2TextForConditionalGeneration.from_pretrained("valhalla/s2t_librispeech_small")
processor = Speech2TextProcessor.from_pretrained("valhalla/s2t_librispeech_small")

features = processor(samples["speech"], sampling_rate=16_000, padding=True, return_tensors="pt")
gen_tokens = model.generate(
    input_ids=features["input_features"],
    attention_mask=features["attention_mask"],
)
generated = processor.batch_decode(gen_tokens, skip_special_tokens=True)
```

TODOs:
- [x] add tests
- [x] implement `Speech2TextProcessor` after #10324 is merged
- [x] finish docs
- [ ] port and eval the CoVoST2 and MuSTc checkpoints
- [ ] add training/fine-tuning script in a follow-up PR


Comments:
@patrickvonplaten, @sgugger , @LysandreJik  The PR is now finalized and ready for your review :) 

I've added proper instructions to install the extra dependencies and addressed Patrick and Sylvain's comments regarding the docs and imports. All slow/non-slow tests are passing!

Merging!

Edit: please see this issue: #10631 

hi @xjdeng

Thanks for reporting this. This PR is now merged. So could you please open an issue with this error, we will discuss it there.
Invalid pytorch_model.bin for TAPAS-large
Author: saichandrapandraju
Pull request: no

Hi,
I was downloading `google/tapas-large` binaries from [HF models](https://huggingface.co/google/tapas-large/tree/main) and for pytorch_model.bin, a zip file was getting downloaded which is not a `bin` file like other models.

folder structure is like - `archive/data/.*File`, `archive/data.pkl` ,` archive/version.File`. Same for tapas-base as well. 
These cannot be used for loading a model (`TapasForQuestionAnswering.from_pretrained('path/to/binaries directory')`)

plz suggest how to proceed further..

Comments:
Hi, not sure why this happens, cc @julien-c.

A workaround is to load the model using `model = TapasModel.from_pretrained("google/tapas-base")` and then use `model.save_pretrained("./")` to save the `config.json` and `pytorch_model.bin` file to a local directory. 

I don't remember how those models were uploaded so not sure why this is happening.

cc'ing @Pierrci for visibility

In the meantime you can just rename the file to .bin

@saichandrapandraju the file on tapas-large is a bin file, but since pytorch 1.6.0, bin files are now zip-based. You can check the documentation of [torch.save](https://pytorch.org/docs/stable/generated/torch.save.html) which states:

> The 1.6 release of PyTorch switched torch.save to use a new zipfile-based file format. torch.load still retains the ability to load files in the old format. If for any reason you want torch.save to use the old format, pass the kwarg _use_new_zipfile_serialization=False.

Are you using a torch version inferior to 1.6.0 to try to load the models?

As pointed out by @julien-c, it is actually downloaded as a zip file, which seems to be the case for several models (my guess is that it does that when the file is zip-based, like all models saved with torch >1.6).

Downloading through git doesn't have that issue.

Ok will close this as:
- we found the root cause but it's outside our control (`lfs` adds an auto content-type)
- there are several workarounds like `git clone`ing the repo or just renaming the file (`from_pretrained` also works as usual)

Thanks for investigating @Pierrci and @LysandreJik 🥇
ImportError: cannot import name 'BERT_PRETRAINED_MODEL_ARCHIVE_MAP' from 'transformers'
Author: lethienhoa
Pull request: no

Hi,

I'm rerunning the code BERT-E2E-ABSA from https://github.com/lixin4ever/BERT-E2E-ABSA, which is based on transformers 2.0.0

I installed the latest transformers 3.0.2 (both from pip and souce) and I have an error that I can not import BERT_PRETRAINED_MODEL_ARCHIVE_MAP.
This simple command will result in errors below:
> python -c "from transformers import BERT_PRETRAINED_MODEL_ARCHIVE_MAP"

> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> ImportError: cannot import name 'BERT_PRETRAINED_MODEL_ARCHIVE_MAP' from 'transformers' (/home/hoa/transformers/src/transformers/__init__.py)

Transformer is well installed as it can do the following:
> python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I hate you'))"
> [{'label': 'NEGATIVE', 'score': 0.9991129040718079}]

Do you know why ? Did I miss something ?

- `transformers` version: 3.0.2
- Platform: Linux-5.4.0-1019-gcp-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.2.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
- Other configurations: Ubuntu 20.04, Python 3.7, CUDA 11

Note: I even installed pytorch on cpu laptop and rerun, still had the same issue !



Comments:
Yeah, there was a breaking change. The name has been changed to `BERT_PRETRAINED_MODEL_ARCHIVE_LIST`.

See https://github.com/huggingface/transformers/pull/4636.

ImportError: cannot import name 'BertweetTokenizer' from 'transformers' (/opt/conda/lib/python3.7/site-packages/transformers/__init__.py)


ImportError: cannot import name 'SquadExample' from 'transformers' (unknown location)

I get this error when I try to import QuestionAnsweringModel from simpletransformers.question_answering

Any help would be appreciated!
saving best model only using modelcheckpoint keras
Author: rohanshingade
Pull request: no
Labels: name: wontfix

while using model.fit for training a Transformer model, how to use `tf.keras.callbacks.ModelCheckpoint` with `'save_best_only=True'` to save only the best model?

`model.fit([input_ids_train, attention_mask_train], train_labels,
          validation_data=([input_ids_valid, attention_mask_valid], valid_labels),
          epochs=50, batch_size=32)`


Comments:
Have you taken a look at the [Keras docs](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint)?

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

@LysandreJik  Yes i looked the following doc and implemented it.

```
config = AutoConfig.from_pretrained(
    PRETRAINED_MODEL_VERSION, num_labels=len(classes), label2id=label2id, id2label=id2label,
    finetuning_task="text-classification")
model = TFAutoModelForSequenceClassification.from_pretrained(PRETRAINED_MODEL_VERSION, config=config)

loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.Accuracy()
optimizer = tf.keras.optimizers.Adam(learning_rate=2e-6, epsilon=1e-08)
model.compile(loss=loss, optimizer=optimizer, metrics=[metric])
print(model.summary())

es_callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, verbose=1)
chkpt_calllback = tf.keras.callbacks.ModelCheckpoint(MODEL_DIRECTORY+"{epoch:02d}",
                                                     monitor='val_loss', verbose=2,
                                                     save_best_only=True, save_weights_only=False)

model.fit([input_ids_train, attention_mask_train], train_labels,
          validation_data=([input_ids_valid, attention_mask_valid], valid_labels),
          epochs=1, batch_size=32, callbacks=[es_callback, chkpt_calllback])
```
but doing so saves the model checkpoint only and I'm not sure how to load these weights again into the **model**. As it only accepts **.h5** format. Facing difficulty in loading these weights
```
checkpoint
02.index
02.data-00000-of-00001
```