ProphetNet

Related to pretrained model cards

# Add [ProphetNet](https://arxiv.org/abs/2001.04063).

This PR implements both ProphetNet and XLM-ProphetNet. The model architectures are identical, but each model uses a different tokenizer.

## Description:

ProphetNet is a new pre-trained language model for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. ProphetNet is able to predict more future tokens with an n-stream decoder. The original implementation is Fairseq version at [github repo](https://github.com/microsoft/ProphetNet).
xProphetNet has the same model structure but is pretrained with wikipedia 100 languages dataset as described in [xGLUE](https://arxiv.org/abs/2004.01401). xGLUE is a benchmark for cross-lingual NLU and NLG tasks. xProphetNet is also served as a baseline model for cross-lingual generation tasks in xGLUE NTG and QG.

## Usage:

Take xGLUE NTG task as an example:
The cross-lingual pretrained model is finetuned with English news title generation data, but inference with both English and other zero-shot language data.
A quick usage is like:
```
from transformers import ProphetNetTokenizer, ProphetNetForConditionalGeneration, ProphetNetConfig

model = ProphetNetForConditionalGeneration.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-ntg')
tokenizer = ProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-ntg')

EN_SENTENCE_TO_QUESTION = "Microsoft Corporation intends to officially end free support for the Windows 7 operating system after January 14, 2020, according to the official portal of the organization. From that day, users of this system will not be able to receive security updates, which could make their computers vulnerable to cyber attacks."
RU_SENTENCE_TO_QUESTION = "орпорация Microsoft намерена официально прекратить бесплатную поддержку операционной системы Windows 7 после 14 января 2020 года, сообщается на официальном портале организации . С указанного дня пользователи этой системы не смогут получать обновления безопасности, из-за чего их компьютеры могут стать уязвимыми к кибератакам."
ZH_SENTENCE_TO_QUESTION = "根据该组织的官方门户网站，微软公司打算在2020年1月14日之后正式终止对Windows 7操作系统的免费支持。从那时起，该系统的用户将无法接收安全更新，这可能会使他们的计算机容易受到网络攻击。"
inputs = tokenizer([EN_SENTENCE_TO_QUESTION, RU_SENTENCE_TO_QUESTION, ZH_SENTENCE_TO_QUESTION], padding=True, max_length=256, return_tensors='pt')

summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=100, early_stopping=True)
print([tokenizer.decode(g) for g in summary_ids])
```
Model will generate news titles like:
```
['[SEP] Microsoft to end Windows 7 free support after January 14, 2020[SEP][PAD][PAD][PAD][PAD]',
 '[SEP] Microsoft намерена прекратить бесплатную поддержку Windows 7 после 14 января 2020 года[SEP]',
 '[SEP]微软打算终止对Windows 7操作系统的免费支持[SEP][PAD][PAD][PAD][PAD][PAD][PAD]']
```

## Released checkpoints:

pretrained:
```
microsoft/prophetnet-large-uncased
microsoft/xprophetnet-large-wiki100-cased
```
fine-tuned:
```
microsoft/prophetnet-large-uncased-cnndm
microsoft/xprophetnet-large-wiki100-cased-xglue-ntg
microsoft/xprophetnet-large-wiki100-cased-xglue-qg
```

## Notes

According to the outputs of original fairseq outputs, integration tests for prophetnet include:

1. encoder hidden states, decoder hidden states, model hidden states of pretrained Prophetnet, xProphetnet checkpoints
2. model hidden states of xProphetnet NTG finetuned model
3. Cross-lingual outputs of xProphetNet NTG finetuned model with different beam sizes
4. CNN/DM outputs of ProphetNet CNN/DM finetuned model with different input lengths

The model was implemented so all of its parts can be used separately. This means that `ProphetNetEncoder` and `ProphetNetEncoder` can be used as stand-alone models. `ProphetNetForCausalLM` can be instantiated easily from pretrained checkpoints and can be used within the EncoderDecoderModel framework.


[doc] [testing] improve/expand the Parametrization section

Fixed some issues, plus documented `@pytest.mark.parametrize` which is used extensively in the `examples` tests.

@sgugger


build/eval/gen-card scripts for fsmt

Here are the various build/eval/gen scripts used for fsmt. We will use them in the future should any updates/corrections need to be done.

They are also a great starter for porting other similar architectures.

Fixes #7092


Inconsistent parameter naming conventions in ModelConfigs

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2 (N/A)

### Who can help

@LysandreJik

## Information

This isn't exactly a bug, but more a suggestion that would be helpful. Across several different models, there are config classes used to initialize a model (e.g. GPT2Config and BertConfig). Some of these configs have parameters that mean the same thing, but use different naming conventions for the parameters, for instance n_layer in GPT2Config is the same, conceptually, as num_hidden_layers for BertConfig.

For ease of development, I think it could be worthwhile to make these parameter names consistent across all of the transformer config classes.

I'd be happy to help out on this, it seems like it should be a relatively straightforward change, though it may break some backwards compatibility.


[model cards] ported allenai Deep Encoder, Shallow Decoder models

model card

Related to pretrained model cards

Once @LysandreJik merges https://github.com/huggingface/transformers/pull/6940, please merge these cards

These are models ported from https://github.com/jungokasai/deep-shallow/

The models are already on s3 under `allenai` - thank you for moving those from my username, @sshleifer.

And added 2 more models from the same author.

Fixes #7049


Gradient checkpointing for GPT-2

# 🚀 Feature request

Gradient checkpointing for GPT-2 would be very useful especially for the larger models in the GPT-2 family. I tried to train the largest release, i.e. "gpt2-xl", on AWS's p3dn.24xlarge instance (has 8 V100 GPUs with 32 GB VRAM on each) with no success. Even making the batch size 1 and using mixed precision training (through torch.cuda.amp) failed with OOM. By the way, I was able to train the second largest model ("gpt2-large") with that configuration i.e. batch size of 1 and enabling mixed precision).


fix the warning message of overflowed sequence

This PR fixes the warning message (of _**PreTrainedTokenizerBase.prepare_for_model**_) when trying to tokenize a model with a sequence that is longer than the specified maximum sequence length.

The current warning message shows the length of the first input_ids, which can be incorrect when a pair is being encoded and the first input length doesn't exceed the limit. The desired warning message should show the overall length of the encoded ids.


Refactoring the TF activations functions

This PR groups all the TF activations functions into a common file like it is done in the PT part.


Ignore me!

I did this PR by mistake, and found out late. Sorry Hugging Face!


add new model prophetnet

Related to pretrained model cards

prophetnet modified

modify codes as suggested v1

add prophetnet test files


Funnel model cards

Related to pretrained model cards

Add the model cards for the 10 Funnel Transformer checkpoints


Fine tune with local model raised `torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'config'`

# ❓ Questions & Help

I've fined tune distilgpt2 with `run_language_modeling.py` under my local `output_dir` and want to fine tune with the model in `output_dir` again, but it raised `torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'config'`.

## Details
<!-- Description of your issue -->
I've fined tune distilgpt2 with `run_language_modeling.py` as follows:
```shell
python3 run_language_modeling.py     --output_dir=/home/xxx/gpt_model/transformers/examples/language-modeling/output_dir --model_type=gpt2  --model_name_or_path=distilgpt2 --per_device_train_batch_size=10     --do_train     --train_data_file=/home/xxx/gpt_model/data_info/data.txt --block_size=64 --save_steps=100 --overwrite_output_dir
```
It works fine and i can get the fine-tuned model after training with my data `data.txt`. Now i want to fine-tune in the `output_dir` so i run `run_language_modeling.py` as follows on my `new_data.txt`:
```shell
 python3 run_language_modeling.py     --output_dir=/home/xxx/gpt_model/transformers/examples/language-modeling/output_new_data --model_type=gpt2  --model_name_or_path=/home/xxx/gpt_model/transformers/examples/language-modeling/output_dir/  --per_device_train_batch_size=20  --do_train     --train_data_file=/home/xxx/gpt_model/data_info/new_data.txt --block_size=64 --save_steps=1000 --overwrite_output_dir
```
But it raise exception and exit. The stderr is as follows
```shell
/home/xxx/gpt_model/pytorch/pytorch/torch/optim/lr_scheduler.py:235: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)
Traceback (most recent call last):
  File "run_language_modeling.py", line 320, in <module>
    main()
  File "run_language_modeling.py", line 284, in main
    trainer.train(model_path=model_path)
  File "/home/xxx/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/trainer.py", line 683, in train
    self.total_flos = getattr(model.config, "total_flos", 0)
  File "/home/xxx/gpt_model/pytorch/pytorch/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'config'
```
##### (1) I'm training on multiple GPUs, but i did'n specify a specific gpu to run. So at first i think this may caused by multiple GPUs, and add the following code under https://github.com/huggingface/transformers/blob/52d250f6aa14844024806e5e4dd1c7882bbd8dd5/src/transformers/trainer.py#L641
```shell
if isinstance(model,torch.nn.DataParallel):
		model = model.module
```
It has no error but doesn't work. The output is as follows . After that, the process is killed and return code `echo $?` is `0`.
```shell
Epoch: 0it [00:00, ?it/s]
/home/lenajin/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/trainer.py:1087: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
  warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)
```
##### (2) Following with https://github.com/huggingface/transformers/issues/1991 , I try to train with following script to train with gpu whose id is `1`.
```shell
CUDA_VISIBLE_DEVICES=1 python3 run_language_modeling.py     --output_dir=/home/xxx/gpt_model/transformers/examples/language-modeling/output_dir --model_type=gpt2  --model_name_or_path=distilgpt2 --per_device_train_batch_size=20     --do_train     --train_data_file=/home/xxx/gpt_model/data_info/data.txt --block_size=64 --save_steps=100 --overwrite_output_dir
```
it return with return code `echo $?` =`0`. At first i don't know what's the problem, but now i changed `--per_device_train_batch_size=20    ` from 20 to 10. It seems all is well, and now my `GPU-Util` is about `98%`.  Maybe it return without training duing to my limit gpu CC？It's wired but at least it works now. Maybe there can give some error message about the return reason?


Load Pre-Trained Model Using Docker

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
I'm trying to load a pre trained model using transformers lib (by hugging-face):

from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')

Using local machine, it starts to download the model. But with docker I get the following:

OSError: Model name 'gpt2-medium' was not found in tokenizers model
name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). We
assumed 'gpt2-medium' was a path, a model identifier, or url to a
directory containing vocabulary files named ['vocab.json',
'merges.txt'] but couldn't find such vocabulary files at this path or url.

Any idea why it happens?


unexpected keyword argument 'force_fusions' when running the onnx notebook

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-5.8.0-1-amd64-x86_64-with-debian-bullseye-sid
- Python version: 3.7.8
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

The problem arises when using:
* [x] the official example scripts: the notebook "04-onnx-export"
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the notebook's cells.
First cell installs onnxruntime in version 1.4.0
The "Benchmarking ONNX quantized model" cell calls transformers/convert_graph_to_onnx.py,
which calls the "quantize" function with parameter force_fusions.

`Error: quantize() got an unexpected keyword argument 'force_fusions'`
The parameter was discarded in onnxruntime 1.4.0
Removing the parameter in convert_graph_to_onnx.py solves the issue.
## Expected behavior

No error.


Tiny typo fix

Small typo in the `generate` function within `generation_tf_utils.py`


Add quotes to paths in MeCab arguments

Without quotes directories with spaces in them will fail to be processed
correctly.

This bug was first reported in the fugashi repo [here](https://github.com/polm/fugashi/issues/24). It's not a bug in fugashi - fugashi handles quoted paths correctly, and the relevant dictionary packages quote paths in `MECAB_ARGS` to handle this case. Looks like it was missed when the dictionary handling code was added, since it doesn't use `MECAB_ARGS` but instead builds args up from component parts.


Adding Fast tokenizers for SentencePiece based tokenizers - Breaking: remove Transfo-XL fast tokenizer

This pull request add the "fast" Rust tokenizer for the SentencePiece tokenizers as well.

Based on unreleased v0.9.0 of `tokenizers`.

Tokenizers:
- [x] Albert
- [x] Bart
- [x] Bert
- [x] Camembert
- [x] DistilBert
- [x] DPR
- [x] Electra
- [x] Funnel
- [x] GPT2
- [x] LongFormer
- [x] LXMert
- [x] MBart
- [x] MobileBert
- [x] OpenAI GPT
- [x] Pegasus
- [x] Reformer
- [x] RetriBert
- [x] Roberta
- [x] T5
- [x] XLM-Roberta
- [x] XLNet

Breaking:
- Fast version of Transformer-XL (which gave different tokenization results) is removed.

Remaining tokenizers without Fast implementations (no fast tokenizers expected in the short/mid-term):
- BertJapanese (special python libs for multi-linguality)
- CTRL (would require a specific BPE to handle missing merges)
- XLM (uses special python libs for multi-linguality)
- Flaubert (same as XLM)
- Transformer-XL (same as XLM)

Other fixes:
- Also allow to tokenizer Bert Japanese with Mecab token splitter and fix https://github.com/huggingface/datasets/issues/665
- deprecation warning in tokenizer methods are limited to one occurence per class instance


Onnx + TensorRT uses CPU not GPU

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@mfuntowicz

## Information

Model I am using Bert:

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

https://colab.research.google.com/drive/1mqn99U2NMm-fdw213h0xsqm_uRwRkNzp?usp=sharing

## Expected behavior

I am trying to run onnx using tensorrt on the gpu, but it always uses the CPU.
I tried to install different version or onnx and cuda with not luck.
Any idea why "TensorrtExecutionProvider" runs on the CPU rather than GPU ? and how to fix it ?


generate text function support part of the  target inputs

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html?highlight=transformer#torch.nn.Transformer

in pytorch transformer class, we can see the forward function can pass target sentence; so when i wanna to generate the target_sentence from the middle , i can pass part of the target_sentence to the model, then i can generate the rest of the sentence.
## Motivation
the generate function is really powerful.
Here is my situation, i have the full source sentence and certain part of the target sentence, then i begin to generate the rest of the target sentence, using beam search etc..
But i found the interface do not support part of target sentence as input.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-47-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in

Question:

After installing transformers, I followed the Quick tour document.

I typed code as follows:

from transformers import pipeline
classifier = pipeline('sentiment-analysis', model="nlptown/bert-base-multilingual-uncased-sentiment")

After running it, I got this error

2020-09-15 16:41:21.434156: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "/home/a/.conda/envs/tf23hf/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 580, in from_pretrained
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hf_pipetest.py", line 2, in <module>
    classifier = pipeline('sentiment-analysis', model='nlptown/bert-base-multilingual-uncased-sentiment')
  File "/home/a/.conda/envs/tf23hf/lib/python3.7/site-packages/transformers/pipelines.py", line 2629, in pipeline
    model = model_class.from_pretrained(model, config=config, **model_kwargs)
  File "/home/a/.conda/envs/tf23hf/lib/python3.7/site-packages/transformers/modeling_tf_auto.py", line 1515, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/home/a/.conda/envs/tf23hf/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 587, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load weights for 'nlptown/bert-base-multilingual-uncased-sentiment'. Make sure that:

- 'nlptown/bert-base-multilingual-uncased-sentiment' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'nlptown/bert-base-multilingual-uncased-sentiment' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.

Please, let me know how to solve this problem..
Thanks in advance


## Environment info

- `transformers` version: 3.1.0
- Platform: ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?): 1.14.0
- Using GPU in script?:  Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using: XLM

The problem arises when using:
When stack two transformes on top of each other and run them on GPUs. (Note that on CPU the following code works fine)

## To reproduce
Steps to reproduce the behavior:

```
import torch
from torch import nn
from transformers import AutoConfig, AutoModel

class Model(nn.Module):

    def __init__(self):

        super().__init__()
        config = AutoConfig.from_pretrained('xlm-mlm-tlm-xnli15-1024')
        self.transformer1 = AutoModel.from_config(config)
        self.transformer2 = AutoModel.from_config(config)

    def forward(
            self,
            input_ids=None,
            input_embeds=None,
    ):

        outputs = self.transformer1(input_ids)
        outputs = self.transformer2(inputs_embeds=outputs[0])
        return outputs

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Model().to(device)

inps = torch.randint(1,256,[1,256],device=device)

model(input_ids=inps)

```

The stack trace:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-35-5ada78c6ff36> in <module>
     34 inps = torch.randint(1, 256, [1, 256], device=device)
     35
---> 36 model(input_ids=inps)

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-35-5ada78c6ff36> in forward(self, input_ids, input_embeds)
     26         print(outputs[0].is_cuda)
     27         outputs = outputs[0].to(input_ids.device)
---> 28         outputs = self.transformer2(inputs_embeds=outputs)
     29         return outputs
     30

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.8/site-packages/transformers/modeling_xlm.py in forward(self, input_ids, attention_mask, langs, token_type_ids, position_ids, lengths, cache, head_mask, inputs_embeds)
    519         # if src_enc is not None:
    520         #     assert self.is_decoder
--> 521         #     assert src_enc.size(0) == bs
    522
    523         # generate masks

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
```


BertForMaskedLM Loss function

# ❓ Questions & Help

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py

1)In BertForMaskedLM, why the loss function don't ignore the not masked tokens?
loss_fct = CrossEntropyLoss()  # -100 index = padding token
masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
2) do i have to set the not masked token to -100 ?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Loss mask for fine-tuning GPT2LMHeadModel model

If we use padding for short-sentence fine-tune data, when fine-tuning GPT2LMHeadModel, should we change the code here https://github.com/huggingface/transformers/blob/48ff6d5109d691e3630169962a5052586aaaf659/src/transformers/modeling_gpt2.py#L744 to exclude the loss for padding tokens?

@patrickvonplaten @thomwolf


evaluate_during_training after each epoch

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
Hello! I just migrated from `pytorch_pretrained_bert` to `transformers` (3.1.0) and I am having problems understanding how to make the model evaluate at the end of each epoch. For this purpose, it is recommended to use `--evaluate_during_training` as mentioned in the issue [#4617](https://github.com/huggingface/transformers/issues/4617). Looking at `Trainer` source code the condition to run an evaluation is the following:
```python
if self.args.evaluate_during_training and self.global_step % self.args.eval_steps == 0:
                        self.evaluate()
```

I am not quite following the logic of why the evaluation depends on the number of steps and not on the number of epochs. I would appreciate if someone helps me on this.

In the same issue mentioned above (#4617) someone said that you should set `--save_steps` to `number_of_samples/batch_size`. If this is true (currently testing if this works on my dataset), shouldn't it be mentioned in the documentation?

Thank you!

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Update README

This PR reorganizes the README to reflect the current state of transformers a bit more. croll down a bit [here](https://github.com/huggingface/transformers/tree/update_readme) for a preview.

Here is the plan I followed:

**Introduction:** Expand a bit on the first two sentences to explain a bit more generally what this library is about.

The idea to have content aimed at both researchers/NLP experts and data scientists/engineers. Since it gets longer, we could move it after the top contributors badges.

**Online demo:** I think it makes more sense to have this right after the introduction, to show what transformers can do.

**Code sample:** Then we switch to how to do it, showcasing two shorts examples: pipeline and a pretrained model loading/example of use. Let’s not overwhelm a reader with more than that and link to our tutorials for more examples.

**Feature list:** This becomes Why use/not-use transformers but the spirit remains the same.

**Installation:** Keeping this simple and to the point, linking to the documentation for more in-depth content.

**Model architectures:** I've kept the full list with papers but I'm not sure it makes sense here as we also have it in the index of our doc. An alternative would be to show this in terms of tasks and give examples for each one of models supporting them (so key names still appear) with links to the docs.

**Learn more:** This is where all the content I removed is linked.

**Citation**


Add tokenizer file save in convert_graph_to_onnx.py

# 🚀 Feature request

Allow saving the associated tokenizer files (to be loaded possibly with the tokenizer library) when running `convert_graph_to_onnx.py`
## Motivation

Having the good vocab file next to the good onnx model to avoid confusion and ease the loading process by any other framework (because models and their tokenizers are deeply linked).

## Your contribution

With [this comment from tokenizer repo](https://github.com/huggingface/tokenizers/issues/59#issuecomment-610645970) and the [convert_graph_to_onnx.py](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_graph_to_onnx.py) one can simply add

L.358
```
if tokenizer is None:
        download_vocab_files_for_tokenizer(nlp.tokenizer, model, output)
```
and before `if __name__ == "__main__"`
```
def download_vocab_files_for_tokenizer(tokenizer, model_type, output_path):
    vocab_files_map = tokenizer.pretrained_vocab_files_map
    vocab_files = {}
    for resource in vocab_files_map.keys():
        download_location = vocab_files_map[resource][model_type]
        f_path = path.join(output_path.parent,
                           path.basename(download_location))
        urllib.request.urlretrieve(download_location, f_path)
```

I can do a PR for it if the interest is big enough.

NB : This is a patch I added in my local lib but feel free to modify it or transform it as you want, it's just to give the idea


[EncoderDecoderModel] fix indentation error

A statement regarding the encoder config init in `EncoderDecoderModel` was falsely indented which might lead to errors for very specific cases.


AssertionError with multiple GPU

## System Info
Red Hat Server 7.7
Pytorch: 1.6.0
Transformers: 3.0.2
Python: 3.7.6
Number of GPU: 4

## Question
I am trying to finetune a GPT2 model using `Trainer` with multiple GPU installed on my machine. However, I get the following error:
```python
Traceback (most recent call last):
  File "run_finetune_gpt2.py", line 158, in <module>
    main()
  File "run_finetune_gpt2.py", line 145, in main
    trainer.train()
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/transformers/trainer.py", line 499, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/transformers/trainer.py", line 622, in _training_step
    outputs = model(**inputs)
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 156, in forward
    return self.gather(outputs, self.output_device)
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 168, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather
    res = gather_map(outputs)
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 55, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/path/to/venvs/my-venv/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 54, in forward
    assert all(map(lambda i: i.is_cuda, inputs))
AssertionError
wandb: Program failed with code 1. Press ctrl-c to abort syncing.
wandb: You can sync this run to the cloud by running:
wandb: wandb sync wandb/dryrun-20200914_134757-1sih3p0q
```
Any ideas about what might be going on? Thanks in advance!


[WIP RAG] Finalize RAG parallel

finalizing RAG in parallel to https://github.com/huggingface/transformers/pull/6813


Fix the HF logger

Hello.

The current HF logger do not behave as expected. As example let's take the `run_tf_ner.py` script. With the following header:

```
import logging
import os
from dataclasses import dataclass, field
from importlib import import_module
from typing import Dict, List, Optional, Tuple

import numpy as np
from seqeval.metrics import classification_report, f1_score, precision_score, recall_score

from transformers import logging as hf_logging

handler = logging.StreamHandler()
formatter = logging.Formatter('[%(levelname)s|%(filename)s:%(lineno)s] %(asctime)s >> %(message)s')
handler.setFormatter(formatter)

logger = hf_logging.get_logger(__name__)

logger.handlers.clear()
logger.addHandler(handler)
hf_logging.enable_propagation()
hf_logging.set_verbosity_info()
hf_logging.enable_default_handler()

from transformers import (
    AutoConfig,
    AutoTokenizer,
    EvalPrediction,
    HfArgumentParser,
    TFAutoModelForTokenClassification,
    TFTrainer,
    TFTrainingArguments,
)
from utils_ner import Split, TFTokenClassificationDataset, TokenClassificationTask
```

The formatter is fully ignored and takes the default logging format of TensorFlow.

This PR fixes the problem with the following new header:
```
import logging
import os
from dataclasses import dataclass, field
from importlib import import_module
from typing import Dict, List, Optional, Tuple

import numpy as np
from seqeval.metrics import classification_report, f1_score, precision_score, recall_score
import tensorflow as tf

from transformers import logging as hf_logging

handler = logging.StreamHandler()
formatter = logging.Formatter('[%(levelname)s|%(filename)s:%(lineno)s] %(asctime)s >> %(message)s')
handler.setFormatter(formatter)

logger = hf_logging.get_logger()

logger.handlers.clear()
logger.addHandler(handler)
logger.setLevel(logging.INFO)

from transformers import (
    AutoConfig,
    AutoTokenizer,
    EvalPrediction,
    HfArgumentParser,
    TFAutoModelForTokenClassification,
    TFTrainer,
    TFTrainingArguments,
)
from utils_ner import Split, TFTokenClassificationDataset, TokenClassificationTask
```

But now the logger is set at library level and not at the class level anymore.

Then I wanted to know if this slitght update is a problem or if it is ok.


only fine tune the encoder part of BART

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
May i ask how can I fine-tune only the BART-encoder?

from transformers import BartTokenizer, BartModel

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartModel.from_pretrained('facebook/bart-base', return_dict=True)

inputs = tokenizer(["i am a girl", "i am from germany"], return_tensors="pt", padding=True)
outputs = model(**inputs).encoder_last_hidden_state

i load the pretrained BART, and get the encoder_last_hiddene_state.

however, during fine-tuning, how about the decoder part?

how could i only load the part of encoder and fine-tune?

Thanks a lot
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Multi predictions trainer

This allows the `Trainer` to properly return predictions when the model has several outputs (for instance, all models `XxxForMultipleChoice`). This should unlock progress in #7032 where the start and end logits are both required.


fix ZeroDivisionError and epoch counting

@sgugger

This PR fix 2 minor bugs in `trainer.py`.

First, the two lines

    num_train_epochs = (
        self.args.max_steps // (len(train_dataloader) // self.args.gradient_accumulation_steps) + 1
    )

and

    epochs_trained = self.global_step // (len(train_dataloader) // self.args.gradient_accumulation_steps)

gives `ZeroDivisionError` when `len(train_dataloader) < self.args.gradient_accumulation_steps`.

The fix takes into account the code below the comment

     # last step in epoch but step is always smaller than gradient_accumulation_steps

therefore when `len(train_dataloader) < self.args.gradient_accumulation_steps`, we still have 1 step in each epoch.

The second bug is due to the `+ 1` in the

    num_train_epochs = (
        self.args.max_steps // (len(train_dataloader) // self.args.gradient_accumulation_steps) + 1
    )

In the example below, we have

    len(train_dataloader) == 40
    self.args.gradient_accumulation_steps = 4
    self.args.max_steps = 10

However we get `num_train_epochs == 2` from the original code, which should be `1`.

    python utils/download_glue_data.py --data_dir ./examples/text-classification/glue/ --tasks all

    python3 run_glue.py \
        --task_name wnli \
        --data_dir ./glue/WNLI \
        --model_name_or_path distilbert-base-uncased \
        --output_dir ./glue/WNLI/ \
        --max_seq_length  16 \
        --num_train_epochs 1 \
        --per_device_train_batch_size 16 \
        --gradient_accumulation_steps 4 \
        --max_steps 10 \
        --logging_steps 1 \
        --save_steps 5 \
        --seed 1 \
        --do_train \
        --do_eval \
        --do_predict \
        --overwrite_output_dir

    (Pdb) len(train_dataloader)
    40
    (Pdb) self.args.gradient_accumulation_steps
    4
    (Pdb) self.args.max_steps
    10
    (Pdb) p num_train_epochs
    2


[s2s] distributed eval in one command

### Issue

One GPU command:
```bash
python run_eval.py Helsinki-NLP/opus-mt-en-ro wmt_en_ro/test.source enro_test_translations.txt --reference_path wmt_en_ro/test.target --task translation --score_path mar_test_bleu.json --fp16 --bs 64
# {'bleu': 27.6865, 'n_obs': 1999, 'runtime': 85, 'seconds_per_sample': 0.0425}
```

Multi GPU:
```
python -m torch.distributed.launch --nproc_per_node=2  run_distributed_eval.py --model_name $opus --input_path wmt_en_ro --type_path test --fp16 --save_dir tmp_gen --fp16 --bs 64
python aggregate_distributed_results.py tmp_gen tmp_gen --calc_bleu
cat tmp_gen/metrics.json
# "bleu": 27.7772
```

### New command

```
python -m torch.distributed.launch --nproc_per_node=2  run_distributed_eval.py --model_name Helsinki-NLP/opus-mt-en-ro --data_dir wmt_en_ro --type_path test --save_dir tmp_gen4 --fp16 --bs 64 --task translation
```

### Future PRs

The actual generations differ slightly vs the single gpu implementation, likely because of sortish sampler/leading spaces.


Seq2SeqDataset experiment: try to use arrow datasets

Help wanted

Extra attention is needed, help appreciated

### High Level Goal
See if [`datasets/arrow_dataset.py`](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L172) can improve `example/seq2seq/utils.py`'s [`Seq2SeqDataset`](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py#L165)

This is more experimental/proof of concept.

### Spec
+ Generate a PR that adds `Seq2SeqArrowDataset` with a similar signature to `Seq2SeqDataset` but uses the arrow dataset as a backend.
+ This means that wherever `linecache` is used, arrow should be used.
+ You can flex the API as much as you'd like, but if you don't call `prepare_seq2seq_batch` you will have to implement lots of collate fns. (If you go this route, you can just implement 1)
+ Feel free to reduce the scope to either translation or summarization.

PR description should answer the following question:

- Can we generate similar batches with an ArrowDataset?
- Is the dataset faster?
- Does it consume as little RAM?
- Does it simplify the code?

Let me know if this is impossible or unclear!


backtranslation script

+ Wait for #7106, then resume in https://github.com/huggingface/transformers/pull/7121.


[blk] backtranslation script

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

@TevenLeScao

## Information

Model I am using (Bert, XLNet ...): xlnet-base-uncased

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
*  [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Simply invoke XLNet using the following

```
last_hidden_state, mems = lm(
                input_ids=input_ids,
                token_type_ids=token_type_ids,
                output_hidden_states=True,
            )
```

I receive a warning as follows:

```
...\anaconda3\envs\newstsc2\lib\site-packages\transformers\modeling_xlnet.py:283: UserWarning: Mixed memory format inputs detected while calling the operator. The operator will output contiguous tensor even if some of the inputs are in channels_last format. (Triggered internally at  ..\aten\src\ATen\native\TensorIterator.cpp:918.)
  attn_score = (ac + bd + ef) * self.scale
```

## Expected behavior

no warning


Fix reproducible tests in Trainer

This fixes the tests that were using hardcoded values for reproducible training by testing instead the training yields the same result as one run with the same seed during setup. As a result, it can also run in a multigpu-environment (which is why I removed the decorator).

I also fixed the batch size to count the number of steps, which should also fix those tests for multigpu-environent. @stas00, when you get a chance to run the tests on that file, it would be great to know if the fix worked as intended.


Temporarily skip failing tests due to dependency change

cc @sgugger @stas00


Feature request: State the goals and none goals of the library in the README

# 🚀 Feature request

I'd like the library's README to say what it's goals and none goals are.
For example, as a (new) user I really like [Typescripts non-goals](https://github.com/microsoft/TypeScript/wiki/TypeScript-Design-Goals) because it made it clear what to expect from Typescript and thus how to best use it. I'd like the same for transformers.

## Motivation

As a new user  I look at the README and think this library can and wants to do everything that I do.

But I think it has a tighter more well defined (implicit) scope and that peripheral use cases are better handled elsewhere, such as aligning [offset annotations](https://github.com/huggingface/transformers/issues/7019#issuecomment-691965153) or having trainers [support other data structures](https://github.com/huggingface/transformers/issues/6860).

I wish that was stated (as clearly as possible) so that I could make the best use of transformers instead of forcing it to be something it doesn't want to be.

## Your contribution

I think it's up to the Huggingface team to decide what transformers is and what it will be. After digging around for a few days I would suggest something like
> Transformers makes pre-trained language models available to everyone through clear, consistent and cross platform APIs. Transformer's goal(s) are to make pretrained models accessible and plug into end users existing infrastructure, as well as support common academic tasks.

> Transformers does not aim to
* Implement end to end applications of NLP
* Implement domain specific models (Custom entities in Swahili, joint classification and entity annotation)
* ....


fix link to paper

change to url of the cited paper (the previous one linked to a different paper)


can i self-define the decoder when i user EncoderDecoderModel?

# ❓ Questions & Help
In my task, i have to define a decoder by myself, because the vocabulary is Chinese words.
But it seems the class method do not support that, i have to init the decoder from the pretrained model.

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


How to return the word embeddings and how to understand the hidden_states in return?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I am new here, transformers is amazing. What I want is to get the embeddings of words with context , “logits” is obviously a bit thin, I think the hidden_states may represent the word embedding, but I can not understand the output of the embeddings and output of each layer.
Which one should I choose?
Thanks very much!

> hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) – Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Generate coherent text with T5

Can we use T5 to generate coherent text given an input line like in case of GPT-2? If yes how do we fine-tune T5 for such a task?


prepare for the label for EncoderDecoderModel

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->
When i using a EncoderDecoderModel for a translation task, do i have to right shift the target token to generate the label?
The [tutorial ](https://github.com/huggingface/transformers/tree/master/model_cards/patrickvonplaten/bert2bert-cnn_dailymail-fp16) not do that, and the source code in  forward  function do neither.

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


MBART/Marian for low resource/backtranslation

Are there any finetuned checkpoints of mBART on any low resource language release by hugging face. I can see English- Romanian but was wondering if there is a way to access other languages. Did facebook release them actually?

@sshleifer


[s2s] distributed eval cleanup


[s2s run_eval] new features

this PR adds several features to `run_eval.py` and adds a new script `run_eval_search.py`

* adding 2 new args:
```
  --dump-args           print the custom hparams with the results
  --info [INFO]         use in conjunction w/ --dump-args to print with the results whatever other info you'd like,
                        e.g. lang=en-ru. If no value is passed, the current datetime string will be used.
```
* changed `parse_numeric_n_bool_cl_kwargs` to support bool args, so now we can pass ` --early_stopping true`

* added a new wrapper script `run_eval_search.py` that performs parametric search using  `run_eval.py`.

Here is the new section in `README.md` that explains all the additions:

#### run_eval tips and tricks

When using `run_eval.py`, the following features can be useful:

* if you running the script multiple times and want to make it easier to track what arguments produced that output, use `--dump-args`. Along with the results it will also dump any custom params that were passed to the script. For example if you used: `--num_beams 8 --early_stopping true`, the output will be:
   ```
   {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True}
   ```

   `--info` is an additional argument available for the same purpose of tracking the conditions of the experiment. It's useful to pass things that weren't in the argument list, e.g. a language pair `--info "lang:en-ru"`. But also if you pass `--info` without a value it will fallback to the current date/time string, e.g. `2020-09-13 18:44:43`.

   If using `--dump-args --info`, the output will be:

   ```
   {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True, 'info': '2020-09-13 18:44:43'}
   ```

   If using `--dump-args --info "pair:en-ru chkpt=best`, the output will be:

   ```
   {'bleu': 26.887, 'n_obs': 10, 'runtime': 1, 'seconds_per_sample': 0.1, 'num_beams': 8, 'early_stopping': True, 'info': 'pair=en-ru chkpt=best'}
   ```

* if you need to perform a parametric search in order to find the best ones that lead to the highest BLEU score, let `run_eval_search.py` to do the searching for you.

   The script accepts the exact same arguments as `run_eval.py`, plus an additional argument `--search`. The value of `--search` is parsed, reformatted and fed to ``run_eval.py`` as additional args.

   The format for the ``--search`` value is a simple string with hparams and the values to try, e.g.:
   ```
    --search "num_beams=5:10 length_penalty=0.8:1.0:1.2 early_stopping=true:false"
   ```
   which will generate `12` `(2*3*2)` searches for a product of each hparam. For example the example that was just used will invoke `run_eval.py` repeatedly with:

   ```
    --num_beams 5 --length_penalty 0.8 --early_stopping true
    --num_beams 5 --length_penalty 0.8 --early_stopping false
    [...]
    --num_beams 10 --length_penalty 1.2 --early_stopping false
   ```

   On completion this function prints a markdown table of the results sorted by the best BLEU score and the winning arguments.
```
bleu  | num_beams | length_penalty | early_stopping
----- | --------- | -------------- | --------------
26.71 |         5 |            1.1 |              1
26.66 |         5 |            0.9 |              1
26.66 |         5 |            0.9 |              0
26.41 |         5 |            1.1 |              0
21.94 |         1 |            0.9 |              1
21.94 |         1 |            0.9 |              0
21.94 |         1 |            1.1 |              1
21.94 |         1 |            1.1 |              0

Best score args:
stas/wmt19-en-ru data/en-ru/val.source data/en-ru/test_translations.txt --reference_path data/en-ru/val.target --score_path data/en-ru/test_bleu.json --bs 8 --task translation --num_beams 5 --length_penalty 1.1 --early_stopping True
```
@sshleifer


[QOL] add signature for prepare_seq2seq_batch

This pr adds the signature, but no implementation of `prepare_seq2seq_batch` to `PretrainedTokenizer` to allow IDE autocompletion when `AutoTokenizer` is used.
+ The signature is quite large and it is nice to see what needs to be supplied.
+ The signature is enforced by the unittests, so it won't be misleading.

On this branch, where `self.tokenizer = AutoTokenizer(...)`:
![image](https://user-images.githubusercontent.com/6045025/93033269-8ed6d480-f603-11ea-9cc4-1947d40de134.png)


Update xsum length penalty to better values

+ Improves ROUGE by 0.4 pts.
+ Already reflected on S3, so this is just a unittest fix.


One command to run+aggregate distributed evaluation results

### Current Situation
In https://github.com/huggingface/transformers/pull/7105, I wrote a three command combo to run distributed eval.
The three commands are:

```
python -m torch.distributed.launch --nproc_per_node=2 run_distributed_eval.py --fp16 --bs 16
python aggregate_distributed_results.py tmp_gen  tmp_gen2
rm -rf tmp_gen
```
+ The first command splits up the data and runs `generate` on a chunk foreach GPU, saving results to `rank_{rank}.json`
+ The second command combines the json results, either just calculating metrics or resaving the generations to disk as `{save_dir}.pred_target`, `save_dir.source` (optionally) `save_dir.target`.
+ the third command deletes the rank.json files.
+ the saving of these independent files to disk in the second command is useful for pseudolabeling, where we train a small model on the predictions of a big model. We have to do more book-keeping than in run_eval.py because I haven't yet determined how to reorder predictions to match the original data. So I just save the original data (roughly, there might be truncation issues) and then write it back to disk.

Goal: 1 command that uses multiple gpus, saves **aggregated** files and computes metrics `metrics.json`. If this command cannot guarantee the ordering of the generations, it must save necessary data to disk.

The design choices in my dirty first attempt do not need to be continued, as long as we can compute metrics and train a second model with the predictions as the new labels

There are many ways to accomplish this, here are a few ideas (not mutually exclusive).

Ideally, this would be 1 command with the order "figured out" somehow, possibly by returning ids from `Seq2SeqDataset`
```
python run_eval.py (existing_args) --gpus 2
```
would just work. No need to save source/labels since they are in the correct order.
To me this sounds hard to implement. I tried briefly and gave up.
The goal now is 1 command, it doesn't need to be the `run_eval.py` command.

### Figuring out ordering by having Seq2Seq dataset return ids
- Ideally, this would be 1 command with the order "figured out" somehow, possibly by returning ids from `Seq2SeqDataset`. Then you don't need to save labels/source documents to disk, you can just reorder the predictions.

### launch n processes in the code rather than from the command line

if we call `torch.multiprocessing.spawn` ourselves, as in
https://github.com/facebookresearch/ParlAI/blob/00efcbebb49524918692638ab580cadeebe70cf8/parlai/scripts/multiprocessing_eval.py#L49

we can wait for the results, join them, and do the reordering in one command.


[s2s] two stage run_distributed_eval.py

```bash
python -m torch.distributed.launch --nproc_per_node=2 run_distributed_eval.py --fp16 --bs 16
python cleanup_distributed.py tmp_gen  tmp_gen2
```


[s2s distill] allow pegasus-12-12


## Environment info

- `transformers` version: 3.1.0
- Platform: Darwin-18.0.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No (Running on Macbook Pro MacOS Mojave 10.14)

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): Bert Multilingual

The problem arises when using:
* [ ] the official example scripts: (give details below)

Hi, this is an error I'm getting running a package someone else created called [simalign](https://github.com/cisnlp/simalign/). I believe their package works with transformers 2.3.0 (I tried it and it worked).

But after trying to upgrade to the newest transformers because I really wanted the fill-mask feature which was more recently released.

(Actually I wanted a way to get the probability of a word in a certain position given the words around it  (left and right) in a sentence and have yet to find a way to do that).

At the same time I also want to run simalign in the same application (so I can't downgrade to the previous transformers make it work again).

I'm not exactly sure what is causing the error (as I'm not experienced enough) but I'll share the traceback below here:

```
>>> import simalign
>>>
>>> source_sentence = "Sir Nils Olav III. was knighted by the norwegian king ."
>>> target_sentence = "Nils Olav der Dritte wurde vom norwegischen König zum Ritter geschlagen ."
>>> model = simalign.SentenceAligner()
2020-09-13 18:02:40,806 - simalign.simalign - INFO - Initialized the EmbeddingLoader with model: bert-base-multilingual-cased
I0913 18:02:40.806071 4394976704 simalign.py:47] Initialized the EmbeddingLoader with model: bert-base-multilingual-cased
>>> result = model.get_word_aligns(source_sentence.split(), target_sentence.split())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simalign/simalign.py", line 181, in get_word_aligns
    vectors = self.embed_loader.get_embed_list(list(bpe_lists))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simalign/simalign.py", line 65, in get_embed_list
    outputs = [self.emb_model(in_ids.to(self.device)) for in_ids in inputs]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simalign/simalign.py", line 65, in <listcomp>
    outputs = [self.emb_model(in_ids.to(self.device)) for in_ids in inputs]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/transformers/modeling_bert.py", line 806, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/transformers/modeling_utils.py", line 248, in get_extended_attention_mask
    input_shape, attention_mask.shape
ValueError: Wrong shape for input_ids (shape torch.Size([18])) or attention_mask (shape torch.Size([18]))
```

I'd love it if anyone could tell me if I could replace a line in the code somewhere to fix this.

Here's a sample of that code region where that error occurs:

```
def get_embed_list(self, sent_pair):
		if self.emb_model is not None:
			sent_ids = [self.tokenizer.convert_tokens_to_ids(x) for x in sent_pair]
			inputs = [self.tokenizer.prepare_for_model(sent, return_token_type_ids=True, return_tensors='pt')['input_ids'] for sent in sent_ids]

			outputs = [self.emb_model(in_ids.to(self.device)) for in_ids in inputs]
			# use vectors from layer 8
			vectors = [x[2][self.layer].cpu().detach().numpy()[0][1:-1] for x in outputs]
			return vectors
		else:
			return None
```

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

I'm trying to use simalign in conjunction with finding the probabilities of a word in a sentence given its position in that sentence

(meaning I want to assess the probability of the word 'to' and 'too' in this sentence: "I went to the store"

Given a function like this:

```
find_probability_of_word_in_given_sentence('to', f'I went {given_word} the store')
find_probability_of_word_in_given_sentence('too', f'I went {given_word} the store')
```

I'd want an output like this:
```
to: 0.849283
too: 0.021412
```

And I don't know if there's a way to do that with transformers version 2.3.0, since it does not have the `fill-mask` feature (although that feature does not return word probabilities for a given word, but word predictions.

## To reproduce

Steps to reproduce the behavior:

1. Install simalign
2. Upgrade transformers
3. Run simalign example here: https://github.com/cisnlp/simalign/blob/master/examples/align_example.py

## Expected behavior

Simalign works as it does with transformers 2.3.0, returns a list of tuples with numbers.


Longformer inference time

The longformer-base-4096 model should be faster than bert-base-cased ,However I find that the former one takes more time to inference


[docs] add testing documentation

This PR adds the initial version of the testing doc.

* This is partially based on the work I did for [fastai last year](https://fastai1.fast.ai/dev/test.html), but rewritten to match the `transformers` environment. It's full of useful tips and tools for running tests. it's very useful for those who do a lot of testing.
* then adding most of the helpers in `testing_utils.py`
* adding CI information

More work is surely needed, but this is a start.

Thanks to @sshleifer for the detailed info on CIs.

@sgugger


[logging] remove no longer needed verbosity override

As the project's verbosity level is now at `logging.WARN` this PR removes the now redundant code.

p.s. another change probably is needed to switch from `logging.getLogger()` to the new API

```
grep -Ir getLogger tests examples | wc -l
56
```


[examples testing] restore code

For some reason https://github.com/huggingface/transformers/pull/5512 re-added temp dir creation code that was removed by
https://github.com/huggingface/transformers/pull/6494 in the process undoing what the latter PR did those tests, leading to the temp dir created twice in a row.

I see now that it was an older PR that was committed much later, so that explains why the new code was not noticed.

@Joel-hanson


broken pypi scipy package that affects tests under `examples`

In case someone runs into this:
```
PYTHONPATH="src" pytest examples/test_examples.py
ImportError while importing test module '/mnt/nvme1/code/huggingface/transformers-examples/examples/test_examples.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/stas/anaconda3/envs/main-38/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
examples/test_examples.py:38: in <module>
    import run_glue
examples/text-classification/run_glue.py:30: in <module>
    from transformers import (
E   ImportError: cannot import name 'glue_compute_metrics' from 'transformers' (/mnt/nvme1/code/huggingface/transformers-examples/src/transformers/__init__.py)
```
Looking into the code, the problem was coming from:
```
if is_sklearn_available():
    from .data import glue_compute_metrics, xnli_compute_metrics
```
for some reason, it was returning `False`.

Looking deeper, I got here:
```
try:
    from sklearn.metrics import f1_score, matthews_corrcoef

    from scipy.stats import pearsonr, spearmanr

    _has_sklearn = True
except (AttributeError, ImportError):
    _has_sklearn = False

def is_sklearn_available():
    return _has_sklearn
```
So next trying:
```
python -c "from sklearn.metrics import f1_score, matthews_corrcoef"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/sklearn/__init__.py", line 80, in <module>
    from .base import clone
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/sklearn/base.py", line 21, in <module>
    from .utils import _IS_32BIT
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/sklearn/utils/__init__.py", line 20, in <module>
    from scipy.sparse import issparse
ModuleNotFoundError: No module named 'scipy.sparse'
```
but the requirement was already installed:

```
pip install scipy
```
A search gave this answer: https://stackoverflow.com/a/59692528/9201239 which solved the problem.
```
pip uninstall scipy
conda install scipy
```
So there is something wrong with that `scipy` package on pypi.


Create README.md

model card

Related to pretrained model cards

Model card for PEGASUS model finetuned for paraphrasing task

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Trying to speed up lost speed of tokenizer.encode

Fixes #6962 (at least tries to fix)

@mfuntowicz


Create README.md

Related to pretrained model cards

Create model card for Pegasus QA

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


fix bug in pegasus converter

<!-- This line specifies which issue to close after the pull request is merged. -->
Reported https://discuss.huggingface.co/t/pegasus-questions/838/14

The bug was that I was passing `**cfg_updates` instead of `**cfg_kwargs`. Then I did some extra cleanup to make the code refer to bart less.


Update convert_pegasus_tf_to_pytorch.py

Instead of full config we were only sending updated config dict.
“sshleifer/pegasus” to “google/pegasus-aeslc” in Tokenizer

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


needing area to put download/convert/eval scripts

# 🚀 Feature request

Would it be useful to allocate a sub-dir in the source code for conversion/eval bash scripts? Some of them are quite complex including a bunch of downloads, moving files around, etc. It'd be good to have those in the repo, so that it'd be easy to re-build data if there was a change/mistake/etc.

note: I'm not proposing to move `src/transformers/convert*py`.

Let the data speak for itself.

I currently have 2 scripts for fairseq transformer models:

```
# Convert fairseq transform wmt19 checkpoint.

# To convert run:
# assuming the fairseq data is under data/wmt19.ru-en.ensemble, data/wmt19.en-ru.ensemble, etc

export ROOT=/code/huggingface/transformers-fair-wmt
cd $ROOT
mkdir data

# get data (run once)
wget https://dl.fbaipublicfiles.com/fairseq/models/wmt19.en-de.joined-dict.ensemble.tar.gz
wget https://dl.fbaipublicfiles.com/fairseq/models/wmt19.de-en.joined-dict.ensemble.tar.gz
wget https://dl.fbaipublicfiles.com/fairseq/models/wmt19.en-ru.ensemble.tar.gz
wget https://dl.fbaipublicfiles.com/fairseq/models/wmt19.ru-en.ensemble.tar.gz
tar -xvzf wmt19.en-de.joined-dict.ensemble.tar.gz
tar -xvzf wmt19.de-en.joined-dict.ensemble.tar.gz
tar -xvzf wmt19.en-ru.ensemble.tar.gz
tar -xvzf wmt19.ru-en.ensemble.tar.gz

# run conversions and uploads

export PAIR=ru-en
PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/wmt19.$PAIR.ensemble --pytorch_dump_folder_path data/fsmt-wmt19-$PAIR

export PAIR=en-ru
PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/wmt19.$PAIR.ensemble --pytorch_dump_folder_path data/fsmt-wmt19-$PAIR

export PAIR=de-en
PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/wmt19.$PAIR.joined-dict.ensemble --pytorch_dump_folder_path data/fsmt-wmt19-$PAIR

export PAIR=en-de
PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/wmt19.$PAIR.joined-dict.ensemble --pytorch_dump_folder_path data/fsmt-wmt19-$PAIR

# upload
cd data
transformers-cli upload -y fsmt-wmt19-ru-en
transformers-cli upload -y fsmt-wmt19-en-ru
transformers-cli upload -y fsmt-wmt19-de-en
transformers-cli upload -y fsmt-wmt19-en-de
cd -

# if updating just small files and not the large models, here is a script to generate the right commands:
perl -le 'for $f (@ARGV) { print qq[transformers-cli upload -y $_/$f --filename $_/$f] for map { "fsmt-wmt19-$_" } ("en-ru", "ru-en", "de-en", "en-de")}' vocab-src.json vocab-tgt.json tokenizer_config.json config.json
# add/remove files as needed
```

Eval script:
```
# to match fairseq you need to set num_beams=50 in `configuration_fsmt.py` and lower BS

# quick estimate version for quick testing

export PAIR=en-ru
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
export NUM_BEAMS=8
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src | head -100 > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref | head -100 > $DATA_DIR/val.target
echo $PAIR
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py stas/fsmt-wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

# ru-en

export PAIR=ru-en
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
export NUM_BEAMS=50
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py stas/fsmt-wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

# (expected BLEU: 41.3 http://matrix.statmt.org/matrix/output/1907?run_id=6937)

# en-ru

export PAIR=en-ru
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
export NUM_BEAMS=50
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
echo $PAIR
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py stas/fsmt-wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

# (expected BLEU: 36.4 http://matrix.statmt.org/matrix/output/1914?score_id=37605)

# en-de

export PAIR=en-de
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
echo $PAIR
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py stas/fsmt-wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

# (expected BLEU: 43.1 http://matrix.statmt.org/matrix/output/1909?run_id=6862)

# de-en

export PAIR=de-en
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
export NUM_BEAMS=50
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
echo $PAIR
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py stas/fsmt-wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

# (expected BLEU: 42.3 http://matrix.statmt.org/matrix/output/1902?run_id=6750)

```

Then I have a different script for 2 sets of other models for wmt from allen nlp, with 2 scripts each:

```

# Convert fairseq transform wmt16 en-de checkpoints from https://github.com/jungokasai/deep-shallow

pip install gdown

# get data (run once)

cd data
gdown 'https://drive.google.com/uc?id=1x_G2cjvM1nW5hjAB8-vWxRqtQTlmIaQU'
gdown 'https://drive.google.com/uc?id=1oA2aqZlVNj5FarxBlNXEHpBS4lRetTzU'
gdown 'https://drive.google.com/uc?id=1Wup2D318QYBFPW_NKI1mfP_hXOfmUI9r'
tar -xvzf trans_ende_12-1_0.2.tar.gz
tar -xvzf trans_ende-dist_12-1_0.2.tar.gz
tar -xvzf trans_ende-dist_6-1_0.2.tar.gz

gdown 'https://drive.google.com/uc?id=1mNufoynJ9-Zy1kJh2TA_lHm2squji0i9'
gdown 'https://drive.google.com/uc?id=1iO7um-HWoNoRKDtw27YUSgyeubn9uXqj'
tar -xvzf wmt16.en-de.deep-shallow.dist.tar.gz
tar -xvzf wmt16.en-de.deep-shallow.tar.gz

cp wmt16.en-de.deep-shallow/data-bin/dict.*.txt trans_ende_12-1_0.2
cp wmt16.en-de.deep-shallow.dist/data-bin/dict.*.txt trans_ende-dist_12-1_0.2
cp wmt16.en-de.deep-shallow.dist/data-bin/dict.*.txt trans_ende-dist_6-1_0.2
cp wmt16.en-de.deep-shallow/bpecodes trans_ende_12-1_0.2
cp wmt16.en-de.deep-shallow.dist/bpecodes trans_ende-dist_12-1_0.2
cp wmt16.en-de.deep-shallow.dist/bpecodes trans_ende-dist_6-1_0.2

# another set wmt19-6-6-de-en
gdown 'https://drive.google.com/uc?id=1j6z9fYdlUyOYsh7KJoumRlr1yHczxR5T'
gdown 'https://drive.google.com/uc?id=1yT7ZjqfvUYOBXvMjeY8uGRHQFWoSo8Q5'
gdown 'https://drive.google.com/uc?id=15gAzHeRUCs-QV8vHeTReMPEh1j8excNE'
tar -xvzf wmt19.de-en.tar.gz
tar -xvzf wmt19_deen_base_dr0.1_1.tar.gz
tar -xvzf wmt19_deen_big_dr0.1_2.tar.gz
cp wmt19.de-en/data-bin/dict.en.txt wmt19_deen_base_dr0.1_1
cp wmt19.de-en/data-bin/dict.en.txt wmt19_deen_big_dr0.1_2
cp wmt19.de-en/data-bin/dict.de.txt wmt19_deen_base_dr0.1_1
cp wmt19.de-en/data-bin/dict.de.txt wmt19_deen_big_dr0.1_2

cd -

# run conversions and uploads

# wmt16-en-de set

PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/trans_ende-dist_12-1_0.2 --pytorch_dump_folder_path data/fsmt-wmt16-en-de-dist-12-1

PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/trans_ende-dist_6-1_0.2 --pytorch_dump_folder_path data/fsmt-wmt16-en-de-dist-6-1

PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/trans_ende_12-1_0.2 --pytorch_dump_folder_path data/fsmt-wmt16-en-de-12-1

# wmt19-de-en set

PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/wmt19_deen_base_dr0.1_1 --pytorch_dump_folder_path data/fsmt-wmt19-de-en-6-6-base

PYTHONPATH="src" python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path data/wmt19_deen_big_dr0.1_2 --pytorch_dump_folder_path data/fsmt-wmt19-de-en-6-6-big
```
Eval:
```

git clone https://github.com/huggingface/transformers
cd transformers
export PAIR=en-de
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=64
export NUM_BEAMS=5
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target

MODEL_PATH=/code/huggingface/transformers-fair-wmt/data/fsmt-wmt16-en-de-dist-12-1
echo $PAIR $MODEL_PATH
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py $MODEL_PATH $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

MODEL_PATH=/code/huggingface/transformers-fair-wmt/data/fsmt-wmt16-en-de-dist-6-1
echo $PAIR $MODEL_PATH
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py $MODEL_PATH $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

MODEL_PATH=/code/huggingface/transformers-fair-wmt/data/fsmt-wmt16-en-de-12-1
echo $PAIR $MODEL_PATH
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py $MODEL_PATH $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

# wmt19-de-en set

export PAIR=de-en
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=64
export NUM_BEAMS=5
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target

MODEL_PATH=/code/huggingface/transformers-fair-wmt/data/fsmt-wmt19-de-en-6-6-base
echo $PAIR $MODEL_PATH
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py $MODEL_PATH $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

MODEL_PATH=/code/huggingface/transformers-fair-wmt/data/fsmt-wmt19-de-en-6-6-big
echo $PAIR $MODEL_PATH
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py $MODEL_PATH $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS

```

So perhaps:
```
model_scripts/
    arch/
         model1-build.sh
         model1-eval.sh
         model2-build.sh
         model2-eval.sh
   [...]
```
So in the case of the above scripts, they could be:
```
model_scripts/fsmt/fairseq-build.sh
model_scripts/fsmt/fairseq-eval.sh
model_scripts/fsmt/allennlp-build.sh
model_scripts/fsmt/allennlp-eval.sh
```
Thoughts?

Of course, I could just start with this proposal as a PR and we can adjust from there.

Thank you.

([fsmt](https://github.com/huggingface/transformers/pull/6940) is not yet merged, in case you wonder about an unfamiliar name)


is config argument necessary for XXModel.from_pretrained method? And when is needed?

# ❓ Questions & Help

I find the model can be loaded with `XXModel.from_pretrained` function even if no `config` argument is given. But if the `config` is given, the argument `gradient_checkpointing` cannot be enabled. I wonder why the config argument can lead to the unable of `gradient_checkpointing`???

I test the conclusion with longformer model. Thanks!


## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-3.10.0_3-0-0-17-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

 Longformer/Reformer: @patrickvonplaten

 -->

## Information

Model I am using (Longformer):

The problem arises when using:
[ * ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SQuAD v1.1

## To reproduce

Steps to reproduce the behavior:

1. add gradient_checkpointing argument for AutoModelForQuestionAnswering in examples/question-answering/run_squad.py
2. run with longformer-base-4096

runtime errors as belows
```
File "run_squad.py", line 821, in <module>
    main()
  File "run_squad.py", line 739, in main
    gradient_checkpointing=True,
  File "/opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py", line 852, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'gradient_checkpointing'
```

even if you explicitly replace AutoModel with LongformerModel like following, error is the same.

![image](https://user-images.githubusercontent.com/24541791/92999722-d1da6e80-f555-11ea-9588-2269fcacee90.png)

When I run with python/ipython interactive mode, the model loaded success.

![image](https://user-images.githubusercontent.com/24541791/92999756-0e0dcf00-f556-11ea-8845-8a661dab1e08.png)

And I have ensured the python enviroment is the same during these two runs.

## Expected behavior

Longformer can be loaded with gradient checkpointing


German electra model card v3 update

Related to pretrained model cards

Model Card Update


train/eval step results log not shown in terminal for tf_trainer.py

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-42-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
  -->
 Trainer: @sgugger
 tensorflow: @jplu
@LysandreJik

## Information

In the current code, which is without setting `logger.setLevel(logging.INFO)` in `trainer_tf.py`:

	09/12/2020 03:42:41 - INFO - absl -   Load dataset info from /home/imo/tensorflow_datasets/glue/sst2/1.0.0
	09/12/2020 03:42:41 - INFO - absl -   Reusing dataset glue (/home/imo/tensorflow_datasets/glue/sst2/1.0.0)
	09/12/2020 03:42:41 - INFO - absl -   Constructing tf.data.Dataset for split validation, from /home/imo/tensorflow_datasets/glue/sst2/1.0.0
	2020-09-12 03:42:57.010229: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 41707 of 67349
	2020-09-12 03:43:03.412045: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
	2020-09-12 03:43:56.636791: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 36279 of 67349
	2020-09-12 03:44:04.474751: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
	09/12/2020 03:44:51 - INFO - __main__ -   *** Evaluate ***
	09/12/2020 03:45:02 - INFO - __main__ -   ***** Eval results *****
	09/12/2020 03:45:02 - INFO - __main__ -     eval_loss = 0.712074209790711
	09/12/2020 03:45:02 - INFO - __main__ -     eval_acc = 0.48977272727272725

You can see that the train/eval step logs are not shown.

If I specify, manually, `logger.setLevel(logging.INFO)` in `trainer_tf.py`:

	09/12/2020 06:04:39 - INFO - absl -   Load dataset info from /home/imo/tensorflow_datasets/glue/sst2/1.0.0
	09/12/2020 06:04:39 - INFO - absl -   Reusing dataset glue (/home/imo/tensorflow_datasets/glue/sst2/1.0.0)
	09/12/2020 06:04:39 - INFO - absl -   Constructing tf.data.Dataset for split validation, from /home/imo/tensorflow_datasets/glue/sst2/1.0.0
	You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.
	To use comet_ml logging, run `pip/conda install comet_ml` see https://www.comet.ml/docs/python-sdk/huggingface/
	***** Running training *****
	  Num examples = 67349
	  Num Epochs = 1
	  Instantaneous batch size per device = 4
	  Total train batch size (w. parallel, distributed & accumulation) = 4
	  Gradient Accumulation steps = 1
	  Steps per epoch = 4
	  Total optimization steps = 4
	2020-09-12 06:04:49.637373: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 39626 of 67349
	2020-09-12 06:04:56.805687: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
	{'loss': 0.6994307, 'learning_rate': 3.7499998e-05, 'epoch': 0.5, 'step': 1}
	{'loss': 0.6897122, 'learning_rate': 2.5e-05, 'epoch': 0.75, 'step': 2}
	Saving checkpoint for step 2 at ./sst-2/checkpoint/ckpt-1
	{'loss': 0.683386, 'learning_rate': 1.25e-05, 'epoch': 1.0, 'step': 3}
	{'loss': 0.68290234, 'learning_rate': 0.0, 'epoch': 1.25, 'step': 4}
	Saving checkpoint for step 4 at ./sst-2/checkpoint/ckpt-2
	Training took: 0:00:43.099437
	Saving model in ./sst-2/
	09/12/2020 06:05:26 - INFO - __main__ -   *** Evaluate ***
	***** Running Evaluation *****
	  Num examples = 872
	  Batch size = 8
	{'eval_loss': 0.6990196158032899, 'eval_acc': 0.49204545454545456, 'epoch': 1.25, 'step': 4}
	09/12/2020 06:05:35 - INFO - __main__ -   ***** Eval results *****
	09/12/2020 06:05:35 - INFO - __main__ -     eval_loss = 0.6990196158032899
	09/12/2020 06:05:35 - INFO - __main__ -     eval_acc = 0.49204545454545456

We see more information like

    {'loss': 0.6994307, 'learning_rate': 3.7499998e-05, 'epoch': 0.5, 'step': 1}

More importantly, we also see this message

	You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.
	To use comet_ml logging, run `pip/conda install comet_ml` see https://www.comet.ml/docs/python-sdk/huggingface/

, which won't be shown if logging level is not set to INFO.

## Related
In the PR #6097, @LysandreJik changed `logger.info(output)` to `print(output)` in `trainer.py` in order to show logs on the screen.
Maybe we should do the same thing for `tf_trainer.py`. If not, could we set logging level to INFO in `tf_trainer.py` - however this would become different from `trainer.py` where the logging level is not set (at least, not in the trainer script).

## To reproduce

    python3 run_tf_glue.py \
    --task_name sst-2 \
    --model_name_or_path distilbert-base-uncased \
    --output_dir ./sst-2/ \
    --max_seq_length  16 \
    --num_train_epochs 2 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --max_steps 4 \
    --logging_steps 1 \
    --save_steps 2 \
    --seed 1 \
    --do_train \
    --do_eval \
    --do_predict \
    --overwrite_output_dir

## Expected behavior

I expect the train/eval step logs will be shown on the screen.

## Remark

I can make a PR once a decision is made by the team.


Transformer-XL: Remove unused parameters

Fixes #6943

Since `tgt_len` and `ext_len` is removed, the method `reset_length` was renamed to `reset_memory_length` to be more meaningful.

I'm not sure whether a deprecation warning should be included in `TransfoXLConfig`. Maybe someone can make suggestions regarding this.


Longformer run error

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
When I train on a classification model by Longformer
def forward(self,input):
        embding=input['enc']
        att_mask=input['mask']
        att_mask[:,[100,300,500,800,1200,]]=2
        labels=input['targets']
        print('jeff:',embding.device,att_mask.device,self.l1.device,embding.shape,att_mask.shape,self.maxlen)

        logit=self.l1(inputs_embeds=embding,attention_mask=att_mask)#[:2]
        return [logit,labels]

Meet error:
pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [205,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed

max_num_extra_indices_per_batch = num_extra_indices_per_batch.max()
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCReduceAll.cuh:327

I checked the length of the attention_mask is same as the config.max_len which is like [bs,max_len]
Do anyone meet the same issue?
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Distilbart's summaries start with an empty space?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.51, gpu
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help

 @sshleifer

Steps to reproduce the behavior:

```
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('sshleifer/distilbart-xsum-12-3')
tokenizer = BartTokenizer.from_pretrained('sshleifer/distilbart-xsum-12-3')
ARTICLE_TO_SUMMARIZE = "\"The accident meant the motorway was closed, making travel to Mourneview Park impossible for the team and fans travelling from Belfast, \" said the Irish Football Association . A new date for the match has yet to be confirmed by Uefa . Northern Ireland have three points from their first two Group Six qualifiers."
inputs = tokenizer.batch_encode_plus([ARTICLE_TO_SUMMARIZE], max_length=512, return_tensors='pt')
summary_ids = model.generate(inputs['input_ids'], num_beams=5, max_length=62, min_length=10, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in summary_ids])
```
[" The Republic of Ireland's Euro 2016 qualifier against Northern Ireland has been postponed because of a motorway crash."]

facebook/large-bart-xsum will give a correct summary. Why this summary starts with an empty space? What shall we pay attention to these inconsitency


How to implement LayoutLM for information extraction

Hi,
I am new to nlp
Can someone please guide me on How to implement the layoutLM using transformers for information extraction (from images like receipt)

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-large-uncased")
model = AutoModel.from_pretrained("microsoft/layoutlm-large-uncased")


SqueezeBERT architecture

Related to pretrained model cards

# Burn-down list:

Here an overview of the general workflow:

- [x] Add model/configuration/tokenization classes.
- [ ] ~~Add conversion scripts.~~ This model was originally developed in PyTorch.
- [x] Add tests and a @slow integration test.
- [x] Document your model.
- [x] Finalize.

Let's detail what should be done at each step.

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] Copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model
  name.
- [x] Edit the files to replace `XXX` (with various casing) with your model name.
- [x] Copy-paste or create a simple configuration class for your model in the `configuration_...` file.
- [x] Copy-paste or create the code for your model in the `modeling_...` files (PyTorch ~~and TF 2.0~~).
- [x] Copy-paste or create a tokenizer class for your model in the `tokenization_...` file.

### loose ends:
- [ ] support for head_mask, encoder_hidden_states, encoder_attention_mask (Not planning to do for this PR)
- [x] Make sure finetuning works.

## ~~Adding conversion scripts~~

Here is the workflow for the conversion scripts:

- [ ] ~~Copy the conversion script (`convert_...`) from the present folder to the main folder.~~
- [ ] ~~Edit this script to convert your original checkpoint weights to the current pytorch ones.~~

## Adding tests:

Here is the workflow for the adding tests:

- [ ] ~~Copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main
  folder and rename them, replacing `xxx` with your model name.~~
- [ ] ~~Edit the tests files to replace `XXX` (with various casing) with your model name.~~
- [ ] ~~Edit the tests code as needed.~~
- [x] Create tests, using the DistilBERT tests as a starting point
    - [x] `test_modeling_squeezebert.py` (based on `test_modeling_distilbert.py`)
    - [x] `test_tokeization_squeezebert.py` (based on `test_tokenization_distilbert.py`)

## Documenting your model:

Here is the workflow for documentation:

- [x] Make sure all your arguments are properly documented in your configuration and tokenizer.
- [x] Most of the documentation of the models is automatically generated, you just have to make sure that
  `XXX_START_DOCSTRING` contains an introduction to the model you're adding and a link to the original
  article and that `XXX_INPUTS_DOCSTRING` contains all the inputs of your model.
- [x] Create a new page `xxx.rst` in the folder `docs/source/model_doc` and add this file in `docs/source/index.rst`.

Make sure to check you have no sphinx warnings when building the documentation locally and follow our
[documentation guide](https://github.com/huggingface/transformers/tree/master/docs#writing-documentation---specification).

## Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] Add import for all the relevant classes in `__init__.py`.
- [x] Add your configuration in `configuration_auto.py`.
- [x] Add your PyTorch and ~~TF 2.0~~ model respectively in `modeling_auto.py` ~~and `modeling_tf_auto.py`~~.
- [x] Add your tokenizer in `tokenization_auto.py`.
- [ ] ~~Add a link to your conversion script in the main conversion utility (in `commands/convert.py`)~~
- [ ] ~~Edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py`
  file.~~
- [x] Add a mention of your model in...
    - [x] `README.md`
    - [x] `docs/source/index.rst`
    - [x] `docs/source/pretrained_models.rst`.
- [x] Upload the vocabulary files, configurations, and pretrained weights .
    - squeezebert-uncased
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-uncased/vocab.txt
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-uncased/pytorch_model.bin
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-uncased/config.json
    - squeezebert-mnli-headless
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli-headless/vocab.txt
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli-headless/pytorch_model.bin
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli-headless/config.json
    - squeezebert-mnli
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli/vocab.txt
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli/pytorch_model.bin
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli/config.json
- [x] Create model card(s) for your models on huggingface.co (These go in the repo, not in the file upload).

For those last two steps, check the [model sharing documentation](https://huggingface.co/transformers/model_sharing.html).

- [ ] Delete these files (I uploaded these to the wrong directories)
    - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert/squeezebert-uncased-vocab.txt
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-uncased-config.json
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-uncased.bin
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-mnli.bin
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-mnli-headless.bin


Longformer output_hidden_states=True outputs sequence length=512 for all inputs of different lengths

Can someone explain why cc[2][0].shape != dd[2][0].shape?

>>> from transformers import LongformerModel, RobertaModel
>>> import torch
>>> from transformers import RobertaTokenizer
>>> tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
>>> aa = RobertaModel.from_pretrained('roberta-base', return_dict=True)
>>> bb = LongformerModel.from_pretrained('allenai/longformer-large-4096', return_dict=True)
Some weights of LongformerModel were not initialized from the model checkpoint at allenai/longformer-large-4096 and are newly initialized: ['longformer.embeddings.position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> cc = aa(**inputs, output_hidden_states=True)
>>> cc[2][0].shape
torch.Size([1, 8, 768])
>>> dd = bb(**inputs, output_hidden_states=True)
>>> dd[2][0].shape
torch.Size([1, 512, 1024])
>>> dd[0].shape
torch.Size([1, 8, 1024])


Clean up autoclass doc

Clean up and standardize the documentation of the auto classes.