Issue with fp16_opt_level default

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-1091-oem-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, 1 GPU
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 tensorflow: @jplu
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): BART @sshleifer

The problem arises when using:
* [ ] the official example scripts:
* [x] my own modified scripts: Running a modified version of finetune.py (via finetune.sh) that specifies "bad word tokens" that BART is prohibited from generating

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task:
* [x] my own task or dataset: Finetuning on a question understanding dataset

## To reproduce

Steps to reproduce the behavior:

1. Run finetune.sh using a GPU with fp16 flag and default settings, instructing the model to perform testing. See set-up below:
Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic

2. At the start of testing, an error related to amp will be encountered:

```python
Traceback (most recent call last):
  File "finetune_prohibition.py", line 444, in <module>
    main(args)
  File "finetune_prohibition.py", line 433, in main
    trainer.test()
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1281, in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in __test_using_best_weights
    results = self.fit(model)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1003, in fit
    results = self.single_gpu_train(model)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 182, in single_gpu_train
    model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py", line 1006, in configure_apex
    model, optimizers = amp.initialize(model, optimizers, opt_level=amp_level)
  File "/home/lib/python3.6/site-packages/apex/amp/frontend.py", line 358, in initialize
    return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
  File "/home/lib/python3.6/site-packages/apex/amp/_initialize.py", line 171, in _initialize
    check_params_fp32(models)
  File "/home/lib/python3.6/site-packages/apex/amp/_initialize.py", line 87, in check_params_fp32
    name, param.type()))
  File "/home/lib/python3.6/site-packages/apex/amp/_amp_state.py", line 32, in warn_or_err
    raise RuntimeError(msg)
RuntimeError: Found param model.model.shared.weight with type torch.cuda.HalfTensor, expected torch.cuda.FloatTensor.
When using amp.initialize, you do not need to call .half() on your model
before passing it, no matter what optimization level you choose.
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
When using the new default of fp16_opt_level=O2, this error is encountered - using fp16_opt_level=O1 solves the issue. I'm unsure if this problem is just related to my machine, and would be interested to see whether others can recreate the problem. Although I used modified scripts, the modifications are simple and shouldn't interact with torch.cuda.HalfTensor classes differently. Let me know if you'd like me to try to recreate this issue on an official GLUE/SQUaD task or if you need any other information from me.


Cannot fine tune my distilbart-cnn-12-6 model because of cuda memory

I'm trying to fine tune my model like this:

```
import os
os.environ['PYTHONPATH'] += ":/content/transformers/examples"
%cd "/content/transformers/examples"

!python /content/transformers/examples/seq2seq/finetune.py \
    --learning_rate=3e-5 \
    --fp16 \
    --gpus 1 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.1 \
    --sortish_sampler \
    --data_dir '/content/dataset' \
    --train_batch_size=1\
    --eval_batch_size=1\
    --output_dir=distilbart_multi_news \
    --num_train_epochs 1 \
    --model_name_or_path /content/model/best_tfmr
```

But even with a batch size of 1 I get this error:
 File "/content/transformers/examples/seq2seq/finetune.py", line 344, in <module>
    main(args)
  File "/content/transformers/examples/seq2seq/finetune.py", line 322, in main
    logger=logger,
  File "/content/transformers/examples/lightning_base.py", line 330, in generic_train
    trainer.fit(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 918, in fit
    self.single_gpu_train(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/distrib_parts.py", line 176, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 1076, in run_pretrain_routine
    False)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 279, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 452, in evaluation_forward
    output = model.validation_step(*args)
  File "/content/transformers/examples/seq2seq/finetune.py", line 136, in validation_step
    return self._generative_step(batch)
  File "/content/transformers/examples/seq2seq/finetune.py", line 163, in _generative_step
    generated_ids = self.model.generate(input_ids=source_ids, attention_mask=source_mask, use_cache=True,)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 459, in generate
    model_specific_kwargs=model_specific_kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 638, in _generate_beam_search
    outputs = self(**model_inputs)  # (batch_size * num_beams, cur_len, vocab_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bart.py", line 1005, in forward
    lm_logits = F.linear(outputs[0], self.model.shared.weight, bias=self.final_logits_bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1676, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA out of memory. Tried to allocate 100.00 MiB (GPU 0; 11.17 GiB total capacity; 10.59 GiB already allocated; 91.81 MiB free; 10.66 GiB reserved in total by PyTorch)

Any idea what to do?


Update model card

model card

Related to pretrained model cards


cleanup torch unittests

a large group of tests has been modified according to the request in https://github.com/huggingface/transformers/issues/5973

If this is what's needed then just running this magic perl sequence should take care of most of them:

```
perl -pi -e 's|^\s+self.check_loss_output\(result\)\n||' tests/test_modeling_bert.py

perl -0777 -pi -e 's|^\s+def check_loss_output\(self, result\):[\s\n]+       self.parent.assertListEqual\(list\(result\["loss"\].size\(\)\), \[]\)\s*\n|\n|msg' tests/test_modeling_bert.py

perl -0777 -pi -e 's#self.parent.assertListEqual\(
[\s\n]*
list\((result\w*)\[" ([^"]+) "\].(?:shape|size\(\))\),[\s\n]+\[ ( [^\]]* ) \],?
[\s\n]*
\)
#self.parent.assertEqual($1.$2.shape, ($3))#xmsg' tests/test_modeling_bert.py
```
(edit, adjusted for multiple various inputs)

well, add:
```
make style
```
to fix the style.

problem: not all results are objects, some are plain `dict` and can't be called with .key_name. See my comment below.

@sshleifer


Encoder decoder config docs

As discussed on #5826, this PR adds more details on how to load encoder/decoder config objects from pretrained folders and how to instantiate encoder_decoder pretrained models given their corresponding configuration objects (useful for loading pre-trained models and modifying some config members for fine-tuning).


longformertokenizerFast gives error

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 tensorflow: @jplu
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ X ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X ] my own task or dataset: (give details below)

## To reproduce

using LongformerTokenizerFast gives error. but using LongformerTokenizer works without any issues keeping everything same

```

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-39-263240bbee7e> in <module>
----> 1 main()

<ipython-input-37-2c27a8a4db79> in main()
     99     )
    100
--> 101     train_dataset = CustomDataset(data_args, tokenizer=tokenizer) if training_args.do_train else None
    102     eval_dataset = CustomDataset(data_args, tokenizer=tokenizer, mode="test") if training_args.do_eval else None
    103

<ipython-input-36-85278feb74ec> in __init__(self, args, tokenizer, limit_length, mode)
    184                     max_length=args.max_seq_length,
    185                     label_list=label_list,
--> 186                     output_mode=self.output_mode,
    187                 )
    188                 start = time.time()

/opt/conda/lib/python3.7/site-packages/transformers/data/processors/glue.py in glue_convert_examples_to_features(examples, tokenizer, max_length, task, label_list, output_mode)
     63         return _tf_glue_convert_examples_to_features(examples, tokenizer, max_length=max_length, task=task)
     64     return _glue_convert_examples_to_features(
---> 65         examples, tokenizer, max_length=max_length, task=task, label_list=label_list, output_mode=output_mode
     66     )
     67

/opt/conda/lib/python3.7/site-packages/transformers/data/processors/glue.py in _glue_convert_examples_to_features(examples, tokenizer, max_length, task, label_list, output_mode)
    133         max_length=max_length,
    134         padding="max_length",
--> 135         truncation=True,
    136     )
    137

/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in __call__(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_pretokenized, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   1918                 return_length=return_length,
   1919                 verbose=verbose,
-> 1920                 **kwargs,
   1921             )
   1922         else:

/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding, truncation, max_length, stride, is_pretokenized, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2103             return_length=return_length,
   2104             verbose=verbose,
-> 2105             **kwargs,
   2106         )
   2107

/opt/conda/lib/python3.7/site-packages/transformers/tokenization_gpt2.py in _batch_encode_plus(self, *args, **kwargs)
    385         )
    386
--> 387         return super()._batch_encode_plus(*args, **kwargs)
    388
    389     def _encode_plus(self, *args, **kwargs) -> BatchEncoding:

/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py in _batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_pretokenized, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    378         else:
    379             encodings = self._tokenizer.encode_batch(
--> 380                 batch_text_or_text_pairs, add_special_tokens=add_special_tokens, is_pretokenized=is_pretokenized
    381             )
    382

/opt/conda/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py in encode_batch(self, inputs, is_pretokenized, add_special_tokens)
    247             raise ValueError("encode_batch: `inputs` can't be `None`")
    248
--> 249         return self._tokenizer.encode_batch(inputs, is_pretokenized, add_special_tokens)
    250
    251     def decode(self, ids: List[int], skip_special_tokens: Optional[bool] = True) -> str:

Exception: Truncation error: Specified max length is too low to respect the various constraints
```


Some weights not initialized in pre-trained RobertaForMaskedLM

The bug is similar to #2202.

I am trying to evaluate MLM perplexity (without training/finetuning) using Roberta with `run_language_modeling.py` (from the [official example](https://github.com/huggingface/transformers/tree/master/examples/language-modeling)). However, some weights seems to be reinitialized instead of getting loading from the pretrained Roberta checkpoint.

## To Reproduce (~~with master branch~~):

```
import logging
logging.basicConfig(level=logging.INFO)
from transformers import RobertaForMaskedLM
_ = RobertaForMaskedLM.from_pretrained('roberta-base')
```

It gives the following warning message:
```
WARNING:transformers.modeling_utils:Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at roberta-base and are newly initialized: ['roberta.embeddings.position_ids', 'lm_head.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The perplexities I get on direct evaluation on Wikitext-2/103 datasets are also much higher than the official Roberta implementation from fairseq. I suspect this could be the reason.


GPT2 crashing at loss.backward()

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Ubuntu
- Python version: 3.6
- PyTorch version (GPU?): 1.5.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

@LysandreJik

## Information

Trying to finetune GPT2 model but the GPU is crashing after `loss.backward()`. I thought it might be just my code but I ran some different code involving finetuning GPT2 and that as well crashed in the same manner.

Getting this warning as well.
```
WARNING - transformers.modeling_utils -   Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

A week or 2 back, everything was working fine but now the same code is crashing on `loss.backward()`.


How to integrate the Pyro module with HuggingFace Transformers?

Hello,
I am trying to convert the HuggingFace Transformer into a Bayesian neural network by using the `Pyro` module.
I provided my code below. Everything works well except I am stuck at the line `svi_loss = svi.step(input_ids = input_ids, attention_mask = attention_mask, labels = label)`. At that line an error is generated, because after converting the HuggingFace Transformer into a Pyro model, the new model does not have any set parameter (since it is a Bayesian model...so the weights for a Pyro model are not fixed, meaning the weights are sampled from a statistical distribution). Is there any way that I can get around this issue? I have also posted the similar question on Pyro forum. Thank you,

CODE:

```python
import torch
from torch import distributions
from transformers import RobertaTokenizer, RobertaForMultipleChoice, AdamW, get_constant_schedule
import pyro
import pyro.infer
import pyro.optim
import pyro.distributions as dist
import pyro.nn.module as module
import pyro.infer.autoguide.guides as guides
from torch import nn
from pyro.optim import Adam
from pyro.infer import SVI
from pyro.infer import Trace_ELBO
from pyro.infer import Predictive

# get the pre-trained HuggingFace RobertaForMultipleChoice and resize the token embeddings
# after adding the special token
model_RobertaForMultipleChoice = RobertaForMultipleChoice.from_pretrained('Roberta-base')

# convert the HuggingFace model into a pyro model
module.to_pyro_module_(model_RobertaForMultipleChoice)

for m in model_RobertaForMultipleChoice.modules():
      for name, value in list(m.named_parameters(recurse=False)):
                setattr(m, name, module.PyroSample(prior=dist.Normal(0, 1)
                                             .expand(value.shape)
                                             .to_event(value.dim())))

# define parameters for training
guide_delta = guides.AutoDelta(model_RobertaForMultipleChoice)
optimizer_2 = Adam({"lr": 0.000000055})
scheduler_2 = pyro.optim.StepLR({'optimizer': optimizer_2, 'optim_args': {'lr': 0.000000055}})
svi_delta = SVI(model_RobertaForMultipleChoice, guide_delta, optimizer_2, loss=Trace_ELBO())

# training loop
for m in range(num_iter):

        # calculate the loss and take a gradient step for svi
        # ERRORS OCCUR HERE
        svi_loss = svi.step(input_ids = input_ids,
                          attention_mask = attention_mask,
                          labels = label)

        # update the with the calculated loss
        total_svi_loss = total_svi_loss + svi_loss

        if m % log_interval == 0 and m > 0:
            cur_svi_loss = total_svi_loss / log_interval
            print('| epoch {:3d} | {:5d}/{:5d} batches | lr {:02.9f} |
                   loss {:5.4f} | ppl {:8.4f}'.format(
                    epoch, m, int(num_lines_train/4), scheduler.get_lr()[0],
                    cur_svi_loss, math.exp(cur_svi_loss)))

            total_svi_loss = 0
```


Add support for truncation argument when calling a Pipeline

# 🚀 Feature request

Currently, only the `padding` argument [is supported](https://github.com/huggingface/transformers/blob/a39dfe4fb122c11be98a563fb8ca43b322e01036/src/transformers/pipelines.py#L500) when calling a pipeline, and it's not possible to pass `truncation` argument. For example, running the following code sample would raise an error:

```python
import transformers as trf

model = trf.pipeline(task='feature-extraction', model='bert-base-cased')
output = model('a sample text', padding=False, truncation=True)
```

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

If toggling padding is supported, then why truncation shouldn't be?

## Your contribution

I think to achieve this, same as `padding`, only a `truncation` argument should be added to `_parse_and_tokenize` method and also when calling the tokenizer. If that's the case, I would be willing to work on a PR.


Support new tokenizers in distillation example

I'm creating a distilled model based on a new transformers model, and needed these two lines from examples changed to make that process easier.

- Change filename of output binarized text vectors to replace '/' with '-'; for example tokenizer 'monsoon-nlp/hindi-bert' will output to a file and not create a new directory
- Load max_model_input_size / max_position_embeddings from teacher_config_class and not from a hardcoded list of common tokenizers in the tokenizer class


taeminlee/kogpt2 not working

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- None: hosted inference API

### Who can help
@LysandreJik
@julien-c
@TevenLeScao

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [O] the official example scripts: Hosted Inference API testing page
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [O] my own task or dataset: Text generation, or anything

## To reproduce

Steps to reproduce the behavior:

1. https://huggingface.co/taeminlee/kogpt2?text=제+이름은+홍길동
2. Or type any text
3. Model returns just the text

## Expected behavior

Generated text should be returned, but isn't.


add new model prophetnet

Related to pretrained model cards

# Add new model structure [ProphetNet](https://arxiv.org/abs/2001.04063).

## Description:

ProphetNet is a new pre-trained language model for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. ProphetNet is able to predict more future tokens with a n-stream decoder. The original implementation is Fairseq version at [github repo](https://github.com/microsoft/ProphetNet).
xProphetNet has the same model structure, but is pretrained with wikipedia 100 languages dataset as described in [xGLUE](https://arxiv.org/abs/2004.01401). xGLUE is a benchmark for corss-lingual NLU and NLG tasks. xProphetNet is also served as a baseline model for cross-lingual generation tasks in xGLUE NTG and QG.

## Usage:

Take xGLUE NTG task as an example:
Cross-lingual pretrained model is finetuned with English news title generation data, but inference with both English and other zero-shot language data.
A quick usage is like:
```
from transformers import ProphetNetTokenizer, ProphetNetForConditionalGeneration, ProphetNetConfig

model = ProphetNetForConditionalGeneration.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-ntg')
tokenizer = ProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-ntg')

EN_SENTENCE_TO_QUESTION = "Microsoft Corporation intends to officially end free support for the Windows 7 operating system after January 14, 2020, according to the official portal of the organization. From that day, users of this system will not be able to receive security updates, which could make their computers vulnerable to cyber attacks."
RU_SENTENCE_TO_QUESTION = "орпорация Microsoft намерена официально прекратить бесплатную поддержку операционной системы Windows 7 после 14 января 2020 года, сообщается на официальном портале организации . С указанного дня пользователи этой системы не смогут получать обновления безопасности, из-за чего их компьютеры могут стать уязвимыми к кибератакам."
ZH_SENTENCE_TO_QUESTION = "根据该组织的官方门户网站，微软公司打算在2020年1月14日之后正式终止对Windows 7操作系统的免费支持。从那时起，该系统的用户将无法接收安全更新，这可能会使他们的计算机容易受到网络攻击。"
inputs = tokenizer([EN_SENTENCE_TO_QUESTION, RU_SENTENCE_TO_QUESTION, ZH_SENTENCE_TO_QUESTION], padding=True, max_length=256, return_tensors='pt')

summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=100, early_stopping=True)
print([tokenizer.decode(g) for g in summary_ids])
```
Model will generate news titles like:
```
['[SEP] Microsoft to end Windows 7 free support after January 14, 2020[SEP][PAD][PAD][PAD][PAD]',
 '[SEP] Microsoft намерена прекратить бесплатную поддержку Windows 7 после 14 января 2020 года[SEP]',
 '[SEP]微软打算终止对Windows 7操作系统的免费支持[SEP][PAD][PAD][PAD][PAD][PAD][PAD]']
```

## Released checkpoints:

pretrained:
```
microsoft/prophetnet-large-uncased
microsoft/xprophetnet-large-wiki100-cased
```
fine-tuned:
```
microsoft/prophetnet-large-uncased-cnndm
microsoft/xprophetnet-large-wiki100-cased-xglue-ntg
microsoft/xprophetnet-large-wiki100-cased-xglue-qg
```


Remove inconsistency between BertTokenizer and BertTokenizerFast

# 🚀 Feature request
`BertTokenizerFast` has the option to specify `strip_accents=False`. The `BertTokenizer` does not have this option. This inconsistency should be removed by adding the `strip_accents` parameter to `BertTokenizer`.

## Motivation
Without adding this, the `BertTokenizer` can not be used for language models which are lowercase but have accents.

In case of a language model with lowercase and with accents you are forced to load the tokenizer by this:

```python
tokenizer = AutoTokenizer.from_pretrained("<model_name_or_path>", use_fast=True, strip_accents=False)
```

This will NOT work: `tokenizer = AutoTokenizer.from_pretrained("<model_name_or_path>")`

And even this would not work: `tokenizer = AutoTokenizer.from_pretrained("<model_name_or_path>", strip_accents=False)`

## Your contribution
With some hints I am willing to contribute.


Fix docstring for `BertTokenizerFast`.

- remove duplicate doc-entry for `tokenize_chinese_chars`
- add doc for `strip_accents` and `wordpieces_prefix`


[s2s] clean up + doc

this is a follow up to https://github.com/huggingface/transformers/pull/6149
- there was no need to add newly added options to finetune.sh - reverted that change
- added a hint to users how to get all the options (--help)

@sshleifer


Fix tokenizer saving/loading with custom token objects

## Summary

This PR fixes issue #5571.

Pre-trained tokenizers might wrap tokens around custom types (e.g. `AddedToken` from **🤗/tokenizers**) which makes (de)serialization without additional meta information difficult. This PR uses `jsonpickle` library which particularly solves  objects (de)serialization problem.

A little drawback of this approach is that type information is a subject of change and such changes will break back-compatibility once happened. But this seems just like an agreement for making careful back-compatible changes.


Failing XLMModelTest

https://github.com/huggingface/transformers/runs/929962952?check_suite_focus=true

FAILED tests/test_modeling_xlm.py::XLMModelTest::test_inputs_embeds - RuntimeError

Failure introduced somewhere in here:
```
* d951c14a Sylvain Gugger: Model output test (#6155) -   (8 hours ago)
* 86caab1e Sylvain Gugger: Harmonize both Trainers API (#6157) -   (8 hours ago)
* 603cd81a Mehrdad Farahani: readme m3hrdadfi/albert-fa-base-v2 (#6153) -   (11 hours ago)
* 838dc06f Suraj Patil: parse arguments from dict (#4869) -   (13 hours ago)
* cf3cf304 Paul O'Leary McCann: Replace mecab-python3 with fugashi for Japanese tokenization (#6086) -   (13 hours ago)
* f250beb8 Stas Bekman: enable easy checkout switch (#5645) -   (13 hours ago)
* 7d50af4b kolk: Create README.md (#6169) -   (13 hours ago)
* 0034a1d2 Prajjwal Bhargava: Add Pytorch Native AMP support in Trainer (#6151) -   (13 hours ago)
* 7231f7b5 Funtowicz Morgan: Enable ONNX/ONNXRuntime optimizations through converter script (#6131) -   (14 hours ago)
* c0b93a1c Stas Bekman: correct the correction (#6163) -   (23 hours ago)
* a2f6d521 Stas Bekman: typos (#6162) -   (24 hours ago)
* f3065abd Sylvain Gugger: Doc tokenizer (#6110) -   (26 hours ago)
* e642c789 guillaume-be: Addition of a DialoguePipeline (#5516) -   (27 hours ago)
* ec026747 Lysandre Debut: Fix FlauBERT GPU test (#6142) -   (30 hours ago)
* 91cb9546 Sylvain Gugger: Switch from return_tuple to return_dict (#6138) -   (32 hours ago)
* 562b6369 Sylvain Gugger: Tf trainer cleanup (#6143) -   (32 hours ago)
* c127d055 Oren Amsalem: add another e.g. to avoid confusion (#6055) -   (32 hours ago)
* d24ea708 Oren Amsalem: Actually the extra_id are from 0-99 and not from 1-100 (#5967) -   (35 hours ago)
* 3212b885 Stas Bekman: [s2s] add support for overriding config params (#6149) -   (2 days ago)
* 54f9fbef Julien Plu: Rework TF trainer (#6038) -   (2 days ago)
* 3f94170a Lysandre Debut: [WIP] Test TF Flaubert + Add {XLM, Flaubert}{TokenClassification, MultipleC… (#5614) -   (2 days ago)
* 8a8ae276 Sylvain Gugger: Use google style to document properties (#6130) -   (2 days ago)
* fc64559c Julien Plu: Fix TF CTRL model naming (#6134) -   (2 days ago)
* 641b873c Lysandre Debut: XLNet PLM Readme (#6121) -   (2 days ago)
* 8d157c93 Timo Moeller: add deepset/xlm-roberta-large-squad2 model card (#6128) -   (2 days ago)
* 6c002853 Funtowicz Morgan: Added capability to quantize a model while exporting through ONNX. (#6089) -   (2 days ago)
* 25de74cc Sylvain Gugger: Use FutureWarning to deprecate (#6111) -   (3 days ago)
* 640550fc Funtowicz Morgan: ONNX documentation (#5992) -   (3 days ago)
```

Any idea on this @sgugger or @LysandreJik  ? Otherwise I'll dig in.


Failing ONNX Export test

https://github.com/huggingface/transformers/runs/929962952?check_suite_focus=true

```
FAILED tests/test_onnx.py::OnnxExportTestCase::test_quantize_pytorch - TypeError
```


Fixed typo in Longformer


HANS Dataset: Incorrect `label_list` and `label`.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: n/a
- Python version: n/a
- PyTorch version (GPU?): n/a
- Tensorflow version (GPU?): n/a
- Using GPU in script?: n/a
- Using distributed or parallel set-up in script?: n/a

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 tensorflow: @jplu
documentation: @sgugger
 -->
@VictorSanh

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
### Incorrect `label_list`
See line: https://github.com/huggingface/transformers/blob/master/examples/adversarial/utils_hans.py#L259
```python
    def get_labels(self):
        """See base class."""
        return ["contradiction", "entailment", "neutral"]
```
HANS dataset has only two labels, non-entailment, and entailment, but here three are given. Similarly, when mapping from text label to label-id, the label "non-entailment" (which exists in the task but not in the afore-defined labels), the below line is used. I'm curious if this is intentional? If so, would be great to add a warning/comment as those might cause subtle errors in the future.

https://github.com/huggingface/transformers/blob/master/examples/adversarial/utils_hans.py#L311
```python
label = label_map[example.label] if example.label in label_map else 0
```

### Incorrect `label` index
The below line uses the last column as the label. However, the HANS dataset uses the first column for `label`
https://github.com/huggingface/transformers/blob/master/examples/adversarial/utils_hans.py#L271
```python
            label = line[-1]
```


Why are the `device()` and `dtype()` functions in `modelling_utils.py` needed?

Hello,

For BERT and RoBERTa HuggingFace pre-trained models, why are the `device()` and `dtype()` functions in `modeling_utils.py` needed?

See: https://github.com/huggingface/transformers/blob/8edfaaa81b9995cedea2f8805e4c18c2b6cb5bfc/src/transformers/modeling_utils.py#L158

https://github.com/huggingface/transformers/blob/8edfaaa81b9995cedea2f8805e4c18c2b6cb5bfc/src/transformers/modeling_utils.py#L177

Would it be possible for my RoBERTa model to function without an error, if I modify these  `device()` and `dtype()` functions in a way that they will always return `cpu` and `torch.float32` (or `torch.float64`), respectively?

Also, while it is easy to modify the original code to do this, I am not sure on how to get my HuggingFace RoBERTa model to take those modified functions into effect. How can I do this?

Thank you (sorry for asking so many questions),


RoBERTa for QuestionAnswering

I am trying to replicate the example in this link
https://github.com/huggingface/transformers/pull/1502/files, but I get the following error :
``
ValueError                                Traceback (most recent call last)
<ipython-input-23-823cc70a5d4f> in <module>
----> 1 token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
      2 start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
      3 all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
      4 print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))

<ipython-input-23-823cc70a5d4f> in <listcomp>(.0)
----> 1 token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
      2 start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
      3 all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
      4 print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))

ValueError: 102 is not in list
``
I see the same error discussed in https://github.com/huggingface/transformers/issues/2261

Any ideas how I could resolve this issue ?
Thanks in advance.


Adds comet_ml to the list of auto-experiment loggers

This PR does three things:

* abstracts the auto-experiment-loggers' is_available() functions into an "integrations.py" file
* adds comet_ml to the list of auto-loggers available
* updates and reorganizes the docs slightly


Doc pipelines

Continue the improvement of the main classes documentation with pipelines.

[Preview](https://67155-155220641-gh.circle-artifacts.com/0/docs/_build/html/main_classes/pipelines.html) of the new pipeline page.
[Preview](https://67155-155220641-gh.circle-artifacts.com/0/docs/_build/html/internal/pipelines_utils.html) of the new pipeline utils page.


t

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


My finetuned gpt2 model is taking wayy too long to generate samples, like 5-8 minutes

I fine tuned the gpt2 model using transformers, i trained it on a lyrics dataset, and after successful training, when i do model.generate(args), it takes like a hell lot of time to genrate results
What Should i do?


🐛 Not adding `token_type_ids` when the model is `electra` (pytorch_lightning example)

### Who can help
@sshleifer (examples issue)

## Information

Model I am using (Bert, XLNet ...): `ELECTRA`

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## About Issue

https://github.com/huggingface/transformers/blob/838dc06ff5a438159ac25f531d622e8f344476f5/examples/text-classification/run_pl_glue.py#L38-L39

As above, it seems that `token_type_ids` is not included if the `model_type=='electra'` (which also have `token_type_ids`)

I think this code should be changed in other way.


Update convert_pytorch_checkpoint_to_tf2.py


[Benchmark]

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!


Create README.md

Related to pretrained model cards

README for  MiniLM-L12-H384-uncased for QA


Albert pretrain datasets/ datacollator

partially fix #5984
Add supports for Albert model pretraining:
- Add `AlbertTextDataset` class
  - Create `segment_ids` and `sentence_order_labels` attributes for sentence order prediction task

- Add `DataCollatorForAlbertPretrain` class
  - inherited from `DataCollatorForLanguageModeling` class
  - create `attention_mask` for both masked and padding tokens

@sgugger


fix the slow tests doc

remove unnecessary duplication wrt `RUN_SLOW=yes`

@sgugger


[wip] diagnose MT metrics regression from pl 0.8.5 upgrade

Help wanted

Extra attention is needed, help appreciated

**Base Command:**
```bash
BS=8
GAS=4
MAX_LEN=128
python finetune.py \
    --learning_rate=3e-5 \
    --do_train \
    --val_check_interval=0.25 \
    --adam_eps 1e-06 \
    --num_train_epochs 6 --src_lang en_XX --tgt_lang ro_RO \
    --data_dir $ENRO_DIR \
    --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
    --train_batch_size=$BS --eval_batch_size=$BS --gradient_accumulation_steps=$GAS \
    --task translation \
    --warmup_steps 500 \
    --freeze_embeds \
    --model_name_or_path=facebook/mbart-large-cc25 \
	--label_smoothing 0.1  --freeze_embeds --gpus 1 --logger_name wandb --sortish_sampler \
	$@
```

**Clues:**
- many more steps per epoch in wandb on `distillmbart` branch
- lr reasonable on both branches.
- loss much **higher** on `distillmbart` branch.
- val_avg_bleu after ¼ epoch much higher  (23 vs 19)
- fp32 loss goes to NaN after .25 epochs. (  `bru_baseline_pl85_fp32`)

**Suspects:**
- not lr scheduler, though lr schedules differ (because steps differs I presume)
- **early stopping**
- maybe fp16_opt_level being used regardless of `--fp16`?
	(maybe scaler line in PL?)
- optimizer_step unchanged besides `lr_scheduler.step` and the scheduler is clearly stepping. Feels wrong.
- dataloader shuffle/`setup`
- src_lens change in LineByLine ds
- just a change in the way val metrics are computed?

**TLDR**:
can get test BLEU = 26.27 with gradient accumulation steps=1 and no early stopping:
```bash
./train_mbart_cc25_enro.sh --output_dir bru_pl85_long  --label_smoothing 0.1  --freeze_embeds --logger_name wandb --sortish_sampler --fp16_opt_level O1 --gpus 1
```


update min tf requirements

All of the test suite is failing w/o this update - need to re-run `pip install -e .[dev]`

note that the failing tests don't show `"You need to run the TensorFlow trainer with at least the version 2.2.0, your version is {` anywhere so perhaps need some extra tweaks for the test fixtures, but then since non-tf tests fail too, the problem is in the core.

Not sure if perhaps this code needs to be replaced with an assert - then the error message will always be there regardless where it's used.
```
if parse(tf.__version__).release < (2, 2, 0):
    logger.info(
        "You need to run the TensorFlow trainer with at least the version 2.2.0, your version is {}".format(
            tf.__version__
        )
    )
    sys.exit(1)
```

Currently, if I run **any** test, including pytorch-only tests, with tf < 2.2 I get:

```
____________________________________________________________ ERROR collecting tests/test_benchmark.py ____________________________________________________________
tests/test_benchmark.py:6: in <module>
    from transformers import AutoConfig, is_torch_available
src/transformers/__init__.py:659: in <module>
    from .trainer_tf import TFTrainer
src/transformers/trainer_tf.py:34: in <module>
    sys.exit(1)
E   SystemExit: 1
======================================================================== warnings summary ========================================================================
```

To reproduce:

```
pip install tensorflow==2.0.1
pytest -ra tests/test_benchmark.py
```

@jplu


RoBERTa ``tokenizer.decode`` does not produce the same sentence.

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-74-generic-x86_64-with-glibc2.27
- Python version: 3.8.0
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.

### Who can help
@mfuntowicz
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 tensorflow: @jplu
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

This code example should reproduce the issue:

```python3
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
s = """Meanwhile, Tucci's 'straight guy', the emphatic doctor Seger, is not developed into a more interesting character, like the fallible 'straight guys' Cuddy and Wilson."""
outputs = tokenizer(s)
input_ids = outputs['input_ids']
ss = tokenizer.decode(input_ids, skip_special_tokens=True)
print('s='+s)
print('ss='+ss)
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I expect ``s`` and ``ss`` should be exactly the same. However, they are not. The outputs are:

```bash
s=Meanwhile, Tucci's 'straight guy', the emphatic doctor Seger, is not developed into a more interesting character, like the fallible 'straight guys' Cuddy and Wilson.

ss=Meanwhile, Tucci's'straight guy', the emphatic doctor Seger, is not developed into a more interesting character, like the fallible'straight guys' Cuddy and Wilson.
```
There are two spaces missing before ``'straight guy'``.
I am not sure if this behavior is expected or it is a bug.
The thing is I want to use the sentence produced by the ``decode`` function and  I find the output is not exactly the same as the original sentence.

Thanks for the help!
<!-- A clear and concise description of what you would expect to happen. -->


correct the correction

proved to be a different file, so extra path corrections.

@sgugger


typos


Padding Strategy Code missing an else case (maybe?)

## Environment info
- `transformers` version: 3.0.2
- Platform: macOS 10.15.5
- Python version: 3.7
- PyTorch version (GPU?): 1.5 GPU-Yes
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 tokenizers: @mfuntowicz
 Summarization: @sshleifer
 T5: @patrickvonplaten

## Information

Model I am using (T5 via Autotokenizer):

The problem arises when using:
`tokenizer([line], max_length=max_length, padding='max_length' if pad_to_max_length else False,
                     truncation=True, return_tensors=return_tensors, **extra_kw)`

In batch encoding, the latest code decides on a padding strategy:
`_get_padding_truncation_strategies(
        self, padding=False, truncation=False, max_length=None, pad_to_multiple_of=None, verbose=True, **kwargs
    ):`

       ` elif padding is not False:
            if padding is True:
                padding_strategy = PaddingStrategy.LONGEST  # Default to pad to the longest sequence in the batch
            elif not isinstance(padding, PaddingStrategy):
                padding_strategy = PaddingStrategy(padding)`

While calling the tokenizer, instead of 'max_length' I first gave the actual PaddingStrategy.MAX_LENGTH Enum as argument,
but the above code throws an error as 'padding_strategy' is not defined.
## To reproduce
Call the tokenizer as:
`tokenizer([line], max_length=max_length, padding=PaddingStrategy.MAX_LENGTH if pad_to_max_length else False,
                     truncation=True, return_tensors=return_tensors, **extra_kw)`

## Expected behavior

The PaddingStrategy enum should be assigned no issue.

##Suggested Solution

                    ` elif padding is not False:
                              if padding is True:
                                 padding_strategy = PaddingStrategy.LONGEST  # Default to pad to the longest sequence in the batch
                      elif not isinstance(padding, PaddingStrategy):
                             padding_strategy = PaddingStrategy(padding)
                      else:
                          padding_strategy = padding`
It's a one line fix basically, I can raise a PR for the same, unless PaddingStrategy wasn't designed to be used directly?


run_squad.py eval metrics meaning

I am having difficulty understanding what exactly the best_f1 and best_exact scores that are outputted in the run_squad.py evaluation mean. (The scores are computed in the squad_metrics script, found [here](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py)).
What are the "scores" the best calculations are working with, what do the metrics represent, and when/is the best_threshold value employed during training?

Thank you!


Hello,

When I am trying to load the `Roberta-large` pre-trained model, I get the following error:

```python
model_RobertaForMultipleChoice = RobertaForMultipleChoice.from_pretrained('roberta-large', output_hidden_states = True)

OUT:
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
```

How can I solve this issue? Thank you,


Add CircleCI config to run TPU tests.

model card

Related to pretrained model cards

For every incoming commit, this PR will create a Docker image containing the commit's latest code and will run that Docker image on Google Kubernetes Engine on a TPU.


Harmonize both Trainers API

As discussed after the latest rework of TFTrainer.

Also removed references to "master" processes in our API to go to main with deprecation warnings.


should mBART-large-en-ro have decoder_start_token_id by default?

translation

machine translation utilities and models

Hypothesis: since the argument `prepend_bos` is set to "False" in fairseq/examples/README.md, mbart-large-en-ro does not need `decoder_start_token_id`.

TODO:
- create branch that deletes `decoder_start_token_id`. Setting it to None in the config might not be enough.
- verify that decoder_start_token_id is in fact not being used by setting a breakpoint in `generate`.
- run_eval.py on wmt-en-ro/test and see if BLEU is >= 26.46, the score with decoder_start_token_id=250020.


Model output test

Step 2 of the strategy for the new model outputs as outlined on the [forum](https://discuss.huggingface.co/t/new-model-output-types/195/8).
Use the `return_dict` argument introduced in #6138 in all tests and remove all unpacking from the tests.


Hidden State Embedding-Transformers

Hi everybody, I want to use Bert model to get the embedding for a sentence after I fine-tuned it with raw texts. I was wondering if is that possible or not and anybody can help me with that?


readme m3hrdadfi/albert-fa-base-v2

model card

Related to pretrained model cards

model_card readme for m3hrdadfi/albert-fa-base-v2


Using BertWordPiece Tokenizer

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
The Bert WordPiece tokenizer only saves a vocab file on saving the model while other tokenizers such as Byte Level BPE also save a merges file.  When I try to call the model after saving it `RobertaTokenizerFast.from_pretrained("./EsperBERTo_italian", max_len=512)` I get the following error
`OSError: Model name './EsperBERTo_italian' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed './EsperBERTo_italian' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
`


Add Pytorch Native AMP support in Trainer

Pytorch 1.6 introduces Native AMP support. This eliminates the need to build and install Apex and provides improvements over problems highlighted in [Apex #818](https://github.com/NVIDIA/apex/issues/818) and flexibility. This is the recommended way to use AMP.

With this PR, Trainer will automatically use Pytorch's native AMP if 1.6 version is installed, otherwise, it will use Apex.

This PR will close [#6115](https://github.com/huggingface/transformers/issues/6115).


🐛 T5 Tokenizer ignores \n \t characters and more than one whitespace together

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2 (master)
- Platform: Linux-4.9.0-12-amd64-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten @sshleifer

## Information
T5 Tokenizer is based out of SentencePiece and in sentencepiece Whitespace is treated as a basic symbol. But huggingface tokenizers just ignores more than one whitespace.
Consider all the following examples tokenize to the same thing.
```
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("t5-base")

print(tokenizer.tokenize("Hi there I'm good"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there     I'm good"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there    I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there \n   I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there \n     I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there \n  \t   I'm good\n"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']

print(tokenizer.tokenize("Hi there\nI'm good"))
>> ['▁Hi', '▁there', '▁I', "'", 'm', '▁good']
```
All these examples should tokenize to different representations. Also ignoring newline outright means that all applications that use newlines fail.
Model I am using (Bert, XLNet ...): T5

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
All the whitespaces have different tokenizations


[s2s] add support for overriding config params

add support for overriding model params:
```
python finetune.py --encoder_layerdrop 0.1 --decoder_layerdrop 0.1 --dropout 0.1 --attention_dropout 0.1
```
as requested at  https://github.com/huggingface/transformers/issues/6018

`README.md` seems to be mostly the editor removing superfluous whitespace - not sure why github shows it - normally it doesn't. The only added doc section is https://github.com/stas00/transformers/blob/seq2seq-train_params-1/examples/seq2seq/README.md#finetuning-training-params


tokenize cache for examples/language-modeling

# 🚀 Feature request

I find transformer already has cache for tokenized result in examples/token-classification.
I think language-modeling which with a much larger dataset also need that


the documents for transformer don't work

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

##   i find the new document for transformer can't work, https://huggingface.co/transformers/v2.5.0/model_doc/bert.html#bertmodel  ; the class link don't work
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


🌟 Mirostat decoding algorithm

# 🌟 Mirostat: A Perplexity-Controlled Neural Text Decoding Algorithm

## Description

Paper : https://arxiv.org/pdf/2007.14966.pdf

Abstract :

> [...] We use this analysis to design a feedbackbased adaptive top-k text decoding algorithm called mirostat that generates text (of any length) with a redetermined value of perplexity, and thereby high-quality text without any tuning. [...] Mirostat avoids both traps: experiments show that cross-entropy has a near-linear relation with repetition in generated text. This relation is almost independent of the sampling method but slightly dependent on the model used. Hence, for a given language model, control over perplexity also gives control over repetitions.

## Open source status

* [x] the model implementation is available: https://github.com/basusourya/mirostat
* [x] the model weights are available: _Not applicable_
* [x] who are the authors: @basusourya


TOKENIZER: truncation not working for batch

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-4.15.0-106-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 examples/distillation: @VictorSanh

## Information

Model I am using: DistilBertForSequenceClassification.

The tokenizer does not truncate when I pass a list of strings. It only works when I pass a single string.

## To reproduce

Copy/paste (or just read) code below.

```python
from transformers import DistilBertTokenizer

tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-uncased")

sentence = "Submit a bug report to help us improve transformers."
output = tokenizer(sentence, padding=True, truncation=True, max_length=4)

print(output['input_ids'])
# 4 tokens as expected from *max_length=4*
# out: [101, 12040, 1037, 102]

# now let's test with multiple sentences
sentences = [
  "Submit a bug report to help us improve transformers.",
  "Benchmark a part of this library and share your results"
]

output = tokenizer(sentences, padding=True, truncation=True, max_length=4)
print(output['input_ids'])
# output is returning all tokens, it is not truncating to max_length!
# out: [[101, 12040, 1037, 11829, 3189, 2000, 2393, 2149, 5335, 19081, 1012, 102, 0],
#       [101, 6847, 10665, 1037, 2112, 1997, 2023, 3075, 1998, 3745, 2115, 3463, 102]]
```

## Expected
```python
# output truncated to max_length (4 as in the example)
# out: [[101, 12040, 1037, 102],
#       [101, 6847, 10665, 102]]
```


Question-Answering pipeline doesn't work anymore with long text

Transformers version: 3.0.2

The question-answering models don't seem to work anymore with long text, any reason why this is happening? I have tried with the default model in `pipeline` as well as with specific models.

e.g

__Sample Code:__

```
from transformers import pipeline

nlp_qa = pipeline('question-answering') # 1st try
nlp_qa = pipeline('question-answering', model='deepset/roberta-base-squad2') # 2nd try

context = """
Coronaviruses are a large family of viruses which may cause illness in animals or humans.
In humans, several coronaviruses are known to cause respiratory infections ranging from the
common cold to more severe diseases such as Middle East Respiratory Syndrome (MERS) and Severe Acute Respiratory Syndrome (SARS).
The most recently discovered coronavirus causes coronavirus disease COVID-19.
COVID-19 is the infectious disease caused by the most recently discovered coronavirus.
This new virus and disease were unknown before the outbreak began in Wuhan, China, in December 2019.
COVID-19 is now a pandemic affecting many countries globally.
The most common symptoms of COVID-19 are fever, dry cough, and tiredness.
Other symptoms that are less common and may affect some patients include aches
and pains, nasal congestion, headache, conjunctivitis, sore throat, diarrhea,
loss of taste or smell or a rash on skin or discoloration of fingers or toes.
These symptoms are usually mild and begin gradually.
Some people become infected but only have very mild symptoms.
Most people (about 80%) recover from the disease without needing hospital treatment.
Around 1 out of every 5 people who gets COVID-19 becomes seriously ill and develops difficulty breathing.
Older people, and those with underlying medical problems like high blood pressure, heart and lung problems,
diabetes, or cancer, are at higher risk of developing serious illness.
However, anyone can catch COVID-19 and become seriously ill.
People of all ages who experience fever and/or  cough associated with difficulty breathing/shortness of breath,
chest pain/pressure, or loss of speech or movement should seek medical attention immediately.
If possible, it is recommended to call the health care provider or facility first,
so the patient can be directed to the right clinic.
People can catch COVID-19 from others who have the virus.
The disease spreads primarily from person to person through small droplets from the nose or mouth,
which are expelled when a person with COVID-19 coughs, sneezes, or speaks.
These droplets are relatively heavy, do not travel far and quickly sink to the ground.
People can catch COVID-19 if they breathe in these droplets from a person infected with the virus.
This is why it is important to stay at least 1 meter) away from others.
These droplets can land on objects and surfaces around the person such as tables, doorknobs and handrails.
People can become infected by touching these objects or surfaces, then touching their eyes, nose or mouth.
This is why it is important to wash your hands regularly with soap and water or clean with alcohol-based hand rub.
Practicing hand and respiratory hygiene is important at ALL times and is the best way to protect others and yourself.
When possible maintain at least a 1 meter distance between yourself and others.
This is especially important if you are standing by someone who is coughing or sneezing.
Since some infected persons may not yet be exhibiting symptoms or their symptoms may be mild,
maintaining a physical distance with everyone is a good idea if you are in an area where COVID-19 is circulating.
"""

nlp_qa(context=context, question='What is a coronavirus ?')
```

__Error Message:__

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-15-ddac1f9cb68e> in <module>()
----> 1 nlp_qa(context=context, question='What is a coronavirus ?')

1 frames
/usr/local/lib/python3.6/dist-packages/transformers/pipelines.py in <listcomp>(.0)
   1314                         ),
   1315                     }
-> 1316                     for s, e, score in zip(starts, ends, scores)
   1317                 ]
   1318

KeyError: 0
```

This used to work before version 3 I remember, would really appreciate some help on this.


Tf trainer cleanup

New version of #6015

Will harmonize the public customization hooks and document everything properly once this is merged.


Fix FlauBERT GPU test


Bug in language_modeling.py calling tokenizer.num_special_tokens_to_add

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-4.4.0-1081-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 tokenizers: @mfuntowicz
 -->

## Information

Model I am using GPT-2

The class TextDataset call tokenizer.num_special_tokens_to_add(pair=False) but the correct argument is called is_pair. I assume the bugfix corresponds to transformers repo.

## To reproduce

https://github.com/huggingface/transformers/blob/e49393c3617e877f0370f7bad7c7e823808c5bfb/src/transformers/data/datasets/language_modeling.py#L27

https://github.com/huggingface/tokenizers/blob/master/bindings/python/tokenizers/implementations/base_tokenizer.py#L20

I'm using transformers 3.0.2 and tokenizers 0.8.1rc1 (try to update but says its incompatible)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Copyright date and owner not filled out in LICENSE file

In transformers/LICENSE the copyright date and owner is not filled out.

https://github.com/huggingface/transformers/blob/master/LICENSE#L179-L190


Applying hugging face transformer in sequence labeling problem

Hello.. Thanks for your great framework. Btw, what I'd like to know is that can I apply these hugging face transformer models in **sequence labeling** problems like **part of speech tagging** and **word segmentation**(because I see only **ner** model in example folder). If I can, **how** can I do that?? Can I get some helps like **example scripts** about how to apply these transformers for sequence labeling problems??


Switch from return_tuple to return_dict

This is the first step in the change of model outputs as described on [the forum](https://discuss.huggingface.co/t/new-model-output-types/195/8).

This PR removes the argument `return_tuple` and introduces `return_dict` (that works the other way round) and all models now return tuple by default (100% full backward compatibility) unless you opt-in the new model output types with `return_dict=True`. The model output class is changed to the dict-like one that should work equally well for TensorFlow.

I have normally updated all examples in the docs to instantiate the model with `return_dict=True` but more docs will follow in other PRs. For the tests, I have set `return_dict=True` in one of the common tests just to make sure it actually works. Step 2 (in a follow-up PR) will be to use it in all tests.

Step 3 is then going to update the TensorFlow models to use this `ModelOutput`.


StopIteration error when using HuggingFace Transformer models

Hello,

I am trying to use the RobertaForMultipleChoice model, and when I try to compute the mc_loss, the following StopIteration error is generated:

```python
>>> mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels = mc_labels)[0]

Traceback (most recent call last):
  File "STAT946_final_project_code_v4.py", line 625, in <module>
    success_rate_list_diag_normal = main_function_diag_normal('/home/ec2-user/test.txt', 'test_ans_num.txt', num_iter, log_interval)
  File "STAT946_final_project_code_v4.py", line 415, in main_function_diag_normal
    best_model_RobertaForMultipleChoice_diag_normal = train_loop(model_RobertaForMultipleChoice, tokenizer, optimizer_1, scheduler_1, log_interval, svi_diag_normal, guide_diag_normal, best_model_RobertaForMultipleChoice_diag_normal)
  File "STAT946_final_project_code_v4.py", line 342, in train_loop
    optimizer, scheduler, log_interval, svi, guide, epoch)
  File "STAT946_final_project_code_v4.py", line 237, in train_mc_head
    mc_loss = model(input_ids = input_ids, attention_mask = attention_mask, labels = mc_labels)[0]
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 441, in forward
    output_hidden_states=output_hidden_states,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyro/nn/module.py", line 413, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 732, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 228, in get_extended_attention_mask
    extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 159, in dtype
    first_tuple = next(gen)
StopIteration
```

The error seem to be generated from the HuggingFace code below:

```python
@property
def device(self) -> device:

     try:
        return next(self.parameters()).device

     except StopIteration:

       # For nn.DataParallel compatibility in PyTorch 1.5

       def find_tensor_attributes(module: nn.Module) -> List[Tuple[str, Tensor]]:
              tuples = [(k, v) for k, v in module.__dict__.items() if torch.is_tensor(v)]
              return tuples

       gen = self._named_members(get_members_fn=find_tensor_attributes)
       first_tuple = next(gen)
       return first_tuple[1].device
```

What is the cause of this error? and how can I fix it?

Thank you


frequent checkpoints have worse performance

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

Hi All, I often notice an issue when training a model and evaluate on dev set.
Usually we may evaluate on the dev after each epoch, let's call this as setting A;
But we often want to check the system more often on the dev set, so we may evaluate for example 1/5 epoch; let's call this as setting B.

What I noticed is that A and B will get totally different performance in the end.  Since B checks more often, I supposed that B can get the same or at least very close performance with A then evaluate at 5/5 of the training set, 10/5 of the training set, etc. But they are very different. For example, when I train textual entailment model on RTE dataset, A can give me about 86% accuracy on dev, but B can only give about 80%.

What's the issue here? thanks

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


MBartTokenizerTrimmed to support truncated embeddings

Core: Tokenization

Internals of the library; Tokenization.

Motivation:
The embeddings table for MBART is huge, but only ~40K of the entries are used/finetuned for most wmt tasks. See https://github.com/pytorch/fairseq/issues/2120

- needs vocab.json (fairseq Dictionary)
- needs to call `encode_as_pieces` with restricted vocabulary.

I will take this.


Enable ONNX/ONNXRuntime optimizations through converter script

Introduce `--optimize` CLI argument and `optimize()` method to allow ONNXRuntime to operates all the possible optimizations on the raw ONNX IR.

Added documentation for this parameter in the ONNX/ONNXRuntime section on the doc.


Use google style to document properties

It's cleaner this way and avoid redundancy.


Add new pre-trained models BERTweet and PhoBERT

model card

Related to pretrained model cards

I'd like to add pre-trained [BERTweet](https://github.com/VinAIResearch/BERTweet/) and [PhoBERT](https://github.com/VinAIResearch/PhoBERT/) models to the `transformers` library.

Users now can use these models directly from `transformers`. E.g:

    bertweettokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-base")
    bertweetmodel = BertweetModel.from_pretrained("vinai/bertweet-base")

    phoberttokenizer = PhobertTokenizer.from_pretrained("vinai/phobert-large")
    phobertmodel = PhobertModel.from_pretrained("vinai/phobert-large")

[BERTweet: A pre-trained language model for English Tweets](https://github.com/VinAIResearch/BERTweet/)
[PhoBERT: Pre-trained language models for Vietnamese](https://github.com/VinAIResearch/PhoBERT/)


add deepset/xlm-roberta-large-squad2 model card

Related to pretrained model cards


Initializing XLMRobertaTokenizer using pretrained tokenizer expects serialized vocab

Hi,
I am training an XLMRoberta model from scratch on Hindi. I am using a sentencepiece tokenizer trained exclusively on monolingual data following the steps mentioned in the [tokenizers repository](https://github.com/huggingface/tokenizers/tree/704cf3fdd2f607ead58a561b892b510b49c301db/bindings/python#using-the-provided-tokenizers). This results in the creation of `vocab.json` and `merges.txt`.
However when I try to initialize the tokenizer using `XLMRobertaTokenizer.from_pretrained` I get an error saying
```assumed 'models/sentencepiece' was a path, a model identifier, or url to a directory containing vocabulary files named ['sentencepiece.bpe.model'] but couldn't find such vocabulary files at this path or url. ```
I am assuming this is a serialized file based on [huggingface.co model](https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-base-sentencepiece.bpe.model) but don't know how to serialize my vocab.json file. I have already tried using `pickle` and `numpy`
Versions used:
transformers: 2.9.1
tokenizers: 0.7.0