finetune_trainer segfault

Hi
I wand to run finetune_trainer on tpu:

export TPU_IB_ADDRESS=10.160.244.2

start=`date +%s`
python xla_spawn.py --num_cores 8 \
    finetune_trainer.py \
    --tokenizer_name t5-base --model_name_or_path t5-base \
    --data_dir /home/rabeeh/ruse/seq2seq/seq2seq/data \
    --output_dir /home/rabeeh/outputs/verify --overwrite_output_dir \
    --learning_rate=3e-4 \
    --warmup_steps 500 \
    --per_device_train_batch_size=256 --per_device_eval_batch_size=256\
    --num_train_epochs=6 \
    --save_steps 500 --eval_steps 500 \
    --logging_steps 200 \
    --max_source_length 128 --max_target_length 128 --val_max_target_length 128 --test_max_target_length 128 \
    --task translation --label_smoothing 0.1 \
    "$@"

end=`date +%s`
runtime=$((end-start))
echo running time $runtime

here are the logs, thank for your help/

2020-11-06 09:05:12.060270: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 	 [[XRTExecute_G12]]
2020-11-06 09:05:12.060282: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
2020-11-06 09:05:12.060292: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 09:05:12.060303: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76]   (1) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
2020-11-06 09:05:12.060314: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 	 [[{{node XRTExecute}}]]
2020-11-06 09:05:12.060325: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
2020-11-06 09:05:12.060336: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 09:05:12.060347: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 0 successful operations.
2020-11-06 09:05:12.060358: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 0 derived errors ignored.
###### Arguments  (ModelArguments(model_name_or_path='t5-base', config_name=None, tokenizer_name='t5-base', cache_dir=None, freeze_encoder=True, freeze_embeds=True), DataTrainingArguments(data_dir='data/wmt_en_de', task='translation', max_source_length=128, max_target_length=128, val_max_target_length=128, test_max_target_length=128, n_train=-1, n_val=-1, n_test=-1, src_lang=None, tgt_lang=None, eval_beams=None, ignore_pad_token_for_loss=True), Seq2SeqTrainingArguments(output_dir='/home/rabeeh/outputs/verify', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=True, evaluation_strategy=<EvaluationStrategy.STEPS: 'steps'>, prediction_loss_only=True, per_device_train_batch_size=64, per_device_eval_batch_size=64, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=0.0003, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=6.0, max_steps=-1, warmup_steps=500, logging_dir='runs/Nov06_09-03-01_eee96b6103a2', logging_first_step=True, logging_steps=200, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=8, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/home/rabeeh/outputs/verify', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, label_smoothing=0.1, sortish_sampler=False, predict_with_generate=False, adafactor=False, encoder_layerdrop=None, decoder_layerdrop=None, dropout=None, attention_dropout=None, lr_scheduler='linear'))
######### data_file  data/wmt_en_de/train.source
######### data_file  data/wmt_en_de/val.source
{'loss': 10624.0, 'learning_rate': 6e-07, 'epoch': 0.00011253657438667567}
Exception in device=TPU:0: Resource exhausted: From /job:tpu_worker/replica:0/task:0:
2 root error(s) found.
  (0) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[XRTExecute_G12]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
    fn(gindex, *args)
  File "/root/home/rabeeh/seq2seq/finetune_trainer.py", line 330, in _mp_fn
    app.run(main, flags_parser=parse_flags)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/root/home/rabeeh/seq2seq/finetune_trainer.py", line 279, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 757, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 31, in __next__
    return self.next()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 37, in next
    xm.mark_step()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 716, in mark_step
    wait=xu.getenv_as('XLA_SYNC_WAIT', bool, False))
RuntimeError: Resource exhausted: From /job:tpu_worker/replica:0/task:0:
2 root error(s) found.
  (0) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[XRTExecute_G12]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.
Traceback (most recent call last):
  File "xla_spawn.py", line 77, in <module>
    app.run(main, flags_parser=parse_args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "xla_spawn.py", line 71, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores, start_method='fork')
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
    start_method=start_method)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code 17
running time 153


update from v3.0.0 to v3.4.0 got an error

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: v3.4.0
- Platform: redhat
- Python version: 3.8.3 64bit
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):
Bert
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

yesterday i used transformers v3.0.0 trained my model and save it use
`torch.save(model.state_dict(), config.save_path)`,
and today i update transformers to v3.4.0, when i use
`torch.load`
i got this error
![image](https://user-images.githubusercontent.com/31238754/98341317-a2be1600-2049-11eb-9672-749896b9c2a4.png)
could anybody help me!!! thx

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


[s2s trainer examples] a tight quality regression test

Continuing from https://github.com/huggingface/transformers/issues/6049 and https://github.com/huggingface/transformers/issues/8154, we are trying to build a test that will be good at regression detection in the finetune training code, quoting @sshleifer:

> try to make a fast command line test/script for summarization that meets some reasonable learning requirements (using rouge score. Try to get rouge2 above 4?)
> For example, if you set dropout=0.95 or freeze all parameters, or set the LR too low, or mess up the special tokens logic, the test should fail.

I have some follow up questions:

1. should I make a subset dataset on s3? or should I attempt to build a tiny one and check it in git?

2. Speed-wise would xsum be the best candidate to derive the dataset from?

3. define fast? do you mean something that would not need `@slow` - but summarization is inherently a slow task so it doesn't go well with "fast"

4. if summarization it is, which model would you recommend that is light enough to fit with not `@slow` to be downloadable from CI

5. if you want a really tight test we will need to first test that the hparams in question do lead to a failure (which will be caught) when they are lower than whatever threshold we choose, which means that ideally we will need to re-run the training multiple times in the same test to make the test really good. Which again leads to the question of how fast is fast.

Of course, one of the difficulties with tight params would be that different GPUs will produce different results, but I suppose we will tune it up as we go.

Thank you!

@sshleifer


I encountered an error when I was running code that the object could not be called.But the BertTokenizer doesn't exist in my code.

## Error details

- `transformers` version:2.11.0

Traceback (most recent call last):
File "main.py", line 108, in
fire.Fire()
File "D:\software\Anaconda3\envs\ynt2\lib\site-packages\fire\core.py", line 138, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "D:\software\Anaconda3\envs\ynt2\lib\site-packages\fire\core.py", line 468, in _Fire
target=component.name)
File "D:\software\Anaconda3\envs\ynt2\lib\site-packages\fire\core.py", line 672, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "main.py", line 69, in train
output = model(x)
File "D:\software\Anaconda3\envs\ynt2\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\YNT\Desktop\重要文件夹\数据集\EMLo相关\pytorch_bert_elmo_example-master\model.py", line 40, in forward
word_embs = self.get_bert(x)
File "C:\Users\YNT\Desktop\重要文件夹\数据集\EMLo相关\pytorch_bert_elmo_example-master\model.py", line 90, in get_bert
ids = self.tokenizer(sentence_lists, padding=True, return_tensors="pt")
TypeError: 'BertTokenizer' object is not callable


Fix typo

# What does this PR do?

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


torch 1.4.0 transformers segment fault

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
torch 1.4.0 transformers segment fault
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


apply_chunking_to_forward should only be the same in the chunking dimension

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: All
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): XLNet

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Simply send in 2 tensor to the apply_chunking_to_forward that have the same batch length, same batch size, but different dimensionality and it will pop up with an exception

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Should only chunk if they are the same in the chunk dimension

```
 assert len(input_tensors) > 0, "{} has to be a tuple/list of tensors".format(input_tensors)
    tensor_shape = input_tensors[0].shape
    assert all(
        input_tensor.shape == tensor_shape for input_tensor in input_tensors
    ), "All input tenors have to be of the same shape"
```
Should be:
```
    tensor_shape = input_tensors[0].shape[chunk_dim]
    assert all(
        input_tensor.shape[chunk_dim] == tensor_shape for input_tensor in input_tensors
    ), "All input tenors have to be of the same shape"
```

In here if there are 2 input tensors with the shapes:
[512,2,768] and [512,2,300] the method throws an exception when it should only chunk based on the chunk dimension (in this case 2).


PEGASUS generation/decoding VERY Slow

## Environment info
- `transformers` version: 3.4.0
- Platform: Linux-4.19.112+-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@sshleifer

## Information

Model I am using: *PEGASUS-large*, *PEGASUS-cnn_dailymail*, *PEGASUS-xsum*

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I compared the generation/decoding performance of BART and PEGASUS both loaded via AutoModelSeq2SeqForLM fine-tuned on a custom dataset for the same amount of time. Generation was done using basic greedy decoding. PEGASUS models are anywhere between 5-15x slower than BART. Fine-tuning speed was on-par for both models.

## To reproduce

Steps to reproduce the behavior:
1. Load PEGASUS and BART from any of the mentioned checkpoints (using AutoModelSeq2SeqForLM)
1. Fine-tune models
2. Decode using greedy decoding
3. Compare fine-tuning performance with other Seq2Seq Model (BART-large)

## Expected behavior
Decoding performance on-par with BART-large?


TFTrainer stuck in evaluation

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Darwin-17.7.0-x86_64-i386-64bit
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: YES

### Who can help
 Trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...): TFCamembertModel and TFTrainer to fine tune model on MLM

The problem arises when using:
* [x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce
Behavior of prediction_loop function causes evaluation to go in infinite loop if prediction_loss_only is set to True. The step==steps condition is never reached
Steps to reproduce the behavior: Read code from TFTrainer below

```python
if not prediction_loss_only:
                if isinstance(logits, tuple):
                    logits = logits[0]

                if isinstance(labels, tuple):
                    labels = labels[0]

                if self.args.n_replicas > 1:
                    for val in logits.values:
                        if preds is None:
                            preds = val.numpy()
                        else:
                            preds = np.append(preds, val.numpy(), axis=0)

                    for val in labels.values:
                        if label_ids is None:
                            label_ids = val.numpy()
                        else:
                            label_ids = np.append(label_ids, val.numpy(), axis=0)
                else:
                    if preds is None:
                        preds = logits.numpy()
                    else:
                        preds = np.append(preds, logits.numpy(), axis=0)

                    if label_ids is None:
                        label_ids = labels.numpy()
                    else:
                        label_ids = np.append(label_ids, labels.numpy(), axis=0)

                if step == steps:
                    break

```

## Expected behavior
Once the number of steps reached, we should be able to get out of the for loop that draws batches


Adding kNN language modeling and Machine Translation

Feature request

Request for a new feature

# 🌟 Adding the kNN-LM and kNN-MT models

## Model description

The kNN [Language Model](https://arxiv.org/pdf/1911.00172.pdf) and [Machine Translation](https://arxiv.org/pdf/2010.00710.pdf) leverage additional data at test time by looking up examples in the data store that are similar to the test example and using them to inform the prediction. This leads to decreased perplexity for language modeling and increased BLEU for MT.

Ideally, we'd have a general KNNDecoder class which can use any model in the library along with a [🤗datasets](https://github.com/huggingface/datasets) data store, similarly to what RAG does currently.

## Open source status

* [x] the model implementation is available: for kNN-LM, an implementation using Fairseq and FAISS can be found [here](https://github.com/urvashik/knnlm)
* [x] the model weights are available: the methods use pre-trained models which are already in the library, [GPT-2](https://huggingface.co/gpt2) and the [Facebook WMT'19 models](https://huggingface.co/facebook/wmt19-de-en)
* [x] who are the authors: first author @urvashik


Error in RAG finetuning script

## Environment info
ut the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Linux-4.18.0-147.5.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 @patrickvonplaten, @lhoestq

## Information

I am using RAG fine-tuning script. During the fine-tuning process, it says

**torch.nn.modules.module.ModuleAttributeError: 'GenerativeQAModule' object has no attribute 'opt'**

The bug exactly appears in [line 332 in finetune.py (https://github.com/huggingface/transformers/blob/master/examples/rag/finetune.py#L332)

## To reproduce
I have installed the transformers library from the source. Not the pip.


model parallelism for BART

Model Parallel

Model Parallelilsm Implementations

For @stas00:

High Level Goal: allow large Seq2Seq transformers, (many of which inherit from BART) to be run on/accross multiple GPUs with model-parallelism.

+ This is a prerequisite for [adding m2m100](https://github.com/huggingface/transformers/issues/8054), and can be done in the same or a separate PR.I prefer separate given all the boilerplate associated with new model additions.
+ This has been attempted for GPT-2, and is days-weeks away from merging: #7772 .
+ fairseq has a different scheme, as shown by the last 4 clargs in [this command](https://github.com/pytorch/fairseq/tree/master/examples/m2m_100#generation-for-the-12b-model)
+ That model requires more hardware than you have locally, which is another good reason to try a 2 GPU case first, then try to
As such, the test you should try to get passing is roughly:

```python
n = 20 # set so that model.cuda() OOMs on 1 GPU (in a few lines)
cfg = BartConfig(encoder_layers=n, decoder_layers=n)
model = BartForConditonalGeneration(cfg)
# this device_map is taken from #7772, feel free to make your own signature, like

model.cuda()
model.generate(**batch) # should OOM here or the line before

device_map = {0: [0, 1, 2, 3, 4, 5, 6, 7, 8],
                          1: [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21],
                          2: [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34],
                          3: [35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]}
model.parallelize(device_map)  # i might call this model.split
batch = tokenizer(['I am a small frog'])

model.save_pretrained('parallelized_model')
model.from_pretrained('parallelized_model') # should be parallelized
model.deparallelize() # Puts the model back on cpu and calls torch.cuda.empty_cache() to liberate GPU memory
```

### Requirements

- user can save_pretrained/load_pretrained without losing the partitioning of layers -> devices.
- some forward/generate call that would `OOM` on a single GPU does not `OOM` after calling `.parallelize`
- user can repartition by loading full model to cpu, calling `model.parallelize(device_map)`

### Brain Dump
- You should read the whole document https://github.com/pytorch/fairseq/tree/master/examples/m2m_100#beyond-english-centric-multilingual-machine-translation before starting  before starting
- I would also read discussion, code for https://github.com/huggingface/transformers/pull/7772
- you have a lot of flexibility in naming/API and should feel empowered to make choices as you see fit.
- To the extent possible, keep the PR small and avoid interfering existing single-gpu functionality.
- You could add a fairscale dependency during your experiments/local dev, but it would be a battle to get `fairscale` added as a dependency. If that is a worthwhile battle, however, you should argue for it.
- I suspect that this will take nearly as long as `FSMT`, but be much less code.

What do you think?


[model_cards] Update Italian BERT models and introduce new Italian XX…

model card

Related to pretrained model cards

Hi,

this PR updates some of our @dbmdz model cards for Italian BERT models:

* Add details about issues with vocab for BERT models
* Introduce new repository for downstream evaluations (NER and PoS tagging)

It also adds new model cards for our newly trained Italian XXL ELECTRA model (discriminator and generator part).


__init__() got an unexpected keyword argument 'early_stop_callback'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


[github CI] add a multi-gpu job for all example tests

As discussed in https://github.com/huggingface/transformers/pull/8315#issuecomment-722398166 we now have a growing set of tests that do distributed testing and require a multi-gpu setup, and which aren't being tested at the moment by any CI.

This PR adds a new job:

* [x] run all example tests on multi-gpu

Since these are slow, I added it only to the scheduled job.

Question: Is this really needed for every step?
```
           source .env/bin/activate
```
It's in all test steps, even when there is multiple test suites run in sequence it's re-run.

Won't it be enough to do it once in the env prep step for all steps in the same job?

@sshleifer, @LysandreJik


Add new token classification example

Related to pretrained model cards

# What does this PR do?

This PR adds a new example script leveraging the dataset library, the offsets from the fast tokenizers of the tokenizers library and the Trainer API.


finetune_trainer being really slow on TPU

Hi
I need to run finetune_trainer for a large scale project on TPU, here is the command I use

`python xla_spawn.py --num_cores 8 finetune_trainer.py     --learning_rate=3e-5         --n_val 1000  --model_name_or_path t5-small --data_dir data/wmt_en_ro/ --output_dir /home/rabeeh/temp/ --overwrite_output_dir  --tpu_num_cores=8 --max_source_length=64 --max_target_length=64 --per_device_train_batch_size=32 --per_device_eval_batch_size=32 --label_smoothing=0.1 --task="translation" --logging_steps=200 --eval_steps=500 --num_train_epochs=1 --save_steps=500 --max_source_length=128  --max_target_length=128 --val_max_target_length=128 --test_max_target_length=128
`

It is running slow for me, and I was wondering if I can make it faster, can I make it run distributedly on multiple host machines? I really appreciate providing me with command to run this code efficiently on TPUs on cluster. Is there any setting I am missing in my command?

thank you.


Update README.md

Related to pretrained model cards

fixes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


pip cannot install transformers with python version 3.X version on Ubuntu 18.04

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform:
```
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
```

- Python version: python3.X
- PyTorch version (GPU?): no
- Tensorflow version (GPU?): na
- Using GPU in script?: na
- Using distributed or parallel set-up in script?:na

### Who can help
  tokenizers: @mfuntowicz
## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] pip install transformers

## To reproduce

Steps to reproduce the behavior:

1. Create a VM with Ubuntu 18.04
2. Install any python3.X version
3. pip3 install transformers

<!--
Requirement already satisfied: tokenizers==0.9.2 in /usr/local/lib/python3.6/dist-packages (from transformers)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers)
Requirement already satisfied: click in /usr/lib/python3/dist-packages (from sacremoses->transformers)
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf->transformers)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3/dist-packages (from requests->transformers)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers)
Building wheels for collected packages: sentencepiece
  Running setup.py bdist_wheel for sentencepiece ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-baldc7a3/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp0rhc8tr_pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/sentencepiece
  copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
  copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
  copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
  running build_ext
  /bin/sh: 1: pkg-config: not found
  Cloning into 'sentencepiece'...
  Note: checking out '8336bbd0c1cfba02a879afe625bf1ddaf7cd93c5'.

  You are in 'detached HEAD' state. You can look around, make experimental
  changes and commit them, and you can discard any commits you make in this
  state without impacting any branches by performing another checkout.

  If you want to create a new branch to retain commits you create, you may
  do so (now or later) by using -b with the checkout command again. Example:

    git checkout -b <new-branch-name>

  ./build_bundled.sh: 15: ./build_bundled.sh: cmake: not found
  make: *** No targets specified and no makefile found.  Stop.
  make: *** No rule to make target 'install'.  Stop.
  env: ‘pkg-config’: No such file or directory
  Failed to find sentencepiece pkg-config

  ----------------------------------------
  Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece
Installing collected packages: sentencepiece, transformers
  Running setup.py install for sentencepiece ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-baldc7a3/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-cfvofwnm-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/sentencepiece
    copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
    copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
    copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
    running build_ext
    /bin/sh: 1: pkg-config: not found
    mkdir: cannot create directory ‘bundled’: File exists
    fatal: destination path 'sentencepiece' already exists and is not an empty directory.
    fatal: destination path 'sentencepiece' already exists and is not an empty directory.
    mkdir: cannot create directory ‘build’: File exists
    ./build_bundled.sh: 15: ./build_bundled.sh: cmake: not found
    make: *** No targets specified and no makefile found.  Stop.
    make: *** No rule to make target 'install'.  Stop.
    env: ‘pkg-config’: No such file or directory
    Failed to find sentencepiece pkg-config

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-baldc7a3/sentencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-cfvofwnm-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-baldc7a3/sentencepiece/
->

## Expected behavior
Successful installation of transformers


Make Trainer evaluation handle dynamic seq_length

# What does this PR do?

In token classification problems or language modeling predictions, logits and labels are usually of shape `[batch_size, seq_length, ...]` and if one uses dynamic padding, the `seq_length` might change from batch to batch. This made the evaluate and predict methods of the `Trainer` fail, because the Trainer wants to concatenate all predictions in one array.

This PR fixes this, by potentially padding up to the maximum seq length when needed, using the -100 pad index (since it's the marker for ignored labels in PyTorch). It add tests that were failing before the PR, to check the fix actually solves the issue.


Language Model to get only the score of predefined tokens

# 🚀 Feature request

Hey,

In Masked Language Model such as [TFDistilBertForMaskedLM](https://huggingface.co/transformers/model_doc/distilbert.html#tfdistilbertformaskedlm). The example explains well how to get the scores for [MASK] token. Nevertheless I did not find anything related to scale it up to 50k texts and especially get the results for a list of predefined tokens (let's say 10 tokens like in [MNLI examples for ZSL](https://huggingface.co/joeddav/xlm-roberta-large-xnli)).

The feature would be to add a _only_ids_ parameter in the `call` function of `TFDistilBertForMaskedLM` to get at the end for each sentence only the scores of the predefines tokens we are interested in.

Maybe I did not search at the right place, however I managed to find something by digging in source code of HuggingFace.

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

Well my motivation came from two things:

I was trying to implement my own pipeline of the [PET Algorithm from Schik et al.](https://arxiv.org/pdf/2001.07676.pdf). And you need to perform weak labeling on a dataset of ~10k texts a few times (as much time as you have PVP couples).

So first, I tried to code with the [TFDistilBertForMaskedLM](https://huggingface.co/transformers/model_doc/distilbert.html#tfdistilbertformaskedlm) class in order to do label the data. But came the second motivation:

```python
ResourceExhaustedError: OOM when allocating tensor
```

This errors occured on colab and also in [Lambda GPU](https://lambdalabs.com) (gpu.1x.rtx6000) whenever I tried to do it for more than 1k sentences. I dug a bit and the errors seemed to come from the `call` function of the class `TFDistilBertLMHead`, because it manipulates vectors of shape `[bs, dim, config.voc_size]`.

As I did not find anything on the issues or StackOverflow, I decided to get into the code of `modeling_tf_distilbert.py` to try to change a bit the `call` functions and try my code on [Lambda GPU](https://lambdalabs.com) (gpu.1x.rtx6000) to look for improvements (I used the test set of yahoo question dataset).

## Your contribution

I changed a couple of things (not complicated things) in the classes and compared it (results below).

First change (easy) - I would call it CS for cutting out the sequence:

What we do is before applying the `vocab_projector` method we reduce the focus only on the mask token. It enables to manipulates tensors of shape (bs, 1, voc_size) instead of tensors of shape (bs, max_length, voc_size).

```python
class CB_TFDistilBertForMaskedLM(TFDistilBertForMaskedLM):

    def call(
        self,
        inputs=None,
        attention_mask=None,
        head_mask=None,
        inputs_embeds=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None,
        labels=None,
        training=False,
        mask_only=False #If you just focus on the mask token id change the False by it (103 here)
    ):
        r"""
        labels (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`):
            Labels for computing the masked language modeling loss.
            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
            Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
            in ``[0, ..., config.vocab_size]``
        """

        return_dict = return_dict if return_dict is not None else self.distilbert.return_dict
        if isinstance(inputs, (tuple, list)):
            labels = inputs[7] if len(inputs) > 7 else labels
            if len(inputs) > 7:
                inputs = inputs[:7]
        elif isinstance(inputs, (dict, BatchEncoding)):
            labels = inputs.pop("labels", labels)

        distilbert_output = self.distilbert(
            inputs,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            training=training,
        )

        hidden_states = distilbert_output[0]  # (bs, seq_length, dim)
        prediction_logits = self.vocab_transform(hidden_states)  # (bs, seq_length, dim)
        prediction_logits = self.act(prediction_logits)  # (bs, seq_length, dim)
        prediction_logits = self.vocab_layer_norm(prediction_logits)  # (bs, seq_length, dim)

        #transform from (bs, seq_length, dim) to (bs, 1, dim) where 1 is the MASK
        if mask_only:
            indexes = tf.where(inputs['input_ids'] == mask_only).numpy()
            prediction_logits = get_mask_rep(prediction_logits, indexes)

        prediction_logits = self.vocab_projector(prediction_logits, ids_only=ids_only)

        loss = None if labels is None else self.compute_loss(labels, prediction_logits)

        if not return_dict:
            output = (prediction_logits,) + distilbert_output[1:]
            return ((loss,) + output) if loss is not None else output

        return TFMaskedLMOutput(
            loss=loss,
            logits=prediction_logits,
            hidden_states=distilbert_output.hidden_states,
            attentions=distilbert_output.attentions
        )

def get_mask_rep(score, indexes):
    """
    Objective: get the score only for the indexes of interest.

    Inputs:
        - score, np.array or tf.tensor: the scores for all vocabulary
        - indexes, np.array: the array indicating [MASK] positionning
    Outputs:
        - final_scores, np.array: the final scores but only for the tokens of interest
    """
    n = score.shape[0]

    embedded_mask = dict(zip(indexes[:, 0], indexes[:, 1]))

    final_scores = []

    for index in range(score.shape[0]):
        final_scores.append(score[index, embedded_mask.get(index), :].numpy().reshape(1, -1))

    return np.array(final_scores)
```

Second change - I would call it RVCS for reducing the vocabulary tensor and cuting the sequence:

The idea is to reduce the tensor of `_linear` method in the _TFEmbeddings_ to manipulates tensors of size (dim, n_tokens_of_interests) instead of tensors of (dim, vocab_size).

```python
class CB_TFEmbeddings(TFEmbeddings):

    def call(self, input_ids=None, position_ids=None, inputs_embeds=None,
             mode="embedding", training=False, ids_only=False):
        """Get token embeddings of inputs.
        Args:
            inputs: list of two int64 tensors with shape [batch_size, length]: (input_ids, position_ids)
            mode: string, a valid value is one of "embedding" and "linear".
        Returns:
            outputs: (1) If mode == "embedding", output embedding tensor, float32 with
                shape [batch_size, length, embedding_size]; (2) mode == "linear", output
                linear tensor, float32 with shape [batch_size, length, vocab_size].
        Raises:
            ValueError: if mode is not valid.

        Shared weights logic adapted from
            https://github.com/tensorflow/models/blob/a009f4fb9d2fc4949e32192a944688925ef78659/official/transformer/v2/embedding_layer.py#L24
        """
        if mode == "embedding":
            return self._embedding(input_ids, position_ids, inputs_embeds, training=training)
        elif mode == "linear":
            return self._linear(input_ids, ids_only)
        else:
            raise ValueError("mode {} is not valid.".format(mode))

    def _linear(self, inputs, ids_only):
        """Computes logits by running inputs through a linear layer.
        Args:
            inputs: A float32 tensor with shape [batch_size, length, hidden_size]
        Returns:
            float32 tensor with shape [batch_size, length, vocab_size].
        """

        batch_size = shape_list(inputs)[0]
        length = shape_list(inputs)[1]

        x = tf.reshape(inputs, [-1, self.dim])

        if ids_only:#reduce the word_embeddings vector to the token ids of interest
            vocab_size = len(ids_only)
            word_embeddings = tf.convert_to_tensor(self.word_embeddings.numpy()[ids_only, :])
        else:
            word_embeddings = self.word_embeddings
            vocab_size = self.vocab_size

        logits = tf.matmul(x, word_embeddings, transpose_b=True)

        return tf.reshape(logits, [batch_size, length, vocab_size])

class CB_TFDistilBertMainLayer(TFDistilBertMainLayer):
    config_class = DistilBertConfig

    def __init__(self, config, **kwargs):
        super().__init__(config, **kwargs)
        self.num_hidden_layers = config.num_hidden_layers
        self.output_attentions = config.output_attentions
        self.output_hidden_states = config.output_hidden_states
        self.return_dict = config.use_return_dict

        self.embeddings = CB_TFEmbeddings(config, name="embeddings")  # Embeddings
        self.transformer = TFTransformer(config, name="transformer")  # Encoder

class CB_TFDistilBertLMHead(TFDistilBertLMHead):

    def call(self, hidden_states, ids_only):
        hidden_states = self.input_embeddings(hidden_states, mode="linear", ids_only=ids_only)
        if ids_only: #if we focus only on some tokens, reduce the bias to those dimensions
            other_bias = tf.convert_to_tensor(self.bias.numpy()[ids_only])
            bias = other_bias
        else:
            bias = self.bias
        hidden_states = hidden_states + bias
        return hidden_states

class CB_TFDistilBertForMaskedLM(TFDistilBertForMaskedLM):

    def __init__(self, config, *inputs, **kwargs):
        super().__init__(config, *inputs, **kwargs)
        self.vocab_size = config.vocab_size

        self.distilbert = CB_TFDistilBertMainLayer(config, name="distilbert")
        self.vocab_transform = tf.keras.layers.Dense(
            config.dim, kernel_initializer=get_initializer(config.initializer_range), name="vocab_transform"
        )
        self.act = get_tf_activation("gelu")
        self.vocab_layer_norm = tf.keras.layers.LayerNormalization(epsilon=1e-12, name="vocab_layer_norm")
        self.vocab_projector = CB_TFDistilBertLMHead(config, self.distilbert.embeddings, name="vocab_projector")

    def call(
        self,
        inputs=None,
        attention_mask=None,
        head_mask=None,
        inputs_embeds=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None,
        labels=None,
        training=False,
        mask_only=False #If you just focus on the mask token id change the False by it (103 here)
        only_ids=False #Otherwise the tokens of interests as a list
    ):
        r"""
        labels (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`):
            Labels for computing the masked language modeling loss.
            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
            Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
            in ``[0, ..., config.vocab_size]``
        """

        return_dict = return_dict if return_dict is not None else self.distilbert.return_dict
        if isinstance(inputs, (tuple, list)):
            labels = inputs[7] if len(inputs) > 7 else labels
            if len(inputs) > 7:
                inputs = inputs[:7]
        elif isinstance(inputs, (dict, BatchEncoding)):
            labels = inputs.pop("labels", labels)

        distilbert_output = self.distilbert(
            inputs,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            training=training,
        )

        hidden_states = distilbert_output[0]  # (bs, seq_length, dim)
        prediction_logits = self.vocab_transform(hidden_states)  # (bs, seq_length, dim)
        prediction_logits = self.act(prediction_logits)  # (bs, seq_length, dim)
        prediction_logits = self.vocab_layer_norm(prediction_logits)  # (bs, seq_length, dim)

        #transform from (bs, seq_length, dim) to (bs, 1, dim) where 1 is the MASK
        if mask_only:
            indexes = tf.where(inputs['input_ids'] == mask_only).numpy()
            prediction_logits = get_mask_rep(prediction_logits, indexes)

        prediction_logits = self.vocab_projector(prediction_logits, ids_only=ids_only)

        loss = None if labels is None else self.compute_loss(labels, prediction_logits)

        if not return_dict:
            output = (prediction_logits,) + distilbert_output[1:]
            return ((loss,) + output) if loss is not None else output

        return TFMaskedLMOutput(
            loss=loss,
            logits=prediction_logits,
            hidden_states=distilbert_output.hidden_states,
            attentions=distilbert_output.attentions
        )

def get_mask_rep(score, indexes):
    """
    Objective: get the score only for the indexes of interest.

    Inputs:
        - score, np.array or tf.tensor: the scores for all vocabulary
        - indexes, np.array: the array indicating [MASK] positionning
    Outputs:
        - final_scores, np.array: the final scores but only for the tokens of interest
    """
    n = score.shape[0]

    embedded_mask = dict(zip(indexes[:, 0], indexes[:, 1]))

    final_scores = []

    for index in range(score.shape[0]):
        final_scores.append(score[index, embedded_mask.get(index), :].numpy().reshape(1, -1))

    return np.array(final_scores)
```

Thus after that I uploaded a notebook on Lambda and looked at the speed of each version of code and especially how many sentences could I process with the same GPU (gpu.1x.rtx6000):

First line is the approximative maximum number of sentences I could process at once. Next lines are the time to process _n_ sentences by batches of 64. If there is a cross it because we got the MOO error before reaching the total number of sentences.

|  Method |  Normal |     CS     |   RVCS   |
|:--------:|:--------:|:--------:|:--------:|
|max batch| 450 | 4500 | 4500  |
|900 stcs| 1.21 | 1.20 | 4.5 |
|30k stcs| x | 42.2 | 134.4 |
|40k tokens| x | 53.5 | 180.7 |
|50k tokens| x | x | 228.4 |
|60k tokens| x | x | 272.7 |

Well it is not much, but at least we can use the GPU for 40k sentences with a little change in the tensors shape and with decent performances. If we add the changes of reducing the tensors in the `_linear` method we reduce by 3 the performance but we can perform the process on more sentence with the same GPU.

Code for the experiment:

```python
import time

patterns = create_patterns('{} [SEP] it talks about [MASK])

verbalizers = ['Society', 'Science', 'Health',
               'Education', 'Computer', 'Sports',
               'Business', 'Entertainment',
               'Relationship', 'Politics']

verb_ids = get_token_id(verbalizers, tokenizer)

model = TFDistilBertForMaskedLM.from_pretrained("distilbert-base-multilingual-cased")
#model = CB_TFDistilBertForMaskedLM.from_pretrained("distilbert-base-multilingual-cased")

step=64
st = time.time()
for i in range(0, n, step):
    texts = apply_pattern(sentences[:step], patterns[1])

    inputs = tokenizer.batch_encode_plus(texts, return_tensors='tf',
                                 max_length=256, truncation=True, padding=True)

    score = model(inputs, return_dict=True)#, mask_only=103, ids_only=verb_ids)
print(time.time() - st)
```


Model card: T5-base fine-tuned on QuaRel

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


FRCNN in the LXMERT demo outputs different features when using a local image vs. an image from a URL

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@eltoto1219

## Information
Model I am using: LXMERT, more specifically FRCNN. The problem arises when using the demo.ipynb file given here: https://github.com/huggingface/transformers/blob/master/examples/lxmert/demo.ipynb

## To reproduce

1. Change the URL to the filename of a local image. Both the visualization and the extracted features are affected by this.

## Expected behavior
I noticed that the order of Red, Green, Blue is changed in this line in the utils file: https://github.com/huggingface/transformers/blob/7abc1d96d114873d9c3c2f1bc81343fb1407cec4/examples/lxmert/utils.py#L552

When I comment out this line, the visualization and the features output by the model for a local image are the same as when we use its URL without commenting this line out.

I would appreciate if you can confirm that commenting out that line does not cause other issues in LXMERT so that I can use local files.

Best,
Ece


Measuring time when using xla_spawn on multiple cores

Hi,
I would like to measure the time of the program, when this is launched in distributed way on mutiple TPU cores with xla_spawn, I tried to write some callback like this which retunrs the same results with num_cores = 1 and 8, I am not sure how to handle it when this is distributed? I want to make sure pytorch XLA is using all cores indeed, thank you for your help.

```
class PrinterTimeCallback(TrainerCallback):

    #A bare :class:`~transformers.TrainerCallback` that just prints the logs.

    def on_epoch_begin(self, args, state, control, **kwargs):
        """
        Event called at the beginning of an epoch.
        """
        if state.is_local_process_zero:
            self.start_time = time.time()

    def on_epoch_end(self, args, state, control, **kwargs):
        """
        Event called at the end of an epoch.
        """
        if state.is_local_process_zero:
            self.end_time = time.time()
            total_time = self.end_time - self.start_time
            print("total_time ", total_time)

```


Flax/Jax documentation

Integrating Flax & JAX into transformers documentation


Docs bart training ref

# What does this PR do?
Following the discussion on #7828 with @sshleifer and others I created a minimal example on the forum and added a reference to the Bart documentation. Let me know if this works for you.


[Seq2SeqDataCollator] dont pass add_prefix_space=False to all tokenizers

T5Tokenizer will warn about an unrecognized kwarg, this fixes that error.

cc @patil-suraj


Return raw outputs in TextClassificationPipeline

Currently the `TextClassificationPipeline` does a softmax over the output values when `num_labels > 1`, and does a sigmoid over the output when `num_labels == 1`.

As seen in https://github.com/huggingface/transformers/issues/8259, this may be problematic when systems depend on a different output range.

Adding a flag `return_raw_outputs` that will not apply the sigmoid or softmax in that case.

closes #8259


Create README.md

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Muti GPU training with torch==1.7.0 not working

Hi,
muti GPU training with torch==1.7.0 is not working.

I have the following configuration:
- CUDA 11.0
- V100 GPU (1 or 4)
- Torch 1.7.0 (1.6.0)
- transformers 3.4.0
- Linux

When I train with one GPU all works. When I run it with 4 GPUs the CPU load goes up to 100% on one core like 1,5 MB memory is used from each GPU but the training does not start. When I downgrade torch to Version 1.6.0 everything works.

Might be an upstream problem but you might want to warn users from using torch 1.7.0

PS: Do you think I should report this at the torch repo?


Adding call back to measure time of each step

Hi
I need to measure the time of each training step, with distributed version of seq2seq_trainer with TPUs, could you provide me with some hints how to measure the time? thanks


Model versioning

Related to pretrained model cards

**Write-up about the context for this change, and the enabled features is at https://discuss.huggingface.co/t/announcement-model-versioning-upcoming-changes-to-the-model-hub/1914**

In short:

1. changes to the file downloading code used in `from_pretrained()` methods to use the new file URLs backed by huggingface-hosted git repos.
2. changes to the model upload CLI to create a model repo then be able to git clone and git push to it.


AlbertTransformer head_mask not subscriptable error when not passed

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Linux version 4.4.0-193-generic / ubuntu
- Python version: 3.6.5
- PyTorch version (GPU?): 1.4.0 (cuda 10.0)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.\
-->
@LysandreJik

## Information
AlbertTransformer has `head_mask` arguments with default value of None. [(modeling_albert.py#L416)](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_albert.py#L416)
But in forward, `head_mask` is indexed by group_idx without checking if head_mask is None. [(modeling_albert.py#L436)](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_albert.py#L436)
So I got the error as below.
(it seems like a similar issue [#1188](https://github.com/huggingface/transformers/issues/1188))

The problem arises when using:
* [-] the official example scripts: (give details below)
* [O] my own modified scripts: (give details below)

The tasks I am working on is:
* [-] an official GLUE/SQUaD task: (give the name)
* [O] my own task or dataset: (give details below)

## To reproduce
```python
from transformers import AlbertTokenizer, AlbertModel

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertModel.from_pretrained('albert-base-v2', return_dict=True)

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
inputs2 = model.embeddings(input_ids=inputs.input_ids,
                 token_type_ids=inputs.token_type_ids,
                 position_ids=None,
                 inputs_embeds=None)
model.encoder(inputs2)
```
error message when i execute the above script.
```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-4fc3cd45df51> in <module>()
      9                  position_ids=None,
     10                  inputs_embeds=None)
---> 11 model.encoder(inputs2)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_albert.py in forward(self, hidden_states, attention_mask, head_mask, output_attentions, output_hidden_states, return_dict)
    434                 hidden_states,
    435                 attention_mask,
--> 436                 head_mask[group_idx * layers_per_group : (group_idx + 1) * layers_per_group],
    437                 output_attentions,
    438                 output_hidden_states,

TypeError: 'NoneType' object is not subscriptable
```

## Expected behavior
`model.encoder(inputs2)` encoder will be output of tensor.
Espeically, shape of the tensor will be (1, 8, 768)

Personally, I think we could add one more line of code before `for loop`
```python
head_mask = [None] * self.config.num_hidden_layers if head_mask is None else head_mask
```


Keyword arguments {'add_prefix_space': False} not recognized.

Hi
I am getting  this message when running finetune_trainer.py on TPUs any idea how to remove it? thanks


Hi
I am running finetune_trainer.py on cloud with TPUs, here is the error, I appreciate your help thanks
```json
{
  "textPayload": "Traceback (most recent call last):\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 330, in _mp_start_fn\n    _start_fn(index, pf_cfg, fn, args)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 324, in _start_fn\n    fn(gindex, *args)\n  File \"/workdir/seq2seq/finetune_trainer.py\", line 303, in _mp_fn\n    app.run(main, flags_parser=parse_flags)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py\", line 300, in run\n    _run_main(main, args)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py\", line 251, in _run_main\n    sys.exit(main(argv))\n  File \"/workdir/seq2seq/finetune_trainer.py\", line 246, in main\n    data_args=data_args,\n  File \"/workdir/seq2seq/seq2seq_trainer.py\", line 37, in __init__\n    super().__init__(*args, **kwargs)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py\", line 318, in __init__\n    self.control = self.callback_handler.on_init_end(self.args, self.state, self.control)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer_callback.py\", line 325, in on_init_end\n    return self.call_event(\"on_init_end\", args, state, control)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer_callback.py\", line 376, in call_event\n    **kwargs,\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/integrations.py\", line 213, in on_init_end\n    self.tb_writer = SummaryWriter(log_dir=args.logging_dir)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py\", line 221, in __init__\n    self._get_file_writer()\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py\", line 252, in _get_file_writer\n    self.flush_secs, self.filename_suffix)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py\", line 62, in __init__\n    log_dir, max_queue, flush_secs, filename_suffix)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/tensorboard/summary/writer/event_file_writer.py\", line 77, in __init__\n    tf.io.gfile.makedirs(logdir)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py\", line 673, in makedirs\n    return get_filesystem(path).makedirs(path)\ntensorboard.compat.tensorflow_stub.errors.AlreadyExistsError: Directory already exists\n",
  "insertId": "5rl5rhwn8m5sobp9q",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "seq2seq",
      "location": "europe-west4-a",
      "project_id": "try-ideas-for-rmi",
      "namespace_name": "ruse-xgcp",
      "cluster_name": "xcloud-v3-donut-europe-west4-a",
      "pod_name": "20201105.df.e2753.0-7mxdc"
    }
  },
  "timestamp": "2020-11-05T11:28:19.975413564Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/job-name": "20201105.df.e2753.0",
    "k8s-pod/controller-uid": "4974cfc1-b3df-4882-9fd1-9095f4a944d9",
    "k8s-pod/jobowner": "ruse-xgcp",
    "k8s-pod/app": "xcloud",
    "k8s-pod/serviceName": "xc-20201105-df-e2753-0"
  },
  "logName": "projects/try-ideas-for-rmi/logs/stderr",
  "receiveTimestamp": "2020-11-05T11:28:23.810886436Z"
}
```


Corrected tpu typo in examples readme

 Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

 documentation: @sgugger

**EDIT**: Tests fail but that is definitely not due to my change


Does Tokenizer provide parameters to split the number？

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I want to split "2004年06月25日" into [2, 0, 0, 5, 年, 0, 6, 月, 2, 5,日],not [2005, 年,  06, 月, 25, 日].
How can i do it the easiest?
now I use these API :
BertTokenizer
BertTokenizerFast
tokenizer.tokenize("2004年06月25日")
Thanks.

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


[s2s] test_bash_script.py - actually learn something

As discussed in https://github.com/huggingface/transformers/issues/6049 this PR replaces the original `test_train_mbart_cc25_enro_script` test with mostly new guts doing the following:

* [x] switches to downloading and caching a bigger custom dataset, which is a subset of the full wmt_en_ro.  I created https://cdn-datasets.huggingface.co/translation/wmt_en_ro-tr40k-va0.5k-te0.5k.tar.gz - hope the name is intuitive - self-documenting. It's just 3.6M (vs 56M original). I made it using this script: https://github.com/stas00/porting/blob/master/transformers/translation/make-wmt_en_ro-subset.md

* [x] performs qualitative checks on the eval/test results:
   - Minimum learning requirement 1: BLEU improves over the course of training by more than 2 pts
   - Minimum learning requirement 2: BLEU finishes above 17
   - Minimum learning requirement 3: test bleu and val bleu within ~1 pt.

also makes some extra improvements:
* [x] uses `require_torch_gpu` decorator
* [x] removes hardcoded paths

Runtime is about ~2min on unoptimized rtx-3090.

Fixes: #6049

@sshleifer


FutureWarning: This config doesn't use attention memories, a core feature of XLNet even though I'm using mem_len

So every time I try to run xl-net using the run_generation.py script, I get the warning : `FutureWarning: This config doesn't use attention memories, a core feature of XLNet. Consider setting mem_len to a non-zero value, for example xlnet = XLNetLMHeadModel.from_pretrained('xlnet-base-cased'', mem_len=1024), for accurate training performance as well as an order of magnitude faster inference.`

The issue is that I am using mem_len, I'm setting it to 384 (have tried other values as well) but I still get this warning.


No loss in model output for TFElectraForPreTraining, even though labels are provided.
This is from docs for ELECTRA:
`loss (optional, returned when labels is provided, tf.Tensor of shape (1,)) – Total loss of the ELECTRA objective.`

The problem is that I want to get loss and logits, but model outputs only logits.

My code snippet:
```
>>> import numpy as np
>>> from transformers import ElectraTokenizer, TFElectraForPreTraining
>>>
>>> model = TFElectraForPreTraining.from_pretrained('google/electra-base-discriminator')
All model checkpoint layers were used when initializing TFElectraForPreTraining.

All the layers of TFElectraForPreTraining were initialized from the model checkpoint at google/electra-base-discriminator.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFElectraForPreTraining for predictions without further training.
>>> tokenizer = ElectraTokenizer.from_pretrained('google/electra-base-discriminator')
>>> inputs = tokenizer('simple phrase', return_tensors='tf')
>>> labels = np.array([0] * 4)
>>> out = model(inputs, labels=labels)
>>> len(out)
1
```


[s2s] test_distributed_eval

This PR:

* [x] adds a helper function `get_gpu_count()`, which returns the number of available gpus (regardless of whether torch or tf is used) (otherwise we have to do all the `if _torch_available: import torch` rigmarole.
* [x] adds a basic test for `run_distributed_eval.py`

Things I wasn't sure about:
- I put the new test into `test_seq2seq_examples_multi_gpu.py`, but it doesn't require multigpu. should I create a new test file  `test_seq2seq_examples_gpu.py` or just leave it here for now and refactor once we have more tests like this?
- I made it `@slow`, but wasn't 100% sure - it takes some 25s - probably too slow.

Fixes: #8297

@sshleifer


[QA examples] fix inconsistent tokenization in _improve_answer_span

Basically, I am trying to fix the QA preprocessing steps when using BPE-based tokenizers (Roberta, Bart, Longformer)

The prior _improve_answer_span has a bug because the way it tokenizes the answer_text is different from the way context is handled. So when the original context is '1987', and the old input ans_span is ['Ġ1987', ','], then the ',' won't be removed.

To fix these cases I use two cleaned forms built by either including prefix_space or not. If a sub_answer_span matches either one of them we'll improve the answer_span. The first form [line 42] is used to handle things with leading punctuations, e.g., orig_answer_context=="73 million" and ans_span is ["Ġ$", "73", "Ġmillion"] (should be cleaned to  ["73", "Ġmillion"] ). The second form is used to handle things with ending punctuations, e.g., the '1987' example mentioned above.


OOMKilled with exit code 137 with finetune_trainer.py

Hi
I am trying to train on WMT with TPUs using seq2seq_trainer.py, and I always get "OOMKilled with exit code 137" on the line when it tries to use xm.spawn to run the main function on multiple TPUs. I decreased the batch size as much as possible, still the error is there. Do you have an idea why this is happening? thank you. I appreciate your help.


Create README.md

Related to pretrained model cards

model card for  ktrapeznikov/gpt2-medium-topic-news


error  'ascii' codec can't decode byte 0xc3 in position 6550: ordinal not in range(128)\n", when running finetune_trainer.py on multiple tpus

Hi
I am running finetune_trainer.py on multiple tpus getting the following error, thank you for your help
{
  "textPayload": "Traceback (most recent call last):\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 330, in _mp_start_fn\n    _start_fn(index, pf_cfg, fn, args)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 324, in _start_fn\n    fn(gindex, *args)\n  File \"/workdir/seq2seq/finetune_trainer.py\", line 299, in _mp_fn\n    app.run(main, flags_parser=parse_flags)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py\", line 300, in run\n    _run_main(main, args)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py\", line 251, in _run_main\n    sys.exit(main(argv))\n  File \"/workdir/seq2seq/finetune_trainer.py\", line 200, in main\n    if training_args.do_train\n  File \"/workdir/seq2seq/utils.py\", line 128, in __init__\n    self.src_lens = self.get_char_lens(self.src_file)\n  File \"/workdir/seq2seq/utils.py\", line 147, in get_char_lens\n    return [len(x) for x in Path(data_file).open().readlines()]\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6550: ordinal not in range(128)\n",
  "insertId": "5rl5rhwn8m5snsv0h",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "namespace_name": "ruse-xgcp",
      "project_id": "try-ideas-for-rmi",
      "pod_name": "20201104.seq2seq.7685c.0-md6t7",
      "container_name": "seq2seq",
      "location": "europe-west4-a",
      "cluster_name": "xcloud-v3-donut-europe-west4-a"
    }
  },
  "timestamp": "2020-11-04T23:53:22.599603005Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/jobowner": "ruse-xgcp",
    "k8s-pod/serviceName": "xc-20201104-seq2seq-7685c-0",
    "k8s-pod/controller-uid": "32981a05-9c69-4234-b64e-6750f0afde11",
    "k8s-pod/app": "xcloud",
    "k8s-pod/job-name": "20201104.seq2seq.7685c.0"
  },
  "logName": "projects/try-ideas-for-rmi/logs/stderr",
  "receiveTimestamp": "2020-11-04T23:53:27.836579865Z"
}


Running seq2seq_trainer with iterable datasets

Hi
Looking into https://github.com/huggingface/transformers/blob/master/examples/seq2seq/seq2seq_trainer.py line 117, if the dataset is iterative, then the code does not return a distributed sampler so in case one wants to train on TPUs on multiple cores with iterable datasets, I was wondering if you could assist me how I can use the codes for iterable datasets on TPUs
thanks


examples/docs: caveat that PL examples don't work on TPU

"Note that this approach does not work for examples that use `pytorch-lightning`."


Clean up data collators and datasets

# What does this PR do?

This PR cleans a bit the `DataCollatorForLanguageModeling` by:
- making sure it keeps additional labels (such a next sentence labels)
- use the `tokenizer.pad` function when possible, rewrite the function that was doing the padding when it's not (to handle padding on the left as well as the right side)
- take advantage of the `special_tokens_mask` that the tokenizer can return, to avoid unnecessary conversions tensor -> list -> tensor again

As a result it deprecates `DataCollatorForSOP` (which can be replaced by `DataCollatorForLanguageModeling`) and changes the elements of `TextDatasetForNextSentencePrediction` to work with `DataCollatorForLanguageModeling`. That dataset change renders `DataCollatorForNextSentencePrediction` unusable, so it's removed (breaking change, though this is relatively contained, users that were using it should just use `DataCollatorForLanguageModeling` instead).

Tests are adapted to check `DataCollatorForLanguageModeling` does work for all those tasks.

In passing, all text datasets get their deprecation warning as we are now encouraging users to move the datasets library.


run_mlm.py: error: argument

I just found today that transformers/examples/language-modeling/run_language_modeling.py"  has now been changed to transformers/examples/language-modeling. Everything fine for me when was using run_language_modeling.py. But with the new change, I am getting the error that, **run_mlm.py: error: argument --mlm_probability: expected one argument** I could not find the reason despite spending many hours on it.

!python "/content/transformers/examples/language-modeling/run_mlm.py" \
    --output_dir "/content/drive/My Drive/Ottawa_citit" \
    --model_type roberta \
    --model_name_or_path roberta-base \
    --do_train \
    --per_gpu_train_batch_size 16 \
    --seed 42 \
    --train_data_file "/content/input_text.txt" \
    --block_size 512 \
    --line_by_line \
    --learning_rate 6e-4 \
    --num_train_epochs 4 \
    --save_total_limit 2 \
    --run_name high_mlm_prob \
    --save_steps 200 \
    --mlm\
    --weight_decay 0.01 \
    --mlm_probability 0.15

Thanks!!


Tokenizers save_pretrained broken when defining vocab and merges file arguments (v3.1)

## Problem Information

Since the upgrade to transformers v3, it appears that issues arise when defining a RoBERTa or BART tokenizer using the vocab_file and merges_file arguments.

Previously defining the tokenizer, by supplying these two arguments resulted in a correctly configured tokenizer and enabled the tokenizer to be saved using the standard .save_pretrained() function. However, since the upgrade to v3, it appears that using such arguments causes the tokenizer to default to incorrect "init_kwargs".

## Replicating the problem

```py
from transformers import RobertaTokenizer

#load the tokenizer using the standard method - results in correct init_kwargs
correct_tokenizer = RobertaTokenizer.from_pretrained("roberta-large")

#save the tokenizer so the vocab and merges file can be read in from disk
correct_tokenizer.save_pretrained("file_path/roberta_tokenizer")

#load the saved vocab and merges files to create a new tokenizer with the same properties
broken_tokenizer = RobertaTokenizer(
          vocab_file="file_path/roberta_tokenizer/vocab.json",
          merges_file = "file_path/roberta_tokenizer/merges.txt"
          )

#attempt to save the second tokenizer using the same function as above
broken_tokenizer.save_pretrained("file_path/saved_folder")
```

In the above script, loading the tokenizer using the ".from_pretrained('roberta-large')" method results in a tokenizer with the correct properties. The tokenizer's init_kwargs are as expected, and in the form of a dictionary as follows:

- merges_file: "file/path/...",
- model_max_length: 512,
- vocab_file: "file/path/...",

This tokenizer can be saved using the ".save_pretrained()" function as intended.

However, when you load a tokenizer, while defining the "vocab_file=vocab.json" and "merges_file=merges.txt", as is the case with "broken_tokenizer", it appears that the init_kwargs defaults to incorrect properties. Unlike before the init_kwargs now present a dictionary with no mention of model configurations, but instead includes the following:

- bos_token: AddedToken(bos_token, lstrip=False, rstrip=False)
- eos_token: AddedToken(eos_token, lstrip=False, rstrip=False)
- sep_token: AddedToken(sep_token, lstrip=False, rstrip=False)
- cls_token: AddedToken(cls_token, lstrip=False, rstrip=False)
- unk_token: AddedToken(unk_token, lstrip=False, rstrip=False)
- pad_token: AddedToken(pad_token, lstrip=False, rstrip=False)

## Error identification

The error in saving appears to arise when you reach the following section of the "save_pretrained()" function within tokenization_utils_base:

```py
tokenizer_config = copy.deepcopy(self.init_kwargs)
if len(self.init_inputs) > 0:
    tokenizer_config["init_inputs"] = copy.deepcopy(self.init_inputs)
for file_id in self.vocab_files_names.keys():
    tokenizer_config.pop(file_id, None)

with open(tokenizer_config_file, "w", encoding="utf-8") as f:
    f.write(json.dumps(tokenizer_config, ensure_ascii=False))
```

In the scenario where the tokenizer has the correct kwargs, the vocab_file and merges_file are popped from "tokenizer_config", leaving a tokenizer config that can be saved in JSON format.

However, where the incorrect kwargs are configured, as is the case with "broken_tokenizer" above, the resulting "tokenizer_config" contains AddedToken objects, and therefore results in the following error:
**"TypeError: Object of type AddedToken is not JSON serializable"**

## Versioning
Transformers = 3.1
Tokenizers = 0.8.1rc2