Does tokenizer.from_pretrained tokenize text on CPU even a GPU is available?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
For a tokenizer from `tokenizer = AutoTokenizer.from_pretrained(model_name)`, if `tokenizer.encode_plus(text)` works on CPUs even a GPU is available. I tried to run such code on a AWS GPU machine instance, but found GPUs are totally not used.
Thanks.


TFBertMode.pre_trained('bert-base-uncased') --> OSError

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform:  Linux-3.10.0-1127.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version:  3.7.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?):  2.3.0 (True)
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

Question :
   I wanted to see the pretrained bert model summary,
   So I opened Jupyter notebook on my computer installed Quadro RTX 5000 GPUs ,
   and typed the following code to load  pretrained bert model using TFBertModel.from_pretrained() function.
   After running cell, but I got  Error messages...

   --- test codes  ---
   from transformers import TFBertModel

   encoder = TFBertModel.from_pretrained('bert-base-uncased')
   ---  end of test codes ---

   ---- error messages start ---
OSError                                   Traceback (most recent call last)
~/anaconda3/envs/tf23/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    354                 resume_download=resume_download,
--> 355                 local_files_only=local_files_only,
    356             )

~/anaconda3/envs/tf23/lib/python3.7/site-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
    729         # File, but it doesn't exist.
--> 730         raise EnvironmentError("file {} not found".format(url_or_filename))
    731     else:

OSError: file bert-base-uncased/config.json not found

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-4-c74bfe775797> in <module>
      1 from transformers import TFBertModel
      2
----> 3 encoder = TFBertModel.from_pretrained('bert-base-uncased')

~/anaconda3/envs/tf23/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    543                 proxies=proxies,
    544                 local_files_only=local_files_only,
--> 545                 **kwargs,
    546             )
    547         else:

~/anaconda3/envs/tf23/lib/python3.7/site-packages/transformers/configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    313
    314         """
--> 315         config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
    316         return cls.from_dict(config_dict, **kwargs)
    317

~/anaconda3/envs/tf23/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    366                 f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a {CONFIG_NAME} file\n\n"
    367             )
--> 368             raise EnvironmentError(msg)
    369
    370         except json.JSONDecodeError:

OSError: Can't load config for 'bert-base-uncased'. Make sure that:

- 'bert-base-uncased' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert-base-uncased' is the correct path to a directory containing a config.json file

   ----- end of error messages ---

  I also tested above codes in Colab, In Colab,  the above code worked  well without errors.

  Please, let me know how to solve this problem..
  Thanks in advance


Support T5 Distillation w/hidden state supervision

Support distilling t5 for summarization and translation with hidden state supervision.

cc @patil-suraj @patrickvonplaten

Here are some very simple commands that work for now:

### Yes Teacher/Traditional Distillation
```bash
python distillation.py --teacher t5-small --data_dir cnn_dm \
--student_decoder_layers 3 --student_encoder_layers 6 --tokenizer_name t5-small \
--learning_rate=3e-4 --freeze_encoder --no_teacher --freeze_embeds \
--do_train --train_batch_size 32 \
--do_predict \
--model_name_or_path t5-small --eval_beams 2 --eval_max_gen_length 142 \
--val_check_interval 0.25 --n_val 1000 \
--output_dir distilt5 --gpus 1 --logger_name wandb

```

### No teacher

```bash
python make_student.py t5-small t5_small_6_3 6 3

python finetune.py --model_name_or_path t5_small_6_3 --data_dir cnn_dm \
--learning_rate=3e-4 --freeze_encoder --freeze_embeds \
--do_train --train_batch_size 32 \
--do_predict \
--model_name_or_path t5_small_6_3 --eval_beams 2 --eval_max_gen_length 142 \
--val_check_interval 0.25 --n_val 1000 \
--output_dir distilt5 --gpus 1 --logger_name wandb
```


Docker GPU Images: Add NVIDIA/apex to the cuda images with pytorch

# What does this PR do?
- Use cuda:10.2 image instead of 10.1 (to address version mismatch  warning with pytorch)
- Use `devel` version that is built on the `runtime` and includes headers and development tools (it was otherwise failing to build apex). For a description of the different flavors, see:  https://hub.docker.com/r/nvidia/cuda -> Overview of Images
- Download and build `apex` for pytorch. https://github.com/NVIDIA/apex#quick-start

## Docs
- https://github.com/NVIDIA/apex
- https://nvidia.github.io/apex/
- https://hub.docker.com/r/nvidia/cuda

## Who can review?
- @mfuntowicz co-authored the Dockerfiles in 71c87119


Enhance TFTrainer.save_model()

# What does this PR do?

Currently, `TFTrainer.save_model()` raises errors if the model is not `TFPreTrainedModel` . However `Trainer` works fine with `torch.nn.modules.Module`.

This is a step to make TFTrainer work with usual `tf.keras.models.Model` models. The idea (from @sgugger) is that a user is building their own models that work like ours (e.g., return the loss as the first output) and can train them with Trainer.

Furthermore, a SavedModel is also saved using `tf.saved_model.save()`.

I tried to avoid duplicated code (check and create output directory before saving), and therefore there is a new method `save_tf_model()` in `modeling_tf_utils`, which is used in `trainer_tf.py`.

For @jplu and @sgugger .


Trainer callbacks

# What does this PR do?

This PR does two things: clean up a bit the files supporting `Trainer` and the `Trainer` class, and add callbacks to `Trainer`.

### Callbacks

This PR introduces a new class called `TrainerCallback` that can access the current state of the training loop and make some decisions (shown in the `TrainerControl` object). This allows us to isolate the pieces of code that do log-reporting on the various ML platforms or report progress in another file and clean up the code of the main `train` method of the `Trainer`.

This way, any new platform we want to integrate with for log-reporting or new behavior (like early stopping) can be implemented in a Callback while `Trainer` focuses on the main aspects of actual training, with or without mixed precision, on one or several GPUs/TPUs.

As an example, integrations with TensorBoard, Wandb and ComeML are moved to the `integrations` module in clean callbacks, while the control flow of logs/saves/evaluations as well as progress reporting are moved to the `trainer_callback` file.

Most of the behavior stays the same as this PR essentially moves code around, but there are a few API changes:
- deprecating the `tb_writer` argument in `Trainer` (with full backward compatibility), people should now use the `TensorBoardCallback`.
- a new `callbacks` argument in the `Trainer` init and new `add_callback`, `pop_callback` and `remove_callback` for the `Trainer`. For all of those, you can either pass an instance of a callback or a callback class.
- Cleaned up the progress bars a bit with only one main progress bar over all the steps we will do for training and evaluation bars that disappear after being done

### Progress bars

Here is the new progress bar behavior in console mode (checked in single and multi GPU envs, to make sure only one progress bar is displayed/logs are only printed once):

![](https://i.ibb.co/Fq60bFS/console-progress.png)

and in a jupyter notebook:

![](https://i.ibb.co/kSzs1fC/notebook-progress.png)

### General cleanup

Not directly related to this PR, but related to the general cleanup of `Trainer`, I moved a bit of stuff around: moved the utils at the start of `Trainer` to a new `trainer_utils_pt`. This way `trainer_utils` can be about the general training utils that work on both PyTorch and TensorFlow, and I moved the ones specific to PyTorch to `trainer_utils_pt`.

Also in `Trainer`, the code for logs, save and evaluation ended being duplicated between the end of a training step and the end of an epoch, so I put it in its private method to improve readability.


change return dicitonary for DataCollatorForNextSentencePrediction from masked_lm_labels to labels

# What does this PR do?

The `masked_lm_labels` argument from  *DataCollatorForNextSentencePrediction*  is deprecated and will be removed in a future version, use `labels` instead. I changed the dictionary key from `masked_lm_labels ` to `labels`.

It will avoid any future errors when `masked_lm_labels` won't be used anymore.

Not a lot of people use the  *DataCollatorForNextSentencePrediction* and I think this will get overlooked in the future if not fixed.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

It fixes warning that appear when using `trainer`

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case. **Not the case**
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation). **Not needed.**
- [x] Did you write any new necessary tests? **Was not needed, the change that I made is very minor.**

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


RagTokenForGeneration.from_pretrained fails while running demo script

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux-4.4.0-1113-aws-x86_64-with-debian-stretch-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@VictorSanh
@patrickvonplaten
@sshleifer
transformers/modeling_utils.py
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): RAG

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. install new conda env py=3.7
2. install RAG requirements
3. run example code from https://huggingface.co/transformers/master/model_doc/rag.html
```python
Python 3.7.9 (default, Aug 31 2020, 12:42:55)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration
>>> import torch
>>> tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
>>> retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True)
Using custom data configuration dummy.psgs_w100.nq.no_index
Reusing dataset wiki_dpr (/homes/thielk/.cache/huggingface/datasets/wiki_dpr/dummy.psgs_w100.nq.no_index/0.0.0/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2)
Using custom data configuration dummy.psgs_w100.nq.exact
Reusing dataset wiki_dpr (/homes/thielk/.cache/huggingface/datasets/wiki_dpr/dummy.psgs_w100.nq.exact/0.0.0/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2)
>>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
```

stack trace:

```python
Traceback (most recent call last):
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 187, in nti
    n = int(s.strip() or "0", 8)
ValueError: invalid literal for int() with base 8: 'del.embe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 2289, in next
    tarinfo = self.tarinfo.fromtarfile(self)
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 1095, in fromtarfile
    obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 1037, in frombuf
    chksum = nti(buf[148:156])
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 189, in nti
    raise InvalidHeaderError("invalid header")
tarfile.InvalidHeaderError: invalid header

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/site-packages/torch/serialization.py", line 595, in _load
    return legacy_load(f)
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/site-packages/torch/serialization.py", line 506, in legacy_load
    with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar, \
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 1593, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 1623, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 1486, in __init__
    self.firstmember = self.next()
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/tarfile.py", line 2301, in next
    raise ReadError(str(e))
tarfile.ReadError: invalid header

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/site-packages/transformers/modeling_utils.py", line 927, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/site-packages/torch/serialization.py", line 426, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/site-packages/torch/serialization.py", line 599, in _load
    raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))
RuntimeError: /homes/thielk/.cache/torch/transformers/06fe449ffe41cbe16aeb1f5976989313464a3c44a605e9a8b91bf6440dfa6026.696574d8c17eafbac08f43f01e951252057f8feb133b64a33b76d4c47d65367a is a zip archive (did you mean to use torch.jit.load()?)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/homes/thielk/miniconda3/envs/transformers-pytorch/lib/python3.7/site-packages/transformers/modeling_utils.py", line 930, in from_pretrained
    "Unable to load weights from pytorch checkpoint file. "
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
be able to completely run example code from RAG documentation
May be related to #7583
<!-- A clear and concise description of what you would expect to happen. -->
```python
from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration
import torch
tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True)
# initialize with RagRetriever to do everything in one forward call
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
input_dict = tokenizer.prepare_seq2seq_batch("How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="pt")
input_ids = input_dict["input_ids"]
outputs = model(input_ids=input_ids, labels=input_dict["labels"])
# or use retriever seperately
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", use_dummy_dataset=True)
# 1. Encode
question_hidden_states = model.question_encoder(input_ids)[0]
# 2. Retrieve
docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt")
doc_scores = torch.bmm(question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2)).squeeze(1)
# 3. Forward to generator
outputs = model(context_input_ids=docs_dict["context_input_ids"], context_attention_mask=docs_dict["context_attention_mask"], doc_scores=doc_scores, decoder_input_ids=input_dict["labels"])
# or directly generate
generated = model.generate(input_ids=input_dict["input_ids"])
generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True)
```


[bart] fix config.classif_dropout


- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-118-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.8
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Just run attached script

## Expected behavior
I dont have '-1' masks with my labels but I get those warnings.
Expected behavior not to have those warnings when I make training.

<!-- A clear and concise description of what you would expect to happen. -->
`python3.6/site-packages/tensorflow/python/autograph/impl/api.py:493: UserWarning: Using `-1` to mask the loss for the token is deprecated. Please use `-100` instead.`
  return py_builtins.overload_of(f)(*args)

Code that reproduces the issue:
```
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

import transformers
import numpy as np
import tensorflow as tf
from transformers import BertConfig, TFTrainer, TFTrainingArguments, TFBertForTokenClassification

transformers.logging.set_verbosity_info()

labels = np.ones((32, 18))
labels_as_tensor = tf.convert_to_tensor(
    labels, dtype=tf.int32, dtype_hint=None, name=None
)

inputs_embeds = np.random.normal(size=(32, 18, 768))
inputs_embeds_as_tensor = tf.convert_to_tensor(
    inputs_embeds, dtype=tf.float32, dtype_hint=None, name=None
)

token_type_ids = np.ones((32, 18))
token_type_ids_as_tensor = tf.convert_to_tensor(
    token_type_ids, dtype=tf.int32, dtype_hint=None, name=None
)

batch = ({
'inputs_embeds': inputs_embeds_as_tensor,
'token_type_ids': token_type_ids_as_tensor
}, labels_as_tensor)

training_args = TFTrainingArguments(output_dir='~/tensorboard',
                                    overwrite_output_dir=True,
                                    learning_rate=0.001,
                                    logging_dir='~/tensorboard',
                                    debug=True,
                                    do_train=True,
                                    do_predict=True,
                                    num_train_epochs=2,
                                    per_device_train_batch_size=32,
                                    per_device_eval_batch_size=32,
                                    save_total_limit=3,
                                    evaluate_during_training=True,
                                    eval_steps=5)

with training_args.strategy.scope():
    config = BertConfig(num_labels=1274, output_hidden_states=False, num_hidden_layers=3)
    model = TFBertForTokenClassification(config)

trainer = TFTrainer(model=model, args=training_args)
trainer.train_loss = tf.keras.metrics.Sum()
trainer.create_optimizer_and_scheduler(20)
trainer.distributed_training_steps(batch)

```


BartConfig saving and loading inconsistency

## Environment info

### Who can help

 Bart: @sshleifer

## Information

Model I am using (Bert, XLNet ...): Bart

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import BartConfig
config_to_save = BartConfig.from_pretrained('facebook/bart-base', classif_dropout=0.42)
config_to_save.save_pretrained('./')
config_loaded = BartConfig.from_pretrained('./')
assert config_to_save.classif_dropout == config_loaded.classif_dropout, "what?"
```

## Expected behavior

Should raise no error.


Update README.md

model card

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


run_language_modeling.py TPU issue during evaluation

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: ubuntu
- Python version: 3.7
- PyTorch version (GPU?): xla-nightly
- Tensorflow version (GPU?):
- Using GPU in script?: no TPU
- Using distributed or parallel set-up in script?: yes

### Who can help
@sgugger @LysandreJik

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
[*] the official example scripts: (give details below)
run_language_modeling.py
The tasks I am working on is:
[*] my own task or dataset: (give details below)
txt file for pretraining Roberta model
## To reproduce
I am trying to launch a pretraining using run_language_modeling.py with TPU unfortunately I got some issue during the evaluation and logging_steps with error message

1. `python run_language_modeling.py --model_name_or_path="roberta-base"  --model_type="roberta" --tokenizer_name="roberta-base" --do_train --evaluate_during_training --mlm --mlm_probability=0.15 --train_data_file="train.txt" --eval_data_file="val.txt" --do_eval --per_device_train_batch=8 --per_device_eval_batch=8 --output_dir="robertaweet" --max_steps=5000000 --logging_dir="log_bertweet" --logging_steps=20 --eval_steps=10 --save_steps=25 --dataloader_num_workers=0 --tpu_num_cores=8  --learning_rate=1e-4`
2.
3.

`020-10-05 15:36:48.946219: W   11372 tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:160] RPC failed with status = "Unavailable: Socket closed" and grpc_error_string = "{"created":"@1601912208.946077389","description":"Error received from peer ipv4:10.255.226.90:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}", maybe retrying the RPC`
## Expected behavior

training without any error message


[makefile] check only .py files

in the `fixup` target add `egrep .py$` to feed black/isort/flake8 only .py files as apparently some of them complain if that's not the case.

Fixes: #7579

@sshleifer


Fix squeezebert docs

Slightly update the SqueezeBERT documentation to fit standards before the documentation gods are angered.


Documentation framework toggle should stick

# What does this PR do?

This PR adds the following feature: when clicking on a `PyTorch` or `TensorFlow` button in the documentation in order to show the corresponding framework code sample, the toggle takes effect on all the current pages' code samples.

TensorFlow users won't need to click on every code sample to convert it to TensorFlow anymore!


Documentation fixes

# What does this PR do?

This PR fixes two issues with the documentation:
- wrong type annotation in the configurations (see #7559)
- wrong example for masked LM models (see this [froum post](https://discuss.huggingface.co/t/questions-on-the-bertmodellmheadmodel/1317/6))

Fixes #7559


XLNet evaluation fails if the size of evaluation set can't be divided by a given evaluation batch size

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-117-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger

## Information

Model I am using (Bert, XLNet ...): XLNet-base-cased

The problem arises when using:
* the official example scripts: run_glue.py

The tasks I am working on is:
* an official GLUE/SQUaD task: SST-2

## To reproduce

Steps to reproduce the behavior:

1. Install transformers from master and download SST-2 data using ```download_glue_data.py```

2. Create the following scripts
```bash
GLUE_DIR=~/glue
CUDA_VISIBLE_DEVICES=0
TASK_NAME=SST-2

python3 ~/applications/transformers/examples/text-classification/run_glue.py \
  --model_name_or_path ~/xlnet \
  --task_name $TASK_NAME \
  --do_eval \
  --data_dir $GLUE_DIR/$TASK_NAME \
  --max_seq_length 64 \
  --per_device_train_batch_size 32 \
  --per_device_eval_batch_size 64 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir ~/result/$TASK_NAME/ \
  --overwrite_output_dir \
  --eval_steps 100

```
3. run this script

## Expected behavior

Trainer should return appropriate evaluation results. Here are logs when evaluating bert-base with above-given hyperparameters.
```bash
10/05/2020 22:28:47 - INFO - filelock -   Lock 140392033291808 acquired on /data/home/liusishun/glue/SST-2/cached_dev_BertTokenizer_64_sst-2.lock
10/05/2020 22:28:47 - INFO - filelock -   Lock 140392033291808 released on /data/home/liusishun/glue/SST-2/cached_dev_BertTokenizer_64_sst-2.lock
10/05/2020 22:28:50 - INFO - __main__ -   *** Evaluate ***
Evaluation: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:01<00:00,  7.22it/s]
{'eval_loss': 0.6916399122378148, 'eval_acc': 0.49770642201834864, 'step': 0}
/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer.py:1168: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
  warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)
10/05/2020 22:28:52 - INFO - __main__ -   ***** Eval results sst-2 *****
10/05/2020 22:28:52 - INFO - __main__ -     eval_loss = 0.6916399122378148
10/05/2020 22:28:52 - INFO - __main__ -     eval_acc = 0.49770642201834864

```

## Observed behavior
```bash
10/05/2020 22:30:05 - INFO - filelock -   Lock 139928226197216 acquired on /data/home/liusishun/glue/SST-2/cached_dev_XLNetTokenizer_64_sst-2.lock
10/05/2020 22:30:05 - INFO - filelock -   Lock 139928226197216 released on /data/home/liusishun/glue/SST-2/cached_dev_XLNetTokenizer_64_sst-2.lock
10/05/2020 22:30:09 - INFO - __main__ -   *** Evaluate ***
Evaluation:  93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉           | 13/14 [00:02<00:00,  4.44it/s]
Traceback (most recent call last):
  File "/data/home/liusishun/applications/transformers/examples/text-classification/run_glue.py", line 247, in <module>
    main()
  File "/data/home/liusishun/applications/transformers/examples/text-classification/run_glue.py", line 197, in main
    eval_result = trainer.evaluate(eval_dataset=eval_dataset)
  File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer.py", line 1297, in evaluate
    output = self.prediction_loop(eval_dataloader, description="Evaluation")
  File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer.py", line 1382, in prediction_loop
    preds = logits if preds is None else nested_concat(preds, logits, dim=0)
  File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in nested_concat
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in <genexpr>
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in nested_concat
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 151, in <genexpr>
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/data/home/liusishun/.conda/envs/myenv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 152, in nested_concat
    return torch.cat((tensors, new_tensors), dim=dim)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 40 and 64 in dimension 1 at /opt/conda/conda-bld/pytorch_1579061855666/work/aten/src/THC/generic/THCTensorMath.cu:71
```


RagRetriever.from_pretrained doesn't get another cache_dir.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux-4.19
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0
- Tensorflow version (GPU?): No
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@VictorSanh

## Information

Model I am using RAG:

The problem arises when using:
* [x] the official example scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Open notebook
2. Run the example code changing the 'TRANSFORMERS_CACHE' path to place the dataset in another place than the default one

```
import os
os.environ['TRANSFORMERS_CACHE'] = '/workspace/notebooks/POCs/cache'

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq") # Here the data is placed in the expected path /workspace...

retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=False) # The dataset is placed in the default place /root/.cache/huggingface/datasets/wiki_dpr/psgs_w100.nq.no_index/0.0.0/
```

## Expected behavior
`RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=False)` should place the data in the expected patch  '/workspace/notebooks/POCs/cache'

I tried with as well with: ` retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", chache_dir='/workspace/notebooks/POCs/cache' use_dummy_dataset=False)` but it doesn't work neither.


[TF generation] Fix typo

# What does this PR do?

Typo + Parameter Assertion Fix

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Create README.md

Related to pretrained model cards

# What does this PR do?
Model card for https://huggingface.co/akhooli/xlm-r-large-arabic-toxic


Expand test to locate flakiness

# What does this PR do?

`test_training_arguments_are_left_untouched` in `test_trainer.py` is a bit flaky, this PR just expands in a loop the assertEqual so we can hopefully locate the source of the flakiness.


make modified_only_fixup complains about non .py files

easy fix @stas00 ?


RobertaTokenizer.get_special_tokens_mask doesn't check for all special tokens, only for the sep and cls tokens

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: macOS-10.15.6-x86_64-i386-64bit
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@LysandreJik @mfuntowicz

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## To reproduce

```python
>>> from transformers import RobertaTokenizer, RobertaTokenizerFast
>>> tokenizer_slow = RobertaTokenizer.from_pretrained('roberta-base')
>>> tokenizer_fast = RobertaTokenizerFast.from_pretrained('roberta-base')
>>> tokenizer_slow.add_special_tokens({'additional_special_tokens': ['<a>']})
1
>>> tokenizer_fast.add_special_tokens({'additional_special_tokens': ['<a>']})
1
>>> tokenizer_slow.get_special_tokens_mask(tokenizer_slow.encode('<a><pad><mask>'), already_has_special_tokens=True)
[1, 0, 0, 0, 1]
>>> tokenizer_fast.get_special_tokens_mask(tokenizer_fast.encode('<a><pad><mask>'), already_has_special_tokens=True)
[1, 1, 1, 1, 1]

```

Steps to reproduce the behavior:

1. Run the above lines

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

RobertaTokenizer should also mask special tokens, like RobertaTokenizerFast does. Let me know if you need any additional info or could do with a PR. Not sure if this issue is present with other tokenizers.

<!-- A clear and concise description of what you would expect to happen. -->


Add support to provide initial tokens to decoder of encoder-decoder type models

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #7502

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Trainer evaluate returns empty dictionary

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-45-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 Trainer: @sgugger

## Information

Model I am using RoBERT:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `python3 finetune_roberta.py -m result/ -d dataset.txt -t dataset.txt`

```python
from transformers import (BertForNextSentencePrediction,
                          BertTokenizer,
                          RobertaModel, RobertaTokenizer, Trainer,
                          TrainingArguments)
from transformers.data.datasets.language_modeling import TextDatasetForNextSentencePrediction
from transformers.data.data_collator import DataCollatorForNextSentencePrediction
from argparse import ArgumentParser

def parse_args():
    parser = ArgumentParser("Fine-tune RoBERTa in Next Sentence Prediction.")
    parser.add_argument("-m", "--model_path", dest="model_path", required=True, help="Path to RoBERTa model.")
    parser.add_argument("-d", "--dataset_path", dest="dataset_path", required=True, help="Path to dataset.")
    parser.add_argument("-t", "--test_dataset_path", dest="test_dataset_path", required=True, help="Path to test dataset.")
    args = parser.parse_args()
    return args

if __name__ == "__main__":
    args = parse_args()
    tokenizer = RobertaTokenizer.from_pretrained(args.model_path)
    finetune_model = BertForNextSentencePrediction.from_pretrained(args.model_path)

    training_args = TrainingArguments(
        output_dir=args.output_path,
        num_train_epochs=3,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        warmup_steps=500,
        weight_decay=0.01,
        logging_dir='./logs',
    )

    data_collator = DataCollatorForNextSentencePrediction(
        tokenizer=tokenizer,
        mlm=False,
        block_size=512,
        nsp_probability=0.5,
      )

    train_dataset = TextDatasetForNextSentencePrediction(
        tokenizer=tokenizer,
        file_path=args.dataset_path,
        block_size=512,
    )

    test_dataset = TextDatasetForNextSentencePrediction(
        tokenizer=tokenizer,
        file_path=args.test_dataset_path,
        block_size=512,
    )

    trainer = Trainer(
        model=finetune_model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
        data_collator=data_collator,
    )

    print(trainer.evaluate(test_dataset))
```
Output in terminal:
```bash
python3 finetune_roberta.py -m result/ -d dataset_fixed_alior.txt -t dataset_fixed_alior.txt -o results_test/
Special tokens have been added in the vocabulary, make sure the associated word emebedding are fine-tuned or trained.
Some weights of the model checkpoint at result/ were not used when initializing RobertaModel: ['bert.embeddings.position_ids', 'bert.embeddings.word_embeddings.weight', 'bert.embeddings.position_embeddings.weight', 'bert.embeddings.token_type_embeddings.weight', 'bert.embeddings.LayerNorm.weight', 'bert.embeddings.LayerNorm.bias', 'bert.encoder.layer.0.attention.self.query.weight', 'bert.encoder.layer.0.attention.self.query.bias', 'bert.encoder.layer.0.attention.self.key.weight', 'bert.encoder.layer.0.attention.self.key.bias', 'bert.encoder.layer.0.attention.self.value.weight', 'bert.encoder.layer.0.attention.self.value.bias', 'bert.encoder.layer.0.attention.output.dense.weight', 'bert.encoder.layer.0.attention.output.dense.bias', 'bert.encoder.layer.0.attention.output.LayerNorm.weight', 'bert.encoder.layer.0.attention.output.LayerNorm.bias', 'bert.encoder.layer.0.intermediate.dense.weight', 'bert.encoder.layer.0.intermediate.dense.bias', 'bert.encoder.layer.0.output.dense.weight', 'bert.encoder.layer.0.output.dense.bias', 'bert.encoder.layer.0.output.LayerNorm.weight', 'bert.encoder.layer.0.output.LayerNorm.bias', 'bert.encoder.layer.1.attention.self.query.weight', 'bert.encoder.layer.1.attention.self.query.bias', 'bert.encoder.layer.1.attention.self.key.weight', 'bert.encoder.layer.1.attention.self.key.bias', 'bert.encoder.layer.1.attention.self.value.weight', 'bert.encoder.layer.1.attention.self.value.bias', 'bert.encoder.layer.1.attention.output.dense.weight', 'bert.encoder.layer.1.attention.output.dense.bias', 'bert.encoder.layer.1.attention.output.LayerNorm.weight', 'bert.encoder.layer.1.attention.output.LayerNorm.bias', 'bert.encoder.layer.1.intermediate.dense.weight', 'bert.encoder.layer.1.intermediate.dense.bias', 'bert.encoder.layer.1.output.dense.weight', 'bert.encoder.layer.1.output.dense.bias', 'bert.encoder.layer.1.output.LayerNorm.weight', 'bert.encoder.layer.1.output.LayerNorm.bias', 'bert.encoder.layer.2.attention.self.query.weight', 'bert.encoder.layer.2.attention.self.query.bias', 'bert.encoder.layer.2.attention.self.key.weight', 'bert.encoder.layer.2.attention.self.key.bias', 'bert.encoder.layer.2.attention.self.value.weight', 'bert.encoder.layer.2.attention.self.value.bias', 'bert.encoder.layer.2.attention.output.dense.weight', 'bert.encoder.layer.2.attention.output.dense.bias', 'bert.encoder.layer.2.attention.output.LayerNorm.weight', 'bert.encoder.layer.2.attention.output.LayerNorm.bias', 'bert.encoder.layer.2.intermediate.dense.weight', 'bert.encoder.layer.2.intermediate.dense.bias', 'bert.encoder.layer.2.output.dense.weight', 'bert.encoder.layer.2.output.dense.bias', 'bert.encoder.layer.2.output.LayerNorm.weight', 'bert.encoder.layer.2.output.LayerNorm.bias', 'bert.encoder.layer.3.attention.self.query.weight', 'bert.encoder.layer.3.attention.self.query.bias', 'bert.encoder.layer.3.attention.self.key.weight', 'bert.encoder.layer.3.attention.self.key.bias', 'bert.encoder.layer.3.attention.self.value.weight', 'bert.encoder.layer.3.attention.self.value.bias', 'bert.encoder.layer.3.attention.output.dense.weight', 'bert.encoder.layer.3.attention.output.dense.bias', 'bert.encoder.layer.3.attention.output.LayerNorm.weight', 'bert.encoder.layer.3.attention.output.LayerNorm.bias', 'bert.encoder.layer.3.intermediate.dense.weight', 'bert.encoder.layer.3.intermediate.dense.bias', 'bert.encoder.layer.3.output.dense.weight', 'bert.encoder.layer.3.output.dense.bias', 'bert.encoder.layer.3.output.LayerNorm.weight', 'bert.encoder.layer.3.output.LayerNorm.bias', 'bert.encoder.layer.4.attention.self.query.weight', 'bert.encoder.layer.4.attention.self.query.bias', 'bert.encoder.layer.4.attention.self.key.weight', 'bert.encoder.layer.4.attention.self.key.bias', 'bert.encoder.layer.4.attention.self.value.weight', 'bert.encoder.layer.4.attention.self.value.bias', 'bert.encoder.layer.4.attention.output.dense.weight', 'bert.encoder.layer.4.attention.output.dense.bias', 'bert.encoder.layer.4.attention.output.LayerNorm.weight', 'bert.encoder.layer.4.attention.output.LayerNorm.bias', 'bert.encoder.layer.4.intermediate.dense.weight', 'bert.encoder.layer.4.intermediate.dense.bias', 'bert.encoder.layer.4.output.dense.weight', 'bert.encoder.layer.4.output.dense.bias', 'bert.encoder.layer.4.output.LayerNorm.weight', 'bert.encoder.layer.4.output.LayerNorm.bias', 'bert.encoder.layer.5.attention.self.query.weight', 'bert.encoder.layer.5.attention.self.query.bias', 'bert.encoder.layer.5.attention.self.key.weight', 'bert.encoder.layer.5.attention.self.key.bias', 'bert.encoder.layer.5.attention.self.value.weight', 'bert.encoder.layer.5.attention.self.value.bias', 'bert.encoder.layer.5.attention.output.dense.weight', 'bert.encoder.layer.5.attention.output.dense.bias', 'bert.encoder.layer.5.attention.output.LayerNorm.weight', 'bert.encoder.layer.5.attention.output.LayerNorm.bias', 'bert.encoder.layer.5.intermediate.dense.weight', 'bert.encoder.layer.5.intermediate.dense.bias', 'bert.encoder.layer.5.output.dense.weight', 'bert.encoder.layer.5.output.dense.bias', 'bert.encoder.layer.5.output.LayerNorm.weight', 'bert.encoder.layer.5.output.LayerNorm.bias', 'bert.encoder.layer.6.attention.self.query.weight', 'bert.encoder.layer.6.attention.self.query.bias', 'bert.encoder.layer.6.attention.self.key.weight', 'bert.encoder.layer.6.attention.self.key.bias', 'bert.encoder.layer.6.attention.self.value.weight', 'bert.encoder.layer.6.attention.self.value.bias', 'bert.encoder.layer.6.attention.output.dense.weight', 'bert.encoder.layer.6.attention.output.dense.bias', 'bert.encoder.layer.6.attention.output.LayerNorm.weight', 'bert.encoder.layer.6.attention.output.LayerNorm.bias', 'bert.encoder.layer.6.intermediate.dense.weight', 'bert.encoder.layer.6.intermediate.dense.bias', 'bert.encoder.layer.6.output.dense.weight', 'bert.encoder.layer.6.output.dense.bias', 'bert.encoder.layer.6.output.LayerNorm.weight', 'bert.encoder.layer.6.output.LayerNorm.bias', 'bert.encoder.layer.7.attention.self.query.weight', 'bert.encoder.layer.7.attention.self.query.bias', 'bert.encoder.layer.7.attention.self.key.weight', 'bert.encoder.layer.7.attention.self.key.bias', 'bert.encoder.layer.7.attention.self.value.weight', 'bert.encoder.layer.7.attention.self.value.bias', 'bert.encoder.layer.7.attention.output.dense.weight', 'bert.encoder.layer.7.attention.output.dense.bias', 'bert.encoder.layer.7.attention.output.LayerNorm.weight', 'bert.encoder.layer.7.attention.output.LayerNorm.bias', 'bert.encoder.layer.7.intermediate.dense.weight', 'bert.encoder.layer.7.intermediate.dense.bias', 'bert.encoder.layer.7.output.dense.weight', 'bert.encoder.layer.7.output.dense.bias', 'bert.encoder.layer.7.output.LayerNorm.weight', 'bert.encoder.layer.7.output.LayerNorm.bias', 'bert.encoder.layer.8.attention.self.query.weight', 'bert.encoder.layer.8.attention.self.query.bias', 'bert.encoder.layer.8.attention.self.key.weight', 'bert.encoder.layer.8.attention.self.key.bias', 'bert.encoder.layer.8.attention.self.value.weight', 'bert.encoder.layer.8.attention.self.value.bias', 'bert.encoder.layer.8.attention.output.dense.weight', 'bert.encoder.layer.8.attention.output.dense.bias', 'bert.encoder.layer.8.attention.output.LayerNorm.weight', 'bert.encoder.layer.8.attention.output.LayerNorm.bias', 'bert.encoder.layer.8.intermediate.dense.weight', 'bert.encoder.layer.8.intermediate.dense.bias', 'bert.encoder.layer.8.output.dense.weight', 'bert.encoder.layer.8.output.dense.bias', 'bert.encoder.layer.8.output.LayerNorm.weight', 'bert.encoder.layer.8.output.LayerNorm.bias', 'bert.encoder.layer.9.attention.self.query.weight', 'bert.encoder.layer.9.attention.self.query.bias', 'bert.encoder.layer.9.attention.self.key.weight', 'bert.encoder.layer.9.attention.self.key.bias', 'bert.encoder.layer.9.attention.self.value.weight', 'bert.encoder.layer.9.attention.self.value.bias', 'bert.encoder.layer.9.attention.output.dense.weight', 'bert.encoder.layer.9.attention.output.dense.bias', 'bert.encoder.layer.9.attention.output.LayerNorm.weight', 'bert.encoder.layer.9.attention.output.LayerNorm.bias', 'bert.encoder.layer.9.intermediate.dense.weight', 'bert.encoder.layer.9.intermediate.dense.bias', 'bert.encoder.layer.9.output.dense.weight', 'bert.encoder.layer.9.output.dense.bias', 'bert.encoder.layer.9.output.LayerNorm.weight', 'bert.encoder.layer.9.output.LayerNorm.bias', 'bert.encoder.layer.10.attention.self.query.weight', 'bert.encoder.layer.10.attention.self.query.bias', 'bert.encoder.layer.10.attention.self.key.weight', 'bert.encoder.layer.10.attention.self.key.bias', 'bert.encoder.layer.10.attention.self.value.weight', 'bert.encoder.layer.10.attention.self.value.bias', 'bert.encoder.layer.10.attention.output.dense.weight', 'bert.encoder.layer.10.attention.output.dense.bias', 'bert.encoder.layer.10.attention.output.LayerNorm.weight', 'bert.encoder.layer.10.attention.output.LayerNorm.bias', 'bert.encoder.layer.10.intermediate.dense.weight', 'bert.encoder.layer.10.intermediate.dense.bias', 'bert.encoder.layer.10.output.dense.weight', 'bert.encoder.layer.10.output.dense.bias', 'bert.encoder.layer.10.output.LayerNorm.weight', 'bert.encoder.layer.10.output.LayerNorm.bias', 'bert.encoder.layer.11.attention.self.query.weight', 'bert.encoder.layer.11.attention.self.query.bias', 'bert.encoder.layer.11.attention.self.key.weight', 'bert.encoder.layer.11.attention.self.key.bias', 'bert.encoder.layer.11.attention.self.value.weight', 'bert.encoder.layer.11.attention.self.value.bias', 'bert.encoder.layer.11.attention.output.dense.weight', 'bert.encoder.layer.11.attention.output.dense.bias', 'bert.encoder.layer.11.attention.output.LayerNorm.weight', 'bert.encoder.layer.11.attention.output.LayerNorm.bias', 'bert.encoder.layer.11.intermediate.dense.weight', 'bert.encoder.layer.11.intermediate.dense.bias', 'bert.encoder.layer.11.output.dense.weight', 'bert.encoder.layer.11.output.dense.bias', 'bert.encoder.layer.11.output.LayerNorm.weight', 'bert.encoder.layer.11.output.LayerNorm.bias', 'bert.pooler.dense.weight', 'bert.pooler.dense.bias', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaModel were not initialized from the model checkpoint at result/ and are newly initialized: ['embeddings.word_embeddings.weight', 'embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.LayerNorm.weight', 'embeddings.LayerNorm.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.9.output.dense.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.10.attention.self.value.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.11.output.dense.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.11.output.LayerNorm.bias', 'pooler.dense.weight', 'pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Evaluation:   0%|                                                                                | 0/7385 [00:00<?, ?it/s]
...
[[[206, 3, 3, 47, 3, 4224, 163, 3, 2197, 3, 3, 3, 91, 324, 3, 2920, 3, 3, 47, 3, 3, 170, 21822, 3, 3, 14139, 3, 301, 3, 1210, 1692, 3, 67, 3, 25690, 358, 3, 258, 84, 3, 447, 3, 2980, 3, 43750, 3, 7388, 3, 15679, 4121, 3, 254, 3, 15390, 223, 21822, 3, 12135, 3, 269, 3, 254, 3, 10577, 3, 88, 611, 90, 3, 4729, 15616, 3, 92, 3, 336, 163, 21822, 3, 526, 3, 269, 3, 98, 3048, 3, 4224, 3, 215, 21822, 3, 88, 358, 3, 88, 611, 84], [254, 3, 67, 3, 88, 611, 74, 21822, 3, 3, 88, 611, 74, 21822, 3, 3, 206, 3, 3, 2048, 74, 21822, 3, 3, 92, 3, 336, 163, 21822, 3, 526]], [[254, 3, 67, 3, 88, 611, 74, 21822, 3, 3, 88, 611, 74, 21822, 3, 3, 206, 3, 3, 2048, 74, 21822, 3, 3, 92, 3, 336, 163, 21822, 3, 526], [1598, 3, 3, 56, 2726, 3, 5704, 3, 2980, 4847, 3, 336, 826, 237, 3, 4224, 163, 3, 254, 3, 10577, 3, 25690, 3, 3412, 23962, 567]], [[301, 6393, 324, 3, 36461, 118, 238, 3, 56, 3, 269, 3, 26672, 84, 3, 93, 3, 169, 210, 3, 3968, 368, 3, 7388, 3, 477, 3, 3, 91, 3, 301, 21176, 3, 94, 3, 94, 3, 301, 21176, 3, 7388, 3, 67, 3, 26605, 110, 21822, 3, 3, 15191, 114, 21822, 3, 3, 3119, 3, 2197, 3, 7208, 3, 95, 661, 3, 227, 3, 3, 3, 2554, 3, 2197, 4847, 3, 2197, 21822, 3, 3, 477, 3, 3, 9746, 3, 4224, 21822, 3, 3, 91, 41872, 311, 3, 1068, 25393, 3, 353, 3, 23635, 3, 75, 3, 1146, 173, 3, 3, 5379, 3, 237, 2736, 3, 3, 93, 3, 1068, 105, 818, 3, 3, 3, 91, 324, 3, 258, 90, 3, 324, 3, 3, 56, 3, 301, 3, 1210, 1692, 3, 25690, 431, 3, 3, 3, 91, 324, 3, 94, 3, 1844, 3, 3, 3, 1146, 163, 21822, 3, 3, 3, 56, 3, 3, 56, 3, 2197, 3, 6073, 3, 3, 29623], [3112, 130, 3, 284, 3, 5704, 3, 94, 3, 95, 661, 3, 2197, 4847]], [[3112, 130, 3, 284, 3, 5704, 3, 94, 3, 95, 661, 3, 2197, 4847], [301, 6393, 324, 3, 36461, 118, 238, 3, 93, 3, 2197, 938, 3, 3190, 75, 91, 3, 388, 3, 336, 3, 4224, 110, 21822, 3, 3, 76, 2018, 3, 215, 518, 74, 3, 3, 93, 88, 21822, 3, 3, 92, 3, 848, 535, 614, 3, 4516, 25307, 3, 353, 3, 95, 94, 3, 10032, 324, 3, 317, 21822, 3, 3, 575, 3, 4224, 3, 92, 7278, 3, 3, 47, 3, 93, 3, 3313, 3, 88, 9813, 1964, 3, 477, 3, 3, 67, 3, 317, 21822, 3, 3, 6494, 114, 21822, 3, 3, 488, 664, 2197, 2518, 21822, 3, 3, 388, 3, 95, 661, 3, 477, 3, 3, 9746, 3, 4224, 21822, 3, 3, 93, 8714, 892, 3, 2197, 12334, 3, 93, 3079, 1451, 3, 15227]], [[301, 6393, 324, 3, 36461, 118, 238, 3, 93, 3, 2197, 938, 3, 3190, 75, 91, 3, 388, 3, 336, 3, 4224, 110, 21822, 3, 3, 76, 2018, 3, 215, 518, 74, 3, 3, 93, 88, 21822, 3, 3, 92, 3, 848, 535, 614, 3, 4516, 25307, 3, 353, 3, 95, 94, 3, 10032, 324, 3, 317, 21822, 3, 3, 575, 3, 4224, 3, 92, 7278, 3, 3, 47, 3, 93, 3, 3313, 3, 88, 9813, 1964, 3, 477, 3, 3, 67, 3, 317, 21822, 3, 3, 6494, 114, 21822, 3, 3, 488, 664, 2197, 2518, 21822, 3, 3, 388, 3, 95, 661, 3, 477, 3, 3, 9746, 3, 4224, 21822, 3, 3, 93, 8714, 892, 3, 2197, 12334, 3, 93, 3079, 1451, 3, 15227], [1569, 359, 3, 404, 3, 3, 3, 7850, 219, 3, 92, 3, 3313, 3, 2854, 81, 3, 3412, 1459, 567, 3, 3412, 1459, 567, 3, 8142, 3, 3412, 1459, 567, 3, 4516, 25307, 98, 21822, 3, 3, 92, 3, 19638, 3, 3968, 176, 3, 2197, 3, 93, 3120, 95, 207, 3, 3, 238, 3, 239, 1881, 132, 3, 10577, 3, 299, 308, 3350, 360, 3, 75, 3, 95, 661, 3, 353, 3, 1068, 7183, 200, 3, 4761, 3, 22443, 3, 11031, 3, 3, 490, 21822, 3, 3, 169, 5252, 4082, 21822, 3, 3, 269, 3, 477, 149, 21822, 3, 3, 227, 3190, 3, 26605, 110, 21822, 3, 3, 383, 98, 21822, 3, 21822, 3, 3, 2197, 3, 2197, 3, 301, 1210, 85, 21822, 3, 3, 1611, 3, 3, 3, 95, 3, 3, 215, 21822, 3, 3, 92, 4655, 3, 47, 3, 575, 3, 12992, 3, 353, 16035, 95, 3, 2923, 21530, 3, 17248, 8268, 3, 47, 24072, 3, 4224, 86, 3, 2197, 3, 10577, 3, 45820, 67, 3, 17248, 5766, 3, 570, 21822, 3, 8325, 21822, 3, 3, 75, 3, 11517, 6412, 21822, 3, 3, 169, 5252, 4082, 21822, 3, 3, 47, 3, 1145, 3, 3, 131, 3322, 3, 1709, 3, 16581, 544, 3, 1328, 683, 3, 388, 3, 1210, 85, 21822, 3, 3, 22443, 3, 27921, 173, 21822, 3, 3, 301, 94, 183, 3, 3, 3, 301, 388, 215, 3, 3, 56, 3, 317, 21822, 3, 3, 21592, 3, 477, 3, 3, 9746, 3, 4224, 163, 3, 67, 3, 46, 2728, 943, 3, 43750, 3, 29317, 5571, 3, 4224, 163, 3, 47, 3482, 84, 3, 2777, 3, 88, 21822, 3, 380, 3, 7850, 4049, 3, 353, 14227, 78, 3, 3, 4224, 2139, 88, 3, 3, 93, 3, 92, 3, 3968, 176, 3, 1709, 3, 227, 3, 3, 3, 488, 664, 19586, 28602, 3, 169, 210, 3, 15390, 223, 21822, 3, 1326, 98, 21822, 3, 206, 3, 673, 3, 1146, 173, 3, 3, 5379, 3, 591, 3, 258, 84, 3, 138, 546, 3, 3, 238, 3, 383, 679, 90, 21822, 3, 3, 269, 3, 1617, 3, 3, 258, 3, 4224, 3, 92, 3, 1642, 3, 2774, 261, 3, 324, 3, 3, 56, 324, 3, 4761, 3, 22443, 3, 301, 5777, 67]], [[1569, 359, 3, 404, 3, 3, 3, 7850, 219, 3, 92, 3, 3313, 3, 2854, 81, 3, 3412, 1459, 567, 3, 3412, 1459, 567, 3, 8142, 3, 3412, 1459, 567, 3, 4516, 25307, 98, 21822, 3, 3, 92, 3, 19638, 3, 3968, 176, 3, 2197, 3, 93, 3120, 95, 207, 3, 3, 238, 3, 239, 1881, 132, 3, 10577, 3, 299, 308, 3350, 360, 3, 75, 3, 95, 661, 3, 353, 3, 1068, 7183, 200, 3, 4761, 3, 22443, 3, 11031, 3, 3, 490, 21822, 3, 3, 169, 5252, 4082, 21822, 3, 3, 269, 3, 477, 149, 21822, 3, 3, 227, 3190, 3, 26605, 110, 21822, 3, 3, 383, 98, 21822, 3, 21822, 3, 3, 2197, 3, 2197, 3, 301, 1210, 85, 21822, 3, 3, 1611, 3, 3, 3, 95, 3, 3, 215, 21822, 3, 3, 92, 4655, 3, 47, 3, 575, 3, 12992, 3, 353, 16035, 95, 3, 2923, 21530, 3, 17248, 8268, 3, 47, 24072, 3, 4224, 86, 3, 2197, 3, 10577, 3, 45820, 67, 3, 17248, 5766, 3, 570, 21822, 3, 8325, 21822, 3, 3, 75, 3, 11517, 6412, 21822, 3, 3, 169, 5252, 4082, 21822, 3, 3, 47, 3, 1145, 3, 3, 131, 3322, 3, 1709, 3, 16581, 544, 3, 1328, 683, 3, 388, 3, 1210, 85, 21822, 3, 3, 22443, 3, 27921, 173, 21822, 3, 3, 301, 94, 183, 3, 3, 3, 301, 388, 215, 3, 3, 56, 3, 317, 21822, 3, 3, 21592, 3, 477, 3, 3, 9746, 3, 4224, 163, 3, 67, 3, 46, 2728, 943, 3, 43750, 3, 29317, 5571, 3, 4224, 163, 3, 47, 3482, 84, 3, 2777, 3, 88, 21822, 3, 380, 3, 7850, 4049, 3, 353, 14227, 78, 3, 3, 4224, 2139, 88, 3, 3, 93, 3, 92, 3, 3968, 176, 3, 1709, 3, 227, 3, 3, 3, 488, 664, 19586, 28602, 3, 169, 210, 3, 15390, 223, 21822, 3, 1326, 98, 21822, 3, 206, 3, 673, 3, 1146, 173, 3, 3, 5379, 3, 591, 3, 258, 84, 3, 138, 546, 3, 3, 238, 3, 383, 679, 90, 21822, 3, 3, 269, 3, 1617, 3, 3, 258, 3, 4224, 3, 92, 3, 1642, 3, 2774, 261, 3, 324, 3, 3, 56, 324, 3, 4761, 3, 22443, 3, 301, 5777, 67], [477, 3, 3, 91, 3, 94, 3, 2980, 3, 67, 3, 3190, 75, 91, 3, 254, 3, 7208, 3, 67, 3, 3190, 75, 91, 3, 94, 3, 94, 3, 94, 3, 308, 4540, 3, 383, 43129, 252, 3, 501]], [[477, 3, 3, 91, 3, 94, 3, 2980, 3, 67, 3, 3190, 75, 91, 3, 254, 3, 7208, 3, 67, 3, 3190, 75, 91, 3, 94, 3, 94, 3, 94, 3, 308, 4540, 3, 383, 43129, 252, 3, 501], [7388, 3, 36461, 118, 238, 3, 2932, 3, 9216, 3, 3, 33118, 3, 1500, 19586, 23671, 3, 673, 3, 4592, 3, 3, 2447, 3, 353, 14227, 132, 3, 94, 3, 93, 1413, 855, 21822, 3, 3, 93, 3, 9486, 21822, 3]], []]
Evaluation: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7385/7385 [37:58<00:00,  3.24it/s]
{}
```

## Expected behavior

Trainer returns dictionary with statistics about model performance.


docs(pretrained_models): fix num parameters

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR corrects the number of parameters of pretrained BERT based models presented in the documentation.

Sometimes the difference between a given model and its pairs is important. For instance:
`bert-base-uncased` has **110M parameters** but `bert-base-multilingual-cased` has more than **178M parameters**, even if both models share the same architecture (12-layer, 768-hidden, 12-heads).

The difference is due to the vocabulary size:
`bert-base-uncased` uses a vocabulary of **30k** entries while `bert-base-multilingual-cased` uses a vocabulary of **119k** entries.

To compute the number of parameters:
``` python
from transformers import AutoModelForMaskedLM

bert_base = AutoModelForMaskedLM.from_pretrained('bert-base-uncased')
print(bert_base.num_parameters())

bert_multiling = AutoModelForMaskedLM.from_pretrained('bert-base-multilingual-cased')
print(bert_multiling.num_parameters())
```

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Particularly: @LysandreJik and @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


# ❓ Questions & Help
Hi, I'm new to gpt2 and also this project! I was trying to run an example in the tutorials **https://huggingface.co/transformers/model_doc/gpt2.html** as follows:
```
import torch
from transformers import GPT2Tokenizer, GPT2DoubleHeadsModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2DoubleHeadsModel.from_pretrained('gpt2', return_dict=True)

# Add a [CLS] to the vocabulary (we should train it also!)
num_added_tokens = tokenizer.add_special_tokens({'cls_token': '[CLS]'})

embedding_layer = model.resize_token_embeddings(len(tokenizer))  # Update the model embeddings with the new vocabulary size

choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]
encoded_choices = [tokenizer.encode(s) for s in choices]
cls_token_location = [tokens.index(tokenizer.cls_token_id) for tokens in encoded_choices]

input_ids = torch.tensor(encoded_choices).unsqueeze(0)  # Batch size: 1, number of choices: 2
mc_token_ids = torch.tensor([cls_token_location])  # Batch size: 1

outputs = model(input_ids, mc_token_ids=mc_token_ids)
lm_logits = outputs.logits
mc_logits = outputs.mc_logits
```
Then I got errors as below:

> Some weights of GPT2DoubleHeadsModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight', 'multiple_choice_head.summary.weight', 'multiple_choice_head.summary.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

**https://github.com/huggingface/transformers/issues/6667**
I've looked it up and find somebody with the same question as me. But I still got confused about what and how I can do to " fine-tune my model on a multiple-choice task". Maybe it's a dumb question though, but I still want to know how to make it work!


[model_card] bert-base-5lang-cased

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds the model card of [amine/bert-base-5lang-cased](https://huggingface.co/amine/bert-base-5lang-cased). A smaller version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) that handles only 5 languages (en, fr, es, de and zh) instead of 104 while reducing its size by 30%.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Finetuning T5: Keyword arguments not recognized.

# ❓ Questions & Help

## Details
Hi, I want to finetune the T5-small model for summarization purposes, to finetune the T5-large later. I prepared my data as as shown in the examples but during training i receive the message:
`Keyword arguments {'src_lang': None, 'tgt_lang': None, 'add_prefix_space': False} not recognized.`
which indicates to me, that somehow my data preparation process is wrong. However, I was not able to find out how the data for T5 has to be prepared properly (as the T5 is a multi-ability model, the data has to be marked somehow, but I dont know how).
Currently my train data looks as the follow:
train.source:
Line 1: A long text
Line 2: Another long Text

train.target:
Line 1: 'target':'Target for the text'
Line 2: 'target':'Another target for the text'

This question is in my view to specific to the transformer huggingface-architecture as I use the `finetune_sh` script in the seq2seq example folder. A small example, of how the data has to be structured for T5 will be very helpful, thank you!


Sequence Classification One-Hot Encoded Data

## Environment info
- `transformers` version: 3.3.1

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik

## Information
I am using Bert and Roberta.

The tasks I am working on is:
Sequence Classification

## Problem
The model does not work with "One-Hot Encoded" data. The model only accepts a list of integers as labels which is feed into the MSELoss. This is undesired in a multi-label classification task with categorical data, because an order of the classes is induced.


Import error for MarianMTModel

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): MarianMTModel

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
I tried importing MarianMTmodel from transformers and it raised an error message

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
My own task

## To reproduce

Steps to reproduce the behavior:

1.from transformers import MarianMTModel, MarianTokenizer
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Imports MarianMTModel correctly
<!-- A clear and concise description of what you would expect to happen. -->

This is the error message I got:
ImportError: cannot import name 'MarianMTModel' from 'transformers' (C:\Users\PRINCE\Anaconda3\envs\Brian's Enviroment\lib\site-packages\transformers\__init__.py)


Add Electra unexpected keys

This PR adds the necessary ELECTRA unexpected keys. Some keys are only used with models that have a different embedding size to their hidden size, in order to do the projection.

Some models (such as the `large` ELECTRA variants), do not leverage these weights, as they have the same embedding/hidden sizes.

Fixes #7530.


[Model card] Java Code Summarizer model

Related to pretrained model cards

Initial version of java code summarizer model for generating code comments.


Is training distilbert with TPU supported yet?

# 🚀 Feature request

Hi! I tried training my own distilbert model with [this code](https://github.com/huggingface/transformers/blob/master/examples/distillation/train.py) using GPU, and it was a success.
I'm wondering if training distilbert model with TPU is supported yet, or if there's any plan to release a new version in which TPU is supported?


Trainer incorrectly checks pytorch version

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-3.10.0-862.el7.x86_64-x86_64-with-glibc2.27
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes(not applicable)
- Using distributed or parallel set-up in script?: not applicable

### Who can help
@sgugger and @prajjwal1 (he added it, according to git blame)

## Information

I'm running token classification example on my own data, and i've faced trouble with fp16 training with torch 1.6.0.
Script says that i need apex installed to use fp16 option. However, apex is not required since torch 1.6.0 came out with native amp support. I've dived into trainer code, and found that there is version checking line: https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/src/transformers/trainer.py#L65

Apparently, it is slightly incorrect. There should be <= instead of <. So it will not try to import apex if torch version is greater OR EQUAL to 1.6

```python
import torch
from packaging import version

print(version.parse(torch.__version__) < version.parse("1.6")) # -> False
print(version.parse(torch.__version__) <= version.parse("1.6")) # -> True
```

## To reproduce
1. Install torch 1.6.0(and do not install apex)
2. clone repository
3. cd into examples/token-classification
4. add '--fp16' to the bottom of run.sh script
5. execute run.sh script

## Expected behavior
 Script works well on torch 1.6.0 without apex installed


Two slow deberta test failures

Tests

Related to tests

https://github.com/huggingface/transformers/runs/1204063498?check_suite_focus=true

```
FAILED tests/test_modeling_deberta.py::DebertaModelIntegrationTest::test_inference_classification_head
FAILED tests/test_tokenization_deberta.py::DebertaTokenizationTest::test_torch_encode_plus_sent_to_model
```

@LysandreJik I think ?


Update normalising method in oneshot classifier

Hi, I know I raised this issue before here: https://github.com/huggingface/transformers/pull/5760#issuecomment-673840015 but I do think that it is worth taking a second look atleast.

So my main concern is, since we are dealing with logits, currently all that happens is to take the softmax over the entailment logits. This does guarantee summing to one, however, it doesn't care for the scale of the logits. For example suppose the logits for two possible classes for a given sentence came out as:
```
[[1000, 10, 10],
 [0.1, 0.1, 0.9]]
```
at the current method would give class1 the higher probability, even though, it can clearly be seen that class1 is a contradiction (since it is 100x contradiction in the log scale), and similarly class2 is clearly an entailment.

Now I do understand that this is an extreme case, but just to cover the bases, what I propose is that:
1. We softmax across every single sentence-class pair to get everything in the same scale.
2. Get a probability measurement over entailment by simply dividing by sum of probabilities. As seen [here](https://github.com/huggingface/transformers/pull/5760#issuecomment-673840015 ) the numbers that we get are different but only slightly for the example shown.

I do apologise about raising this again, just simply trying to help.


Error Loading Gpt-2 model after training from scratch.

```
 Error(s) in loading state_dict for GPT2LMHeadModel:
	size mismatch for transformer.h.0.mlp.c_fc.weight: copying a param with shape torch.Size([768, 6]) from checkpoint, the shape in current model is torch.Size([768, 3072]).
```
I trained a GPT-2 model from scratch. When I tried loading the model using
```
config = GPT2Config.from_json_file('../input/hindigpt/config.json')
# config.type_vocab_size=3072
model = GPT2LMHeadModel(config)
model.load_state_dict(torch.load('../input/hindigpt/pytorch_model.bin'))
```
I am getting the above error. Here is the copy of my config.json file
```
{
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "gradient_checkpointing": false,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": 6,
  "n_layer": 6,
  "n_positions": 512,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "vocab_size": 36021
}
```
Can anyone help me with this?