BertforSequenceClassification MSELoss() without normalizing using sigmoid/softmax

- `transformers` version: 3.3.0
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.4
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

@LysandreJik @sshleifer

I'm trying to use BartForSequenceClassification() to do a regression, which should use a MSELoss() when (num_labels=1), as stated in the documentation. However, when I went over the source code for modeling_Bart.py, it didn't seem like the regression functionality with MSELoss() was added in the source code. Within the class BartForSequenceClassfication(), It only has CrossEntropyLoss() to do classification. I wonder if you will be adding this functionality.

So I went to BertForSequenceClassification() class to see how it did it, and I found that it might have a problem.

In the class BertForSequenceClassification(BertPreTrainedModel) (line 1352):

if labels is not None:
            if self.num_labels == 1:
                #  We are doing regression
                loss_fct = MSELoss()
                loss = loss_fct(logits.view(-1), labels.view(-1))
            else:
                loss_fct = CrossEntropyLoss()
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

Here the MSELoss() and CrossEntropyLoss() are both loss functions from pytorch.

So you passed in the logits, which are unnormalized probabilities, to both of the loss functions. It is ok to do so for the CrossEntropyLoss(), since from pytorch's documentation they expect the inputs to be unnormalized logits and from their source code they first log softmax it before actually computing the loss, but I don't think it's okay to do the same for the MSELoss() functions. If you look at their implementation of it, it did not normalize it first using either softmax or sigmoid, and they also indicate that the function expects unnormalized logits in their documentation. I'm not sure if this behavior is intended (which is unlikely since we want to normalize it first before loss functions or there could be gradient explosions), but I think this confusion/inconsistency from pytorch may cause a problem and you probably want to change it. Please correct me if I'm wrong and thanks for this wonderful package!

Thanks!


quick questions about the `BertModelLMHeadModel`.

Hello,
I have a few questions about the `BertModelLMHeadModel`:

1. Is `BertModelLMHeadModel`  used to conduct the causal language modeling (next token prediction), as it is the case for the `GPT2LMHeadModel`?

2. For `GPT2LMHeadModel`, I can just specify `labels = input_ids` for convenience. I just specify the `labels` in this way for the `BertModelLMHeadModel` as well?

Thanks,


Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained?

Hello,
Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained as well?
I am asking because like for the pre-trained `GPT2DoubleHeadsModel`, the main body of the model is trained but its multiple-choice head is not.

Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model untrained, just like the case for the `GPT2DoubleHeadsModel`?

Thank you,


Sharing Microsoft's DialogRPT (new dialog ranking model)

# 🌟 New model addition

## Model description

Thanks for the awesome work!

[DialogRPT](https://github.com/golsun/DialogRPT) (Dialog Ranking Pretrained Transformers) is a set of GPT-2 based dialogue ranking models recently released with an [EMNLP paper](https://arxiv.org/abs/2009.06978) by Microsoft Research. It's a follow-up work of [DialoGPT](https://huggingface.co/transformers/model_doc/dialogpt.html) (thanks for hosting it!)
The architecture is pretty simple: a `GPT2Model` followed by a `torch.nn.Linear(n_embd, 1, bias=False)`, and implemented based on a [previous HuggingFace commit](https://github.com/huggingface/transformers/commit/4d456542e9d381090f9a00b2bcc5a4cb07f6f3f7)
At first, I'm trying to create a model card for it, but then realized that it seems there's no existing model architecture in HuggingFace is compatible with DialogRPT. I noticed a lot of BERT-based sequence classification models, but ours is GPT-2 based.

If there's a simple fix (or I missed something) please let me know!
If implementation in modeling_gpt2.py is necessary, I'm also glad to help!

## Open source status

* [x] the model implementation is available: (https://github.com/golsun/DialogRPT)
* [x] the model weights are available: (https://github.com/golsun/DialogRPT)
* [x] who are the authors: @golsun @dreasysnail


`run_squad_trainer` doesn't actually use a Rust tokenizer + errors in `squad_convert_example_to_features` when using a Rust tokenizer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux-4.14.35-1902.303.4.1.el7uek.x86_64-x86_64-with-oracle-7.8
- Python version: 3.6.8
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

and

- `transformers` version: 3.3.1
- Platform: macOS-10.15.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@mfuntowicz
@LysandreJik
@patil-suraj

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

Firstly, in `run_squad_trainer.py`, I noticed that the "use_fast" arg doesn't get propagated into the tokenizer instantiation: https://github.com/huggingface/transformers/blob/0acd1ffa09a06084efa7cfa0e4e9d97cffdda5f9/examples/question-answering/run_squad_trainer.py#L107  Probably should be
```
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast
    )
```
However, when I make that change, the script hangs at the call to `squad_convert_examples_to_features` in SquadProcessor.

So, I did a little digging. The error is in `squad_convert_example_to_features` and seems to be due to inconsistencies in the behavior of `tokenizer.encode_plus` between the Python and Rust tokenizers, detailed below. I've also provided a [gist](https://gist.github.com/k8si/a143346dfa875c28d98e95cba1f82f1b) that hopefully elucidates & will help reproduce each of these points. I tested both BertTokenizer/BertTokenizerFast and GPT2Tokenizer/GPT2TokenizerFast.

1) Python tokenizers handle negative values for `stride`, Rust tokenizers throw an exception (`OverflowError: can't convert negative int to unsigned`)

2) For sequence pairs, Python tokenizers are fine if the first arg (`text`) is a list of ints and the second arg (`text_pair`) is a list of strings. The Rust tokenizers throw an exception `ValueError: PreTokenizedInputSequence must be Union[List[str], Tuple[str]]`. (Furthermore, the typehints for these arguments indicate that a string, a list of strings, or a list of ints are all fine.)

3) Leaving the `is_split_into_words` kwarg at its default value (`False`), then running `tokenizer.encode_plus(list of ints)` works fine for the Python tokenizers. The Rust tokenizers raise an exception `ValueError: TextInputSequence must be str`.

4) When running on a pair of sequences and setting `return_tensors=None`, the Python tokenizers return an output dict with input_ids (and other elements) as a list of ints i.e.`input_ids = [id1, id2, ...]` whereas the Rust tokenizers return a dict with input_ids as a list of list of ints i.e. `input_ids = [[id1, id2, ...]]`. I also noticed that if you set `return_tensors="pt"`, both the Python and Rust tokenizers return `input_ids = tensor([[id1, id2, ...]])`.

5) When `return_overflowing_tokens=True`, the Python tokenizers return a list of the overflowing tokens at key `overflowing_tokens` as expected. The Rust tokenizers return them at key `overflow_to_sample_mapping` which is not documented anywhere, as far as I can tell. The values seem to be different for the Python output vs. Rust output.

6) Running the same procedure on the same input twice produces the same result each time for the Python tokenizer. For the Rust tokenizer, the result of the second run is **different**. I am not familiar enough with the Rust tokenizer internals at this point to have a theory as to why this is the case. Anyway, this is the point at which I stopped debugging and decided to file an issue.

## To reproduce

Steps to reproduce the behavior:

1. Download squad 2.0 dataset from ["official" squad website](https://rajpurkar.github.io/SQuAD-explorer/)
2. Make fix in `run_squad_training.py` described above to correctly instantiate a Rust tokenizer
3. Run script: `python examples/question-answering/run_squad_trainer.py --model_name_or_path bert-base-uncased --use_fast --output_dir "./outputs-squad" --do_train --data_dir "./squad-data" --version_2_with_negative`

Also see gist detailing issues described above: https://gist.github.com/k8si/a143346dfa875c28d98e95cba1f82f1b

## Expected behavior

1) I expected `run_squad_trainer.py` to use a Rust tokenizer when the `use_fast` arg was set to True
2) I expected `SquadProcessor.squad_convert_example_to_features` to not raise exceptions when processing squad data when using a Rust tokenizer
3) I expected `tokenizer.encode_plus` to return the same outputs given the same inputs, regardless of whether the tokenizer is a Rust tokenizer or a Python tokenizer


Update README.md

model card

Related to pretrained model cards

@julien-c
Model is now fine-tuned on Transformers 3.1.0.  Previous model fine-tuned on Transformers 2.3.0 is out-of-date.


Clean the Trainer state

# What does this PR do?

This PRs clean the fields used inside the `Trainer` to store state and gather them all in a clear class named `TrainerState` that is typed so the user knows exactly what they can access when subclassing and overriding methods (and for my next step, when writing callbacks).

As a result, the `log_history` does not need to be saved and loaded separately and the hack to get the steps we were at from the checkpoint folder name is removed (the user can copy all the checkpoint saved by the `Trainer` in a previous training in any folder and use it for resuming training). This PRs adds a test of the full reproducibility of a training resumed from a checkpoint.

There is a tiny breaking change that would affect users that trained a model using an earlier version of transformers and would like to resume it with a version obtained after this commit but I don't think this matters much.

It also enforces that the `TrainingArguments` passed to a `Trainer` are not changed during training, to avoid subtle bugs when launching several trainings in a row. This is verified by a new test.


Use of global attention of Longformer when generating

I'm training Longformer2Roberta, the encoder part of this Seq2Seq model is Longformer. The one feature Longformer brings is global attention, I found the use of it during training, but it is never used at the point of generation. I guess it should be used somewhere here https://github.com/huggingface/transformers/blob/03e46c1de3864b8464a1b40d2a414b35f6b7f0df/src/transformers/generation_utils.py#L402.

I guess @patrickvonplaten is working on these models.


[s2s] fix kwargs style

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


[s2s] Fix t5 warning for distributed eval


Using BERT for spelling correction

I am currently working on the task of spelling correction. I used BERT by masking the misspelled word to get predictions with their probability score. However, the results are not so good. Hence, I want to fine-tune the BERT. For this spelling correction task, I would like to know which is the suitable method to fine-tune the BERT.

I would be glad if anyone could help me in this regard.


Tenosrflow Loading the saved Model For GPT2

```py
from time import time
from transformers import TFGPT2LMHeadModel, GPT2Tokenizer
import tensorflow as tf

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = TFGPT2LMHeadModel.from_pretrained('gpt2')

text = "What are you doing after you have finished working?"
generated = tokenizer.encode(text)
context = tf.constant([generated])
past = None
start = time()
for i in range(100):
    output, past = model([context, past])
    logits = output[0, -1, :]
    tok = tf.argmax(logits)
    generated.append(tok.numpy())
    context = tf.expand_dims(tf.expand_dims(tok, 0), 0)
sequence = tokenizer.decode(generated)
print(time() - start, sequence)

#save the model
tf.saved_model.save(model,"temp")

#loading back the model
nm = tf.saved_model.load("temp")
infer = nm.signatures['serving_default']
```
Not able to call the model as shown below

```py
text = "What are you doing after you have finished working?"
generated = tokenizer.encode(text)
context = tf.constant([generated])
past = None
start = time()
for i in range(100):
    output, past = **infer**([context, past])
    logits = output[0, -1, :]
    tok = tf.argmax(logits)
    generated.append(tok.numpy())
    context = tf.expand_dims(tf.expand_dims(tok, 0), 0)
sequence = tokenizer.decode(generated)
print(time() - start, sequence)
```


Bump isort version.

# What does this PR do?

Had some problem on my local setup with isort wanting to change `test_modeling_deberta.py`. Updating to 5.5.4 (from 5.4.2) fixed the issue so I think we should pin our setup to it.


Add forgotten return_dict argument in the docs

# What does this PR do?

The documentation wasn't updated to `return_dict=True` not being the default for all models. This PR fixes that.

Fixes #7482


Issue with Summary of the tasks - Named Entity Recognition in Docs

- `transformers` version: 3.3.1
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.3
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 examples/token-classification: @stefan-it
 documentation: @sgugger

## Information

I am trying to run the Pytorch version of named entity recognition from the "Summary of the tasks" section in the documentation.

## To reproduce

Steps to reproduce the behavior:

I'm running the exact example from the docs, but will attach the code below

```
from transformers import AutoModelForTokenClassification, AutoTokenizer

import torch

model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

label_list = [
    "O",       # Outside of a named entity
    "B-MISC",  # Beginning of a miscellaneous entity right after another miscellaneous entity
    "I-MISC",  # Miscellaneous entity
    "B-PER",   # Beginning of a person's name right after another person's name
    "I-PER",   # Person's name
    "B-ORG",   # Beginning of an organisation right after another organisation
    "I-ORG",   # Organisation
    "B-LOC",   # Beginning of a location right after another location
    "I-LOC"    # Location
]

sequence = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very" \
           "close to the Manhattan Bridge."

tokens = tokenizer.tokenize(tokenizer.decode(tokenizer.encode(sequence)))

inputs = tokenizer.encode(sequence, return_tensors="pt")

outputs = model(inputs).logits

predictions = torch.argmax(outputs, dim=2)
```

Running this leads to the error:

```
Traceback (most recent call last):

  File "test.py", line 21, in <module>

    outputs = model(inputs).logits

AttributeError: 'tuple' object has no attribute 'logits'
```

## Expected behavior

I expect this to run successfully and produce predictions for the example sequence. I've run this example before and it succeeded so I'm not sure what's happening differently now. I feel like I'm making a dumb mistake somewhere, but idk.
Thanks!

Note:
changing line 21 to
`outputs = model(inputs)[0]`
seems to lead to the expected output, but this might not be the kind of behavior you all are looking for.


Minor dead code clean-up

Hello,

I am not sure how sensitive you generally are about dead code in the repository. I have identified a few places with dead code, where I believe a clean-up would improve readability.

- removal of a couple of unused dropouts I came across in Albert and XLNet
- removal of an unused code block for relative attention shift for XLNet

## Who can review?

@LysandreJik , @TevenLeScao


Upload models using transformers-cli fails

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-112-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 Model Cards: @julien-c
 T5: @patrickvonplaten

## Information

Model I am using T5:

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Command:
`transformers-cli upload ./prot_t5_xl_bfd/ --organization Rostlab`

Error:
```
About to upload file /mnt/lsf-nas-1/lsf/job/repo/elnaggar/prot-transformers/models/transformers/prot_t5_xl_bfd/pytorch_model.bin to S3 under filename prot_t5_xl_bfd/pytorch_model.bin and namespace Rostl
ab
Proceed? [Y/n] y
Uploading... This might take a while if files are large
  0%|▌                                                                                | 48242688/11276091454 [00:02<14:55, 12534308.31it/s]
Traceback (most recent call last):
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1049, in _send_output
    self.send(chunk)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 971, in send
    self.sock.sendall(data)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

Traceback (most recent call last):
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 1049, in _send_output
    self.send(chunk)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/http/client.py", line 971, in send
    self.sock.sendall(data)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/transformers/commands/transformers_cli.py", line 33, in main
    service.run()
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/transformers/commands/user.py", line 232, in run
    access_url = self._api.presign_and_upload(
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/transformers/hf_api.py", line 167, in presign_and_upload
    r = requests.put(urls.write, data=data, headers={"content-type": urls.type})
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/requests/api.py", line 134, in put
    return request('put', url, data=data, **kwargs)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/lsf-nas-1/lsf/job/repo/elnaggar/anaconda3/envs/transformers_covid/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

```

## Expected behavior

I am trying to upload our T5-3B model using transformers-cli, but it always fails and gives "BrokenPipeError".
It only uploads small files like configuration files but it fails for the model files.
I have tried two different machines and both of them gives the same error.


Loading saved model not working

Do we know do to load the saved model pipeline back up and make predictions again locally? The from_pretrained()is not working.
Pls provide few instructions how to load the model using from pretrained


Alphabetize model lists

# What does this PR do?

The model lists have grown a bit and so, like the doc navbar, I think we'll find our way better by alphabetizing them.
Adding new models will be easy in the README since Markdwon supports enumerated lists with 1. for all items. RestructuredText is more annoying but I'll make one script generate the proper part of index.rst automatically to make sure it's in sync with the README while I'm procrastinating something more important.


[s2strainer] fix eval dataset loading

`eval_dataset` should be loaded if either `--do_eval` or`EvaluationStrategy` is not `no`

@sshleifer


RAG: Can we have a document that explains the fine-tuning mechanism?

I want to fine-tune RUG with a custom dataset. Please help me.


Small QOL improvements to TrainingArguments

# What does this PR do?

Some small QOL improvements as discussed on the [forum](https://discuss.huggingface.co/t/seq2seqtrainer-questions/1276/):
- make `do_eval` defaults to `evaluation_strategy != "no"` so there is no need to pass the two
- make `run_name` defaults to `output_dir`.


[Seq2Seq] Fix a couple of bugs and clean examples

# What does this PR do?

## **IMPORTANT** - BREAKING CHANGES
This PR changes the behavior of **T5** and **TFT5** due to 3 bugs and 1 small change in forward API to support onnx and torchscript.
It also slighlty changes the behavior of **Bart** and **EncoderDecoderModel**

## Description

_1st Bug_: Due to a sloppy review from my part these lines got merged into the PyTorch T5 model: https://github.com/huggingface/transformers/pull/5518/files#r496058908, which set `decoder_input_ids = input_ids` if `decoder_input_ids` were not provided. This is misleading and also just wrong. Never should `decoder_input_ids=input_ids` in T5. This is not done during training nor during inference, so these lines don't make much sense. Because T5 is mostly either used with `.generate()` or in training with `model(input_ids=input_ids, labels=labels)`, in which cases the change has no effect we only received one issue recently about it #7358. The change was done to make T5 work with onnx, but is just wrong IMO.

_2nd Bug_: T5 was implemented with a small bug regarding the relative distance bias calculation for the cross-attention layer. It was spotted here: #7323. The correction leads to slightly different results when doing beam search. @sshleifer - if it's easy for you could maybe run a quick eval on WMT to see if bleu improves in this PR?

_3rd Bug_: T5 currently cuts the `input_ids` to the last token when `past` is used. This is a convenient function for the user, but has the potential to lead to bugs as mentioned here: https://github.com/huggingface/transformers/issues/4368#issuecomment-630244541. It's not really in the spirit of the library to do some magic under-the-hood which make certain use-cases easier for the user, but prevents other edge cases as shown in the issue above.

Feature request: support torchscript and onnx. This PR allows to use T5 with torchscript and onnx.

Now the difficult part: **This PR has breaking changes!**

For once, all three bugs lead to breaking changes. Then in order to solve the torchscript, onnx problem we are having with T5 (and we actually have with all Seq2Seq models), I had to change the positional ordering of T5's forward pass slightly, which should have minimal breaking changes because I doubt anybody has used T5 with positional arguments as follows:

`tf_model(input_ids, None, None, decoder_input_ids)`.

We had a couple of issues *e.g.* #5647 about supporting torchscript and onnx for Bart/T5. If we ever want to support onnx and torchscript in the future, I think we need to do this positional reordering. As shown by @mfuntowicz onnx can lead to great speed improvements and we also know now that `torchscript` can give ~30% speed improvement on dynamic input sizes. => I would be really happy if we could accept this slight breaking change here.

I thought about this quite a bit and I think it's very important that we agree on ONE positional argument ordering for the forward pass of Seq2Seq models. At the moment the ordering of Bart, EncoderDecoder, T5, ... is not coherent and done in a way that does not support onnx and torchscript. At the moment no seq2seq model really supports torchscript (Bart does in the test, but one cannot provide `decoder_input_ids` when using torchscript which effectively makes torchscript useless for inference). The ordering should be as follows IMO:

`input_ids`
`attention_mask`
`decoder_input_ids`
`decoder_attention_mask`
`encoder_outputs`
...,

meaning that all `required` inputs should come first to comply with onnx and torchscript and optional ones should come after.

I changed the ordering of all seq2seq models to comply with this format even though we have some positional ordering breaking changes for `T5`, `Bart` and `EncoderDecoder`.

## UPDATE:

Added tests that the encoder decoder forward signature stays the same
Applied changes to all Seq2Seq models
Cleaned docs
Fixed TF slow tests


Make transformers install check positive

# What does this PR do?

When transformers is correctly installed, I feel you should get a positive message. It's called huggingface not angryface, after all ;-)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed.
Given this concerns the documentation, mentioning @sgugger


Number of GPUs for multi-gpu

Print number of GPUs when running the multi-gpu testing suites.


Fix LXMERT with DataParallel

This PR fixes LXMERT when using DataParallel, similar to https://github.com/huggingface/transformers/pull/4300


Seq2SeqDataset: avoid passing src_lang everywhere

Changed Constructor argument for AbstractSeq2SeqDataset to kwargs to avoid passing unwanted parameters to tokenizers,
eg. src and tgt lang to T5 tokenizer.

# What does this PR do?
tokenization_util.py was generating unnecessary warning statements continuously for tokenizer arguments for seq2seq batch processing that shouldn't have been passed in the first place. Fixed this phenomenon and added a test case as suggested.

Fixes # (issue)
#7454

## Before submitting
- This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
N/A
-Did you read the [contributor guideline]
Yes
- Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
Yes, #7454
- Did you make sure to update the documentation with your changes? Here are the
  N/A
-Did you write any new necessary tests?
Yes, added a relevant test in examples/seq2seq/test_datasets.py

## Who can review?
@sshleifer


fix the first chunk's lower triangle

Correct the first chunk's lower triangle.
For details, please look at Page 4 in a google shared document.
Link is: https://docs.google.com/document/d/12rv879j2m5VkfTvk0F-WSOPFF5gqbE0kgk60PvY5nHc/edit?usp=sharing

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Create README.md

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


[s2sTrainer] test + code cleanup

- add a 10 second test for Seq2SeqTrainer
- general code cleanup
- pass `data_args` to Seq2SeqTrainer

@patil-suraj


Seq2SeqTrainer: add a fast test that doesn't learn anything but can run on CPU

@patil-suraj do you want to take this or should I?


RAG - reproducing RAG-Sequence QA score

I'm trying to reproduce RAG-Sequence NQ score of 44.5 presented in Table 1 of the paper at https://arxiv.org/abs/2005.11401.

I used the command in the examples/rag readme
```bash
python examples/rag/eval_rag.py \
    --model_name_or_path facebook/rag-sequence-nq \
    --model_type rag_sequence \
    --evaluation_set path/to/test.source \
    --gold_data_path path/to/gold_data \
    --predictions_path path/to/e2e_preds.txt \
    --eval_mode e2e \
    --gold_data_mode qa \
    --n_docs 5 \
    --print_predictions \
    --recalculate \
```

For gold_data_path I used data.retriever.qas.nq-test from DPR repo, consisting of 3610 questions and answers: https://github.com/facebookresearch/DPR/blob/master/data/download_data.py#L91-L97

For evaluation_set, my understanding it should be the questions, so I extracted just the questions from the qas.nq-test csv file.

I tried the above command with n_docs 5 and 10, with the following results:

n_docs 5
INFO:__main__:F1: 49.67
INFO:__main__:EM: 42.58

n_docs 10
INFO:__main__:F1: 50.62
INFO:__main__:EM: 43.49

With n_docs 10 it's still 1 point below the score in paper. What would be the proper setup to reproduce the number, is the pretrained model loaded different, higher n_docs, or different test data?

Thanks in advance!


Remove config assumption in Trainer

# What does this PR do?

This PR tries to limit the access to `model.config` in `Trainer` to the minimum so that it works with regular PyTorch modules (as long as they accept dict inputs and return loss first like our models). The most challenging part was the storing/restoring of the `total_flos`, which I moved to the newly created `TrainerState`. It should work as before and be saved along the rest of the training state.


Trainer should not modify its TrainingArguments

# What does this PR do?

This fixes a bug that took me some time to track in a notebook with several trainings. The bottom line is that `Trainer` should not modify its `TrainingArguments` so this fixes that part by saving the number of `max_steps` desired in the state instead of in the args. Also storing the number of training epochs for easy access in subclasses.


RAG - how to precompute custom document index?

Was wondering if there was any code snippet / blog post showing how one could load their own documents and index them, so they can be used by the RAG retriever.

Cheers!


Distributed Trainer: 2 little fixes

1) fix DDP access to `model.config`. We could also set `self.config = model.config` earlier in `__init__`
2) switch torch.Tensor -> torch.tensor. The latter "infers the dtype automatically"
After which the command in #7460 works.

CC @patil-suraj , @TevenLeScao


Seq2SeqTrainer Distributed: AttributeError and the RuntimeError

The following command (on 8 GPUS) fails with

```python
AttributeError: DistributedDataParallel has no attribute "config"
```

### Command
```bash
export WANDB_PROJECT=dmar
export BS=64
export GAS=1
export m=sshleifer/student_marian_en_ro_6_3
export MAX_LEN=128
python -m torch.distributed.launch --nproc_per_node=8 finetune_trainer.py \
    --tokenizer_name $m --model_name_or_path $m \
    --data_dir wmt_mar_pl \
    --output_dir marian_en_ro_6_3 --overwrite_output_dir --predict_with_generate \
    --learning_rate=3e-4 \
    --warmup_steps 500 --sortish_sampler \
    --fp16 \
    --gradient_accumulation_steps=$GAS \
    --per_device_train_batch_size=$BS --per_device_eval_batch_size=$BS \
    --freeze_encoder --freeze_embeds \
    --num_train_epochs=6 \
    --save_steps 3000 --eval_steps 3000 \
    --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
    --do_train --do_eval --do_predict --evaluate_during_training\
    --predict_with_generate --logging_first_step \
    --task translation --label_smoothing 0.1 --n_gpu 8 \
    --run_name builtin_trainer_63_v8_pl \
    "$@"
```


Update README.md

Related to pretrained model cards

Update/reference v2 model


Fix Trainer tests in a multiGPU env

# What does this PR do?

Should fix the multiple GPU CI test (tests are passing locally). Will merge as soon as the CI passes to make the CI green.


Get a better error when check_copies fails

# What does this PR do?

Prints a cleaner error message when `check_copies.py` encounters a bad copy.


Catch import datasets common errors

# What does this PR do?

This PR adds more checks when trying to import datasets to check we actually are using the datasets library and not a local folder/module.

Fixes #7430


Adding the Streamlit demo app code for the RAG model

# Adding RAG demo code

This PR shares the code for the RAG demo running [here](https://huggingface.co/rag/) for future reference. The code is added in `examples/rag`


Seq2seq example for T5 keeps on generating warning

In the latest version from Master, on running finetune.sh for T5, was getting the following warning continuously:
**Keyword arguments {'src_lang':None,'tgt_lang':None,'add_prefix_space':False} not recognized.**

I found out this is because the translation and BART parameters are being passed to prepare_seq2seq_batch of T5Tokenizer which it cannot handle and the tokenizer in the end spits out warning for unused kwargs.
I made a small change in utils.py to the constructor at https://github.com/huggingface/transformers/blob/9e9a1fb8c75e2ef00fea9c4c0dc511fc0178081c/examples/seq2seq/utils.py#L100:
```python
        **dataset_kwargs
    ):
        super().__init__()
        self.src_file = Path(data_dir).joinpath(type_path + ".source")
        self.tgt_file = Path(data_dir).joinpath(type_path + ".target")
        self.len_file = Path(data_dir).joinpath(type_path + ".len")
        if os.path.exists(self.len_file):
            self.src_lens = pickle_load(self.len_file)
            self.used_char_len = False
        else:
            self.src_lens = self.get_char_lens(self.src_file)
            self.used_char_len = True
        self.max_source_length = max_source_length
        self.max_target_length = max_target_length
        assert min(self.src_lens) > 0, f"found empty line in {self.src_file}"
        self.tokenizer = tokenizer
        self.prefix = prefix if prefix is not None else ""

        if n_obs is not None:
            self.src_lens = self.src_lens[:n_obs]
        self.pad_token_id = self.tokenizer.pad_token_id
        self.dataset_kwargs = dataset_kwargs
        dataset_kwargs.update({'add_prefix_space' : True} if isinstance(self.tokenizer, BartTokenizer) else {})
```

since src_lang and tgt_lang weren't being used anywhere else other than passing on to prepare_seq2seq_batch as parameters.

While calling the method I used dataset_kwargs as the paremeter which sorted out the issue:
```python
self.tokenizer.prepare_seq2seq_batch(
            [x["src_texts"] for x in batch],
            tgt_texts=[x["tgt_texts"] for x in batch],
            max_length=self.max_source_length,
            max_target_length=self.max_target_length,
            return_tensors="pt",
            **self.dataset_kwargs
        )

```

If this seems reasonable I can raise a PR and check it in?
@sshleifer @patil-suraj


Multi-GPU Testing setup

# What does this PR do?

This PR contributes a testing suite that runs on a multi-GPU machine.
The machine has two T4 GPUs (better than k80 in almost every way, and cheaper), and the testing suite is identical to the single-GPU machine testing suite. Two jobs are run:
- One job on each commit to the `master` branch
- One job on a scheduled basis, that additionally runs all the slow tests.


LayoutLM: add exception handling for bbox values

# What does this PR do?

Fixes unhandled error when trying to use bbox balues greater than maximum allowed threshold 1000. To replicate error:

- In `test_modelling_layoutlm.py` set `range_bbox=1025`, i.e. greater 1024
- Run `pytest tests/test_modeling_layoutlm.py`

Requirement for bbox values to be within the range 0-1000 is documented
but if it is violated then it is not clear what is the issue from error
message.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

@sgugger
@liminghao1630 @vblagoje


T5 unsupervised training

I want to train T5 in a new language from scratch an I think the best way to do this is through the unsupervised denoising task (because you can have all text you want! no labels required! hurra!)

However I have some doubts, I've seen [the HuggingFace  documentation about this](https://huggingface.co/transformers/model_doc/t5.html#training) and I wonder how to create the train data, I mean, is there any function in the library to add the sentinel tokens? In my own research I've worked with [the original T5 library](https://github.com/google-research/text-to-text-transfer-transformer) and I've seen that in this library they have some functions to do that, but this functions not apply the sentinel_tokens to the text but it replace the "noising" tokens by " always.

My questions are:

1. It exists some functions to do that in HuggingFace?
2.  If the question 1 answer is not, anybody have any?
3. If the question 1 and 2 answers are not, What are the rules I must follow to create this function? Rules like:
           - How much concatenated tokens must the sentinel tokens mask? I mean, in the sentence "The cute dog walks in the park” they put the sentinel tokens in "cute dog" and my question is why this words election? I can take 1 token forever?
           - In each sentence I must start using the sentinel token number 1?

Thank you in advance! :)


deleted

Deleted


What's the most straightforward way to initialise BertForSequenceClassification for different token rather than [CLS]?

# ❓ Questions & Help

## Details
BertForSequenceClassification uses [CLS] token's representation to feed a linear classifier. I want to leverage another token (say [X] in the input sequence) rather than [CLS]. What's the most straightforward way to implement that in Transformers?

**A link to original question on the forum/Stack Overflow**: https://stackoverflow.com/questions/64094098/how-to-initialize-bertforsequenceclassification-for-different-input-rather-than


v3.3.0 - Issue with name conflict in transformers & datasets - AttributeError: module 'datasets' has no attribute '__version__'

Version 3.3.0 tries to import the module [datasets](https://pypi.org/project/datasets/):
https://github.com/huggingface/transformers/blob/v3.3.0/src/transformers/file_utils.py#L69

However, this can cause some undesirable behavior if there is a "datasets" folder in the same folder.

An example to re-produce the error:
```
datasets/   <= Folder that contains your own data files
myscript.py
```

myscript.py with the following content:
```
import transformers
```

This produces the following error:
```
python myscript.py
Traceback (most recent call last):
  File "myscript.py", line 1, in <module>
    import transformers
  File "/home/user/miniconda3/envs/sberttest/lib/python3.7/site-packages/transformers/__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip
  File "/home/user/miniconda3/envs/sberttest/lib/python3.7/site-packages/transformers/integrations.py", line 42, in <module>
    from .trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun  # isort:skip
  File "/home/user/miniconda3/envs/sberttest/lib/python3.7/site-packages/transformers/trainer_utils.py", line 6, in <module>
    from .file_utils import is_tf_available, is_torch_available, is_torch_tpu_available
  File "/home/user/miniconda3/envs/sberttest/lib/python3.7/site-packages/transformers/file_utils.py", line 72, in <module>
    logger.debug(f"Succesfully imported datasets version {datasets.__version__}")
AttributeError: module 'datasets' has no attribute '__version__'
```

The issue is with the import logic of Python. The datasets-folder will be treated as a module and transformers tries to load this module. This obviously fails, as we talk here about the datasets-folder and not [datasets package](https://pypi.org/project/datasets/).

As *datasets* is a quite common folder name in many setups to contain the files for the own datasets, I can image that this name collision will appear frequently. As soon as there is a datasets folder, you can no-longer import transformers.

## Solution
I am not sure what the best solution is for this. One quick fix would be to change:
https://github.com/huggingface/transformers/blob/v3.3.0/src/transformers/file_utils.py#L74

to
```
except:
    _datasets_available = False
```

This would catch all exceptions. Old scripts, that have a `datasets/` folder would then still be working.

## Environment info
- `transformers` version: 3.3.0
- Platform: Linux-4.15.0-39-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (False)
- datasets package is not installed


Getting Bert Embeddings in Batch

Hi all

I have a list of sentences(a batch during training) and for every word in each sentence, I need an aligned bert embedding which should be the mean of every word-piece that word was split into.

Right now, I am doing it sentence by sentence and obtain the aligned embedding for every word by reiterating over the sentence, tokenise the individual word, note the number of word-pieces it was split into and lookup into the Bert embedding matrix to average out those rows of the matrix. Following is the code to get the aligned embeddings:

`

       def get_bert_aligned_embeddings(self, last_hidden_states, tokens):

            count = 0
            aligned_embeddings = []

            for i in tokens:

                 tokenisation_length = len(self.tokenizer.tokenize(i))

                 emb = torch.mean(last_hidden_states[count:count+tokenisation_length], axis = 0)

                 count += tokenisation_length

                 aligned_embeddings.append(emb)

            aligned_embeddings = torch.stack(aligned_embeddings)

            return aligned_embeddings
`

tokens is the list of words in a sentence, last_hidden_states are the embeddings I obtained from Bert.

So, the above function runs for every sentence and every sentence is passed one by one to Bert Model.

I want to know if there is any faster way of doing this? Can this entire process be done in batches? Any suggestions that could help me speed up this process would be great. Thanks


Adding gradient checkpointing to GPT2

This PR adds gradient checkpointing capabilities to GPT-2, imitating the Longformer and Bert checkpointing code. It also disables `find_unused_parameters` in Trainer if the model is using gradient checkpointing, as per #4659 they are incompatible.


Add is_split_into_words as an argument to tokenize

# What does this PR do?
Two calls to `self.tokenize` in `tokenization_utils.py` were missing the argument `is_split_into_words`. Since `is_split_into_words` is not present in the `kwargs`, `self.tokenize` resorts to the default behavior of not adding a space before every word.
This PR fixes this issue by adding the missing arguments in the calls to self.tokenize().

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Environment info

- `transformers` version: 3.3.0
- Platform: Linux-4.15.0-99-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.3.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?:No
- Using distributed or parallel set-up in script?: No

### Who can help
 tokenizers: @mfuntowicz
 Trainer: @sgugger (because this might be related to https://github.com/huggingface/transformers/pull/7236)
## Information

`tokenizer.encode`, `tokenizer.encode_plus`, and `tokenizer.batch_encode_plus` ignore the flag `is_split_into_words`.

## To reproduce

Steps to reproduce the behavior:

```py
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
print(tokenizer.encode("happened", is_split_into_words=True))
print(tokenizer.encode_plus("happened", is_split_into_words = True)
)
print(tokenizer.batch_encode_plus(["happened"], is_split_into_words=True))
```
## Actual behavior
The word ``happened`` is tokenized without a space prefix which would be expected because of `is_split_into_words=True`:

```
[0, 298, 3340, 4490, 2]
{'input_ids': [0, 298, 3340, 4490, 2], 'attention_mask': [1, 1, 1, 1, 1]}
{'input_ids': [[0, 298, 3340, 4490, 2]], 'attention_mask': [[1, 1, 1, 1, 1]]}
```

## Expected behavior
a space should be prefixed in front of "happened" before tokenization, giving the following outputs:

```
[0, 1102, 2]
{'input_ids': [0, 1102, 2], 'attention_mask': [1, 1, 1]}
{'input_ids': [[0, 1102, 2]], 'attention_mask': [[1, 1, 1]]}
```


Update README.md

Related to pretrained model cards

Hi, just corrected the example code, add 2 links and fixed some typos

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Error training GPT-2 from scratch on Hindi

I was trying to retrain GPT-2 from scratch. I was able able to train a tokenizer but was facing issues while running run_language_modeling.py.
```
2020-09-29 06:00:14.450718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
/usr/local/lib/python3.6/dist-packages/transformers/training_args.py:299: FutureWarning: The `evaluate_during_training` argument is deprecated in favor of `evaluation_strategy` (which has more options)
  FutureWarning,
09/29/2020 06:00:16 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: False
09/29/2020 06:00:16 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/content/data', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=True, evaluation_strategy=<EvaluationStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=2, per_device_eval_batch_size=2, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Sep29_06-00-16_fcba31604e1d', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=2, no_cuda=False, seed=108, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name=None, disable_tqdm=False, remove_unused_columns=True, label_names=None)
Traceback (most recent call last):
  File "/content/transformers/examples/language-modeling/run_language_modeling.py", line 313, in <module>
    main()
  File "/content/transformers/examples/language-modeling/run_language_modeling.py", line 205, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_auto.py", line 251, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1428, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1575, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_albert.py", line 155, in __init__
    self.sp_model.Load(vocab_file)
  File "/usr/local/lib/python3.6/dist-packages/sentencepiece.py", line 367, in Load
    return self.LoadFromFile(model_file)
  File "/usr/local/lib/python3.6/dist-packages/sentencepiece.py", line 177, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
```
Here is the link to my colab file. https://colab.research.google.com/drive/1rWHwWCB_U_rTOnfGyXdgZ9Kb4HpbNdVs?usp=sharing


Setting up transformers/examples/seq2seq

 @sshleifer

I'm following the instructions in the README for seq2seq. In particular, I forked, cloned, and then ran "pip install -e ."

But then when I tried to run finetune.sh, a number of libraries had not been installed. I had to just manually install them with pip (e.g. rouge_score, git, sacrebleu). Should these all have been included automatically? Or was there a better way to get them?

Is this still the best way to go about finetuning a seq2seq model on a custom task (in my case I am doing T5)?


Faced the TypeError:forward() got an unexpected keyword argument 'output_all_encoded_layers'

## Environment info
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.3
- PyTorch version (GPU?): 1.1.0 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

  Trainer: @sgugger
 TransfoXL/XLNet: @TevenLeScao

 -->

## Information

Model I am using XLNet...:

The problem arises when using:

* [ ] my own modified scripts like below url:
[chinese-bert-pytorch](https://github.com/649453932/Bert-Chinese-Text-Classification-Pytorch)

The tasks I am working on is:
My own dataset Chinese Text Classification
## To reproduce

Steps to reproduce the behavior:

1.Firstly, load the xlnet model and tokenize:
```heisenberg
from pytorch_transformers import XLNetModel,XLNetTokenizer,XLNetConfig
class Model(nn.Module):

    def __init__(self, config):
        super(Model, self).__init__()
        self.bert =XLNetModel.from_pretrained(config.bert_path)
        for param in self.bert.parameters():
            param.requires_grad = True
        self.fc = nn.Linear(config.hidden_size, config.num_classes)

    def forward(self, x):
        context = x[0]  # 输入的句子
        mask = x[2]  # 对padding部分进行mask，和句子一个size，padding部分用0表示，如：[1, 1, 1, 1, 0, 0]
        _, pooled = self.bert(context, attention_mask=mask, output_all_encoded_layers=False)
        out = self.fc(pooled)
        return out
```
2.Then start training finetuning and evaluating:
```
def train(config, model,model_name, train_iter, dev_iter, test_iter):
    model.train()
    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}]
    # optimizer = torch.optim.Adam(model.parameters(), lr=config.learning_rate)
    optimizer = BertAdam(optimizer_grouped_parameters, lr=config.learning_rate,warmup=0.05,t_total=len(train_iter) * config.num_epochs)
    total_batch = 0
    dev_best_loss = float('inf')
    last_improve = 0
    flag = False
    model.train()
    for epoch in range(config.num_epochs):
        print('Epoch [{}/{}]'.format(epoch + 1, config.num_epochs))
        for i, (trains, labels) in enumerate(train_iter):
            outputs = model(trains)
            model.zero_grad()
            loss = F.cross_entropy(outputs, labels)
            loss.backward()
            optimizer.step()
            if total_batch % 100 == 0:
                true = labels.data.cpu()
                predic = torch.max(outputs.data, 1)[1].cpu()
                train_acc = metrics.accuracy_score(true, predic)
                dev_acc, dev_loss = evaluate(config, model, dev_iter)
```

3. But Error Occured:
```
$ python run.py --model xlnet_base
Loading data...
401it [00:01, 225.08it/s]
140it [00:00, 260.37it/s]
135it [00:00, 240.91it/s]
Time usage: 0:00:03
Epoch [1/1]
Traceback (most recent call last):
  File "run.py", line 40, in <module>
    train(config, model,model_name, train_iter, dev_iter, test_iter)
  File "F:\PycharmProjects\Bert-Chinese-Text-Classification-Pytorch-master\train_eval.py", line 52, in train
    outputs = model(trains)
  File "D:\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "F:\PycharmProjects\Bert-Chinese-Text-Classification-Pytorch-master\models\xlnet_base.py", line 47, in forward
    _, pooled = self.bert(context, attention_mask=mask, output_all_encoded_layers=False)
  File "D:\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'output_all_encoded_layers'
```
4. I googled the error and found the [issuses ](https://github.com/huggingface/transformers/issues/3541) in transformers:
So I changed the Model Load code like below:
```
class Model(nn.Module):

    def __init__(self, config):
        super(Model, self).__init__()
        model_config = XLNetConfig.from_pretrained(config.bert_path,output_hidden_states=False)
        self.bert = XLNetModel.from_pretrained(config.bert_path,config=model_config)
        for param in self.bert.parameters():
            param.requires_grad = True
        self.fc = nn.Linear(config.hidden_size, config.num_classes)

    def forward(self, x):
        context = x[0]
        mask = x[2]
        _, pooled = self.bert(context, attention_mask=mask, output_all_encoded_layers=False)
        out = self.fc(pooled)
        return out
```
BUT I STILL encounter the same problem, I dunno why.

Hope Ur Reply. Thanks A Lot!


creating readme for bert-base-mongolian-uncased

Related to pretrained model cards

I am adding the model card for bert-base mongolian uncased.

Can you review this for me please!


Creating readme for bert-base-mongolian-cased

Related to pretrained model cards

I am adding pretrained BERT-base models to model hub.
Please review this for me


CUDA out of memory (ALBERT) - run_squad.py ignores --per_gpu_train_batch_size

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.0
- Platform: Colab Pro (P100) / Anaconda (Windows / 2080 Ti)
- Python version: 3.6
- PyTorch version (GPU?): 1.6.0 / CUDA 10.2
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): ALBERT

The problem arises when using:
* [x] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD 2.0
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fine-tune ALBERT-xlarge or xxlarge and  set --per_gpu_train_batch_size 8 or 10
2. try to finetune

```
!python transformers\examples\question-answering\run_squad.py \
  --model_type albert \
  --model_name_or_path albert-large-v2 \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file train-v2.0.json \
  --predict_file dev-v2.0.json \
  --per_gpu_train_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 1.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/model_output \
  --save_steps 1000 \
  --threads 4 \
  --version_2_with_negative \
  --overwrite_output_dir
```
## Expected behavior

The model should be training, but despite the 8gb limit there is an out of memory error:
```
RuntimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 15.90 GiB total capacity; 15.01 GiB already allocated; 7.88 MiB free; 15.03 GiB reserved in total by PyTorch)
```


RAG Retriever  (NameError: name 'load_dataset' is not defined   in retrieval_rag.py)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.0
- Platform: Linux-4.19.0-11-cloud-amd64-x86_64-with-debian-10.6
- Python version: 3.7.3
- PyTorch version (GPU?): 1.6.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help
@sshleifer
RAG model is not on the list, but this is summarization related
 -->

## Information

Model I am using RAG

The problem arises when using:
* [ +] the official example scripts: (give details below)
``` python from transformers import RagTokenizer, RagRetriever, RagSequenceForGeneration
import torch

tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True)
# initialize with RagRetriever to do everything in one forward call
model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
```

The tasks I am working on is:
Model coudln't load, didn't perform any task

## To reproduce

Steps to reproduce the behavior:

1. run the code
``` python from transformers import RagTokenizer, RagRetriever, RagSequenceForGeneration
import torch

tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True)
# initialize with RagRetriever to do everything in one forward call
model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
```

## Expected behavior

Raise a NameError, load_dataset is not defined.

```python
NameError                                 Traceback (most recent call last)
<ipython-input-6-752205d4a1c8> in <module>
      3
      4 tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq")
----> 5 retriever = RagRetriever.from_pretrained("facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True)
      6 # initialize with RagRetriever to do everything in one forward call
      7 model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)

/mnt/disks/nlp/env_nlp_main/lib/python3.7/site-packages/transformers/retrieval_rag.py in from_pretrained(cls, retriever_name_or_path, **kwargs)
    307         generator_tokenizer = rag_tokenizer.generator
    308         return cls(
--> 309             config, question_encoder_tokenizer=question_encoder_tokenizer, generator_tokenizer=generator_tokenizer
    310         )
    311

/mnt/disks/nlp/env_nlp_main/lib/python3.7/site-packages/transformers/retrieval_rag.py in __init__(self, config, question_encoder_tokenizer, generator_tokenizer)
    287                 config.retrieval_vector_size,
    288                 config.index_path,
--> 289                 config.use_dummy_dataset,
    290             )
    291         )

/mnt/disks/nlp/env_nlp_main/lib/python3.7/site-packages/transformers/retrieval_rag.py in __init__(self, dataset_name, dataset_split, index_name, vector_size, index_path, use_dummy_dataset)
    218
    219         logger.info("Loading passages from {}".format(self.dataset_name))
--> 220         self.dataset = load_dataset(
    221             self.dataset_name, with_index=False, split=self.dataset_split, dummy=self.use_dummy_dataset
    222         )

NameError: name 'load_dataset' is not defined
```


Create README.md

Related to pretrained model cards

MagBERT-NER : Added widget (Text)


[s2s] consistent output format across eval scripts


Document new features of make fixup

# What does this PR do?

This is a small follow-up on #7403 documenting the behavior it introduced, as instructed by @stas00.


Add a code of conduct

# What does this PR do?

This PR adds a code of conduct to the project, inspired by the [Contributor Covenant](https://www.contributor-covenant.org/). To make it clearly visible it also adds:
- a badge that displays under Transformers on the main README.
- a link in the contributing guide.


Fine-tune BERTForMaskedLM

Hello,

I am doing a project on spelling correction. I used pre-trained "bert-base-cased" model. However, the results are not that accurate. Therefore, I planned to fine-tune the BERT for Masked LM task. I couldn't find any examples for fine-tuning BERT model for Masked LM. I tried to use "run_language_modeling.py" for fine-tuning. But, I came across with the following error:

```
C:\Users\ravida6d\spell_correction\transformers\examples\language-modeling>python run_language_modeling.py --output_dir ="C:\\Users\\ravida6d\\spell_correction\\contextualSpellCheck\\fine_tune\\" --model_type = bert --model_name_or_path = bert-base-cased --do_train --train_data_file =$TRAIN_FILE --do_eval --eval_data_file =$TEST_FILE –mlm

C:\Users\ravida6d\AppData\Local\Continuum\anaconda3\envs\contextualSpellCheck\lib\site-packages\transformers\training_args.py:291: FutureWarning: The `evaluate_during_training` argument is deprecated in favor of `evaluation_strategy` (which has more options)
  FutureWarning,

Traceback (most recent call last):
  File "run_language_modeling.py", line 313, in <module>
    main()
  File "run_language_modeling.py", line 153, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "C:\Users\ravida6d\AppData\Local\Continuum\anaconda3\envs\contextualSpellCheck\lib\site-packages\transformers\hf_argparser.py", line 151, in parse_args_into_dataclasses

    raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")
ValueError: Some specified arguments are not used by the HfArgumentParser: ['bert', 'bert-base-cased']
```

I am not understanding how to use this script. Can anyone give some information for understanding the fine-tuning of BERT Masked LM.


Add automatic best model loading to Trainer

# What does this PR do?

This PR cleans up a bit the part that saves the training state inside `Trainer` and adds an API that can track which was the best model during any of the evaluation phases to load it back at the end.

When fine-tuning a model on a dataset that can easily overfit the model, it's quite common to have the last model not be the best one (in terms of metrics). This PR adds a `TrainingArgument` named `load_best_model_at_end` that triggers the following behavior:
- `save_steps` gets ignored and the model is saved every time there is an evaluation (determined by `evaluation_strategy` and `eval_steps`)
- It keeps track in a `TrainerState` of when the best model was encountered (that state is saved along the checkpoints so it can work with resuming a training)
- The best model is determined by the new `TrainingArgument`s `metric_for_best_model` (defaults to the loss) and `greater_is_better` (default to False for the loss, True otherwise).
- The best model is loaded once the training is finished.

In passing I've added some tests of the saving API in Trainer and made sure it can handle both `PreTrainedModel` and regular `nn.Module` (a feature asked in #6901). Both are now tested in the CI, as is the new API.

Fixes #6901

Those newly introduced arguments and APIs can then be leveraged to have early stopping supported in `Trainer`.


import error in version 3.3.0, conflict with local directory "datasets"

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.0
- Platform: Google Colab

Model I am using ：Bert

## To reproduce

Steps to reproduce the behavior:

Traceback (most recent call last):
  File "train.py", line 19, in <module>
    from mydataset import load_data,dist_load_data,load_data2
  File "/content/drive/My Drive/mrc4ner/mydataset.py", line 5, in <module>
    from transformers import BertTokenizer
  File "/usr/local/lib/python3.6/dist-packages/transformers/__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 42, in <module>
    from .trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun  # isort:skip
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_utils.py", line 6, in <module>
    from .file_utils import is_tf_available, is_torch_available, is_torch_tpu_available
  File "/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py", line 72, in <module>
    logger.debug(f"Succesfully imported datasets version {datasets.__version__}")
AttributeError: module 'datasets' has no attribute '__version__'
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

My code works well before, and there is a "datasets" folder in my working directory.  When my transformers version upgraded to 3.3.0, I get this error. If I change the name of the folder "datasets" or downgrade transformers to version 3.2.0, the error is get fixed.
Is this a bug?  Because it doesn't allow me to use "datasets" as a folder name.


Update README.md

Related to pretrained model cards

Add links to models fine-tuned on a downstream task

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Train T5 in Tensoflow 2 Community Notebook

# What does this PR do?
This adds a link to **Training T5 in Tensoflow 2 Community Notebook** under the notebooks/Readme.md community notebook section.

This notebook demonstrates how to train T5 for any task using Tensorflow 2. Trains a Question & Answer task implemented in Tensorflow 2 using SQUAD

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case). **Yes**
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section? **Yes**
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case. [Forum Link](https://discuss.huggingface.co/t/how-to-train-t5-with-tensorflow/641)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation). **Not Applicable**
- [ ] Did you write any new necessary tests?  **Not Applicable**

## Who can review?
@patrickvonplaten @jplu


Problem while using tokenizer.encode_plus for sentence pairs

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
Hi, thanks for this great work.
I was trying to use tokenizer.encode_plus to encode sentence pairs. The code looks like

```py
training_encoded_dict = tokenizer.encode_plus(
            seq0,
            seq1,
            add_spicial_tokens = True,
            max_length = 256,
            truncation_strategy = 'only_second',
            pad_to_max_length = True,
            return_attention_mask = True,
            return_token_type_ids = True,
            return_tensors = 'pt',

        )
```

However, the problem looks like
```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-17-ae3ce93d62ba> in <module>()
     24             return_attention_mask = True,
     25             return_token_type_ids = True,
---> 26             return_tensors = 'pt',
     27
     28         )

2 frames
/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py in truncate_sequences(self, ids, pair_ids, num_tokens_to_remove, truncation_strategy, stride)
   2068             ids = ids[:-num_tokens_to_remove]
   2069         elif truncation_strategy == "only_second":
-> 2070             assert pair_ids is not None and len(pair_ids) > num_tokens_to_remove
   2071             window_len = min(len(pair_ids), stride + num_tokens_to_remove)
   2072             overflowing_tokens = pair_ids[-window_len:]

AssertionError:
```

This issue doesn't occur while using truncation_strategy = 'longest_first', and it happens with other truncation_strategy such as 'only_second' and 'only_first'.
I was wondering does anyone have the same issue or have any idea about how to fix it? Thanks a lot in advance.
**A link to original question on the forum/Stack Overflow**:


[T5] Automatic setting of decoder_input_ids is misleading and does not correspond to the expected behavior of T5

These lines: https://github.com/huggingface/transformers/blob/f62f2ffdcc2df75cf01438bebc7ae281d921d21d/src/transformers/modeling_t5.py#L1020 were added in this PR: https://github.com/huggingface/transformers/pull/5518 .

@mfuntowicz - do we need this hack to make onnx work? I would prefer to revert this change.

The lines do not make much sense IMO and should be deleted. Also the T5 error message when only `input_ids` are passed should be updated to be clearer.

And the docs should be cleaned as well: https://huggingface.co/transformers/model_doc/t5.html#t5model.

Also see:https://github.com/huggingface/transformers/issues/7358


Getting Import error ImportError: cannot import name 'quantize' from 'transformers.convert_graph_to_onnx' (/opt/conda/lib/python3.7/site-packages/transformers/convert_graph_to_onnx.py)

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-19-60bf04c6de64> in <module>
----> 1 from simpletransformers.classification import MultiLabelClassificationModel
      2
      3
      4 model = MultiLabelClassificationModel('roberta', 'roberta-base', num_labels=6, args={'train_batch_size':2, 'gradient_accumulation_steps':16, 'learning_rate': 3e-5, 'num_train_epochs': 3, 'max_seq_length': 512})

/opt/conda/lib/python3.7/site-packages/simpletransformers/classification/__init__.py in <module>
----> 1 from simpletransformers.classification.classification_model import ClassificationModel
      2 from simpletransformers.classification.multi_label_classification_model import MultiLabelClassificationModel
      3 from simpletransformers.classification.multi_modal_classification_model import MultiModalClassificationModel
      4 from simpletransformers.config.model_args import (
      5     ClassificationArgs,

/opt/conda/lib/python3.7/site-packages/simpletransformers/classification/classification_model.py in <module>
     62     get_linear_schedule_with_warmup,
     63 )
---> 64 from transformers.convert_graph_to_onnx import convert, quantize
     65
     66 from simpletransformers.classification.classification_utils import (

ImportError: cannot import name 'quantize' from 'transformers.convert_graph_to_onnx' (/opt/conda/lib/python3.7/site-packages/transformers/convert_graph_to_onnx.py)


[draft] codecov no comment


Reorganize documentation navbar

With the library containing so many models now, the documentation navigation bar was started to get unreadable, especially if someone was looking for a specific model. To make this cleaner I:
- removed the PACKAGE REFENCE caption (not happy about this but there is no way to have a caption with empty content :-( )
- added three captions to separate the documentation between main classes, models and internals
- sorted alphabetically the sections

Also, make the background color of the section headers a bit darker to make the distinction with the rest of the toc clearer.

The result can be checked [here](https://92916-155220641-gh.circle-artifacts.com/0/docs/_build/html/index.html).


Custom TF weights loading

This PR provides a custom weight loading function in order to take into account dynamic model architecture building. More precisely, the brand new loading function takes into account the `authorized_unexpected_keys` and `authorized_missing_keys` class attributes enabling the possibility to ignore some layers in the models.