[Fix] github actions CI by reverting #5138

On github actions, even when the tests fail the job is green, presumably because of my artifacts change( #5318 ).
I am not clear on why this happens, but have verified that this does not happen on CircleCI.
In the screenshot below, there is a green check mark, but also 2 tests have failed:

![image](https://user-images.githubusercontent.com/6045025/87226810-6103b400-c364-11ea-875a-dcbd7c1e49ca.png)


Fix Trainer in DataParallel setting

The new output types seem to break data parallel FYI, see comment on #5671. This is is because of the line
```
return type(out)(map(gather_map, zip(*outputs)))
```
in `scatter_gather` which tries to reconstruct an output of the same type as ours (and fails since it does not provide the necessary arguments). There is no  way to fix our `ModelOutput` to work with this AFAICT.

However, we have the `return_tuple` argument to fix the issue :-)


fix incorrect docstring on bart summarization example

Change summarization example  for BART from
```
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
```
to
```
inputs = tokenizer.encode_plus(ARTICLE_TO_SUMMARIZE, max_length=1024, return_tensors='pt')
```


Add Microsoft's CodeBERT

model card

Related to pretrained model cards

@guoday


What is the decoder_input for encoder-decoder transformer in training time?

https://datascience.stackexchange.com/questions/76261/whats-the-input-dimension-for-transformer-decoder-during-training

Is the link's answer right?

Thank you very much!


[pipelines] Update fill mask pipeline to remove special tokens in the output

Small fix to remove the special tokens from the output of the fill mask pipeline.


How to produce customized attention mask for BertModel?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->

The attention mask for BertModel needs a tensor sized (batch, seq-length). But what if I need to customize the attention for each token, just like UniLM, or some diagonal attention like GPT?


Pipeline model type check

Add model type check for pipelines.

https://github.com/huggingface/transformers/issues/5678


Weird output when using unexpected model type for pipelines

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): CodeBERT

Language I am using the model on (English, Chinese ...): Code

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

This is the right code and right outputs:
```python
from transformers import RobertaConfig, RobertaTokenizer, RobertaForMaskedLM, pipeline

model = RobertaForMaskedLM.from_pretrained('microsoft/codebert-base-mlm')
tokenizer = RobertaTokenizer.from_pretrained('microsoft/codebert-base-mlm')

CODE = "if (x is not None) <mask> (x>1)"
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)

outputs = fill_mask(CODE)
print(outputs)
```
Output:
```python
[{'sequence': '<s>if (x is not None) and(x>1)</s>', 'score': 0.7236990928649902, 'token': 8, 'token_str': 'Ġand'}, {'sequence': '<s>if (x is not None) &(x>1)</s>', 'score': 0.10633797943592072, 'token': 359, 'token_str': 'Ġ&'}, {'sequence': '<s>if (x is not None)and(x>1)</s>', 'score': 0.021604137495160103, 'token': 463, 'token_str': 'and'}, {'sequence': '<s>if (x is not None) AND(x>1)</s>', 'score': 0.02122747339308262, 'token': 4248, 'token_str': 'ĠAND'}, {'sequence': '<s>if (x is not None) if(x>1)</s>', 'score': 0.016991324722766876, 'token': 114, 'token_str': 'Ġif'}]
```
But if we load the model with `RobertaModel` and proceed with the same pipeline:
```python
from transformers import RobertaConfig, RobertaTokenizer, RobertaModel, pipeline

model = RobertaModel.from_pretrained('microsoft/codebert-base-mlm')
tokenizer = RobertaTokenizer.from_pretrained('microsoft/codebert-base-mlm')

CODE = "if (x is not None) <mask> (x>1)"
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)

outputs = fill_mask(CODE)
print(outputs)
```
Then the output makes no sense at all:
```python
[{'sequence': '<s>if (x is not None) real(x>1)</s>', 'score': 0.9961338043212891, 'token': 588, 'token_str': 'Ġreal'}, {'sequence': '<s>if (x is not None)n(x>1)</s>', 'score': 1.70519979292294e-05, 'token': 282, 'token_str': 'n'}, {'sequence': '<s>if (x is not None) security(x>1)</s>', 'score': 1.5919968063826673e-05, 'token': 573, 'token_str': 'Ġsecurity'}, {'sequence': '<s>if (x is not None) Saturday(x>1)</s>', 'score': 1.5472969607799314e-05, 'token': 378, 'token_str': 'ĠSaturday'}, {'sequence': '<s>if (x is not None) here(x>1)</s>', 'score': 1.543204598419834e-05, 'token': 259, 'token_str': 'Ġhere'}]
```

- `transformers` version: 3.0.1
- Platform: Colab
- Python version: Doesn't matter
- PyTorch version (GPU?): Doesn't matter
- Tensorflow version (GPU?): Doesn't matter
- Using GPU in script?: Doesn't matter
- Using distributed or parallel set-up in script?: Doesn't matter


[WIP] Added indexes in grouped entity NER

Based on [issue #5676](https://github.com/huggingface/transformers/issues/5676)

Any application that requires users to locate grouped named entities would require some sort of index. This feature is present in the standard NER pipeline and should also exist in the grouped entity NER pipeline as well. This is a very short addition to the model and is a relevant use case to many developers.


Add indexes to grouped entity NER pipeline

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
**There should be indexes in the output of the grouped entity NER pipeline**

The standard NER pipeline from transformers outputs entities that contain the word, score, entity type, and index. The following snippet demonstrates the normal behavior of the NER pipeline with the default `grouped_entities=False` option.

```python
from transformers import pipeline
nlp_without_grouping = pipeline("ner")
sequence = "Hugging Face Inc. is a company based in New York City."
print(nlp_without_grouping(sequence))

[
    {'word': 'Hu', 'score': 0.9992662668228149, 'entity': 'I-ORG', 'index': 1},
    {'word': '##gging', 'score': 0.9808881878852844, 'entity': 'I-ORG', 'index': 2},
    {'word': 'Face', 'score': 0.9953625202178955, 'entity': 'I-ORG', 'index': 3},
    {'word': 'Inc', 'score': 0.9993382096290588, 'entity': 'I-ORG', 'index': 4},
    {'word': 'New', 'score': 0.9990268349647522, 'entity': 'I-LOC', 'index': 11},
    {'word': 'York', 'score': 0.9988483190536499, 'entity': 'I-LOC', 'index': 12},
    {'word': 'City', 'score': 0.9991773366928101, 'entity': 'I-LOC', 'index': 13}
]

```

However, the NER pipeline with `grouped_entities=True` outputs only word, score, and entity type. Here's the code snippet and output. There's also the problem of 'New York City' being duplicated, but I will address that in a new issue.
```python
from transformers import pipeline
nlp_with_grouping = pipeline("ner", grouped_entities=True)
sequence = "Hugging Face Inc. is a company based in New York City."
print(nlp_with_grouping(sequence))

[
    {'entity_group': 'I-ORG', 'score': 0.9937137961387634, 'word': 'Hugging Face Inc'},
    {'entity_group': 'I-LOC', 'score': 0.9990174969037374, 'word': 'New York City'},
    {'entity_group': 'I-LOC', 'score': 0.9990174969037374, 'word': 'New York City'}
]
```

I believe that the grouped entities returned should also include the tokens of the entities. Sample output would look as such
```python
[
    {'entity_group': 'I-ORG', 'score': 0.9930560886859894, 'word': 'Hugging Face Inc', 'indexes': [1, 2, 3, 4]},
    {'entity_group': 'I-LOC', 'score': 0.998809814453125, 'word': 'New York City', 'indexes': [11, 12, 13]},
    {'entity_group': 'I-LOC', 'score': 0.998809814453125, 'word': 'New York City', 'indexes': [11, 12, 13]}
]
```

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

**Any application that requires users to locate grouped named entities would require some sort of index.** This feature is present in the standard NER pipeline and should also exist in the grouped entity NER pipeline as well.

In my case, I am trying to append the type to the text right after the named entity ("Apple" would become "Apple \<I-ORG\>") so I need to be able to locate the named entity within my phrase.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I have been able to fix this by adding two lines to `group_sub_entities` function https://github.com/huggingface/transformers/blob/7fad617dc1fc681a7f5da5e0172c8b83f4bf0024/src/transformers/pipelines.py#L1042
```python
    def group_sub_entities(self, entities: List[dict]) -> dict:
        """
        Returns grouped sub entities
        """
        # Get the first entity in the entity group
        entity = entities[0]["entity"]
        scores = np.mean([entity["score"] for entity in entities])
        tokens = [entity["word"] for entity in entities]
        indexes = [entity["index"] for entity in entities]    # my added line

        entity_group = {
            "entity_group": entity,
            "score": np.mean(scores),
            "word": self.tokenizer.convert_tokens_to_string(tokens),
            "indexes": indexes    # my added line
        }
        return entity_group
```


Deepset model not loading using default code

# 🐛 Bug

## Information

Model I am using (Bert): Bert

Language I am using the model on (English): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run this script below

```
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
tokenizer = AutoTokenizer.from_pretrained("deepset/bert-large-uncased-whole-word-masking-squad2")
model = AutoModelForQuestionAnswering.from_pretrained("deepset/bert-large-uncased-whole-word-masking-squad2")
```

2. Notice this error stack:
```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-29-2a5e47891fb0> in <module>
----> 1 tokenizer = AutoTokenizer.from_pretrained("deepset/bert-large-uncased-whole-word-masking-squad2")
      2 model = AutoModelForQuestionAnswering.from_pretrained("deepset/bert-large-uncased-whole-word-masking-squad2")
      3
      4 reg_tokenizer = RegexpTokenizer(r'\w+')

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    107             return RobertaTokenizer.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    108         elif 'bert' in pretrained_model_name_or_path:
--> 109             return BertTokenizer.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    110         elif 'openai-gpt' in pretrained_model_name_or_path:
    111             return OpenAIGPTTokenizer.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    280
    281         """
--> 282         return cls._from_pretrained(*inputs, **kwargs)
    283
    284

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
    344                         pretrained_model_name_or_path, ', '.join(s3_models),
    345                         pretrained_model_name_or_path,
--> 346                         list(cls.vocab_files_names.values())))
    347
    348         # Get files from url, cache, or disk depending on the case

OSError: Model name 'deepset/bert-large-uncased-whole-word-masking-squad2' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed 'deepset/bert-large-uncased-whole-word-masking-squad2' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
```

## Expected behavior

I switched to a conda environment and reinstalled transformers package with conda. Before, having used just pip install, this segment of code was working. Now, it no longer even finds the relevant model. The expected behavior is to load this specific model.

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-3.10.0-1062.12.1.el7.x86_64-x86_64-with-redhat-7.8-Maipo
- Python version: 3.6.8
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: n/a
- Using distributed or parallel set-up in script?: no


Can't get BART to generate EOS token.

# 🐛 Bug
I finetuned Bart on a few seq2seq tasks. It seems to learn the right thing, but it never seems to stop generating text unless I set `max_length`, i.e. it never generates the EOS token on its own. This seems to be the case for the pretrained model as well: if I run the example [here](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration), it always produces very long summaries even if the input text is quite small.

To make the bug easy to reproduce, I set up a toy task below where I am finetuning Bart to just repeat the input.

## Information

Model I am using (Bert, XLNet ...): Bart

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a dataset for seq2seq training where the target is a copy of the source, e.g. below I create a dataset of sentences with `4,000` examples in the training set.
```python
import nlp
d = nlp.load_dataset('snli')
d = list(set(d['train']['premise'][:20000]))
import os
folder = '/tmp/bartz2'
if os.path.exists(folder):
    os.system('rm -rf %s' % folder)
os.mkdir(folder)
N = 4000
f = open(os.path.join(folder, 'train.source'), 'w')
f.write('\n'.join(d[:N]))
f = open(os.path.join(folder, 'val.source'), 'w')
f.write('\n'.join(d[N:]))
f.close()
f = open(os.path.join(folder, 'test.source'), 'w')
f.write('\n'.join(d[N:]))
f.close()
f = open(os.path.join(folder, 'train.target'), 'w')
f.write('\n'.join(d[:N]))
f.close()
f = open(os.path.join(folder, 'val.target'), 'w')
f.write('\n'.join(d[N:]))
f.close()
f = open(os.path.join(folder, 'test.target'), 'w')
f.write('\n'.join(d[N:]))
f.close()
```
2. Finetune bart on this dataset, i.e.
```sh
./finetune.sh --data_dir /tmp/bartz2/ --train_batch_size=4 --eval_batch_size=4 --output_dir=copybart --num_train_epochs 3 --model_name_or_path facebook/bart-large --n_val=1000 --n_test=1000 --task=translation
```
Caveat: I commented out `--fp16` in `finetune.sh`

3. Generate text with the finetuned model
```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model = AutoModelForSeq2SeqLM.from_pretrained(f'/home/marcotcr/work/transformers/examples/seq2seq/copybart//best_tfmr')
tokenizer = AutoTokenizer.from_pretrained(f'/home/marcotcr/work/transformers/examples/seq2seq/copybart/best_tfmr')
model.to('cuda');
for text in d[N+1:N+10]:
    print(text)
    inputs = tokenizer([text], max_length=1024, return_tensors='pt', truncation=True).to('cuda')
    a = model.generate(**inputs,  early_stopping=True, num_beams=4, max_length=100, early_stoppy=True)
    dec = tokenizer.batch_decode(a, skip_special_tokens=True, clean_up_tokenization_spaces=False)
    print(dec[0])
    print()
```

Output (first few lines):
```
A black man walks away with a large basket full of items on his head.
A black man walks away with a large basket full of items on his head of a woman-on-shoreline-a-chosie-centre-of items in a basket full-items on their head.A fellow-side of the basket.-

A person is doing a bicycle trick over the rocks while a bystander gives a thumbs up.
A person is doing a bicycle trick over the rocks while a bystander gives a thumbs up.A couple of people are afoot in a bicycle trunk over the Rocks while a passerer gives an thumbs up to the bystander is a thumb up.T-up.

Female with long black hair and wearing eyeglasses, a blue shirt, sitting next to a male in a striped shirt, at a table, cutting into a chocolate cake with a single red candle in it.
Female with long black hair and wearing eyeglasses, a blue shirt, sitting next to a male in a striped shirt, at a table, cutting into a chocolate cake with a single red candle in it.The cake is a blue shirts, a purple-colored tissue-
```

## Expected behavior
I would expect it to emit the EOS token after copying the sentence. It seems to learn the right behavior up to that point (it copies the sentence), and I saw this in my real seq2seq tasks as well (i.e. it did the right thing but failed to stop). This behavior is also present in `test_generations.txt`.

I also tried training on a dataset where the target sentences ended in `</s>`, but that didn't change it.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 - `transformers` version: 3.0.2 (pulled from master this morning)
- Platform: Linux-5.3.0-61-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Document model outputs

This PR adds proper documentation for all model outputs introduced in #5226. There are a few fixes to some docstrings for proper sphinx formatting, and a tiny change in the function that generates docstring to reference to `ModelOutput` types by their full names (since they are not in the init).


Added first description of the model

Related to pretrained model cards

Added general description, information about the tags and also some example usage code.


Deprecate old past arguments

As discussed internally, previous arguments `past`, `decoder_cached_states` and `decoder_past_key_value_states` are deprecated and replaced by either `past_key_values` or `decoder_past_key_values`.
This also fixes the mentions to those arguments in the input docstrings, the output docstrings already refer to the correct arg (this was done in #5226 ).

In passing, replace `DeprecationWarning` in the other deprecated args by `FutureWarning`, since it's the right way to do it.


[WIP] add DeFormer (ACL 2020) example

Hi there, I'm one of the authors of the [DeFormer paper](https://www.aclweb.org/anthology/2020.acl-main.411/), and I'd like to adapt the [DeFormer codebase](https://github.com/StonyBrookNLP/deformer) to this awesome transformers library. To get the adaptation done, I put a few high-level todos in the README (also here). I plan to get a working example by/before August but don't have a precise timeline yet. Let me hear what you think. Thanks.

- [ ] use HF preprocessing (use HF nlp library)
- [ ] convert original TF DeFormer to HF version
- [ ] convert pre-trained checkpoints
- [ ] compare and test accuracy for SQuAD, RACE, and BoolQ
- [ ] prepare demo and upload to model cards


[squad] add version tag to squad cache

This diff is to add a version number to the SQuAD cache file so that cached SQuADv1.1 features are not mistakenly read when you request SQuADv2. Addresses #5668


SquadDataset should use version number in cache file name

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): N/A

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts:
* [x] my own modified scripts: simple example code given below

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

1. Load `SquadDataset` with `args.version_2_with_negative = False`. You will see the progress bars for creating cached features.

2. Load `SquadDataset` with `args.version_2_with_negative = True`. Rather than seeing it create a new cache for the v2 dataset, you will see it automatically use the already made cache file for v1.

Example code:

```
from transformers import AutoTokenizer
from transformers import SquadDataset
from transformers import SquadDataTrainingArguments

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
args = SquadDataTrainingArguments()

# FIXME: Change this path to your local SQuAD dataset path
args.data_dir = os.path.expanduser("~/.torch/nlp/SQuAD")

args.version_2_with_negative = False
squadv1 = SquadDataset(args, tokenizer)

args.version_2_with_negative = True
squadv2 = SquadDataset(args, tokenizer)
```
## Expected behavior

Separate cache files should be created for the v1.1 and v2 versions of SQuAD

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.0.0-1035-azure-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: no


pytorch_model.bin file is different after uploading to HuggingFace Models

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `distilroberta-base`

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behaviour:

1. Save my model with `.save_pretrained()`
2. Download the models `pytorch_model.bin`
3. Check the `diff` between `pytorch_model.bin` _before_ uploading and _after_ downloading, it is not the same.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

I first noticed that a model I have trained produced different outputs when I loaded it from a local directory compared to uploading it to https://huggingface.co/models and downloading it. Unfortunately, the error is a little hard to reproduce as you need access to my saved model. I have uploaded it [here](https://drive.google.com/file/d/1C8okSoS4tJHtZllQ8qIJbmXRLb8ITL6N/view?usp=sharing). With that model downloaded, I compare its outputs before/after uploading:

_Before uploading_ (e.g. loading the model from a local directory)

```python
import torch
from scipy.spatial.distance import cosine

from transformers import AutoModel, AutoTokenizer

# Load the model
tokenizer = AutoTokenizer.from_pretrained("declutr-small")
model = AutoModel.from_pretrained("declutr-small")

# Prepare some text to embed
text = [
    "A smiling costumed woman is holding an umbrella.",
    "A happy woman in a fairy costume holds an umbrella.",
]
inputs = tokenizer(text, padding=True, truncation=True, return_tensors="pt")

# Embed the text
with torch.no_grad():
    sequence_output, _ = model(**inputs, output_hidden_states=False)

# Mean pool the token-level embeddings to get sentence-level embeddings
embeddings = torch.sum(
    sequence_output * inputs["attention_mask"].unsqueeze(-1), dim=1
) / torch.clamp(torch.sum(inputs["attention_mask"], dim=1, keepdims=True), min=1e-9)

# Compute a semantic similarity via the cosine distance
semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
print(semantic_sim)  # => ~0.83
```

_After uploading_ (e.g. loading the model from https://huggingface.co/models)

```python
# Load the model
tokenizer = AutoTokenizer.from_pretrained("johngiorgi/declutr-small", force_download=True)
model = AutoModel.from_pretrained("johngiorgi/declutr-small", force_download=True)

# Prepare some text to embed
text = [
    "A smiling costumed woman is holding an umbrella.",
    "A happy woman in a fairy costume holds an umbrella.",
]
inputs = tokenizer(text, padding=True, truncation=True, return_tensors="pt")

# Embed the text
with torch.no_grad():
    sequence_output, _ = model(**inputs, output_hidden_states=False)

# Mean pool the token-level embeddings to get sentence-level embeddings
embeddings = torch.sum(
    sequence_output * inputs["attention_mask"].unsqueeze(-1), dim=1
) / torch.clamp(torch.sum(inputs["attention_mask"], dim=1, keepdims=True), min=1e-9)

# Compute a semantic similarity via the cosine distance
semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
print(semantic_sim)  # => ~0.99, NOT the same as the local model!
```

The embeddings must be different, as their semantic similarity is. After some more digging, I realized that the `pytorch_model.bin` of the local model and the uploaded then downloaded model are not the same, which I checked with `diff`.

I tried everything I could think of, deleting my `transformers` cache folder, deleting the model from https://huggingface.co/models and re-uploading. I also tried uploading it from both macOS/Linux. The error persists.

Does anyone have any clue how this could happen? It's such a frustrating error b/c it essentially passes silently until you look at your models outputs.

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

I expect that the outputs of my model to be identical when I load it from a local directory, and when I upload it and then download it from https://huggingface.co/models.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.


How do you connect Convolutional layers to Transformers?

# ❓ Questions & Help
Hi everyone!
Where can I find code example that makes clear how to connect Convolutional layers to Transformers and how it needs to be shaped in order to make such a connection?
I have a bit of a hard time figuring it out.

Thank you for your support.


[AutoModels] Fix config params handling of all PT and TF AutoModels

As shown in #5474, currently, a command like:

```python
from transformers import AutoModelForCausalLM]
model = AutoModelForCausalLM.from_pretrained('bert-base-uncased', is_decoder=True)
```

fails because `is_decoder` is carried on as a model init argument even though it should *only* be used as a config init argument.

This PR fixes one `AutoModelFor....` for this, but this still has be applied for other `AutoModelFor...` classes.

Pinging @LysandreJik @sgugger @thomwolf - are you guys ok with this change (bug fix) in general?


[PyTorch] Load and run a model CPU which was traced and saved on GPU

# ❓ Questions & Help

## Details
I am trying to trace/save openai_gpt on GPU and use that model on a CPU and facing issues. Is this possible to do? I have attached the link to the question posted on the discuss forum as well.

### Sample Script
```python
from transformers import OpenAIGPTTokenizer, OpenAIGPTModel
import torch

tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTModel.from_pretrained('openai-gpt')

inputs = torch.tensor([tokenizer.encode("Hello, my dog is cute")])
outputs = model(inputs)
print(outputs)

print("To CUDA:")
inputs = inputs.to("cuda")
model = model.to("cuda")
traced_model = torch.jit.trace(model, (inputs,))
torch.jit.save(traced_model, "openai_gpt_cuda.pt")
print(traced_model.graph)
print("\n")
print("Load model onto CPU")
loaded = torch.jit.load("openai_gpt_cuda.pt", map_location=torch.device("cpu"))
inputs = inputs.to("cpu")
print("\n")
print(loaded.graph)
outputs = loaded(inputs)
print(outputs)
```

### Error seen
```
Traceback (most recent call last):
  File "gpt.py", line 23, in <module>
    outputs = loaded(inputs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select
The above operation failed in interpreter.
Traceback (most recent call last):
Serialized   File "code/__torch__/torch/nn/modules/module/___torch_mangle_147.py", line 35
    position_ids = torch.arange(_20, dtype=4, layout=0, device=torch.device("cuda:0"), pin_memory=False)
    input0 = torch.view(torch.unsqueeze(position_ids, 0), [-1, _19])
    _21 = torch.add((_14).forward(input, ), (_13).forward(input0, ), alpha=1)
                                             ~~~~~~~~~~~~ <--- HERE
    input1 = torch.add(_21, CONSTANTS.c0, alpha=1)
    _22 = (_12).forward(input1, )
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/functional.py(1484): embedding
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/sparse.py(114): forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(516): _slow_forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(530): __call__
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_openai.py(433): forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(516): _slow_forward
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(530): __call__
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(1034): trace_module
/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(882): trace
gpt.py(14): <module>
Serialized   File "code/__torch__/torch/nn/modules/module/___torch_mangle_0.py", line 8, in forward
  def forward(self: __torch__.torch.nn.modules.module.___torch_mangle_0.Module,
    input: Tensor) -> Tensor:
    position_embeds = torch.embedding(self.weight, input, -1, False, False)
                      ~~~~~~~~~~~~~~~ <--- HERE
    return position_embeds

The above operation failed in interpreter.
Traceback (most recent call last):
```
https://discuss.huggingface.co/t/pytorch-trace-on-cpu-and-use-on-gpu/181/3


Request for Support to Adapt a Model (Human Dignity Observatory: Non-Profit Project)

Dear @huggingface community from the Knowmad Institut we need your contribution and support for the Observatory of Human Dignity that we have developed.

You can access to the observatory here: (https://bit.ly/MQHHRRES)

We need your support to adapt the hugging face tools. We need to get the sentiments out of the content of the tweets and also (based on the data we have filtered out one by one) filter out the ones that are really human rights violations, like classifying from 1 to 5 with 1 being the one that looks the least like a human rights violation and 5 being the one that most matches the data we already have.

We thank the community and the @huggingface team for their support in disseminating this request for support.

|[![FB18.png](https://knowmadinstitut.org/wp-content/uploads/2020/04/Black-and-Red-Geometric-Technology-Keynote-Presentation-1.png)](https://bit.ly/MQHHRRES) |

Dear @huggingface community from the Knowmad Institut we need your contribution and support for the Observatory of Human Dignity that we have developed.

You can access to the observatory here: https://t.co/3TgKwjs3PJ

1/3 pic.twitter.com/ly4Z139qHb
— Knowmad Institut (@KnowmadInstitut) July 8, 2020


[WIP - don't merge][TF generate] Make tf generate compatible with tf.function

The tf generate function shoud be cleaned up so that it can be used with tf.function.


Create Model card for RoBERTa-hindi-guj-san

Related to pretrained model cards


"How to train a new language model from scratch" colab stuck at training

Hello,
I am following the tutorial : https://huggingface.co/blog/how-to-train
At command :
trainer.train()
It gets stuck (Nothing displayed except "Using deprecated `--per_gpu_train_batch_size` argument").
Any idea ?


[Longformer] fix longformer global attention output

This PR fixes the attention probs that are outputted when longformer uses global attention and sets `output_attention=True`.

Thanks a million to @k141303 for very clean issue + perfect proposed solution in https://github.com/huggingface/transformers/issues/5646 .


Create README.md - Model card

Related to pretrained model cards

Model card for sentence-transformers/bert-base-nli-max-tokens


Create README.md - Model card

Related to pretrained model cards

Model card for sentence-transformers/bert-base-nli-cls-token


Truncated Outputs by t5 fine-tuned models

I fine-tuned t5-small over CNN/DM dataset using the finetune_t5.sh script. The outputs produced by the saved fine-tuned model is okayish but it's getting cut i.e., producing incomplete sentence at the end.

Example :

Artcile:

(CNN)The only thing crazier than a guy in snowbound Massachusetts boxing up the powdery white stuff and offering it for sale online? People are actually buying it. For $89, self-styled entrepreneur Kyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box -- enough for 10 to 15 snowballs, he says.Kyle Waring died last week. But not if you live in New England or surrounding states. "We will not ship snow to any states in the northeast!" says Waring's website, ShipSnowYo.com. "We're in the business of expunging snow!" His website and social media accounts claim to have filled more than 133 orders for snow -- more than 30 on Tuesday alone, his busiest day yet. With more than 45 total inches, Boston has set a record this winter for the snowiest month in its history. Most residents see the huge piles of snow choking their yards and sidewalks as a nuisance, but Waring saw an opportunity. According to Boston.com, it all started a few weeks ago, when Waring and his wife were shoveling deep snow from their yard in Manchester-by-the-Sea, a coastal suburb north of Boston. He joked about shipping the stuff to friends and family in warmer states, and an idea was born. His business slogan: "Our nightmare is your dream!" At first, ShipSnowYo sold snow packed into empty 16.9-ounce water bottles for $19.99, but the snow usually melted before it reached its destination. So this week, Waring began shipping larger amounts in the Styrofoam cubes, which he promises will arrive anywhere in the U.S. in less than 20 hours. He also has begun selling a 10-pound box of snow for $119. Many of his customers appear to be companies in warm-weather states who are buying the snow as a gag, he said. Whether Waring can sustain his gimmicky venture into the spring remains to be seen. But he has no shortage of product. "At this rate, it's going to be July until the snow melts," he told Boston.com. "But I've thought about taking this idea and running with it for other seasonal items. Maybe I'll ship some fall foliage."

Summary produced by t5-small fine-tuned over CNN/DM :

Kyle Waring will ship you 6 pounds of snow in an insulated Styrofoam box for $89 . The self-styled entrepreneur says he will not ship snow to any states in the northeast . Waring's website and social media accounts claim to have filled more than 133 orders for snow . "We're in the business of expunging snow!" Waring says . He has begun selling a 10-pound box of snow for $119 . His business slogan: "Our nightmare is your

At first I thought this might be because the model hasn't converged as I just ran for 1 epoch but it's producing similar truncated outputs even for t5-small fine-tuned over cnn/dm for 5 epochs.Also this problem is not related to min_length or max_length parameters I think, as it produced similar outputs for all combinations of those two parameters. Tried changing --max_source_length, --max_target_length, --val_max_target_length, --test_max_target_length(these 4 parameters are present in finetune.py) parameter's values too from their default values before fine-tuning but no use.

What might be the reason for this truncation? Is this a problem of the fine-tuning code used to fine-tune pretrained models as pre-trained models don't produce this kind of outputs.


Create model card

Related to pretrained model cards

Create model card for T5-small fine-tuned on SQUAD v2


❓ Difficulties to reproduce BART results on CNN/DM by fine-tuning bart-large

# ❓ Help

I'm trying to fine-tune BART on CNN/DM by myself (so, starting from `facebook/bart-large` checkpoint).

However I can't reproduce the results so far... BART authors report a R1 score of `44.16` in their paper, but my best checkpoint so far is only `42.53`.

It's not an issue with the eval script, as I can reproduce the authors' results from the checkpoint `facebook/bart-large-cnn`. I get a score of `44.09` using this checkpoint.

I tried several hyper-parameters : the ones provided in the example folder, but also the ones used in fairseq repo. It doesn't change anything...

---

I'm a bit at loss on how to reproduce these fine-tuning score...
Could anyone fine-tune BART successfully using `transformers` repo ? If yes, can you share your parameters ?
Any help would be greatly appreciated !

@sshleifer


# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
```
> from transformers import AutoTokenizer, AutoModelWithLMHead
> tokenizer = AutoTokenizer.from_pretrained("hfl/chinese-roberta-wwm-ext")

I0710 17:52:53.548153 139925919450880 tokenization_utils_base.py:1167] Model name 'hfl/chinese-roberta-wwm-ext' not found in model shortcut name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). Assuming 'hfl/chinese-roberta-wwm-ext' is a path, a model identifier, or url to a directory containing tokenizer files.
I0710 17:52:59.942922 139925919450880 tokenization_utils_base.py:1254] loading file https://s3.amazonaws.com/models.huggingface.co/bert/hfl/chinese-roberta-wwm-ext/vocab.json from cache at None
I0710 17:52:59.943219 139925919450880 tokenization_utils_base.py:1254] loading file https://s3.amazonaws.com/models.huggingface.co/bert/hfl/chinese-roberta-wwm-ext/merges.txt from cache at None
I0710 17:52:59.943420 139925919450880 tokenization_utils_base.py:1254] loading file https://s3.amazonaws.com/models.huggingface.co/bert/hfl/chinese-roberta-wwm-ext/added_tokens.json from cache at /home/ubuntu/.cache/torch/transformers/23740a16768d945f44a24590dc8f5e572773b1b2868c5e58f7ff4fae2a721c49.3889713104075cfee9e96090bcdd0dc753733b3db9da20d1dd8b2cd1030536a2
I0710 17:52:59.943602 139925919450880 tokenization_utils_base.py:1254] loading file https://s3.amazonaws.com/models.huggingface.co/bert/hfl/chinese-roberta-wwm-ext/special_tokens_map.json from cache at /home/ubuntu/.cache/torch/transformers/6f13f9fe28f96dd7be36b84708332115ef90b3b310918502c13a8f719a225de2.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4
I0710 17:52:59.943761 139925919450880 tokenization_utils_base.py:1254] loading file https://s3.amazonaws.com/models.huggingface.co/bert/hfl/chinese-roberta-wwm-ext/tokenizer_config.json from cache at /home/ubuntu/.cache/torch/transformers/5bb5761fdb6c8f42bf7705c27c48cffd8b40afa8278fa035bc81bf288f108af9.1ade4e0ac224a06d83f2cb9821a6656b6b59974d6552e8c728f2657e4ba445d9
I0710 17:52:59.943786 139925919450880 tokenization_utils_base.py:1254] loading file https://s3.amazonaws.com/models.huggingface.co/bert/hfl/chinese-roberta-wwm-ext/tokenizer.json from cache at None

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 217, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1140, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1288, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/transformers/tokenization_roberta.py", line 171, in __init__
    **kwargs,
  File "/home/ubuntu/anaconda3/envs/deeplearning/lib/python3.6/site-packages/transformers/tokenization_gpt2.py", line 167, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

Does it support `hfl/chinese-roberta-wwm-ext` now? Or what should i do. Hope for help, thx!
@julien-c

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Create README.md - Model card for sentence-transformers/bert-base-nli-mean-tokens

Related to pretrained model cards

Model card for https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens


T5 fp16 overflow in forward (T5DenseReluDense)

# 🐛 Bug
Using `AutoModelWithLMHead.from_pretrained("t5-base")` for fine-tuning, after 34 iterations I get nan loss from the forward method.
After debugging it, I found that the source of the nan is due to an overflow that happens in `T5DenseReluDense`, when running `h = self.wo(h)`.  The result of this forward is a tensor that has `inf` in one of its values, which later on causes the nan loss.
I looked into this calculation with fp32 and I saw that his `inf` is caused due to a value of 66246.3906, which is over the maximum value of 65504 in fp16.

This issue only happens with fp16 (opt_level="O1"), with opt_level="O0" everything is fine.

## Information

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
I don't have step by step instructions, because I will need to upload my entire dataset for that.
I have a pickle for the vector `h` and the weights of `self.wo` that causes the overflow in `T5DenseReluDense`, I can upload it if it might help.

## Expected behavior

get a numeric loss

## Environment info

- `transformers` version:  3.0.2
- Platform: Linux-5.3.0-1030-aws-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Wrong answers from Longformer model even on simple questions

I am using the pretrained model `allenai/longformer-large-4096-finetuned-triviaqa`, however upon inspecting it on my system and the demo on the huggingface website, the outputs seem off even for very simple examples and samples from the dataset.

1. [Example 1](https://www.dropbox.com/s/9h3dcqpwq0n1b05/download%20%283%29.png?dl=0)
2. [Example 2](https://www.dropbox.com/s/40e93m2odix8x1p/download%20%284%29.png?dl=0)
3. [Example 3](https://www.dropbox.com/s/s5t1k6jluyzfs33/download%20%286%29.png?dl=0)
4. [Example 4](https://www.dropbox.com/s/oyps5a5gr2e4c25/download%20%287%29.png?dl=0)

Other models for QA (like `bert-large-uncased-whole-word-masking`) get such simple examples right.


Bugs due to design choices in LongformerTokenizer

I am using the transformers library from source (version: 3.0.2).

```python
import transformers
from transformers import *
longformer_tokenizer = LongformerTokenizer.from_pretrained("allenai/longformer-base-4096")
longformer_tokenizer.tokenize("This is a sample sentence for the tokenizer.")
```
The output I get is
``` ['This', 'Ġis', 'Ġa', 'Ġsample', 'Ġsentence', 'Ġfor', 'Ġthe', 'Ġtoken', 'izer']```

The design choice here is to use the `Ġ` as a start of every new word (except for the first word). This is in contrast with other tokenizers which insert `##` tokens for suffixes of broken words.

Due to this slightly different tokenization quirk, many things could break, one of which is the following piece of code in `squad.py`:

https://github.com/huggingface/transformers/blob/02a0b43014ac333a169e99d76aaba023a316e384/src/transformers/data/processors/squad.py#L106-L112

The `doc_tokens` from the processor are whitespace separated tokens, which are to be further tokenized using this code. But since each word is treated individually, and `LongformerTokenizer` doesn't insert `Ġ` for the first token, there is a problem. The resulting `all_doc_tokens` can not be correctly converted to original string using `tokenizer.convert_tokens_to_string` because it is missing the `Ġ`  at the start.


Classification accuracy on validation set didn't improve while fine-tuning BERT

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

I was fine-tuning BERT with BertForSequenceClassification on my own dataset. 0.8 of the whole dataset was used as training set and the others as validation set. The training loss decreased during the process. However, the accuracy on validation set was always around 0.5, which is similar to random guessing. And the accuracy on validation didn't improve a lot after fine-tuning. For example, from epochs 1-3, accuracy was from 0.48-0.52. So I was wondering whether this problem was caused by my dataset itself or if I did something wrong while fine-tuning it? Does anybody have any ideas on this? By the way, before this, I was fine-tuning BERT on another dataset and it did improve the classification accuracy a lot .
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


T5 TorchScript (Trace) Conversion

# ❓ Questions & Help

How can we correctly set inputs for t5 TorchScript?

## Details
<!-- Description of your issue -->

```python
from transformers import T5Model
import torch

tokens_tensor = torch.ones(1, 10, dtype=torch.long)
model = T5Model.from_pretrained("t5-small", torchscript=True)
model.eval()
scripted_model = torch.jit.trace(model, (tokens_tensor, tokens_tensor))
```

Error:
```
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```


Can't get (global) attention probs using Longformer

# 🐛 Bug

## Information

Model I am using **Longformer**:

Language I am using the model on Japanese:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Set config.output_attentions=True
2. Use global attention (sum(global_attention_mask)>0)

The following is the minimum code to reproduce the error.

~~~python3:test.py
import torch
from transformers import AutoModel, AutoTokenizer, AutoConfig

if __name__ == '__main__':
    config = AutoConfig.from_pretrained("allenai/longformer-base-4096", output_attentions=True)
    model = AutoModel.from_pretrained("allenai/longformer-base-4096", config=config)
    tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
    token_ids = [[
        tokenizer.cls_token_id, 10, 11, 12,
        tokenizer.sep_token_id, 21, 22, 23,
        tokenizer.sep_token_id
    ]]
    global_attention_mask = [[1,1,1,1,1,0,0,0,0]]
    logit, *_, attention_probs = model(
        torch.LongTensor(token_ids),
        global_attention_mask=torch.LongTensor(global_attention_mask)
    )
    print(attention_probs[0].size())
~~~

~~~bash
$ python3 test.py
Traceback (most recent call last):
  File "test_longformer.py", line 16, in <module>
    global_attention_mask=torch.LongTensor(global_attention_mask)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/transformers/modeling_longformer.py", line 1004, in forward
    output_hidden_states=output_hidden_states,
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/transformers/modeling_longformer.py", line 695, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, output_attentions,)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/transformers/modeling_longformer.py", line 658, in forward
    self_attn_outputs = self.attention(hidden_states, attention_mask, output_attentions=output_attentions,)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/transformers/modeling_longformer.py", line 642, in forward
    self_outputs = self.self(hidden_states, attention_mask, output_attentions,)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/uge_mnt/home/username/.local/lib/python3.6/site-packages/transformers/modeling_longformer.py", line 435, in forward
    attn_probs = attn_probs.view(batch_size, self.num_heads, max_num_global_attn_indices, seq_len)
RuntimeError: shape '[1, 12, 5, 512]' is invalid for input of size 3182592
~~~

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

The model can output attention probs for each attention head.

~~~bash
$ python3 test.py
torch.Size([1, 12, 4096, 5])
~~~

It would seem to work if I rewrite the target line as follows.

https://github.com/huggingface/transformers/blob/02a0b43014ac333a169e99d76aaba023a316e384/src/transformers/modeling_longformer.py#L435

~~~python3
#attn_probs = attn_probs.view(batch_size, self.num_heads, max_num_global_attn_indices, seq_len)
attn_probs = attn_probs[:,:,:,:max_num_global_attn_indices]
attn_probs = attn_probs.permute(0, 2, 1, 3)
~~~

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:3.0.2
- Platform:Ubuntu 18.04.4 LTS
- Python version:Python 3.6.9 :: Anaconda, Inc.
- PyTorch version (GPU?):1.5.1 (Yes)
- Tensorflow version (GPU?):
- Using GPU in script?:Yes
- Using distributed or parallel set-up in script?:Yes


enable easy checkout switch

allow having multiple repository checkouts and not needing to remember to rerun `pip install -e .[dev]` when switching between checkouts and running tests. This code will automatically do the right thing for the test suite.

Note that `python -m pytest` automatically adds `.` to the path, so normally most packages get automatically tested against the local checkout. However, since this project is under a sub-dir `src/` this feature doesn't help.


FlaubertForTokenClassification

implement FlaubertForTokenClassification as a subclass of XLMForTokenClassification.

Based on an item from https://github.com/huggingface/transformers/projects/17


Help with Using TFXLNet on custom embeddings

# 🐛 Bug

## Information

Hi, I am working on implementing the multimodal bitransformer: https://arxiv.org/pdf/1909.02950.pdf
I have already gotten this working using your implementation of TFBertModel, but I want to try using TFXLNetModel in place of BERT to see if it makes an improvement. What I've done for using TFBertModel is extract the word_embeddings and pass the word embeddings with token_type_ids = 0 and image embeddings with token_type_ids = 1.

## To reproduce

I have the following definition of my model

`
class BERT(transformers.TFXLNetModel):
    def __init__(self, config, *inputs, **kwargs):
        super(BERT, self).__init__(config, *inputs, **kwargs)
        self.call = tf.function(self.call)

class MyModel(tf.keras.Model):

    def __init__(self):
        super(MyModel, self).__init__()
        self.resnet = tf.keras.applications.ResNet152V2(include_top=False,
                                                             weights='imagenet',
                                                             input_shape=(224, 224, 3))
        self.bert = BERT.from_pretrained('xlnet-base-cased')

        self.text_embedding = self.bert.get_input_embeddings().weights[0]

        self.pooling = layers.AveragePooling2D(pool_size=(2, 2), padding='same')
        self.reshape = layers.Reshape((4 * 4, 2048)) # 4 is from 7//2 + 1
        self.W_ns = [layers.Dense(self.bert.config.hidden_size) for _ in range(self.reshape.target_shape[0])]

        self.concat = layers.Concatenate(axis=1)

        self.dropout = layers.Dropout(0.1)
        self.denseout = layers.Dense(1, activation='sigmoid')

    def call(self, inputs):
        text, image = inputs

        # handle image
        image = tf.keras.applications.resnet_v2.preprocess_input(image)
        image_emb = self.resnet(image)
        image_emb = self.pooling(image_emb)
        image_emb = self.reshape(image_emb)
        image_embeds = [self.W_ns[i](image_emb[:, i]) for i in range(self.reshape.target_shape[0])]
        image_emb = tf.keras.backend.stack(image_embeds, axis=1)

        # handle text
        text_emb = tf.gather(self.text_embedding, text)

        # concat and feed to bert
        concat_emb = self.concat([text_emb, image_emb])
        seg_ids = np.concatenate((np.zeros(max_len, dtype=np.int64),
                                  np.ones(self.reshape.target_shape[0], dtype=np.int64)))
        print('input shapes to xlnet', concat_emb.shape, seg_ids.shape)
        bert_encodings = self.bert(inputs={'inputs_embeds': concat_emb,
                                            'token_type_ids': seg_ids})[0]
        doc_encoding = tf.squeeze(bert_encodings[:, 0:1, :], axis=1)
        doc_encoding = self.dropout(doc_encoding)

        output = self.denseout(doc_encoding)

        return output
`

In the line that prints "input shapes to xlnet", I get (None, 116, 768) for the inputs_embeds and (116,) for the token_type_ids, which I expect because I have 100 word embeddings and 16 image embeddings. When I call fit() on this model, it gives the error:

> ValueError: in converted code:
>
>     <ipython-input-12-b6bccd3c83e0>:50 call  *
>         bert_encodings = self.bert(inputs={'inputs_embeds': concat_emb,
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py:891 __call__
>         outputs = self.call(cast_inputs, *args, **kwargs)
>     /homes/awl27/python36env/lib/python3.6/site-packages/transformers/modeling_tf_xlnet.py:824 call  *
>         outputs = self.transformer(inputs, **kwargs)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py:842 __call__
>         outputs = call_fn(cast_inputs, *args, **kwargs)
>     /homes/awl27/python36env/lib/python3.6/site-packages/transformers/modeling_tf_xlnet.py:530 call  *
>         token_type_ids = tf.transpose(token_type_ids, perm=(1, 0)) if token_type_ids is not None else None
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py:1780 transpose_v2
>         return transpose(a=a, perm=perm, name=name, conjugate=conjugate)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py:1870 transpose
>         ret = transpose_fn(a, perm, name=name)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py:11455 transpose
>         "Transpose", x=x, perm=perm, name=name)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py:793 _apply_op_helper
>         op_def=op_def)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py:548 create_op
>         compute_device)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:3429 _create_op_internal
>         op_def=op_def)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1773 __init__
>         control_input_ops)
>     /homes/awl27/python36env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1613 _create_c_op
>         raise ValueError(str(e))
>
>     ValueError: Dimension must be 1 but is 2 for 'transformer/transpose_1' (op: 'Transpose') with input shapes: [116], [2].
>

## Expected behavior

I expected this to work just like TFBertModel did. If I just change the definition in the BERT class to use TFBertModel instead of TFXLNetModel, it works fine.

## Environment info
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: No
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla K40c
Nvidia driver version: 418.87.00
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip3] numpy==1.16.6
[pip3] torch==1.5.0
[pip3] torchtext==0.5.0
[pip3] torchvision==0.6.0
[conda] Could not collect


Improvements to PretrainedConfig documentation

Preview is [here](https://59194-155220641-gh.circle-artifacts.com/0/docs/_build/html/main_classes/configuration.html)


Multiple Mask Tokens

For those wishing to [MASK] several tokens, here this is.

My question, however, relates to the output. I added "top_k" assuming I'd be able to return multiple sentences, but that was not the case. I am not sure how exactly I can achieve this.

```
import torch
from transformers import BertTokenizer, BertModel,BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

input_tx = "[CLS] [MASK] [MASK] [MASK] of the United States mismangement of the Coronavirus is its distrust of science. [SEP]"
tokenized_text = tokenizer.tokenize(input_tx)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
top_k = 10
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([[0]*25])

model = BertForMaskedLM.from_pretrained('bert-base-cased')

outputs = model(tokens_tensor, token_type_ids=segments_tensors)
predictions = outputs[0]
predicted_index = [torch.argmax(predictions[0, i]).item() for i in range(0,24)]
predicted_token = [tokenizer.convert_ids_to_tokens([predicted_index[x]])[0] for x in range(1,24)]
print(predicted_token)

```

`Output: 'The', 'main', 'cause', 'of', 'the', 'United', 'States', 'mi', '##sman', '##gement', 'of', 'the', 'Co', '##rona', '##virus', 'is', 'its', 'di', '##st', '##rust', 'of', 'science', '`


Cleanup bart caching logic

Previously, we had a helper function that checked 4 possible cases to determine whether we should:

(a) combine a cached attention mask with a new one.
(b) just use the cached one
(c) just use the new/passed one

This consolidates that logic into 3 branches and deletes the helper func, which was only called once.


test suite fails due to pytorch bug in torch.seed

# 🐛 Bug

## Information

This is on a dual-gpu machine.

Almost all tests/test_modeling_reformer.py sub-tests fail with:

```
    def cb():
        for i in range(device_count()):
            default_generator = torch.cuda.default_generators[i]
>           default_generator.manual_seed(seed)
E           RuntimeError: Overflow when unpacking long
```

when run after any test_multigpu_data_parallel_forward sub-test, e.g.:

`python -m pytest -n 1 --dist=loadfile -v    tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward  tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs`

**The failure gets triggered here:**
```
transformers/modeling_reformer.py:1102: in _init_attention_seed
    self.attention_seed = int((torch.seed() % sys.maxsize))
```

I reduced the failing sequence of code to this:

```
# test.py
import torch
print(f"Torch version: {torch.__version__}")

x = torch.tensor(data=[[1,2],[3,4]], dtype=torch.long, device=None)
x = x.to('cuda:0')
seed = torch.seed()
```

```
$ python tests/test.py
Torch version: 1.5.1

Traceback (most recent call last):
  File "tests/test.py", line 10, in <module>
    seed = torch.seed()
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/random.py", line 45, in seed
    torch.cuda.manual_seed_all(seed)
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py", line 111, in manual_seed_all
    _lazy_call(cb)
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/__init__.py", line 99, in _lazy_call
    callable()
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py", line 109, in cb
    default_generator.manual_seed(seed)
RuntimeError: Overflow when unpacking long
```

It fails about 75% of time.

It happens after moving the tensor to gpu.

This seems to be related to this [pytorch bug](https://github.com/pytorch/pytorch/issues/33546), albeit somewhat different sequence of code.

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.1 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Full trace of failing sub-tests (one of them):

```
python -m pytest -n 1 --dist=loadfile -v tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs
====================================================================== test session starts =======================================================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /home/stas/anaconda3/envs/main/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/mnt/nvme1/code/huggingface/transformers-FlaubertForTokenClassification/.hypothesis/examples')
rootdir: /mnt/nvme1/code/huggingface/transformers-FlaubertForTokenClassification
plugins: hypothesis-5.5.4, filter-subpackage-0.1.1, arraydiff-0.3, flaky-3.6.1, ipynb-1.1.1.dev0, cov-2.10.0, astropy-header-0.1.2, forked-1.2.0, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2, xdist-1.32.0
[gw0] linux Python 3.7.5 cwd: /mnt/nvme1/code/huggingface/transformers-FlaubertForTokenClassification
[gw0] Python 3.7.5 (default, Oct 25 2019, 15:51:11)  -- [GCC 7.3.0]
gw0 [2]
scheduling tests via LoadFileScheduling

tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward
[gw0] [ 50%] PASSED tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward
tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs
[gw0] [100%] FAILED tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs

============================================================================ FAILURES ============================================================================
_______________________________________________________ ReformerLocalAttnModelTest.test_attention_outputs ________________________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <tests.test_modeling_reformer.ReformerLocalAttnModelTest testMethod=test_attention_outputs>

    def test_attention_outputs(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
        seq_len = getattr(self.model_tester, "seq_length", None)
        decoder_seq_length = getattr(self.model_tester, "decoder_seq_length", seq_len)
        encoder_seq_length = getattr(self.model_tester, "encoder_seq_length", seq_len)
        decoder_key_length = getattr(self.model_tester, "key_length", decoder_seq_length)
        encoder_key_length = getattr(self.model_tester, "key_length", encoder_seq_length)
        chunk_length = getattr(self.model_tester, "chunk_length", None)
        if chunk_length is not None and hasattr(self.model_tester, "num_hashes"):
            encoder_seq_length = encoder_seq_length * self.model_tester.num_hashes

        for model_class in self.all_model_classes:
            inputs_dict["output_attentions"] = True
            inputs_dict["output_hidden_states"] = False
            model = model_class(config)
            model.to(torch_device)
            model.eval()
            with torch.no_grad():
>               outputs = model(**self._prepare_for_class(inputs_dict, model_class))

tests/test_modeling_common.py:149:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/nn/modules/module.py:550: in __call__
    result = self.forward(*input, **kwargs)
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1623: in forward
    output_attentions=output_attentions,
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/nn/modules/module.py:550: in __call__
    result = self.forward(*input, **kwargs)
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1371: in forward
    output_attentions,
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1267: in forward
    output_attentions=output_attentions,
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/nn/modules/module.py:550: in __call__
    result = self.forward(*input, **kwargs)
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1138: in forward
    self._init_attention_seed()
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1102: in _init_attention_seed
    self.attention_seed = int((torch.seed() % sys.maxsize))
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/random.py:45: in seed
    torch.cuda.manual_seed_all(seed)
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py:111: in manual_seed_all
    _lazy_call(cb)
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/__init__.py:99: in _lazy_call
    callable()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def cb():
        for i in range(device_count()):
            default_generator = torch.cuda.default_generators[i]
>           default_generator.manual_seed(seed)
E           RuntimeError: Overflow when unpacking long

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py:109: RuntimeError
======================================================================== warnings summary ========================================================================
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(locations, collections.Iterable), 'Must provide locations for directive.'

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(args, collections.Mapping), '{} args must be a dict with argument names as keys.'.format(name)

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import OrderedDict, Sequence, defaultdict

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================================================================== short test summary info =====================================================================
FAILED tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs - RuntimeError: Overflow when unpacking long
============================================================ 1 failed, 1 passed, 4 warnings in 5.57s =======================================
```
```


Create README.md

Related to pretrained model cards

Create model card for T5-small fine-tuned on SQUAD v1.1


Add forum link in the docs


Should check that torch TPU is available

fix #5634


[WIP][Examples] Adding more examples and more introductory tutorials


T5 has no module  ```torch_xla``` when using T5 fine-tuned on SQUADv2

# 🐛 Bug

## Information

I get this error:
``` ModuleNotFoundError: No module named 'torch_xla'```
Full error message:

```
      2
      3 tokenizer = AutoTokenizer.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
----> 4 model = AutoModelWithLMHead.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
      5
      6 def get_answer(question, context):

1 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    796
    797         if hasattr(config, "xla_device") and config.xla_device:
--> 798             import torch_xla.core.xla_model as xm
    799
    800             model = xm.send_cpu_data_to_device(model, xm.xla_device())

ModuleNotFoundError: No module named 'torch_xla'
```

## To reproduce
```python
from transformers import AutoModelWithLMHead, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
model = AutoModelWithLMHead.from_pretrained("mrm8488/t5-base-finetuned-squadv2")

def get_answer(question, context):
  input_text = "question: %s  context: %s </s>" % (question, context)
  features = tokenizer([input_text], return_tensors='pt')

  output = model.generate(input_ids=features['input_ids'],
               attention_mask=features['attention_mask'])

  return tokenizer.decode(output[0])

context = "Manuel have created RuPERTa-base with the support of HF-Transformers and Google"
question = "Who has supported Manuel?"

get_answer(question, context)
```
I used this example code a few weeks ago and had no problem...


More explicit error when failing to tensorize overflowing tokens


Fixed use of memories in XLNet (caching for language generation + warning when loading improper memoryless model)

The default XLNet model is loaded with 0 memory length, which is an issue both at training time (improper performance) and inference time (as there's no caching speed-up since it doesn't return former attentions). As discussed with @LysandreJik , this PR introduces a warning that in the future, the default XLNet model will have 1024 memory length, in accordance with [the original paper](https://arxiv.org/abs/1906.08237). It also fixes the re-use of cached memory, which was broken similarly to TransfoXL (#4752).


Correct extension for model summary links


How can I fine-tune on custom model

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

I would like to use TFBert as encoder
then I will add additional layer on-top of it (with custom model class)

So, I would like to fine-tune all layer down to encoder layer

Specifically, I would like to do BERT-BiLSTM-CRF for NER task

Is there a way to do it?

Thank you for your answer

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Fixed TextGenerationPipeline on torch + GPU

Fixes #5622 .


Support for Polyencoder and other retriever based models

# ❓ Questions & Help

Is there any way I can load Polyencoder and other retriever based models from ParlAI in huggingface/transformers because as of now, there seems no support of loading huggingface/transformers models in ParlAI other than GPT.

Polyencoder: https://arxiv.org/abs/1905.01969
ParlAI Implementation : https://github.com/facebookresearch/ParlAI/blob/master/parlai/agents/transformer/polyencoder.py


Model doc failed

As title, model doc failed in page https://huggingface.co/transformers/model_summary.html, e.g. https://huggingface.co/model_doc/distilbert


doc: fix apparent copy-paste error in docstring


Cannot reproduce roberta-large on SQuAD

## Information
Using the same hyper-paramters as [paper](https://arxiv.org/abs/1907.11692), I fine-tund roberta-large on SQuAD1.1 resulting disappointing results as below. I am wondering that the reason of it might be the gradient noralization is different from the [official implementation](), although it hasn't been released yet.

```
{'exact': 0.21759697256385999,
'f1': 7.113439302309792, '
total': 10570,
'HasAns_exact': 0.21759697256385999,
'HasAns_f1': 7.113439302309792,
'HasAns_total': 10570,
'best_exact': 0.21759697256385999,
'best_exact_thresh': 0.0,
'best_f1': 7.113439302309792,
'best_f1_thresh': 0.0}
```

## To reproduce
```
python3 -m torch.distributed.launch --nproc_per_node=4 ./examples/question-answering/run_squad.py \
    --model_type roberta \
    --model_name_or_path roberta-large \
    --do_train \
    --do_eval \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 1.5e-5 \
    --weight_decay 0.01 \
    --max_grad_norm 0.0 \
    --num_train_epochs 2 \
    --warmup_steps 222 \
    --adam_betas '(0.9, 0.98)' \
    --adam_epsilon 1e-6 \
    --max_seq_length 512 \
    --doc_stride 128 \
    --output_dir ./examples/models/finetuned_squad1.1/ \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=2   \
    --gradient_accumulation_steps=6 \
    --threads 8 \
    --overwrite_cache \
```


Inference widgets for self-hosted models?

# 🚀 Feature request
I'm loving the new [huggingface](https://huggingface.co/bert-base-uncased?) dataset browsing & hosted model interfaces 🤯
So firstly a huge thank you to everyone <3

This is a question/feature request ~
- Can we use inference widgets for self-hosted models?

I see that there is a [serving.py](https://github.com/huggingface/transformers/blob/master/src/transformers/commands/serving.py) ( transformer-cli ) but nothing about widgets as far as I can see. If this is possible I would love an example on how; if not, will it be in the future?

## Motivation
Inference widgets would be nice to have during model testing & demos

## Your contribution
I was planning on doing something similar using `huggingface -> spacy(displacy) -> streamlit`  (for NER)


Predictor in Streamlit Docker eating all memory and OOM

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): DistilBert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Training on my datatset and creation of a model on Colab
2. Saving the said model locally
3. Loading the predictor in my app, with Streamlit library
4. Deployment of said app in a Docker container

This is my app, which use a predictor with Streamlit.

```
import pprint
import re

import ktrain
import numpy as np
import pandas as pd
import streamlit as st
import trafilatura
from googlesearch import search
from ktrain import text, predictor

st.title(':crystal_ball: schemaPredictor :crystal_ball:')

add_selectbox = st.selectbox(
    'How would you like to predict?',
    ('Text', 'Url'))

predictor = ktrain.load_predictor('./tmp/schema_mapping')

def get_prob(p):
    i = 0
    for x in p:
        if x > i:
            i = x
    return i

if add_selectbox == "Text":
    body = st.text_area('Insert your text here, as clean as possible.')
    if st.button("Predict"):
        st.success(":crystal_ball: " + predictor.predict(body) + " :crystal_ball:")
        st.success("With a probability of " + "{:.1%}".format(get_prob(predictor.predict_proba(body))))

elif add_selectbox == "Url":
    body = st.text_input('Insert your url here')
    if st.button("Predict"):
        page = body
        downloaded = trafilatura.fetch_url(page)
        result = trafilatura.extract(downloaded, include_tables=False, include_formatting=False,
                                     include_comments=False)
        st.success(":crystal_ball: " + predictor.predict(result) + " :crystal_ball:")
        st.success("With a probability of " + "{:.1%}".format(get_prob(predictor.predict_proba(result))))
```

## Expected behavior

If I run this app locally, without a Docker container, but in a conda env it works differently, it still takes memory at each iteration, but when it gets to around 10/11gb it frees memory to use it again. That on my 12gb ram laptop.
So I expected this to happen in my container too, but what happens in a container is that at each 'predict' it takes some memory, but it goes on untill it runs OOM. I tried with a Docker cointainer with 4 CPU, 12 gb of RAM and 1 gb SWAP

## Environment info
 Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2020-07-09 11:56:21.400037: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-07-09 11:56:21.404641: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3800005000 Hz
2020-07-09 11:56:21.404943: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55dce5a7cca0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-09 11:56:21.405087: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-09 11:56:21.406403: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-07-09 11:56:21.406441: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-07-09 11:56:21.406509: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (f8630e8d0e49): /proc/driver/nvidia/version does not exist

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 2.11.0
- Platform: Linux-4.19.76-linuxkit-x86_64-with-debian-10.4
- Python version: 3.7.7
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

#


TextGenerationPipeline breaks when used with device=0

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): model-agnostic (breaks with GPT2 and XLNet)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
[x] my own modified scripts: (give details below)

The tasks I am working on is:
[x] my own task or dataset: plain old language generation

## To reproduce

Steps to reproduce the behavior:

```
#!/usr/bin/env python3
import random
from transformers import pipeline, XLNetLMHeadModel
import torch
import time
random.seed(0)
torch.manual_seed(0)

generator = pipeline("text-generation", model="xlnet-base-cased", tokenizer="xlnet-base-cased", device=0)
output_to_check = generator("Today is a beautiful day and I, ", offset=offset, do_sample=True, top_k=50, max_len=100)
```

## Expected behavior

What should happen : text generation
What actually happens :

```
Traceback (most recent call last):
  File "/home/teven/dev_transformers/perso/transformers/generation_script.py", line 15, in <module>
    output_to_check = generator("Today is a beautiful day and I, ", offset=offset, do_sample=True, top_k=50, max_len=100)
  File "/home/teven/dev_transformers/perso/transformers/src/transformers/pipelines.py", line 692, in __call__
    generated_sequence = generated_sequence.numpy().tolist()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
```

Just missing a conversion before the `.numpy()` call

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-62-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Add freshly trained `codegram/calbert-base-uncased`

Related to pretrained model cards

Trained from the rewrite mentioned in #5599, just finished training last night. The model card now reflects both models, with tested code examples and links to Exbert.


Fix re-tokenization (ignoring is_pretokenized=True) when passing a pretokenized batch to both batch_encode_plus and tokenizer.__call__ methods

# Bug

> Fix unexpected behavior when passing already tokenized tokens (ignoring `is_pretokenized=True`) using `batch_encode_plus` and `self.__call__()`

```python
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

batch_sentences = ['The rapid expansion of the current COVID - 19 pandemic.',
                   'Hospitals around the globe have had to implement drastic changes']
batch_tokenized = [tokenizer.tokenize(x) for x in batch_sentences]
```

Correct output when passing either a single string or batch  ✅
- Applies to types:
 - `str` : one sentence
 - `List[str]` : batch of sentences

```python
# also applies when using batch_encode_plus
inputs = tokenizer(batch_sentences, add_special_tokens=False)
for ids in inputs['input_ids']: print(tokenizer.decode(ids))
...
"The rapid expansion of the current COVID - 19 pandemic."
"Hospitals around the globe have had to implement drastic changes"
```

Incorrect output when passing either a sequence of tokens or in batch  ❌
- Applies to types:
 - `List[str]`  : one sequence of string tokens
 - `List[List[str]]`  : batch of sequences of string tokens

```python
# also applies when using batch_encode_plus
inputs = tokenizer(batch_tokenized, add_special_tokens=False, is_pretokenized=True)
for ids in inputs['input_ids']: print(tokenizer.decode(ids))
...
"The rapid expansion of the current CO # # VI # # D - 19 pan # # de # # mic."
"Hospital # # s around the globe have had to implement drastic changes"
```
## Cause of issue

> The problem is when the sequences provided are a list of strings (pretokenized | batch of tokens) as we can observe in the output above, and the second condition in `get_input_ids()` completely disregards the truth that `is_pretokenized=True` by *re-tokenizing* a previously tokenized input!

```python
...
def get_input_ids(text):
    if isinstance(text, str):
        tokens = self.tokenize(text, **kwargs)
        return self.convert_tokens_to_ids(tokens)
    elif isinstance(text, (list, tuple)) and len(text) > 0 and isinstance(text[0], str):
        if is_pretokenized:
            # If the user set is_pretokenized=True, then the input is a batch of token string sequences.
            # The expected behavior is then to convert tokens to ids and not to re-tokenize - ``self.tokenize()``
            tokens = list(itertools.chain(*(self.tokenize(t, is_pretokenized=True, **kwargs) for t in text)))
            return self.convert_tokens_to_ids(tokens)
        else:
            return self.convert_tokens_to_ids(text)
            ...
```

## Fix

> All I needed to do is flip the behavior. Easy fix!

```python
...
if is_pretokenized:    # If already tokenized then, convert string tokens to token_ids
    return self.convert_tokens_to_ids(text)

else:  # Otherwise, tokenize to string tokens before converting to token_ids
    tokens = list(itertools.chain(*(self.tokenize(t, is_pretokenized=True, **kwargs) for t in text)))
    return self.convert_tokens_to_ids(tokens)
...
```


Should t5-small generate coherent text as summaries without finetuning?

I am following the summarization example and if I run `run_eval.py` for `t5-small` and `xsum` without finetuning I still get coherent, new (similar to source but not the same) and meaningful texts as summaries. The doc does not mention that it was pretrained on any kind of summarization task.


Generate up to max_target_length sequences

* Modifies the generate() call to allow for generation of sequences up to and including max_target_length number of tokens.
* Previous to this commit, implementation caps generation at 20 tokens and may result in poor performance.
* See related recent generation_utils.py commit: https://github.com/huggingface/transformers/blob/c4d4e8bdbd25d9463d41de6398940329c89b7fb6/src/transformers/generation_utils.py#L139


Update README.md

Related to pretrained model cards


fix 404


🐛 Bart Tokenization difference between 2.11.0 and 3.0.2

# 🐛 Bug

Running this code :

```python
from transformers import BartTokenizer
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large")
print(tokenizer.batch_encode_plus(["This is an example"]))
```

in `transformers` `2.11.0` and `3.0.2` gives different results.

`transformers` `2.11.0` :

> {'input_ids': [[0, 152, 16, 41, 1246, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1]]}

`transformers` `3.0.2` :

>  {'input_ids': [[0, 713, 16, 41, 1246, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1]]}

---

Colab for reproducing :

* [`2.11.0`](https://colab.research.google.com/drive/1qwYkcZoD1JtuLLDjABngJFoDUD06RiXm?usp=sharing)
* [`3.0.2`](https://colab.research.google.com/drive/1qUWcCUYInpa9Lwy2Ur-t3N3hImF1grCT?usp=sharing)

---

Is it from the refactoring of `generation_utils.py` ?


[WIP] Test TF Flaubert + Add {XLM, Flaubert}{TokenClassification, MultipleC…

…hoice} models and tests

The remaining TF tests pass with TF2.3. Waiting to unpin TF before merge.


doc fixes

a few minor doc improvements.


Did the run_language_model support TPU?

# ❓ Questions & Help

I try to using own dataset on tpu with running run_language_model.py, the command is I use below:
python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_language_modeling.py --model_name_or_path hfl/chinese-bert-wwm --output_dir model/tpu --train_data_file /Language_masked_model/data/toy_MLM_data.txt --line_by_line --mlm --block_size 512 --do_train --evaluate_during_training --per_device_train_batch_size 10 --tpu_num_cores 8 --debug --num_train_epochs 1 --save_steps 20000

**No errors but I assume it not use TPU,** I mentor the usage of TPU, get info below:

Cloud TPU Monitoring Results (Sample  20 ):

   TPU type: TPU v3
  Utilization of TPU Matrix Units (higher is better): 0.000%

Cloud TPU Monitoring Results (Sample  21 ):

   TPU type: TPU v3
  Utilization of TPU Matrix Units (higher is better): 0.000%

Cloud TPU Monitoring Results (Sample  22 ):

   TPU type: TPU v3
  Number of TPU cores: 1 (Replica count = 8, num cores per replica = 1)
  TPU idle time (lower is better): 0.027%
  Utilization of TPU Matrix Units (higher is better): 0.039%
  Step time: 11.1ms (avg), 11.1ms (min), 11.1ms (max)
  Infeed percentage: 0.000% (avg), 0.000% (min), 0.000% (max)

Cloud TPU Monitoring Results (Sample  23 ):

   TPU type: TPU v3
  Utilization of TPU Matrix Units (higher is better): 0.000%

Cloud TPU Monitoring Results (Sample  24 ):

   TPU type: TPU v3
  Utilization of TPU Matrix Units (higher is better): 0.000%

**My question is did  run_language_model.py support TPU?**

tpu: V3.8  on Google Cloud Platform
tensorflow==2.2.0
torch==1.7.0a0+12b5bdc
torch-xla==1.6+5430aca
I use offical docker on XLA (gcr.io/tpu-pytorch/xla:nightly_3.6) repo


IndexError: index out of range in self

# 🐛 Bug

## Information

The model I am using Bert ('bert-large-uncased') and I am facing two issues related to this model

The language I am using the model on English

The problem arises when using:

When I am trying to encode a large sentence ( sentence length 500 words ), I am getting this error :

`IndexError: index out of range in self`

I tried to set max_words length as 400, still getting same error :

Data I am using can be downloaded like this :

```
from sklearn.datasets import fetch_20newsgroups
import re

categories = ['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med']
twenty_train = fetch_20newsgroups(subset='train',categories=categories, shuffle=True, random_state=42)

print("\n".join(twenty_train.data[0].split("\n")[:3]))

X_tratado = []

for email in range(0, len(twenty_train.data)):

    # Remover caracteres especiais
    texto = re.sub(r'\\r\\n', ' ', str(twenty_train.data[email]))
    texto = re.sub(r'\W', ' ', texto)

    # Remove caracteres simples de uma letra
    texto = re.sub(r'\s+[a-zA-Z]\s+', ' ', texto)
    texto = re.sub(r'\^[a-zA-Z]\s+', ' ', texto)

    # Substitui multiplos espaços por um unico espaço
    texto = re.sub(r'\s+', ' ', texto, flags=re.I)

    # Remove o 'b' que aparece no começo
    texto = re.sub(r'^b\s+', '', texto)

    # Converte para minúsculo
    texto = texto.lower()

    X_tratado.append(texto)

dr = {}
dr ['text'] = X_tratado
dr ['labels'] = twenty_train.target
```

Now I am  using bert model to encode the sentences :

```
from transformers import BertModel, BertConfig, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
model     = BertModel.from_pretrained('bert-large-uncased')
inputs    = tokenizer(datar[7], return_tensors="pt")
outputs   = model(**inputs)
features  = outputs[0][:,0,:].detach().numpy().squeeze()
```

Which is giving this error  :

```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-41-5dcf440b245f> in <module>
      5 model     = BertModel.from_pretrained('bert-large-uncased')
      6 inputs    = tokenizer(datar[7], return_tensors="pt")
----> 7 outputs   = model(**inputs)
      8 features  = outputs[0][:,0,:].detach().numpy().squeeze()

~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/tfproject/tfenv/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states)
    751
    752         embedding_output = self.embeddings(
--> 753             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    754         )
    755         encoder_outputs = self.encoder(

~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/tfproject/tfenv/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    177         if inputs_embeds is None:
    178             inputs_embeds = self.word_embeddings(input_ids)
--> 179         position_embeddings = self.position_embeddings(position_ids)
    180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    181

~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1722         # remove once script supports set_grad_enabled
   1723         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1724     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1725
   1726

IndexError: index out of range in self
```

The second issue I am facing, When I am using this bert model to encode many sentences, It seems Bert is not using GPU :

![Screenshot 2020-07-09 at 12 45 14 AM](https://user-images.githubusercontent.com/17107749/86960748-9c905980-c17d-11ea-8d1e-bb72141cbf37.png)

How to accelerate GPU while using bert model?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: '3.0.0'
- Platform:  Ubuntu 18.04.4 LTS
- Python version: python3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): '2.2.0
- Using GPU in script?:
- Using distributed or parallel set-up in script?: