Unable to load Longformer pretrained weights

Hi,

I am following the official example to load longformer model by:

```
from transformers import LongformerModel, LongformerTokenizer
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
model = LongformerModel.from_pretrained('allenai/longformer-base-4096')

```
I can load tokenizer and config, but when I try to load model, it gave me the error "OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True." It happened on my other transformer models like BERT/XLNet, and I solved it by their corresponding functions "convert_tf_checkpoint_to_pytorch". I don't understand why this happen on Longformer as it is not from any tensorflow module?

I'm using
- `transformers` version: 3.0.0
- Python version: 3.7.4
- PyTorch version: 1.2.0


Update dependency mecab to at least version 1.0

Require at least version 1.0 of mecab, which comes with prebuilt wheels for all major platforms (OS X, Linux, Windows). This should do away with some reported incompatibility issues for non-linux systems.

See https://github.com/SamuraiT/mecab-python3/issues/31#issuecomment-651053281

Note: code untested but nothing much to test. [Version 1.0.0 is on PyPi](https://pypi.org/project/mecab-python3/1.0.0/) so it should work as written.


How to share model cards with the CLI

As seen offline with @julien-c, adding a new recommended way to upload model cards along with models.


Update README.md

model card

Related to pretrained model cards

- T5 pic uploaded to a more permanent place


Albert pooling dimension mismatches

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): **albert-base-v2**

Language I am using the model on (English, Chinese ...): **English**

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the following code snippet. You will need the **gdown** package. Note that I am using a modified version of modeling_albert.py to avoid the bug described [here](https://github.com/huggingface/transformers/pull/4095) and [here](https://github.com/huggingface/transformers/issues/1188).

```shell
# Install Transformers
pip install transformers

# Create a working folder (skip if using Colab)
mkdir /content/

# Clone BAT repository to the folder
git clone https://github.com/akkarimi/Adversarial-Training-for-ABSA.git

# Download patches for BAT scripts in src/
cd /content/
git clone https://github.com/williamsdaniel888/reimagined-octo-tribble.git
mv /content/reimagined-octo-tribble/modeling_albert.py /usr/local/lib/python3.6/dist-packages/transformers/modeling_albert.py
mv /content/reimagined-octo-tribble/absa_data_utils.py /content/Adversarial-Training-for-ABSA/src/absa_data_utils.py
mv /content/reimagined-octo-tribble/asc_bert_pt.py /content/Adversarial-Training-for-ABSA/src/asc_bert_pt.py
mv /content/reimagined-octo-tribble/bat_asc.py /content/Adversarial-Training-for-ABSA/src/bat_asc.py
mv /content/reimagined-octo-tribble/run_asc.py /content/Adversarial-Training-for-ABSA/src/run_asc.py

# Training
cd /content/Adversarial-Training-for-ABSA/script
chmod +x ./run_absa.sh
./run_absa.sh asc laptop_pt laptop pt_asc 1 0
```

2. Open **/content/Adversarial-Training-for-ABSA/run/pt_asc/laptop/1/train_log.txt** to observe the stack trace.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

**Stack trace:**
```
Traceback (most recent call last):
  File "../src/run_asc.py", line 280, in <module>
    main()
  File "../src/run_asc.py", line 275, in main
    train(args)
  File "../src/run_asc.py", line 122, in train
    _loss, adv_loss = model(input_ids, segment_ids, input_mask, label_ids)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/Adversarial-Training-for-ABSA/src/bat_asc.py", line 20, in forward
    _loss = self.loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2113, in nll_loss
    .format(input.size(0), target.size(0)))
ValueError: Expected input batch_size (1600) to match target batch_size (16).
```

A diagnostic printout of intermediate variables' sizes within the model's forward() method:
```
Shape of input_ids: torch.Size([16, 100])
Shape of embedding_output: torch.Size([16, 100, 128])
Shape of last encoded layer: torch.Size([16, 100, 768])
Shape of pooled_output: torch.Size([16, 100, 768])
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The pooling unit should pool over the second dimension of the last encoded layer, so that the shape of pooled_output is ```torch.Size([16, 768])``` instead of ```torch.Size([16, 100, 768])```.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Possible fix
I have worked around this issue by defining an AlbertPooler class for AlbertModel, similar to the BertPooler method for BertModel in modeling_bert.py:
```python
class AlbertPooler(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states):
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)
        return pooled_output
```

Make sure to then update **AlbertModel's init()** method:
```
self.pooler = AlbertPooler(config)
```
and **AlbertModel's forward()** method:
```python
pooled_output = self.pooler(sequence_output)
```


Update README.md

Related to pretrained model cards

- Model pic uploaded to a more permanent place


Update README.md

Related to pretrained model cards

- Fix missing ```-``` in language meta
- T5 pic uploaded to a more permanent place


Update README.md

Related to pretrained model cards


Fix model card folder name so that it is consistent with model hub

Related to pretrained model cards


Add link to file and fix typos in model card

Related to pretrained model cards


Doc for v3.0.0


[seq2seq docs] Move evaluation down, fix typo


# ❓ Questions & Help

## Details
I wanna build a new NLU model extented by TFRobertaPreTrainedModel.
And I write code with reference to 'TFRobertaForSequenceClassification' class.
```python
from transformers import TFRobertaPreTrainedModel, TFRobertaMainLayer
class TFRobertaForNLU(TFRobertaPreTrainedModel):
    def __init__(self, config, *inputs, **kwargs):
        super().__init__(config, *inputs, **kwargs)
        self.roberta = TFRobertaMainLayer(config, name="roberta")

    def call(self,inputs=None,attention_mask=None,token_type_ids=None,position_ids=None,head_mask=None,inputs_embeds=None,output_attentions=None,output_hidden_states=None,labels=None,training=False,):
        pass

model = TFRobertaForNLU.from_pretrained("jplu/tf-xlm-roberta-base")
```
And I got an error:
```python
Traceback (most recent call last):
  File "test.py", line 449, in <module>
  File ".../lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 489, in from_pretrained
    model.load_weights(resolved_archive_file, by_name=True)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 181, in load_weights
    return super(Model, self).load_weights(filepath, by_name)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1175, in load_weights
    saving.load_weights_from_hdf5_group_by_name(f, self.layers)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 751, in load_weights_from_hdf5_group_by_name
    str(len(weight_values)) + ' element(s).')
ValueError: Layer #0 (named "roberta") expects 0 weight(s), but the saved weights have 199 element(s).
```

It seems like the 'roberta' layer not properly initialized. Am I wrong in somewhere?


[Benchmark] Readme for benchmark

At @sshleifer, @julien-c, @clmnt  - shifting discussion about README.me here to include prev PR: #5360
in v3.0.0


Pin mecab for now

This fixes the tests locally, so will hopefully fix the CI :-)


[WIP] update pl=0.8.5

1. update to use from_argparse_args

2. deleted args that were manual but trainer has already.

3. fixed optimizer_step.

4. moved the LR logging to the correct hook

5. dropped the rank_zero_only stuff since that's not needed. All loggers and print ops happen only on rank_zero already. This was redundant.

6. proper use of setup()

@sshleifer let me know how I can test these changes


[Docs] Benchmark docs

This PR updates the docs for benchmarks and adds a README.md where the community can post their benchmark results.

Would be happy about Feedback from @sgugger and @LysandreJik .

@LysandreJik - I deleted the part about "This work was done by [Timothy Liu](https://github.com/tlkh)." because the links were broken.


Segmentation fault (core dumped) after importing transformers

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.import torch
2.import transformers
3.torch.rand(4,5)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```python
import torch
import transformers
torch.rand(4,5)
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I got an error 'segmentation fault (core dumped)' while trying to generate a tensor after importing transformers, but if I removed 'import transformers', the tensor could be generated.
![image](https://user-images.githubusercontent.com/32594371/86000735-83caca00-ba40-11ea-9f25-cf8617f843aa.png)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Ubuntu 18.04.4 LTS
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0+cu100
- Tensorflow version (GPU?): 2.2.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Do T5 have the next-sentence-predict loss?

Input the previous sentence and predict/generate the next sentence.

Thank you very much.


Fix table format fot test tesults

Related to pretrained model cards


Create model card

Related to pretrained model cards


Update Bertabs example to work again

Fix the bug 'Attempted relative import with no known parent package' when using the bertabs example. Also change the used model from bertabs-finetuned-cnndm, since it seems not be accessible anymore


Added data collator for XLNet language modeling and related calls

Related to pretrained model cards

Added `DataCollatorForXLNetLanguageModeling` in `data/data_collator.py` to return necessary inputs (applies masking and generates revelant tensors i.e. input_ids, perm_mask, target_mask and labels as per https://github.com/zihangdai/xlnet/blob/master/data_utils.py) for language modeling training with XLNetLMHeadModel. Also added related arguments, logic and calls in `examples/language-modeling/run_language_modeling.py`.

Resolves: #4739, #2008 (partially)


Create model card for asafaya/bert-large-arabic

Related to pretrained model cards


Create model card for asafaya/bert-mini-arabic

Related to pretrained model cards


Create model card for asafaya/bert-medium-arabic

Related to pretrained model cards


Move tests/utils.py -> transformers/testing_utils.py

Fixes #5350
The motivation is to allow examples/ tests to use these utilities.

for both groups of tests, the import is
```python
from transformers.testing_utils import slow
```

Motivation: I was about to rewrite the @slow decorator today and felt that this was cleaner.


T5 FP16: bad generations from my converted checkpoint

`t5-base` can generate reasonable summaries in fp16, but my checkpoint `sshleifer/t5-base-cnn` cannot. Is there something besides the conversion script I need to do to make fp16 work?

[Colab](https://colab.research.google.com/drive/1k8agNDPdzfF38aTrz5o1kcqvWDHUxAnD?usp=sharing) showing good generations for t5-base, bad for my checkpoint.

Without colab,
```python
from transformers import *
device = 'cuda'
tokenizer = T5Tokenizer.from_pretrained('t5-base')
my_model = T5ForConditionalGeneration.from_pretrained('sshleifer/t5-base-cnn').to(device)
TXT = """summarize:  Marseille, France (CNN)The French prosecutor leading an investigation into the crash of Germanwings Flight 9525 insisted Wednesday that he was not aware of any video footage from on board the plane. Marseille prosecutor Brice Robin told CNN that "so far no videos were used in the crash investigation." He added, "A person who has such a video needs to immediately give it to the investigators." Robin\'s comments follow claims by two magazines, German daily Bild and French Paris Match, of a cell phone video showing the harrowing final seconds from on board Germanwings Flight 9525 as it crashed into the French Alps. All 150 on board were killed. Paris Match and Bild reported that the video was recovered from a phone at the wreckage site. The two publications described the supposed video, but did not post it on their websites. The publications said that they watched the video, which was found by a source close to the investigation. "One can hear cries of \'My God\' in several languages," Paris Match reported. "Metallic banging can also be heard more than three times, perhaps of the pilot trying to open the cockpit door with a heavy object.  Towards the end, after a heavy shake, stronger than the others, the screaming intensifies. Then nothing." "It is a very disturbing scene," said Julian Reichelt, editor-in-chief of Bild online. An official with France\'s accident investigation agency, the BEA, said the agency is not aware of any such video. Lt. Col. Jean-Marc Menichini, a French Gendarmerie spokesman in charge of communications on rescue efforts around the Germanwings crash site, told CNN that the reports were "completely wrong" and "unwarranted." Cell phones have been collected at the site, he said, but that they "hadn\'t been exploited yet." Menichini said he believed the cell phones would need to be sent to the Criminal Research Institute in Rosny sous-Bois, near Paris, in order to be analyzed by specialized technicians working hand-in-hand with investigators. But none of the cell phones found so far have been sent to the institute, Menichini said. Asked whether staff involved in the search could have leaked a memory card to the media, Menichini answered with a categorical "no." Reichelt told "Erin Burnett: Outfront" that he had watched the video and stood by the report, saying Bild and Paris Match are "very confident" that the clip is real. He noted that investigators only revealed they\'d recovered cell phones from the crash site after Bild and Paris Match published their reports."""
batch = tokenizer.batch_encode_plus([TXT], return_tensors='pt').to(device)
tokenizer.batch_decode(my_model.generate(**batch, skip_special_tokens=True)) # way shorter than t5-base but english
# ['prosecutor in crash investigation says he is not aware of any video footage from on board ']
my_model = my_model.half()
my_model.generate(**batch, skip_special_tokens=True)
# [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
```

**Update:** On master +brutasse, (torch 1.5), t5-base produces bad generations in fp16. They are similar to the recently converted checkpoint -all bos.


T5 Warning: embeddings are not initialized

```python
from transformers import *
model = T5ForConditionalGeneration.from_pretrained("t5-base")
```

Is this concerning?

```
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at t5-base and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Relatedly, I am porting a summarization checkpoint and wondering whether I should initialize lm_head.


Training with a large dataset

I'm currently following the tutorial on how to train a new language model ([here](https://huggingface.co/blog/how-to-train)) and facing some issues on Colab because of my large training corpus (+40 mil lines, 5 GB).

I tried to use IterableDataset so I can load data on the fly, and I'm getting this error when I try to train with the script provided in the tutorial:
"ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.RandomSampler object at 0x7f777bdbe710>"

What's the best way to resolve this?


Upload model card with the CLI

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Upload model card README.md with transformers CLI.

## Motivation

Some applications may require to upload multiple models. It would be nice to be able to upload associated model card with model/tokenizer/config files.

At the moment a PR is required making it more difficult for users to upload their models and nearly impossible for apps relying on training and uploading a lot of different models.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I could help propose a PR for the client side.


Massive text generation slowdown when using repetition_penalty param on GPU

# 🐛 Bug

## Information

Text generation when using the `repetition_penalty` takes about 2x-10x longer on a GPU, which is disproportionate and implies the GPU may not be used in that instance.

Reported from https://github.com/minimaxir/aitextgen/issues/34

The `enforce_repetition_penalty_` function at

https://github.com/huggingface/transformers/blob/08c9607c3d025f9f1a0c40e6d124d5d5d446208e/src/transformers/modeling_utils.py#L817

may be using CPU ops instead of Tensor ops.

## To reproduce

Demo Colab notebook w/ time benchmarks: https://colab.research.google.com/drive/1SzYUEC0xikHN8OEp2WOTWHlT6JxT9rRf?usp=sharing


[examples] fix example links

Fix links for summarization and translation examples in BIG TABLE OF TASKS.
Regarding issue #5309

@sshleifer


[Reformer] Simpler reverse sort backward implementation

After reviewing code of similar PyTorch implementation: https://github.com/lucidrains/reformer-pytorch/pull/104 and original trax code again: https://github.com/google/trax/blob/master/trax/layers/research/efficient_attention.py#L1265-L1266, there is a simpler way to implement the backward reverse sort function.
The code is taken from https://github.com/lucidrains/reformer-pytorch/blob/42a8682ff8e7cec3122eff6febc9087f1c53f370/reformer_pytorch/reformer_pytorch.py#L414.

This simpler code is also implemented in the Reformer test branch and all tests are checked for correctness: https://github.com/huggingface/transformers/tree/branch_to_save_trax_integration_tests

**Note**: There was no bug in the code before / the logic of the code has not changed. This PR just makes the backward function of ReverseSort simpler.


Added support for XLNet language modelling training in examples

Updated src/transformers/data/data_collator.py with a new XLNet-specific collator that applies masking and generates revelant tensors (input_ids, perm_mask, target_mask, labels) as per https://github.com/zihangdai/xlnet/blob/master/data_utils.py. Also added relevant calls and imports in examples/language-modeling/run_language_modeling.py. Relevant issues #4739 #2008


In Tensorflow the serving is very slow

# ❓ I'm using the gpt2 model, In Tensorflow the serving is very slow, Tensorflow is much faster.

## Details
I use tf.saved_model save model
`tf.saved_model.save(model, export_dir="/model/2/")`

Tensorflow reads the model
```
imported = tf.saved_model.load("/model/2/")
f = imported.signatures["serving_default"]
```

The speed of Tensorflow is much higher than that of tensorflow
In the same model without the use of transformers, tensorFlow and TensorFlow serve at the same speed


Core: Tokenization

Internals of the library; Tokenization.

# ❓ Questions & Help
GPT2Tokenizer remove the ' ' (space) if it is at the end of text?
## Details
<!-- Description of your issue -->
Hello~ I'm using transformers v2.11.0 to run some GPT2 model.
When I tested the code, I found that the token id list generated by GPT2Tokenizer had confusing result when I input a text string end with a space.

To reproduce:
```
from transformers import GPT2Tokenizer
tokenizer_file = 'path_to_your_tokenzier_files'
tokenizer = GPT2Tokenizer.from_pretrained(tokenizer_file)

# case 1: without a space at the end input text string
text_1 = 'import tensorflow as'
encoded_1 = tokenizer.encode(text_1, add_special_tokens=False)
print(encoded_1)

# case 2: with a space at the end input text string
text_2 = 'import tensorflow as '   # noted there is a space at the end
encoded_2 = tokenizer.encode(text_2, add_special_tokens=False)
print(encoded_2)
```
the output is confusing
```
# case 1:
 [618, 1969, 573]

# case 2:
 [618, 1969, 573]
```
both these two case output the same token id list, with my tokenzier is [618, 1969, 573]. Which means the space at the end of case2 is ignored in the tokenization process.

I debug the code and found that the space is ignored at this step
https://github.com/huggingface/transformers/blob/b42586ea560a20dcadb78472a6b4596f579e9043/src/transformers/tokenization_utils.py#L1288

Is this step with some propose?

I also tried the ByteLevelBPETokenizer in tokenizers project
```
from tokenizers.implementations import ByteLevelBPETokenizer

tokenizer_file = 'path_to_your_tokenzier_files'
tokenizer = ByteLevelBPETokenizer(tokenizer_file)

# case 1: without a space at the end input text string
text_1 = 'import tensorflow as'
encoded_1 = tokenizer.encode(text_1)
print(len(encoded_1))
print(encoded_1.ids)
print(encoded_1.tokens)

# case 2: with a space at the end input text string
text_2 = 'import tensorflow as '
encoded_2 = tokenizer.encode(text_2)
print(len(encoded_2))
print(encoded_2.ids)
print(encoded_2.tokens)
```
the output are as expected:
```
# case 1
3
[618, 5015, 573]
['import', 'Ġtensorflow', 'Ġas']

# case 2
4
[618, 5015, 573, 231]
['import', 'Ġtensorflow', 'Ġas', 'Ġ']
```


Predefined tasks in T5

```
  "task_specific_params": {
    "summarization": {
      "early_stopping": true,
      "length_penalty": 2.0,
      "max_length": 200,
      "min_length": 30,
      "no_repeat_ngram_size": 3,
      "num_beams": 4,
      "prefix": "summarize: "
    },
    "translation_en_to_de": {
      "early_stopping": true,
      "max_length": 300,
      "num_beams": 4,
      "prefix": "translate English to German: "
    },
    "translation_en_to_fr": {
      "early_stopping": true,
      "max_length": 300,
      "num_beams": 4,
      "prefix": "translate English to French: "
    },
    "translation_en_to_ro": {
      "early_stopping": true,
      "max_length": 300,
      "num_beams": 4,
      "prefix": "translate English to Romanian: "
    }
  },

```
The config file generated while saving a T5 model mentions the above-predefined tasks which can be done by just prepending the inputs with appropriate prefixes. I wanted to know if there is a list that shows any other predefined tasks which can be performed by T5.


Confuse by  "All learning rates are 0"

transformers/examples/seq2seq/finetune.py  219row :
        if max(scheduler.get_last_lr()) > 0:
            warnings.warn("All learning rates are 0")

why, is there something i left?


arxiv-ai-gpt2 model card

model card

Related to pretrained model cards

Add model card and generation script for model arxiv_ai_gpt2.


BillSum dataset finetuning

(from distilbart issue).

Hey @sshleifer , thanks for the distilled BART version I was able to fine tune it with the same script on BillSum dataset as T5 but the numbers are way different between the two. I just wanted to understand if I might be doing something wrong with regards to fine tuning distilBART, does it require student training everytime?
Reference numbers on BillSum Dataset:

T5-base:
avg_train_loss = tensor(1.5333, device='cuda:0')
avg_val_loss = tensor(1.4528, device='cuda:0')
epoch = 1
loss = tensor(1.6734, device='cuda:0')
rouge1 = 0.49188267841912325
rouge2 = 0.26436589848185027
rougeL = 0.3591894400892483
train_loss = tensor(1.6734, device='cuda:0')
val_loss = tensor(1.4528, device='cuda:0')

dBART-cnn-12-6:
avg_train_loss = tensor(1.3013, device='cuda:0')
avg_val_loss = tensor(1.4013, device='cuda:0')
epoch = 1
loss = tensor(1.4901, device='cuda:0')
rouge1 = 0.3681518923769047
rouge2 = 0.15683286277623087
rougeL = 0.23453727441540043
train_loss = tensor(1.4901, device='cuda:0')
val_loss = tensor(1.4013, device='cuda:0')

PS. I am using a modified version of the older finetune.py so it doesn't have Rouge for validation epochs.

Thanks


BertForPreTraining and BertModel when loading TF checkpoints

Hello,
I want to load tensorflow checkpoints and when I run code below

```python
config = BertConfig.from_json_file('./bert_config.json')
config.output_hidden_states = True
bert_model = BertForPreTraining.from_pretrained('./model.ckpt', from_tf=True, config=config)
......
self.bert = bert_model
.......
bert_seq_out, _ = self.bert(input_ids, token_type_ids=segment_ids, attention_mask=input_mask,
output_hidden_states=False)

```

got error like this

```
Traceback (most recent call last):
  File "train_tf.py", line 738, in <module>
    neg_log_likelihood = model.neg_log_likelihood(input_ids, segment_ids, input_mask, label_ids)
  File "train_tf.py", line 605, in neg_log_likelihood
    bert_feats = self._get_bert_features(input_ids, segment_ids, input_mask)
  File "train_tf.py", line 542, in _get_bert_features
    bert_seq_out, _ = self.bert(input_ids, token_type_ids=segment_ids, attention_mask=input_mask, output_hidden_states=False)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'output_hidden_states'
```

I think it is because I use `BertForPreTraining` instead of `BertModel`,
but when I use `BertModel`, I got error like below

```
/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    592                 return modules[name]
    593         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 594             type(self).__name__, name))
    595
    596     def __setattr__(self, name, value):

AttributeError: 'BertModel' object has no attribute 'bias'
```

How can I fix and load TF checkpoint correctly?
any help will be appreciated


Add "labels" functionality for all TF Causal LM and Masked LM models

# 🚀 Feature request

Currently, it is not possible to calculate the loss of a CLM or MLM model using TF:

```python
from transformers import TFBertForMaskedLM, BertTokenizerFast

model = TFBertForMaskedLM.from_pretrained("bert-base-uncased")
tok = BertTokenizerFast.from_pretrained("bert-base-uncased")

input_ids = tok("This is a test string.").input_ids
loss = model(input_ids, labels=input_ids)[0]  # This is currently not possible
```
Currently the `call(...)` function does not accept a `labels` input and does not return the
loss. This should be implemented similar to how it was done for `BertForSequenceClassification`:
https://github.com/huggingface/transformers/blob/393b8dc09a97197df1937a7e86c0c6b4ce69c7e9/src/transformers/modeling_tf_bert.py#L918

All CLM and MLM TF models should be updated to accept `labels` as an input and return the corresponding loss.

## Motivation

1. This allows to use TFTrainer for CLM and MLM models.
2. This aligns TF with PT API

## Your contribution

Starting from 29.06/30.06 I want to implement all these features (with some help from @jplu if possible ;-))

Pinging @LysandreJik @julien-c @sgugger for notification.


XLNet with high CPU usage

The same issue as these two:

https://github.com/huggingface/transformers/issues/1722#issue-517212978
https://github.com/huggingface/transformers/issues/1529#issue-507567684


Link to the example/summarization in doc is broken

The link to the example/summarization folder in [the-big-table-of-tasks](https://huggingface.co/transformers/examples.html#the-big-table-of-tasks) is broken because the folder's name has been changed to seq2seq


Adds train_batch_size, eval_batch_size, and n_gpu to to_sanitized_dict output for logging.

Closes #5330

```
>>> from transformers import TrainingArguments
>>> args = TrainingArguments("dir")
>>> args.to_sanitized_dict()
{
    "output_dir": "dir",
    "overwrite_output_dir": False,
    "do_train": False,
    ...
    "train_batch_size": 8,
    "eval_batch_size": 8,
    "n_gpu": 0,
}
```


Better hyperparameter tensorboard logging in Trainer.

# 🚀 Feature request

`Trainer` should write `train_batch_size` and `eval_batch_size` to tensorboard. Currently only `per_device_` batch sizes are logged as hyperparamters, which means unless you know how many gpus you trained on (which is also not logged), you can't know the actual batch sizes used in training.

## Your contribution

Submitted a PR: [Adds train_batch_size, eval_batch_size, and n_gpu to to_sanitized_dict output for logging.](https://github.com/huggingface/transformers/pull/5331#issue-440857548)


Add option to keep tb_writer open after training is done.

Currently, `Trainer` automatically closes the tensorboard writer `tb_writer` after the training loop is done. Often I'll want to add extra observations, metrics, data, etc ... to the tensorboard event that requires the trained model (ie after the training loop is done). For example.

```
tb_writer = SummaryWriter(log_dir="some_dir")

# do some stuff with tb_writer before training

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
    compute_metrics=compute_metrics,
    tb_writer=tb_writer,
)
trainer.train()

# do extra stuff with tb_writer after the model is trained (that requires a trained model)
```

## Your contribution

Submitted a PR: [Adds option to keep tb_writer open after training finishes](https://github.com/huggingface/transformers/pull/5328#issue-440856807)


Adds option to keep tb_writer open after training finishes

Set the `close_tb_writer` parameter to `False` in `Trainer.train()` to keep the tensorboard writer open.

```
tb_writer = SummaryWriter(log_dir="some_dir")

# do some stuff with tb_writer before training

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
    compute_metrics=compute_metrics,
    tb_writer=tb_writer,
)
trainer.train(close_tb_writer=False)

# you can now use tb_writer even after training!
```

Closes #5329


[mBART] skip broken forward pass test, stronger integration test

translation

machine translation utilities and models

(1) I fixed the config of mbart to go from BLEU score 12 to BLEU score 26. This requires that the `decoder_start_token_id=lang_code['ro']` but that we filter that out when we decode.
(2) I added another, harder, integration test case. The expected result is only 1 word different than GT. This also adds implicit coverage for batched generation.
(3) tried for awhile to fix the broken slow GPU `test_enro_forward`.
On commits where the test passed, it doesn't pass any more. And the model hadn't been updated on S3 since March 26. So for now we skip it.


In the run_ner.py example, give the optional label arg a default value

Otherwise, if label is not specified, the following error occurs:

	Traceback (most recent call last):
	  File "run_ner.py", line 303, in <module>
	    main()
	  File "run_ner.py", line 101, in main
	    model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
	  File "/home/user/anaconda3/envs/bert/lib/python3.7/site-packages/transformers/hf_argparser.py", line 159, in parse_json_file
	    obj = dtype(**inputs)
	TypeError: __init__() missing 1 required positional argument: 'labels'


Added a model card README.md for my pretrained model.

model card

Related to pretrained model cards


More model cards

Related to pretrained model cards


New model sharing tutorial

This PR removes the old serialization guide which was a bit outdated and adds the new instructions to the current model sharing tutorial (rewritten in rst to have proper links to the rest of the docs).

Also it adds a link to the trainer tutorial recently merged in the quicktour.

Preview is [here](https://53867-155220641-gh.circle-artifacts.com/0/docs/_build/html/model_sharing.html)


examples/seq2seq/run_eval.py fixes and docs


[{m}bart] Fix final_logits bias warning

This warning is safe to ignore, but can be easily fixed.

```bash
Some weights of BartForConditionalGeneration were not initialized from the model checkpoint at facebook/mbart-large-en-ro and are newly initialized: ['final_logits_bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```


Reformer model axial.position.shape config not working

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->
I am trying to play around with the config parameters of the Reformer model;
I change the axial position shape parameters to ;
"axial_pos_shape": [
    64,
    256
  ]
and the "max_position_embeddings": 8192 (multiplication of axial_pos_shape)

But it get the following error;
**RuntimeError: Error(s) in loading state_dict for ReformerForSequenceClassification:
	size mismatch for reformer.embeddings.position_embeddings.weights.0: copying a param with shape torch.Size([512, 1, 64]) from checkpoint, the shape in current model is torch.Size([64, 1, 64]).
	size mismatch for reformer.embeddings.position_embeddings.weights.1: copying a param with shape torch.Size([1, 1024, 192]) from checkpoint, the shape in current model is torch.Size([1, 256, 192]).**

The rest of parameters are unchanged. What's wrong with the config?
## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Fix `xxx_length` behavior when using XLNet in pipeline

When using a `pipeline` for text gerenation with XLNet and transformer-XL, the pipeline adds a long prompt at the beginning to help the model. It messes up with `min_length` and `max_length` so I fixed that by adding to those kwargs (if passed) the length of the prompt.

Also made the prompt use the tokenizer eos token instead of putting the three possibilities at the end.


[CI] GH-runner stores artifacts like CircleCI

Split this out to merge when GH runner is working.


Clearer lr schedule math


[pl_examples] default warmup steps=0


Add BART-base modeling and configuration

Fix 404 Not Found for BART-base


Transformer-XL not working with DistributedDataParallel

# 🐛 Bug

## Information
I'm using the pretrained `TransfoXLLMHeadModel` for finetuning but I get an error when trying to use it with pytorch's `DistributedDataParallel`:
> self.reducer.prepare_for_backward(list(_find_tensors(output)))
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable). (prepare_for_backward at /opt/conda/conda-bld/pytorch_1579022034529/work/torch/csrc/distributed/c10d/reducer.cpp:514)

I'm using the pytorch distributed launch utility.
I've already gone through similar issues regarding this topic, but none of the solutions solved my problem. A guess from me is, that this is caused by the returned `mems` and `DistributedDataParallel` could think that they require a gradient?

Model I am using: Transformer-XL

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-4.15.0-108-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes `torch.distributed.launch`


Model cards for finance-koelectra models

Related to pretrained model cards

Added new cards for the models.


Add benchmark notebook

This PR adds a general notebook on how to use the Benchmark tools


AllenNLP SPECTER model

# 🌟 New model addition

SPECTER: Document-level Representation Learning using Citation-informed Transformers

## Model description

[SPECTER](https://arxiv.org/pdf/2004.07180.pdf) by AllenAI is a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph. It is based on SciBERT.

This model is showing us promising results for scientific paper similarity tasks. However, the repo is quite rough around the edges and inference performance could be better. According to the AllenNLP repo issues, they do not plan to adapt the models to ONNX runtime.

It would be great to have a plug'n'play implementation for Transformers. Instructions on how to do this are also welcome 🤝

## Open source status

* [x] the model implementation is available: https://github.com/allenai/specter
* [x] the model weights are available: see repo above
* [x] who are the authors: AllenAI, @armancohan, @sergeyf


overflowing tokens are now always returned

closes https://github.com/huggingface/transformers/issues/5293

@thomwolf may be superseded by https://github.com/huggingface/transformers/pull/5308/files


Links out of date under transformers/examples/README.md

# 🐛 Typos

## Information

Under the Big Table of Tasks in `transformers/examples/README.md`, the links for summarization and translation are out of date. They should lead to `examples/seq2seq` instead of `examples/summarization` and `examples/translation`.


[tokenizers] Updates data processors, docstring, examples and model cards to the new API

Related to pretrained model cards

Updates the data-processors to the new recommended tokenizers' API instead of the old one.
Also update the docstrings, examples, and model-cards which were using the old API.

Supersede #5310

@sshleifer you have a couple of methods only your models use (bart and marian). I'm not sure about the consequences of updating those API so I'll let you update them. Here is the doc on the new tokenizer API if you need it: https://huggingface.co/transformers/master/preprocessing.html
Recommended updates:
- use `__call__` instead of `encode_plus` and `batch_encode_plus`
- use `padding` and `truncation` instead of `max_length` only and `pad_to_max_length`


More model cards

Related to pretrained model cards


[Generation] fix docs for decoder_input_ids

Fix docs for decoder input ids.


# ❓ Questions & Help
Hi, I have seen BERT public the pre-training language model like BERT-based-uncased. Does BERT public an embedding file like glove.840B.300d.txt? Many thanks.


Bert Abs not using GPU

# 🐛 Bug

## Information

Model I am using: Bert abs (https://github.com/huggingface/transformers/tree/master/examples/seq2seq/bertabs)
-> remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization model

Language I am using the model on: English

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Install Conda Environment with proper pytorch installation
2. Verify that `torch.cuda.is_available()` return True
3. Start run_summarization.py as stated in the Readme

## Expected behavior

I would expect, that the models uses GPU for Training / Generation. Instead I have a very high CPU usage, Gpu usage is nearly zero and GPU clock stays idle. Since I verified that the GPU and cuda is available, I expect it to use it.

--no_cuda is not set, so the default of False is taken used

## Environment info

- `transformers` version: 2.11.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Attempted relative import with no known parent package

# 🐛 Bug

## Information

Model I am using: Bertabs (https://github.com/huggingface/transformers/tree/master/examples/seq2seq/bertabs)

Language I am using the model on: English

The problem arises when using:
 [x ] the official example scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce
Hi, I did as the Readme says but the following error is thrown, when I want to start the training via:

`python run_summarization.py --documents_dir "data/stories" --summaries_output_dir "out" --no_cuda false --batch_size 4 --min_length 50 --max_length 200 --beam_size 5 --alpha 0.95 --block_trigram true
`:

` File "run_summarization.py", line 15, in <module>
    from .utils_summarization import (
ImportError: attempted relative import with no known parent package`

I am using Win10, and an Anaconda Env.

Steps to reproduce the behavior:

1. Install a new Anaconda Env with torch
2. Do as the Readme says
3. Put the code into a file or start it via console.

## Expected behavior
Start Training, or any error message, which I can resolve.

## Environment info
Using Windows 10, and an Anaconda Env.

Thank you


Transformer-XL: Fixed tokenization of brackets, numbers etc.

Fixes #5136

As explained in the above issue, this PR fixes the tokenization of the `TransfoXLTokenizer` by using the `sacremoses` library with an extended feature of tokenizing comma-separated and floating point numbers. That way the input text is tokenized the same way as in the WikiText-103 dateset used for pretraining.

Open questions:
- [ ] Should the method `prepare_for_tokenization` be removed, since it just issues a warning for no whitespace before punctuation? This is solved by the `MosesTokenizer` anyway.


Request for inclusion of PEGASUS for text summarization by Google.

# 🚀 Feature request

Google has recently released PEGASUS, a new model for text summarization: https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html. I think It would be interesting to have it included in Transformers library. The code and checkpoints are here: https://github.com/google-research/pegasus. This model seems to accomplish human-like summarization.


T5ForConditionalGeneration fp16 nan loss

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

(Pdb) net_inputs["attention_mask"]
tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0')
(Pdb) net_inputs["lm_labels"]
tensor([[    3,  1489,    89,  -100,  -100,  -100],
        [ 5441,  1511,    13,     3, 10314,   152]], device='cuda:0')
(Pdb) net_inputs["lm_labels"]
tensor([[    3,  1489,    89,  -100,  -100,  -100],
        [ 5441,  1511,    13,     3, 10314,   152]], device='cuda:0')
(Pdb)

1.  model = T5ForConditionalGeneration.from_pretrained("t5-large")
2. outputs = model(input_ids=net_inputs["input_ids"], attention_mask=net_inputs["attention_mask"], lm_labels=net_inputs["lm_labels"])
3. loss, lm_logits = outputs[0], outputs[1]; print(loss)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Seems like a masking issue.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: V100
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0
- Tensorflow version (GPU?):N/A
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: parallel


No documentation for MMBT on official docs

I tried finding MMBT on the official documentation: https://huggingface.co/transformers/ but I could not find any references to it, even though there is an implementation for it in the source code.


The start and end position of BertForQuestionAnswering

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am confused with the start and end position of Bert for QA model because of wordpiece. My question is: what is the value of position based on? For example, if start_position=10, it means the 10th word of the input or the 10th subword of input after wordpiece? Also, is the position value contains the length of question?
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Can we have a way for a tokenizer to transform word level or character level annotations?

Core: Tokenization

Internals of the library; Tokenization.

# 🚀 Feature request

If I have a string with some annotations and I want to tokenize it I'd like the tokenizer to be able to transform the annotations as well. For example suppose I have `s = "therein lies the problem."` and I'm interested in the substring `"the problem"`. So I have a string `s` and I know that the substring I'm interested in is at index 13:24. But then I tokenize `s` so that I can put it into a huggingface model and get out `['there', '##in', 'lies', 'the', 'problem', '.']` and it doesn't match up with my annotation anymore. Could we add an annotation as an additional argument to the tokenizer.tokenize function so that I could get something like the following:

```
tokenizer.tokenize("therein lies the problem.", selection=(13,24))
> ['there', '##in', 'lies', 'the', 'problem', '.'], [0, 0, 0, 1, 1, 0]
```

Substrings of interest are almost always going to line up with token boundaries so it doesn't matter too much what happens to a token that is partially in and partially outside of the selected region.

Is there a way of doing something like this now?


Update outdated TensorFlow -> PyTorch model transfer CLI example

The example is outdated and reports an error.


Is summing of attention_mask intended?

# ❓ Questions & Help
Is summing of `attention_mask` intended?

## Details
Documents describe `attention_mask` as a mask to avoid performing attention on padding token indices.
However from the [code](https://github.com/huggingface/transformers/blob/2ffef0d0c7a6cfa5a59c4b883849321f66c79d62/src/transformers/modeling_bert.py#L243) in `BertSelfAttention` the attention mask is added to the scores.
```
attention_scores = attention_scores + attention_mask
```

Is this intended? Does this fully mask out the context?


Slow Integration Test for examples/seq2seq/finetune.py

Help wanted

Extra attention is needed, help appreciated

Train `sshleifer/student-xsum-12-3` for N batches on xsum data.
Ensure that loss goes down. and is below ~2.
This test should ONLY run on GPU and takes between 30s and 3 mins.

python code to fetch xsum data:

```
import wget
import tarfile
wget.download('https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz')
tarball = tarfile.open('xsum.tar.gz')
tarball.extractall()
data_dir='xsum'
```
cc @williamFalcon


run_squad.py :: ValueError: Input [] is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
[*] the official example scripts: (give details below)
[ ] my own modified scripts: (give details below)

The tasks I am working on is:
[*] an official GLUE/SQUaD task: (give the name)
[ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. while running run_squad.py
2.  Training & testing with
https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Error
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 320, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
ValueError: Input [] is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.

## Expected behavior

To work

## Environment info
- `transformers` version: 2.11.0
- Platform: Linux-5.3.0-1017-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Saving and loading tokenizers with torch.save fails

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Albert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load albert base tokenizer using `AutoTokenizer.from_pretrained`
2. Save it to a file using `torch.save`
3. Delete `~/.cache/torch/transformers` directory
4. Now try to load from the file using `torch.load`
5. Loading fails as the cached file does not exist

```
import transformers
import torch
token = transformers.AutoTokenizer.from_pretrained("albert-base-v2")
torch.save({"token":token}, "./token.pt")
```
Delete `~/.cache/torch/` directory
Then Run
```
import torch
torch.load("./token.pt")
```

## Expected behavior
Tokenizer should load successfully.

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-4.19.104-microsoft-standard-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.3.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


CircleCI stores cleaner output at test_outputs.txt

Changes:

- Don't run pytest with `-v` or `--cov`. I left run_tests_torch_and_tf using `--cov` but would love to delete that. I have never used circleci to determine code coverage.

- the run* jobs create artifact files called test_output.txt that are easier to read than scrolling in the circleci gui

- self scheduled runner also attempts to make a test_output.txt

Before:
![image](https://user-images.githubusercontent.com/6045025/85790787-d33a8e80-b6fe-11ea-9e10-d4c48dfe713f.png)

After:

[artifact](https://53278-155220641-gh.circle-artifacts.com/0/test_output.txt) is very manageable

and [ui](https://app.circleci.com/pipelines/github/huggingface/transformers/8144/workflows/816b1f60-bda9-4456-9f51-92d6cff7b266/jobs/53278/steps) is less noisy.


Model with fastest inference?

Just curious whether there is any benchmark for inference speed of models in the transformers library? I am interested in the question-answer task but I think a benchmark on any tasks that BERT can do should be good. Thank you.


[pipelines] Change summarization default to distilbart-cnn-12-6

Core: Pipeline

Internals of the library; Pipeline.

- Also adds an integration test that runs on GPU if available.
- Other pipelines could do the same if that would be helpful.


Is there a Longformer For Sequence Classification?

I noticed that Longformer does not have a "LongformerForSequenceClassification". Is there a reason for this and is this something that would be added in the near future?


[tokenizers] Several small improvements and bug fixes

Various improvements for tokenizers:
- Avoid recursion loop for special tokens id look-up in Fast tokenizers
- Fix #5232 by removing the unsupported method `convert_tokens_to_string` for Fast tokenizers
- Fix #5256 by aligning the behavior of the slow tokenizer on the behavior of the fast tokenizer for special tokens inside the input.

A little bit of background on the modifications in Roberta tokenizer:
We now align the behavior of the byte-level BPE tokenizer to the Fast version which is the most consistent with the way the original tokenizer behaved: all the special tokens are assumed to not have a prefix space so the user can control whether he wants to have a space or not in the string.

We do an exception for the mask token in Roberta which is assumed to represent a word and thus has a prefix space by default (can be overided at initialization). This is necessary to be able to use Roberta in filled-mask completion easily.

This is already built-in for the Fast tokenizer. Here I update the slow tokenizer to have this behavior using the newly introduced `AddedToken` which lets you control the space behaviors of the special tokens.


save_pretrained on master results in tokenizers that cannot be loaded in v2.11

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using sshleifer/distilbart- *

The problem arises when using:
tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-xsum-12-3")  -it fails here

## To reproduce

Steps to reproduce the behavior:

tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-xsum-12-3")  -it fails here

![image](https://user-images.githubusercontent.com/40685761/85780041-de0c1800-b72c-11ea-840e-ed4402da3a01.png)

## Environment info

- `transformers` version:2.8.0
- Platform:
- Python version: 3.6
- PyTorch version (GPU?): https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp37-cp37m-linux_x86_64.whl
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?:  No


Roberta's Positional Embedding Offset

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_bart.py#L754

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_bart.py#L763

So this offset is added because the function `create_position_ids_from_input_ids` shifts the position ids by padding_idx + 1. However, I wonder if other models should also include this?

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_roberta.py#L54
For instance, when I am using `Longformer`, it looks like the offset is not added to `Roberta`, so I need to add such a offset to config.max_position_embeddings


Tokenizer batch_encode_plus unexpected behavior

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
bert-base-multilingual-cased

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
A script to tokenize sentences into inputs for the model

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Standard text that is used for intent and domain classification

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=True)['input_ids']
>>>tensor([[  101, 61694, 10133, 15127, 11324, 10124, 14268,   102]])
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=False)['input_ids']
>>>tensor([[61694, 10133, 15127, 11324, 10124, 14268,     0,     0]])
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=True)['input_ids']
>>>tensor([[    0, 33600,    31,   759,  9351,    83,  3362,     2]])
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=False)['input_ids']
>>>tensor([[33600,    31,   759,  9351,    83,  3362,     1,     1]])
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

When running batch_encode_plus for a single example without adding special tokens and pad_to_max_length set to True, I would not expect any pad tokens. I did not see any mention in the documentation as to why this behavior is the expected norm.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: MacOS
- Python version: 3.7.3
- PyTorch version (GPU?): 1.5.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Gpt2 model card

model card

Related to pretrained model cards


Bart: Instatiate lm_head once without wasting memory

Marking this down since I agreed to do it https://github.com/huggingface/transformers/pull/4803#discussion_r443381438

cc @patrickvonplaten


Segmentation fault when trying to load models

We are using `Azure ML` pipelines to train our `transformers` models.  We have had it working for a few weeks, and then recently (just noticed it a few days ago), when trying to initialize a model, we are getting `Segmentation fault`.

I tried just loading the models locally this morning and have the same issues.  See snippet below.

```
config = config_class.from_pretrained(model_name, num_labels=10)
tokenizer = tokenizer_class.from_pretrained(model_name, do_lower_case=False)
model = model_class.from_pretrained("distilroberta-base", from_tf=False, config=config)
```

I also tried to download the `*_model.bin` and pass a *local path* instead of the model *name* and also got a `Segmentation fault`.  I also tried to use `bert-base-uncased` instead of `distilroberta-base` and had the same issue.

I am running on Ubuntu, with the following package versions:

```
torch==1.3.0
tokenizers=0.0.11
transformers==2.4.1
```

**UPDATE**:

I hacked some example scripts and had success, so I *think* the issue is that our code uses...

```
    "roberta": (RobertaConfig, RobertaForTokenClassification, RobertaTokenizer),
    "mroberta": (RobertaConfig, RobertaForMultiLabelTokenClassification, RobertaTokenizer),    # our custom multilabel class
```

instead of what the example scripts use...

```
    AutoConfig,
    AutoModelForTokenClassification,
    AutoTokenizer,
```

Was there a breaking change to model files recently that would mean that our use of the "non-auto" classes are no longer usable?

**UPDATE 2**:

Our original code does *not* cause a `Segmentation fault` on Windows.


Remove links for all docs

Now that someone has done an awesome version selector for the docs, there's no need to list all versions in the README.


Add DPR model

# Dense Passage Retrieval

## Intro

The Dense Passage Retrieval (DPR) model from facebook ([github](https://github.com/facebookresearch/DPR), [arxiv](https://arxiv.org/abs/2004.04906)).
It is used to do Open Domain Question Answering by extracting answer spans from a set of documents.
This model actually comes in three different parts:
- a context encoder
- a question encoder
- a reader for span prediction

I did a schema to show the roles and the pipeline that one could build with those parts.
The components in RED are the one in transformers.
You can use whatever you want for the retrieval part, but there will be a new [retrieval feature](https://github.com/huggingface/nlp/pull/298) in the 🤗nlp library soon that will make the use of models like DPR easier.
<img src="https://user-images.githubusercontent.com/42851186/85740169-b05da980-b701-11ea-975d-fbff4f368a5e.png" height="300">

## Implementation

- All three components share an encoding part with bert so I factorized this into the class `DprBertEncoder `.
- The reader has a `.generate` method that finds the best spans and return them, while the `.forward` method only returns the logits.
- In the config I allow to specify to load the weights from [files provided in the official repo](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py). I've already added one pretrained weight file per component in S3.

## Things I'd like to improve:

- I think we can probably improve the tokenization step. Right now the reader inputs are currently two sets of input_ids. One for the question with text_pair=context_title, and one for the context_text (i.e. the content in which we are looking for answer spans). This is because they all need to be combined like
```
[CLS] <question_input_ids> [SEP] <context_title_input_ids> [SEP] <context_text_input_ids>
```
I was thinking of making a custom tokenizer just for the reader, let me know if it sounds reasonable.

## Example of usage (outdated)

Provided we have a retrieval module (here named`wiki`) we can do:
```python
tokenizer = DprTokenizer.from_pretrained('dpr-model-base')
ctx_encoder = DprContextEncoder.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base')
question_encoder = DprQuestionEncoder.from_pretrained('facebook/dpr-question_encoder-single-nq-base')
reader = DprReader.from_pretrained('facebook/dpr-reader-single-nq-base')

# First step: retrieve the best wikipedia passages
question = 'Who created the Pokemon games ?'
question_emb = question_encoder(tokenizer.encode(question, return_tensors="pt")).numpy()
scores, passages = wiki.get_nearest_examples("embeddings", question_emb, k=10)
# Second step: Feed the reader with the question and the retrieved snippets
encoded_question_and_titles = [
    tokenizer.encode(question, text_pair=passage["title"], return_tensors="pt")
    for passage in passages]
encoded_texts = [
    tokenizer.encode(passage["text"], return_tensors="pt", add_special_tokens=False)
    for passage in passages]
predicted_spans = reader.generate(encoded_question_and_titles, encoded_texts)
# Last step: print the result
best_span = predicted_spans[0]
best_span_ids = encoded_texts[best_span.doc_id].numpy().flatten()
best_span_ids = best_span_ids[best_span.start_index:best_span.end_index + 1]
print(tokenizer.decode(best_span_ids))
# >>> satoshi tajiri
```
-----------------
I'd be very happy to have some feedbacks on this one, as it is my first contribution to the library :D


[dbart] push picture


can't open file 'transformers-cli'

when running
```!python transformers-cli convert --model_type xlnet```

I'm getting the following error :

```python: can't open file 'transformers-cli': [Errno 2] No such file or directory```

Any possible fix to this problem? thank you


Bert base model card

Related to pretrained model cards


Description of how to preprocess text corpus for roBERTa LM training

# 🚀 Feature request

There are different locations where you desribe how to train roBERTa models. For example here:
- https://huggingface.co/blog/how-to-train
- https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
- https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b

But nobody sais how to preprocess the text corpus. I think it must be one sentence per row. But does it need empty lines between documents? Is it ok to shuffle the text line by line?

Could you please claify this?


[examples/seq2seq] more README improvements


I have some problems with the "bert-large-uncased" model

This is the wrong code segment：

from transformers import BertTokenizer, BertModel, BertConfig
config = BertConfig.from_pretrained(args.bert_model, output_hidden_states = True, output_attentions = True)
tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=True, output_hidden_states=True)
bert = BertModel.from_pretrained(args.bert_model, config = config) # args.bert_model is 'bert-large-uncased'

And below is the bug report:

Traceback (most recent call last):
  File "main.py", line 119, in <module>
    main(args)
  File "main.py", line 64, in main
    net = BertProber(rel_vec_representation, args)
  File "/home/jzhao/program/interpret_bert/re/model.py", line 62, in __init__
    self.bert = BertModel.from_pretrained(args.bert_model, config = config)
  File "/home/jzhao/anaconda3/envs/python36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 466, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/jzhao/anaconda3/envs/python36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 615, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/jzhao/anaconda3/envs/python36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 149, in __init__
    self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=0)
  File "/home/jzhao/anaconda3/envs/python36/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 97, in __init__
    self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))
TypeError: new() received an invalid combination of arguments - got (str, int), but expected one of:
 * (torch.device device)
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, torch.device device)
      didn't match because some of the arguments have invalid types: (str, int)
 * (object data, torch.device device)
      didn't match because some of the arguments have invalid types: (str, int)


A question about the test accuracy of BERT-based-uncased model on the MNLI dataset

# 🐛 Bug
I am curious about how accuracy you guys can reach by using the BERT-based-uncased model on the MNLI task? I got 61% test accuracy on the MNLI-matched dataset and 62% accuracy on the MNLI-unmatched dataset.
## Information

Model I am using Bert-based-uncased:

Language I am using the model on English:

The problem arises when using:
* I use the exactly the same official example scripts: (give details below)
https://github.com/huggingface/transformers#quick-tour
https://github.com/huggingface/transformers#run_gluepy-fine-tuning-on-glue-tasks-for-sequence-classification

The tasks I am working on is:
* An official GLUE task: (MNLI)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: v2.11.0
- Platform: Linux
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0
- Using GPU in script?: 2 Tesla V100 GPUs
- Using distributed or parallel set-up in script?: No


BART finetune.py: model not learning anything

Examples

Which is related to examples in general

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
./examples/summarization/finetune.sh
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Finetuning on PubMed dataset.

## To reproduce

Steps to reproduce the behavior:

1. set DATA_DIR & OUT_DIR
2. Run command:
python finetune.py \
    --model_name_or_path=facebook/bart-large-cnn \
    --learning_rate=3e-5 \
    --gpus 1 \
    --do_predict \
    --do_train \
    --n_val 1000 \
    --val_check_interval 0.1 \
    --sortish_sampler \
    --max_target_length=80 \
    --val_max_target_length=200 \
    --test_max_target_length=200 \
    --max_source_length=850 \
    --train_batch_size=1 \
    --eval_batch_size=1 \
    --data_dir=$DATA_DIR \
    --output_dir=$OUT_DIR \
    --logger=wandb \
    $@
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The model is not learning anything. The generated eval summaries are identical throughout training and identical across training instances with different hyperparams. Appears as though backprop is not happening but there is not error message. (Probably I am missing something simple)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-5.0.0-37-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Create README.md

model card

Related to pretrained model cards

Create README.md for finance-koelectra-base-discriminator model


Fix LR decay in TF Trainer

Revival of #5051

I have accidentally deleted the previous branch so I recreated this same one.


Fix LR decay in TF Trainer


ValueError in T5 community colab notebook.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the community notebook https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb:
  * It raises subclassing error while defining `class T2TDataCollator(DataCollator)`. DataCollator is not a class anymore on the latest master branch, just `class T2TDataCollator()` fixes the error.
  * Even that bug is fixed, the notebook raises another ValueError while importing dataset.pt:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 231, in _start_fn
    fn(gindex, *args)
  File "<ipython-input-18-4f8aea5d9d8b>", line 191, in _mp_fn
    main()
  File "<ipython-input-18-4f8aea5d9d8b>", line 145, in main
    train_dataset  = torch.load(data_args.train_file_path)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 589, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 847, in _load
    result = unpickler.load()
  File "/usr/local/lib/python3.6/dist-packages/nlp/splits.py", line 493, in __setitem__
    raise ValueError("Cannot add elem. Use .add() instead.")
ValueError: Cannot add elem. Use .add() instead.
```

## To reproduce

Steps to reproduce the behavior:

1. Run the community colab notebook: https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb
2. You will get the error.

## Expected behavior
* Not applicable

## Environment info
* It's running on colab.


Finetune T5 on other Dataset, AssertionError: assert tokenized.input_ids.shape[1] == max_length

Core: Tokenization

Internals of the library; Tokenization.

# ❓ Questions & Help
## Details
I was trying to finetune Amazon Food Review Dataset on T5 using **Latest code from master**.

I formatted the data, as .source and .target.

When running [finetune_t5.sh](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py),

First it gives error :  **Keyword arguments {'add_prefix_space': True} not recognized.**

And then  : `File "/content/transformers/examples/summarization/utils.py", line 51, in encode_file
    assert tokenized.input_ids.shape[1] == max_length
AssertionError`

Could you please say, whether it is a bug or I am doing something wrong?


test_torch_fillmask failing on GPU

```bash
FAILED tests/test_modeling_bart.py::MBartIntegrationTests::test_enro_forward

        expected_slice = torch.tensor([9.0078, 10.1113, 14.4787], device=torch_device, dtype=model.dtype)
        result_slice = logits[0][0][:3]
>       self.assertTrue(torch.allclose(expected_slice, result_slice, atol=TOLERANCE))
E       AssertionError: False is not true

tests/test_modeling_bart.py:258: AssertionError

FAILED tests/test_pipelines.py::MonoColumnInputTestCase::test_torch_fill_mask_results

tests/test_pipelines.py:147: in _test_mono_column_pipeline
    set([o[key] for o in result]), set([o[key] for o in expect]),
E   AssertionError: Items in the first set but not the second:
E   '<s>My name is John</s>'
E   '<s>My name is Chris</s>'
E   Items in the second set but not the first:
E   '<s> My name is John</s>'
E   '<s> My name is:</s>'

```

cc @julien-c I'll figure it out.


Set the number of times to evaluate per epoch when using Trainer

Basically, I want to see how the training is going against the evaluation dataset twice per epoch when using `Trainer`. To calculate `logging_steps` so that `evaluate()` is called a certain number of times per epoch when training, will this hold-up to single and multi-device training?

```
evals_per_epoch = 2
logging_steps = len(train_dataset) / train_batch_size / gradient_accumulation_steps // evals_per_epoch
```

I'm getting myself confused over the implementation of `global_steps` and `logging_steps` inside of `Trainer.train`.


[examples] Verify marian and mbart BLEU scores with examples/seq2seq/run_eval.py

Examples

Which is related to examples in general

Got MBART 26.8 before postprocessing on wmt_en_ro,  37.1 after.
6 minutes. (first # same as fairseq, second should be also).
Marian: 27.7/37.4 90 Seconds


Cannot control wandb metadata when running examples/seq2seq/finetune.py

There is no easy way to control the wandb project_name kwarg.
How can we facilitate this without massive number of command line args?


[proposal] Move tests/utils.py to src/transformers/testing_utils.py so that examples can import

for both groups of tests, the import would be
```python
from transformers.testing_utils import slow
```

Motivation: I was about to rewrite the @slow decorator today and felt that this was cleaner.

Any objections? @julien-c @LysandreJik @thomwolf @sgugger @patrickvonplaten @mfuntowicz
@anyoneelseimforgetting


BertTokenizerFast does not support `pad_to_max_length` argument

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

The fast tokenizer has different behavior from the normal tokenizer.

```python
from transformers import BertTokenizer, BertTokenizerFast

BertTokenizer.from_pretrained("bert-base-uncased").encode("hello world", max_length=128, pad_to_max_length="right")
# succeeds
BertTokenizerFast.from_pretrained("bert-base-uncased").encode("hello world", max_length=128, pad_to_max_length="right")
*** TypeError: enable_padding() got an unexpected keyword argument 'max_length'
```

## Environment info

- `transformers` version: 2.11.0
- `tokenizers` version: 0.8.0rc3
- Platform: Ubuntu 18.04
- Python version: 3.7


Create README.md

model card

Related to pretrained model cards


save_pretrained: mkdir(exist_ok=True)

Old Logic:
if you pass something that is not a pre-existing directory -> Error

New Logic:
if you pass a file that exists -> Error.
if you pass a path that doesn't exist -> we call `mkdir path`, no error.
if you pass an existing directory -> no error.

This is not a breaking change, since no calls that previously succeeded produce different results.

Costs:
- you might occasionally make a directory called `pytorch_model.bin` for a confused user.
- a little bit of error checking code

Benefits:
- fewer late failures during training because you forgot to mkdir. This happens to me a lot.
- It feels like the spirit of the lib is to make NLP easier for the user, and I think this change is a small step in that direction.

Feedback much appreciated, I want to see if people like this before I add tests.


Tokenization tutorial

This takes most of the description in #4510 and organizes it as a tokenizer tutorial for section 2 of the documentation.

Preview is [here](https://52645-155220641-gh.circle-artifacts.com/0/docs/_build/html/preprocessing.html).


RobertaTokenizerFast produces a different output than RobertaTokenizer

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

`RobertaTokenizerFast.tokenize()` produces a different output than `RobertaTokenizer.tokenize()`. I am not sure if this is an issue that will impact model performance. Is this intended? I assumed the fast tokenizers should be consistent with the normal ones in terms of outputs.

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
from transformers import RobertaTokenizer, RobertaTokenizerFast

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
tokens = tokenizer.tokenize("This is a test. </s> <s> Another one. </s> <s> Yet another one.")
print("Normal Tokens: " + str(tokens))

ids = tokenizer.convert_tokens_to_ids(tokens)
print("Normal IDs: " + str(ids))

tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
tokens = tokenizer.tokenize("This is a test. </s> <s> Another one. </s> <s> Yet another one.")
print("Fast Tokens: " + str(tokens))

ids = tokenizer.convert_tokens_to_ids(tokens)
print("Fast IDs: " + str(ids))
```

Output:

```
Normal Tokens: ['This', 'Ġis', 'Ġa', 'Ġtest', '.', '</s>', '<s>', 'ĠAnother', 'Ġone', '.', '</s>', '<s>', 'ĠYet', 'Ġanother', 'Ġone', '.']
Normal IDs: [713, 16, 10, 1296, 4, 2, 0, 2044, 65, 4, 2, 0, 3507, 277, 65, 4]
Fast Tokens: ['ĠThis', 'Ġis', 'Ġa', 'Ġtest', '.', 'Ġ', '</s>', 'Ġ', '<s>', 'ĠAnother', 'Ġone', '.', 'Ġ', '</s>', 'Ġ', '<s>', 'ĠYet', 'Ġanother', 'Ġone', '.']
Fast IDs: [152, 16, 10, 1296, 4, 1437, 2, 1437, 0, 2044, 65, 4, 1437, 2, 1437, 0, 3507, 277, 65, 4]
```

Using `tokenizer.enocde()` instead of `tokenizer.convert_tokens_to_ids(tokenizer.tokenize())` solves the discrepancy with the first token but still inserts token id `1437` between `</s>` and `<s>`.

## Expected behavior

`RobertaTokenizerFast` produces the same output as `RobertaTokenizer`.

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Fix first test


Move GenerationMixin to separate file

This PR splits the `modeling_utils.py` and `modeling_tf_utils.py` by moving the code and methods related to generation to `modeling_generation.py` and `modeling_tf_generation.py` respectively.

Both of these files were getting pretty long, with the code dedicated to generation taking about 1000 LOC in each while being completely disjoint from the rest. This re-organization should make the code easier to read and contribute to.

There are no functional changes, I literally just created a new Mixin class for each and moved the functions as is.


Use master _static

Make all doc versions use the _static from master to make sure they all have the same version controller, link to hugging face logo etc.


[Tokenization] Fix #5181 - make #5155 more explicit - move back the default logging level in tests to WARNING

Padding to max sequence length while truncating to another length did not behave as expected on slow tokenizers as raised in #5181 by @sshleifer (it was truncating and then padding back to original length...)

This PR adds more tests to cover various combinations of padding + truncation strategies.

Fix #5181

This PR also:
- make #5155 clearer by changing the assertion in a cleaner error message (until the data processors are refactored)
- move back the default level of logging in tests to `logging.WARNING`
- switch some really slow tokenizations tests on CPU (when the inputs go in the full models) to `@slow` and speed-up testing by limiting the max sequence length used for testing


Fix version controller links (for realsies)

This time tested on pretty much all situations and add proper links as a result.

In particular:
- not sure that `location.toString()` will end with a '/' or not when on the index, this works with both
- when nested, use the previous value and not an absolute one
- the base url needs to be slice up to the version, including it for stable


Fix tensor label type inference in default collator

Quick fix to `default_data_collator` allowing it to recognize the correct type of PT tensor label inputs rather than always casting them to float (#5060)


Distilroberta Tokenizer and Encoder not aligning

Internals of the library; Tokenization.

I working on a sequence tagging task where I to extract cause and effect sub spans for a given text. For example, extract <e1> and <e2> for the text below.

`"<e2>The Sunshine State drew in a net influx of about $17.7 billion in adjusted gross income (AGI)  -  most of which (72 percent) came from those aged 55 and older.</e2> <e1>It is consistently one of the most popular destinations for retirees due to affordability and low taxes.</e1> Florida's $17.7 billion in net AGI dwarves the remaining 19 states that saw a positive net influx of income  -  which combined for a total of $19.4 billion."`

For the model, I'm try to generate a BIO style tags  that align with the tokenized input from the Distilroberta tokenizer. How I'm finding the after encoding the tokenized input, there is misalignment with the expected tags.

```
text = """The Sunshine State drew in a net influx of about $17.7 billion in adjusted gross income (AGI)  -  most of which (72 percent) came from those aged 55 and older. It is consistently one of the most popular destinations for retirees due to affordability and low taxes. Florida's $17.7 billion in net AGI dwarves the remaining 19 states that saw a positive net influx of income  -  which combined for a total of $19.4 billion."""

cause = 'It is consistently one of the most popular destinations for retirees due to affordability and low taxes.'
effect = 'The Sunshine State drew in a net influx of about $17.7 billion in adjusted gross income (AGI)  -  most of which (72 percent) came from those aged 55 and older.'

# Test that cause and effect are valid substring in text
print(text.find(cause)) # 160
print(text.find(effect)) # 0

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilroberta-base")

# Convert cause into Tags
cause_toks = tokenizer.tokenize(cause)
cause_tags = ["B-cause"] + ["I-cause"] * (len(cause_toks) -1)

# Convert effect into Tags
effect_toks = tokenizer.tokenize(effect)
effect_tags = ["B-cause"] + ["I-cause"] * (len(cause_toks) -1)

# Convert text tokinzed string
text_toks = tokenizer.tokenize(text)
text_toks_string = " ".join(text_toks)
text_toks_string = text_toks_string.replace(" ".join(cause_toks), " ".join(cause_tags))
text_toks_string = text_toks_string.replace(" ".join(effect_toks)," ".join(effect_tags))
text_toks = [tok if tok in ["B-cause", "I-cause", "B-effect", "I-effect"] else "O" for tok in text_toks_string.split()]

#['B-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'O', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']

print("text toks len: ", len(text_toks) + 2) # include start and end tokens
print("encoded text len: ", len(tokenizer.encode(text)))
# text toks len:  77
# encoded text len:  100
```

I also found I don't seem to get consistent behavior encoding a text and subspan seperately either.
```
text = "Cats are super coolio"
subtext = "super coolio"

print(tokenizer.encode(text))
# [0, 20913, 32, 2422, 3035, 1020, 2]

print(tokenizer.encode(subtext, add_special_tokens=False))`
# [16101, 3035, 1020]
```

Is my understanding on encode wrong? I thought encode converts the BPE tokens to numerical values and adds the cls and sep tokens according to the beginning and end. But it seems like something else is going on.


Fix links in version selector

This fixes the links provided by the version selector.


[WIP] Support label_smoothed_cross_entropy

By default there is no implementation of cross entropy with soft labels in Pytorch as discussed in #5168

I found a feature request in pytorch for it but it is still not done.

https://github.com/pytorch/pytorch/issues/7455

There is also discussion where i found an implemented version of this loss. I checked that it performs the same as nn.CrossEntropyLoss given smoothing=0.0 and accurately smoothes labels given nonzero smoothing. I ported it with small refactoring.

Key improvements that are planned in this PR:

1) Add label smoothing cross entropy loss
2) support and parametrise loss choice in `finetune.py`


Fix deploy doc

Try to update the master doc like the stable docs to see if this fixes the problem of things not being copied over.


[Benchmarks] improve Example Plotter

This PR make it possible to plot csv files that have "N/A" values.


Add some prints to debug deploy script


Don't recreate old docs

Change the check to look at a directory on the doc hosts instead of CircleCI to avoid creating old docs at each commit.


[Benchmark] fix print in benchmark

Tiny change pretty print results.


[Benchmark] Extend Benchmark to all model type extensions

This PR does the following changes:

1) - The default model class to benchmark is the one that can be found under config.architectures
2) - Improve plotting file


[WIP] Add 🤗nlp in examples using the updated tokenizer API

This PR superseded #4864

This PR examines how to best make use of all the features of 🤗nlp in the examples.

First example studied is GLUE. The main goal is to have explicit data processing (target: no data processing happening inside transformers) as well as add some efficiency features like dynamic/optimized batching. The second goal is to make this a lot more efficient, fast, and reproducible.


Multilingual MNLI model

Hi,

I'm trying Zero-Shot Learning  with 'facebook/bart-large-mnli' and it's pretty well, but I want to do it in Spanish. Is there any multilingual mnli model?

Thanks for your attention!


Not Implemented Error

TensorFlow

Anything TensorFlow

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): TFBert

Language I am using the model on (English, Chinese ...): Any

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## To reproduce
 I am trying to run the Bert model on a sequence of sentences.
Steps to reproduce the behavior:
```
import tensorflow as tf
from transformers import BertTokenizer, TFBertModel
inputs = tf.keras.Input(shape=(50, 64), dtype='int32')
model = TFBertModel.from_pretrained('bert-base-uncased')
outputs = tf.keras.layers.TimeDistributed(model)(inputs)

```
I get a not implemented error,
NotImplementedError                       Traceback (most recent call last)
<ipython-input-5-631f3cd2e8b2> in <module>
----> 1 outputs = tf.keras.layers.TimeDistributed(model)(inputs)

The same code works fine for
```
inputs = tf.keras.Input(shape=(10, 128, 128, 3))
conv_2d_layer = tf.keras.layers.Conv2D(64, (3, 3))
outputs = tf.keras.layers.TimeDistributed(conv_2d_layer)(inputs)
outputs.shape
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I should be able to run the bert model on the sequence of sentences.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.0.0
- Platform: Linux-5.0.0-1028-azure-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


BART(base) - Finetune Is this a bug ? Or I am doing something wrong?

## I have finetuned my dataset using latest piece of code from master, by cloning and the building.

Now when I try to load my finetuned model as shown in the examples of "summarization" (previous commit).

`from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('FinetuneOutput/best_tfmr')
tokenizer = BartTokenizer.from_pretrained('FinetuneOutput/best_tfmr')`

I got the following error :

`TypeError: special token mask_token has to be either str or AddedTokenFast but got: <class 'dict'>`

Am I missing something, or it is a BUG again ?


Model cards for Hate-speech-CNERG models

model card

Related to pretrained model cards

Made minor updates in previous model cards and added new cards for the newly updated models.


Does to T5 Transformer training scale to multiple GPUs?

Hello team,

  I have a large set of sequence to sequence dataset. Basically, a huge bunch of input text sequences to output text sequences. I want to train a T5 network on this. I have the following specific questions.

a. Can I use the sample code here (along with my own code) to train T5 on my data? - https://huggingface.co/transformers/model_doc/t5.html

b. Will that automatically scale to multiple GPUs? What if I want to further scale to tens of GPUs across different machines. Does HuggingFace support that?

Thanks


Fix model path


Fix PABEE division by zero error

Fix the `division_by_zero` error when `patience` is set to `0` during inference.

https://github.com/JetRunner/PABEE/issues/2