Pipeline for text generation

Hello,
* pipelines concise syntax and features are really nice, but there is none for text generation from left context
* `examples/run_generation.py` concise syntax (and some model-specific preprocessing) are really nice, but it is made for use via CLI and not from code

Any chance we see a text generation pipeline (optionally with some of the `run_generation.py` features) coming to 🤗 Transformers ?


[Config, Caching] Remove `output_past` everywhere and replace by `use_cache` argument

The `config.output_past` variable is removed and replaced by a function argument `use_cache`.
The reasons for this were explained in PR: https://github.com/huggingface/transformers/pull/3682

Affected models are:
T5, Bart, GPT2, XLNet, CTRL, TFGPT2, TFCTRL, TFXLNET

It is made sure that the change **does not break backwards compatibility** by setting `use_cache=True` by default since `config.output_past` was set to `True` by default before.
This can also be checked by seeing that none of the tests had to be changed (except T5's where the last PR: https://github.com/huggingface/transformers/pull/3682 for GPT2 and CTRL broke the backward compatibility of the default output length of T5)

I made the behavior of using the `past` variable the same in GPT2, T5, and CTRL. The logic is the following:
If the user decides to use `past` the `past` key-value states are cached and output.
The user can then optionally only input the last `input_ids` instead of all previous ones.
if the user decides to use `past`, the `last_hidden_states` output is reduced to only the last tensor instead of the same length as the `input_ids` (this is the same as before and cannot really be changed anyways because when caching keys and values earlier outputs cannot be calculated anymore and should not to improve speed).

It is made sure that if `use_cache` is False, nothing is cached! This means that a lot of memory can be saved when the user needs to be memory efficiency (this was not the case before).

All of this should be documented well in each of the models docstrings - let me know if something is badly documented and I'll change it :-)

Would be nice if you could take a look @sshleifer @thomwolf @LysandreJik @yjernite


How i take an OpenAIGPTDoubleHeadsModel from run_language_modeling.py script?

I train a gpt2 type model from scratch with run_language_modeling.py .
But i want to take an OpenAIGPTDoubleHeadsModel as my model.
My config.json is below. What i should change?

config = {
    "architectures": [
		"gpt2"
	],
    "model_name_or_path": None ,
    "model_type": "gpt2",
    "vocab_size":5000,
    "n_positions":1024,
    "n_ctx":1024,
    "n_embd":768,
    "n_layer":6,
    "n_head":12,
    "activation_function":'gelu_new',
    "resid_pdrop":0.1,
    "embd_pdrop":0.1,
    "attn_pdrop":0.1,
    "layer_norm_epsilon":1e-05,
    "initializer_range": 0.02,
    "summary_type":"cls_index",
    "summary_use_proj":True,
    "summary_activation":None,
    "summary_proj_to_labels":True,
    "summary_first_dropout":0.1,
    "bos_token_id":4999,
    "eos_token_id":4999,
}


Fine tuning XLMRoberta for Question Answering

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am trying to fine-tune XLM Roberta for SQuAD using the run_squad.py script available in examples. I have simply written an `XLMRobertaForQuestionAnswering` class as suggested at https://github.com/huggingface/transformers/issues/3694. However, the performance is extremely poor. I am wondering, do I need to perform anything special during preprocessing the SQuAD dataset when I use XLMRoberta?

I am using all the default hyper-parameters provided in the example script.


Loading pipeline("summarization") failed

Hi guys, when I try to load pipeline("summarization") I get the following error:

`Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "PATH", line 11, in <module>
    summarizer = pipeline("summarization")
  File "PATH\venv\lib\site-packages\transformers\pipelines.py", line 1626, in pipeline
    return task_class(model=model, tokenizer=tokenizer, modelcard=modelcard, framework=framework, task=task, **kwargs,)
  File "PATH\venv\lib\site-packages\transformers\pipelines.py", line 367, in __init__
    task_specific_params = self.model.config.task_specific_params
AttributeError: 'NoneType' object has no attribute 'config'
`

**Setup:**
Python: 3.7.6
transformers==2.8.0
tensorboard==2.0.2
tensorflow==2.0.0
tensorflow-estimator==2.0.1
tensorflow-hub==0.7.0

```
from transformers import pipeline
from transformers import TFAutoModelWithLMHead, AutoTokenizer

summarizer = pipeline("summarization")
```


# 🐛 Bug

Some previous issues #2954 described a memory leak when resuming training from a checkpoint.  I still get an OOM error when resuming training from a checkpoint.


exbert links for my albert model cards

model card

Related to pretrained model cards

Adding links for exbert visualization.


Checking that the LM actually trained

I have trained a gpt2 from scratch with the way that is decribed in that post https://huggingface.co/blog/how-to-train  .
Just in the step 4, where he checks if the trained model actually works, he uses from pipeline the
"fill-mask" but that works only for models with  masked language modeling objective.
Exists something similar i could use like "fill-mask" for my case?


ValueError: Cannot reshape a tensor - TFBertForSequenceClassification

I'm building a multiclass text classification model using Keras and BERT.

To convert my inputs to the required bert format, I'm using the `encode_plus` method found in the BertTokenizer class [found here][1]

The data is a paragraph of sentences per feature, and has a single label (of 45 labels in total)

**The code to convert the inputs is :**

    def create_input_array(df, tokenizer):
    sentences = df.text.values
    labels = df.label.values

    input_ids = []
    attention_masks = []
    token_type_ids = []

    # For every sentence...
    for sent in sentences:
        # `encode_plus` will:
        #   (1) Tokenize the sentence.
        #   (2) Prepend the `[CLS]` token to the start.
        #   (3) Append the `[SEP]` token to the end.
        #   (4) Map tokens to their IDs.
        #   (5) Pad or truncate the sentence to `max_length`
        #   (6) Create attention masks for [PAD] tokens.

        encoded_dict = tokenizer.encode_plus(
            sent,  # Sentence to encode.
            add_special_tokens=True,  # Add '[CLS]' and '[SEP]'
            max_length=128,  # Pad & truncate all sentences.
            pad_to_max_length=True,
            return_attention_mask=True,  # Construct attn. masks.
            return_tensors='tf',  # Return tf tensors.
        )

        # Add the encoded sentence to the list.
        input_ids.append(encoded_dict['input_ids'])

        # And its attention mask (simply differentiates padding from non-padding).
        attention_masks.append(encoded_dict['attention_mask'])

        token_type_ids.append(encoded_dict['token_type_ids'])

    return [np.asarray(input_ids, dtype=np.int32),
            np.asarray(attention_masks, dtype=np.int32),
            np.asarray(token_type_ids, dtype=np.int32)]

**The model in it's most basic form which still reproduces the error:**

    model = TFBertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels = labellen,
    output_attentions = False,
    output_hidden_states = False
    )

**Compile and fit:**

    optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3, epsilon=1e-08, clipnorm=1.0)
    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
    model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
    model.fit(x_train, y[:100], epochs=1, batch_size=3)

**The error when I run this :**

> ValueError: Cannot reshape a tensor with 768 elements to shape
> [1,1,128,1] (128 elements) for '{{node
> tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape}}
> = Reshape[T=DT_FLOAT, Tshape=DT_INT32](tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape/ReadVariableOp,
> tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape/shape)'
> with input shapes: [768], [4] and with input tensors computed as
> partial shapes: input[1] = [1,1,128,1].

I understand that BERT converts every token into a 768 value array, but that is the only knowledge I have of that particular number, so I'm stuck on how to proceed.

I would also appreciate your thoughts on whether TFBertForSequenceClassification is appropriate for paragraph classification.

Many thanks.

  [1]: https://huggingface.co/transformers/main_classes/tokenizer.html


Separate input_ids and decoder_input_ids in model.generate()

This makes the generation behavior more similar for sequence-to-sequence and language models, and allows us to initialize decoding with a prefix for the encoder-decoder setting.


Fine-tuning for paraphrasing tasks

Discussion

Discussion on a topic (keep it focused or open a new issue though)

# ❓ Questions & Help
I asked on SO and was downvoted since it is considered "off-site" and is against their Terms of Service. My question is somewhat simple. How do I fine-tune a GPT-2 model for the task of paraphrasing like the paper: https://www.aclweb.org/anthology/D19-5623.pdf

A link to my SO question : https://stackoverflow.com/questions/61115488/how-to-fine-tune-gpt-2-for-paraphrasing?noredirect=1#comment108120354_61115488

## Details
My question is somewhat simple. How do I fine-tune a GPT-2 model for the task of paraphrasing like the paper: https://www.aclweb.org/anthology/D19-5623.pdf

Is there a way to achieve this with huggingface-transformers ?


Has anyone used run_language_modelling.py to train a gpt 2 from scratch?

Ex: LM (Pretraining)

Related to language modeling pre-training

I have read this post https://huggingface.co/blog/how-to-train and i would like to do the train from scratch, for a gpt2 type model.
One question also, that i have, is the special tokens that he uses in that post for tokenizer will be the same and for a tokenizer that i will use for a gpt2 model?


How to get multiple answers from the context using BertForQuestionAnswering

How do I get multiple answers from the text using **BertForQuestionAnswering**, just like for the below question there are two possible answers:

1. a nice puppet
2. a software engineer

**Below is the code snippet for the same:**

```
from transformers import BertTokenizer, BertForQuestionAnswering
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet.Jim Henson was a software engineer."
input_ids = tokenizer.encode(question, text)
token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))

all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
answer = ' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1])
print(answer)

'a software engineer'
```

Thanks in advance!!


Integrate Bert-like model on Flax runtime.

This Pull Request attempts to bring support for [Flax](https://github.com/google/flax) framework as part of transformers.

Main focus as been put on providing BERT-like models, principally by making it possible to load PyTorch checkpoints and doing the necessary conversions (few) directly on the fly. Supports also providing a **msgpack** formatted file from Flax.

`save_pretrained` will save the model through **msgpack** format to avoid dependency on torch inside Jax code.

**Targeted models:**
- [x] Bert
- [x] RoBERTa
- [ ] DistilBERT
- [ ] DistilRoBERTa

**If not too hard**

- [ ] CamemBERT


DistributedSampler can't shuffle the dataset

# 🐛 Bug

## Information

I'm trying to fine-tune BERT model using ```run_language_modeling.py```.

Language I am using the model on is Persian:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

But according to this [issue](https://github.com/pytorch/pytorch/issues/31771) there is a bug in ```torch.utils.data.distributed.DistributedSampler``` so that during different epochs shuffling operation doesn't work properly(it's not working).
To solve this problem: according to pytorch official example [here](https://github.com/pytorch/examples/blob/ad775ace1b9db09146cdd0724ce9195f7f863fff/imagenet/main.py#L238), we should add ```train_sampler.set_epoch(epoch)``` before each new epoch at this [line](https://github.com/huggingface/transformers/blob/f8208fa456039b46873a2e497b6318d30a4fc84e/examples/run_language_modeling.py#L322)

## To reproduce

Steps to reproduce the behavior:

1. compare batches between different epoch like mentioned [issue](https://github.com/pytorch/pytorch/issues/31771)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: transformers==2.8.0
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): torch==1.4.0 (Yes)
- Tensorflow version (GPU?): tensorflow-gpu==2.1.0 (Yes)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: distributed


Disable @torch.no_grad() for model.generate() ?

Ex: Generation

Natural Language Generation

# ❓ Questions & Help

Is there any way to do this?


Unable to load german BERT model

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): German

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import *
model = TFBertModel.from_pretrained('bert-base-german-dbmdz-cased')
```

I get the following error trace -

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/daksh/miniconda3/envs/1.8/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 351, in from_pretrained
    assert os.path.isfile(resolved_archive_file), "Error retrieving file {}".format(resolved_archive_file)
  File "/Users/daksh/miniconda3/envs/1.8/lib/python3.6/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
```

## Expected behavior

Model should be loaded

## Environment info

- `transformers` version: 2.4.1
- Platform: Mac OS
- Python version: 3.6.5
- Tensorflow version (GPU?): 2.1.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


loading from tf_ckp and this showed up: AttributeError: 'BertCrf' object has no attribute 'bias' .


how to use transformers with gpu

I want to load a model with gpu in transformers, but it seem like the model always load in cpu
my os deepin 15.11 python 3.7.5 pytorch-gpu 1.4 transformers 2.8


Shift labels internally within TransfoXLLMHeadModel when called with labels

Fixes #3711 .


How can i conditional fine-tuning with GPT2?

I can use run_generation.py to create a statement by adding context.

But is there a way to do fine-tuning based on condition (context)?
For example, when data of "context [SEP] sentence" is input, the "context" is used to obtain the hidden state without learning.
In addition, the "sentence" is learned with the language model.


Zero shot multilingual BERT

❓ Questions & Help

I have a doubt about the usage of multilingual BERT.

I did a domain adaptation on the language model **[`BERT-Base, Multilingual Cased`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)** with a dataset of a slightly different kind of text. This dataset is unbalanced on English language but contains also other languages as Italian.

Then I did a finetuning on an Italian dataset on NER.

Has this kind of training a name, like **zero shot** classification (beacuse I adapted on a multilingual dataset but unbalanced on English and then we finetune on Italian)?
Or maybe the 0 shot would be in the case in which I finetuned with an Italian dataset and then I evaluated on a corpus of another language?

Thanks


cannot determine what will be the cardinality of the output after applying glue_convert_examples_to_features [TF  2.2.0rcx]

# 🐛 Bug

## Information
With Tensorflow 2.2.0 (2.2.0rc2) we should be able to see the number of entries in the data without looking over them and using tf.data.experimental.cardinality.

One issue that I found is that after applying  `glue_convert_examples_to_features` tf.data.experimental.cardinality is not able to find the total number of entry. I thought first that it was bug in this TF 2.2.0 release candidate.https://github.com/tensorflow/tensorflow/issues/37998.

When using data from tensorflow dataset tf.data.experimental.cardinality is returning the number of event

```
print(data['train'])
print(tf.data.experimental.cardinality(data['train']))
```

```
<DatasetV1Adapter shapes: {idx: (), label: (), sentence: ()}, types: {idx: tf.int32, label: tf.int64, sentence: tf.string}>
tf.Tensor(67349, shape=(), dtype=int64)
```

Now when I am using Huggingface transformer that modify the structure of the data:
```
train_dataset = glue_convert_examples_to_features(data['train'],
                                                  tokenizer,
                                                  max_length=128,
                                                  task='sst-2')

print(tf.data.experimental.cardinality(train_dataset))
```

```
<FlatMapDataset shapes: ({input_ids: (None,), attention_mask: (None,), token_type_ids: (None,)}, ()), types: ({input_ids: tf.int32, attention_mask: tf.int32, token_type_ids: tf.int32}, tf.int64)>
tf.Tensor(-2, shape=(), dtype=int64)
```

When the input pipeline contains a flat_map, it is generally not possible to statically determine what will be the cardinality of the output from the cardinality from the input. I don't see any flatmap in this function. I am trying to identify which part of the code is responsible. I am not 100% sure this is a transformer issue.

## To reproduce

Steps to reproduce the behavior:

```
import tensorflow as tf
import tensorflow_datasets

from transformers import (
    BertConfig,
    BertTokenizer,
    TFBertModel,
    TFBertForSequenceClassification,
    glue_convert_examples_to_features,
    glue_processors
)

data, info = tensorflow_datasets.load(name='glue/sst2',
                                      data_dir='/tmp/',
                                      with_info=True)

pretrained_weights = 'bert-base-multilingual-uncased'

# Load tokenizer
tokenizer = BertTokenizer.from_pretrained(pretrained_weights)

# recap of input dataset
print(data['train'])
print(tf.data.experimental.cardinality(data['train']))

# Prepare data for BERT
train_dataset = glue_convert_examples_to_features(data['train'],
                                                  tokenizer,
                                                  max_length=128,
                                                  task='sst-2')

# recap of pre processing dataset
print(train_dataset)
print(tf.data.experimental.cardinality(train_dataset))

```
## Expected behavior
I am expecting tf.data.experimental.cardinality to still be able to report the total number of entries after transforming the data with `glue_convert_examples_to_features`

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: MacOS 0.14.6
- Python version: 3.7.5
- Tensorflow version (CPU): 2.2.0rc2 (v2.2.0-rc1-34-ge6e5d6df2a 2.2.0-rc2)


Text Generation with XLNet is very Slow

Natural Language Generation

Using the run_generation script to generate text with XLNet is currently extremely slow compared to GPT-2 as mentioned in [this issue](https://github.com/huggingface/transformers/issues/789)

> To generate 100 tokens, XLNet takes **3m22s** while GPT-2 takes **14s**. And it grows exponentially : for 500 tokens, XLNet takes **51m46s** while GPT-2 takes **2m52s**.

More information on why this might be happening is included in the issue. However, it was closed before it could be resolved. If anyone could look into this and maybe reopen the original issue, that would be greatly appreciated.
Thanks!


TransfoXLLMHead doesn't shift labels internally when called for loss

# 🐛 Bug

When called with labels to get the language-modeling loss, `TransfoXLLMHead.forward` computes the NLLLoss of the outputs directly against the labels, rather than against the shifted labels like the documentation indicates (and like the other models). This makes it impossible to train with `lm_labels = input_ids` as suggested by the doc.

## Information

Model I am using: TransformerXL

Language I am using the model on: English

The problem arises when using:
* [x] my own modified scripts:

The task I am working on is:
* [x] my own task or dataset:

## To reproduce

```
import torch
from transformers import TransfoXLConfig, TransfoXLLMHeadModel

config = TransfoXLConfig()
lm = TransfoXLLMHeadModel(config)
test_tensor = torch.LongTensor([[0]])
print(lm(input_ids=test_tensor, labels=test_tensor)[0])
```

A 1x1 loss tensor is returned.

## Expected behavior

As there is only 1 token in the input tensor, no loss should be returned: there's no next label to compare the output against. For example, running this with GPT2

```
import torch
from transformers import GPT2Config, GPT2LMHeadModel

config = GPT2Config()
lm = GPT2LMHeadModel(config)
test_tensor = torch.LongTensor([[0]])
print(lm(input_ids=test_tensor, labels=test_tensor)[0])
```

returns `tensor(nan, grad_fn=<NllLossBackward>)`.

## Environment info

- `transformers` version: 2.8.0
- Platform: Linux-5.3.0-45-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False


inconsistent tokenize output

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): Chinese

The problem arises when using: BertTokenizer

I am using the BertModel. When predicting the result, it got inconsistent output (the output differs from time to time). It turns out that the tokenizer gets different output when the input string is "\n" or "\n\n".

for example:

from transformers.tokenization_bert import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
tokens = tokenizer.tokenize("\n\n")
output: ['[SEP]'] or ['[PAD]'] or ['[CLS]']


Add model tag

model card

Related to pretrained model cards

Add model tag to be correctly indexed while working on card description


Add model tag

Related to pretrained model cards

Add model tag to be correctly indexed while working on description


Distributed training on multiple GPU nodes is slower than on single GPU node

Our team uses pre-trained model and transformers has been a great help.

While trying to benchmark the training speed of our computing infrastructure, we were running this example:
https://github.com/huggingface/transformers/blob/v2.3.0/examples/run_lm_finetuning.py

We found that on a single p3.16xlarge GPU instance, DDP would took about 36:33 to train wikitext-103-raw.
However, once we move on to two p3.16xlarge GPU instance, it would take more time (1:45:49) for the same dataset.

I would like to know what could possibly cause this?

There are something i am suspecting:
1. The synchronization of gradiants/parameters between different process
2. The optimizer that has been used in this scripts.

Any help is appreciated. Thanks!


Cleanup fast tokenizers integration

First PR to clean up the fast tokenizers integration.


Update tokenizers to 0.7.0-rc5

This includes some bug fix (around added tokens), and a small breaking change.


Queries about the Notation and Model training of T5 and ELECTRA sentiment classification.

I have a few questions about the model notation. And also short info about T5 and ELECTRA. I would like to make separate issues but things are not too complex. I mainly working on CV, so sorry if I being so silly.

### 1 Cased or Uncased
What is mean by cased and uncased?
```
bert-base-uncased
bert-base-cased
```

### 2 Suffix

I was trying to run the XLM model but in the pre-train model, I've found the following weights, I understood about XML-MLM but couldn't get the rest of the part, ex: `enfr-1024, enro-1024` etc.

```
xlm-mlm-enfr-1024
xlm-mlm-enro-1024
xlm-mlm-tlm-xnli15-1024
```

### 3 Sentiment Analysis using T5 and ELECTRA

Is it possible to use these two models for sentiment classification, simply just a binary classification? How can we implement these two transformers? I have a high-level overview of T5, it transforms both (input/target) as a text. I [found it](https://github.com/google-research/text-to-text-transfer-transformer/issues/109) useful though but bit trouble to implement. Using transformers, is it possible to go within a convenient way?


Token-level regression mode added in ForTokenClassification models

This is related to the issue #3646 I opened 2 days ago and which was considered interesting by @julien-c

I added the **support for token-level regression in Bert, Roberta, Albert, XLNet, XLM, DistilBert and in the template for adding a new model** when `self.num_labels == 1`, fixing the docstring to match the new changes (and correcting the one for XLNetForTokenClassification which was copied from the XLNForMultipleChoice one).

Given two different approaches to compute `active_labels`, I privileged the one used in more recent models (e.g. Albert). The change was tested on tests and examples (without RUN_SLOW and RUN_CUSTOM_TOKENIZERS though) and passed all tests.

I didn't felt like adding this to Electra since its TokenClassification implementation seems to be missing the `num_labels` variable for some reason. I didn't add new tests since the sentence regression case wasn't covered by the testing suite either.

**Edit:** After the fix on `ElectraForTokenClassification`'s `num_labels` attribute, I added support for token-level regression there too.


Add `run_glue_tpu.py` that trains models on TPUs


Problem with https://transformer.huggingface.co/doc/gpt2-xl

Hi, when using it, simple switch the button up to xl one. When you try to generate a prediction of words; it will simply never load regardless of what all the other setting are on. Every other gpt2 model size works; expect the xl one. Tried it on Android and Linux same problem and different browsers Firefox and Google Chrome same problem nothing to with addons cause the same problem occurs with them all off.


Would the weights for the main body of the pertained GPT2Model and pertained GPT2DoubleHeadsModel be identical?

Hello,

From my understanding, the difference between the GPT2Model and GPT2DoubleHeadsModel is that GPT2Model does not include any output head, whereas GPT2DoubleHeadsModel includes two types of output heads (lmhead and mchead).

I am wondering, would the weights used in the pertained GPT2Model identical to the weights used in the main body (every parts of the model except the output heads) of the pertained GPT2DoubleHeadsModel? or would the two sets of weights be different since GPT2DoubleHeadsModel was trained after including the output heads, whereas GPT2Model was trained without any output head?

Thank you (I hope my question is understandable),


Bug in ElectraForTokenClassification

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ElectraForTokenClassification

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [√] the official example scripts: (give details below)
  huggingface.co/transformers/model_doc/electra.html#electrafortokenclassification
* [×] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ×] an official GLUE/SQUaD task: (give the name)
* [ √] my own task or dataset: (give details below)
  Ran the example given in the documentation

        from transformers import ElectraTokenizer, ElectraForTokenClassification
        import torch

        tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-discriminator')
        model = ElectraForTokenClassification.from_pretrained('google/electra-small-discriminator')

        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute",
        add_special_tokens=True)).unsqueeze(0)  # Batch size 1
        labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids, labels=labels)

        loss, scores = outputs[:2]
## To reproduce

Steps to reproduce the behavior:

1. Start Google Colab
2. Install Transformer and  requriements
3. run the code

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
AttributeError: 'ElectraForTokenClassification' object has no attribute 'num_labels'
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Windows 10
- Python version: 3.7.6
- PyTorch version (GPU?): Colab
- Tensorflow version (GPU?): Colab
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


More doc for model cards

see https://github.com/huggingface/transformers/pull/3679#pullrequestreview-389368270


Fix force_download of files on Windows

On Windows, `os.rename` gives the error
```
FileExistsError: [WinError 183] Cannot create a file when that file already exists:
```
when trying to re-download a model that already exists in cache using `force_download=True`.


Can not set different token and model dir in `run_glue.py`

# 🐛 Bug

## Information
the original code as follows, when i use args.tokenizer_name different from model_name_or_path, it still call for the model_name_or_path config file.
tokenizer = AutoTokenizer.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
        do_lower_case=args.do_lower_case,
        cache_dir=args.cache_dir if args.cache_dir else None,
)

It would be solved  by adding config params as follows:
tokenizer = AutoTokenizer.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
        do_lower_case=args.do_lower_case,
        config=config,
        cache_dir=args.cache_dir if args.cache_dir else None,
   )


Deserialize BERT Sequence Lassifier Quantized Model & Inferencing Issue

Dynamic Quantization:  Loading Quantized Model issue here:

Currently am looking at below collab link for quantization, where all Linear layers are quantized in corresponding BERTForSequence Classification.

Step:1,
Serialized Quantized Model

Step 2:
Deserialized Using below code:

Screenshot#1:
Linear layers of the encoder are converted to DynamicQuantizedLinear Layers just after conversion
Once loaded serialized quantized model for future use couldn't show up DynamicQuantizedLinear layers rather Linear layers for the query, key, values as shown in screenshots.

Also, inference these deserialized models shows error predictions as shown here, so someone guide me how to deserialize quantized models as it is taking hell lot of time to figure out the same.

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb?authuser=1#scrollTo=dUJ1NGinLAa1

## Information


Extending XLM Roberta for Question Answering

# ❓ Questions & Help

The AutoModelForQuestionAnswering is supported by many models, but not yet by XLM Roberta. In the current implementation I could see that most task-specific classes for XLM-R, e.g. XLMRobertaForSequenceClassification are just inheriting from Roberta. However, when I try to extend the class analogously, the process fails. This is my extension:

`from transformers.modeling_roberta import (
    RobertaForQuestionAnswering,
)`

and

`@add_start_docstrings(
    XLM_ROBERTA_START_DOCSTRING,
)`

`class XLMRobertaForQuestionAnswering(RobertaForQuestionAnswering):
    config_class = XLMRobertaConfig
    pretrained_model_archive_map = XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
`

The error message I get is

> Traceback (most recent call last):
  File "..miniconda3/lib/python3.6/site-packages/torch/serialization.py", line 289, in _check_seekable
    f.seek(f.tell())
AttributeError: 'NoneType' object has no attribute 'seek'

> During handling of the above exception, another exception occurred:

> Traceback (most recent call last):
  File "../miniconda3/lib/python3.6/site-packages/transformers/modeling_utils.py", line 516, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "../miniconda3/lib/python3.6/site-packages/torch/serialization.py", line 525, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "../miniconda3/lib/python3.6/site-packages/torch/serialization.py", line 217, in _open_file_like
    return _open_buffer_reader(name_or_buffer)
  File "../miniconda3/lib/python3.6/site-packages/torch/serialization.py", line 202, in __init__
    _check_seekable(buffer)
  File "../miniconda3/lib/python3.6/site-packages/torch/serialization.py", line 292, in _check_seekable
    raise_err_msg(["seek", "tell"], e)
  File "/home/anlausch/miniconda3/lib/python3.6/site-packages/torch/serialization.py", line 285, in raise_err_msg
    raise type(e)(msg)
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

>  File "../modeling_auto.py", line 968, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "..miniconda3/lib/python3.6/site-packages/transformers/modeling_utils.py", line 519, in from_pretrained
    "Unable to load weights from pytorch checkpoint file. "

Any idea what's going on here?


How can I pinpoints Logs directory to Google Drive while finetuning GPT-2 model, which helps in visualizing data via tensorboard?

Hi,

I am facing an issue in retrieving logs, while fine-tuning GPT-2 model by using [Google Colab](https://colab.research.google.com/github/interactive-fiction-class/interactive-fiction-class.github.io/blob/master/homeworks/language-model/hw4_transformer.ipynb).

As the finetunining takes several hours, Google Colab halts the running process at certain point in time, even there are remaining epochs left behind.

In that case, I can be able to successfully continue my finetuning by including a parameter "should_continue" while running the script using [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py).

However, my logs vanished, and I could not be to retrieve data of tensorbaord, which has been run with these commands

```
%load_ext tensorboard
%tensorboard --logdir=runs
```

**Is there a way to pin point my logs to the Google Drive, so that I can retrieve and visualize them with tensor-board at any point in time?**


How to use Huggingface pytorch bert to generate the prediction TSV file from the test set of a GLUE task?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello folks! Can you provide some simple example of using pytorch bert to generate the prediction TSV file from the test set of a GLUE task (such as MRPC) based on the fine-tuned model so that I can submit the prediction TSV file for each GLUE task to GLUE leaderboard? Thank you very much.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


cannot import name AddedToken

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Albert

Language I am using the model on (English, Chinese ...): English

     27 from typing import List, Optional, Sequence, Tuple, Union
     28
---> 29 from tokenizers import AddedToken, Encoding
     30 from tokenizers.decoders import Decoder
     31 from tokenizers.implementations import BaseTokenizer

ImportError: cannot import name 'AddedToken'

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name):
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


BertSelfAttention have not Add and Norm layer, why???


Can't update the train_batch_size and eval_batch_size for the training image in a docker container

Hi,

I tried to train a couple multi-label models with the fast-bert libary using the container files to build up the docker image and uploaded to AWS ECR and used the aws helper notebook that's included in the 'sample notebook' folder in the repo. I have trained for 3 models and **regardless I changed to a different train_batch_size in the hyperparameters.json file, the model when training still outputs total train batch size is 64 and eval batch size is 128.**

**My question here:**

-  Am i not able to update the train batch size?  if the training is happening in a container?

-  Does the training and eval batch size have some relationship? from a glance, it looks like eval_batch_size is doubled train_batch_size. I'm gonna say there shouldn't be any relationship. However, **why there's no parameter set in the hyperparameters.json to specify the eval_batch_size?**

-  The three models i have trained all got really good accuracy_thresh with above 0.97. **However, one of the models only outputs 2 classes as the top probability class.** The original data has about 9455 rows and 113 classes. I have also trained it on BERT tensorflow version, i was able to get multiple labels as top predicted class. **what could be possibly wrong?** Note that my other 2 models has about 36 classes and 11 classes. The top predicted classes all came out reasonable, meaning all the 36 and 11 classes showed up for the top predicted class.  In addition, i don't see the performance change whenever i changed the accuracy_thresh change after epoch 2.

**Please provide some guidance as this is going into deployment soon** but I'm still struggling to figure out why....


Big cleanup of `glue_convert_examples_to_features`


Is it possible to use multiprocessing for pipelines?

I am trying to use multiprocessing for pipelines, but it seems that it's not working. I think it's because the pipeline already uses multiprocessing features and so you can't have multiprocessing inception. Anyone able to get it to work?


Bug in variable name in NER

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Irrelevant

Language I am using the model on (English, Chinese ...): Irrelevant

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: NER
* [ ] my own task or dataset: (give details below)

## Error

```
File "/transformers/examples/ner/utils_ner.py", line 123, in convert_examples_to_features special_tokens_count = tokenizer.num_added_tokens()
AttributeError: 'BertTokenizer' object has no attribute 'num_added_tokens'
```

## Issue

After update to new Tokenizers, some util files are broken. Found one in examples/ner/utils_ner.py.
Need to change line 123 from num_added_tokens to num_special_tokens_to_add.


Requesting model for TFAlbertForQuestionAnswering

# 🌟 New model addition

Is there support for adding a TensorFlow version of the AlbertForQuestionAnswering model? I would be happy to contribute the work. This would also enable the `run_squad.py` example script for TensorFlow.

It also looks like [the preprocessing of SQuAD data is different](https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/squad.py#L349) for PyTorch and TensorFlow. PyTorch has an option to return `all_example_index`, while TensorFlow does not. This means that running the model evaluation script (which takes the max F1/EM over all answers) is only possible in PT. Running SQuAD evaluation in TensorFlow is important for my use case, and I would love to understand the design decisions or difficulties that went into that decision. Again, happy to contribute - any support or objections for aligning the data preprocessing techniques?


Updating the TensorFlow models to work as expected with tokenizers v3.0.0

Models and tokenizers should work in harmony; this is why it is an API design choice to be able to send the output of `encode_plus` and `batch_encode_plus` straight to the model, in both PyTorch and TensorFlow:

```py
encoded_sequence = tokenizer.encode_plus(sequence)
model(encoded_sequence)  # for TensorFlow
model(**encoded_sequence)  # for PyTorch
```

With the recent changes of tokenizers-v3.0.0 and the introduction of `BatchEncoding`, the way the TensorFlow models usually identified such inputs didn't work, as it was looking for a `dict` instead of a `BatchEncoding`. This PR patches this.

This feature was previously untested; this PR addresses this by adding four different tests on each tokenizers; testing that the tokenizers return correct `BatchEncoding` objects in both PyTorch and TensorFlow which can be fed directly to the model. Both `encode_plus` and `batch_encode_plus` are tested.

Some issues were found, and were patched with this PR.


question-answering pipeline error : too many values to unpack (expected 2)

Core: Pipeline

Internals of the library; Pipeline.

For some `question-answering` models the pipleline encounters extra tuples from the `.model()` call, where we must have exactly 2.

> This stub results in the following error:
```python
from transformers.pipelines import pipeline

model_name = "mrm8488/bert-uncased-finetuned-qnli"

nlp = pipeline("question-answering", model=model_name, tokenizer=model_name)
QA_input = {
    "question": "Why is model conversion important?",
    "context": "The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.",
}
res = nlp(QA_input)
```

> error message
```bash
Traceback (most recent call last):
  File "test3.py", line 10, in <module>
    res = nlp(QA_input)
  File "/opt/conda/lib/python3.7/site-packages/transformers/pipelines.py", line 1010, in __call__
    start, end = self.model(**fw_args)
ValueError: too many values to unpack (expected 2)
```

> env
```text
- `transformers` version: 2.8.0
- Platform: Linux-4.9.184-linuxkit-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: nope
- Using distributed or parallel set-up in script?: nope
```


[T5, generation] Add decoder caching for T5

This PR greatly speeds up the autoregressive decoding for T5 by storing past key / value states.
The summarization test: https://github.com/huggingface/transformers/blob/500aa12318ce5acd289d5edb6cb8266b3c3b162e/tests/test_modeling_t5.py#L260 now takes only 44s whereas before it took 311s -> 7.5x Speed up

This will also significantly speed up the translation and summarization pipelines when using T5.

- [x] Add key value state caching
- [x] Test for equal output on hard-coded tests
- [x] Add simple past tests including using an attention mask
- [x] update the docstring
- [x] clean up code

The caching design was already somewhat outcommented in place. It was cleaned, made functional and implemented very similar to GPT2's one.

### IMPORTANT:
This PR has a breaking change, in that it increases the default output length of T5Model and T5ForConditionalGeneration from 4 to 5 (including the `past_key_value_states`).

### Future PR:
- [ ] Do the same for TF if this PR is accepted.

Would be nice if you could take a look @craffel @thomwolf @LysandreJik @sshleifer


updating to new transformer


How to use GPT2DoubleHeadsModel?

Hello,

I have a question about the example shown in the GPT2DoubleHeadsModel documentation page:
https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel

In the example, the input to the GPT2DoubleHeadsModel is simply a set of choices. But what if the multiple choice question that I want to process also includes a question text? so for example,

Bob likes candy; what does Bob like?
a. Bag
b. Burger
c. Candy
d. Pencil

In the example above, the question text would be " Bob likes candy; what does Bob like?"
and the choices will be the Bag, Burger, Candy and Pencil. How should I pre-process my multiple choice questions to be used with the GPT2DoubleHeadsModel?

For example, given that the token that will be used for the classification is "<|endoftext|>" (this is the default eos token for the GPT2 models), would the following be fine?

```python
import torch
from transformers import GPT2Tokenizer, GPT2DoubleHeadsModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2DoubleHeadsModel.from_pretrained('gpt2')

choices = [ "Bob likes candy ; what does Bob like ?  Bag <|endoftext|>",
                   "Bob likes candy ; what does Bob like ?  Burger <|endoftext|>",
                   "Bob likes candy ; what does Bob like ?  Candy <|endoftext|>",
                  "Bob likes candy ; what does Bob like ?  Pencil <|endoftext|>"]

encoded_choices = [tokenizer.encode(s) for s in choices]
eos_token_location = [tokens.index(tokenizer.eos_token_id) for tokens in encoded_choices]

input_ids = torch.tensor(encoded_choices).unsqueeze(0)
mc_token_ids = torch.tensor([eos_token_location])
outputs = model(input_ids, mc_token_ids=mc_token_ids)
lm_prediction_scores, mc_prediction_scores = outputs[:2]
```

Thank you,


Update doc for {Summarization,Translation}Pipeline and other tweaks


run_generation.py with empty input

Ex: Generation

Natural Language Generation

Hi,

I would like to generate text following some context but also from scratch as seen in *Write with Transformer*. Using both
* `python run_generation_callable.py --model_type=gpt2 --model_name_or_path=gpt2` and feeding an empty prompt to the `Model prompt >>> `
* or changing the line
`prompt_text = args.prompt if args.prompt else input("Model prompt >>> ")`
in the source to
`prompt_text = args.prompt`
and using
`python run_generation_callable.py --model_type=gpt2 --model_name_or_path=gpt2 --prompt ""`

result in a `RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous` (tell me if you need the complete stack trace or more detail for the reproduction).

Thanks by advance for any tips, workarounds or directions on achieving this!


Does anyone have the XLNet (and ALBERT) NER performance on CONLL-2003

# ❓ Questions & Help
Most transformer models in the library can be fine-tuned for NER tasks.

In https://huggingface.co/transformers/v2.2.0/examples.html#named-entity-recognition, the performances of the roberta, bert, and distilbert have been reported.

However, I did not find performances achieved by other models like XLNet.

By any chance, does anyone experiment with other models and can report the performances for models like XLNet and albert on CONLL-2003?


gpt2-medium fine-tuned model.generate joins words and sentences together without space or newline

Hi,

I have successfully fine-tuned and used a gpt2 model to generate text. My training corpus consist of short sentences - 3-5 words and longer ones 10-15 words. All separated by new line character. Sometimes ending with [ . ! ? ] sometimes not

`outputs = model.generate(
            input_ids=input_ids, max_length=max_length, temperature=temperature,
            repetition_penalty=repetition_penalty,
            bos_token_id=tokenizer.bos_token_id,
            top_k=top_k,
            top_p=top_p
        )`

`ret = tokenizer.decode(outputs[0], skip_special_tokens=True)`

Then I fine-tuned a gpt2-medium model. The training corpus was slightly different, but structured the same as described above.

I had to use --fp16 and --block_size=512 to fit in the GPU memory limits.

The result:
using the fine-tuned a gpt2-medium model, I am experiencing a couple of issues:

1. I get frequent issues with lines or words stuck, without any new line or space:
example:
**word1Word2Word3**
or:
**line 1 with some words!Another line with some words™Next line...**

2. I get a 'warning':
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

I've tried playing with the decode parameters with no luck:
`ret = tokenizer.decode(outputs[0], skip_special_tokens=False, clean_up_tokenization_spaces=False)`

Help appreciated,

thanks in advance,
Albert


Wrong tokenizer configuration in sentiment-analysis pipeline

Core: Pipeline

Internals of the library; Pipeline.

# 🐛 Bug

## Information

When following the Pipelines Notebook 03-pipelines.ipynb, Sentiment Analysis tasks gives wrong result ("NEGATIVE") for example 'Such a nice weather outside !'.
```
nlp_sentence_classif = pipeline('sentiment-analysis')
nlp_sentence_classif('Such a nice weather outside !')
[{'label': 'NEGATIVE', 'score': 0.97545063}]
```

Probable reason: pipelines.py configuration uses uncased model, but cased tokenizer. Tokenizer should probably be 'distilbert-base-uncased'.

```
"sentiment-analysis": {
        "impl": TextClassificationPipeline,
        "tf": TFAutoModelForSequenceClassification if is_tf_available() else None,
        "pt": AutoModelForSequenceClassification if is_torch_available() else None,
        "default": {
            "model": {
                "pt": "distilbert-base-uncased-finetuned-sst-2-english",
                "tf": "distilbert-base-uncased-finetuned-sst-2-english",
            },
            "config": "distilbert-base-uncased-finetuned-sst-2-english",
            "tokenizer": "distilbert-base-cased",
        },
    },
```

Model I am using (Bert, XLNet ...): distilbert-base-uncased-finetuned-sst-2-english (preconfigured sentiment-analysis pipeline)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: 03-pipelines.ipynb
* [ ] my own modified scripts: (give details below)

## Expected behavior

Example sentence should be labeled as POSITIVE.

## Environment info

- `transformers` version: 2.8.0
- Platform: Linux Mint
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0 (no)
- Tensorflow version (GPU?): 2.1.0 (no)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


[Examples, Benchmark] Improve benchmark utils

This PR improves the `benchmarks.py` file a bit:
- "results, memory" are renamed to "time, memory"
- all print statements can optionally be saved in a log file
- the CSV file output format is improved
- better naming in general


TypeError while loading the model built from scratch using transformer

# 🐛 Bug

> TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType`

 ---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)
<ipython-input-33-cd040b700e71> in <module>()
      3 from transformers import BertTokenizer, AdamW, BertForNextSentencePrediction
      4
----> 5 tokenizer = BertTokenizer.from_pretrained('/content/drive/My Drive/Colab Notebooks/data/test/')

3 frames
/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    391
    392         """
--> 393         return cls._from_pretrained(*inputs, **kwargs)
    394
    395     @classmethod

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
    542         # Instantiate tokenizer.
    543         try:
--> 544             tokenizer = cls(*init_inputs, **init_kwargs)
    545         except OSError:
    546             raise OSError(

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_bert.py in __init__(self, vocab_file, do_lower_case, do_basic_tokenize, never_split, unk_token, sep_token, pad_token, cls_token, mask_token, tokenize_chinese_chars, **kwargs)
    186         self.max_len_sentences_pair = self.max_len - 3  # take into account special tokens
    187
--> 188         if not os.path.isfile(vocab_file):
    189             raise ValueError(
    190                 "Can't find a vocabulary file at path '{}'. To load the vocabulary from a Google pretrained "

/usr/lib/python3.6/genericpath.py in isfile(path)
     28     """Test whether a path is a regular file"""
     29     try:
---> 30         st = os.stat(path)
     31     except OSError:
     32         return False

TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

## Information
I am trying to fine-tune the model that I built from scratch using transformers. When I am trying to load the tokenizer from the model that is just made, it is giving Type Error
Model I am using (Bert, XLNet ...): Model is built from scratch using https://huggingface.co/blog/how-to-train

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
```
import torch
import transformers
from transformers import BertTokenizer, AdamW, BertForNextSentencePrediction

tokenizer = BertTokenizer.from_pretrained('/content/drive/My Drive/Colab Notebooks/data/model/')

```
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Google Colab
- Python version: 3.x
- PyTorch version (GPU?):'1.4.0'
- Tensorflow version (GPU?):'2.8.0'
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


How to train BART text summarization with your own data?

# ❓ Questions & Help

## Details
This is actually a two part question. I have noticed that in [1](https://github.com/huggingface/transformers/blob/master/examples/summarization/bart/run_train.sh) instructions have been given to train with the cnn/dm data. How would we train it with our own data? Should the file format be .story?
And secondly how exactly do we handle the python path in google colab?

![image](https://user-images.githubusercontent.com/23182970/78650840-49d6a580-78e1-11ea-8796-d4267bf44714.png)

I have tried in both these ways and failed.

Link to this question in SO: [2] (https://stackoverflow.com/questions/61058171/no-module-named-transformer-base/61070453#61070453)