Can't export TransfoXLModel model

## 🐛 Bug

<!-- Important information -->
I am trying to export TransfoXLModel and use it for inference from C++ API.
I tried torch.jit.trace(), torch.jit.script() and torch.onnx.export(). But none of these work.

Model I am using - TransfoXLModel:

Language I am using the model on - English

The problem arise when using:
```
model = TransfoXLModel.from_pretrained("transfo-xl-wt103", torchscript=True)
model.eval()
tokenizer = TransfoXLTokenizer.from_pretrained("transfo-xl-wt103", torchscript=True)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
torch.jit.script(model, (input_ids))
```

The tasks I am working on is:
Running inference using C++ API

## To Reproduce
Executing above python code throws error.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py:1200: UserWarning: `optimize` is deprecated and has no effect. Use `with torch.jit.optimized_execution() instead
  warnings.warn("`optimize` is deprecated and has no effect. Use `with torch.jit.optimized_execution() instead")
Traceback (most recent call last):
  File "test_bert_jit.py", line 28, in <module>
    torch.jit.script(model, (input_ids))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1203, in script
    return torch.jit.torch.jit._recursive.recursive_script(obj)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/_recursive.py", line 172, in recursive_script
    stubs = list(map(make_stub, filtered_methods))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/_recursive.py", line 169, in make_stub
    return torch.jit.script_method(func, _jit_internal.createResolutionCallbackFromClosure(func))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1280, in script_method
    ast = get_jit_def(fn, self_name="ScriptModule")
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 169, in get_jit_def
    return build_def(ctx, py_ast.body[0], type_line, self_name)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 209, in build_def
    build_stmts(ctx, body))
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in build_stmts
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in <listcomp>
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 185, in __call__
    return method(ctx, node)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 352, in build_If
    build_stmts(ctx, stmt.body),
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in build_stmts
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 127, in <listcomp>
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 185, in __call__
    return method(ctx, node)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 283, in build_Assign
    rhs = build_expr(ctx, stmt.value)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 185, in __call__
    return method(ctx, node)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 442, in build_Call
    args = [build_expr(ctx, py_arg) for py_arg in expr.args]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 442, in <listcomp>
    args = [build_expr(ctx, py_arg) for py_arg in expr.args]
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/jit/frontend.py", line 184, in __call__
    raise UnsupportedNodeError(ctx, node)
torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren't supported:
at /home/user/transformers/transformers/modeling_transfo_xl.py:767:24
        core_out = self.drop(core_out)

        new_mems = self._update_mems(hids, mems, mlen, qlen)

        # We transpose back here to shape [bsz, len, hidden_dim]
        outputs = [core_out.transpose(0, 1).contiguous(), new_mems]
        if self.output_hidden_states:
            # Add last layer and transpose to library standard shape [bsz, len, hidden_dim]
            hids.append(core_out)
            hids = list(t.transpose(0, 1).contiguous() for t in hids)
                        ~ <--- HERE
            outputs.append(hids)
        if self.output_attentions:
            # Transpose to library standard shape [bsz, n_heads, query_seq_len, key_seq_len]
            attentions = list(t.permute(2, 3, 0, 1).contiguous() for t in attentions)
            outputs.append(attentions)

        return outputs  # last hidden state, new_mems, (all hidden states), (all attentions)
```

## Expected behavior

torch.jit.script() succeeds without any error

## Environment

* OS: Ubunut 18.04
* Python version: 3.7.4
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): master @ ae1d03fc51bb22ed59517ee6f92c560417fdb049
* Using GPU ? Yes
* Distributed of parallel setup ? No.
* Any other relevant information:
Using torch.onnx.export() throws below error:
```
/home/user/transformers/transformers/modeling_transfo_xl.py:452: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator mul_. This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  embed.mul_(self.emb_scale)
/home/user/transformers/transformers/modeling_transfo_xl.py:725: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if mask_len > 0:
/home/user/transformers/transformers/modeling_transfo_xl.py:729: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  dec_attn_mask = (torch.triu(all_ones, 1+mlen)
/home/user/transformers/transformers/modeling_transfo_xl.py:730: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  + torch.tril(all_ones, -mask_shift_len))[:, :, None] # -1
/home/user/transformers/transformers/modeling_transfo_xl.py:290: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w_head_q = w_head_q[-qlen:]
/home/user/transformers/transformers/modeling_transfo_xl.py:321: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_mask is not None and torch.sum(attn_mask).item():
/home/user/transformers/transformers/modeling_transfo_xl.py:684: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  end_idx = mlen + max(0, qlen - 0 - self.ext_len)
/home/user/transformers/transformers/modeling_transfo_xl.py:685: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  beg_idx = max(0, end_idx - self.mem_len)
/home/user/transformers/transformers/modeling_transfo_xl.py:689: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  new_mems.append(cat[beg_idx:end_idx].detach())
/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py:617: UserWarning: ONNX export failed on ATen operator triu because torch.onnx.symbolic_opset10.triu does not exist
  .format(op_name, opset_version, op_name))
Traceback (most recent call last):
  File "test_bert_jit.py", line 37, in <module>
    output_names = ['output']) # the model's output names
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/__init__.py", line 143, in export
    strip_doc_string, dynamic_axes, keep_initializers_as_inputs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 66, in export
    dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 382, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 262, in _model_to_graph
    fixed_batch_size=fixed_batch_size)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 132, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/__init__.py", line 174, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 618, in _run_symbolic_function
    op_fn = sym_registry.get_registered_op(op_name, '', opset_version)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/onnx/symbolic_registry.py", line 91, in get_registered_op
    return _registry[(domain, version)][opname]
KeyError: 'triu'
```

## Additional context

<!-- Add any other context about the problem here. -->


fine tuning bert and roberta_base model

could you please let me know how to fine tune the BERT/ ROBERTA_Base models?


request for a Bert_base uncase model.bin file

## 🚀 Feature

currently we r using BERT_Large uncased which slows down our application we like to use BERT_Base uncased model but BERT_base uncased model does not contain bin file.could  you please let me know where I get model.bin file.


OpenAIGPTDoubleHeadsModel Not working (even with the official example...)

## 🐛 Bug

<!-- Important information -->

Model I am using : OpenAI GPT  (DoubleHeadsModel)

Language I am using the model on :  English

The problem arise when using:
* [ ] the official example scripts: [link](https://huggingface.co/transformers/model_doc/gpt.html)
```
tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTDoubleHeadsModel.from_pretrained('openai-gpt')
tokenizer.add_special_tokens({'cls_token': '[CLS]'})  # Add a [CLS] to the vocabulary (we should train it also!)
choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]
input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0)  # Batch size 1, 2 choices
mc_token_ids = torch.tensor([input_ids.size(-1), input_ids.size(-1)]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, mc_token_ids=mc_token_ids)
lm_prediction_scores, mc_prediction_scores = outputs[:2]
```
This codes doesn't work.
Maybe need to add `model.resize_token_embeddings(len(tokenizer))`
However, it still doesn't work with the following error
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-53e79c250ad3> in <module>
      7 input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0)  # Batch size 1, 2 choices
      8 mc_token_ids = torch.tensor([input_ids.size(-1), input_ids.size(-1)]).unsqueeze(0)  # Batch size 1
----> 9 outputs = model(input_ids, mc_token_ids=mc_token_ids)
     10 lm_prediction_scores, mc_prediction_scores = outputs[:2]

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_openai.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, mc_token_ids, lm_labels, mc_labels)
    603
    604         lm_logits = self.lm_head(hidden_states)
--> 605         mc_logits = self.multiple_choice_head(hidden_states, mc_token_ids).squeeze(-1)
    606
    607         outputs = (lm_logits, mc_logits) + transformer_outputs[1:]

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in forward(self, hidden_states, cls_index)
    728                 cls_index = cls_index.expand((-1,) * (cls_index.dim()-1) + (hidden_states.size(-1),))
    729             # shape of cls_index: (bsz, XX, 1, hidden_size) where XX are optional leading dim of hidden_states
--> 730             output = hidden_states.gather(-2, cls_index).squeeze(-2) # shape (bsz, XX, hidden_size)
    731         elif self.summary_type == 'attn':
    732             raise NotImplementedError

RuntimeError: Invalid index in gather at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:657
```

## Environment

* OS: Linux
* Python version: 3.6
* PyTorch version: 1.3
* Using GPU ? yes


Unpickling errors when running examples

## ❓ Questions & Help

Hi there, when I run the examples

```
%run run_generation.py \
    --model_type=gpt2 \
    --model_name_or_path=gpt2
```

I keep getting the following errors:

```
---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
~/research/transformers/examples/run_generation.py in <module>
    258
    259 if __name__ == '__main__':
--> 260     main()

~/research/transformers/examples/run_generation.py in main()
    186     model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
    187     tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path)
--> 188     model = model_class.from_pretrained(args.model_name_or_path)
    189     model.to(args.device)
    190     model.eval()

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    343
    344         if state_dict is None and not from_tf:
--> 345             state_dict = torch.load(resolved_archive_file, map_location='cpu')
    346
    347         missing_keys = []

~/anaconda3/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    385         f = f.open('rb')
    386     try:
--> 387         return _load(f, map_location, pickle_module, **pickle_load_args)
    388     finally:
    389         if new_fd:

~/anaconda3/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    562             f.seek(0)
    563
--> 564     magic_number = pickle_module.load(f, **pickle_load_args)
    565     if magic_number != MAGIC_NUMBER:
    566         raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: invalid load key, '<'.

```


Access denied to pretrained GPT2 model

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: I cannot load the GPT2 small pretrained model.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: I am trying to instantiate a GPT2 pretrained model.

## To Reproduce

Steps to reproduce the behavior:

```
from pytorch_transformers import AutoModel
model = AutoModel.from_pretrained('gpt2')
```

This only happens with the 'gpt2' shortcut, not others ('gpt2-medium', etc.)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

Error message:
```
---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
<ipython-input-14-e057f5f0ba3e> in <module>
      1 from pytorch_transformers import AutoModel
----> 2 model = AutoModel.from_pretrained('gpt2')

~/pipeline/.venv/lib/python3.7/site-packages/pytorch_transformers/modeling_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    246             return OpenAIGPTModel.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
    247         elif 'gpt2' in pretrained_model_name_or_path:
--> 248             return GPT2Model.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
    249         elif 'transfo-xl' in pretrained_model_name_or_path:
    250             return TransfoXLModel.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)

~/pipeline/.venv/lib/python3.7/site-packages/pytorch_transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    537
    538         if state_dict is None and not from_tf:
--> 539             state_dict = torch.load(resolved_archive_file, map_location='cpu')
    540         if from_tf:
    541             # Directly load from a TensorFlow checkpoint

~/pipeline/.venv/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    385         f = f.open('rb')
    386     try:
--> 387         return _load(f, map_location, pickle_module, **pickle_load_args)
    388     finally:
    389         if new_fd:

~/pipeline/.venv/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    562             f.seek(0)
    563
--> 564     magic_number = pickle_module.load(f, **pickle_load_args)
    565     if magic_number != MAGIC_NUMBER:
    566         raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: invalid load key, '<'.
```

Contents of the downloaded file:
```
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>D027DE3363BB3D26</RequestId><HostId>GPDcAN+fZerpFZ5ZR9ZnATk3XIJ4GgLjCDMLnzvs48MRKG8soooyb8HM+zjBA0Gnn7HJc4CRqpA=</HostId></Error>%
```

## Expected behavior

Successfully load the pretrained model.

## Environment

* OS: macOS Catalina
* Python version: 3.7.4
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Add ALBERT to the library

This PR adds ALBERT to the library. It offers two new model architectures:

- AlbertModel
- AlbertForMaskedLM

AlbertModel acts in a similar way to BertModel as it returns a sequence output as well as a pooled output. AlbertForMaskedLM exposes an additional language modeling head.

A total of four pre-trained checkpoints are available, which are the checkpoints discussed in the official ALBERT paper, available on the TensorFlow hub page:

- albert-base
- albert-large
- albert-xlarge
- albert-xxlarge

These are currently available on the S3 bucket: an ALBERT model may be loaded like other models with the following code.

```py
from transformers import AlbertTokenizer, AlbertModel

tokenizer = AlbertTokenizer.from_pretrained("albert-base")
model = AlbertModel.from_pretrained("albert-base")
```

What is left to implement:

- ~PyTorch model & tests~
- ~Tokenizer & tests~
- ~Export PyTorch checkpoints~
- ~TensorFlow 2 model & tests~
- ~Export TensorFlow 2 checkpoints~
- Replicate the results obtained in the paper; **currently obtained 81 acc on MNLI with albert-base**

# Workflow for including a model from [README.md](https://github.com/huggingface/transformers/blob/master/templates/adding_a_new_model/README.md)

Here an overview of the general workflow:

- [ ] add model/configuration/tokenization classes
- [ ] add conversion scripts
- [ ] add tests
- [ ] finalize

Let's details what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-past or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-past or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [x] copy-past or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this script to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] [high-level-API] add your models and tokenizer to `pipeline.py`
- [x] [high-level-API] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation it-self at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.


xnli benchmark

adapted from `run_glue.py`


Wrong Roberta special tokens in releases on GitHub

## 🐛 Bug

Model I am using (Bert, XLNet....): Roberta

Language I am using the model on (English, Chinese....): Potentially wrong on any language

The problem arise when using the official example scripts: see https://github.com/huggingface/transformers/releases/tag/1.1.0

In the section `Tokenizer sequence pair handling` the special tokens for Roberta are wrong if I'm not mistaken. The example reads:
```
[CLS] SEQUENCE_0 [SEP] [SEP] SEQUENCE_1 [SEP]
```

whereas Roberta's actual representation for a sequence pair including special tokens should be (also following transformer's official documentation, cf. https://huggingface.co/transformers/model_doc/roberta.html):
```
<s> SEQUENCE_0 </s> <s> SEQUENCE_1 </s>
```
Note the <s> or </s> instead of [SEP]. I am not sure about the [CLS], though, but I think for Roberta it should not be there.


Error when creating RobertTokenizer for distilroberta-base

## 🐛 Bug

Model I am using (Bert, XLNet....): DistilRoberta

Language I am using the model on (English, Chinese....): EN

The problem arise when using the official example scripts: https://github.com/huggingface/transformers/tree/master/examples/distillation

## To Reproduce
```
RobertaTokenizer.from_pretrained('distilroberta-base')
```

this will yield an error:
```
OSError: Model name 'distilroberta-base' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli). We assumed 'distilroberta-base' was a path or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
```

## Expected behavior
should produce a RobertaTokenizer instance, which can also be used for `distilroberta-base`.

## Environment

* OS: MacOS
* Python version: 3.7
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch):  2.1.1
* Using GPU ? no


Fix https://github.com/huggingface/transformers/issues/1673


Download assets directly to the specified cache_dir

## 🚀 Feature

```
import torch
from transformers import *
TRANSFORMERS_CACHE='/path/to/my/transformers-cache'
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', cache_dir=TRANSFORMERS_CACHE)
```

Actual behavior: It downloads the asset into a temp folder and then copies it to the specified cache_dir.

Proposed behavior: Download the asset directly to the specified cache_dir.

Impacted code part: https://github.com/huggingface/transformers/blob/master/transformers/file_utils.py#L295-L322

## Motivation

We have an environment setup where the tmp folders have limited space, because it is not a mounted docker volume. If the `"asset size" > 10GB - "docker image size"` then it won't be able to download the asset. (The 10GB limitation is a docker limitation)


i want to use bert pre-trained modle in a text classification problem which the text with Multi-label. But，there are some problems .

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
i want to use bert pre-trained modle in a text classification problem which the text with Multi-labels. Which program and task should i select between run_glue.py、run_multiple_choice.py、run_squad.py and so on？
for example of my text：“I'd like 2 tickets to see Zoolander 2 tomorrow at Regal Meridian 16 theater in Seattle at 9:25 PM”
 in this text include this labels: request_ticket;inform_moviename;inform_date;inform_theater;inform_city;inform_starttime;inform_numberofpeople
which program  should i select?  Thanks very much!


🌟 BART

# 🌟New model addition

## Model description

method for pre-training seq2seq models by de-noising text. BART outperforms previous work on a bunch of generation tasks (summarization/dialogue/QA), while getting similar performance to RoBERTa on SQuAD/GLUE

[BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/pdf/1910.13461.pdf)

Amazing results on text summarization.

## Open Source status

* [x] the model implementation is available: not yet
* [x] the model weights are available: not yet
* [ ] who are the authors: @yinhanliu @ernamangoyal


Any example of how to do multi-class classification with TFBertSequenceClassification

## ❓ Questions & Help

I am trying to create a multi-class text classification model using BertSequenceClassifier for Tensorflow 2.0. Any help with the implementation strategy would be appreciated. Also, are there any recommendations as to how to convert a simple CSV file containing text and labels into TF dataset format such as Glue?


possible issues with run_summarization_finetuning.py

Hi,
thanks for pushing summarization codes, here are my comments on this file:

- line 473: checkpoints = [] is empty and will not be evaluated. Also, the evaluation script is not called.

- line 482: results = "placeholder" is set to the placeholder, I was wondering if the function could return the generated text,so user could see the performance of the method visually.

- line 463: the model is only saved after the training, would be great to have the saving option like run_glue, "eval_during_training" also during the training activated.

- line 272 /transformers/modeling_encoder_decoder.py, the tie of weights is not done, this is a part of the model, would be great to have it implemented

- line 139 of transformers/modeling_auto.py, here you check the "path" if this starts with the name of bert, ..., you load the relevant one, but in run_summarization_finetuning, the user does not need to save the model in args.output_dir which starts with the name of model, so the codes wont work if the model is not saved with the path starting with the name of the model

- line 152 of /run_summarization_finetuning.py for param_group in optimizer.param_groups: I think this should be optimizer[stack] not optimizer alone

- utils_summarization.py: line 180, the comparison should not work, since s is a sequence, and then it is compared with special token,

- utils_summarization.py:  line 182: embeddings.append(sentence_num % 2), to me you need to add sentence_sum%2 for the length of the sentence, but not 1 for each sentence.

thanks.


BertModel.from_pretrained is failing with "HTTP 407 Proxy Authentication Required" during model weight download when running behing a proxy

## 🐛 Bug

<!-- Important information -->
Hello,
I'am using transformers behind a proxy. `BertConfig.from_pretrained(..., proxies=proxies)` is working as expected, where `BertModel.from_pretrained(..., proxies=proxies)` gets a
`OSError: Tunnel connection failed: 407 Proxy Authentication Required` . This could be the symptom of `proxies` parameter not being passed through the `request` package commands.

Model I am using (Bert, XLNet....): Bert, base, cased.

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. put your endpoint behind a proxy
2. configure the proxies variable accordingly `proxies={"https": 'foo.bar:3128'}
3. run any script calling BertConfig.from_pretrained( ...,proxies=proxies)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
Stack Trace :
```
bash-3.2$ cd /Users/xxxx/_Data.science/NLP ; env PYTHONIOENCODING=UTF-8 PYTHONUNBUFFERED=1 /Users/xxxx/anaconda3/envs/farm-nlp/bin/python  /Users/xxxx/FARM/examples/embeddings_extraction.py
10/29/2019 13:10:21 - INFO - transformers.file_utils -   PyTorch version 1.2.0 available.
10/29/2019 13:10:22 - INFO - transformers.modeling_xlnet -   Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .
10/29/2019 13:10:22 - WARNING - farm.utils -   TensorboardX not installed. Required if you use tensorboard logger.
10/29/2019 13:10:22 - INFO - farm.utils -   device: cpu n_gpu: 0, distributed training: False, 16-bits training: False
10/29/2019 13:10:22 - INFO - farm.modeling.tokenization -   Loading tokenizer of type 'BertTokenizer'
10/29/2019 13:10:23 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt not found in cache or force_download set to True, downloading to /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpwaag8tam
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 231508/231508 [00:01<00:00, 154673.07B/s]
10/29/2019 13:10:25 - INFO - transformers.file_utils -   copying /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpwaag8tam to cache at /Users/xxxx/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/29/2019 13:10:25 - INFO - transformers.file_utils -   creating metadata file for /Users/xxxx/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/29/2019 13:10:25 - INFO - transformers.file_utils -   removing temp file /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpwaag8tam
10/29/2019 13:10:25 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt from cache at /Users/xxxx/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/29/2019 13:10:26 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json not found in cache or force_download set to True, downloading to /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmprex2n__s
Traceback (most recent call last):
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 662, in urlopen
    self._prepare_proxy(conn)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 948, in _prepare_proxy
    conn.connect()
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/urllib3/connection.py", line 342, in connect
    self._tunnel()
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/http/client.py", line 919, in _tunnel
    message.strip()))
OSError: Tunnel connection failed: 407 Proxy Authentication Required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /models.huggingface.co/bert/bert-base-cased-config.json (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/transformers/configuration_utils.py", line 133, in from_pretrained
    resolved_config_file = cached_path(config_file, cache_dir=cache_dir, force_download=force_download, proxies=proxies)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/transformers/file_utils.py", line 176, in cached_path
    return get_from_cache(url_or_filename, cache_dir=cache_dir, force_download=force_download, proxies=proxies)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/transformers/file_utils.py", line 302, in get_from_cache
    http_get(url, temp_file, proxies=proxies)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/transformers/file_utils.py", line 238, in http_get
    req = requests.get(url, stream=True, proxies=proxies)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/requests/adapters.py", line 510, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /models.huggingface.co/bert/bert-base-cased-config.json (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxxx/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/Users/xxxx/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/__main__.py", line 432, in main
    run()
  File "/Users/xxxx/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/xxxx/_Data.science/NLP/FARM/examples/embeddings_extraction.py", line 38, in <module>
    language_model = Bert.load(lang_model_conf)
  File "/Users/xxxx/_Data.science/NLP/FARM/farm/modeling/language_model.py", line 253, in load
    bert.model = BertModel.from_pretrained(pretrained_model_name_or_path)
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/transformers/modeling_utils.py", line 287, in from_pretrained
    **kwargs
  File "/Users/xxxx/anaconda3/envs/farm-nlp/lib/python3.6/site-packages/transformers/configuration_utils.py", line 145, in from_pretrained
    raise EnvironmentError(msg)
OSError: Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json' to download pretrained model configuration file.
Terminated: 15
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
Model-weights.bin file download,  after silent, behind-the-scene correct proxy authentication :

```
cd /Users/xxxxx/_Data.science/NLP ; env PYTHONIOENCODING=UTF-8 PYTHONUNBUFFERED=1 /Users/xxxxx/anaconda3/envs/farm-nlp/bin/python  /Users/xxxxx/FARM/examples/embeddings_extraction.py
10/29/2019 15:28:48 - INFO - transformers.file_utils -   PyTorch version 1.2.0 available.
10/29/2019 15:28:48 - INFO - transformers.modeling_xlnet -   Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .
10/29/2019 15:29:00 - WARNING - farm.utils -   TensorboardX not installed. Required if you use tensorboard logger.
10/29/2019 15:29:00 - INFO - farm.utils -   device: cpu n_gpu: 0, distributed training: False, 16-bits training: False
10/29/2019 15:29:00 - INFO - farm.modeling.tokenization -   Loading tokenizer of type 'BertTokenizer'
10/29/2019 15:29:00 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt from cache at /Users/xxxxx/.cache/torch/transformers/5e8a2b4893d13790ed4150ca1906be5f7a03d6c4ddf62296c383f6db42814db2.e13dbb970cb325137104fb2e5f36fe865f27746c6b526f6352861b1980eb80b1
10/29/2019 15:29:03 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json not found in cache or force_download set to True, downloading to /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpxtz55r5f
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 313/313 [00:00<00:00, 88643.97B/s]
10/29/2019 15:29:04 - INFO - transformers.file_utils -   copying /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpxtz55r5f to cache at /Users/xxxxx/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.d7a3af18ce3a2ab7c0f48f04dc8daff45ed9a3ed333b9e9a79d012a0dedf87a6
10/29/2019 15:29:04 - INFO - transformers.file_utils -   creating metadata file for /Users/xxxxx/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.d7a3af18ce3a2ab7c0f48f04dc8daff45ed9a3ed333b9e9a79d012a0dedf87a6
10/29/2019 15:29:04 - INFO - transformers.file_utils -   removing temp file /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpxtz55r5f
10/29/2019 15:29:04 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json from cache at /Users/xxxxx/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.d7a3af18ce3a2ab7c0f48f04dc8daff45ed9a3ed333b9e9a79d012a0dedf87a6
10/29/2019 15:29:04 - INFO - transformers.configuration_utils -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}

10/29/2019 15:29:05 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin not found in cache or force_download set to True, downloading to /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpaz0jbgo4
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 435779157/435779157 [04:19<00:00, 1677901.46B/s]
10/29/2019 15:33:25 - INFO - transformers.file_utils -   copying /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpaz0jbgo4 to cache at /Users/xxxxx/.cache/torch/transformers/35d8b9d36faaf46728a0192d82bf7d00137490cd6074e8500778afed552a67e5.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2
10/29/2019 15:33:26 - INFO - transformers.file_utils -   creating metadata file for /Users/xxxxx/.cache/torch/transformers/35d8b9d36faaf46728a0192d82bf7d00137490cd6074e8500778afed552a67e5.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2
10/29/2019 15:33:26 - INFO - transformers.file_utils -   removing temp file /var/folders/kg/qbqf751d6r13qchghq3vs15w0000gn/T/tmpaz0jbgo4
10/29/2019 15:33:26 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin from cache at /Users/xxxxx/.cache/torch/transformers/35d8b9d36faaf46728a0192d82bf7d00137490cd6074e8500778afed552a67e5.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2
```
## Environment

* OS: MacOS
* Python version: 3.6
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information: Proxy

## Additional context

<!-- Add any other context about the problem here. -->


Hello,

Is HuggingFace TransfoXLLMHeadModels trainable from scratch? The documentation makes it look like it is possible to train the TransfoXLLMHeadModel from scratch, since (according to the documentation) the loss can be returned via TransfoXLLMHeadModel( ) as long as labels are provided (https://huggingface.co/transformers/model_doc/transformerxl.html#transfoxllmheadmodel) .

However, the code for TransfoXLLMHeadModels shown in the Github repository
(https://github.com/huggingface/transformers/blob/master/transformers/modeling_transfo_xl.py#L780) seem to suggest that the loss is, in fact, not returned even when the labels are provided.

Is HuggingFace TransfoXLLMHeadModels trainable from scratch?

Thank you,


Quick Tour TF2.0 Training Script has Control Flow Error when Replacing TFBERT with TFRoberta

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....): TFRoberta.

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: Quick Tour TF2.0 Training Script.
* [ ] my own modified scripts:

Details of the issue:

When replacing TF BERT with TF Roberta (and the relevant tokenizer) in the quick tour script, I get the following error:

```
TypeError: You are attempting to use Python control flow in a layer that was not declared to be dynamic. Pass `dynamic=True` to the class constructor.
Encountered error:
"""
using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph is disabled in this function. Try decorating it directly with @tf.function.
"""
```

I suspect this extends to all models, though I haven't verified this. Any thoughts?

## Environment

* OS: Catalina
* Python version: 3.7.14
* PyTorch version: NA
* PyTorch Transformers version (or branch): Transformers
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Checklist

- [x] I have read the migration guide in the readme.
- [x] I checked if a related official extension example runs on my machine.

## Additional context

<!-- Add any other context about the problem here. -->


Templates and explanation for adding a new model and example script

This PR adds:
- templates and explantations for all the steps needed to add a new model
- a simple template for adding a new example script (basically the current `run_squad` example).
- links to them in the `README` and `CONTRIBUTING` docs.

@LysandreJik and @rlouf, feel free to update if you want to add stuff or fix the wording.


How to load trained model of distilbert

## ❓ Questions & Help
Hi,
 I have trained distilbert using the steps mentioned in example/distillation.
saved the checkpoints into one directory.

But i cant use run_glue.py using the checkpoint path i saved for distilbert.
It throws error for tokenizer missing.

Would you please help me, how to achieve that. If i am doing any mistake in my step.
TIA!!!
<!-- A clear and concise description of the question. -->


Fixed training for TF XLM

This PR fixes `model.fit()` training for TF XLM model, and tested in a script similar to `run_tf_glue.py`. It also is tested and works with AMP and tf.distribute for mixed precision and multi-GPU training.

This changes some Python `assert` statements to `tf.debugging.assert_equal` both in `TFXLMMainLayer.call()` and `gen_mask()`

Otherwise, errors encountered:
* `TypeError: You are attempting to use Python control flow in a layer that was not declared to be dynamic. Pass 'dynamic=True' to the class constructor.`
* `OperatorNotAllowedInGraphError: using a 'tf.Tensor' as a Python 'bool' is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.`


Added FP16 support to benchmarks.py

This PR adds in FP16 support for the inference benchmarks for TensorFlow and PyTorch, and presents the collected results. This is a "re-do" of a previous PR (#1567) taking into account changes to `benchmark.py` and also adding in the PyTorch component with additional results collected.

**TensorFlow**

Added a automatic mixed precision (AMP) option to the benchmark script. As you can see, we can get between 1.2x to up to 4.5x inference speed depending on model, batch size and sequence length. (1.0x refers to no change in speed)

| Batch Size | Speedup (XLA only) | Speedup (XLA + AMP) | Min. Seq Len* |
| -------------- | --------------------------- | ------------------------------- | ------------------ |
| 1                | 1.1 ~ 1.9                    | 1.4 ~ 2.9                       | 512                 |
| 2                | 1.1 ~ 1.9                    | 1.4 ~ 3.4                       | 256                 |
| 4                | 1.1 ~ 2.1                    | 1.2 ~ 3.8                       | 128                 |
| 8                | 1.1 ~ 3.1                    | 1.2 ~ 4.5                       | 64                   |

*Min. Seq Len refers to minimum sequence length required to not see **any** performance regression at all. For example, at batch size 1:
* Seq Len of 512 tokens see speed up of 1.4~2.1x depending on model
* Seq Len of 256 tokens see speed up of 0.8~1.2x depending on model

**PyTorch**

Added a FP16 (half precision) option to the benchmark script. As you can see, we can get between up to 4.2x inference speed depending on model, batch size and sequence length. (1.0x refers to no change in speed)

| Batch Size | Speedup (TorchScript only) | Speedup (FP16 Only) |
| -------------- | ------------------------------------- | ----------------------------- |
| 1                | 1.0 ~ 1.7                               | 1.0 ~ 3.0                      |
| 2                | 1.0 ~ 1.8                               | 1.0 ~ 3.5                      |
| 4                | 1.0 ~ 1.7                               | 1.0 ~ 4.0                      |
| 8                | 1.0 ~ 1.7                               | 1.4 ~ 4.2                      |

*FP16 and CTRL result in performance regression below 1x256, 2x128, 4x64.

**Summary of Collected Results**

Google Sheets with the TF/PyTorch results [here](https://docs.google.com/spreadsheets/d/1IW7Xbv-yfE8j-T0taqdyoSehca4mNcsyx6u0IXTzSJ4/edit#gid=1307979840). GPU used is a single V100 (16GB).


Question: Token sequence length longer maximum sequence length

## ❓ Questions & Help

I'm using `run_glue.py` with a task name of `SST-2` to fine-tune a binary classifier on my data, which I put into the required format. However, some of my data's sentences are longer than the `max_seq_length` of `512` for `BERT` and `RoBERTa`; so, I get

`WARNING - transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (length_of_my_string > 512). Running this sequence through the model will result in indexing errors`.

What exactly is happening here? Are the training examples with more than `510` tokens still being used? If so, is the string being truncated down to `[CLS]` + the `first 510 tokens` + `[SEP]`? Is there any way to increase the `max_seq_length` or implement something like `head+tail`, which selects the `first 128` and the `last 382` tokens like suggested in this [paper](https://arxiv.org/pdf/1905.05583.pdf). That paper also uses `discriminative learning rate` as suggested [here](https://arxiv.org/pdf/1801.06146.pdf). Is there any plan to implement this?


Allowing PR#1455 to be merged in the master

Hi Thomas
Remi was saying in PR:#1455 it has the bert seq2seq ready, could you move in a gradual way please and allow this PR to be merged at this stage that is working for BERT? Then people can use the BERT one, this is already great, then after a while when this is ready for also other encoders, you can add them later, I really appreciate adding the BERT ones thanks


Moving model from GPU -> CPU doesn't work

## 🐛 Bug

Hi,

I tried creating a model (doesn't matter which one from my experiments), moving it first to multiple GPUs and then back to CPU. But I think it doesn't work as intended.

The following is the code to reproduce the error:

```python
import torch
import torch.nn as nn
from transformers import BertTokenizer, BertModel

>>> model = BertModel.from_pretrained('bert-base-uncased')
>>> model.to('cuda:0')
>>> model = nn.DataParallel(model, device_ids=range(torch.cuda.device_count()))
>>> print(model.device_ids)
[0, 1]
>>> model.to('cpu')
>>> print(model.device_ids) # Still on GPUs
[0, 1]
```


Problem with restoring GPT-2 weights

Hello, I've been debugging an issue for a while and it seem it's a model-specific issue.

I'm training GPT-2 on a TPU and I can't save and restore it. It looks like there is a code that silently changes parameter values right in `load_state_dict()`.

```
    print(state_dict['transformer.wte.weight'])
    print(state_dict['transformer.wte.weight'].shape)

    cpu_model = model_class(config=config)
    cpu_model.load_state_dict(state_dict)
    print(cpu_model.state_dict()['transformer.wte.weight'])
    print(cpu_model.state_dict()['transformer.wte.weight'].shape)
```

```
tensor([[-0.1101, -0.0393,  0.0331,  ..., -0.1364,  0.0151,  0.0453],
        [ 0.0417, -0.0488,  0.0485,  ...,  0.0827,  0.0097,  0.0454],
        [-0.1275,  0.0479,  0.1841,  ...,  0.0899, -0.1297, -0.0879],
        ...,
        [-0.0439, -0.0579,  0.0103,  ...,  0.1113,  0.0919, -0.0724],
        [ 0.1846,  0.0156,  0.0444,  ..., -0.0974,  0.0785, -0.0211],
        [ 0.0471, -0.0284,  0.0492,  ...,  0.0048,  0.1511,  0.1202]])
torch.Size([50257, 768])
tensor([[-0.1317, -0.0305,  0.0339,  ..., -0.1310,  0.0113,  0.0262],
        [ 0.0413, -0.0491,  0.0451,  ...,  0.0930, -0.0019,  0.0457],
        [-0.1465,  0.0565,  0.1839,  ...,  0.0962, -0.1339, -0.1074],
        ...,
        [-0.0432, -0.0628,  0.0088,  ...,  0.1002,  0.1045, -0.0654],
        [ 0.1725,  0.0160,  0.0444,  ..., -0.0944,  0.0760, -0.0289],
        [ 0.0330, -0.0182,  0.0455,  ...,  0.0136,  0.1487,  0.0975]])
torch.Size([50257, 768])
```

For the context
https://github.com/pytorch/xla/issues/1245
https://discuss.pytorch.org/t/problem-with-model-accuracy-after-restore-on-tpu/59304/3

Full code is here
https://github.com/mgrankin/ru_transformers/blob/9d52a4caef16df5b921c386f4841c879877d03a4/debug_lm_finetuning.py


Tokenizer.tokenize return none on some utf8 string in current pypi version
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts:

The tasks I am working on is:
* [ ] my own task or dataset: SQUaD format, Chinese, DRCD

## To Reproduce
Current seems not updated
Cause returning null result in  Tokenizer.tokenize when input some special utf8 string

Steps to reproduce the behavior:

1.
```
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-uncased')
```
2.
```
text = "ุ"
tokenized_text = tokenizer.tokenize(text)
print(len(text.split()),len(text.strip().split()),text,tokenized_text,"\n")
```
3.
```
1 1 ุ []
```

## Expected behavior

I have try the implementation from GitHub, It seems to be fine :
```
def whitespace_tokenize(text):
    """Runs basic whitespace cleaning and splitting on a piece of text."""
    text = text.strip()
    if not text:
        return []
    tokens = text.split()
    return tokens

def tokenize(text):
        output_tokens = []
        for token in whitespace_tokenize(text):
            chars = list(token)
            if len(chars) > 100:
                output_tokens.append("[UNK]")
                continue

            is_bad = False
            start = 0
            sub_tokens = []

            while start < len(chars):
                end = len(chars)
                cur_substr = None
                while start < end:
                    substr = "".join(chars[start:end])
                    if start > 0:
                        substr = "##" + substr
                    if substr in tokenizer.vocab:
                        cur_substr = substr
                        break
                    end -= 1
                if cur_substr is None:
                    is_bad = True
                    break
                sub_tokens.append(cur_substr)
                start = end

            if is_bad:
                output_tokens.append("[UNK]")
            else:
                output_tokens.extend(sub_tokens)
        return output_tokens

print(len(text.split()),len(text.strip().split()),text,tokenize(text),"\n")
```
Return
```
1 1 ุ ['[UNK]']
```

## Colab demo : https://colab.research.google.com/drive/1WGu4dYLWtaPRPBq_YZEPvrmMALEFlCBn


BERT multi heads attentions

Hello,
I would like to analysis the effect of specific heads' attention.
Is it possible to turn off some heads attentions in a particular layer?
if yes, can you please tell me how to do that or share any helpful document?

Thank you in advance


How to fine-tune CTRL?

How to fine-tune CTRL on a custom dataset with custom control codes using the transformers package?

I'm aware of the [guide](https://github.com/salesforce/ctrl/tree/master/training_utils) for tensorflow users. However, as a PyTorch user, the guide is not friendly to me.

I'm also aware of the language modelling fine-tuning script [here](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py). However, it does not support CTRL right now.

Thanks,
Peixiang


How is the interactive GPT-2 implemented?

## ❓ Questions & Help
I came across this online demo from HuggingFace for GPT-2 writing: https://transformer.huggingface.co/doc/gpt2-large. The demo is really amazing, both accurate and fast. My major observation is that the service actually uses user's earlier writing examples in later prediction, almost instantly. I'm very curious how it is implemented? It seems to me that it is not fine-tuned in real-time, then is there some other mechanism behind it? Any ideas are appreciated.
Context examples I typed in:

> Set the a c to low level = the room is very cold
> turn down the volume = the music is too loud

Then when i try:
> turn on the lights =

It gives me
> the room is too bright

Also, I tried fine-tuning the entire model with much more than 2 examples (around 30), however the result for "turn on the lights = "  after fine-tuning is a lot worse than the demo:

> about 0.016 (lit a second and a 100 pixels)

Is it that the demo only fine-tune, e.g. the very last layer of the model?


How to fine tune xlm-mlm-100-128 model.

## ❓ Questions & Help

How to fine tune xlm-mlm-17-128 model for own dataset. Since, run_lm_finetuning.py has no option to fine tune XLM models.


[WIP] Raise error if larger sequences

Hi,

I suggest to improve a bit user experience when using pretrained models by raising more errors if some of parameters are incoherent.
For example, in this PR, there is a suggestion to raise error and thus inform user about potential error as "RuntimeError: cublas runtime error ..." which can be harder to find if running on GPU.

What do you think ?


Parallel data preprocessing for distillation

## 🚀 Feature

Use the `multiprocessing.Pool` function to parallelize the text tokenization and uint16 conversion in `transformers/examples/distillation/scripts/binarized_data.py`.

## Motivation

I tried to preprocess a 2.6 GB txt file using the python script, but the expected time is 2.4 hours. I tried to parallelize it myself and the total time decreased to 10 minutes on my server.

## Additional context

My code is something like this:

```
def process_data(text):
    return tokenizer.encode(f'{bos} {text.strip()} {sep}')

pool = Pool()
rslt = pool.map(process_data, data)
rslt_ = pool.map(np.uint16, rslt)
```


Missing a line in examples/distillation/README.md

In How to train Distil* -> B, in both of the training commands, you should add `--alpha_clm 0.0 \`, otherwise an assertion error will be triggered (https://github.com/huggingface/transformers/blob/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c/examples/distillation/train.py#L49).


Can I load a CTRL model that was fine-tuned using the Salesforce code?

## ❓ Questions & Help

I have a custom CTRL model that I trained using the Salesforce TF code and I was hoping that I could convert it into the transformers format and load it there. Any advice?


No way to control ID of special chars e.g. mask IDs

## Summary

Hi, many thanks for the library - this is a fantastic tool for the NLP community!

I notice there are a number of constants defined in code that the user cannot inject whilst initialising.
Some examples are:
- `padding_idx = 1` in `RobertaEmbeddings`
- `CrossEntropyLoss(ignore_index=-1)` in `RobertaForMaskedLM`.
- `padding_idx=0` in `BertEmbeddings`

`RobertaModel` also raises a warning if there are no tokens with index `0`, but it is not clear which control character this corresponds to.

1. Would it be a good idea to allow these parameters to be injectable so a user can control the ID of the special tokens?
2. Is it possible to provide a list of what the expected indices for special characters are?
I think for example:
```
    -1 => Ignore target during loss
    0 => `[CLS]`
    1 => `[PAD]`
```
but 0 could also be `[SEP]` as I believe both are always used in roBERTa.
Is there an index I must respect other than these? E.g. does `[SEP]` need a specific index?
3. Why is the ignore index -1? Is this just to stay true to the original papers? Wouldn't the index of the `[PAD]` token make sense? I notice this index is different in the different embedding classes.

Many thanks for your thoughts,
Dom


Missing required argument 'mode' in run_ner.

## 🐛 Bug

<!-- Important information -->

Model I am using: BERT

Language I am using the model on: Polish (irrelevant for the error)

The problem arise when using:
* [x] the official example scripts: run_ner.py

The tasks I am working on is:
* [x] my own task or dataset: token classification (aka NER)

## To Reproduce

Steps to reproduce the behavior:

1. Start run_ner.py with --evaluate_during_training
2. During evaluation the error will happen

## Expected behavior

Evaluation should run fine

## Additional context

There is a missing argument in line 156 `mode`, which (I belive) should be `"dev"`.
I can provide a PR if the above solution is confirmed.


How to set local_rank argument in run_squad.py

Hi!

I would like to try out the run_squad.py script (with AWS SageMaker in a PyTorch container).
I will use 8 x 100V 16 GB GPUs for the training.
How should I set the the local_rank parameter in this case?
( I tried to understand it from the code, but I couldn't really.)

Thank you for the help!


Custom language text generation

## ❓ Questions & Help

How to generate text in non english languages. Is xlm-mlm-100-1280 best for this, I tried but results are too worst.
Also tried things mentioned here:
https://github.com/huggingface/transformers/issues/1414
https://github.com/huggingface/transformers/issues/1068
https://github.com/huggingface/transformers/issues/1407
https://github.com/Morizeyao/GPT2-Chinese

Any better suggestion Please.


ALBERT

# 🌟New model addition

## Model description

ALBERT is "A Lite" version of BERT, a popular unsupervised language representation learning algorithm. ALBERT uses parameter-reduction techniques that allow for large-scale configurations, overcome previous memory limitations, and achieve better behavior with respect to model degradation.

For a technical description of the algorithm, see our paper:

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut

## Open Source status

* [ ] the model implementation is available: https://github.com/google-research/google-research/tree/master/albert.

I just want to ask whether you guys have plan to add ALBERT in near future.


Changing LM loss function

Hi all,

I am modifying gpt2 loss function. My new code looks like that:

        lm_logits = self.lm_head(hidden_states)

        outputs = (lm_logits,) + transformer_outputs[1:]
        if labels is not None:
            # Shift so that tokens < n predict n
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = labels[..., 1:].contiguous()
            all_logits = shift_logits[0].cpu().data.numpy()
            tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
            sampled_words = []
            for elem in all_logits:
                logits = elem.reshape(-1)
                exps = np.exp(numbers - np.max(logits))
                output_logits_normalized = exps / np.sum(exps)
                sampled_word = np.array(np.argmax(output_logits_normalized)).reshape([1,1])
                sampled_words.append(sampled_word)
            text = tokenizer.decode(np.array(sampled_words).reshape(-1))

            # Flatten the tokens
            loss_fct = CrossEntropyLoss(ignore_index=-1)
            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
                            shift_labels.view(-1))
            print("CE Loss:", loss.cpu().data.numpy())
            l = Lyrics(text=text, language='en-us', lookback=15)
            rl = l.get_avg_rhyme_length()
            beta = 1
            rl_loss = rl * beta
            print("RL loss: ", rl_loss)
            total_loss = loss * 1/rl_loss
            print("Total loss: ", total_loss.cpu().data.numpy())
            outputs = (total_loss,) + outputs

        return outputs  # (loss), lm_logits, presents, (all hidden_states), (attentions)

But after evaluation, every model checkpoint returns the same loss on the test set, so it seems that parameters are never updated. Could you please tell me why and how I could solve this?

Thank you a lot.


distilroberta-base unavailable in pip install transformers

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,

Would you please update the `pip install transformers` with the addition of `distilroberta-base`?

As of 28 Nov 2019, I tried `pip install transformers` or `pip install --upgrade transformers` but the `distilroberta-base` model is not available. But I can see it from the master branch and if I install it from source, it will work btw.

Thanks!


Undefined behavior

## 🐛 Bug

There is an undefined behavior in `get_from_cache()` method in `transformers/transformers/file_utils.py`:
```python3
if not os.path.exists(cache_path) and etag is None:
    matching_files = fnmatch.filter(os.listdir(cache_dir), filename + '.*')
    matching_files = list(filter(lambda s: not s.endswith('.json'), matching_files))
    if matching_files:
        cache_path = os.path.join(cache_dir, matching_files[-1])
```

According to [docs](https://docs.python.org/3/library/os.html)  `os.listdir()`
> Return a list containing the names of the entries in the directory given by path. The list is in **arbitrary order**, ...

so taking last element from list returned by `os.listdir()` in the last row of snippet doesn't make sense because of arbitrary order. A possible solution is to add `sorted()`:
```python3
cache_path = os.path.join(cache_dir, sorted(matching_files)[-1])
```
I can make a PR if you agree.


Error while importing RoBERTa model

I tried to import RoBERTa model.

But running the following snippet:
# Load the model in fairseq
`from fairseq.models.roberta import RobertaModel`
`roberta = RobertaModel.from_pretrained('./roberta.large', checkpoint_file='model.pt')`
`roberta.eval() # disable dropout (or leave in train mode to finetune)`

I got the following error:
`RuntimeError: Error(s) in loading state_dict for RobertaModel: Missing key(s) in state_dict: "decoder.sentence_encoder.layers.0.self_attn.k_proj.weight", "decoder.sentence_encoder.layers.0.self_attn.k_proj.bias", "decoder.sentence_encoder.layers.0.self_attn.v_proj.weight", "decoder.sentence_encoder.layers.0.self_attn.v_proj.bias", "decoder.sentence_encoder.layers.0.self_attn.q_proj.weight", "decoder.sentence_encoder.layers.0.self_attn.q_proj.bias", "decoder.sentence_encoder.layers.1.self_attn.k_proj.weight", "decoder.sentence_encoder.layers.1.self_attn.k_proj.bias", "decoder.sentence_encoder.layers.1.self_attn.v_proj.weight", "decoder.sentence_encoder.layers.1.self_attn.v_proj.bias", "decoder.sentence_encoder.layers.1.self_attn.q_proj.weight", "decoder.sentence_encoder.layers.1.self_attn.q_proj.bias", "decoder.sentence_encoder.layers.2.self_attn.k_proj.weight", "decoder.sentence_encoder.layers.2.self_attn.k_proj.bias", "decoder.sentence_encoder.layers.2.self_attn.v_proj.weight", "decoder.sentence_encoder.layers.2.self_attn.v_proj.bias", "decoder.sentence_encoder.layers.2.self_attn.q_proj.weight", "decoder.sentence_encoder.layers.2.self_attn.q_proj.bias", "decoder.sentence_encoder.layers.3.self_attn.k_proj.weight", "decoder.sentence_encoder.layers.3.self_attn.k_proj.bias", "decoder.sentence_encoder.layers.3.self_attn.v_proj.weight", "decoder.sentence_encoder.layers.3.self_attn.v_proj.bias", "decoder.sentence_encoder.layers.3.self_attn.q_proj.weight", "decoder.sentence_encoder.layers.3.self_attn.q_proj.bias", "decoder.sentence_encoder.... Unexpected key(s) in state_dict: "decoder.sentence_encoder.layers.0.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.0.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.1.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.1.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.2.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.2.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.3.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.3.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.4.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.4.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.5.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.5.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.6.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.6.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.7.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.7.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.8.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.8.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.9.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.9.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.10.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.10.self_attn.in_proj_bias", "decoder.sentence_encoder.layers.11.self_attn.in_proj_weight", "decoder.sentence_encoder.layers.11.self_attn.in_proj_bi...`

Is it related to the above error? How can we fix it? Using the hub I get the same error.


Maximum length of out put generated in run_generation.py is of length (1021 ) despite changing position id length and length parameter

I would like to generate a text of about 3000 words.
However the run_generation.py file limits it to 1024, and produces only 1021 words. I have tried changing the internal parameters for the same but in vain.


how to use  BertForMaskedLM

Hi
I want to use  BertForMaskedLM as a decoder, apparently I need to give ids and then this function generates the ids and computes the loss. could you tell me how the generation with this function work? I see for instance in run_generation.py codes you use neucleus sampling or beam search, I see none of them used here, could you explain how this works. Also I want to see the generated sequence as text, could you tell me how from the output of this function I can get this information? thanks


How to compute loss with HuggingFace transformers?

Hello,

Is it possible to train HuggingFace TransfoXLLMHeadModel on a dataset different than WikiText103, say, on the combined WikiText2 and WikiText103 dataset?

Below are my code:
```js

# Import packages
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import TransfoXLConfig, TransfoXLTokenizer, TransfoXLLMHeadModel
from transformers import AdamW, WarmupLinearSchedule
import spacy
import torchtext
from torchtext.data.utils import get_tokenizer
from torchtext.data import Field, BPTTIterator, TabularDataset
import tensorflow as tf
import math
import random
import numpy as np
import pandas as pd
import time

# set hyperparameters for this experiment
bptt = 30
batch_size = 64
lr = 0.01 # learning rate
criterion = nn.CrossEntropyLoss() # loss criterion

# define tokenizer
en = spacy.load('en')

def Sp_Tokenizer(text):
    return [tok.text for tok in en.tokenizer(text)]

# define the English text field
TEXT = Field(tokenize = Sp_Tokenizer,
             init_token='< sos >',
             eos_token='< eos >',
             unk_token='< unk >',
             tokenizer_language='en',
             lower=True)

# load the datasets
train_Wiki2, val_Wiki2, test_Wiki2 = torchtext.datasets.WikiText2.splits(TEXT)
train_Wiki103, val_Wiki103, test_Wiki103 = torchtext.datasets.WikiText103.splits(TEXT)

# Define device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# build vocabulary based on the defined field and
# the combined WikiText2 and WikiText103 datasets.
TEXT.build_vocab(train_Wiki2, val_Wiki2, test_Wiki2,
                 train_Wiki103, val_Wiki103, test_Wiki103)

# set hyperparameter ntokens
ntokens = len(TEXT.vocab.stoi)

## specify the transformer-XL model that we are going to use.
#
# define transformer-XL configuration.
transfoXLconfig = TransfoXLConfig(vocab_size_or_config_json_file = ntokens,
                                  cutoffs = [20000, 40000, 200000],
                                  d_model = 64,
                                  d_embed = 64,
                                  n_head = 16,
                                  d_head = 64,
                                  n_layer = 5,
                                  attn_type = 0,
                                  dropout = 0.1,
                                  output_hidden_states = True,
                                  output_attentions = True)

# define the transformer-XL model based on the specified configuration.
model = TransfoXLLMHeadModel(transfoXLconfig)

# add new tokens to the embeddings of our model
model.resize_token_embeddings(ntokens)

# define BPTTiterators
#
train_iter, val_iter, test_iter = BPTTIterator.splits(
            (train_Wiki2, val_Wiki2, test_Wiki2),
            batch_size = batch_size,
            bptt_len= bptt,
            sort_key=lambda x: len(x.text),
            sort_within_batch = True,
            shuffle = False,
            device= device,
            repeat=False)

train = next(iter(train_iter))
val = next(iter(train_iter))
test = next(iter(test_iter))

```

and now I am trying to write the train function but I am not sure how exactly I should proceed.
Below is what I tried:

```js

# define the hyperparameters for running the train function.
train = train
optimizer = AdamW(model.parameters())
scheduler = WarmupLinearSchedule(optimizer = optimizer,
                                 warmup_steps = 200,
                                 t_total = 1000,
                                 last_epoch = -1)
model.train()

# define the train function
def train(model, train, bptt, criterion, optimizer, scheduler, ntokens, log_interval):

    # initialize total_loss to 0
    total_loss = 0
    # measure the computation time
    start_time = time.time()
    # number of tokens in the vocabulary
    ntokens = ntokens

    for i in range(train.text.size()[1]):

        batch = i
        input_ids, targets = train.text[:,i], train.target[:,i]
        input_ids = torch.tensor(input_ids.tolist()).unsqueeze(0)
        targets = torch.tensor(targets.tolist()).unsqueeze(0)
        optimizer.zero_grad()

        # I intend this 'output' to be the final output of the Transformer-XL....
        output = model(input_ids)
       #... to execute this line
        loss = criterion(output.view(-1, ntokens), targets)

        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()
```

But I don't think the line `loss = criterion(output.view(-1, ntokens), targets)` shouldn't work since the line `output = model(input_ids)` does not actually give out the final output from the model, but it rather gives out (according to the HuggingFace documentation) prediction_scores, mems, attention, etc. How can I train TransfoXLLMHeadModel on a dataset different than just WikiText103?

Thank you,


How to use custom built Torchtext vocabulary with HuggingFace TransfoXLLMHeadModel?

Hello,

I am trying to use my custom built vocabulary which I defined using Torchtext functions with the HuggingFace TransfoXLLMHeadModel, and I am having some troubles with it.
I defined my text field as below:
```js

# Import packages
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import TransfoXLConfig, TransfoXLTokenizer, TransfoXLLMHeadModel
from transformers import AdamW, WarmupLinearSchedule
import spacy
import torchtext
from torchtext.data.utils import get_tokenizer
from torchtext.data import Field, BPTTIterator, TabularDataset
import tensorflow as tf
#import lineflow as lf
#import lineflow.datasets as lfds
import math
import random
import numpy as np
import pandas as pd
import time

# define tokenizer
en = spacy.load('en')

def Sp_Tokenizer(text):
    return [tok.text for tok in en.tokenizer(text)]

# define the English text field
TEXT = Field(tokenize = Sp_Tokenizer,
             init_token='< sos >',
             eos_token='< eos >',
             unk_token='< unk >',
             tokenizer_language='en',
             lower=True)

# load WikiText-2 dataset and split it into train and test set
train_Wiki2, val_Wiki2, test_Wiki2 = torchtext.datasets.WikiText2.splits(TEXT)
train_Wiki103, val_Wiki103, test_Wiki103 = torchtext.datasets.WikiText103.splits(TEXT)
train_Penn, val_Penn, test_Penn = torchtext.datasets.PennTreebank.splits(TEXT)

# build custom vocabulary based on the field that we just defined.
TEXT.build_vocab(train_Wiki2, val_Wiki2, test_Wiki2,
                 train_Wiki103, val_Wiki103, test_Wiki103,
                 train_Penn, val_Penn, test_Penn)
```
and then I defined the HuggingFace transformer's configuration as below:
```js

# set hyperparameter ntokens
ntokens = len(TEXT.vocab.stoi)

# define transformer-XL configuration.
transfoXLconfig = TransfoXLConfig(vocab_size_or_config_json_file = ntokens,
                                  cutoffs = [20000, 40000, 200000],
                                  d_model = 64,
                                  d_embed = 64,
                                  n_head = 16,
                                  d_head = 64,
                                  n_layer = 5,
                                  attn_type = 0,
                                  dropout = 0.1,
                                  output_hidden_states = True,
                                  output_attentions = True)

# define the transformer-XL model based on the specified configuration.
model = TransfoXLLMHeadModel(transfoXLconfig)

# add new tokens to the embeddings of our model
model.resize_token_embeddings(ntokens)
```
and then I want to somehow specify that I want to use my `TEXT.vocab` that I defined earlier via Torchtext for my vocabulary along with the TransfoXLLMHeadModel, but I am not sure how to do this. Can someone help me on this? Thank you!


Why DistilBertTokenizer and BertTokenizer are creating different number of features??

Hi,

I tried working with DistilBertTokenizer and BertTokenizer  from transformers. And according to documentation DistilBertTokenizer was identical to the BertTokenizer . But while creating features for a particular Dataset it creates different number of examples. Why? I also tried using distilbert model with BertTokenizer but still it doesnot work.

could you please explain me this ?? or How can i get same number of features ??


Add Transformer-XL fine-tuning support.

## 🚀 Feature

Add Transformer-XL fine-tuning support.

## Motivation

This model archieves good language modeling result while having a "saner" number of parameters compared with GPT-2 or other language modeling.


how can I pre-training my own model from the existed model or from scratch

## ❓ Questions & Help

I want to load the pre-training model like bert offered by google,and train language model on more corpus,how can I do it ?thanks


Installation error :Command "python setup.py egg_info" failed with error code 1

[puttyerrortransformers2.log](https://github.com/huggingface/transformers/files/3774258/puttyerrortransformers2.log)
## 🐛 Bug

Hello Everyone,

I am trying to install transformers using the command:

pip3 install -v --no-binary :all: --prefix=/short/oe7/uk1594 transformers

 * Python version: Python 3.6.7
* PyTorch version:1.12.0
* CentOS release 6.10 (Final)

I get the below error:

 Using cached https://files.pythonhosted.org/packages/1b/87/c3c2fa8cbec61fffe031ca9f0da512747520bec9be7f886f748457daac31/sentencepiece-0.1.83.tar.gz
  Downloading from URL https://files.pythonhosted.org/packages/1b/87/c3c2fa8cbec61fffe031ca9f0da512747520bec9be7f886f748457daac31/sentencepiece-0.1.83.tar.gz#sha256=d194cf7431dd87798963ff998380f1c02ff0f9e380cc922a07926b69e21c4e2b (from https://pypi.org/simple/sentencepiece/)
  Running setup.py (path:/short/oe7/uk1594/tmp/pip-install-5c0k51ol/sentencepiece/setup.py) egg_info for package sentencepiece
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/short/oe7/uk1594/tmp/pip-install-5c0k51ol/sentencepiece/setup.py", line 29, in <module>
        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
      File "/apps/python3/3.6.7/lib/python3.6/codecs.py", line 897, in open
        file = builtins.open(filename, mode, buffering)
    FileNotFoundError: [Errno 2] No such file or directory: '../VERSION'
Cleaning up...
  Removing source in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/transformers
  Removing source in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/boto3
  Removing source in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/requests
  Removing source in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/tqdm
  Removing source in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/regex
  Removing source in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/sentencepiece
Command "python setup.py egg_info" failed with error code 1 in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/sentencepiece/
Exception information:
Traceback (most recent call last):
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 324, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 154, in prep_for_dist
    self.req.run_egg_info()
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 486, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/apps/python3/3.6.7/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 698, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /short/oe7/uk1594/tmp/pip-install-5c0k51ol/sentencepiece/

Please find the logs attached.

Appreciate your help.

Thanks.
[puttyerrortransformers2.log](https://github.com/huggingface/transformers/files/3774257/puttyerrortransformers2.log)


## 🐛 Bug

I can't seem to get ctrl generation working.

This is with a pull of the repo from master, and pip3 install as recommended during installation:

The problem arise when using:
* [ X ] the official example scripts:

```bash
$ uname -a
Linux ctrl 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```bash
python3 ./examples/run_generation.py     --model_type=ctrl     --length=20     --model_name_or_path=ctrl     --temperature=0     --repetition_penalty=1.2
/home/vessenes/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/vessenes/.local/lib/python3.5/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
10/25/2019 22:11:28 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json from cache at /home/vessenes/.cache/torch/transformers/a858ad854d3847b02da3aac63555142de6a05f2a26d928bb49e881970514e186.285c96a541cf6719677cfb634929022b56b76a0c9a540186ba3d8bbdf02bca42
10/25/2019 22:11:28 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt from cache at /home/vessenes/.cache/torch/transformers/aa2c569e6648690484ade28535a8157aa415f15202e84a62e82cc36ea0c20fa9.26153bf569b71aaf15ae54be4c1b9254dbeff58ca6fc3e29468c4eed078ac142
10/25/2019 22:11:29 - INFO - transformers.configuration_utils -   loading configuration file https://storage.googleapis.com/sf-ctrl/pytorch/ctrl-config.json from cache at /home/vessenes/.cache/torch/transformers/d6492ca334c2a4e079f43df30956acf935134081b2b3844dc97457be69b623d0.1ebc47eb44e70492e0c20494a084f108332d20fea7fe5ad408ef5e7a8f2baef4
10/25/2019 22:11:29 - INFO - transformers.configuration_utils -   Model config {
  "attn_pdrop": 0.1,
  "dff": 8192,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "from_tf": false,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-06,
  "n_ctx": 512,
  "n_embd": 1280,
  "n_head": 16,
  "n_layer": 48,
  "n_positions": 50000,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 246534
}
10/25/2019 22:00:19 - INFO - transformers.modeling_utils -   loading weights file https://storage.googleapis.com/sf
-ctrl/pytorch/seqlen256_v1.bin from cache at /home/vessenes/.cache/torch/transformers/c146cc96724f27295a0c3ada1fbb3
632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
10/25/2019 22:01:17 - INFO - __main__ -   Namespace(device=device(type='cpu'), length=20, model_name_or_path='ctrl'
, model_type='ctrl', n_gpu=0, no_cuda=False, padding_text='', prompt='', repetition_penalty=1.2, seed=42, stop_toke
n=None, temperature=0.0, top_k=0, top_p=0.9, xlm_lang='')
Model prompt >>> Link Thid is a test article
Traceback (most recent call last):
  File "./examples/run_generation.py", line 256, in <module>
    main()
  File "./examples/run_generation.py", line 228, in main
    if not any(context_tokens[0] == x for x in tokenizer.control_codes.values()):
AttributeError: 'CTRLTokenizer' object has no attribute 'control_codes'
```

## Expected behavior

I expect to be able to type in a prompt and see text generated.

## Environment

* OS: Debian 4.9
* Python version: 3.5.3
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): not sure how to find this info
* Using GPU ?  Yes - V100 on GCP
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Training DistilBert - RuntimeError: index out of range at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237

## 🐛 Bug

Hello,

<!-- Important information -->

Model I am using (Bert, XLNet....): DistilBert

Language I am using the model on (English, Chinese....): French

The problem arise when using:
* [ ] the official example scripts: examples/distillation/train.py

The tasks I am working on is:
* [ ] the official training DistilBert from scratch task

## To Reproduce

I followed the required steps to train distil* from scratch :

```bash
python ./scripts/binarized_data.py \
        --file_path ./data/dataset.txt \
        --tokenizer_type bert \
        --tokenizer_name bert-base-multilingual-cased \
        --dump_file ./data_output/binarized_text &
```

The only modification I made was to increase the vocab_size, otherwise I had a bug:
```bash
python ./scripts/token_counts.py \
    --data_file ./data/binarized_text.bert-base-multilingual-cased.pickle \
    --token_counts_dump ./data/token_counts.bert-base-multilingual-cased.pickle \
    --vocab_size 65536
```

Then, I launched the training with the following :
```bash
python train.py \
    --student_type distilbert \
    --student_config ./training_configs/distilbert-base-uncased.json \
    --teacher_type bert \
    --teacher_name bert-base-multilingual-cased \
    --alpha_ce 5.0 --alpha_mlm 2.0 --alpha_cos 1.0 --mlm \
    --dump_path ./serialization_dir/my_first_training \
    --data_file ./data/binarized_text.bert-base-multilingual-cased.pickle \
    --token_counts ./data/token_counts.bert-base-multilingual-cased.pickle \
    --force
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

Error message :
```bash
-Iter:   0% 0/586181 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train.py", line 289, in <module>
    main()
  File "train.py", line 284, in main
    distiller.train()
  File "/dds/work/distil/transformers/examples/distillation/distiller.py", line 339, in train
    self.step(input_ids=token_ids, attention_mask=attn_mask, lm_labels=lm_labels)
  File "/dds/work/distil/transformers/examples/distillation/distiller.py", line 369, in step
    s_logits, s_hidden_states = self.student(input_ids=input_ids, attention_mask=attention_mask)     # (bs, seq_length, voc_size)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/transformers/modeling_distilbert.py", line 528, in forward
    head_mask=head_mask)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/transformers/modeling_distilbert.py", line 461, in forward
    embedding_output = self.embeddings(input_ids)   # (bs, seq_length, dim)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/transformers/modeling_distilbert.py", line 92, in forward
    word_embeddings = self.word_embeddings(input_ids)                   # (bs, max_seq_length, dim)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/dds/miniconda/envs/dds/lib/python3.5/site-packages/torch/nn/functional.py", line 1467, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 61578 out of table with 30521 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237
-Iter:   0% 0/586181 [00:00<?, ?it/s]
```

## Environment

* OS: Debian
* Python version: 3.5
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? No
* Distributed of parallel setup ?
* Any other relevant information:

Thanks in advance !


How to initialize AdamW optimizer in HuggingFace transformers?

Hello,
I am new to Python and NLP and so I have some questions that may sound a bit funny to the experts.

I had been trying to set my optimizer by setting
`optimizer = AdamW()`

but of course it failed, because I did not specify the required parameter `'param'` (for lr, betas, eps, weight_decay, and correct_bias, I am just going to use the default values).

As a beginner, I am not so clear on what `'param'` stands for in this case. What kind of input should I provide for `'param'`?

Thank you,


Fix for mlm evaluation in run_lm_finetuning.py

No masking is done in the original evaluation code, so the resulting perplexity is always something like 1.0. In this PR a simple fix is proposed, using just the same masked scheme as in a training code.