CUDA out of memory for 8x V100 GPU

```
python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-cased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_train_batch_size 24 \
    --gradient_accumulation_steps 12
```
We are trying the same command (except  bert-base-cased, we are using bert-large-uncased-whole-word-masking) on 8x V100 GPU but getting CUDA out of memory error (CUDA out of memory. Tried to allocate 216.00 MiB....)

As per the https://github.com/huggingface/transformers/tree/master/examples it should work but it's giving error and stopping in the middle. Any tips would be appreciated.


ALBERT how to obtain the embedding matrix?

## ❓ Questions & Help

Hi, I'm trying to use ALBERT for word embedding with this library.
ALBERT's doc mentioned an embedding size of 128 independently of the model version (base, large, ...) while the hidden_size changes.
I would like to obtain the 128 word (or subword) vectors but the model gives me only the output of the last hidden state (so for xxlarge a 4096 tensor for each token).

What am I doing wrong?


ImportError: cannot import name 'WarmupLinearSchedule'

$ pip show transformers
Name: transformers
Version: 2.2.1
Summary: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch
Home-page: https://github.com/huggingface/transformers
Author: Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors
Author-email: thomas@huggingface.co
License: Apache
Location: /home/ubuntu/anaconda3/lib/python3.6/site-packages
Requires: numpy, requests, regex, sacremoses, tqdm, sentencepiece, boto3
Required-by:


handle string with only whitespaces as empty


Encoding special tokens

## 🐛 Bug

<!-- Important information -->
In version 2.2.1 encoding special tokens changed.

```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer.decode(tokenizer.encode("[CLS] hello world [SEP]", add_special_tokens=False))
```
output: `'[ cls ] hello world [ sep ]'`

For version `transformers==2.2.0` the output is: `'[CLS] hello world [SEP]'`


Hi~

How to average sub-words embeddings to obtain word embeddings?
I only want word-level embedding instead of sub-word-level, how can I get them?

Is there any tokenizer that provides a method that can output the index/mask of sub-words or something?


[cli] Uploads: add progress bar

see https://github.com/huggingface/transformers/pull/2044#discussion_r354057827 for context

There might be a more pythonic way (to do a "simple" method overriding) but I couldn't find it.


corrected documentation for past tensor shape for ctrl and gpt2 model

fix issue #1904


Text Generation in Hebrew

## ❓ Questions & Help

Hi all,

I have 30K tweets in Hebrew and I want to create a sort of chatbot that will answer in the style of those tweets, similar to [this](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313). The only multilingual models that were trained on Hebrew are BERT and XLM, and they are both MLMs which are not too good at text generation. I thought I could fine-tune XLM and then run `run_generation.py`, but `run_lm_finetuning.py` doesn't support XLM.

Is there a way I can go about my task?

Thanks!


Check link validity

We would like to make sure that every download link in the code base works. The best way to do this is to check automatically with the CI; this also prevents us from merging code with broken links.

This PR adds a small script that:

- Lists all source code files
- Extracts links with a regexp
- Performs HEAD requests to check the validity of each link
- Returns an error if at least one link is broken, along with the list of all broken links.

I also add a Circle CI workflow `repository-consistency` with a small machine that runs this script. It could be used to enforce things such as coding styles etc in the future.

For now the links are checked sequentially; if it turns out to take too long we can use `aiohttp` to run the queries concurrently.

_Edit:_ commits squashed


Check the validity of download links

We would like to make sure regularly that every download link in the codebase works. The best way to do this is to check automatically with the CI; this also prevents us from merging code with broken links.

This PR adds a small script that:
- Lists all source code files
- Extract links with a regexp
- Perform HEAD requests to check the validity of each links
- Returns an error if at least one link is broken, along with the list of all broken links.

I also add a Circle CI workflow `repository-consistency` with a small machine that runs this script. It could be used to enforce things such as coding styles etc in the future.


How to structure text data to finetune distilGPT2 using tf.keras.model.fit()?

here is the relevant section of code where I get my text data via a txt file "file_path":

```
examples=[]

with open(file_path, encoding="utf-8") as f:
    text = f.read()

tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))

block_size = 256
for i in range(0, len(tokenized_text)-block_size+1, block_size): # Truncate in block of block_size
    examples.append(tokenized_text[i:i+block_size])
```

This looks to be the way it is structured in the run_lm_finetuning.py script?

then:

```
dataset = tf.data.Dataset.from_tensor_slices(examples)

BATCH_SIZE = 32
BUFFER_SIZE = 10000
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=[loss, None, None, None, None, None, None], metrics=[metric])
model.fit(dataset, epochs=1)
```

and I get this error:
```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-15-acd4de99eacb> in <module>
----> 1 model.fit(dataset1, epochs=10)

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    726         max_queue_size=max_queue_size,
    727         workers=workers,
--> 728         use_multiprocessing=use_multiprocessing)
    729
    730   def evaluate(self,

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
    322                 mode=ModeKeys.TRAIN,
    323                 training_context=training_context,
--> 324                 total_epochs=epochs)
    325             cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)
    326

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
    121         step=step, mode=mode, size=current_batch_size) as batch_logs:
    122       try:
--> 123         batch_outs = execution_function(iterator)
    124       except (StopIteration, errors.OutOfRangeError):
    125         # TODO(kaftan): File bug about tf function and errors.OutOfRangeError?

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in execution_function(input_fn)
     84     # `numpy` translates Tensors to values in Eager mode.
     85     return nest.map_structure(_non_none_constant_value,
---> 86                               distributed_function(input_fn))
     87
     88   return execution_function

~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in __call__(self, *args, **kwds)
    455
    456     tracing_count = self._get_tracing_count()
--> 457     result = self._call(*args, **kwds)
    458     if tracing_count == self._get_tracing_count():
    459       self._call_counter.called_without_tracing()

~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _call(self, *args, **kwds)
    501       # This is the first call of __call__, so we have to initialize.
    502       initializer_map = object_identity.ObjectIdentityDictionary()
--> 503       self._initialize(args, kwds, add_initializers_to=initializer_map)
    504     finally:
    505       # At this point we know that the initialization is complete (or less

~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    406     self._concrete_stateful_fn = (
    407         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 408             *args, **kwds))
    409
    410     def invalid_creator_scope(*unused_args, **unused_kwds):

~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   1846     if self.input_signature:
   1847       args, kwargs = None, None
-> 1848     graph_function, _, _ = self._maybe_define_function(args, kwargs)
   1849     return graph_function
   1850

~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   2148         graph_function = self._function_cache.primary.get(cache_key, None)
   2149         if graph_function is None:
-> 2150           graph_function = self._create_graph_function(args, kwargs)
   2151           self._function_cache.primary[cache_key] = graph_function
   2152         return graph_function, args, kwargs

~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2039             arg_names=arg_names,
   2040             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2041             capture_by_value=self._capture_by_value),
   2042         self._function_attributes,
   2043         # Tell the ConcreteFunction to clean up its graph once it goes out of

~/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    913                                           converted_func)
    914
--> 915       func_outputs = python_func(*func_args, **func_kwargs)
    916
    917       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    356         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    357         # the function a weak reference to itself to avoid a reference cycle.
--> 358         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    359     weak_wrapped_fn = weakref.ref(wrapped_fn)
    360

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in distributed_function(input_iterator)
     71     strategy = distribution_strategy_context.get_strategy()
     72     outputs = strategy.experimental_run_v2(
---> 73         per_replica_function, args=(model, x, y, sample_weights))
     74     # Out of PerReplica outputs reduce or pick values to return.
     75     all_outputs = dist_utils.unwrap_output_dict(

~/.local/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py in experimental_run_v2(self, fn, args, kwargs)
    758       fn = autograph.tf_convert(fn, ag_ctx.control_status_ctx(),
    759                                 convert_by_default=False)
--> 760       return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    761
    762   def reduce(self, reduce_op, value, axis):

~/.local/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py in call_for_each_replica(self, fn, args, kwargs)
   1785       kwargs = {}
   1786     with self._container_strategy().scope():
-> 1787       return self._call_for_each_replica(fn, args, kwargs)
   1788
   1789   def _call_for_each_replica(self, fn, args, kwargs):

~/.local/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py in _call_for_each_replica(self, fn, args, kwargs)
   2130         self._container_strategy(),
   2131         replica_id_in_sync_group=constant_op.constant(0, dtypes.int32)):
-> 2132       return fn(*args, **kwargs)
   2133
   2134   def _reduce_to(self, reduce_op, value, destinations):

~/.local/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    290   def wrapper(*args, **kwargs):
    291     with ag_ctx.ControlStatusCtx(status=ag_ctx.Status.DISABLED):
--> 292       return func(*args, **kwargs)
    293
    294   if inspect.isfunction(func) or inspect.ismethod(func):

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in train_on_batch(model, x, y, sample_weight, class_weight, reset_metrics)
    262       y,
    263       sample_weights=sample_weights,
--> 264       output_loss_metrics=model._output_loss_metrics)
    265
    266   if reset_metrics:

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py in train_on_batch(model, inputs, targets, sample_weights, output_loss_metrics)
    309           sample_weights=sample_weights,
    310           training=True,
--> 311           output_loss_metrics=output_loss_metrics))
    312   if not isinstance(outs, list):
    313     outs = [outs]

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py in _process_single_batch(model, inputs, targets, output_loss_metrics, sample_weights, training)
    250               output_loss_metrics=output_loss_metrics,
    251               sample_weights=sample_weights,
--> 252               training=training))
    253       if total_loss is None:
    254         raise ValueError('The model cannot be run '

~/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py in _model_loss(model, inputs, targets, output_loss_metrics, sample_weights, training)
    164
    165         if hasattr(loss_fn, 'reduction'):
--> 166           per_sample_losses = loss_fn.call(targets[i], outs[i])
    167           weighted_losses = losses_utils.compute_weighted_loss(
    168               per_sample_losses,

IndexError: list index out of range
```

Any ideas? it looks like maybe I'm supposed to provide labels? I could not find the relevant section of run_lm_finetuning.py that deals with that.


Accessing roberta embeddings

## Finetune Setup
* Model: roberta-base
* Language: english
* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Script inputs:
```
python run_lm_finetuning.py \
--output_dir=$OUTPUT_DIR \
--model_type=roberta \
--model_name_or_path=roberta_base \
--do_train \
--train_data_file=$TRAIN_FILE \
--do_eval \
--eval_data_file=$TEST_FILE \
--mlm \
--no_cuda
```
## ❓ Questions & Help
I would like to compare the embeddings of a sentence produced by `roberta-base` and my finetuned model (which is based on roberta-base using my domain specific data), but I am not sure how to access them. Any pointers on how to do this? Thanks in advance.


## 🚀 Feature

Hey, thanks for everything,
The generation script could fail when there's a double space in the prompt, e.g. " I  go to"
![image](https://user-images.githubusercontent.com/1544039/70262743-e4bc4900-1762-11ea-9041-9bee082a0054.png)

I know it's not important, but it would be good if the tokenize is more "robust"


XLMWithLMHeadModel forwarding questions

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
1. Why is the labels argument named 'labels' instead of 'masked_lm_labels' like in BertForMaskedLM?

2. When I change labels for masked tokens to -1 as suggested in documentation, I got an error from NLLLoss for label being outside valid num_classes range. When I instead change labels for masked tokens to -100 (default ignore_index), it seems to work. Why is this happening?


clean up PT <=> TF conversion

Cleaning up PT <=> TF conversion method.

cc @VictorSanh


Nicer error message when Bert's input is missing batch size

Currently it fails in the computation of the attention_mask.

Let's fail with a shape error message instead.


Save model for tensorflow serving

Hello,
Thanks for the library. I tried your Multi label classification. I trained it with my data. It worked very accurate and fast. Now i want to use this model with tensorflow. I am new on pytorch and i looked some tutorials. As i understand i need to save model then convert to Onnx then to tensorflow. So I tried to save model first but it gave me this error;
```
AttributeError                            Traceback (most recent call last)
<ipython-input-10-4f1e1257e6e8> in <module>()
      1
----> 2 torch.save(model.state_dict(), 'output/multilabel.pth')

AttributeError: 'MultiLabelClassificationModel' object has no attribute 'state_dict'
```

How can i save the model and export to onnx?


CPU RAM out of memory when detach from GPU

## ❓ Questions & Help

I am using the following code to get embedding layer from BERT:
```
class BertEmbedding():
    def __init__(self, load_model=None, load_config=None, model='bert-base-uncased', max_len=512, batch_size=6):
        self.pre_trained_model = model
        self.max_len = max_len
        self.batch_size = batch_size
        self.model = BertModel.from_pretrained(self.pre_trained_model)
        self.tokenizer = BertTokenizer.from_pretrained(self.pre_trained_model)
        #self.optimizer = AdamW(params = self.model.parameters(), lr=1e-5)

    def create_ids(self, sentences):
        logging.getLogger("transformers.tokenization_utils").setLevel(logging.ERROR) #Disable tokenizer logs, it's really annoy
        input_ids = []
        for sen in tqdm_notebook(sentences, desc="Create Ids"):
            tmp = self.tokenizer.encode(sen)
            input_ids.append(tmp)
        input_ids = pad_sequences(input_ids,
                                  maxlen=self.max_len,
                                  dtype='int64',
                                  truncating='post',
                                  padding='post')
        return input_ids

    def generate(self, inputs):
        test_ids = self.create_ids(inputs)
        test_dataloader = DataLoader(torch.tensor(test_ids), batch_size=self.batch_size)

        embedding = []
        self.model.to(device)
        self.model.eval()
        for input_ids in tqdm_notebook(test_dataloader, desc="Generating"):
            with torch.no_grad():
                last_state = self.model(input_ids.to(device))[0]
            last_state = last_state.detach().cpu().numpy()
            embedding.extend(last_state)
        return embedding

bert_embedding = BertEmbedding(batch_size=100)

embedding = bert_embedding.generate(train.sentence.values)
```

The problem is when it generate embedding layer from model (train on GPU and detach to CPU), RAM is increasing significantly (1GB --> 30GB for a list of 25,000 arrays (512,768)). While I checked with `sys.getsizeof(embedding)` = `224208` and size of `bert_embedding` is `56` only.

If I delete both `embedding` and `bert_embedding`, RAM ~ 20GB. I think that the model is still existing. How can I optimize this for CPU RAM?


Fixing camembert tokenization

The original fairseq implmentation of Camembert has a bunch of duplicate tokens in the dictionary, in particular there are two `<unk>` tokens but only the index of the first `<unk>` should be used:
```
import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
list(camembert.task.source_dictionary[i] for i in range(10))
>>> ['<s>', '<pad>', '</s>', '<unk>', '<unk>', '<s>', '</s>', ',', '▁de', '.']
```

This PR updates Camembert tokenizer to fix this behavior and as a consequence fixes #2019 and #2020


[ Structure of LM vocab trained from scratch ]

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,
I am trying to create a BERT LM trained from scratch and I have a question about the tokenizer.
I have a big text corpus and I trained a tokenizer with SentencePiece with 32K as dimension of the vocabulary. Then I applied a transformation from SentencePiece notation to WordPiece notation in order to be compatible with BERT.
The result is that my dictionary has this structure:

----------------------------------------

MY DICTIONARY
```
0 [PAD]
1 [UNK]
2 [CLS]
3 [SEP]
4 [MASK]
5 ,
6 .
7 '
...
.... [ ALL USED TOKENS ]
....
100 ##ndo
...
.... [ ALL USED TOKENS ]
....
31741 [ disgustat ]
31742 [ UNUSED TOKEN]
.......
.......  [ ALL UNUSED TOKENS ]
.......
31999 [ UNUSED TOKEN]
```
And all the unused tokens are at the end of the vocab, in my case from 31742 to 31999.

---------------------------

And this is the STANDARD VOCAB for BERT:

BERT cased_L-12_H-768_A-12 VOCABULARY

```
0  [PAD]
1 [unused1]
2  ........
3  ........ [ ALL UNUSED TOKENS ]
4  .......
...........
100 [UNK]
101 [CLS]
102 [SEP]
103 [MASK]
[unused100]
[unused101]
!
"
..... [ ALL USED TOKENS ]
```
----------------------------

My question is: the fact that the SPECIAL TOKENS in MY DICTIONARY are in different positions than the Standard BERT VOCABULARY , can be a problem? Do you think I should keep the same positions of BERT vocab also for my dictionary? Also for unused token I have the same doubts.

( I saw that in the sentence piece training it's possibile to specify the exact positions of special tokens, but my question is if the position of special tokens will affect the LM training in some way )

Thank you


special_tokens_mask value was unused and calculated twice

In the current master, in the `prepare_for_model` method of the `PreTrainedTokenizer` class, the sepcial_tokens_mask is calculated but not used: https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/transformers/tokenization_utils.py#L904.

```python
        # Handle special_tokens
        if add_special_tokens:
            sequence = self.build_inputs_with_special_tokens(ids, pair_ids)
            token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids)
            special_tokens_mask = self.get_special_tokens_mask(ids, pair_ids)
        else:
            sequence = ids + pair_ids if pair else ids
            token_type_ids = [0] * len(ids) + ([1] * len(pair_ids) if pair else [])
            special_tokens_mask = [0] * (len(ids) + (len(pair_ids) if pair else 0))
        if return_special_tokens_mask:
            encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
```

The proposed change is to use the `special_tokens_mask ` computed in the if/else statement in the output dictionary.


TypeError: argument of type 'PosixPath' is not iterable (in modeling_utils.py)

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):BERT

Language I am using the model on (English, Chinese....):English

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce
TypeError: argument of type 'PosixPath' is not iterable

Steps to reproduce the behavior:

1.install transformers by pip
2.make test function with model=Bert.....    line, put your own values as arguements

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
Traceback (most recent call last):
  File "run_bert.py", line 226, in <module>
    main()
  File "run_bert.py", line 219, in main
    run_train(args)
  File "run_bert.py", line 70, in run_train
    model = BertForMultiLable.from_pretrained(config['bert_model_dir'], num_labels=len(label_list))
  File "/home/aditya/anaconda3/envs/RD/lib/python3.7/site-packages/transformers/modeling_utils.py", line 321, in from_pretrained
    if "albert" in pretrained_model_name_or_path and "v2" in pretrained_model_name_or_path:
TypeError: argument of type 'PosixPath' is not iterable

```
run_train():
```

def run_train(args):
    # --------- data
    processor = BertProcessor(vocab_path=config['bert_vocab_path'], do_lower_case=args.do_lower_case)
    label_list = processor.get_labels()
    label2id = {label: i for i, label in enumerate(label_list)}
    id2label = {i: label for i, label in enumerate(label_list)}

    train_data = processor.get_train(config['data_dir'] / f"{args.data_name}.train.pkl")
    train_examples = processor.create_examples(lines=train_data,
                                               example_type='train',
                                               cached_examples_file=config[
                                                    'data_dir'] / f"cached_train_examples_{args.arch}")
    train_features = processor.create_features(examples=train_examples,
                                               max_seq_len=args.train_max_seq_len,
                                               cached_features_file=config[
                                                    'data_dir'] / "cached_train_features_{}_{}".format(
                                                   args.train_max_seq_len, args.arch
                                               ))
    train_dataset = processor.create_dataset(train_features, is_sorted=args.sorted)
    if args.sorted:
        train_sampler = SequentialSampler(train_dataset)
    else:
        train_sampler = RandomSampler(train_dataset)
    train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size)

    valid_data = processor.get_dev(config['data_dir'] / f"{args.data_name}.valid.pkl")
    valid_examples = processor.create_examples(lines=valid_data,
                                               example_type='valid',
                                               cached_examples_file=config[
                                                                        'data_dir'] / f"cached_valid_examples_{args.arch}")

    valid_features = processor.create_features(examples=valid_examples,
                                               max_seq_len=args.eval_max_seq_len,
                                               cached_features_file=config[
                                                                        'data_dir'] / "cached_valid_features_{}_{}".format(
                                                   args.eval_max_seq_len, args.arch
                                               ))
    valid_dataset = processor.create_dataset(valid_features)
    valid_sampler = SequentialSampler(valid_dataset)
    valid_dataloader = DataLoader(valid_dataset, sampler=valid_sampler, batch_size=args.eval_batch_size)

    # ------- model
    logger.info("initializing model")
    if args.resume_path:
        args.resume_path = Path(args.resume_path)
        model = BertForMultiLable.from_pretrained(args.resume_path, num_labels=len(label_list))
    else:
        model = BertForMultiLable.from_pretrained(config['bert_model_dir'], num_labels=len(label_list))
    t_total = int(len(train_dataloader) / args.gradient_accumulation_steps * args.epochs)

    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],'weight_decay': args.weight_decay},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
    ]
    warmup_steps = int(t_total * args.warmup_proportion)
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    lr_scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=t_total)

    if args.fp16:
        try:
            from apex import amp
        except ImportError:
            raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
        model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)

    # ---- callbacks
    logger.info("initializing callbacks")
    train_monitor = TrainingMonitor(file_dir=config['figure_dir'], arch=args.arch)
    model_checkpoint = ModelCheckpoint(checkpoint_dir=config['checkpoint_dir'],mode=args.mode,
                                       monitor=args.monitor,arch=args.arch,
                                       save_best_only=args.save_best)

    # **************************** training model ***********************
    logger.info("***** Running training *****")
    logger.info("  Num examples = %d", len(train_examples))
    logger.info("  Num Epochs = %d", args.epochs)
    logger.info("  Total train batch size (w. parallel, distributed & accumulation) = %d",
                args.train_batch_size * args.gradient_accumulation_steps * (
                    torch.distributed.get_world_size() if args.local_rank != -1 else 1))
    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
    logger.info("  Total optimization steps = %d", t_total)

    trainer = Trainer(n_gpu=args.n_gpu,
                      model=model,
                      epochs=args.epochs,
                      logger=logger,
                      criterion=BCEWithLogLoss(),
                      optimizer=optimizer,
                      lr_scheduler=lr_scheduler,
                      early_stopping=None,
                      training_monitor=train_monitor,
                      fp16=args.fp16,
                      resume_path=args.resume_path,
                      grad_clip=args.grad_clip,
                      model_checkpoint=model_checkpoint,
                      gradient_accumulation_steps=args.gradient_accumulation_steps,
                      batch_metrics=[AccuracyThresh(thresh=0.5)],
                      epoch_metrics=[AUC(average='micro', task_type='binary'),
                                     MultiLabelReport(id2label=id2label)])
    trainer.train(train_data=train_dataloader, valid_data=valid_dataloader, seed=args.seed)

```

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:Ubuntu 18.04
* Python version:3.6
* PyTorch version:
* PyTorch Transformers version (or branch):2.2.1
* Using GPU ?No
* Distributed of parallel setup ?Non Distrubuted
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

<!-- Add any other context about the problem here. -->
I used this script to load the bert model i had finetuned for a classification task following google_research_bert.
I want to convert those TF checkpoints to pytorch.

```python
config = BertConfig.from_pretrained('bert-base-uncased')
config.num_labels=4
# You will need to load a BertForSequenceClassification model
model = BertForSequenceClassification(config)

tf_checkpoint_path = init_checkpoint
# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)

pytorch_dump_path = "./pytorch_bert_output"
# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
```

When i execute this, i get the following error, BertForSequenceClassification' object has no attribute 'bias. Any leads would be helpful.
Thanks


Pr for pplm

Updated paper link and better commands to generate samples.


How to run a batch of data through BERT model?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I understand how to run **1 data** point of d words through a BERT model, but how can I run **n data** sequence of words through the BERT model?

Nvm solved this issue.

I can just pass sth like a 2xd data that looks like this:
tensor([[2182, 2003, 1996, 6251, 1045, 2215, 7861, 8270, 4667, 2015, 2005,
           1012],
          [2182, 2003, 1996, 6251, 1045, 2215, 7861, 8270, 4667, 2015, 2005,
           1012]], device='cuda:0')
through the BERT forward function


Automatically allocates memory in GPU, always OOM when create TFALBERT model

## 🐛 Bug

<!-- Important information -->

Model I am using :ALBERT

Language I am using the model on (English, Chinese....):English

> from transformers import TFAlbertModel

> model2=TFAlbertModel.from_pretrained('albert-base-v1')

Then:

> ---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-a440a0748e94> in <module>
----> 1 model2=TFAlbertModel.from_pretrained('albert-base-v1')

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    287         # 'by_name' allow us to do transfer learning by skipping/adding layers
    288         # see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1339-L1357
--> 289         model.load_weights(resolved_archive_file, by_name=True)
    290
    291         ret = model(model.dummy_inputs, training=False)  # Make sure restore ops are run

~/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in load_weights(self, filepath, by_name)
    179         raise ValueError('Load weights is not yet supported with TPUStrategy '
    180                          'with steps_per_run greater than 1.')
--> 181     return super(Model, self).load_weights(filepath, by_name)
    182
    183   @trackable.no_automatic_dependency_tracking

~/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py in load_weights(self, filepath, by_name)
   1169           'first, then load the weights.')
   1170     self._assert_weights_created()
-> 1171     with h5py.File(filepath, 'r') as f:
   1172       if 'layer_names' not in f.attrs and 'model_weights' in f:
   1173         f = f['model_weights']

~/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds)
    392                 fid = make_fid(name, mode, userblock_size,
    393                                fapl, fcpl=make_fcpl(track_order=track_order),
--> 394                                swmr=swmr)
    395
    396             if swmr_support:

~/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    168         if swmr and swmr_support:
    169             flags |= h5f.ACC_SWMR_READ
--> 170         fid = h5f.open(name, flags, fapl=fapl)
    171     elif mode == 'r+':
    172         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5f.pyx in h5py.h5f.open()

OSError: Unable to open file (file signature not found)

I tried to trace the GPU state, the memory usage is 15513MiB / 16130MiB, it is obvious that
when I create a model, it automatically allocates memory in GPU, but when I tried this in colab and use the same TF version, it works well, after creating model, there still are much free memory.
*OS: Linux version 4.9.0-11-amd64
* Python version:3.7
* TF version:TF2.0
* Transformers version (or branch):2.2
* Using GPU ?GPU

## Additional context

<!-- Add any other context about the problem here. -->


`distilroberta-base` link missing

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

According to the current master code, link for `distilroberta-base` isn't provided.
https://github.com/huggingface/transformers/blob/1ab8dc44b3d84ed1894f5b6a6fab58fb39298fc7/transformers/configuration_distilbert.py#L28-L33


## ❓ Questions & Help
cannot import name 'get_linear_schedule_with_warmup' from 'transformers.optimization'
<!-- A clear and concise description of the question. -->


Remove dependency on pytest for running tests


Find dot product of query and key vectors

Hi,

I am following [this popular article](http://jalammar.github.io/illustrated-transformer/) to understand the Transformers. Alongside this, I am using [huggingface transformers](https://huggingface.co/transformers/model_doc/bert.html#bertmodel) to get the attention scores.

On running the following code:
`from transformers import BertTokenizer, BertModel, BertConfig, BertForTokenClassification
import torch

config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True, output_attentions=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased', config=config)

input_ids = torch.tensor(tokenizer.encode("Hello my dog is cute", add_special_tokens=False)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)

print(len(outputs))

last_hidden_states, pooler_outputs, hidden_states, attentions = outputs  # The last hidden-state is the first element of the output tuple

print(attentions)`

I get the weighted sum attention matrix of size 5x5. I am actually trying to find the softmax values like 0.88 and 0.12. I was wondering if there is any way I can obtain the dot-product scores.

![image](https://user-images.githubusercontent.com/22553367/70174075-394bbf80-16a2-11ea-972b-4aeb539dbd7e.png)

Thanks!


Crosslingual classification with XLM, loss does not converge

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to use the XLM pretrained model `xlm-mlm-tlm-xnli15-1024` for a cross lingual classification task, but I cannot get the loss to converge and the final accuracy is random.

To check this was not an implementation error of my own doing, I ran the `run_xnli.py` example and found using `xlm-mlm-tlm-xnli15-1024` results in an accuracy of 30% while using `bert-base-multilingual-cased` results in the expected accuracy of 70%.

system config:
```
Platform Linux-4.4.0-1098-aws-x86_64-with-debian-stretch-sid
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0]
PyTorch 1.2.0+cu92
Tensorflow 2.0.0
```


Missing "do_lower_case" action for special token (e.g. mask_token)

## 🐛 Bug

Model I am using (Bert, XLNet....): 'bert-base-uncased'

Language I am using the model on (English, Chinese....): English

After upgrading to 2.2.1 version, the BERT tokenizer cannot tokenize special word while it works in 2.1.1 version.

According to [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_bert.py#L71), 'bert-base-uncased' should perform lower case operation. Inputs follow this config to [perform lower case operation](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L615), while no corresponding action for [special tokens](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L1100). Eventually, it tokenizes '[MASK]' to 3 subwords (e.g. [, mask and ]) rather than skip the tokenization operation in [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L658).

Error occurs after this [commit](https://github.com/huggingface/transformers/commit/7246d3c2f93c4461f3ec8ada7a26a002d8f196ea).

## To Reproduce

Steps to reproduce the behavior:
```
import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer.tokenize('The quick brown [MASK] jumps over the lazy dog.')
```

## Expected behavior

Expected output is
['the', 'quick', 'brown', **'[MASK]'**, 'jumps', 'over', 'the', 'lazy', 'dog', '.']
while actual output is
['the', 'quick', 'brown', **'[', 'mask', ']'**, 'jumps', 'over', 'the', 'lazy', 'dog', '.']


Fix bug which lowercases special tokens

A previous PR (#1592), which lowercases input and added tokens if `do_lower_case` is set to `True` for a given tokenizer, introduced a bug which lowercases text without considering whether parts of the input are special tokens. The result is that special tokens may not be tokenized properly, e.g. "[CLS]" becomes 4 separate tokens when using the BERT tokenizer: "[", "cl", "##s", "]".

This change fixes that by only applying lowercasing to non-special tokens. The do_lower_case test case has also been expanded to use some special token based on the subclass.

Closes #2047


[CamemBert] About SentencePiece training

## ❓ Questions & Help

According to the paper, SentencePiece uses a vocabulary of size of 32k subword tokens, learned on 107 sentences sampled from the pretraining dataset. How the sampling was performed? The chosen size of the vocabulary (32K subwords token) is related to the pretraining dataset in some way? Or it is an arbitrary choice?

Thank you.


## 🐛 Bug
`
Traceback (most recent call last):

  File "train.py", line 32, in <module>

    from distiller import Distiller

  File "~/transformers/examples/distillation/distiller.py", line 40, in <module>

    from utils import logger

  File "~/transformers/examples/distillation/utils.py", line 18, in <module>

    import git

ModuleNotFoundError: No module named 'git'

`
how to install git package?


Changing the number of hidden layers for BERT

## ❓ Questions & Help

Hello,

when reducing the number of hidden layers for BERT, say from 12 to 3, which layers are loaded from the pretrained model, the first 3 layers or the last 3 ones? and is there a way to control this?

Thanks in advance


Tokenization in quickstart guide fails

## 🐛 Bug

<!-- Important information -->

The same issue as in #226 re-appears in transformers==2.2.1 (it works on 2.1!)

I just encountered the same issue as @dhirajmadan1 with `transformers==2.2.1`. Is this expected somehow?

I am following the quickstart guide: https://huggingface.co/transformers/quickstart.html

## To Reproduce

Steps to reproduce the behavior:

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Run an example text through this:
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

masked_index = 8
tokenized_text[masked_index] = '[MASK]'
predicted_tokenized_sentence = ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']

```

## Expected behavior

This should not fail:

```assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']```

## Environment

* OS: Mac
* Python version: 3.6
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.1 (latest-minor)
* Using GPU no
* Distributed of parallel setup no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Add NER TF2 example.

Create a NER example similar to the Pytorch one. It takes the same options, and can be run the same way.

As you asked @julien-c I prefered I did a fresh new PR :)


Remove dead code in tests.


CLI for authenticated file sharing

ping review @mfuntowicz & @thomwolf

(I'll fix the tests for Python 2 and Python 3.5 tomorrow)

To create an account in `staging` (used by the tests): https://moon-staging.huggingface.co/join
To create an account in `production` (used by the CLI): https://huggingface.co/join


Missing xlm-mlm-100-1280

## 🐛 Bug

For some reason I can't download the xlm-mlm-100-1280 model for tensorflow 2.0

Model I am using (Bert, XLNet....): XLM

Language I am using the model on (English, Chinese....): 100 languages

The problem arise when using:
```TFXLMForSequenceClassification.from_pretrained("xlm-mlm-100-1280")```

## Expected behavior

Being able to download the model as for the other configuration

## Environment

* OS: Ubuntu 16.04
* Python version: 3.7.5
* Using GPU : yes
* Distributed of parallel setup : distributed
* Tensorflow 2.0
* transformers version 2.1.1


## Finetuning Setup
* Model: roberta-base
* Language: english
* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Script inputs:
```
python run_lm_finetuning.py \
--output_dir=$OUTPUT_DIR \
--model_type=roberta \
--model_name_or_path=roberta_base \
--do_train \
--train_data_file=$TRAIN_FILE \
--do_eval \
--eval_data_file=$TEST_FILE \
--mlm \
--no_cuda
```

## ❓ Questions & Help
I finetuned roberta on some domain specific data I have and was trying to follow the example in the Quick Tour section for getting the output, however I get the following error:
```python
Traceback (most recent call last):
  File "/path/to/code/roberta_compare.py", line 26, in <module>
    last_hidden_states = model(input_ids)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 246, in forward
    inputs_embeds=inputs_embeds)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 688, in forward
    extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
UnboundLocalError: local variable 'extended_attention_mask' referenced before assignment
```
Here is my code:
```python
import torch
from transformers import RobertaTokenizer, RobertaForMaskedLM

model_class = RobertaForMaskedLM
model = model_class.from_pretrained('/path/to/models/roberta_finetuned/model')
tokenizer_class = RobertaTokenizer
tokenizer = tokenizer_class.from_pretrained('/path/to/models/roberta_finetuned/model')
tmp = tokenizer.encode('A sentence to encode with roberta.', add_special_tokens=True)
input_ids = torch.tensor(tmp)

with torch.no_grad():
    last_hidden_states = model(input_ids)[0]
```

Any thoughts on what I might be messing up?


How do I load a pretrained file offline?

## ❓ Questions & Help

Hi, when I use "RobertaModel.from_pretrained(roberta.large)" to load model. A progress bar appears to download the pre-training model. I've already downloaded files like "roberta-large-pytorch_model.bin ". How can I stop automatically downloading files to the ".cache" folder and instead specify these pre-training files I downloaded?


XLM-R Support

## ❓ Questions & Help

Hello!

Is there a way to use XLM-R (https://github.com/pytorch/fairseq/blob/master/examples/xlmr/README.md) with the library of transformers? maybe via RoBERTa? can you provide some guidance on this please?

Thank you in advance


Meaning of run_lm_finetuning.py output

## ❓ Questions & Help
Is there documentation somewhere about what the various output files that get created when running `run_lm_finetuning.py` are and what the meaning of their contents is? Concretely, what are the files and directories:
```
added_tokens.json
checkpoint-50/
checkpoint-100/
checkpoint-150/
checkpoint-200/
checkpoint-250/
checkpoint-300/
checkpoint-350/
checkpoint-400/
config.json
eval_results.txt
merges.txt
pytorch_model.bin
runs/
special_tokens_map.json
tokenizer_config.json
training_args.bin
vocab.json
```
and what is the meaning of their contents? the `checkpoint` directories contain:
```
config.json
pytorch_model.bin
training_args.bin
```

and `runs/Dec03_09-15-51_MACHINENAME` contains:
```
events.out.tfevents.20414.0
```

## Finetuning Setup
* Model: roberta-base
* Language: english
* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): 2.2.1, I believe
* Using GPU ? No
* Distributed of parallel setup ? No
* Script inputs:
```
python run_lm_finetuning.py \
--output_dir=$OUTPUT_DIR \
--model_type=roberta \
--model_name_or_path=roberta_base \
--do_train \
--train_data_file=$TRAIN_FILE \
--do_eval \
--eval_data_file=$TEST_FILE \
--mlm \
--no_cuda
```

Thanks in advance!


run_squad with xlm: Dataparallel has no attribute config.

## 🐛 Bug

<!-- Important information -->

Model I am using XLM.

Language I am using the model on English:

The problem arise when using:
* [x] the official example scripts: run_squad.py

## To Reproduce

Steps to reproduce the behavior:

1. Azure VM with 2 GPUs
2. run_squad with XLM
3. Everything fine until evaluation step.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
12/03/2019 15:54:26 - INFO - __main__ -   ***** Running evaluation  *****
12/03/2019 15:54:26 - INFO - __main__ -     Num examples = 10918
12/03/2019 15:54:26 - INFO - __main__ -     Batch size = 16
Evaluating: 100%|█████████████████████████████████████████████████████| 683/683 [05:16<00:00,  2.16it/s]
12/03/2019 15:59:42 - INFO - __main__ -     Evaluation done in total 316.178766 secs (0.028959 sec per example)
Traceback (most recent call last):
  File "transformers/examples/run_squad.py", line 575, in <module>
    main()
  File "transformers/examples/run_squad.py", line 564, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "transformers/examples/run_squad.py", line 280, in evaluate
    model.config.start_n_top, model.config.end_n_top,
  File "/home/wallis/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'DataParallel' object has no attribute 'config'
```
## Expected behavior

Calculate scores and prints them.

## Possible suggestion for parallel use:

Change that line 280 in run_squad.py to:
```
model.module.config.start_n_top, model.module.config.end_n_top,
```
as suggested [here](https://discuss.pytorch.org/t/dataparallel-throws-an-error-attributeerror-dataparallel-object-has-no-attribute-loss/34228).

With this change, it seems to progress but only reach another
[error](https://github.com/huggingface/transformers/issues/1771)
so not sure.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: ubuntu 16.04 on azure
* Python version: 3.7
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? yes
* Distributed of parallel setup ? I think its trying to do parallel
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


how to select best model in run_glue

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
i'm a green hand and i know it is a rediculous problem.I just saw
`    # Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()
    if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0) and not args.tpu:
        # Create output directory if needed
        if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]:
            os.makedirs(args.output_dir)`

in row 526,but it seems like that i can hardly find any word about selecting the best model.Would be very appriciate if you can tell me.


error

## ❓ Questions & Help

12/03/2019 09:12:25 - INFO - transformers.modeling_utils -   loading weights file model_check_points112/pytorch_model.bin
12/03/2019 09:12:40 - INFO - __main__ -   Creating features from dataset file at dev-v1.1.json
Traceback (most recent call last):
  File "run_squad.py", line 558, in <module>
    main()
  File "run_squad.py", line 547, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "run_squad.py", line 195, in evaluate
    dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)
  File "run_squad.py", line 296, in load_and_cache_examples
    version_2_with_negative=args.version_2_with_negative)
  File "/content/drive/My Drive/examples/utils_squad.py", line 97, in read_squad_examples
    input_data = json.load(reader)["data"]
  File "/usr/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 4194293 (char 4194292)


Doubts on modeling_gpt2.py

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I've been going through the gpt2 source code and i was tracing out how the self attention and feed forward work basically we have `Block` which is a decoder consisting of other 2 segments `Attention` and `MLP`. I was also reading a blog where it mentions the `queries` has to be learned i finally saw the class responsible for that is `Conv1D`
```
class Conv1D(nn.Module):
    def __init__(self, nf, nx):
        """ Conv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2)
            Basically works like a Linear layer but the weights are transposed
        """
        super(Conv1D, self).__init__()
        self.nf = nf
        w = torch.empty(nx, nf)
        nn.init.normal_(w, std=0.02)
        self.weight = nn.Parameter(w)
        self.bias = nn.Parameter(torch.zeros(nf))

    def forward(self, x):
        size_out = x.size()[:-1] + (self.nf,)
        x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
        x = x.view(*size_out)
        return x

```

Here we have a affine function `addmm` is how we do... but i was expecting `train optimer.step, loss` basically all the thing that goes into training in pytorch. Can anyone elaborate on it?

Then in `GPT2LMHeadModel` we use a linear layer to represent `vocab_embed` and we produce `logits` by multiplying it with the transformer output so if the linear layer is trained what is the use of `from_pretrained` anyway?

I am sure this may perhaps be silly questions but i'd like to get some help here.

Thanks a lot.


Updated examples/README and parser for run_summarization_finetuning

1. Updated `examples/README.md` to change default `--model_type` and `--model_name_or_path` to `bert` and `bert_base_cased` because `bert2bert` just won't work
2. Updated `examples/run_summarization_finetuning.py` parser to take in `--do-train` instead of `--do-train=True` for consistency with other examples and `--model_type` + `--model_name_or_path`
3. Changed `add_special_tokens_single_sequence` to `build_inputs_with_special_tokens` in `examples/utils_summarization.py`


run_lm_finetuning.py script CLM inputs and labels preparing

I'm trying to finetune the GPT-2 on my own dataset, while I'm reading the code in `run_lm_finetuning.py` script, I found a weird thing in line `227`. When the script preparing CLM batch inputs and labels, it gives the model the same `batch` variable as inputs and labels:

```
inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
```

In this way, the model will learn to take the token as an input and predict it directly, right?

Can anyone explain what happens?


Any workaround to extend the embeddings on TFGPT2DoubleHeadsModel?

Getting access to Keras' `model.fit()` method makes life so much easier for transfer learning/fine-tuning, but TFGPT2DoubleHeadsModel doesn't currently support extending embeddings, so it really restricts practical applications. You almost always have to add something to the vocabulary / generate special tokens. Does anyone know of a workaround?


Typo in modeling_albert.py for mask_token

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Albert

Language I am using the model on (English, Chinese....): English

## To Reproduce

```
tokenizer_class, pretrained_weights = AlbertTokenizer, "albert-base-v1"
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
print(tokenizer.mask_token)   # [MASK]>
print(tokenizer.mask_token_id)  # 1 (same as  <unk>)
```

I think the typo lies here https://github.com/huggingface/transformers/blob/fbaf05bd92249b6dd961f5f8d60eb0892c541ac8/transformers/tokenization_albert.py#L69

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

mask_token should be "[MASK]" and mask_token_id should be 4

## Environment

* OS: Windows 10
* Python version: 3.6.9
* PyTorch version:  1.2.0
* PyTorch Transformers version (or branch): 2.2.0


cannot import name 'WEIGHTS_NAME'

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): gpt2

Language I am using the model on (English, Chinese....): english

The problem arise when using:
* [X] the official example scripts: `run_lm_finetuning.py`
* [ ] my own modified scripts: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Obtain `transformers` from zip file on github.
2. try to run `run_lm_finetuning.py` using the example in the documentation.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
``` python
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 45, in <module>
    from transformers import (WEIGHTS_NAME, AdamW, get_linear_schedule_with_warmup,
ImportError: cannot import name 'WEIGHTS_NAME' from 'transformers' (unknown location)
```

## Environment

* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): whichever version is included in the zip file off github
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information: None.

## Additional context

<!-- Add any other context about the problem here. -->
I tried to obtain transformers from source using `git clone https://github.com/huggingface/transformers.git` but I got a timeout error (which is why I opted to try to zip file).


gpt-2 generation examples

## ❓ Questions & Help

Hi! Thanks for everything, I want to try generation with the gpt-2 model, following:
```
python ./examples/run_generation.py \
    --model_type=gpt2 \
    --length=20 \
    --model_name_or_path=gpt2 \
```
But it does not seem to work very well, for example (Prompt -> Generation):
i go to ->  the Kailua Islands? Eh? Ahh. Although they did say the
i like reading -> -_-/- 40:25:13 7d 9h 25m We battle trainer. Before we
i like running ->  from someone which can easily overwhelm your battery in those moments and through the rest of your day

I mean, the generation don't really look good to me, is that anything I should mind during trying this?
Thanks!

Additional info:
`12/02/2019 15:41:46 - INFO - __main__ -   Namespace(device=device(type='cuda'), length=20, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_samples=1, padding_text='', prompt='', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')`


[CamemBERT] Potential error in the docs

Thanks for the great work on this repo! As I was going through the details about the available pre-trained models (https://huggingface.co/transformers/v2.2.0/pretrained_models.html), I spotted what I think is an error in the description of camembert-base (12-layer, 768-hidden, 12-heads, 110M parameters; CamemBERT using the BERT-base architecture). Isn't it RoBERTa-based?


Tokenization differs for different intepreter instances

## 🐛 Bug

<!-- Important information -->
Tokenization of `" "` changes for each python interpreter instance.

## To Reproduce

```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
for i in range(5):
    print(tokenizer.encode(" "))
```

## Environment

* Python version: 3.7.2
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.0


## ❓ Does GPT2LMHeadModel need <|startoftext|> and <|endoftext|> tokens?

Hey!

I'm using GPT2LMHeadModel to get a good representation of a Language Model - I want to get probabilities for each word. The problem is - the **model predicts probabilities very well for all tokens except for the first one**. The first's token probability is often very small no matter what word I choose.
I've read that there is "<|startoftext|>" token, but have not found information on how to use it. It also doesn't exist in GPT2Tokenizer.vocabulary. Do we have to use it?

### Example code:

```
import torch
from pytorch_transformers import *

pretrained_weights='gpt2'

tokenizer = GPT2Tokenizer.from_pretrained(pretrained_weights)
model = GPT2LMHeadModel.from_pretrained(pretrained_weights)
model.eval()

def show_probabilities(INPUT_TEXT):
    input_ids = torch.tensor([tokenizer.encode(INPUT_TEXT)])

    with torch.no_grad():
        index=0
        outputs = model(input_ids=input_ids)
        logits = outputs[0][0]
        probs = torch.softmax(logits, 1)
        for index in range(0, len(input_ids[0])):
            token_id = input_ids[0][index]
            probability = probs[index - 1][token_id].item()
            print(f"Probability for the token \"{tokenizer.decode(token_id.item())}\" is {probability}")
    print("\n")

show_probabilities('To be or not to be <|endoftext|>')
show_probabilities('<|startoftext|> To be or not to be <|endoftext|>')
show_probabilities('<|endoftext|> To be or not to be <|endoftext|>')
show_probabilities('Hello world is so wierd?')

```

### Output:
###### (so that you dont have to run it)
```
Probability for the token " To" is 6.045737421800368e-08
Probability for the token " be" is 0.01369183138012886
Probability for the token " or" is 0.0001948970602825284
Probability for the token " not" is 0.7490634322166443
Probability for the token " to" is 0.5098284482955933
Probability for the token " be" is 0.9639962911605835
Probability for the token "<|endoftext|>" is 0.00017062896222341806

Probability for the token " <" is 1.5030431086415774e-06
Probability for the token "|" is 0.0006586791132576764
Probability for the token "start" is 7.143173570511863e-05
Probability for the token "of" is 0.0012107481015846133
Probability for the token "text" is 0.0007207148591987789
Probability for the token "|" is 0.4524894058704376
Probability for the token ">" is 0.027218399569392204
Probability for the token " To" is 0.0003593114379327744
Probability for the token " be" is 0.015610950998961926
Probability for the token " or" is 0.0021431492641568184
Probability for the token " not" is 0.46310704946517944
Probability for the token " to" is 0.8615797162055969
Probability for the token " be" is 0.9770862460136414
Probability for the token "<|endoftext|>" is 0.0008418861543759704

Probability for the token "<|endoftext|>" is 3.0863736810715636e-06
Probability for the token " To" is 3.549279790604487e-05
Probability for the token " be" is 0.04548846557736397
Probability for the token " or" is 0.0003993543505202979
Probability for the token " not" is 0.8718274831771851
Probability for the token " to" is 0.9372356534004211
Probability for the token " be" is 0.9853253960609436
Probability for the token "<|endoftext|>" is 0.0009108908707275987

Probability for the token " Hello" is 0.00041539999074302614
Probability for the token " world" is 0.00014912338519934565
Probability for the token " is" is 0.029302824288606644
Probability for the token " so" is 0.01128558162599802
Probability for the token " w" is 0.00020273651171009988
Probability for the token "ier" is 0.008098911494016647
Probability for the token "d" is 0.8924543857574463
Probability for the token "?" is 0.0036612364929169416
```


How to convert a tf2 pre-trained model to pytorch model?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I have trained a pre-trained model from scratch using a tensorflow 2.0 official script (run_pretraining.py).
https://github.com/tensorflow/models/tree/master/official/nlp/bert

My question is how to convert the pre-trained model to pytorch model?
Thanks in advance.


[ALBERT] : ValueError: Layer #1 (named "predictions") expects 11 weight(s), but the saved weights have 10 element(s).

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): ALBERT

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

import tensorflow as tf
from transformers import *
#Download AlbertMaskedLM model
model = TFAlbertForMaskedLM.from_pretrained('albert-large-v2')

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details) Initial validation

## To Reproduce

Steps to reproduce the behavior:
import tensorflow as tf
from transformers import *
#Download AlbertMaskedLM model
model = TFAlbertForMaskedLM.from_pretrained('albert-large-v2')
1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
The code throws an error as follows :

100%|██████████| 484/484 [00:00<00:00, 271069.99B/s]
100%|██████████| 87059544/87059544 [00:03<00:00, 28448930.07B/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-a2c768b76a32> in <module>()
----> 1 model = TFAlbertForMaskedLM.from_pretrained('albert-large-v2')

3 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    287         # 'by_name' allow us to do transfer learning by skipping/adding layers
    288         # see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1339-L1357
--> 289         model.load_weights(resolved_archive_file, by_name=True)
    290
    291         ret = model(model.dummy_inputs, training=False)  # Make sure restore ops are run

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in load_weights(self, filepath, by_name)
    179         raise ValueError('Load weights is not yet supported with TPUStrategy '
    180                          'with steps_per_run greater than 1.')
--> 181     return super(Model, self).load_weights(filepath, by_name)
    182
    183   @trackable.no_automatic_dependency_tracking

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py in load_weights(self, filepath, by_name)
   1173         f = f['model_weights']
   1174       if by_name:
-> 1175         saving.load_weights_from_hdf5_group_by_name(f, self.layers)
   1176       else:
   1177         saving.load_weights_from_hdf5_group(f, self.layers)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py in load_weights_from_hdf5_group_by_name(f, layers)
    749                          '") expects ' + str(len(symbolic_weights)) +
    750                          ' weight(s), but the saved weights' + ' have ' +
--> 751                          str(len(weight_values)) + ' element(s).')
    752       # Set values.
    753       for i in range(len(weight_values)):

ValueError: Layer #1 (named "predictions") expects 11 weight(s), but the saved weights have 10 element(s).
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
TFAlbertMaskedLM model can not be loaded from pre-trained
## Environment

* OS: Linux (Colab)
* Python version: 3.6
* PyTorch version: Tensorflow 2.0
* PyTorch Transformers version (or branch):
* Using GPU ? Yes
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Is it possible to fine-tune models on TPUs using TensorFlow?

I have looked at the release notes and found out that:

"Training on TPU using free TPUs provided in the TensorFlow Research Cloud (TFRC) program is possible but requires to implement a custom training loop (not possible with keras.fit at the moment).
We will add an example of such a custom training loop soon." (Note from September 26).

Is this observation still true? Can we train the transformer models on TPUs in TF?


How to convert the ALBERT tfhub model to pytorch model?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to apply ALBERT to other QA datasets. But the first question is how to convert the tf_hub model. I downloaded the tfhub model from the repo "google research".  The script ```convert_albert_original_tf_checkpoint_to_pytorch.py``` need the parameter ```--tf_checkpoint_path```. I attempted the directory 'variables', 'assets' and the root directory but all failed.  I haven't solve it by document. Is there some demos for ALBERT?


save as tensorflow saved model format and how to inference?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, l follow the script in readme, train a model and save as tensorflow saved_model format instead of h5 format.
When inferencing, I get some problem, I don't know how to feed the inputs to the model. Here is code.

```python
import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

tf.saved_model.save(model,"/content/saved")
```
I change the last line code to get a tensorflow saved_model. I get a problem when inferencing.

```python
loaded = tf.saved_model.load("/content/saved")
inference_func = loaded.signatures["serving_default"]
for inputs,_ in valid_dataset:
  print(inference_func(inputs))
```
Then I get:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-7c90c411776e> in <module>()
      1 for inputs,_ in valid_dataset:
----> 2   print(inference_func(inputs))

1 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py in _call_impl(self, args, kwargs, cancellation_manager)
   1098            "of {}), got {}. When calling a concrete function, positional "
   1099            "arguments may not be bound to Tensors within nested structures."
-> 1100           ).format(self._num_positional_args, self._arg_keywords, args))
   1101     args = list(args)
   1102     for keyword in self._arg_keywords[len(args):]:

TypeError: Expected at most 0 positional arguments (and the rest keywords, of ['attention_mask', 'input_ids', 'token_type_ids']), got ({'input_ids': <tf.Tensor: id=130383, shape=(64, 128), dtype=int32, numpy=
array([[ 101, 1284, 5376, ...,    0,    0,    0],
       [ 101, 2061,  117, ...,    0,    0,    0],
       [ 101, 1130, 1103, ...,    0,    0,    0],
       ...,
       [ 101, 1109, 3302, ...,    0,    0,    0],
       [ 101, 1556, 1292, ...,    0,    0,    0],
       [ 101, 1109,  158, ...,    0,    0,    0]], dtype=int32)>, 'attention_mask': <tf.Tensor: id=130382, shape=(64, 128), dtype=int32, numpy=
array([[1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       ...,
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>, 'token_type_ids': <tf.Tensor: id=130384, shape=(64, 128), dtype=int32, numpy=
array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>},). When calling a concrete function, positional arguments may not be bound to Tensors within nested structures.
```
Has anyone encountered this problem before?


Camenbert length Tokenizer not equal config vocab_size

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi there,
when I load the pretrained Camenbert model and tokenizer via

`model = CamembertForMaskedLM.from_pretrained('camembert-base')
tokenizer = CamembertTokenizer.from_pretrained('camembert-base')`

the length of the tokenizer is 32004 but the vocab_size of the model is 32005.
`print(len(tokenizer))`
'print(model.config.vocab_size'

This throws me an error
> Index out of range

when I try to adapt the lm_finetuning example because of
`model.resize_token_embeddings(len(tokenizer))`

It runs when I comment out this line. So my question is, is this the intended behaviour resp. what's the reason for the unevenness between the tokenizer and the model vocab_size?


[CamemBert] Tokenizer function add_tokens doesn't work

## ❓ Questions & Help

Hi,

I am trying to add new tokens to the CamemBert tokenizer, but when I run the function tokenizer.add_tokens, it doesn't seem to add any token at all :

`from transformers import CamembertTokenizer`
`tokenizer = CamembertTokenizer.from_pretrained('camembert-base')`
`tokenizer.add_tokens(['notfrenchword'])`

`Out[12]: 0`

Whereas with Bert model it works perfectly. Is this a bug or am I doing something wrong ?

Thanks


FileNotFoundError: [Errno 2] No such file or directory: 'data/dump.txt'

@stefan-it
Hello, I am new learner in BERT and I want to have a try the excellent work - distilBert.
But the problem happened when I ran the training step, and Could you tell me where can I download the `dump.txt` file ?
Thank you very much!


How to use GPT-2 text generator in spanish

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I would like to know if there is a way to use the gpt-xl model for text generation in spanish.

The command I use to run the english text generation model is the following:

$ python ./examples/run_generation.py     --model_type=gpt2     --length=50     --model_name_or_path=gpt2-xl

What other parameter must I use to allow spanish text generation?


GPT-2 finetuning with run_lm_finetuning.py script

## ❓ Questions & Help
I tried to finetune gpt-2 model using `run_lm_finetuning.py` script with the following parameters:
```
python run_lm_finetuning.py \
    --train_data_file=text8.train \
    --output_dir=/content/gpt2 \
    --eval_data_file=text8.val \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --do_eval \
    --per_gpu_train_batch_size=32 \
    --per_gpu_eval_batch_size=32 \
    --gradient_accumulation_steps=1 \
    --num_train_epochs=3 \
    --warmup_steps=200
```
and it throws memory error no matter what my machine type is. In extreme case I wanted to run this demo on Google Cloud with 32 cpus and 120GB RAM - not possible. It just eats the whole RAM and does not even make a single iteration. On the other hand, I was able to do finetuning from this project [gpt-2-simple](https://github.com/minimaxir/gpt-2-simple) on Google Colab with 124M model (CPU). What is going on? Am I doing something wrong?


[CamemBERT] Add CamembertForQuestionAnswering

Firstly, a huge thanks to Hugging Face team for their great work ! As we have now Camembert, it would be nice to use it for question answering using transformers ! You can find SQuAD in French on GitHub so it would be so easy to a lot of people to fine-tune Camembert for this task.

Please consider it in future releases 😉


Mark tests in TFAutoModelTest as slow.

Each test forces downloading the same 536MB file, which is slow
even with a decent internet connection.


What is the real parameters to weight the triple loss (L_{ce}, L_{mlm}, L_{cos}) in DistilBert?

Hello! Thanks for your great work DistilBert. I want to ask what is the real parameters "alpha" you used in DistilBert to weight the triple loss (L_{ce}, L_{mlm}, L_{cos})?

You did not mention this detail in your NIPS workshop paper (http://arxiv.org/abs/1910.01108). In the [README](https://github.com/huggingface/transformers/blob/master/examples/distillation/README.md) file, you listed two different setups: `--alpha_ce 5.0 --alpha_mlm 2.0 --alpha_cos 1.0 --alpha_clm 0.0` for single GPU training and `--alpha_ce 0.33 --alpha_mlm 0.33 --alpha_cos 0.33 --alpha_clm 0.0` for distributed training. Can you tell me what is the best setting?

Actually, I have tried to reproduce your results of DistilBert. I trained the DistilBert with the corpus used by BERT, but the performance of GLUE seemed slightly fall behind your pre-trained `distilbert-base-uncased` by 2 points. I would be appreciated if you can tell me the parameters for reproducibility. Thanks!


How to output the vectors of the last four layers of BERT_Model.

E.g
output = [the_last_one_layer_output, second_last_layer_output, ...]


typo fix on the docs as per Pytorch v1.1+


Changing the docs as per Pytorch v1.1+

## ❓ Questions & Help

[Docs Link](https://huggingface.co/transformers/migration.html#optimizers-bertadam-openaiadam-are-now-adamw-schedules-are-standard-pytorch-schedules)

```
# From the Docs
### In Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
    scheduler.step()
    optimizer.step()
```

As per the Pytorch 1.1+,

>Prior to PyTorch 1.1.0, the learning rate scheduler was expected to be called before the optimizer’s update; 1.1.0 changed this behavior in a BC-breaking way. If you use the learning rate scheduler (calling scheduler.step()) before the optimizer’s update (calling optimizer.step()), this will skip the first value of the learning rate schedule. If you are unable to reproduce results after upgrading to PyTorch 1.1.0, please check if you are calling scheduler.step() at the wrong time.

[Pytorch Reference Link](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate)

Thanks.

PS Not sure if the issue category selected is apt.


Reason for using einsum in xlnet?

## ❓ Questions & Help

Hello.

This might be a newbie question, so I apologize in advance.

While reading your implementation of xlnet, I ran into several usages of `torch.einsum` function.

example) `k_head_h = torch.einsum('ibh,hnd->ibnd', cat, self.k) `

After studying the definition of einsum, I came to a conclusion that the above statement is exactly like using a linear layer (without bias) (from dimension h to n*d), and then resizing the output to be ibnd.

So if I'm not wrong, is there any reason to prefer using `torch.einsum` over `nn.Linear`?

Is it related to performance issues?

I ran a simple test, and `nn.Linear` seems to be a bit faster than `torch.einsum`.

I would really appreciate your help.

Thank you.


Expand run_lm_finetuning.py to all models

## 🚀 Feature

[run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/b0ee7c7df3d49a819c4d6cef977214bd91f5c075/examples/run_lm_finetuning.py) is a very useful tool for finetuning many models the library provided. But it doesn't cover all the models. Currently available models are:

- gpt2
- openai-gpt
- bert
- roberta
- distilbert
- camembert

And not available ones:

- ctrl
- xlm
- xlnet
- transfo-xl
- albert

## Motivation

Most important part of such a library is that it can be easily finetuned. `run_lm_finetuning.py` gives us that opportunity but why say no more :)


fixed XLNet attention output for both attention streams whenever target_mapping is provided

XLNet uses two separate attention streams, i.e. there are two separate tensors for representing the model's attention. Both of them need to have their dimensions permuted.

The problem has been described in #1994 .


[ALBERT]: 'AlbertForMaskedLM' object has no attribute 'bias'

Hi,

I wanted to convert an own trained ALBERT model with the `convert_albert_original_tf_checkpoint_to_pytorch.py` script:

```bash
$ python3 convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path /mnt/albert-base-secrect-language-cased/ --albert_config_file /mnt/albert-base-secrect-language-cased/config.json --pytorch_dump_path pytorch_model.bin
```

Unfortunately, the following error message is returned:

```bash
<--snip-->
bert/pooler/dense/bias
bert/pooler/dense/bias/adam_m
bert/pooler/dense/bias/adam_v
bert/pooler/dense/kernel
bert/pooler/dense/kernel/adam_m
bert/pooler/dense/kernel/adam_v
cls/predictions/output_bias
cls/predictions/output_bias/adam_m
cls/predictions/output_bias/adam_v
cls/predictions/transform/LayerNorm/beta
cls/predictions/transform/LayerNorm/beta/adam_m
cls/predictions/transform/LayerNorm/beta/adam_v
cls/predictions/transform/LayerNorm/gamma
cls/predictions/transform/LayerNorm/gamma/adam_m
cls/predictions/transform/LayerNorm/gamma/adam_v
cls/predictions/transform/dense/bias
cls/predictions/transform/dense/bias/adam_m
cls/predictions/transform/dense/bias/adam_v
cls/predictions/transform/dense/kernel
cls/predictions/transform/dense/kernel/adam_m
cls/predictions/transform/dense/kernel/adam_v
cls/seq_relationship/output_bias
cls/seq_relationship/output_bias/adam_m
cls/seq_relationship/output_bias/adam_v
cls/seq_relationship/output_weights
cls/seq_relationship/output_weights/adam_m
cls/seq_relationship/output_weights/adam_v
global_step
INFO:transformers.modeling_albert:Skipping bert/embeddings/attention/LayerNorm/beta
INFO:transformers.modeling_albert:Skipping bert/embeddings/attention/LayerNorm/beta
INFO:transformers.modeling_albert:Skipping bert/embeddings/attention/LayerNorm/beta
INFO:transformers.modeling_albert:Skipping bert/embeddings/attention/LayerNorm/beta
Traceback (most recent call last):
  File "convert_albert_original_tf_checkpoint_to_pytorch.py", line 66, in <module>
    args.pytorch_dump_path)
  File "convert_albert_original_tf_checkpoint_to_pytorch.py", line 37, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_albert(model, config, tf_checkpoint_path)
  File "/mnt/transformers/transformers/modeling_albert.py", line 92, in load_tf_weights_in_albert
    pointer = getattr(pointer, 'bias')
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'AlbertForMaskedLM' object has no attribute 'bias'
```

I'm using the latest commit in `google-research` for training the ALBERT model. Configuration is:

```json
{
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "embedding_size": 128,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_hidden_groups": 1,
  "net_structure_type": 0,
  "gap_size": 0,
  "num_memory_blocks": 0,
  "inner_group_num": 1,
  "down_scale_factor": 1,
  "type_vocab_size": 2,
  "vocab_size": 32000
}
```


## ❓ Questions & Help

I want to use `mixed_precision`, and I found [tf.keras.mixed_precision.experimental.Policy](https://www.tensorflow.org/api_docs/python/tf/keras/mixed_precision/experimental/Policy).

So I put `tf.keras.mixed_precision.experimental.set_policy("mixed_float16")` before `TFBertModel.from_pretrained(pretrained_weights)`. When I run the code, I got the following error:

> InvalidArgumentError: cannot compute AddV2 as input #1(zero-based) was expected to be a half tensor but is a float tensor [Op:AddV2] name: tf_bert_model_1/bert/embeddings/add/

which happened at `ret = model(model.dummy_inputs, training=False)  # build the network with dummy inputs`.

I am not sure if I used it correctly. I think `tf.keras.mixed_precision.experimental.set_policy` is supposed to be used before constructing / build the model, as the tf page says `Policies can be passed to the 'dtype' argument of layer constructors, or a global policy can be set with 'tf.keras.mixed_precision.experimental.set_policy'`.

I wonder if I can use AMP with tf based transformer models and how. Thanks.

[error.txt](https://github.com/huggingface/transformers/files/3907032/error.txt)


Can we use tf.keras.mixed_precision.experimental.set_policy ?


Where I could find the vocab.json for XLNet

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I notice in tokenizer_xlnet.py there is not the vocab,json only spiece model. So I want to know where I could find the vocab.json? And what I should rename the file ?


Always use SequentialSampler during evaluation

When evaluating, shouldn't we always use the SequentialSampler instead of DistributedSampler? Evaluation only runs on 1 GPU no matter what, so if you use the DistributedSampler with N GPUs, I think you'll only evaluate on 1/N of the evaluation set. That's at least what I'm finding when I run an older/modified version of this repo.


GPT2: how to construct batch for Language Modeling

I am a little confused about how to prepare input bathces for GPT2LMHeadModel. I want to use GPT2 as an LM. For instance, I want to generate probability distributions over the vocabulary at each timestep, as well as computing the perplexities of sentences. It is important to note that I am working with sentences and not documents, so I will have to pad the inputs in the batch.

```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel

# Prepare model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
model.to('cuda')

# input sentences
batch = ['this is a sentence.',
        'this is another sentence.',
        'this is another even longer sentence.']
```

## Question 1: Special tokens
a) Do I have to add a bos token id on my own or is it handled internally by GPT2Tokenizer? Same for the eos token.
```
# tokenize
tokens = [tokenizer.encode(x) for x in batch]

# add BOS and EOS
tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]  for x in tokens]
```
```
[[50256, 428, 318, 257, 6827, 13, 50256],
 [50256, 428, 318, 1194, 6827, 13, 50256],
 [50256, 428, 318, 1194, 772, 2392, 6827, 13, 50256]]
```
b) `tokenizer.encode(x)` gives me a warning "This tokenizer does not make use of special tokens. Input is returned with no modification." I replaced it with `tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x, add_prefix_space=True))` and the warning went away, but I am not sure what is the difference. Which tokenization should I use?

c) By looking at the properties of an instance of GPT2Tokenizer, I see that `bos_token` and `eos_token` are the same. Is this correct?

## Question 2: Padding
I want to pad based on the longest sentence in the batch. This is how I usually do it.
```
batch = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True).to('cuda')
```
```
tensor([[50256,   428,   318,   257,  6827,    13, 50256,     0,     0],
        [50256,   428,   318,  1194,  6827,    13, 50256,     0,     0],
        [50256,   428,   318,  1194,   772,  2392,  6827,    13, 50256]])
```
a) What id does the model expect for the padded tokens? Do I have to pass the token id as an argument to the model or the tokenizer or you have a predifined one?

## Question 3: Model input
How is GPT2 made aware of the padded steps? For instance for an RNN I would do something like this:
```
lengths = (batch != 0).sum(-1) # tensor([7, 7, 9])
packed = pack_padded_sequence(x, lengths, batch_first=True)
out_packed, hn = rnn(packed)
```
but for GPT2 I havent found an example. The only ones i found are with batch size 1. So something like this wont work as expected:
```
outputs = model(x, labels=x)  # labels are shifted inside the model, right?
loss, logits = outputs[:2]
```
---
# Update
So there are still things unclear, but from reading other issues this is my current understanding:
 - GPT2 has no padding token, as it was trained on documents and not sentences.
 - In order to use GPT2 with variable length inputs, we can apply padding with an arbitrary token and ensure that those tokens are not used by the model with an `attention_mask`.
 - As for the labels, we should replace **only** on the `labels` variable the padded token ids with `-1`.
So based on that, here is my current toy implementation:
```python
inputs = [
    'this is a sentence.',
    'this is another sentence.',
    'this is another even longer sentence.', ]

# tokenize
# tokens = [tokenizer.encode(x) for x in batch]
tokens = [tokenizer.convert_tokens_to_ids(
    tokenizer.tokenize(x, add_prefix_space=True))
    for x in inputs]

# add BOS and EOS
tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]
          for x in tokens]

# padding_value can be whatever...
inputs = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True, padding_value=0).to('cuda')
# 1 for real tokens and 0 for padded tokens
mask = (inputs != 0).float()
# replace the ids of the padded tokens (where token_id==padded_id) with `-1`
labels = inputs.masked_fill(inputs == 0, -1)

outputs = model(inputs, attention_mask=mask, labels=labels)
loss, logits = outputs[:2]
```
Is this correct??

### Bug: Padded tokens are not excluded from the loss
However, I computed the loss on my own and found your implementation does not take into account the padded tokens when averaging, unless I am missing something
```python
_logits = logits.view(-1, logits.size(-1))  # flatten logits
_labels = torch.cat([inputs[:, 1:], inputs[:, :1] * 0], dim=1).view(-1) # shift inputs one position to the left and flatten
loss_real_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='sum') / mask.sum() # ignore padded timesteps
loss_naive_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='mean')

print("GPT2 loss:", loss.item())
print("loss_naive_avg:", loss_naive_avg.item())
print("loss_real_avg:", loss_real_avg.item())
```
```
GPT2 loss: 4.664564609527588
loss_naive_avg: 4.664564609527588
loss_real_avg: 4.056143283843994
```


Wrong tokenization in Transformer-XL documentation

## 🐛 Bug

This is a documentation-related bug. In the [TransfoXL documentation](https://huggingface.co/transformers/model_doc/transformerxl.html), the tokenization example is wrong. The snippet goes:
```
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
...
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
```
This code outputs the tokens `[24, 617, 3225, 23, 16072]`, of which `24` is `<unk>`.

The problem comes from the fact that Transformer-XL does **not** use a wordpiece vocabulary, but a regular (whole-word) one. Also, in WT-103, punctuation marks are split from the words. Consequently, the example should read instead (note that space in from of `,`):
```
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
...
input_ids = torch.tensor(tokenizer.encode("Hello , my dog is cute")).unsqueeze(0)  # Batch size 1
```

It would also be nice to warn the user about this fact in the documentation, perhaps in `TransfoXLTokenizer`'s docstring?


Training masked language model with Tensorflow

## ❓ Questions & Help

I'm trying to fine-tune a masked language model starting from bert-base-multilingual-cased with Tensorflow using the PyTorch-based example _examples/run_lm_finetuning_ as starting point. I'd like to take the multilingual model and adapt it to the Italian language.
Unfortunately I'm unable to find examples over the internet for the TFBertForMaskedLM model in training mode, so I hope this is the appropriate place for this question.

System and libraries

> Platform Linux-5.0.0-36-generic-x86_64-with-debian-buster-sid
> Python 3.7.5 (default, Oct 25 2019, 15:51:11)
> [GCC 7.3.0]
> PyTorch 1.3.1
> Tensorflow 2.0.0
> Transformers 2.2.0

I first convert my train sentences in 4 arrays:
1) train_ids_masked: tokens ids with special tokens and masking + padding up to max_seq_length = 10
2) train_attnmasks: masks for attention (padding masks)
3) train_segments: masks for sentence (constant array since sentences are independent)
4) train_labels: original masked tokens + UNK tokens everywhere else

Every array has shape (num sentences, max_seq_length) = (72,10)

Then I define the model and print the summary
```python
pre_trained_model = 'bert-base-multilingual-cased'

config = transformers.BertConfig.from_pretrained(pre_trained_model)

model = transformers.TFBertForMaskedLM.from_pretrained(pre_trained_model, config=config)

model.compile(optimizer=tf.optimizers.Adam(lr=params['learning_rate']), loss='binary_crossentropy')

print(model.summary())
```

which outputs
```
Model: "tf_bert_for_masked_lm_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
bert (TFBertMainLayer)       multiple                  177853440
_________________________________________________________________
mlm___cls (TFBertMLMHead)    multiple                  92920059
=================================================================
Total params: 178,565,115
Trainable params: 178,565,115
Non-trainable params: 0
```
Then I try to train the model
```python
model.fit([train_ids_masked, train_attnmasks, train_segments], train_labels, epochs=1, batch_size=20)
```
The model trains over the first batch but returns the following error
```
Train on 72 samples
20/72 [=======>......................] - ETA: 7sTraceback (most recent call last):
  File "/home/andrea/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1610, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimensions must be equal, but are 10 and 119547 for 'loss/output_1_loss/mul' (op: 'Mul') with input shapes: [?,10], [?,?,119547].
```
when calculating the loss, trying to compare the padding length max_seq_length (= 10) to the vocabulary size (= 119547).

I've also tried to define the model in the following way
```python
inp_ids = tf.keras.layers.Input(shape=(max_seq_length, ), dtype='int32', name="bert_input_ids")
inp_attnmasks = tf.keras.layers.Input(shape=(max_seq_length, ), dtype='int32', name="bert_input_attention_masks")
inp_segments = tf.keras.layers.Input(shape=(max_seq_length, ), dtype='int32', name="bert_input_segment_ids")
inputs = [inp_ids, inp_attnmasks, inp_segments]
outputs = transformers.TFBertForMaskedLM.from_pretrained(pre_trained_model)(inputs)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
model.compile(optimizer=tf.optimizers.Adam(lr=params['learning_rate']), loss='binary_crossentropy')
```
but I get the same error.
My input and label arrays have the same shape as the ones in the _run_lm_finetuning_ example and my model is simply the Tensorflow equivalent to the model used there.

What am I doing wrong?
Is it possible that this is related to the loss calculation rather than the definition of the model?
I've noticed that in the _run_lm_finetuning_ example the model has an additional argument **masked_lm_labels**
```python
outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
```
that allows to compute the loss only on masked tokens using PyTorch, but this option is not present in TFBertForMaskedLM, how can I achieve that?


Added Camembert to available models

Added Camembert to the available models in the `run_lm_finetuning.py` example.


## ❓ Questions & Help
How to get a spiece.model from customize chinese vocab.txt in Albert xlnet ?


ALBERT is missing from AutoClasses

Pull request to fix this: https://github.com/huggingface/transformers/pull/1995


Add ALBERT to AutoClasses

Adds ALBERT to AutoClasses and also fixes some documentation mistakes along the way