unk_token not set when loading TransformerXLTokenizer.from_pretrained() from a save_pretrained()

Core: Tokenization

Internals of the library; Tokenization.

Using `TransfomerXLTokenizer.from_pretrained()` when loading from `.save_pretrained()` generated files.

Python complains about missing <unknown> token:

```
line 175, in _build_from_file
    raise ValueError("No <unkown> token in vocabulary")
ValueError: No <unkown> token in vocabulary
```


Create README.md for the new model fine tuned for Spanish POS tagging


kwargs are passed to both model and configuration in AutoModels

AutoModel doc says it passes arguments to config, but it actually also passes them to the models which makes them crash.


add explaining example to XLNet LM modeling

1) adds example explaining how XLNet could be used for standard auto-regressive modelling.
2) puts add_special_tokens=True for simple example to make sure no <sep> and <cls> are added to input. The are used for special training (similar to BERT) and might be confusing to user


Trying to Use AlbertTokenizer With my own custom Vocab file

Internals of the library; Tokenization.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->
## Details
<!-- Description of your issue -->
Good Afternoon All,

I am trying to used AlbertTokenizer to tokenize my corpus with a custom vocab file.  I tried the command :  **AlbertTokenizer("my-custom-vocab", do_lower_case=True, keep_accents=False, bos_token='[CLS]', eos_token='[SEP]', unk_token='<unk>', sep_token='[SEP]', pad_token='<pad>', cls_token='[CLS]', mask_token='[MASK]', )**  but it gives an error : " **_RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())]"_**  . a subser of my Vocab file is attached, what am I doing wrong?

Regards,
[vocab.txt](https://github.com/huggingface/transformers/files/4246094/vocab.txt)

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


No optimizer steps when gradient_accumulation_steps smaller than epoch_iterator length

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Doesn't matter.

Language I am using the model on (English, Chinese ...): Doesn't matter.

The problem arises when using:
* [x] the official example scripts
* [ ] my own modified scripts

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Any of the tasks.
* [ ] my own task or dataset

## To reproduce

Steps to reproduce the behavior:

1. Run `run_glue.py` where number of batches for each epoch is smaller than gradient_accumulation_steps.

In https://github.com/huggingface/transformers/blob/8194df8e0cff8e5866ec2bcbda34e3892f10eb39/examples/run_glue.py#L233

`step` is at most `len(train_dataloader)/batch_size`. If the latter is small, then we would never enter the if condition and never call `optimizer.step()` and so on.

I know it's the user responsibility to be aware of this, but this can be easily solved by by altering the condition to be:
```
if (step + 1) % args.gradient_accumulation_steps == 0 or ((step + 1) < args.gradient_accumulation_steps and (step + 1) == len(epoch_iterator)):
```

Should I create a small PR?


Tf qa pipelines test


Too many bugs in Version 2.5.0

Internals of the library; Tokenization.

Should Fix

This has been identified as a bug and should be fixed.

1. It cannot be installed on MacOS. By runing `pip install -U transformers`, I got the following errors:
>   Building wheels for collected packages: tokenizers
>   Building wheel for tokenizers (PEP 517) ... error
>   ERROR: Command errored out with exit status 1:
>   command: /anaconda/bin/python /anaconda/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/5h/fr2vhgsx4jd8wz4bphzt22_8p1v0bf/T/tmpfh6km7na
>   cwd: /private/var/folders/5h/fr2vhgsx4jd8wz4bphzt22_8p1v0bf/T/pip-install-fog09t3h/tokenizers
>   Complete output (36 lines):
>   running bdist_wheel
>   running build
>   running build_py
>   creating build
>   creating build/lib
>   creating build/lib/tokenizers
>   copying tokenizers/__init__.py -> build/lib/tokenizers
>   creating build/lib/tokenizers/models
>   copying tokenizers/models/__init__.py -> build/lib/tokenizers/models
>   creating build/lib/tokenizers/decoders
>   copying tokenizers/decoders/__init__.py -> build/lib/tokenizers/decoders
>   creating build/lib/tokenizers/normalizers
>   copying tokenizers/normalizers/__init__.py -> build/lib/tokenizers/normalizers
>   creating build/lib/tokenizers/pre_tokenizers
>   copying tokenizers/pre_tokenizers/__init__.py -> build/lib/tokenizers/pre_tokenizers
>   creating build/lib/tokenizers/processors
>   copying tokenizers/processors/__init__.py -> build/lib/tokenizers/processors
>   creating build/lib/tokenizers/trainers
>   copying tokenizers/trainers/__init__.py -> build/lib/tokenizers/trainers
>   creating build/lib/tokenizers/implementations
>   copying tokenizers/implementations/byte_level_bpe.py -> build/lib/tokenizers/implementations
>   copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib/tokenizers/implementations
>   copying tokenizers/implementations/base_tokenizer.py -> build/lib/tokenizers/implementations
>   copying tokenizers/implementations/__init__.py -> build/lib/tokenizers/implementations
>   copying tokenizers/implementations/char_level_bpe.py -> build/lib/tokenizers/implementations
>   copying tokenizers/implementations/bert_wordpiece.py -> build/lib/tokenizers/implementations
>   copying tokenizers/__init__.pyi -> build/lib/tokenizers
>   copying tokenizers/models/__init__.pyi -> build/lib/tokenizers/models
>   copying tokenizers/decoders/__init__.pyi -> build/lib/tokenizers/decoders
>   copying tokenizers/normalizers/__init__.pyi -> build/lib/tokenizers/normalizers
>   copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib/tokenizers/pre_tokenizers
>   copying tokenizers/processors/__init__.pyi -> build/lib/tokenizers/processors
>   copying tokenizers/trainers/__init__.pyi -> build/lib/tokenizers/trainers
>   running build_ext
>   running build_rust
>   error: Can not find Rust compiler
>   ----------------------------------------
>   ERROR: Failed building wheel for tokenizers
>   Running setup.py clean for tokenizers
> Failed to build tokenizers
> ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

2. On Linux, it can be installed, but failed with the following code:
> import transformers
> transformers.AutoTokenizer.from_pretrained("bert-base-cased").save_pretrained("./")
> transformers.AutoModel.from_pretrained("bert-base-cased").save_pretrained("./")
> transformers.AutoTokenizer.from_pretrained("./")
> transformers.AutoModel.from_pretrained("./")

Actually, it is the second line that generates the following errors:
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/anaconda/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 587, in save_pretrained
>     return vocab_files + (special_tokens_map_file, added_tokens_file)
> TypeError: unsupported operand type(s) for +: 'NoneType' and 'tuple'

3. The vocabulary size of xlm-roberta is wrong, so it failed with the following code,  (this bug also exist in Version 2.4.1):
> import transformers
> tokenizer = transformers.AutoTokenizer.from_pretrained("xlm-roberta-base")
> tokenizer.convert_ids_to_tokens(range(tokenizer.vocab_size))

The error is actually caused by the wrong vocab size:
> [libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1506] CHECK failed: (index) < (current_size_):
> terminate called after throwing an instance of 'google::protobuf::FatalException'
>   what():  CHECK failed: (index) < (current_size_):
> zsh: abort      python


# ❓ Questions & Help

Train TFXLNetForSequenceClassification model failed.

## Details
**train_dataset,dev_dataset:**
`<RepeatDataset shapes: ({input_ids: (None, None), attention_mask: (None, None), token_type_ids: (None, None)}, (None,)), types: ({input_ids: tf.int32, attention_mask: tf.int32, token_type_ids: tf.int32}, tf.int64)>`
```
model = TFXLNetForSequenceClassification.from_pretrained(path)
model.config.num_labels=1
train_steps = 10
valid_steps = 5
model.fit(train_dataset,
          epochs=6,
          steps_per_epoch=train_steps,
          validation_data=dev_dataset,
          validation_steps=valid_steps,)

ValueError                                Traceback (most recent call last)
<ipython-input-76-7d07613f7463> in <module>
      5           steps_per_epoch=train_steps,
      6           validation_data=dev_dataset,
----> 7           validation_steps=valid_steps,)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    817         max_queue_size=max_queue_size,
    818         workers=workers,
--> 819         use_multiprocessing=use_multiprocessing)
    820
    821   def evaluate(self,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    233           max_queue_size=max_queue_size,
    234           workers=workers,
--> 235           use_multiprocessing=use_multiprocessing)
    236
    237       total_samples = _get_total_number_of_samples(training_data_adapter)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in _process_training_inputs(model, x, y, batch_size, epochs, sample_weights, class_weights, steps_per_epoch, validation_split, validation_data, validation_steps, shuffle, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    591         max_queue_size=max_queue_size,
    592         workers=workers,
--> 593         use_multiprocessing=use_multiprocessing)
    594     val_adapter = None
    595     if validation_data:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in _process_inputs(model, mode, x, y, batch_size, epochs, sample_weights, class_weights, shuffle, steps, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    704       max_queue_size=max_queue_size,
    705       workers=workers,
--> 706       use_multiprocessing=use_multiprocessing)
    707
    708   return adapter

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in __init__(self, x, y, sample_weights, standardize_function, **kwargs)
    700
    701     if standardize_function is not None:
--> 702       x = standardize_function(x)
    703
    704     # Note that the dataset instance is immutable, its fine to reusing the user

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in standardize_function(dataset)
    682           return x, y
    683         return x, y, sample_weights
--> 684       return dataset.map(map_fn, num_parallel_calls=dataset_ops.AUTOTUNE)
    685
    686   if mode == ModeKeys.PREDICT:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in map(self, map_func, num_parallel_calls)
   1589     else:
   1590       return ParallelMapDataset(
-> 1591           self, map_func, num_parallel_calls, preserve_cardinality=True)
   1592
   1593   def flat_map(self, map_func):

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in __init__(self, input_dataset, map_func, num_parallel_calls, use_inter_op_parallelism, preserve_cardinality, use_legacy_function)
   3924         self._transformation_name(),
   3925         dataset=input_dataset,
-> 3926         use_legacy_function=use_legacy_function)
   3927     self._num_parallel_calls = ops.convert_to_tensor(
   3928         num_parallel_calls, dtype=dtypes.int32, name="num_parallel_calls")

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in __init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
   3145       with tracking.resource_tracker_scope(resource_tracker):
   3146         # TODO(b/141462134): Switch to using garbage collection.
-> 3147         self._function = wrapper_fn._get_concrete_function_internal()
   3148
   3149         if add_to_graph:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _get_concrete_function_internal(self, *args, **kwargs)
   2393     """Bypasses error checking when getting a graph function."""
   2394     graph_function = self._get_concrete_function_internal_garbage_collected(
-> 2395         *args, **kwargs)
   2396     # We're returning this concrete function to someone, and they may keep a
   2397     # reference to the FuncGraph without keeping a reference to the

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2387       args, kwargs = None, None
   2388     with self._lock:
-> 2389       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2390     return graph_function
   2391

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _maybe_define_function(self, args, kwargs)
   2701
   2702       self._function_cache.missed.add(call_context_key)
-> 2703       graph_function = self._create_graph_function(args, kwargs)
   2704       self._function_cache.primary[cache_key] = graph_function
   2705       return graph_function, args, kwargs

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2591             arg_names=arg_names,
   2592             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2593             capture_by_value=self._capture_by_value),
   2594         self._function_attributes,
   2595         # Tell the ConcreteFunction to clean up its graph once it goes out of

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    976                                           converted_func)
    977
--> 978       func_outputs = python_func(*func_args, **func_kwargs)
    979
    980       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in wrapper_fn(*args)
   3138           attributes=defun_kwargs)
   3139       def wrapper_fn(*args):  # pylint: disable=missing-docstring
-> 3140         ret = _wrapper_helper(*args)
   3141         ret = structure.to_tensor_list(self._output_structure, ret)
   3142         return [ops.convert_to_tensor(t) for t in ret]

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in _wrapper_helper(*args)
   3080         nested_args = (nested_args,)
   3081
-> 3082       ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
   3083       # If `func` returns a list of tensors, `nest.flatten()` and
   3084       # `ops.convert_to_tensor()` would conspire to attempt to stack

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\autograph\impl\api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

ValueError: in converted code:

    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py:677 map_fn
        batch_size=None)
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py:2410 _standardize_tensors
        exception_prefix='input')
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_utils.py:510 standardize_input_data
        'for each key in: ' + str(names))

    ValueError: No data provided for "inputs". Need data for each key in: ['attention_mask', 'inputs', 'token_type_ids']


run_ner.py / bert-base-multilingual-cased can output empty tokens

This can happen when using bert-base-multilingual-cased with an input containing an unique space.
In this case, the tokenizer will output just an empty word_tokens thus leading to an non-consistent behavior
over the labels_ids tokens adding one more tokens than tokens vector.


run_ner.py example

# ❓ Questions & Help

## Details
It seem that examples/run_ner.py has been removed from github repo. Is there any issue with this code?
Thanks


Documentation

Updating documentation for tokenizers.

~Still left to do:~ will do in a future PR


Speed up GELU computation with torch.jit

Currently, the implementation of the GELU activation uses several unfused pointwise operations. In my experiments, computing this activation takes about 10% of forward time for GPT2-like networks for inputs of size similar to (32,128). This PR speeds up the execution of gelu_new during both forward (~3-5x) and backward (~2-3x) passes with the help of torch.jit, which might be helpful for both training and inference.

Below are the benchmarking results, done on pytorch v1.4.0 and transformers v2.5.0 with RTX 2080Ti and GTX 1080Ti. The benchmarking code is available [here](https://gist.github.com/mryab/d639d7ba5e741cdd6a6c712a118e97f8).

1080Ti:
```
torch.float32   (32, 128)       gelu 2.6e-04    4.1e-04 jit 1.1e-04     1.8e-04 speedup forward 2.50    backward 2.27
torch.float32   (32, 512)       gelu 2.6e-04    4.1e-04 jit 6.5e-05     1.5e-04 speedup forward 4.06    backward 2.67
torch.float32   (32, 1024)      gelu 2.6e-04    4.0e-04 jit 6.7e-05     1.6e-04 speedup forward 3.94    backward 2.59
torch.float32   (32, 4096)      gelu 2.5e-04    3.9e-04 jit 6.6e-05     1.6e-04 speedup forward 3.75    backward 2.51
torch.float32   (256, 128)      gelu 2.7e-04    4.1e-04 jit 6.7e-05     1.6e-04 speedup forward 3.96    backward 2.61
torch.float32   (256, 512)      gelu 2.5e-04    4.0e-04 jit 6.5e-05     1.5e-04 speedup forward 3.88    backward 2.57
torch.float32   (256, 1024)     gelu 2.5e-04    4.0e-04 jit 6.2e-05     1.5e-04 speedup forward 4.05    backward 2.62
torch.float32   (256, 4096)     gelu 2.6e-04    4.2e-04 jit 1.0e-04     1.7e-04 speedup forward 2.52    backward 2.45
torch.float32   (1024, 128)     gelu 2.5e-04    3.9e-04 jit 6.5e-05     1.5e-04 speedup forward 3.82    backward 2.57
torch.float32   (1024, 512)     gelu 2.5e-04    3.8e-04 jit 7.2e-05     1.5e-04 speedup forward 3.43    backward 2.52
torch.float32   (1024, 1024)    gelu 2.6e-04    4.2e-04 jit 1.0e-04     1.7e-04 speedup forward 2.52    backward 2.44
torch.float32   (1024, 4096)    gelu 8.8e-04    1.3e-03 jit 3.2e-04     3.5e-04 speedup forward 2.71    backward 3.79
torch.float32   (8192, 128)     gelu 2.6e-04    4.2e-04 jit 1.0e-04     1.7e-04 speedup forward 2.51    backward 2.43
torch.float32   (8192, 512)     gelu 8.8e-04    1.3e-03 jit 3.2e-04     3.5e-04 speedup forward 2.72    backward 3.80
torch.float32   (8192, 1024)    gelu 1.7e-03    2.5e-03 jit 6.4e-04     5.9e-04 speedup forward 2.69    backward 4.30
torch.float32   (8192, 4096)    gelu 6.7e-03    1.0e-02 jit 2.7e-03     2.5e-03 speedup forward 2.53    backward 4.05
torch.float16   (32, 128)       gelu 2.6e-04    4.0e-04 jit 9.4e-05     1.8e-04 speedup forward 2.79    backward 2.24
torch.float16   (32, 512)       gelu 2.5e-04    3.9e-04 jit 6.2e-05     1.4e-04 speedup forward 4.09    backward 2.74
torch.float16   (32, 1024)      gelu 2.6e-04    4.0e-04 jit 6.2e-05     1.5e-04 speedup forward 4.22    backward 2.68
torch.float16   (32, 4096)      gelu 2.4e-04    3.8e-04 jit 6.3e-05     1.5e-04 speedup forward 3.84    backward 2.56
torch.float16   (256, 128)      gelu 2.6e-04    4.0e-04 jit 6.1e-05     1.4e-04 speedup forward 4.34    backward 2.81
torch.float16   (256, 512)      gelu 2.5e-04    3.9e-04 jit 6.4e-05     1.5e-04 speedup forward 3.98    backward 2.59
torch.float16   (256, 1024)     gelu 2.4e-04    3.7e-04 jit 6.3e-05     1.4e-04 speedup forward 3.82    backward 2.65
torch.float16   (256, 4096)     gelu 2.3e-04    3.2e-04 jit 7.6e-05     1.4e-04 speedup forward 3.00    backward 2.32
torch.float16   (1024, 128)     gelu 2.2e-04    3.2e-04 jit 6.3e-05     1.4e-04 speedup forward 3.47    backward 2.32
torch.float16   (1024, 512)     gelu 2.2e-04    3.2e-04 jit 6.3e-05     1.4e-04 speedup forward 3.47    backward 2.31
torch.float16   (1024, 1024)    gelu 2.3e-04    3.2e-04 jit 7.6e-05     1.4e-04 speedup forward 3.01    backward 2.31
torch.float16   (1024, 4096)    gelu 5.4e-04    8.9e-04 jit 2.2e-04     2.6e-04 speedup forward 2.44    backward 3.40
torch.float16   (8192, 128)     gelu 2.5e-04    3.8e-04 jit 7.6e-05     1.5e-04 speedup forward 3.29    backward 2.61
torch.float16   (8192, 512)     gelu 5.4e-04    8.9e-04 jit 2.2e-04     2.5e-04 speedup forward 2.43    backward 3.49
torch.float16   (8192, 1024)    gelu 1.0e-03    1.7e-03 jit 4.8e-04     4.6e-04 speedup forward 2.18    backward 3.60
torch.float16   (8192, 4096)    gelu 4.2e-03    6.5e-03 jit 2.3e-03     2.0e-03 speedup forward 1.83    backward 3.30
```

RTX 2080Ti:
```
torch.float32   (32, 128)       gelu 3.0e-04    6.2e-04 jit 1.2e-04     2.2e-04 speedup forward 2.50    backward 2.80
torch.float32   (32, 512)       gelu 3.2e-04    6.8e-04 jit 6.8e-05     2.1e-04 speedup forward 4.66    backward 3.20
torch.float32   (32, 1024)      gelu 3.4e-04    7.2e-04 jit 6.8e-05     2.1e-04 speedup forward 4.96    backward 3.38
torch.float32   (32, 4096)      gelu 3.3e-04    7.0e-04 jit 6.4e-05     1.8e-04 speedup forward 5.07    backward 3.83
torch.float32   (256, 128)      gelu 3.3e-04    6.9e-04 jit 6.5e-05     1.9e-04 speedup forward 5.07    backward 3.57
torch.float32   (256, 512)      gelu 3.0e-04    6.2e-04 jit 6.4e-05     1.9e-04 speedup forward 4.73    backward 3.21
torch.float32   (256, 1024)     gelu 3.3e-04    6.9e-04 jit 6.6e-05     2.1e-04 speedup forward 4.95    backward 3.35
torch.float32   (256, 4096)     gelu 3.3e-04    6.8e-04 jit 9.3e-05     2.2e-04 speedup forward 3.53    backward 3.09
torch.float32   (1024, 128)     gelu 3.1e-04    6.2e-04 jit 6.5e-05     1.9e-04 speedup forward 4.70    backward 3.32
torch.float32   (1024, 512)     gelu 3.4e-04    6.4e-04 jit 7.7e-05     1.9e-04 speedup forward 4.41    backward 3.30
torch.float32   (1024, 1024)    gelu 3.1e-04    6.1e-04 jit 9.5e-05     2.2e-04 speedup forward 3.26    backward 2.73
torch.float32   (1024, 4096)    gelu 6.2e-04    9.9e-04 jit 2.7e-04     3.1e-04 speedup forward 2.26    backward 3.15
torch.float32   (8192, 128)     gelu 3.1e-04    4.9e-04 jit 9.7e-05     1.9e-04 speedup forward 3.13    backward 2.55
torch.float32   (8192, 512)     gelu 6.1e-04    1.0e-03 jit 2.7e-04     3.4e-04 speedup forward 2.27    backward 2.99
torch.float32   (8192, 1024)    gelu 1.2e-03    1.9e-03 jit 5.3e-04     5.5e-04 speedup forward 2.21    backward 3.38
torch.float32   (8192, 4096)    gelu 4.5e-03    6.7e-03 jit 2.2e-03     1.6e-03 speedup forward 2.04    backward 4.24
torch.float16   (32, 128)       gelu 3.2e-04    6.3e-04 jit 1.1e-04     2.2e-04 speedup forward 2.84    backward 2.92
torch.float16   (32, 512)       gelu 3.3e-04    6.9e-04 jit 6.2e-05     1.6e-04 speedup forward 5.23    backward 4.29
torch.float16   (32, 1024)      gelu 3.0e-04    5.9e-04 jit 6.5e-05     1.7e-04 speedup forward 4.58    backward 3.46
torch.float16   (32, 4096)      gelu 3.0e-04    6.1e-04 jit 6.4e-05     1.8e-04 speedup forward 4.63    backward 3.34
torch.float16   (256, 128)      gelu 3.0e-04    5.9e-04 jit 6.4e-05     1.7e-04 speedup forward 4.61    backward 3.49
torch.float16   (256, 512)      gelu 3.0e-04    5.9e-04 jit 6.3e-05     1.7e-04 speedup forward 4.68    backward 3.41
torch.float16   (256, 1024)     gelu 2.9e-04    5.7e-04 jit 6.5e-05     1.6e-04 speedup forward 4.40    backward 3.54
torch.float16   (256, 4096)     gelu 2.9e-04    5.5e-04 jit 7.5e-05     2.0e-04 speedup forward 3.87    backward 2.74
torch.float16   (1024, 128)     gelu 3.7e-04    6.3e-04 jit 8.0e-05     2.3e-04 speedup forward 4.59    backward 2.75
torch.float16   (1024, 512)     gelu 3.4e-04    6.0e-04 jit 6.6e-05     1.6e-04 speedup forward 5.13    backward 3.81
torch.float16   (1024, 1024)    gelu 3.0e-04    5.9e-04 jit 7.2e-05     1.9e-04 speedup forward 4.12    backward 3.08
torch.float16   (1024, 4096)    gelu 4.1e-04    6.9e-04 jit 1.6e-04     2.6e-04 speedup forward 2.49    backward 2.68
torch.float16   (8192, 128)     gelu 3.6e-04    6.6e-04 jit 7.0e-05     1.8e-04 speedup forward 5.08    backward 3.73
torch.float16   (8192, 512)     gelu 4.1e-04    7.0e-04 jit 1.6e-04     2.5e-04 speedup forward 2.57    backward 2.76
torch.float16   (8192, 1024)    gelu 7.4e-04    1.2e-03 jit 3.2e-04     4.1e-04 speedup forward 2.30    backward 2.81
torch.float16   (8192, 4096)    gelu 2.8e-03    3.9e-03 jit 1.5e-03     1.2e-03 speedup forward 1.86    backward 3.34
```


Add preprocessing step for transfo-xl tokenization to avoid tokenizing words followed by punction to <unk>

The problem is well shown in Issue #2000 .
This PR adds a preprocessing step to transfo-xl tokenization to better deal with the problem.


How to generate BERT/Roberta word/sentence embedding?

Discussion

Discussion on a topic (keep it focused or open a new issue though)

Usage

General questions about the library

I know the stanford operation.
```python
tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
model = RobertaModel.from_pretrained('roberta-large')

input_ids = torch.tensor(tokenizer.encode("Hello, my <span class="highlighter highlight-on">dog</span> is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)

last_hidden_states = outputs[0] #(batch_size, input_len, embedding_size) But I need single vector for each sentence
```

But. I am working on improving RNN with incorporating Bert-like pretrain model embedding. How to get a sentence embedding so in this case(one vector for entire sentence)?   Averaging or some transformation of the last_hidden_states?  Is `add_special_token` necessary? Any suggested papers to read?


`AutoModel.from_pretrained` sends config kwargs to model

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert (may apply to more)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
from transformers import AutoModel
AutoModel.from_pretrained('bert-base-uncased', output_attention=True)
```

([example from the docs](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoModel.from_pretrained))

It crashes:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[...]/transformers/src/transformers/modeling_auto.py", line 384, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "[...]/transformers/src/transformers/modeling_utils.py", line 463, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'output_attention'
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

That the code returns a correct model, without crashing

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.0 (master)
- Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: no


add_ctags_to_git_ignore


NER support for Albert in run_ner.py and NerPipeline

I have added a class, AlbertForTokenClassification, based on BertForTokenClassification and added it to lists used for checking NER capabilities in run_ner.py and NerPipeline.

I tested NER fine-tuning on albert-base-v2, albert-large-v2 and albert-xlarge-v2 on the english CONLL 2003 dataset and they all get F1 of around 0.93-0.94, so it seems to be working. The fine-tuned models are published [here](https://huggingface.co/KB).

I've also added some command-line options to better control tokenization since different tokenizers have different possible arguments and defaults. I guess that in the end, when all tokenizers behave the same, these options will be unnecessary.

I changed how NerPipeline outputs tokens, from .decode(..) to .convert_ids_to_tokens(...) since it removes '_' at the beginning of tokens making it impossible (for sentencepiece tokens) to know which tokens form a word. Using .convert(...) would make sense if it were  outputting whole words and not words split into tokens. It might make sense to change this so that NerPipeline outputs whole words. That would assume that all the tokens in a word gets classified with the same label, which is not always the case.

I had one weird thing happening: when fine-tuning albert-large-v2 specifically for 3 or 4 epochs F1 would be reported as exactly 0. When setting num_train_epochs to 2 or 5 this did not happen. I'm going to assume that this has nothing to do with the code submitted :)


Change masking to direct labeling for TPU support.

Torch XLA conversion is very sensitive to certain core pytorch operations. These result in TPU being slower than CPU operation. The most obvious are binary masking and calls to item().

https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md

This PR replaces some of these calls that are in the example/NER pathway.


Strange bug when Finetuning own pretrained model (with an even stranger solution)

# 🐛 Bug

## Information
Roberta

Language I am using the model on (English, Chinese ...): Latin script(migh have a mix of languages)

The problem arises when using:
run_glue on model obtained from run_language_modeling

The tasks I am working on is:
Sequence Classification(single)

Steps to reproduce the behavior:

1. Train model using run_language_modeling
2. Use trained model in run_glue script

Error:
File "run_glue.py", line 148, in train
    optimizer.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "optimizer.pt")))
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 116, in load_state_dict
    raise ValueError("loaded state dict contains a parameter group "
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group
Note: I searched what might cause this error(freezing some layers and passing an incorrect params_group). But I have not done anything like that, so this error should not occur.

## Quick Hack/Solution:
There is a strange solution simply by deleting optimizer.pt and setting number of epochs to an arbitrarily large number. Not setting epochs to a very high number causes the script to proceed directly to evaluation and not do any training.

## Environment info
Google Colab
Tokenizers 0.5
Transformers 2.5
GPU:P4