GPT2 should not store/compute cached activations during finetuning

This PR tries to fix the issue with large memory usage from GPT2 during fine-tuning.

## Quick estimations

@LysandreJik compared memory usage with @minimaxir GPT2-simple (https://github.com/minimaxir/gpt-2-simple):

*Small model*, batch size 4, sequence length 512 (roughly similar):
- us => 9.9GB,
- GPT2-simple => 8.5GB

Increasing to a 1024 length:
- us => 20.4GB...,
- GPT2-simple => still 8.5GB

*Medium model*, batch size de 4, sequence length de 512
- us => 23.36GB. OOM on a titan with 1024 seq len.
- GPT2-simple throws an error related to layers not contained in the checkpoint

## Possible reason

Investigating our `run_lm_finetuning` script and GPT2 model showed that we are alway computing/storing cached hidden-states (which are normally only useful for decoding).

This PR attempt to fix this most probable source of large memory usage.
It cleans up a little bit GPT2 codebase at the same time.

I haven't tried it yet on a large scale test.

cc @LysandreJik @arnicas


transformers command not found after installing transformers using pip

I wanted to convert TF checkpoints to pytorch saved files and thus I followed instructions as mentioned in the link https://huggingface.co/transformers/converting_tensorflow_models.html

Thus, I installed PyTorch, Tensorflow and then transformers. But after doing so, when I ran the command, my system prompted me transformers command not found.

What can be the issue here.


[debug] Debug Heisenbug, the old school way.


[http] Tweak http user-agent


Cli tweaks


GLUE Benchmark Hyperparameters

## ❓ Questions & Help

In the `run_glue.py` script, are the hyperparameters for running BERT, RoBERTa, ALBERT, etc. the exact same? The documentation does not seem to outline separate hyperparameters, but the papers of each respective model show different hyperparameter ranges. I'm wondering if this was taken account when reporting the benchmark results.


Trouble fine tuning BERT language model

## 🐛 Bug

Hello, I'm having trouble running **run_lm_finetuning.py** script. I'm using pytorch 1.2, python 3.5, CUDA 9.2, Ubuntu 18.04.

When I run

```
$python run_lm_finetuning.py
--output_dir= my_output_dir/
--model_type=bert
--model_name_or_path=bert-base-uncased
--do_train
--train_data_file=$TRAIN_FILE
--do_eval
--eval_data_file=$TEST_FILE
--mlm
```

I obtain:

```
***** Running training *****
Num examples = 4517
Num Epochs = 1
Instantaneous batch size per GPU = 4
Total train batch size (w. parallel, distributed & accumulation) = 4
Gradient Accumulation steps = 1
Total optimization steps = 1130
Epoch:   0%|                                                                                | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):                                                               | 0/1130 [00:00<?, ?it/s]
  File "lm_fine_backup.py", line 712, in <module>
    main()
  File "lm_fine_backup.py", line 662, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "lm_fine_backup.py", line 298, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mbugueno/.local/lib/python3.5/site-packages/transformers/modeling_bert.py", line 899, in forward
    masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/functional.py", line 2009, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97
```

I'm using the official example script in WikiText-2 dataset.
Any insights?


Enforce target version for black.

This should stabilize formatting.

As suggested by @julien-c.


CamembertForQuestionAnswering

Hi,
is it possibile tu add _CamembertForQuestionAnswering_ class that extend _RobertaForQuestionAnswering_ into **src/transformers/modeling_camembert.py**, **src/transformers/__init__.py** and **examples/run_squad.py** ?

I had to manually force it in order to execute _run_squad.py_ with a CamemBERT-like network.

Thanks.


revise T5 code to support one step decoding during generation

@thomwolf Hi, I'm new to contribute to this project. I revised the T5 code to support one step decoding during generation based on your implementation. Besides adding `decode_step` function, I also revised some others to pass in `cache` variable. Meanwhile, I added `bos_token` in tokenizer_t5 so that `bos_token` can be used as the first token during decoding. The example usage is as follows:
```
Examples:
            encoder_hidden_states = model.encode(input_ids)
            cache = model.init_state_from_encoder(encoder_hidden_states)
            next_token = input_ids.new_full((batch_size, 1), tokenizer.bos_token_id)
            generated = [next_token]
            for i in range(100):
                output, cache = model.decode_step(cache, input_ids=next_token)
                next_token = torch.argmax(logits, dim=-1).unsqueeze(-1)
                generated += [next_token]
            generated = torch.cat(generated, dim=1).tolist()
```
Let me know whether it is useful and can be merged.


Why does the BertForQuestionAnswering sample code duplicate the [CLS] token?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
The BertForQuestionAnswering sample code creates duplicate [CLS] tokens. Wondering why:

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
input_ids = tokenizer.encode(input_text)
token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# a nice puppet
tokenizer.decode(input_ids)
#'[CLS] [CLS] who was jim henson? [SEP] jim henson was a nice puppet [SEP] [SEP]'
```

If I remove the extra [CLS], the extraction doesn't work. It's exactly two tokens off:
```
input_ids = tokenizer.encode(input_text, add_special_tokens=False)
...rerun same code as above...
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# was a
```
What am I doing wrong? How can I get the extraction working without duplicate [CLS] tokens? (and duplicate final [SEP] tokens BTW).

The sample code comes right from the docs: https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering


Feature Request: Pipeline for Query/Document relevance

Core: Pipeline

Internals of the library; Pipeline.

# Pipelines for IR tasks

## Justification

In the last few years, a bunch of deep architectures were proposed for Ad-hoc retrieval, most with limited success (if any). However, BERT(et al)-based models are finally pushing the state of the art for Ad-hoc retrieval. In fact in the last TREC had a [Deep Learning track](https://microsoft.github.io/TREC-2019-Deep-Learning/) whre "NNLM" (neural network language models) [dominated](https://twitter.com/UnderdogGeek/status/1206595356017848324/photo/1) any other traditional (Mostly BM25 and variations) and other deep models.

So, it's a current trend that BERT should be the new baseline for any proposed model for IR.

## Description

There should be a [pipeline-like](https://github.com/huggingface/transformers#quick-tour-of-pipelines) feature that is able to score pairs of documents and user queries. Probably, pre-trained on a dataset like the [MSMarco dataset for TREC'19](https://microsoft.github.io/TREC-2019-Deep-Learning/). Ideally, this would also support a list of documents to rank and return scores.

In real-life applications, one would probably want to combine BERT scores with a traditional baseline scores (like QL or BM25). So, the score is needed (or, even better, combine something like [pyserini](https://github.com/castorini/pyserini) in the backend?)

## Usage
```
from transformers import pipeline

# Allocate a pipeline for sentiment-analysis
nlp = pipeline('document-relevancy')
nlp({
    'query 'can hives be a sign of pregnancy',
    'context' '<document content>'
})
>>> {'score': 0.28756016668193496'}
```

I have already used DistilBERT on a paper to appear on ECIR2020 (Diagnosing BERT with Retrieval Heuristics), and would be able to contribute the model for this (even for bert-base).

I would also love to contribute with this, but will probably need some guidance, if anyone is willing to help.

Thanks!


How to run bert without checkpoints

I would like to run BERT from scratch with no checkpoints for my language (PT-BR) and make a comparison with the multilingual model!

I am currently running BERT-Native provided by google to get checkpoints from scratch and then converting to pytorch. But it is a time consuming process!

can anybody help me?


How to finetune PreTrainedEncoderDecoder

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
PreTrainedEncoderDecoder is great.
Now I have the following questions :
(1) How to use my data to finetune the PreTrainedEncoderDecoder?
(2) If I want to use the pretrained RoBERTa as encoder and deocder, what should I do ?


Tokenizers as optional dependency

- `tokenizers` as an optional dependency (`pip install -e .[fast]`)
- code formating with `make style` `make quality`


"Reformer: The Efficient Transformer" looks awesome. I'd love to see it in the library.

# 🌟New model addition

## Model description
Efficient Transformer with locality-sensitive hashing and reversible layers
https://openreview.net/forum?id=rkgNKkHtvB
<!-- Important information -->

## Open Source status

* [. ] the model implementation is available: (give details)
There is an implementation from google
https://github.com/google/trax/blob/master/trax/models/research/reformer.py
* [ ] the model weights are available: (give details)
* [. ] who are the authors: (mention them)
Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya

## Additional context

<!-- Add any other context about the problem here. -->


Bert cross attention

## ❓ Questions & Help

In the standard Transformer/Bert architecture, what is the intuition behind cross attention doing a weighted average over the encoder_hidden_states? What happens if we set the value layer to decoder hidden_state instead?

See the cross attention value layer being set to the encoder_hidden_states at [modeling_bert.py#L238](https://github.com/huggingface/transformers/blob/8c67b529f615cc24c46864b8323d2d47a15ccd58/src/transformers/modeling_bert.py#L238), and the weighted average being taken at [modeling_bert.py#L266](https://github.com/huggingface/transformers/blob/8c67b529f615cc24c46864b8323d2d47a15ccd58/src/transformers/modeling_bert.py#L266)


read each lines, require less memory

The original code reads whole data at once, so it requires so much memory to handle huge corpus.
This change is:
* read corpus by each lines
* flatten 2-dimension array by itertools.chain, it requies less memory and fast


Summarization ROGUE scores don't equal that of the paper ...

## ❓ Questions & Help

Just ran the `run_summarization.py` script, with the parameters specified [here](https://github.com/huggingface/transformers/tree/master/examples/summarization) and the ROGUE scores are far off from what is reported in the related paper.

The ROGUE scores reported in [PreSumm paper](https://github.com/nlpyang/PreSumm) (R1, R2, RL):

> BertSumExtAbs | 42.13 | 19.60 | 39.18

The ROGUE scores after running the HF script:

> ROGUE 1:
> F1 = .275
> Precision = .299
> Recall = .260
>
> ROGUE 2:
> F1 = .161
> Precision = .184
> Recall = .149
>
> ROGUE L:
> F1 = .305
> Precision = .326
> Recall = .290

The README file seems to suggest that running the script as is, with all the stories in a single directory, will give you ROGUE scores similar to that of the paper.  That doesn't seem the case.

***Any ideas why? Or what I may be doing wrong here?***

Thanks much!

FYI ... ran the script as in the README:

```
python run_summarization.py \
    --documents_dir $STORIES_DIR
    --summaries_output_dir $OUTPUT_SUM_DIR
    --no_cuda false \
    --batch_size 4 \
    --min_length 50 \
    --max_length 200 \
    --beam_size 5 \
    --alpha 0.95 \
    --block_trigram true \
    --compute_rouge true
```


Dropout rates to be updated in all ALBERT v2 configs

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): ALBERT v2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: run_squad.py
* [ ] my own modified scripts:

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQuAD v2.0
* [ ] my own task or dataset:

## To Reproduce

Steps to reproduce the behavior:

1. Just choose one of the following default ALBERT v2 configs: base, large, xlarge

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

As stated in the updated ALBERT Tensorflow repo from Google Research, model v2 introduces no dropout at all for downstream tasks, like SQuAD, MRPC or COLA.
Following the discussing from this [issue](https://github.com/google-research/ALBERT/issues/23), model configurations on TF-Hub were wrong and thus, the ones used by transformers (loaded on Amazon S3).

<s>While configs on TF-Hub will be updated in the near future</s> As configs on TF-Hub have already been updated, transformers' ones should be updated too: parameters `attention_probs_dropout_prob` and `hidden_dropout_prob` should be both 0 for all v2 configs.

## Environment

* OS: Platform Linux-4.14.152-98.182.amzn1.x86_64-x86_64-with-glibc2.9
* Python version: Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
* PyTorch version: PyTorch 1.3.1
* PyTorch Transformers version (or branch):
* Using GPU ? YES, 4x NVIDIA V100 16GB
* Distributed of parallel setup ? No
* Any other relevant information: Using Amazon AWS Deep Learning Linux AMI 26.0

## Additional context

Hope this will be useful!


TypeError: Expected Operation, Variable, or Tensor, got None while saving tensorflow model

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): TFAlbert

Language I am using the model on (English, Chinese....): English

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name): GLUE
* [ ] my own task or dataset: (give details)

## Environment

* OS: Linux
* Python version: Python 3.7.5 /
* Tensorflow version: 2.0.0
* Using GPU ? GPU

**System information**
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux
- TensorFlow installed from (source or binary): Source
- TensorFlow version: '2.0.0'
- Python version: Python 3.7.5 /Conda
- CUDA/cuDNN version: cuda10.0_0/cudnn-7.6.5
- GPU model and memory: Tesla V100-PCIE / 32 GB memory

**Describe the current behavior**
**I am getting TypeError: Expected Operation, Variable, or Tensor, got None while saving the model using model.save('../output/my_model')**

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-49-5ab71d0ebc23> in <module>
----> 1 model.save('../output/my_model')

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options)
    973     """
    974     saving.save_model(self, filepath, overwrite, include_optimizer, save_format,
--> 975                       signatures, options)
    976
    977   def save_weights(self, filepath, overwrite=True, save_format=None):

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options)
    113   else:
    114     saved_model_save.save(model, filepath, overwrite, include_optimizer,
--> 115                           signatures, options)
    116
    117

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save.py in save(model, filepath, overwrite, include_optimizer, signatures, options)
     72   # default learning phase placeholder.
     73   with K.learning_phase_scope(0):
---> 74     save_lib.save(model, filepath, signatures, options)
     75
     76   if not include_optimizer:

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/saved_model/save.py in save(obj, export_dir, signatures, options)
    868   if signatures is None:
    869     signatures = signature_serialization.find_function_to_export(
--> 870         checkpoint_graph_view)
    871
    872   signatures = signature_serialization.canonicalize_signatures(signatures)

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/saved_model/signature_serialization.py in find_function_to_export(saveable_view)
     62   # If the user did not specify signatures, check the root object for a function
     63   # that can be made into a signature.
---> 64   functions = saveable_view.list_functions(saveable_view.root)
     65   signature = functions.get(DEFAULT_SIGNATURE_ATTR, None)
     66   if signature is not None:

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/saved_model/save.py in list_functions(self, obj)
    139     if obj_functions is None:
    140       obj_functions = obj._list_functions_for_serialization(  # pylint: disable=protected-access
--> 141           self._serialization_cache)
    142       self._functions[obj] = obj_functions
    143     return obj_functions

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _list_functions_for_serialization(self, serialization_cache)
   2420   def _list_functions_for_serialization(self, serialization_cache):
   2421     return (self._trackable_saved_model_saver
-> 2422             .list_functions_for_serialization(serialization_cache))
   2423
   2424

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/base_serialization.py in list_functions_for_serialization(self, serialization_cache)
     89         `ConcreteFunction`.
     90     """
---> 91     fns = self.functions_to_serialize(serialization_cache)
     92
     93     # The parent AutoTrackable class saves all user-defined tf.functions, and

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/layer_serialization.py in functions_to_serialize(self, serialization_cache)
     77   def functions_to_serialize(self, serialization_cache):
     78     return (self._get_serialized_attributes(
---> 79         serialization_cache).functions_to_serialize)
     80
     81   def _get_serialized_attributes(self, serialization_cache):

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/layer_serialization.py in _get_serialized_attributes(self, serialization_cache)
     92
     93     object_dict, function_dict = self._get_serialized_attributes_internal(
---> 94         serialization_cache)
     95
     96     serialized_attr.set_and_validate_objects(object_dict)

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/model_serialization.py in _get_serialized_attributes_internal(self, serialization_cache)
     45     # cache (i.e. this is the root level object).
     46     if len(serialization_cache[constants.KERAS_CACHE_KEY]) == 1:
---> 47       default_signature = save_impl.default_save_signature(self.obj)
     48
     49     # Other than the default signature function, all other attributes match with

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in default_save_signature(layer)
    204   original_losses = _reset_layer_losses(layer)
    205   fn = saving_utils.trace_model_call(layer)
--> 206   fn.get_concrete_function()
    207   _restore_layer_losses(original_losses)
    208   return fn

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in get_concrete_function(self, *args, **kwargs)
    774       if self._stateful_fn is None:
    775         initializer_map = object_identity.ObjectIdentityDictionary()
--> 776         self._initialize(args, kwargs, add_initializers_to=initializer_map)
    777         self._initialize_uninitialized_variables(initializer_map)
    778

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    406     self._concrete_stateful_fn = (
    407         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 408             *args, **kwds))
    409
    410     def invalid_creator_scope(*unused_args, **unused_kwds):

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   1846     if self.input_signature:
   1847       args, kwargs = None, None
-> 1848     graph_function, _, _ = self._maybe_define_function(args, kwargs)
   1849     return graph_function
   1850

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   2148         graph_function = self._function_cache.primary.get(cache_key, None)
   2149         if graph_function is None:
-> 2150           graph_function = self._create_graph_function(args, kwargs)
   2151           self._function_cache.primary[cache_key] = graph_function
   2152         return graph_function, args, kwargs

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2039             arg_names=arg_names,
   2040             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2041             capture_by_value=self._capture_by_value),
   2042         self._function_attributes,
   2043         # Tell the ConcreteFunction to clean up its graph once it goes out of

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    913                                           converted_func)
    914
--> 915       func_outputs = python_func(*func_args, **func_kwargs)
    916
    917       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    356         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    357         # the function a weak reference to itself to avoid a reference cycle.
--> 358         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    359     weak_wrapped_fn = weakref.ref(wrapped_fn)
    360

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saving_utils.py in _wrapped_model(*args)
    141     with base_layer_utils.call_context().enter(
    142         model, inputs=inputs, build_graph=False, training=False, saving=True):
--> 143       outputs_list = nest.flatten(model(inputs=inputs, training=False))
    144
    145     try:

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    845                     outputs = base_layer_utils.mark_as_return(outputs, acd)
    846                 else:
--> 847                   outputs = call_fn(cast_inputs, *args, **kwargs)
    848
    849             except errors.OperatorNotAllowedInGraphError as e:

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    290   def wrapper(*args, **kwargs):
    291     with ag_ctx.ControlStatusCtx(status=ag_ctx.Status.DISABLED):
--> 292       return func(*args, **kwargs)
    293
    294   if inspect.isfunction(func) or inspect.ismethod(func):

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/transformers/modeling_tf_albert.py in call(self, inputs, **kwargs)
    783
    784     def call(self, inputs, **kwargs):
--> 785         outputs = self.albert(inputs, **kwargs)
    786
    787         pooled_output = outputs[1]

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    845                     outputs = base_layer_utils.mark_as_return(outputs, acd)
    846                 else:
--> 847                   outputs = call_fn(cast_inputs, *args, **kwargs)
    848
    849             except errors.OperatorNotAllowedInGraphError as e:

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    290   def wrapper(*args, **kwargs):
    291     with ag_ctx.ControlStatusCtx(status=ag_ctx.Status.DISABLED):
--> 292       return func(*args, **kwargs)
    293
    294   if inspect.isfunction(func) or inspect.ismethod(func):

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/transformers/modeling_tf_albert.py in call(self, inputs, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, training)
    680
    681         embedding_output = self.embeddings(
--> 682             [input_ids, position_ids, token_type_ids, inputs_embeds], training=training)
    683         encoder_outputs = self.encoder(
    684             [embedding_output, extended_attention_mask, head_mask], training=training)

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    889           with base_layer_utils.autocast_context_manager(
    890               self._compute_dtype):
--> 891             outputs = self.call(cast_inputs, *args, **kwargs)
    892           self._handle_activity_regularization(inputs, outputs)
    893           self._set_mask_metadata(inputs, outputs, input_masks)

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in return_outputs_and_add_losses(*args, **kwargs)
     55     inputs = args[inputs_arg_index]
     56     args = args[inputs_arg_index + 1:]
---> 57     outputs, losses = fn(inputs, *args, **kwargs)
     58     layer.add_loss(losses, inputs)
     59     return outputs

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in wrap_with_training_arg(*args, **kwargs)
    109         training,
    110         lambda: replace_training_and_call(True),
--> 111         lambda: replace_training_and_call(False))
    112
    113   # Create arg spec for decorated function. If 'training' is not defined in the

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py in smart_cond(pred, true_fn, false_fn, name)
     57         pred, true_fn=true_fn, false_fn=false_fn, name=name)
     58   return smart_module.smart_cond(
---> 59       pred, true_fn=true_fn, false_fn=false_fn, name=name)
     60
     61

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/smart_cond.py in smart_cond(pred, true_fn, false_fn, name)
     54       return true_fn()
     55     else:
---> 56       return false_fn()
     57   else:
     58     return control_flow_ops.cond(pred, true_fn=true_fn, false_fn=false_fn,

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in <lambda>()
    109         training,
    110         lambda: replace_training_and_call(True),
--> 111         lambda: replace_training_and_call(False))
    112
    113   # Create arg spec for decorated function. If 'training' is not defined in the

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in replace_training_and_call(training)
    104     def replace_training_and_call(training):
    105       set_training_arg(training, training_arg_index, args, kwargs)
--> 106       return wrapped_call(*args, **kwargs)
    107
    108     return tf_utils.smart_cond(

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in __call__(self, *args, **kwargs)
    531     if not self.call_collection.tracing:
    532       self.call_collection.add_trace(*args, **kwargs)
--> 533     return super(LayerCall, self).__call__(*args, **kwargs)
    534
    535   def get_concrete_function(self, *args, **kwargs):

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in __call__(self, *args, **kwds)
    455
    456     tracing_count = self._get_tracing_count()
--> 457     result = self._call(*args, **kwds)
    458     if tracing_count == self._get_tracing_count():
    459       self._call_counter.called_without_tracing()

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _call(self, *args, **kwds)
    492       # In this case we have not created variables on the first call. So we can
    493       # run the first trace but we should fail if variables are created.
--> 494       results = self._stateful_fn(*args, **kwds)
    495       if self._created_variables:
    496         raise ValueError("Creating variables on a non-first call to a function"

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in __call__(self, *args, **kwargs)
   1820   def __call__(self, *args, **kwargs):
   1821     """Calls a graph function specialized to the inputs."""
-> 1822     graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
   1823     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
   1824

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   2148         graph_function = self._function_cache.primary.get(cache_key, None)
   2149         if graph_function is None:
-> 2150           graph_function = self._create_graph_function(args, kwargs)
   2151           self._function_cache.primary[cache_key] = graph_function
   2152         return graph_function, args, kwargs

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2039             arg_names=arg_names,
   2040             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2041             capture_by_value=self._capture_by_value),
   2042         self._function_attributes,
   2043         # Tell the ConcreteFunction to clean up its graph once it goes out of

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    913                                           converted_func)
    914
--> 915       func_outputs = python_func(*func_args, **func_kwargs)
    916
    917       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    356         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    357         # the function a weak reference to itself to avoid a reference cycle.
--> 358         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    359     weak_wrapped_fn = weakref.ref(wrapped_fn)
    360

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in wrapper(*args, **kwargs)
    513         layer, inputs=inputs, build_graph=False, training=training,
    514         saving=True):
--> 515       ret = method(*args, **kwargs)
    516     _restore_layer_losses(original_losses)
    517     return ret

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in wrap_with_training_arg(*args, **kwargs)
    109         training,
    110         lambda: replace_training_and_call(True),
--> 111         lambda: replace_training_and_call(False))
    112
    113   # Create arg spec for decorated function. If 'training' is not defined in the

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py in smart_cond(pred, true_fn, false_fn, name)
     57         pred, true_fn=true_fn, false_fn=false_fn, name=name)
     58   return smart_module.smart_cond(
---> 59       pred, true_fn=true_fn, false_fn=false_fn, name=name)
     60
     61

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/smart_cond.py in smart_cond(pred, true_fn, false_fn, name)
     54       return true_fn()
     55     else:
---> 56       return false_fn()
     57   else:
     58     return control_flow_ops.cond(pred, true_fn=true_fn, false_fn=false_fn,

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in <lambda>()
    109         training,
    110         lambda: replace_training_and_call(True),
--> 111         lambda: replace_training_and_call(False))
    112
    113   # Create arg spec for decorated function. If 'training' is not defined in the

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in replace_training_and_call(training)
    104     def replace_training_and_call(training):
    105       set_training_arg(training, training_arg_index, args, kwargs)
--> 106       return wrapped_call(*args, **kwargs)
    107
    108     return tf_utils.smart_cond(

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in call_and_return_conditional_losses(inputs, *args, **kwargs)
    555   layer_call = _get_layer_call_method(layer)
    556   def call_and_return_conditional_losses(inputs, *args, **kwargs):
--> 557     return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs)
    558   return _create_call_fn_decorator(layer, call_and_return_conditional_losses)
    559

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in get_losses_for(self, inputs)
   1382     losses = [l for l in self.losses if not l._unconditional_loss]
   1383     inputs = nest.flatten(inputs)
-> 1384     reachable = tf_utils.get_reachable_from_inputs(inputs, losses)
   1385     return [l for l in losses if l in reachable]
   1386

/app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py in get_reachable_from_inputs(inputs, targets)
    132       outputs = x.consumers()
    133     else:
--> 134       raise TypeError('Expected Operation, Variable, or Tensor, got ' + str(x))
    135
    136     for y in outputs:

TypeError: Expected Operation, Variable, or Tensor, got None
**Describe the expected behavior**

**Code to reproduce the issue**

`import tensorflow as tf`
`import pandas as pd`

`from sklearn.model_selection import train_test_split`

`import transformers`
`from transformers import AlbertConfig`
`from transformers import AlbertTokenizer`
`from transformers import TFAlbertForSequenceClassification`
`from transformers import glue_convert_examples_to_features`

`data_df = pd.read_excel("../input/test.xlsx")`
`model_dir = '../input/albert_xxlarge_v2/'`
`EPOCHS = 3`
`MAX_SEQ_LENGTH = 256`
`label_list = [0,1]`

`config = AlbertConfig.from_pretrained('albert-xxlarge-v2')`
`tokenizer = AlbertTokenizer.from_pretrained('albert-xxlarge-v2', cache_dir=model_dir)`
`model = TFAlbertForSequenceClassification.from_pretrained('albert-xxlarge-v2', ``cache_dir=model_dir, config=config)`

`train_df, test_df = train_test_split(data_df[['id','text1', 'text2', 'LABEL']],
                                                    random_state=42, shuffle=True,
                                                    test_size=0.20, stratify=data_df['LABEL'])`

`train_InputExamples = train_df.apply(lambda x: InputExample(guid=x['id'],
                                                                           text_a=x['text1'],
                                                                           text_b=x['text2'],
                                                                           label=x['LABEL']), axis=1)`

`train_dataset = glue_convert_examples_to_features(examples=train_InputExamples, tokenizer=tokenizer,
                                                  max_length=MAX_SEQ_LENGTH,
                                                  label_list = label_list, output_mode="classification")`

`optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)`
`loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)`
`metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')`

`input_ids_train = []`
`attention_mask_train = []`
`token_type_ids_train = []`
`output_label_train = []`
`for f in train_dataset:`
    `input_ids_train.append(f.input_ids)`
    `attention_mask_train.append(f.attention_mask)`
    `token_type_ids_train.append(f.token_type_ids)`
    `output_label_train.append(f.label)`
`model.compile(optimizer=optimizer, loss=loss, metrics=[metric])`

`input_ids_train = np.array(input_ids_train)`
`attention_mask_train = np.array(attention_mask_train)`
`token_type_ids_train = np.array(token_type_ids_train)`
`output_label_train = np.array(output_label_train)`

`model.fit([input_ids_train,attention_mask_train, token_type_ids_train], y=output_label_train,
          epochs = EPOCHS, batch_size=4)`

`model.save('../output/my_model')`


XLNet and RoBERTa embeddings

Referring to Jay Alammar's awesome blog post wherein he showed how to create sentence embeddings from BERT (DistilBert as well), can we use the workings he showed here for XLNet and RoBERTa models as well?

http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/

I was thinking majorly to use everything same what he did for RoBERTa assuming `<s>` token in RoBERTa contains the classification output and  change following lines for XLNet since `<cls>` is at the end of the sequence length unlike in case of BERT where it is at the beginning

`features = last_hidden_states[0][:,-1,:].numpy()`

Any idea if my assumptions are correct?


relativeattentionbias.weight in block 0 EncDecAttention of T5 Model not in original tf model. Where do we get it from?

## ❓ Questions & Help

Hi, I was comparing the weights in original tf model and the pytorch t5 model and it looks like there is an extra embedding in the EncDecAttention layer (layer_1) in block_0 (relative_attention_bias.weight). I could find and compare the other embedding weights in the model but not this particular one. Was this new parameter randomly initialized using the initializer and stored as is for pre-trained model or was it fine-tuned somehow and stored?


Add 'keep_accents' flag to basic tokenizer

Hello!

Recently we released our Spanish Bert Model (https://github.com/dccuchile/beto) and we found problems with the tokenization for Spanish.
The problem relates to that the basic tokenizer convert the text to NFD.

For example:

```
text = "[CLS] compañera [SEP]"
tokenized_text = tokenizer.tokenize(text)
tokenized_text
['[CLS]', 'compa', '##ner', '##a', '[SEP]']
```

It changes *ñ* to *n*.

Another:

```
text = "[CLS] acción [SEP]"
tokenized_text = tokenizer.tokenize(text)
tokenized_text
['[CLS]',  'accion' ,'[SEP]']
```

It changes *ó* to *o*.

That behavior is not wanted for our Spanish model so in this PR I'm adding a flag to control that.

Waiting for your comments, thank you!


What does 'output of the embeddings' mean?

Hello,

According to Hugging Face Transformers documentation, (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel)
the transformer's output ```hidden_state``` is defined as the following:

```
hidden_states: (optional, returned when config.output_hidden_states=True)
list of torch.FloatTensor (one for the output of each layer + the output of the embeddings)
of shape (batch_size, sequence_length, hidden_size): Hidden-states of the model
at the output of each layer plus the initial embedding outputs.
```

I am a bit confused by the statement ```list of torch.FloatTensor (one for the output of each layer + the output of the embeddings) ```. Does the ```output of the embeddings``` at the end of the statement refer to 'output of the uppermost output layer'?

Thank you,


Learning Rate is not being updated by the Scheduler

Hello,

Outside of the training function, I set:
```python
# define the hyperparameters for running the train function.
optimizer_ch2 = AdamW(model_ch2.parameters(), lr = lr, correct_bias = True)

scheduler_ch2 = get_linear_schedule_with_warmup(optimizer = optimizer_ch2,
                                                num_warmup_steps = 200,
                                                num_training_steps = 1000,
                                                last_epoch = -1)
```

and here is my train function:
```python
def train_lm_head(model, train_iter, optimizer, scheduler, log_interval, pad_index):

    # turn on a training mode
    model.train()

    # initialize total_loss to 0
    total_loss = 0

    for batch_index, batch in enumerate(train_iter):

        input_ids = [instance for instance in batch.text]

        ## NOTE: Positions embeddings can be automatically created by the GPT2DoubleHeadsModel as (0, 1, ..., N)

        # set the gradient back to 0 (necessary step)
        optimizer.zero_grad()

        # notice here that we are only placing lm_labels
        # as mc_label is unnecessary for language modelling purpose.
        lm_labels = [-1] + input_ids[:(len(input_ids)-1)]
        lm_labels = torch.tensor([lm_labels], dtype=torch.long)

        input_ids = torch.tensor([input_ids], dtype=torch.long)

        output = model(input_ids, lm_labels = lm_labels)
        loss = output[0]
        # 'loss' here is the cross entropy.
        # recall: 'input_ids' is defined above.

        # calculate gradient by backwarding the loss
        # calculate gradient of the loss w.r.t weights
        loss.backward()

        # clips norm of the gradient of an iterable of parameters.
        # The norm is computed over all gradients together, as if they were
        # concatenated into a single vector. Gradients are modified in-place.
        # so basically just normalizes the weights and returns them.
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)

        optimizer.step() # update the weights by following the WarmupLinearSchedule for the lr.

        scheduler.step() # update the learning rate

        # update the with the calculated loss
        total_loss = total_loss + loss

        # python format: 's' for string, 'd' to display decimal integers (10-base), and 'f' for floats.
        # ex: print("Sammy ate {0:.3f} percent of a pizza!".format(75.765367))
        #     >> Sammy ate 75.765 percent of a pizza!
        #     print("Sammy ate {0:f} percent of a {1}!".format(75, "pizza"))
        #     >> Sammy ate 75.000000 percent of a pizza!
        #
        # Below is good enough since we are doing the Stochastic Gradient Descent.
        # (i.e. 1 batch = 1 sample)

        if batch_index % log_interval == 0 and batch_index > 0:
            print('| epoch {:3d} | {:5d}/{:5d} batches | lr {:02.2f} |'.format(
                    epoch, batch_index, len(train_iter), scheduler.get_lr()[0]))

            total_loss = 0
```

and when I iterate the train function above for 5 epoch, I am getting the following output:
```{python}
# ...
| epoch   1 |   138/ 4957 batches | lr 0.00 |
| epoch   1 |   139/ 4957 batches | lr 0.00 |
| epoch   1 |   140/ 4957 batches | lr 0.00 |
| epoch   1 |   141/ 4957 batches | lr 0.00 |
| epoch   1 |   142/ 4957 batches | lr 0.00 |
| epoch   1 |   143/ 4957 batches | lr 0.00 |
| epoch   1 |   144/ 4957 batches | lr 0.00 |
| epoch   1 |   145/ 4957 batches | lr 0.00 |
| epoch   1 |   146/ 4957 batches | lr 0.00 |
| epoch   1 |   147/ 4957 batches | lr 0.00 |
| epoch   1 |   148/ 4957 batches | lr 0.00 |
| epoch   1 |   149/ 4957 batches | lr 0.00 |
| epoch   1 |   150/ 4957 batches | lr 0.00 |
| epoch   1 |   151/ 4957 batches | lr 0.00 |
| epoch   1 |   152/ 4957 batches | lr 0.00 |

#... list goes on
```
I am a bit concerned about this output because the learning rate does not seem to be changing, although I have specified in my train function ```scheduler.step()```, right underneath the ```optimizer.step()```.

What am I doing wrong here?

Thank you,


BERT adapted to time series

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Is there a better way of modifying BERT to take time series as input (i.e. numerical data instead of text) than editing my local library to skip the word embedding? If not, what is the easiest way to do the latter?

Thanks!


refactoring the code

code formatting, following PEP8 convention


Refactoring the code

Making the code formatting appropriate


load_and_cache_examples crashes on windows

## 🐛 Bug

<!-- Important information -->

Model I am using ALBERT:

Language I am using the model on (English):

The problem arise when using:
[examples/run_squad.py] the official example scripts: (run evaluation for offline model)

It crashes in "load_and_cache_examples" for paths with windows format. It's due to split being done using ('/'). For windows, it needs to be ('\\\\')

The tasks I am working on is:
SQUaD

## To Reproduce

Steps to reproduce the behavior:

1. Get one model cached into a local folder by running Evaluation for SQUAD 2.0 using run_squad.py. This will load online model to the system
2. Run evaluation again with model_name_or_path needs to be a local relative path with "..\\" in it.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows
* Python version: Python 3.6
* PyTorch version: torch 1.3.1
* PyTorch Transformers version (or branch): Latest

## Additional context

Not needed


run_generation.py gives TypeError when using xlnet due to empty dict being passed as token

## 🐛 Bug

When I run
```
python run_generation.py  --model_type=xlnet  --model_name_or_path=xlnet-large-cased
```
I get the following error
```
Traceback (most recent call last):
  File "run_generation.py", line 236, in <module>
    main()
  File "run_generation.py", line 214, in main
    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 820, in encode
    **kwargs
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 912, in encode_plus
    first_ids = get_input_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 904, in get_input_ids
    return self.convert_tokens_to_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 751, in convert_tokens_to_ids
    ids.append(self._convert_token_to_id_with_added_voc(token))
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 758, in _convert_token_to_id_with_added_voc
    if token in self.added_tokens_encoder:
TypeError: unhashable type: 'dict'
```

I think the problem lies in the following code in run_generation.py:
```
def prepare_xlnet_input(args, _, tokenizer, prompt_text):
    prompt_text = (args.padding_text if args.padding_text else PADDING_TEXT) + prompt_text
    return prompt_text, {}
```
This returns a tuple of (string, dict). As this gets passed down to _convert_token_to_id_with_added_voc(), it will first try to check whether prompt_text is in self.added_tokens_encoder, and then whether {} is in self.added_tokens_encoder (which gives a TypeError, because you cannot check ```{} in list```.

I'm not yet sure where the empty dict is supposed to be used, so I can't fix it myself. Would be happy to contribute though.

## Important information

Model I am using (Bert, XLNet....): XLNet
Language I am using the model on (English, Chinese....): English
The problem arise when using:
* [x] the official example scripts: run_generation.py
* [ ] my own modified scripts: (give details)
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: NA
* [ ] my own task or dataset: NA

## To Reproduce

Steps to reproduce the behavior:

1. cd examples
2. python run_generation.py  --model_type=xlnet  --model_name_or_path=xlnet-large-cased

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

```
Traceback (most recent call last):
  File "run_generation.py", line 236, in <module>
    main()
  File "run_generation.py", line 214, in main
    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 820, in encode
    **kwargs
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 912, in encode_plus
    first_ids = get_input_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 904, in get_input_ids
    return self.convert_tokens_to_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 751, in convert_tokens_to_ids
    ids.append(self._convert_token_to_id_with_added_voc(token))
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 758, in _convert_token_to_id_with_added_voc
    if token in self.added_tokens_encoder:
TypeError: unhashable type: 'dict'
```

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

Running without an error.

## Environment

* OS: Mac OS Mojave 10.14.4
* Python version: 3.7.3
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): current master (8c67b529f615cc24c46864b8323d2d47a15ccd58)
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information: NA

## Additional context

<!-- Add any other context about the problem here. -->
NA


How to make FP16 quantization on gpt/xl?

How could I fix this error?
`ValueError: Message tensorflow.GraphDef exceeds maximum protobuf size of 2GB: 6234365906`


Typo in serving.py


Where does the pre-trained bert model gets cached in my system by default?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I used model_class.from_pretrained('bert-base-uncased') to download and use the model. The next time when I use this command, it picks up the model from cache. But when I go into the cache, I see several files over 400M with large random names. How do I know which is the bert-base-uncased or distilbert-base-uncased model? Maybe I am looking at the wrong place


I am getting repetitive output when running "python run_generation.py"

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Here is the command I used to run the code:
python  run_generation.py --model_type=gpt2 --model_name_or_path=gpt2 --length 100
Here is the input and output I got:
Model prompt >>> nice to meet you
nice to meet you.

"I'm sorry, but I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet you. I'm not going to be able to meet!

I have tried different inputs but the output is always repeated.


Bert Decoder using is_decoder and encoder_hidden_states

```
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = "[CLS] For an unfamiliar eye, the Porsche Cayenne and the Cayenne Coupe would look similar"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 3

tokenized_text[masked_index] = '[MASK]'
print(tokenized_text)

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
tokens = tokenizer.convert_ids_to_tokens(indexed_tokens)
string = tokenizer.convert_tokens_to_string(tokens)

# # Define sentence A and B indices associated to 1st and 2nd sentences (see paper)
segments_ids = [0 for x in range(len(tokenized_text))]
#
# # Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
#
model = BertForMaskedLM.from_pretrained('bert-base-uncased', is_decoder=True)
model.eval()
#
# # Predict all tokens
with torch.no_grad():
    outputs = model(tokens_tensor, token_type_ids=segments_tensors, tokens=tokenized_text, encoder_hidden_states=tokens_tensor)
    predictions = outputs[0]

print('state_dict',len(model.state_dict()))
predicted_indices = []

# # confirm we were able to predict 'henson'
for i in range(len(tokenized_text)):
    predicted_indices.append(torch.argmax(predictions[0, i]).item())
# predicted_index = torch.argmax(predictions[0, masked_index]).item()

predicted_token = tokenizer.convert_ids_to_tokens(predicted_indices)[0]
print('indexed_tokens', indexed_tokens)
print('predicted_indices', predicted_indices)
predicted_text = tokenizer.decode(predicted_indices)
print(predicted_text)
```

In `modeling_bert` it's mentioned
```
        To behave as an decoder the model needs to be initialized with the
        `is_decoder` argument of the configuration set to `True`; an
        `encoder_hidden_states` is expected as an input to the forward pass.

```

So i did the same in my code but i get 2 error saying

`INFO:transformers.modeling_utils:Weights of BertForMaskedLM not initialized from pretrained model: ['bert.encoder.layer.0.crossattention.self.query.weight`
and
```
  File "/Volumes/Data/transformers-master/transformers/modeling_bert.py", line 679, in forward
    extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
RuntimeError: expected device cpu and dtype Float but got device cpu and dtype Bool

```
Am i missing something or is this the wrong way to configure bert decoder? In General, i'd like to know how encoder-decoder transformer work in BERT


how to do a simple multi-classifier by bert 2.0,training set ,and label set all lines


help: couldn't find such vocabulary files at this path or url

I want to load the Chinese Roberta model  of pre-trained. When I use RobertaModel.from_pretrained() to load pre-trained model, it can't work.

The  Chinese Roberta model is download from https://github.com/brightmart/roberta_zh
I am not sure if it is a problem with the pre-trained model or the transformers framework problem


How can I read my bert model by using transformers?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
when I use pytorch_pretrained_bert, i can read my model like this:

from pytorch_pretrained_bert import BertModel, BertTokenizer
tokenizer = BertTokenizer.from_pretrained(bert_vocab_path)
bert = BertModel.from_pretrained(bert_model_path)

when i use transformers, how can i do this?


Fix beam search when sampling in language generation

I think there is a problem with beam search when setting `do_sample=True`
As it was implemented before, the variable `next_words` in previous line 829 would always contains
word ids < `vocab_size` which forces all `beam_idx` to always be == 0.
This way all words would actually always be appended to the `input_ids` of the first beam.

In the proposed PR, the words are sampled over the scores of size `(batch_size, num_beams * vocab_size)`, which is similar to what is done in greedy decoding.

I tried generating a couple of sequences with the proposed change and it seems to be important that the temperature is set relatively high (~1.5) to avoid repeating words.

Not 100% whether the proposed PR is the best fix. In general, beam search seems to work better when doing greedy decoding.


Delete [dev] behind pip install -e .

I might be wrong here, but I think it should simply be
```bash
$ pip install -e .
```
without the [dev]
When executing
```bash
$ pip install -e .[dev]
```
in my terminal I get the error:
`no matches found: .[dev]`


Add hint to install pytest-xdist

Just a small hint that pytest-xdist should be installed before running the make test step


Is there a uncased gpt2?

## ❓ Questions & Help

Hi, thanks for everything. Quick question: Is there a pre-trained uncased gpt2, like bert-uncased?


Add dropout to WordpieceTokenizer and BPE

We can add dropout not only to model weights but and to a tokenizer. The paper, proposed by Ivan Provilkov (2019, https://arxiv.org/pdf/1910.13267.pdf), describes all benefits from this approach and shows that it's almost always better to use dropout during tokenization. (use only for training, for inference dropout should be equal to 0)

Example:
```
import transformers
tokenizer = transformers.RobertaTokenizer.from_pretrained("roberta-base")
tokenizer.tokenize("Dropout is very important")
# default value is 0
# ['Drop', 'out', 'Ġis', 'Ġvery', 'Ġimportant']
tokenizer.tokenize("Dropout is very important", dropout=0.1)
# ['Drop', 'out', 'Ġis', 'Ġvery', 'Ġimport', 'ant']
```


Correct tokenization for special and added tokens

When a tokenizer is being loaded with `PreTrainedTokenizer._from_pretrained`, it should set `added_tokens` and `all_special_tokens` to `unique_added_tokens_encoder`.
If we don't do it, it will corrupt the tokenization.

Example:
```
import transformers
tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-uncased")
tokenizer.tokenize("[CLS] token should not be splitted.")
# correct output
# ['[CLS]', 'token', 'should', 'not', 'be', 'split', '##ted', '.']
# incorrect output
# ['[', '[UNK]', ']', 'token', 'should', 'not', 'be', 'split', '##ted', '.']
```


Can I use BERT / gpt-2 for text generation

## ❓ Questions & Help
I want to get a list of possible completions and their probabilities.
For example,
For the sentence "I put the glass of the _"
I want to get a vector with word and probabilities from a pre-trained model, such as :
desk = 0.1
table = 0.2
car = 0.05
shirt = 0.001
Is that possible?


revert erroneous fix #2276

I based #2276 on having an error pop up on an older pytorch version, and also on the erroneous (current!) documentation for pytorch.Tensor.scatter():

> `scatter(dim, index, source)` → Tensor
>
>     Out-of-place version of torch.Tensor.scatter_()
>
> `scatter_(dim, index, src)` → Tensor
> ...

The argument was called `source`, inconsistently, in the version I was using, but somewhere along the way it went back to being `src` without the docs changing, which caused this confusion...


Bug: Tokenization of Special Tokens

## 🐛 Bug

The commit https://github.com/huggingface/transformers/commit/deceb001616995199a6a5dca866ffec95c3ebe74 introduces a bug in the tokenization of special tokens when using `from_pretrained` to initialize the tokenizer.

```
from transformers import AutoTokenizer

bert_tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
bert_text = "[CLS] An example for [MASK] change. [SEP]"
bert_tokenizer.tokenize(bert_text)
# Before: ['[CLS]', 'an', 'example', 'for', '[MASK]', 'change', '.', '[SEP]']
# After: ['[', '[UNK]', ']', 'an', 'example', 'for', '[', '[UNK]', ']', 'change', '.', '[', '[UNK]', ']']

roberta_tokenizer = AutoTokenizer.from_pretrained('roberta-base')
roberta_text = "<s> An example for <mask> change. </s>"
roberta_tokenizer.tokenize(roberta_text)
# Before: ['<s>', 'An', 'Ġexample', 'Ġfor', '<mask>', 'change', '.', '</s>']
# After: ['<', 's', '>', 'ĠAn', 'Ġexample', 'Ġfor', 'Ġ<', 'mask', '>', 'Ġchange', '.', 'Ġ</', 's', '>']
````

Fixed by https://github.com/huggingface/transformers/pull/2312.


pytorch_pretrained_bert giving different scores for BertForNextSentencePrediction

## ❓ Questions & Help
from pytorch_transformers.modeling_bert import BertForNextSentencePrediction
from pytorch_transformers import BertTokenizer, BertConfig
import torch

#Load pretrained model from local
config = BertConfig.from_json_file('resources/bert_config.json')
token = BertTokenizer('resources/vocab.txt')
model = BertForNextSentencePrediction.from_pretrained('resources/pytorch_model.bin', config=config)
model.eval()
textA_ids = token.tokenize("How old are you?")
textB_ids = token.tokenize("The Eiffel Tower is in Paris")
text_ids = token.convert_tokens_to_ids(["[CLS]"] + textA_ids + ["[SEP]"] + textB_ids + ["[SEP]"])
segments_ids = [0]\*(len(textA_ids)+2) + [1]\*(len(textB_ids)+1)
text_inputs = torch.tensor([text_ids])
segments_inputs = torch.tensor([segments_ids])
with torch.no_grad():
    outputs = model(text_inputs, token_type_ids=segments_inputs)
print(outputs)

Outputs changed everytime when I ran the code. I followed many ways in other issues to solve the problem, but they didin't work.  And I've used local pretrained model for many other tasks, but never happened this thing before. So I don't think model caused this problem.

Version Information:
torch 1.1.0
pytorch-transformers 1.2.0


What's the exact name of BERT large in results ( GermEval 2014)?

## ❓ Questions & Help
I use BERT large cased model downloaded by run_ner.py script. But I can't get the result in the table below.
<!-- A clear and concise description of the question. -->
![image](https://user-images.githubusercontent.com/13817269/71427541-cb207880-26f4-11ea-9b15-5f6f13b6f31f.png)


Non-Deterministic Behavior in BertTokenizer

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BertTokenizer only

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: Jupyter Notebook

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details) tokenize a string

## To Reproduce
Jupyter Notebook with one cell, with a cloned version of the transformers repo.

```
import sys
sys.path.insert(0, 'transformers')

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', sep_token='[SEP]', do_lower_case=True)

tokenizer.tokenize("[PAD] [SEP] [SEP] [PAD]")
```

with outputs (varying between kernel restarts and runs):
`['[PAD]', '[', 'sep', ']', '[', 'sep', ']', '[PAD]']`
`['[PAD]', '[SEP]', '[SEP]','[PAD]']`
`['[PAD]', '[SEP]', '[SEP]', '[', 'pad', ']']`
## Expected behavior
Expected the output to be
`['[PAD]', '[SEP]', '[SEP]','[PAD]']`
and have deterministic behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

Platform Linux-3.10.0-1062.el7.x86_64-x86_64-with-redhat-7.7-Verona
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0]
PyTorch 1.3.1
Tensorflow 1.15.0

* PyTorch Transformers version (or branch): on master branch of repo
* Using GPU ? Yes

## Additional context

This happens sometimes when the notebook kernel is restarted and the cell is re-run. I haven't observed this happening when running a python script.

<!-- Add any other context about the problem here. -->


[CLS] token /  is used as the aggregate sequence representation for classification tasks

[CLS]  is  fed into an output layer for classification. How will be built this token. Is there something special done for it during training? in which way does happen this aggregations of sequences?

Thanks for response


Why are you getting just the last encoder states in the summarization code?

The line is here:

https://github.com/huggingface/transformers/blob/v2.3.0/examples/summarization/modeling_bertabs.py#L142

By changing the line to `encoder_hidden_states = encoder_output` I was able to fine-tune the model successfully, as well as, run the inference code from the `run_summarization.py` script.

So just wondering why you're indexing into the the encoder output rather than passing all of it along to the decoder?


fix repetition penalty error in modeling_utils.py

fix bug mention in #2302


Repetition penalty work falsely in case the logit of the token is negativ

## 🐛 Bug

<!-- Important information -->

Model I am using (LMHeadModels; distilgpt2 in this example but holds true for all LMHeadModels):

Language I am using the model on English:

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: language genaration

## To Reproduce

Run the following code:

```
input_sentence = 'The dog'
tokenizer = AutoTokenizer.from_pretrained('distilgpt2')
model = AutoModelWithLMHead.from_pretrained('distilgpt2')
input_ids = torch.tensor(tokenizer.encode(input_sentence)).unsqueeze(0)
outputs = model.generate(input_ids=input_ids, do_sample=True, bos_token_id=tokenizer.bos_token_id, eos_token_ids=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id, repetition_penalty=1.5)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Expected behavior
Output:
`"The dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog"`

In the output, the word dog is repeated multiple times. It can be noticed that the higher the `repetition_penalty`, the more likely already occurring words are to be repeated. Thus, the penalty achieves exactly the opposite of what it is supposed to do.

## Environment

* OS: Linux
* Python version: 3.6.8
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): main branch v.2.3.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

The reason for this behavior can be understood when looking at line https://github.com/huggingface/transformers/blob/81db12c3ba0c2067f43c4a63edf5e45f54161042/src/transformers/modeling_utils.py#L731 :

If the logit `next_token_logits[i, previous_tokens]` is < 0, then dividing by a number > 1 is actually going to increase the probability of sampling that token the next time instead of reducing it.


Can I use run_lm_finetuning.py for training models in an uncovered language?

Is it possibile to use run_lm_finetuning.py script to train one of the models from scratch in a language not covered by the available pretrained models? (like spanish, italian, german).

My idea is to replicate something like camemBERT for a language different from french, given that I have the corpora needed for the training.

What are some suggestions that you could give me? What are the changes to make in the script in order to run it correctly for this purpose? How can I deal with a corpus of ~150GB?

Thanks for any help


run_ner.py RobertaForTokenClassification.from_pretrained "size mismatch for classifier.bias"

## ❓ Questions & Help
I have a trouble on run_ner.py([https://github.com/huggingface/transformers/blob/master/examples/run_ner.py](url)) evaluation using **Roberta**. My error is in this snippet:

```python
    # Load pretrained model and tokenizer
    if args.local_rank not in [-1, 0]:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

    args.model_type = args.model_type.lower()
    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
    config = RobertaConfig.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
                                          num_labels=num_labels,
                                          cache_dir=args.cache_dir if args.cache_dir else None)
    tokenizer = RobertaTokenizer.from_pretrained(args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
                                                do_lower_case=args.do_lower_case,
                                                cache_dir=args.cache_dir if args.cache_dir else None)
    model = RobertaForTokenClassification.from_pretrained(args.model_name_or_path,
                                        from_tf=bool(".ckpt" in args.model_name_or_path),
                                        config=config,
                                        cache_dir=args.cache_dir if args.cache_dir else None)
```

If I run with both training and evaluation, it works fine.
If I want only to evaluate my model I get this error:
```
Traceback (most recent call last):
  File "run_pos.py", line 560, in <module>
    main()
  File "run_pos.py", line 477, in main
    cache_dir=args.cache_dir if args.cache_dir else None)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 479, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RobertaForTokenClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([19, 768]) from checkpoint, the shape in current model is torch.Size([18, 768]).
        size mismatch for classifier.bias: copying a param with shape torch.Size([19]) from checkpoint, the shape in current model is torch.Size([18]).
```

I used a different set of labels (18 labels rather than CoNLL format) passing them through the flag `--labels /path/to/labels.txt`. As you can see, when it loads the model, it sees 19 labels and not the expected 18. I think the 19th is added during the training to tag the subwords. In particular it should be:

```python
# Use cross entropy ignore index as padding label id so that only real label ids contribute to the loss later
    pad_token_label_id = CrossEntropyLoss().ignore_index
```

I don't know if I have to remove a label from its mapping (how to do it?) or if there are other solutions. I also don't know why this error doesn't occur if I train and evaluate sequentially in the same process.

Thank you!


Model2Model inference

## ❓ Questions & Help
I am trying to implement a simple Model2Model question-answer task, where the input is question and the answer needs to be generated. At inference time I feed in the "[CLS]" token to let it generate but it only generates a single token instead of the entire sentence. The ppl at which I do the inference is ~5 ppl on valid set.

Is there a fundamental issue with my model?

Training time
`outputs = model(input_ids,
                        batch['speakableAnswer'],
                        decoder_lm_labels=batch['speakableAnswer'])`
print(outputs.size() -> batch x seq_len x vocab_size)

Test time
`outputs = model(input_ids,
                        batch['speakableAnswer'])`
print(outputs.size() -> batch x 1 x vocab_size)
<!-- A clear and concise description of the question. -->


Why cosine similarity of BERT, ALBERT, Robert is so big, almost near 1.0?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I tried to use bert models to do similarity comparison of words/sentences, but I found that the cosine similarities are all very high, even for very different words/sentences in meaning. Why?

Does all the vector are located in a small portion the vector-space?


RunTimeError in "run_summarization": expected device cuda:0 and dtype byte but got device cuda: 0 and dtype Bool

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
bertabs-finetuned-cnndm-extractive-abstractive-summarization-pytorch_model

Language I am using the model on (, Chinese....):
English

The problem arise when using:
* [ ] the official example scripts: (give details)  run_summarization.py in examples

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: summarization

## To Reproduce

Steps to reproduce the behavior:

1.  python run_summarization.py --documents_dir .\data --sumaries_output_dir .\output

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Win10
* Python version: 3.6.1
* PyTorch version:1.2.0
* PyTorch Transformers version (or branch): master at 24 Dec
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

errors come from modeling_berabs.py, line 328, in forward


A question about BERT position embedding.

## ❓ Questions & Help

I noticed that in Transformers, it turn position_ids into position_embedding through `nn.Embedding(config.max_position_embeddings, config.hidden_size)`

Here `config.max_position_embeddings` is 512 ,and `config.hidden_size` is 768.

So, when I input a sentence shorter than 512 ,such as "Today is a nice day", will this sentence's position embedding still be 512?

Or just as long as position_ids? Here is 7 with [CLS] and [SEP].


How do you handle large documents?

## ❓ Questions & Help

I have been a huge fan of this library for a while now. I've used it to accomplish things like sentence classification, a chat bot, and even stock market price prediction, this is truly a fantastic library. But I have not yet learned how to tackle large documents (e.g. documents 10x the size of the model's max length).

An example. A task I would love to accomplish is document abstraction, however the documents I am dealing with are upwards of 3,000+ words long and I'm afraid that taking the first 512 or 768 tokens will not yield a quality summary.

One idea that I was kicking around, but have not put code to yet, involved taking a window of 512-tokens to produce a model output and then repeating this process, shifting the window of 512-tokens, until I have covered my entire corpus. Then I will repeat the process until I have an input that can fit into my model.

There must be a better way. I have heard of developers using these NLP models to summarize large legal documents and legislation, which can be hundreds of pages, let alone thousands of words. Am I missing something, am I overthinking this problem?


Is there any efficient way to convert BERT outputs to fit token-level tasks?

Say I have a sentence consisting of two words: S = [“Definitely”, “not”], and what I want is to transfer S into an embedding matrix T with a size of (2, 100), where each row represents a word.

I want to adopt BERT embeddings. But in BERT, each word is represented as a sub-word unit. This means that S will be represented as [“Def”, “##in”, “##ite”, “##ly”, “not”] ( “Definitely” is tokenized as “Def”, “##in”, “##ite”, “##ly”). BERT will output an embedding matrix H with a size of (5, 100) :(.

My goal is to merge some rows of H according to the sub-word units.
For example, for “Definitely”, I should merge the embeddings of [“Def”, “##in”, “##ite”, “##ly”] to get its presentation.

In my current method, I use a head mask vector h = [1, 0, 0, 0, 1] to record the “head” of each word, where 1 indicates the head position:
h = [
1, -> “Def”
0, -> “##in”
0, -> “##ite”
0, -> “##ly”
1 -> “not”
]
So I should merge rows which have a head mask of 0 to that having a head mask of 1. I have to use the `for` computation to enumerate each element in h, which is slow and can not batchfy.

Is there any efficient method to do the above computation?


Train custom NER model with new Pipeline

Internals of the library; Pipeline.

## 🚀 Feature

New `Pipelines` feature is great! I am wondering whether it will be possible to implement a pre-training on domain specific data (similar to ULMFiT approach, unsupervised encoder-decoder) and then train a custom NER model with annotated data (similar to spaCy)?


Add cached past for language generation

add past input for gpt2 and ctrl for faster decoding for language generation.

1. add `prepare_inputs_for_generation` fn for gpt2 and ctrl
2. add private `_do_output_past` fn for PretrainModel class to check whether model outputs past key-value states
      - fn only covers cases for gpt2 and ctrl for the moment and needs to add 'xlnet' and 'transfo_xl' via `mem_len`
      - might be better to move `_do_output_past` to each individual LMHeadModel
3. rename `pasts` to `past`

can also add dummy tests for language generation


Fix F841 flake8 warning

This PR completes the "fix all flake8 warnings" effort of the last few days.

There's a lot of judgment in the fixes here: when the result of an expression is assigned to a variable that isn't used:

- if the expression has no side effect, then it can safely be removed
- if the expression has side effects, then it must be kept and only the assignment to a variable must be removed
- or it may be a coding / refactoring mistake that results in a badly named variable

I'm not sure I made the right call in all cases, so I would appreciate a review.

E203, E501, W503 are still ignored because they're debatable, black disagrees with flake8, and black wins (by not being configurable).


duplicated line for repeating_words_penalty_for_language_generation

length_penalty has a duplicated wrong documentation for language generation -> delete two lines


fix bug in prepare inputs for language generation for xlm for effective batch_size > 1

if multiple sentence are to be generated the masked tokens to be appended have to equal the effective batch size


Improve handling of optional imports


Do Hugging Face GPT-2 Transformer Models Automatically Does the Absolute Position Embedding for Users?

Hello,

According to Hugging Face ```GPT2DoubleHeadsModel``` documentation (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel)

```
"Indices of input sequence tokens in the vocabulary.
GPT-2 is a model with absolute position embeddings"
```

So does this mean that, when we implement any Hugging Face GPT-2 Models (```GPT2DoubleHeadsModel```,```GPT2LMHeadsModel```, etc.) via the ```model( )``` statement, the 'absolute position embedding' is _automatically_ done for the user, so that the user actually does not need to specify anything in the ```model( )``` statement to ensure the absolute position embedding?

If the answer is 'yes', then why do we have an option of specifying ```position_ids``` in the ```model( )``` statement?

Thank you,


Typo in tokenization_utils.py

avoir -> avoid


BertTokenizer custom UNK unexpected behavior

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: (give details)
Importing transformers to my own project but using BertTokenizer and BertModel with pretrained weights, using 'bert-base-multilingual-cased' for both tokenizer and model.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)
Fine tuning BERT for English NER on a custom news dataset.

## To Reproduce

Steps to reproduce the behavior:

1. Initialize a tokenizer with custom UNK
2. Try to convert the custom UNK to ID
3. Receive None as

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
>>>tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False, pad_token="<pad>", unk_token="<unk>")
>>> tokenizer.tokenize("<unk>")
['<unk>']
>>> tokenizer.convert_tokens_to_ids(["<unk>"])
[None]
```

## Expected behavior

My custom UNK should have an ID. (instead, I get None)

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Mac OSX
* Python version: 3.7.3
* PyTorch version: 1.1.0.post2
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


[ALBERT]: Albert base model itself consuming 32 GB GPU memory..

## 🐛 Bug

<!-- Important information -->

Model I am using TFALBERT:

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts:

`from transformers import TFAlbertForSequenceClassification`
`model = TFAlbertForSequenceClassification.from_pretrained('albert-base-v2')`

After this GPU memory is consumed almost 32 GB.... base V2 is model is roughly around 50 MB which is occupying 32 GB on GPU

![image](https://user-images.githubusercontent.com/14140685/71364084-64d41280-259b-11ea-9191-094c6e1cd2c3.png)

## Environment

* OS: Linux
* Python version: 3.7


Loading sciBERT failed

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to compare the effect of different pre-trained models on RE, the code to load bert is:
`self.bert = BertModel.from_pretrained(pretrain_path)`

When the "pretrain_path" is "pretrain/bert-base-uncased" , everything is fine, but after i changed it to "pretrain/scibert-uncased", i got error:

`-OSError: Model name 'pretrain/scibert-uncased' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed 'pretrain/scibert-uncased/config.json' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.`

The scibert is pytorch model and the two directories are same in structure.
It seems that if the model name is not in the name list, it won't work.

Thank you very much!


Maybe some parameters are error in document for distributed training ?

Based on [Distributed training document](https://huggingface.co/transformers/examples.html#id1) , one can use `bert-base-cased` model to fine-tune MR model and reaches very high score.

> Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.0:

```
python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-cased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_train_batch_size 24 \
    --gradient_accumulation_steps 12
```

```
f1 = 93.15
exact_match = 86.91
```

**But based on [google bert repo](https://github.com/google-research/bert#squad-11) , the model `bert-base-cased` perfermance is**
```
{"f1": 88.41249612335034, "exact_match": 81.2488174077578}
```

Maybe the right pretrained model is `bert-large-uncased` ?
Thanks~


Add Dutch pre-trained BERT model

We trained a Dutch cased BERT model at the University of Groningen.

Details are on [Github](https://github.com/wietsedv/bertje/) and [Arxiv](https://arxiv.org/abs/1912.09582).


Do anyone have solution for this

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


Help with finetune BERT pretraining

Hi
could you please assist me how I can pretrain the BERT model, so not like SNLI/MNLI when we finetune a pretrained model, but doing pretraining objective.
thanks a lot
and Merry Christmas and happy new year in advance to the team


where is the script of a second step of  knwoledge distillation on SQuAD 1.0?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
In Distil part, there is a paragraph description which is "distilbert-base-uncased-distilled-squad: A finetuned version of distilbert-base-uncased finetuned using (a second step of) knwoledge distillation on SQuAD 1.0. This model reaches a F1 score of 86.9 on the dev set (for comparison, Bert bert-base-uncased version reaches a 88.5 F1 score)."
so where is the script of "a second step of  knwoledge distillation on SQuAD 1.0" mentioned above?
Thanks a lot, it will be very helpful to me!


Does the calling order need to be changed?

## ❓ Questions & Help
pytorch: 1.3.0

torch/optim/lr_scheduler.py:100: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)


fix error due to wrong argument name to Tensor.scatter()

The named argument is called "source", not "src" in the out of place version for some reason, despite it being called "src" in the in-place version of the same Pytorch function. This causes an error.


Gpt2/xl Broken on "Write With Transformer" site

If you navigate to the [GPT-2 section of the Write With Transformer site](https://transformer.huggingface.co/doc/gpt2-large), select gpt2/xl, and try to generate text, the process will not generate anything.


## 🐛 Bug

<!-- Important information -->

The example script `run_generation.py` is broken with the error message `AttributeError: 'GPT2LMHeadModel' object has no attribute 'generate'`

## To Reproduce

Steps to reproduce the behavior:

1. In a terminal, cd to `transformers/examples` and then `python run_generation.py --model_type=gpt2 --model_name_or_path=gpt2`
2. After the model binary is downloaded to cache, enter anything when prompted "`Model prompt >>>`"
3. And then you will see the error:

```
Traceback (most recent call last):
  File "run_generation.py", line 236, in <module>
    main()
  File "run_generation.py", line 216, in main
    output_sequences = model.generate(
  File "C:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'GPT2LMHeadModel' object has no attribute 'generate'
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows 10
* Python version: 3.7.3
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.2
* Using GPU ? N/A
* Distributed of parallel setup ? N/A
* Any other relevant information:

I'm running the latest version of `run_generation.py`. Here is the permanent link: https://github.com/huggingface/transformers/blob/ce50305e5b8c8748b81b0c8f5539a337b6a995b9/examples/run_generation.py

## Additional context

<!-- Add any other context about the problem here. -->


adding special tokens after truncating in run_lm_finetuning.py

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details)

In `run_lm_finetuning.py`, we have the following
```
for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
    self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size]))
```

If we add special tokens after truncating to `block_size`, the example is no longer of length `block_size`, but longer.

In the help information for `block_size` as an input argument, it says "Optional input sequence length after tokenization. The training dataset will be truncated in block of this size for training. Default to the model max input length for single sentence inputs (**take into account special tokens**)." This may be confusing, because the `block_size` written as the default input is 512, but if you use BERT-base as the model you're pretraining from, the `block_size` input in that function is actually 510.

The [original BERT code](https://github.com/google-research/bert/blob/master/extract_features.py) makes sure all examples are `block_size` after adding special tokens:
```
if tokens_b:
      # Modifies `tokens_a` and `tokens_b` in place so that the total
      # length is less than the specified length.
      # Account for [CLS], [SEP], [SEP] with "- 3"
      _truncate_seq_pair(tokens_a, tokens_b, seq_length - 3)
else:
      # Account for [CLS] and [SEP] with "- 2"
      if len(tokens_a) > seq_length - 2:
        tokens_a = tokens_a[0:(seq_length - 2)]
```


Run_tf_ner.py error on TPU

## 🐛 Bug
run_tf_ner.py does not work with TPU. The first error is:
`File system scheme '[local]' not implemented `
When the script is changed and .tfrecord file is moved to gs:// address (and also hardcoded "/tmp/mylogs" is replaced with gs:/// dir)  there is an error with optimiser:
`AttributeError: 'device_map' not accessible within a TPU context.`

## To Reproduce

Steps to reproduce the behaviour:

python run_tf_ner.py.1 --tpu grpc://10.240.1.2:8470 --data_dir gs://nomentech/datadir --labels ./datasets/labels.txt --output_dir gs://nomentech/model1 --max_seq_length 40 --model_type bert --model_name_or_path bert-base-multilingual-cased --do_train --do_eval --cache_dir gs://nomentech/cachedir --num_train_epochs 5 --per_device_train_batch_size 96

## Environment

* OS: Ubuntu 18
* Python version: 3.7
* Tensorflow version: 2.1.0-dev20191222 (tf-nightly)
* PyTorch Transformers version (or branch): 2.3.0
* Distributed of parallel setup ? TPU


Improve setup and requirements

- Clean up several requirements files generated with pip freeze, with no clear update process
- Rely on extra_requires for managing optional requirements
- Update contribution instructions


Remove support for Python 2


Bad F1 Score for run_squad.py on SQuAD2.0

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When I run the run_squad.py for SQuAD2.0 like this

python3 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --train_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/train-v2.0.json \
  --predict_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/dev-v2.0.json \
  --per_gpu_train_batch_size 4 \
  --learning_rate 4e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /share/nas165/Wendy/transformers/examples/SQuAD2.0_debug_bert/
  --version_2_with_negative=True \
  --null_score_diff_threshold=-1.967471694946289

It runs very fast and the f1 score only 7.9% What is wrong with it
The log message like this
<img width="960" alt="擷取" src="https://user-images.githubusercontent.com/32416416/71322341-939daa80-2501-11ea-9313-e179d1760b99.PNG">
Thanks a lot for your help.By the way I clone it today so it is the new version


Improve repository structure

This PR builds on top of #2255 (which should be merged first).

Since it changes the location of the source code, once it's merged, contributors must update their local development environment with:

        $ pip uninstall transformers
        $ pip install -e .

I'll clarify this when I update the contributor documentation (later).

I checked that:

- `python setup.py sdist` packages the right files (only from `src`)
- I didn't lose any tests — the baseline for `run_tests_py3_torch_and_tf` is `691 passed, 68 skipped, 50 warnings`, see [here](https://app.circleci.com/jobs/github/huggingface/transformers/10684))


Does Pre-Trained Weights Work Internally in pytorch?

I am using bert’s pretrained model in from_pretrained and coming across it’s fine tuning code we can save the new model weights and other hyper params in save_pretrained.
My doubt is in there modeling_bert code there is no explicit code that takes the pre-trained weights in acount and then trains as it generally takes attention matrices and puts it in a feed forward network in the class `BertPredictionHeadTransform`
```
class BertPredictionHeadTransform(nn.Module):
    def __init__(self, config):
        super(BertPredictionHeadTransform, self).__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        if isinstance(config.hidden_act, str) or (sys.version_info[0] == 2 and isinstance(config.hidden_act, unicode)):
            self.transform_act_fn = ACT2FN[config.hidden_act]
        else:
            self.transform_act_fn = config.hidden_act
        self.LayerNorm = BertLayerNorm(config.hidden_size, eps=config.layer_norm_eps)

    def forward(self, hidden_states):
        hidden_states = self.dense(hidden_states)
        hidden_states = self.transform_act_fn(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)
        print('BertPredictionHeadTransform', hidden_states.shape)
        return hidden_states

```
And here i do not see any kind of “inheritance” of the pre-trained weights…
So is it internally handled by pytorch or am i missing something in the code itself?


Imports likely broken in examples

## 🐛 Bug

While cleaning up imports with isort, I classified them all. I failed to identify the following four imports:

1. model_bertabs
2. utils_squad
3. utils_squad_evaluate
4. models.model_builder

These modules aren't available on PyPI or in the transformers code repository.

I think they will result in ImportError (I didn't check).

I suspect they used to be in transformers, but they were renamed or removed.


Only the Bert model is currently supported

## 🐛 Bug

<!-- Important information -->

Model I am using Model2Model and BertTokenizer:

Language I am using the model on English:

The problem arise when using:
*  the official example scripts: (give details)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
# Let's re-use the previous question
question = "Who was Jim Henson?"
encoded_question = tokenizer.encode(question)
question_tensor = torch.tensor([encoded_question])

# This time we try to generate the answer, so we start with an empty sequence
answer = "[CLS]"
encoded_answer = tokenizer.encode(answer, add_special_tokens=False)
answer_tensor = torch.tensor([encoded_answer])

# Load pre-trained model (weights)
model = Model2Model.from_pretrained('fine-tuned-weights')
model.eval()

# If you have a GPU, put everything on cuda
question_tensor = encoded_question.to('cuda')
answer_tensor = encoded_answer.to('cuda')
model.to('cuda')

# Predict all tokens
with torch.no_grad():
    outputs = model(question_tensor, answer_tensor)
    predictions = outputs[0]

# confirm we were able to predict 'jim'
predicted_index = torch.argmax(predictions[0, -1]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Colab
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:


Fix doc link in README

close https://github.com/huggingface/transformers/issues/2252

- [x] Update `.circleci/deploy.sh`
- [x] Update `deploy_multi_version_doc.sh`

Set commit hash before "Release: v2.3.0".


BertModel sometimes produces the same output during evaluation

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I finetune BertModel as a part of my model to produce word embeddings. I found sometimes the performance was very bad, then i run the code again without any change, the performance was normal. It is very strange. I check my code to try to find the bug. I found the word embeddings produced by BertModel were all the same. Then I followed the code of BertModel, and found the BertEncoder would make the output become similar gradually which was consisted of 12 BertLayers. I have no idea about this situation.


How to do_predict on run_glue?

## ❓ Questions & Help
I have fine-tuned BERT for sequence classification task by running run_glue script. Now I have trained and evaluated model. My question is how do I make prediction with it on new instances (test set)? Thanks!