glue.py when using mrpc and similar data does not work

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): ENGLISH

The problem arise when using:
* [ ] the official example scripts: (give details):
I am using my dataset with format [idx, sentence1, sentence2, label] in form of dict of tf.data.Dataset and using glue_convert_examples_to_features() to convert my dataset to features

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
I am using MRPC task

## To Reproduce

Steps to reproduce the behavior:
0- Using BertTokenizer.from_pretrained('bert-base-cased') to create tokenizer
1-read csv file as dataframe
2-convert dataframe to dict of tensors using tf.data.Dataset.from_tensor_slices(dict(train))
the element_spec of my data is :

{'idx': TensorSpec(shape=(), dtype=tf.string, name=None),
 'sentence1': TensorSpec(shape=(), dtype=tf.string, name=None),
'sentence2': TensorSpec(shape=(), dtype=tf.string, name=None),
'label': TensorSpec(shape=(), dtype=tf.int32, name=None)}

3- when using glue_convert_examples_to_features(train_data, tokenizer, label_list=[1,0], max_length=128 , task='mrpc'), I will have an error in glue.py in glue_convert_examples_to_features : label = label_map[example.label]

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version: 3.7
* PyTorch version: TF 2.1.0
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed or parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

I think you can't access the dictionary by example.label. When I copy paste it to my code, it will actually work with example["label"].


Can not upload BertTokenizer.from_pretrained() from an AWS S3 bucket

## 📚 Migration

the model I am using is BertForSequenceClassification

The problem arises when I serialize my Bert model, and then upload to an AWS S3 bucket. Once my model is inside of S3, I can not import the model via BertTokenizer.from_pretrained()

For example, in order to save my model to S3, my code reads,

```
byte_obj = pickle.dumps(model)
s3_resource = boto3.resource('s3')
s3_resource.Object("s3 name", "bertClassifier").put(Body=byte_obj)
```

This saves the BertForSequenceClassification model. I can not use the  `model.save_pretrained("s3 name")`, as I get an error from AWS. I beleive in order to transfer files in AWS, one needs to first pickle the file.

When I want to re upload the model, I can not use

`BertTokenizer.from_pretrained("s3 name")`
or
`BertForSequenceClassification.from_pretrained("s3 name")`

because the object I am trying to load is serialized via pickle module. Instead I upload the file this way.

```
session = boto3.session.Session()
s3client = session.client('s3')
response = s3client.get_object(Bucket='s3 name', Key='bertClassifier')

body_string = response['Body'].read()
bert_nn = pickle.loads(body_string)
```

This succesfully loads the BertForSequenceClassification model, but I have no way of loading BertTokenizer from this same pre trained model. Again, becuase I am not able to upload via the BertTokenizer.from_pretrained("s3 name") function.

Is there a work around for this?


XLNet: Incorrect segment id for CLS token

Thanks for the great work!

It seems that the XLNetTokenizer assigns an incorrect segment id to the CLS token when a single sequence of token ids is provided. If token_ids_1 is None, all segment ids are '0', including the segment id of the CLS token. In my understanding, the segment ids should always differ.

```python
if token_ids_1 is None:
    return len(token_ids_0 + sep + cls) * [0]
return len(token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1] + cls_segment_id
```
https://github.com/huggingface/transformers/blob/983c484fa2fcad307d37cb81f3e1125aa7b9dc37/src/transformers/tokenization_xlnet.py#L243

The original implementation assigns a different segment id to the CLS token in both cases (single sequence of tokens and pair of sequences): https://github.com/zihangdai/xlnet/blob/bbaa3a6fa0b3a2ee694e8cf66167434f9eca9660/classifier_utils.py#L109


Edit a way to get `projected_context_layer`

I edited a way to get `projected_context_layer`.

Instead of doing
'''
w = (
    self.dense.weight.t()
    .view(self.num_attention_heads, self.attention_head_size, self.hidden_size)
    .to(context_layer.dtype)
)
b = self.dense.bias.to(context_layer.dtype)
projected_context_layer = torch.einsum("bfnd,ndh->bfh", context_layer, w) + b
'''

I added `self.merge_last_ndims` at `AlbertAttention`.
'''
def merge_last_ndims(self, x, n_dims):
    s = x.size()
    assert n_dims > 1 and n_dims < len(s)
    return x.view(*s[:-n_dims], -1)
'''

I commited this one yesterday, but that one didn't pass the tests, so I re-wrote my code and re-committing now.


unexpected keyword argument 'encoder_hidden_states' when using PreTrainedEncoderDecoder

## ❓ Questions & Help

After I have defined my seq2seq class using Encoder Decoder Architecture in the following way:
```
from transformers import PreTrainedEncoderDecoder
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased','gpt2')
```
I try to forward tensor through the model in this following way:
```
model(question_batch, answer_batch)
```
 but I got this error:

```
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-16-bb34e9576e8b> in <module>()
----> 1 model(test_history, test_knowledge)

2 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
```

can anyone help me?


Trouble fine tuning multiple choice

## ❓ Questions & Help

Hi! I have issues with fine-tuning the multi-choice BERT and I am stuck on an error and I can use some help. When I tried to fine-tune it with my own dataset, it threw the Error  ```RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /opt/conda/conda-bld/pytorch_1570711300255/work/aten/src/THNN/generic/ClassNLLCriterion.c:97````

According to what I have found, this normally happened due to the dimension mismatch between the label and the output layer. When I printed the model, it seems that the model does not have a suitable output layer.

```
` (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=1, bias=True)
)`
```
I have made sure that the Bert Config has received the correct number of the label (`num_labels`)

```
{
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 3,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 30522
}
```

I am using the code from examples/run_multiple_choice.py and examples/utils_multiple_choice.py without touching its logic but I have modified it the data processor.

```
class CSProcessor(object):

    def get_train_examples(self):
        return self._create_examples(df_all[:8000] ,
                                     df_answer[:8000], "train")

    def get_test_examples(self):
        """See base class."""
        return self._create_examples(df_sentence[8000:] ,
                                     df_answer[8000:], "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1", "2"]

    def _create_examples(self, df_sentence, df_answer, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for I in range(len(df_sentence)):
            race_id = "%s-%s" % (set_type, I)

            truth = str(ord(df_answer[1][I]) - ord("A"))
            illogicalAnswer = df_sentence["FalseSent"][I]
            examples.append(
                InputExample(
                example_id=race_id,
                question="Why it doesn't make senses?",
                contexts=[illogicalAnswer],
                endings=[df_sentence['OptionA'][I],
                         df_sentence['OptionB'][I],
                         df_sentence['OptionC'][I]],
                label=truth,
                )
            )
        return examples
```

I appreciate any help :)


Xlnet, Alberta, Roberta are not finetuned for CoLA task

## 🐛 Bug

I am currently trying to finetune pretrained models on CoLA task by using run_glue.py. Some of the models such as Bert and DistilBert are finetuned correctly as it is expected (the training loss goes down and the evaluation result is as what has been reported). Even though, for other models such as Roberta, Albert and Xlnet the training loss remains the same. I finetune the models for CoLA accordingly:

python run_glue.py --data_dir=./glue_data/CoLA/  --model_type=roberta
--model_name_or_path=roberta-base --task_name=CoLA
--output_dir=./model_roberta/  --max_seq_len=128 --do_train  --do_eval --num_train_epochs=3.0 --save_steps=50  --learning_rate=5e-5

## Observed behavior
(for Roberta-base)
"loss": 0.5889998215436936, "step": 50
"loss": 0.649243945479393, "step": 100
"loss": 0.6612952649593353, "step": 150
"loss": 0.6241107112169266, "step": 200
...
"loss":  0.6236384356021881, "step": 50
...
"loss": 0.6253059101104737, "step": 800
...
(I trained for more epochs (different lr )and still, the loss is near 0.5-0.6)
By debugging the code it seems that the model's output (softmax of logits) during training no matter what is the input is always label 1.

Another hint: I tried to finetune Roberta for other tasks such as STS-B and it finetuned well (loss was going down and the output of the model was not identical for all different inputs).

I was wondering if someone also has faced the same issue. How should I solve this?

OS type and version: Linux-3.10.0
Python: 3.7
Pytorch: 1.3.1


load  tf2 roberta model meet error

## ❓ Questions & Help
`
config = RobertaConfig() # print(config) to see settings
config.output_hidden_states = False # Set to True to obtain hidden states
model = TFRobertaModel.from_pretrained('/home/wk/Bert_Pretrained/robert_base/roberta-base-tf_model.h5', config=config)   `
errors
`ValueError                                Traceback (most recent call last)
<ipython-input-19-eac9e3228d6c> in <module>
      1 config = RobertaConfig() # print(config) to see settings
      2 config.output_hidden_states = False # Set to True to obtain hidden states
----> 3 model = TFRobertaModel.from_pretrained('/home/wk/Bert_Pretrained/robert_base/roberta-base-tf_model.h5', config=config)

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    315         # see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1339-L1357
    316         try:
--> 317             model.load_weights(resolved_archive_file, by_name=True)
    318         except OSError:
    319             raise OSError("Unable to load weights from h5 file. "

~/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in load_weights(self, filepath, by_name)
    179         raise ValueError('Load weights is not yet supported with TPUStrategy '
    180                          'with steps_per_run greater than 1.')
--> 181     return super(Model, self).load_weights(filepath, by_name)
    182
    183   @trackable.no_automatic_dependency_tracking

~/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py in load_weights(self, filepath, by_name)
   1173         f = f['model_weights']
   1174       if by_name:
-> 1175         saving.load_weights_from_hdf5_group_by_name(f, self.layers)
   1176       else:
   1177         saving.load_weights_from_hdf5_group(f, self.layers)

~/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py in load_weights_from_hdf5_group_by_name(f, layers)
    758                                symbolic_weights[i])) +
    759                            ', but the saved weight has shape ' +
--> 760                            str(weight_values[i].shape) + '.')
    761
    762         else:

ValueError: Layer #0 (named "roberta"), weight <tf.Variable 'tf_roberta_model_5/roberta/embeddings/word_embeddings/weight:0' shape=(30522, 768) dtype=float32, numpy=
array([[-0.02175204,  0.01785859, -0.01712652, ...,  0.0088525 ,
        -0.00240036,  0.01757819],
       [ 0.01320856,  0.01548896,  0.0290868 , ..., -0.01266216,
         0.00756532, -0.01283411],
       [ 0.02433892,  0.00970818, -0.01082115, ...,  0.01121136,
         0.01314066,  0.0088822 ],
       ...,
       [-0.00798688, -0.03137787, -0.00074065, ...,  0.03188593,
         0.02637535,  0.02540809],[ 0.01545427, -0.02784344,  0.01380141, ..., -0.02135191,
        -0.01506698, -0.00579444],
       [-0.01216899,  0.00676558,  0.01336646, ..., -0.00323554,
         0.02038151,  0.02287306]], dtype=float32)> has shape (30522, 768), but the saved weight has shape (50265, 7)`
<!-- A clear and concise description of the question. -->


Transfer Learning on Text Summarization Model

Hi all,
Is there any way to do transfer learning on the Text Summarization model (bertabs-finetuned-cnndm)? I would like to continue training it on my dataset.
The code run_summarization.py only does prediction. Thanks!


changing the attention head size in MultiBert

## 🐛 Bug

<!-- Important information -->
I'm trying to use a MultiBERT model with not all 12 attention heads but just 8 attention heads. so, in config file, I changed the following keys
    config.num_attention_heads = 8
    config.hidden_size = 512
    config.pooler_fc_size = 512

I assumed similar to layer size that we have flexibility in changing it, we may have a similar freedom in changing the head size. however, the run_xnli.py code throws the following error.

>         size mismatch for bert.encoder.layer.11.output.LayerNorm.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in curr
> ent model is torch.Size([512]).
>         size mismatch for bert.pooler.dense.weight: copying a param with shape torch.Size([768, 768]) from checkpoint, the shape in current model is t
> orch.Size([512, 512]).
>         size mismatch for bert.pooler.dense.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Si
> ze([512]).

Model I am using MultiBERT.

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [X ] an official GLUE/SQUaD task: run_xnli.py
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. get the run_xnli.py script and added the following config change codes
    config.num_attention_heads = 8
    config.hidden_size = 512
    config.pooler_fc_size = 512
    config.pooler_num_attention_heads = 8

## Environment

* OS: windows
* Python version: 3.6
* PyTorch version: 1.1
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? yes
* Distributed or parallel setup ? parallel
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


RAM leakage when trying to retrieve the hidden states from the GPT-2 model.

Hello,

I am trying to retrieve hidden state vectors from my trained GPT-2 model from a loop, and there is a huge RAM leakage associated with the operation. Below are my code:

```python
# for loop to calculate TVD
def TVD_loop(test_i, test_dummy_i, nlayer, best_model):

    TVD_tensor = torch.zeros(test_i.size()[1], (nlayer+1), test_i.size()[0]).float()

    # replace every 0's in TVD_tensor to -2
    TVD_tensor = torch.where(TVD_tensor == 0.0, torch.tensor(-2.0), TVD_tensor)

    for m in range(test_i.size()[1]):

        input_ids = test_i[:,m]
        input_ids = torch.tensor(input_ids.tolist()).unsqueeze(0)

        # NOTE: Hidden states are in torch.FloatTensor,
        #       (one for the output of each layer + the output of the embeddings)
        # jth layer
        for j in range(nlayer+1):

            tst_hidden_states = best_model(input_ids)[3][j][0, (test_i.size()[0] - 1), :]

            for l in range(m * test_i.size()[0], (m+1) * test_i.size()[0]):

                input_ids_dummy = test_dummy_i[:,l]
                input_ids_dummy = torch.tensor(input_ids_dummy.tolist()).unsqueeze(0)

                tst_hidden_states_dummy = best_model(input_ids_dummy)[3][j][0, (test_i.size()[0] - 1), :]

                # TVD_tensor[i,j,k] denotes for TVDC calculated at
                # batch i, layer j, and dummy output k
                TVD_tensor[m,j,(l % (test_i.size()[0]))] = TVD(tst_hidden_states, tst_hidden_states_dummy)

    return TVD_tensor
```

I have about ~400GB of RAM, but each time the hidden state vector is retrieved (e.g. ```tst_hidden_states = best_model(input_ids)[3][j][0, (test_i.size()[0] - 1), :]```), it uses up about ~2GB of RAM. How can I prevent this? Thank you,


edited a way to get  at AlbertAttention.forward

'''
# Should find a better way to do this
w = (
    self.dense.weight.t()
    .view(self.num_attention_heads, self.attention_head_size, self.hidden_size)
    .to(context_layer.dtype)
)
b = self.dense.bias.to(context_layer.dtype)
'''

I thought the above code is not necessary.
it can be simply fixed by "merging" `context_layer` at forward().
I committed my "merging" function at AlbertAttention

'''
def merge_tensor(self, x):
    s = x.size()[-2]
    return torch.cat([x[:,:,i,:] for i in range(s)], dim=-1)
'''

I wanted to make a test by "make test' as described in CONTRIBUTING.md
but I couldn't because i faced some make error.

This is my first open source contribution. If i forgot something, please let me know, so I can fix and follow up.


Added custom model dir to PPLM train

Just an option to save the model to other than the working directory.

Default functionality hasn't changed.


RuntimeError: The expanded size of the tensor (449) must match the existing size (2) at non-singleton dimension 2.  Target sizes: [4, 2, 449].  Tensor sizes: [1, 2] while using ALBERT

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I wanted to use ALBERT with a double head
as we have one  for openaigpt with the name OpenAIGPTDoubleHeadsModel
I am doing it with taking inspiration from OpenAIGPTDoubleHeadsModel
but I am getting this error

` File "train.py", line 266, in <module>
    train()
  File "train.py", line 258, in train
    trainer.run(train_loader, max_epochs=args.n_epochs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 446, in run
    self._handle_exception(e)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 410, in _handle_exception
    raise e
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 433, in run
    hours, mins, secs = self._run_once_on_dataset()
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 399, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 410, in _handle_exception
    raise e
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/ignite/engine/engine.py", line 391, in _run_once_on_dataset
    self.state.output = self._process_function(self, batch)
  File "train.py", line 180, in update
    mc_labels=mc_labels, lm_labels=lm_labels
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/transformers/modeling_albert.py", line 956, in forward
    inputs_embeds=inputs_embeds)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/transformers/modeling_albert.py", line 499, in forward
    inputs_embeds=inputs_embeds)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/moha/venv/huggtrans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 171, in forward
    position_ids = position_ids.unsqueeze(0).expand(input_shape)
RuntimeError: The expanded size of the tensor (449) must match the existing size (2) at non-singleton dimension 2.  Target sizes: [4, 2, 449].  Tensor sizes: [1, 2]`

the code block for AlbertDoubleHeadsModel

`class AlbertDoubleHeadsModel(AlbertPreTrainedModel):`
`    def __init__(self, config):`
`        super(AlbertDoubleHeadsModel, self).__init__(config)`

        self.albert = AlbertModel(config)
        self.lm_head = nn.Linear(config.embedding_size, config.vocab_size, bias=False)
        self.multiple_choice_head = SequenceSummary(config)

        self.init_weights()

    def get_output_embeddings(self):
        return self.lm_head

    def forward(self, input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None,
                mc_token_ids=None, lm_labels=None, mc_labels=None):
        transformer_outputs = self.albert(input_ids,
                                               attention_mask=attention_mask,
                                               token_type_ids=token_type_ids,
                                               position_ids=position_ids,
                                               head_mask=head_mask,
                                               inputs_embeds=inputs_embeds)
        hidden_states = transformer_outputs[0]

        lm_logits = self.lm_head(hidden_states)
        mc_logits = self.multiple_choice_head(hidden_states, mc_token_ids).squeeze(-1)

        outputs = (lm_logits, mc_logits) + transformer_outputs[1:]
        if mc_labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(mc_logits.view(-1, mc_logits.size(-1)),
                            mc_labels.view(-1))
            outputs = (loss,) + outputs
        if lm_labels is not None:
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = lm_labels[..., 1:].contiguous()
            loss_fct = CrossEntropyLoss(ignore_index=-1)
            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
                            shift_labels.view(-1))
            outputs = (loss,) + outputs

        return outputs  # (lm loss), (mc loss), lm logits, mc logits, (all hidden_states), (attentions)`

Is there anything that I am missing?
Please do tell me.


What is the f1 score of Squad v2.0 on bert-base? I only got f1 score 74.78.

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello, I am doing some experiment of squad v2.0 on bert-base (NOT bert-large).
According to the BERT paper, bert-large achieves f1 score 81.9 with squad v2.0.
Since I couldn't find the official result for bert-base, I am not sure if I am getting the right f1 score.
Has anyone tried running squad v2.0 on bert base?

I got f1 score **74.78** for squad v2.0 result on bert-base, using below command:
sudo python3 ../../../run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --train_file $SQUAD2_DIR/train-v2.0.json \
  --predict_file $SQUAD2_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 4 \
  --learning_rate 4e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --version_2_with_negative \
  --overwrite_output_dir \
  --output_dir ../../../bert_base/$TASK_NAME/


run_glue.py, CoLA : MCC goes to 0, in some hyperparameter cases

## 🐛 Bug

<!-- Important information -->

Model I am using: roberta-large
Language I am using the model on: English

The problem arise when using:
* [x] the official example scripts: run_glue.py

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: CoLA

## To Reproduce
Steps to reproduce the behavior:
1. Running the following command is one way to create an mcc of 0.  Possible 'non-standard' items here are: LR up to 3e-5, different seeds, warmup, and evaluate-during-training.
```
python run_glue_orig.py --model_type roberta --model_name_or_path roberta-large --task_name CoLA --do_train --do_eval --do_lower_case --evaluate_during_training --logging_steps 50 --save_steps 1000000 --data_dir /home/_DBS/GLUE/CoLA --max_seq_length 128 --per_gpu_eval_batch_size 8 --per_gpu_train_batch_size 8 --learning_rate 3e-5 --num_train_epochs 10 --warmup_steps 641 --output_dir try_fail_1 --seed 3
```
For reference, the args are:
```
Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir='/home/_DBS/GLUE/CoLA', do_eval=True, do_lower_case=True, do_train=True, eval_all_checkpoints=False, evaluate_during_training=True, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=3e-05, local_rank=-1, logging_steps=50, max_grad_norm=1.0, max_seq_length=128, max_steps=-1, model_name_or_path='roberta-large', model_type='roberta', no_cuda=False, num_train_epochs=10.0, output_dir='try_fail_1', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=8, per_gpu_train_batch_size=8, save_steps=1000000, seed=3, server_ip='', server_port='', task_name='CoLA', tokenizer_name='', warmup_steps=641, weight_decay=0.0)
```
 ## Expected behavior
Non-zero mcc values for CoLA, after warmup, even for LR = 3e-5.  (Agree?)

## Environment
* OS: Ubuntu 18.04   Platform Linux-4.15.0-74-generic-x86_64-with-debian-buster-sid
* Python version:    Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
* PyTorch version:  1.2.0.dev20190702 py3.6_cuda10.0.130_cudnn7.5.1_0
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU : TitanXP, qty. 1
* Distributed or parallel setup : no
* Any other relevant information:

## Additional context

* I am trying to reproduce the CoLA results for BERT and RoBERTa.  I've built a test harness that calls a lightly modified version of run_glue.py, following the approach of paragraph 5.1 [here](https://arxiv.org/pdf/1907.11692.pdf).
* Once I saw the mcc=0 values in my version, I went back and reproduced the problem in the standard run_glue.py version.
* I have not seen this (mcc->0) behavior when running the hyperparameter search on bert-base-cased, or roberta-base.  Just on roberta-large.
* The output of the standard run_glue.py is too large to include here, but here is the output from my version, showing what mcc looks like during the run, with the same parameters as above (you need to scroll to the right to see the mcc values):
```
2020-01-19 13:14:10 Ep  0/10 Gstep    50 Step    49:  Loss=  0.680712  LR=2.340094e-06   mcc=  0.000000,
2020-01-19 13:14:45 Ep  0/10 Gstep   100 Step    99:  Loss=  0.610892  LR=4.680187e-06   mcc=  0.000000,
2020-01-19 13:15:19 Ep  0/10 Gstep   150 Step   149:  Loss=  0.627466  LR=7.020281e-06   mcc=  0.000000,
2020-01-19 13:15:54 Ep  0/10 Gstep   200 Step   199:  Loss=  0.603764  LR=9.360374e-06   mcc=  0.000000,
2020-01-19 13:16:28 Ep  0/10 Gstep   250 Step   249:  Loss=  0.576851  LR=1.170047e-05   mcc=  0.222055,
2020-01-19 13:17:03 Ep  0/10 Gstep   300 Step   299:  Loss=  0.581703  LR=1.404056e-05   mcc=  0.046356,
2020-01-19 13:17:37 Ep  0/10 Gstep   350 Step   349:  Loss=  0.580515  LR=1.638066e-05   mcc=  0.176324,
2020-01-19 13:18:12 Ep  0/10 Gstep   400 Step   399:  Loss=  0.499536  LR=1.872075e-05   mcc=  0.452767,
2020-01-19 13:18:46 Ep  0/10 Gstep   450 Step   449:  Loss=  0.586593  LR=2.106084e-05   mcc=  0.459277,
2020-01-19 13:19:21 Ep  0/10 Gstep   500 Step   499:  Loss=  0.619785  LR=2.340094e-05   mcc=  0.465695,
2020-01-19 13:19:55 Ep  0/10 Gstep   550 Step   549:  Loss=  0.554341  LR=2.574103e-05   mcc=  0.480823,
2020-01-19 13:20:29 Ep  0/10 Gstep   600 Step   599:  Loss=  0.573441  LR=2.808112e-05   mcc=  0.272740,
2020-01-19 13:21:04 Ep  0/10 Gstep   650 Step   649:  Loss=  0.581185  LR=2.997313e-05   mcc=  0.347692,
2020-01-19 13:21:38 Ep  0/10 Gstep   700 Step   699:  Loss=  0.662212  LR=2.982386e-05   mcc=  0.000000,
2020-01-19 13:22:13 Ep  0/10 Gstep   750 Step   749:  Loss=  0.601005  LR=2.967459e-05   mcc=  0.000000,
2020-01-19 13:22:47 Ep  0/10 Gstep   800 Step   799:  Loss=  0.648288  LR=2.952533e-05   mcc=  0.000000,
2020-01-19 13:23:22 Ep  0/10 Gstep   850 Step   849:  Loss=  0.637533  LR=2.937606e-05   mcc=  0.000000,
2020-01-19 13:23:56 Ep  0/10 Gstep   900 Step   899:  Loss=  0.633061  LR=2.922679e-05   mcc=  0.000000,
2020-01-19 13:24:30 Ep  0/10 Gstep   950 Step   949:  Loss=  0.647681  LR=2.907752e-05   mcc=  0.000000,
2020-01-19 13:25:05 Ep  0/10 Gstep  1000 Step   999:  Loss=  0.608788  LR=2.892825e-05   mcc=  0.000000,
2020-01-19 13:25:39 Ep  0/10 Gstep  1050 Step  1049:  Loss=  0.620082  LR=2.877898e-05   mcc=  0.000000,
2020-01-19 13:26:14 Ep  1/10 Gstep  1100 Step    30:  Loss=  0.604326  LR=2.862971e-05   mcc=  0.000000,
2020-01-19 13:26:48 Ep  1/10 Gstep  1150 Step    80:  Loss=  0.599047  LR=2.848045e-05   mcc=  0.000000,
2020-01-19 13:27:22 Ep  1/10 Gstep  1200 Step   130:  Loss=  0.635962  LR=2.833118e-05   mcc=  0.000000,
2020-01-19 13:27:57 Ep  1/10 Gstep  1250 Step   180:  Loss=  0.576796  LR=2.818191e-05   mcc=  0.000000,
2020-01-19 13:28:31 Ep  1/10 Gstep  1300 Step   230:  Loss=  0.627478  LR=2.803264e-05   mcc=  0.000000,
2020-01-19 13:29:06 Ep  1/10 Gstep  1350 Step   280:  Loss=  0.584176  LR=2.788337e-05   mcc=  0.000000,
2020-01-19 13:29:40 Ep  1/10 Gstep  1400 Step   330:  Loss=  0.604549  LR=2.773410e-05   mcc=  0.000000,
2020-01-19 13:30:14 Ep  1/10 Gstep  1450 Step   380:  Loss=  0.619358  LR=2.758483e-05   mcc=  0.000000,
2020-01-19 13:30:49 Ep  1/10 Gstep  1500 Step   430:  Loss=  0.582581  LR=2.743557e-05   mcc=  0.000000,
2020-01-19 13:31:23 Ep  1/10 Gstep  1550 Step   480:  Loss=  0.599344  LR=2.728630e-05   mcc=  0.000000,
2020-01-19 13:31:58 Ep  1/10 Gstep  1600 Step   530:  Loss=  0.628334  LR=2.713703e-05   mcc=  0.000000,
2020-01-19 13:32:32 Ep  1/10 Gstep  1650 Step   580:  Loss=  0.619170  LR=2.698776e-05   mcc=  0.000000,
2020-01-19 13:33:07 Ep  1/10 Gstep  1700 Step   630:  Loss=  0.647814  LR=2.683849e-05   mcc=  0.000000,
2020-01-19 13:33:41 Ep  1/10 Gstep  1750 Step   680:  Loss=  0.573997  LR=2.668922e-05   mcc=  0.000000,
2020-01-19 13:34:16 Ep  1/10 Gstep  1800 Step   730:  Loss=  0.591928  LR=2.653995e-05   mcc=  0.000000,
2020-01-19 13:34:50 Ep  1/10 Gstep  1850 Step   780:  Loss=  0.636912  LR=2.639069e-05   mcc=  0.000000,
2020-01-19 13:35:25 Ep  1/10 Gstep  1900 Step   830:  Loss=  0.647883  LR=2.624142e-05   mcc=  0.000000,
2020-01-19 13:35:59 Ep  1/10 Gstep  1950 Step   880:  Loss=  0.612844  LR=2.609215e-05   mcc=  0.000000,
2020-01-19 13:36:34 Ep  1/10 Gstep  2000 Step   930:  Loss=  0.661776  LR=2.594288e-05   mcc=  0.000000,
2020-01-19 13:37:08 Ep  1/10 Gstep  2050 Step   980:  Loss=  0.648593  LR=2.579361e-05   mcc=  0.000000,
2020-01-19 13:37:42 Ep  1/10 Gstep  2100 Step  1030:  Loss=  0.628153  LR=2.564434e-05   mcc=  0.000000,
2020-01-19 13:38:17 Ep  2/10 Gstep  2150 Step    11:  Loss=  0.589748  LR=2.549507e-05   mcc=  0.000000,
2020-01-19 13:38:51 Ep  2/10 Gstep  2200 Step    61:  Loss=  0.583273  LR=2.534581e-05   mcc=  0.000000,
2020-01-19 13:39:26 Ep  2/10 Gstep  2250 Step   111:  Loss=  0.631948  LR=2.519654e-05   mcc=  0.000000,
2020-01-19 13:40:00 Ep  2/10 Gstep  2300 Step   161:  Loss=  0.628461  LR=2.504727e-05   mcc=  0.000000,
2020-01-19 13:40:35 Ep  2/10 Gstep  2350 Step   211:  Loss=  0.644976  LR=2.489800e-05   mcc=  0.000000,
2020-01-19 13:41:09 Ep  2/10 Gstep  2400 Step   261:  Loss=  0.612494  LR=2.474873e-05   mcc=  0.000000,
2020-01-19 13:41:44 Ep  2/10 Gstep  2450 Step   311:  Loss=  0.628107  LR=2.459946e-05   mcc=  0.000000,
2020-01-19 13:42:18 Ep  2/10 Gstep  2500 Step   361:  Loss=  0.663605  LR=2.445019e-05   mcc=  0.000000,
2020-01-19 13:42:52 Ep  2/10 Gstep  2550 Step   411:  Loss=  0.586811  LR=2.430093e-05   mcc=  0.000000,
2020-01-19 13:43:27 Ep  2/10 Gstep  2600 Step   461:  Loss=  0.620628  LR=2.415166e-05   mcc=  0.000000,
2020-01-19 13:44:01 Ep  2/10 Gstep  2650 Step   511:  Loss=  0.608283  LR=2.400239e-05   mcc=  0.000000,
2020-01-19 13:44:36 Ep  2/10 Gstep  2700 Step   561:  Loss=  0.620366  LR=2.385312e-05   mcc=  0.000000,
2020-01-19 13:45:10 Ep  2/10 Gstep  2750 Step   611:  Loss=  0.631265  LR=2.370385e-05   mcc=  0.000000,
2020-01-19 13:45:44 Ep  2/10 Gstep  2800 Step   661:  Loss=  0.636871  LR=2.355458e-05   mcc=  0.000000,
2020-01-19 13:46:19 Ep  2/10 Gstep  2850 Step   711:  Loss=  0.639280  LR=2.340531e-05   mcc=  0.000000,
2020-01-19 13:46:53 Ep  2/10 Gstep  2900 Step   761:  Loss=  0.603878  LR=2.325605e-05   mcc=  0.000000,
2020-01-19 13:47:28 Ep  2/10 Gstep  2950 Step   811:  Loss=  0.647503  LR=2.310678e-05   mcc=  0.000000,
2020-01-19 13:48:02 Ep  2/10 Gstep  3000 Step   861:  Loss=  0.580940  LR=2.295751e-05   mcc=  0.000000,
2020-01-19 13:48:37 Ep  2/10 Gstep  3050 Step   911:  Loss=  0.612520  LR=2.280824e-05   mcc=  0.000000,
2020-01-19 13:49:11 Ep  2/10 Gstep  3100 Step   961:  Loss=  0.607909  LR=2.265897e-05   mcc=  0.000000,
2020-01-19 13:49:46 Ep  2/10 Gstep  3150 Step  1011:  Loss=  0.596036  LR=2.250970e-05   mcc=  0.000000,
2020-01-19 13:50:20 Ep  2/10 Gstep  3200 Step  1061:  Loss=  0.597132  LR=2.236043e-05   mcc=  0.000000,
2020-01-19 13:50:55 Ep  3/10 Gstep  3250 Step    42:  Loss=  0.611773  LR=2.221117e-05   mcc=  0.000000,
2020-01-19 13:51:29 Ep  3/10 Gstep  3300 Step    92:  Loss=  0.605382  LR=2.206190e-05   mcc=  0.000000,
2020-01-19 13:52:03 Ep  3/10 Gstep  3350 Step   142:  Loss=  0.619225  LR=2.191263e-05   mcc=  0.000000,
2020-01-19 13:52:38 Ep  3/10 Gstep  3400 Step   192:  Loss=  0.628465  LR=2.176336e-05   mcc=  0.000000,
2020-01-19 13:53:12 Ep  3/10 Gstep  3450 Step   242:  Loss=  0.626908  LR=2.161409e-05   mcc=  0.000000,
2020-01-19 13:53:47 Ep  3/10 Gstep  3500 Step   292:  Loss=  0.636135  LR=2.146482e-05   mcc=  0.000000,
2020-01-19 13:54:21 Ep  3/10 Gstep  3550 Step   342:  Loss=  0.624727  LR=2.131555e-05   mcc=  0.000000,
2020-01-19 13:54:55 Ep  3/10 Gstep  3600 Step   392:  Loss=  0.620360  LR=2.116629e-05   mcc=  0.000000,
2020-01-19 13:55:30 Ep  3/10 Gstep  3650 Step   442:  Loss=  0.578948  LR=2.101702e-05   mcc=  0.000000,
2020-01-19 13:56:04 Ep  3/10 Gstep  3700 Step   492:  Loss=  0.644757  LR=2.086775e-05   mcc=  0.000000,
2020-01-19 13:56:39 Ep  3/10 Gstep  3750 Step   542:  Loss=  0.599062  LR=2.071848e-05   mcc=  0.000000,
2020-01-19 13:57:13 Ep  3/10 Gstep  3800 Step   592:  Loss=  0.623814  LR=2.056921e-05   mcc=  0.000000,
2020-01-19 13:57:48 Ep  3/10 Gstep  3850 Step   642:  Loss=  0.607594  LR=2.041994e-05   mcc=  0.000000,
2020-01-19 13:58:22 Ep  3/10 Gstep  3900 Step   692:  Loss=  0.636492  LR=2.027067e-05   mcc=  0.000000,
2020-01-19 13:58:57 Ep  3/10 Gstep  3950 Step   742:  Loss=  0.596911  LR=2.012141e-05   mcc=  0.000000,
2020-01-19 13:59:31 Ep  3/10 Gstep  4000 Step   792:  Loss=  0.615585  LR=1.997214e-05   mcc=  0.000000,
2020-01-19 14:00:06 Ep  3/10 Gstep  4050 Step   842:  Loss=  0.583996  LR=1.982287e-05   mcc=  0.000000,
2020-01-19 14:00:40 Ep  3/10 Gstep  4100 Step   892:  Loss=  0.616359  LR=1.967360e-05   mcc=  0.000000,
2020-01-19 14:01:15 Ep  3/10 Gstep  4150 Step   942:  Loss=  0.610478  LR=1.952433e-05   mcc=  0.000000,
2020-01-19 14:01:49 Ep  3/10 Gstep  4200 Step   992:  Loss=  0.604057  LR=1.937506e-05   mcc=  0.000000,
2020-01-19 14:02:23 Ep  3/10 Gstep  4250 Step  1042:  Loss=  0.646733  LR=1.922579e-05   mcc=  0.000000,
2020-01-19 14:02:58 Ep  4/10 Gstep  4300 Step    23:  Loss=  0.603115  LR=1.907653e-05   mcc=  0.000000,
2020-01-19 14:03:32 Ep  4/10 Gstep  4350 Step    73:  Loss=  0.569349  LR=1.892726e-05   mcc=  0.000000,
2020-01-19 14:04:07 Ep  4/10 Gstep  4400 Step   123:  Loss=  0.603698  LR=1.877799e-05   mcc=  0.000000,
2020-01-19 14:04:41 Ep  4/10 Gstep  4450 Step   173:  Loss=  0.616832  LR=1.862872e-05   mcc=  0.000000,
2020-01-19 14:05:15 Ep  4/10 Gstep  4500 Step   223:  Loss=  0.638386  LR=1.847945e-05   mcc=  0.000000,
2020-01-19 14:05:50 Ep  4/10 Gstep  4550 Step   273:  Loss=  0.587255  LR=1.833018e-05   mcc=  0.000000,
2020-01-19 14:06:24 Ep  4/10 Gstep  4600 Step   323:  Loss=  0.616687  LR=1.818091e-05   mcc=  0.000000,
2020-01-19 14:06:59 Ep  4/10 Gstep  4650 Step   373:  Loss=  0.593398  LR=1.803164e-05   mcc=  0.000000,
2020-01-19 14:07:33 Ep  4/10 Gstep  4700 Step   423:  Loss=  0.612735  LR=1.788238e-05   mcc=  0.000000,
2020-01-19 14:08:07 Ep  4/10 Gstep  4750 Step   473:  Loss=  0.612634  LR=1.773311e-05   mcc=  0.000000,
2020-01-19 14:08:42 Ep  4/10 Gstep  4800 Step   523:  Loss=  0.597561  LR=1.758384e-05   mcc=  0.000000,
2020-01-19 14:09:16 Ep  4/10 Gstep  4850 Step   573:  Loss=  0.630737  LR=1.743457e-05   mcc=  0.000000,
2020-01-19 14:09:51 Ep  4/10 Gstep  4900 Step   623:  Loss=  0.639448  LR=1.728530e-05   mcc=  0.000000,
2020-01-19 14:10:25 Ep  4/10 Gstep  4950 Step   673:  Loss=  0.607357  LR=1.713603e-05   mcc=  0.000000,
2020-01-19 14:11:00 Ep  4/10 Gstep  5000 Step   723:  Loss=  0.601610  LR=1.698676e-05   mcc=  0.000000,
2020-01-19 14:11:34 Ep  4/10 Gstep  5050 Step   773:  Loss=  0.604433  LR=1.683750e-05   mcc=  0.000000,
2020-01-19 14:12:08 Ep  4/10 Gstep  5100 Step   823:  Loss=  0.643115  LR=1.668823e-05   mcc=  0.000000,
2020-01-19 14:12:43 Ep  4/10 Gstep  5150 Step   873:  Loss=  0.639950  LR=1.653896e-05   mcc=  0.000000,
2020-01-19 14:13:17 Ep  4/10 Gstep  5200 Step   923:  Loss=  0.638993  LR=1.638969e-05   mcc=  0.000000,
2020-01-19 14:13:51 Ep  4/10 Gstep  5250 Step   973:  Loss=  0.633510  LR=1.624042e-05   mcc=  0.000000,
2020-01-19 14:14:26 Ep  4/10 Gstep  5300 Step  1023:  Loss=  0.581198  LR=1.609115e-05   mcc=  0.000000,
2020-01-19 14:15:00 Ep  5/10 Gstep  5350 Step     4:  Loss=  0.592722  LR=1.594188e-05   mcc=  0.000000,
2020-01-19 14:15:34 Ep  5/10 Gstep  5400 Step    54:  Loss=  0.614371  LR=1.579262e-05   mcc=  0.000000,
2020-01-19 14:16:09 Ep  5/10 Gstep  5450 Step   104:  Loss=  0.607973  LR=1.564335e-05   mcc=  0.000000,
2020-01-19 14:16:43 Ep  5/10 Gstep  5500 Step   154:  Loss=  0.605945  LR=1.549408e-05   mcc=  0.000000,
2020-01-19 14:17:18 Ep  5/10 Gstep  5550 Step   204:  Loss=  0.620083  LR=1.534481e-05   mcc=  0.000000,
2020-01-19 14:17:52 Ep  5/10 Gstep  5600 Step   254:  Loss=  0.625456  LR=1.519554e-05   mcc=  0.000000,
2020-01-19 14:18:27 Ep  5/10 Gstep  5650 Step   304:  Loss=  0.633456  LR=1.504627e-05   mcc=  0.000000,
2020-01-19 14:19:01 Ep  5/10 Gstep  5700 Step   354:  Loss=  0.608953  LR=1.489700e-05   mcc=  0.000000,
2020-01-19 14:19:35 Ep  5/10 Gstep  5750 Step   404:  Loss=  0.590640  LR=1.474774e-05   mcc=  0.000000,
2020-01-19 14:20:10 Ep  5/10 Gstep  5800 Step   454:  Loss=  0.608632  LR=1.459847e-05   mcc=  0.000000,
2020-01-19 14:20:44 Ep  5/10 Gstep  5850 Step   504:  Loss=  0.615661  LR=1.444920e-05   mcc=  0.000000,
2020-01-19 14:21:19 Ep  5/10 Gstep  5900 Step   554:  Loss=  0.602201  LR=1.429993e-05   mcc=  0.000000,
2020-01-19 14:21:53 Ep  5/10 Gstep  5950 Step   604:  Loss=  0.593200  LR=1.415066e-05   mcc=  0.000000,
2020-01-19 14:22:27 Ep  5/10 Gstep  6000 Step   654:  Loss=  0.623690  LR=1.400139e-05   mcc=  0.000000,
2020-01-19 14:23:02 Ep  5/10 Gstep  6050 Step   704:  Loss=  0.608412  LR=1.385212e-05   mcc=  0.000000,
2020-01-19 14:23:36 Ep  5/10 Gstep  6100 Step   754:  Loss=  0.637673  LR=1.370286e-05   mcc=  0.000000,
2020-01-19 14:24:11 Ep  5/10 Gstep  6150 Step   804:  Loss=  0.606503  LR=1.355359e-05   mcc=  0.000000,
2020-01-19 14:24:45 Ep  5/10 Gstep  6200 Step   854:  Loss=  0.575204  LR=1.340432e-05   mcc=  0.000000,
2020-01-19 14:25:19 Ep  5/10 Gstep  6250 Step   904:  Loss=  0.619873  LR=1.325505e-05   mcc=  0.000000,
2020-01-19 14:25:54 Ep  5/10 Gstep  6300 Step   954:  Loss=  0.587598  LR=1.310578e-05   mcc=  0.000000,
2020-01-19 14:26:28 Ep  5/10 Gstep  6350 Step  1004:  Loss=  0.637348  LR=1.295651e-05   mcc=  0.000000,
2020-01-19 14:27:03 Ep  5/10 Gstep  6400 Step  1054:  Loss=  0.613608  LR=1.280724e-05   mcc=  0.000000,
2020-01-19 14:27:37 Ep  6/10 Gstep  6450 Step    35:  Loss=  0.631628  LR=1.265798e-05   mcc=  0.000000,
2020-01-19 14:28:12 Ep  6/10 Gstep  6500 Step    85:  Loss=  0.640207  LR=1.250871e-05   mcc=  0.000000,
2020-01-19 14:28:46 Ep  6/10 Gstep  6550 Step   135:  Loss=  0.620651  LR=1.235944e-05   mcc=  0.000000,
2020-01-19 14:29:20 Ep  6/10 Gstep  6600 Step   185:  Loss=  0.620827  LR=1.221017e-05   mcc=  0.000000,
2020-01-19 14:29:55 Ep  6/10 Gstep  6650 Step   235:  Loss=  0.600443  LR=1.206090e-05   mcc=  0.000000,
2020-01-19 14:30:29 Ep  6/10 Gstep  6700 Step   285:  Loss=  0.617364  LR=1.191163e-05   mcc=  0.000000,
2020-01-19 14:31:04 Ep  6/10 Gstep  6750 Step   335:  Loss=  0.640479  LR=1.176236e-05   mcc=  0.000000,
2020-01-19 14:31:38 Ep  6/10 Gstep  6800 Step   385:  Loss=  0.636902  LR=1.161310e-05   mcc=  0.000000,
2020-01-19 14:32:12 Ep  6/10 Gstep  6850 Step   435:  Loss=  0.596365  LR=1.146383e-05   mcc=  0.000000,
2020-01-19 14:32:47 Ep  6/10 Gstep  6900 Step   485:  Loss=  0.608958  LR=1.131456e-05   mcc=  0.000000,
2020-01-19 14:33:21 Ep  6/10 Gstep  6950 Step   535:  Loss=  0.619341  LR=1.116529e-05   mcc=  0.000000,
2020-01-19 14:33:56 Ep  6/10 Gstep  7000 Step   585:  Loss=  0.626361  LR=1.101602e-05   mcc=  0.000000,
2020-01-19 14:34:30 Ep  6/10 Gstep  7050 Step   635:  Loss=  0.613354  LR=1.086675e-05   mcc=  0.000000,
2020-01-19 14:35:05 Ep  6/10 Gstep  7100 Step   685:  Loss=  0.612054  LR=1.071748e-05   mcc=  0.000000,
2020-01-19 14:35:39 Ep  6/10 Gstep  7150 Step   735:  Loss=  0.613521  LR=1.056822e-05   mcc=  0.000000,
2020-01-19 14:36:13 Ep  6/10 Gstep  7200 Step   785:  Loss=  0.577611  LR=1.041895e-05   mcc=  0.000000,
2020-01-19 14:36:48 Ep  6/10 Gstep  7250 Step   835:  Loss=  0.605257  LR=1.026968e-05   mcc=  0.000000,
2020-01-19 14:37:22 Ep  6/10 Gstep  7300 Step   885:  Loss=  0.611978  LR=1.012041e-05   mcc=  0.000000,
2020-01-19 14:37:57 Ep  6/10 Gstep  7350 Step   935:  Loss=  0.595855  LR=9.971141e-06   mcc=  0.000000,
2020-01-19 14:38:31 Ep  6/10 Gstep  7400 Step   985:  Loss=  0.585623  LR=9.821873e-06   mcc=  0.000000,
2020-01-19 14:39:05 Ep  6/10 Gstep  7450 Step  1035:  Loss=  0.590831  LR=9.672604e-06   mcc=  0.000000,
2020-01-19 14:39:40 Ep  7/10 Gstep  7500 Step    16:  Loss=  0.621975  LR=9.523336e-06   mcc=  0.000000,
2020-01-19 14:40:14 Ep  7/10 Gstep  7550 Step    66:  Loss=  0.602145  LR=9.374067e-06   mcc=  0.000000,
2020-01-19 14:40:49 Ep  7/10 Gstep  7600 Step   116:  Loss=  0.620748  LR=9.224798e-06   mcc=  0.000000,
2020-01-19 14:41:23 Ep  7/10 Gstep  7650 Step   166:  Loss=  0.602158  LR=9.075530e-06   mcc=  0.000000,
2020-01-19 14:41:58 Ep  7/10 Gstep  7700 Step   216:  Loss=  0.573956  LR=8.926261e-06   mcc=  0.000000,
2020-01-19 14:42:32 Ep  7/10 Gstep  7750 Step   266:  Loss=  0.585606  LR=8.776993e-06   mcc=  0.000000,
2020-01-19 14:43:06 Ep  7/10 Gstep  7800 Step   316:  Loss=  0.585316  LR=8.627724e-06   mcc=  0.000000,
2020-01-19 14:43:41 Ep  7/10 Gstep  7850 Step   366:  Loss=  0.643286  LR=8.478456e-06   mcc=  0.000000,
2020-01-19 14:44:15 Ep  7/10 Gstep  7900 Step   416:  Loss=  0.607292  LR=8.329187e-06   mcc=  0.000000,
2020-01-19 14:44:50 Ep  7/10 Gstep  7950 Step   466:  Loss=  0.599829  LR=8.179918e-06   mcc=  0.000000,
2020-01-19 14:45:24 Ep  7/10 Gstep  8000 Step   516:  Loss=  0.651290  LR=8.030650e-06   mcc=  0.000000,
2020-01-19 14:45:58 Ep  7/10 Gstep  8050 Step   566:  Loss=  0.631367  LR=7.881381e-06   mcc=  0.000000,
2020-01-19 14:46:33 Ep  7/10 Gstep  8100 Step   616:  Loss=  0.602171  LR=7.732113e-06   mcc=  0.000000,
2020-01-19 14:47:07 Ep  7/10 Gstep  8150 Step   666:  Loss=  0.649055  LR=7.582844e-06   mcc=  0.000000,
2020-01-19 14:47:42 Ep  7/10 Gstep  8200 Step   716:  Loss=  0.588507  LR=7.433575e-06   mcc=  0.000000,
2020-01-19 14:48:16 Ep  7/10 Gstep  8250 Step   766:  Loss=  0.649030  LR=7.284307e-06   mcc=  0.000000,
2020-01-19 14:48:50 Ep  7/10 Gstep  8300 Step   816:  Loss=  0.622789  LR=7.135038e-06   mcc=  0.000000,
2020-01-19 14:49:25 Ep  7/10 Gstep  8350 Step   866:  Loss=  0.586203  LR=6.985770e-06   mcc=  0.000000,
2020-01-19 14:49:59 Ep  7/10 Gstep  8400 Step   916:  Loss=  0.597735  LR=6.836501e-06   mcc=  0.000000,
2020-01-19 14:50:34 Ep  7/10 Gstep  8450 Step   966:  Loss=  0.636263  LR=6.687233e-06   mcc=  0.000000,
2020-01-19 14:51:08 Ep  7/10 Gstep  8500 Step  1016:  Loss=  0.612294  LR=6.537964e-06   mcc=  0.000000,
2020-01-19 14:51:42 Ep  7/10 Gstep  8550 Step  1066:  Loss=  0.576644  LR=6.388695e-06   mcc=  0.000000,
2020-01-19 14:52:17 Ep  8/10 Gstep  8600 Step    47:  Loss=  0.627220  LR=6.239427e-06   mcc=  0.000000,
2020-01-19 14:52:51 Ep  8/10 Gstep  8650 Step    97:  Loss=  0.613375  LR=6.090158e-06   mcc=  0.000000,
2020-01-19 14:53:26 Ep  8/10 Gstep  8700 Step   147:  Loss=  0.555401  LR=5.940890e-06   mcc=  0.000000,
2020-01-19 14:54:00 Ep  8/10 Gstep  8750 Step   197:  Loss=  0.593526  LR=5.791621e-06   mcc=  0.000000,
2020-01-19 14:54:34 Ep  8/10 Gstep  8800 Step   247:  Loss=  0.659518  LR=5.642352e-06   mcc=  0.000000,
2020-01-19 14:55:09 Ep  8/10 Gstep  8850 Step   297:  Loss=  0.626100  LR=5.493084e-06   mcc=  0.000000,
2020-01-19 14:55:43 Ep  8/10 Gstep  8900 Step   347:  Loss=  0.634845  LR=5.343815e-06   mcc=  0.000000,
2020-01-19 14:56:18 Ep  8/10 Gstep  8950 Step   397:  Loss=  0.608417  LR=5.194547e-06   mcc=  0.000000,
2020-01-19 14:56:52 Ep  8/10 Gstep  9000 Step   447:  Loss=  0.631753  LR=5.045278e-06   mcc=  0.000000,
2020-01-19 14:57:27 Ep  8/10 Gstep  9050 Step   497:  Loss=  0.596399  LR=4.896010e-06   mcc=  0.000000,
2020-01-19 14:58:01 Ep  8/10 Gstep  9100 Step   547:  Loss=  0.612060  LR=4.746741e-06   mcc=  0.000000,
2020-01-19 14:58:35 Ep  8/10 Gstep  9150 Step   597:  Loss=  0.612795  LR=4.597472e-06   mcc=  0.000000,
2020-01-19 14:59:10 Ep  8/10 Gstep  9200 Step   647:  Loss=  0.615351  LR=4.448204e-06   mcc=  0.000000,
2020-01-19 14:59:44 Ep  8/10 Gstep  9250 Step   697:  Loss=  0.643801  LR=4.298935e-06   mcc=  0.000000,
2020-01-19 15:00:19 Ep  8/10 Gstep  9300 Step   747:  Loss=  0.598972  LR=4.149667e-06   mcc=  0.000000,
2020-01-19 15:00:53 Ep  8/10 Gstep  9350 Step   797:  Loss=  0.624094  LR=4.000398e-06   mcc=  0.000000,
2020-01-19 15:01:28 Ep  8/10 Gstep  9400 Step   847:  Loss=  0.628975  LR=3.851129e-06   mcc=  0.000000,
2020-01-19 15:02:02 Ep  8/10 Gstep  9450 Step   897:  Loss=  0.589227  LR=3.701861e-06   mcc=  0.000000,
2020-01-19 15:02:36 Ep  8/10 Gstep  9500 Step   947:  Loss=  0.609399  LR=3.552592e-06   mcc=  0.000000,
2020-01-19 15:03:11 Ep  8/10 Gstep  9550 Step   997:  Loss=  0.599416  LR=3.403324e-06   mcc=  0.000000,
2020-01-19 15:03:45 Ep  8/10 Gstep  9600 Step  1047:  Loss=  0.583097  LR=3.254055e-06   mcc=  0.000000,
2020-01-19 15:04:20 Ep  9/10 Gstep  9650 Step    28:  Loss=  0.609004  LR=3.104787e-06   mcc=  0.000000,
2020-01-19 15:04:54 Ep  9/10 Gstep  9700 Step    78:  Loss=  0.599495  LR=2.955518e-06   mcc=  0.000000,
2020-01-19 15:05:28 Ep  9/10 Gstep  9750 Step   128:  Loss=  0.627960  LR=2.806249e-06   mcc=  0.000000,
2020-01-19 15:06:03 Ep  9/10 Gstep  9800 Step   178:  Loss=  0.641110  LR=2.656981e-06   mcc=  0.000000,
2020-01-19 15:06:37 Ep  9/10 Gstep  9850 Step   228:  Loss=  0.594384  LR=2.507712e-06   mcc=  0.000000,
2020-01-19 15:07:12 Ep  9/10 Gstep  9900 Step   278:  Loss=  0.639790  LR=2.358444e-06   mcc=  0.000000,
2020-01-19 15:07:46 Ep  9/10 Gstep  9950 Step   328:  Loss=  0.606976  LR=2.209175e-06   mcc=  0.000000,
2020-01-19 15:08:21 Ep  9/10 Gstep 10000 Step   378:  Loss=  0.591287  LR=2.059906e-06   mcc=  0.000000,
2020-01-19 15:08:55 Ep  9/10 Gstep 10050 Step   428:  Loss=  0.591274  LR=1.910638e-06   mcc=  0.000000,
2020-01-19 15:09:29 Ep  9/10 Gstep 10100 Step   478:  Loss=  0.611863  LR=1.761369e-06   mcc=  0.000000,
2020-01-19 15:10:04 Ep  9/10 Gstep 10150 Step   528:  Loss=  0.623777  LR=1.612101e-06   mcc=  0.000000,
2020-01-19 15:10:38 Ep  9/10 Gstep 10200 Step   578:  Loss=  0.631314  LR=1.462832e-06   mcc=  0.000000,
2020-01-19 15:11:13 Ep  9/10 Gstep 10250 Step   628:  Loss=  0.641188  LR=1.313564e-06   mcc=  0.000000,
2020-01-19 15:11:47 Ep  9/10 Gstep 10300 Step   678:  Loss=  0.634291  LR=1.164295e-06   mcc=  0.000000,
2020-01-19 15:12:22 Ep  9/10 Gstep 10350 Step   728:  Loss=  0.609055  LR=1.015026e-06   mcc=  0.000000,
2020-01-19 15:12:56 Ep  9/10 Gstep 10400 Step   778:  Loss=  0.609741  LR=8.657578e-07   mcc=  0.000000,
2020-01-19 15:13:30 Ep  9/10 Gstep 10450 Step   828:  Loss=  0.617953  LR=7.164892e-07   mcc=  0.000000,
2020-01-19 15:14:05 Ep  9/10 Gstep 10500 Step   878:  Loss=  0.628554  LR=5.672206e-07   mcc=  0.000000,
2020-01-19 15:14:39 Ep  9/10 Gstep 10550 Step   928:  Loss=  0.567698  LR=4.179520e-07   mcc=  0.000000,
2020-01-19 15:15:14 Ep  9/10 Gstep 10600 Step   978:  Loss=  0.603719  LR=2.686835e-07   mcc=  0.000000,
2020-01-19 15:15:48 Ep  9/10 Gstep 10650 Step  1028:  Loss=  0.587463  LR=1.194149e-07   mcc=  0.000000,
```
* Finally, here is what the results look like over a lot of hyperparameter choices, for roberta-large.  See the 3rd column from the right, 'last_mcc', and notice that several (but not all) finish with mcc = 0.
* The command for run_glue.py I included at the top corresponds with the last line in this table.  Note that mcc finishes =  0 for all seeds for LR=3e-5, but changing to LR=2e-5, and things are better. (But: see line 7)
```
#     bs       LR  seed  best_mcc  warmup  best_mcc_step  last_mcc  last_mcc_step  last_loss
0   12.0  0.00001   1.0  0.681680   427.0         3600.0  0.623916         7100.0   0.066959
1   12.0  0.00001   2.0  0.678343   427.0         1800.0  0.643239         7100.0   0.028075
2   12.0  0.00001   3.0  0.683793   427.0         3550.0  0.648869         7100.0   0.069469
3   12.0  0.00002   1.0  0.680998   427.0         5450.0  0.635618         7100.0   0.021568
4   12.0  0.00002   2.0  0.651091   427.0         4350.0  0.631009         7100.0   0.042258
5   12.0  0.00002   3.0  0.661668   427.0         4500.0  0.645933         7100.0   0.091654
6   12.0  0.00003   1.0  0.650776   427.0         5150.0  0.598263         7100.0   0.056770
7   12.0  0.00003   2.0  0.403664   427.0          300.0  0.000000         7100.0   0.641559
8   12.0  0.00003   3.0  0.637731   427.0         1800.0  0.597579         7100.0   0.078028
9    8.0  0.00001   1.0  0.667611   641.0         7400.0  0.640759        10650.0   0.046092
10   8.0  0.00001   2.0  0.685714   641.0         4050.0  0.639093        10650.0   0.035902
11   8.0  0.00001   3.0  0.691234   641.0         2650.0  0.625711        10650.0   0.016033
12   8.0  0.00002   1.0  0.670903   641.0         6100.0  0.655837        10650.0   0.071015
13   8.0  0.00002   2.0  0.660777   641.0         8700.0  0.623216        10650.0   0.183001
14   8.0  0.00002   3.0  0.633514   641.0         8450.0  0.585573        10650.0   0.061994
15   8.0  0.00003   1.0  0.521248   641.0          500.0  0.000000        10650.0   0.622017
16   8.0  0.00003   2.0  0.494827   641.0          600.0  0.000000        10650.0   0.614100
17   8.0  0.00003   3.0  0.480823   641.0          550.0  0.000000        10650.0   0.587463
```


run_lm_finetuning.py regenerates examples cache when restored from a checkpoint, is this intended?

## ❓ Questions & Help

Hello,

I am finetuning a gpt2-medium model on a large (800mb+) input via run_lm_finetuning.py on a windown/conda env with recent git checkout of transformers and apex installed (--per_gpu_train_batch_size=1 --fp16 --fp16_opt_level O2 --gradient_accumulation_steps=10 --block_size=512).
On initiall run a cache file is created next to the input file using the model type as filename prefix, i.e. gpt2-medium_cached_lm_512_dataset.txt

This WAI and if I restart the finetuning process the cache is picked up, but when I try to resume by restoring a checkpoint (--model_name_or_path=FULL_PATH\output\checkpoint-200\) the cache is regenerated in the /output folder under a name of checkpoint-200_cached_lm_512_dataset.txt.
(also providing a relative path to model_name_or_path does not seem to work[1])

- Tracing the cache generation code it looks like it depends on the tokenizer which is restored from the checkpoint, but I am not sure if any state from the checkpoint is actually affecting the cache generation. Both files have the same checksum, so it does seem like unnecessary work?
- If not, Is it possible to store current version of the cache alongside other checkpoint data for faster resume?
- Am I restoring the model incorrectly? Even after cache regeneration the models OOMs, but starting from scratch I can finetune way past the checkpoint state. [2]

Thank you!

[1]
```
Saving features into cached file C:\data\.\output\checkpoint-200\_cached_lm_512_dataset.txt
Traceback (most recent call last):
  File ".\transformers\examples\run_lm_finetuning.py", line 721, in <module>
    main()
  File ".\transformers\examples\run_lm_finetuning.py", line 666, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File ".\transformers\examples\run_lm_finetuning.py", line 130, in load_and_cache_examples
    block_size=args.block_size,
  File ".\transformers\examples\run_lm_finetuning.py", line 115, in __init__
    with open(cached_features_file, "wb") as handle:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\data\\.\\output\\checkpoint-200\\_cached_lm_512_dataset.txt'
```

[2]

```
  File ".\transformers\examples\run_lm_finetuning.py", line 721, in <module>
    main()
  File ".\transformers\examples\run_lm_finetuning.py", line 671, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File ".\transformers\examples\run_lm_finetuning.py", line 316, in train
    with amp.scale_loss(loss, optimizer) as scaled_loss:
  File "...\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "...\apex\amp\handle.py", line 111, in scale_loss
    optimizer._prepare_amp_backward()
  File "...\apex\amp\_process_optimizer.py", line 137, in prepare_backward_with_master_weights
    self._amp_lazy_init()
  File "...\apex\amp\_process_optimizer.py", line 309, in _amp_lazy_init
    self._lazy_init_maybe_master_weights()
  File "...\apex\amp\_process_optimizer.py", line 90, in lazy_init_with_master_weights
    self.load_state_dict(self.state_dict())
  File "...\torch\optim\optimizer.py", line 147, in load_state_dict
    state[param] = cast(param, v)
  File "...\torch\optim\optimizer.py", line 134, in cast
    return {k: cast(param, v) for k, v in value.items()}
  File "...\torch\optim\optimizer.py", line 134, in <dictcomp>
    return {k: cast(param, v) for k, v in value.items()}
  File "...\torch\optim\optimizer.py", line 130, in cast
    value = value.to(param.dtype)
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.00 GiB total capacity; 8.61 GiB already allocated; 10.74 MiB free; 8.69 GiB reserved in total by PyTorch)
```


how can i download the model manually?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to download the model manually because of my network. But now I can only find the download address of bert. Where is the address of all models? Such as XLNET。


The accuracy of XLNet

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....):
XLNet
Language I am using the model on (English, Chinese....):
English
The problem arise when using:
 [ ] the official example scripts: (give details)
 [ * ] my own modified scripts: (give details)
I use my own scripts under this library

The tasks I am working on is:
[ ] an official GLUE/SQUaD task: (give the name)
[ * ] my own task or dataset: (give details)
TVQA dataset, for question answering task

Details of the issue:

I was using pytorch-pretrained-BERT library on my scripts. Last week, I switched to the "transformer" library. After that, I tried to use BERT with "bert-base-cased" (do_lower_case=true) model, the valid acc is around 64% after 800 it (batch size=8). However, when I modify the model type to "xlnet" and the model with "xlnet-base-cased" (I have set left padding as true and pad_token_segment_id=4. The other parts are kept the same as using BERT), the valid acc is only about 28% after 800 it (batch size=8). When more iterations are made, the valid acc will drop to about 22%. I think this is quite strange.

## Environment

* OS: Windows 10
* Python version: 3.6
* PyTorch version: 1.0
* PyTorch Transformers version (or branch): The newest? (Downloaded last Monday)
* Using GPU ? Yes
* Distributed or parallel setup ? No
* Any other relevant information:

## Checklist

- [ * ] I have read the migration guide in the readme.
- [ * ] I checked if a related official extension example runs on my machine.

## Additional context

<!-- Add any other context about the problem here. -->


PyTorch 1.2 has released API 'torch.nn.Transformer'，so it's better to modify the source code with the official  python API

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

## Motivation

It's better to Modify modeling_bert.py with official API 'torch.nn.Transformer' of PyTorch 1.2

## Additional context

https://pytorch.org/docs/stable/nn.html?highlight=transformer#torch.nn.Transformer


Attibute Error：‘NoneType’ object has no attribute 'seek' and OSError

My pytorch version is 1.4.0+cpu and tensorflow version is 2.0.0-dev20191002.
/torch/serialization.py,line 289,in_check_seekable
‘NoneType’ object has no attribute 'seek'
You can only torch.load from a file that is seekable.Please pre_load the data into a buffer like io.BytesIO and try to load from it instead.
But how should I do to sovle the quetions?
Another question.
OSError:Unable to load weights from pytorch checkpoint file.If you tried to load a PyTorch model from a TF 2.0 checkpoint,please set from_tf=true.


what's the structure of the model saved after fine-tuning ?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
 Hello ! I'm wondering what is the structure of the model saved after fine-tuning. For example, after the sequence classification fine-tuning , how to show the layers information of newly-formed model ? Is new model's sentence vector different from the original one which is extracted from pre-trained model ?


## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
How to start a server and client to get feature vectors，or Which part of the code should I study in https://github.com/huggingface/transformers.git.


XLM-Roberta checkpoint redundant weight

deleted


I'm trying to run TFBertForTokenClassification with tensorflow_datasets.load('glue/sst2'):

```py
import tensorflow as tf
import tensorflow_datasets
from transformers import *

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertForTokenClassification.from_pretrained('bert-base-uncased')
data = tensorflow_datasets.load('glue/sst2')

train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='sst-2')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='sst-2')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)
```

while model.fit I get this error:

```
Train for 115 steps, validate for 7 steps
Epoch 1/2
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_for_token_classification_1/bert/pooler/dense/kernel:0', 'tf_bert_for_token_classification_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_for_token_classification_1/bert/pooler/dense/kernel:0', 'tf_bert_for_token_classification_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_for_token_classification_1/bert/pooler/dense/kernel:0', 'tf_bert_for_token_classification_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_for_token_classification_1/bert/pooler/dense/kernel:0', 'tf_bert_for_token_classification_1/bert/pooler/dense/bias:0'] when minimizing the loss.
  1/115 [..............................] - ETA: 25:00
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-15-f52b3b390355> in <module>()
      1 history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
----> 2                     validation_data=valid_dataset, validation_steps=7)

11 frames
/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  assertion failed: [Condition x == y did not hold element-wise:] [x (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/Shape_1:0) = ] [32 1] [y (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/strided_slice:0) = ] [32 128]
	 [[node loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/assert_equal_1/Assert/Assert (defined at <ipython-input-15-f52b3b390355>:2) ]]
	 [[Reshape_824/_584]]
  (1) Invalid argument:  assertion failed: [Condition x == y did not hold element-wise:] [x (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/Shape_1:0) = ] [32 1] [y (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/strided_slice:0) = ] [32 128]
	 [[node loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/assert_equal_1/Assert/Assert (defined at <ipython-input-15-f52b3b390355>:2) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_distributed_function_477134]

Function call stack:
distributed_function -> distributed_function
```


glue_convert_examples_to_features in  glue.py runs to  errors

## 🐛 Bug

<!-- Important information -->

Model I am using: Bert

Language I am using the model on (English)

The problem arise when using:
* [x] the official example scripts: (give details):
I have a venv running with TF2.0 and transformers, and I am running mrpc dataset with BERT. Here's the code:

train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)

## To Reproduce

Steps to reproduce the behavior:

I am using the official code on hugging face for BERT and  sequence calssification and it is not working...

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased', force_download=True)
data = tensorflow_datasets.load('glue/mrpc')

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
Error message:

File "hugf-bert.py", line 20, in <module>
    train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
  File "/Users/ns5kn/Documents/insight/transformers/src/transformers/data/processors/glue.py", line 84, in glue_convert_examples_to_features
    logger.info("Writing example %d/%d" % (ex_index, len(examples)))
TypeError: object of type '_OptionsDataset' has no len()

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: mac 10.14
* Python version: 3.6
* Tensorflow version: 2.0
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed or parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Added saving to custom dir in PPLM train


GPT2TokenizerFast object has no attribute 'with_pre_tokenizer'

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')

The tasks I am working on is:

* [ ] my own task or dataset:

## To Reproduce

Steps to reproduce the behavior:

1.tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. --><ipython-input-97-b0a0cde738fe> in <module>
----> 1 tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')

/media/bahram/New Volume/Projects/Python_LM/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    307
    308         """
--> 309         return cls._from_pretrained(*inputs, **kwargs)
    310
    311     @classmethod

/media/bahram/New Volume/Projects/Python_LM/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
    459         # Instantiate tokenizer.
    460         try:
--> 461             tokenizer = cls(*init_inputs, **init_kwargs)
    462         except OSError:
    463             raise OSError(

/media/bahram/New Volume/Projects/Python_LM/transformers/tokenization_gpt2.py in __init__(self, vocab_file, merges_file, unk_token, bos_token, eos_token, pad_to_max_length, add_prefix_space, max_length, stride, truncation_strategy, **kwargs)

AttributeError: 'Tokenizer' object has no attribute 'with_pre_tokenizer'

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
AttributeError: 'Tokenizer' object has no attribute 'with_pre_tokenizer'
## Environment

* OS: Ubuntu 18
* Python version: 3.7
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU yes
* Distributed or parallel setup No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


always occur error:AssertionError

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I put the wiki.train.raw and the wiki.test.raw in /dataset,
then run the command:
python run_lm_finetuning.py     --output_dir=output     --model_type=roberta     --model_name_or_path=roberta-base     --do_train     --train_data_file=/dataset/wiki.train.raw     --do_eval     --eval_data_file=/dataset/wiki.test.raw     --mlm

errors:
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 717, in <module>
    main()
  File "run_lm_finetuning.py", line 662, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 127, in load_and_cache_examples
    block_size=args.block_size,
  File "run_lm_finetuning.py", line 86, in __init__
    assert os.path.isfile(file_path)
AssertionError

then I find the code in run_lm_finetuning.py,
I don't know change file_path in def __init__:
def __init__(self, tokenizer, args, file_path="train", block_size=512):
I hope you can help me out.


fill_mask helper


Fix examples/run_tf_ner.py label encoding error #2559

This is an explanation and a proposed fix for #2559
The code set `pad_token_label_id = 0`, and increase the total number of labels `num_labels = len(labels) + 1`, but made no change to the label list. Thus the first label in label list has the same index as pad_token_label_id.

Following instructions in README take GermEval 2014 as an example, for one sentence in test dataset the token `Aachen` is labeled as `B-LOC` (`B-LOC` is the first label in label list), yet because of the collision with pad_token_label_id, both pad tokens and `Aachen` are encoded as 0:
![image](https://user-images.githubusercontent.com/1331543/72657322-bb90fa00-3957-11ea-980d-33ac12e54545.png)
And the test_predictions.txt is also off by one:
```
1951 I-PERpart
bis I-PERpart
1953 I-PERpart
wurde I-PERpart
...
```

The fix adds a placeholder label `[PAD]` at position 0 when loading the datasets and all labels positions are shifted by 1. The resulting encoding for the same sample sentence:
![image](https://user-images.githubusercontent.com/1331543/72657627-5e974300-395b-11ea-905e-4a3028f4b5c4.png)

And the test_predictions.txt thus has correct index:
```
1951 O
bis O
1953 O
wurde O
...
```


is RoBERTa-base.json in s3  wrong?

Q:
when i open the json file downloaded from this url:
`https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-vocab.json`,
i found there is so many wrong strange code in it,like this:
> {"<s>": 0, "<pad>": 1, "</s>": 2, "<unk>": 3, ".": 4, "Ġthe": 5, ",": 6, "Ġto": 7, "Ġand": 8, "Ġof": 9, "Ġa": 10, "Ġin": 11, "-": 12, "Ġfor": 13, "Ġthat": 14, "Ġon": 15, "Ġis": 16, "âĢ": 17, "'s": 18, "Ġwith": 19, "ĠThe": 20, "Ġwas": 21, "Ġ\"": 22, "Ġat": 23, "Ġit": 24, "Ġas": 25, "Ġsaid": 26, "Ļ": 27, "Ġbe": 28, "s": 29, "Ġby": 30, "Ġfrom": 31, "Ġare": 32, "Ġhave": 33, "Ġhas": 34, ":": 35, "Ġ(": 36, "Ġhe": 37, "ĠI": 38, "Ġhis": 39, "Ġwill": 40, "Ġan": 41, "Ġthis": 42, ")": 43, "ĠâĢ": 44, "Ġnot": 45, "Ŀ": 46, "Ġyou": 47, "ľ": 48, "Ġtheir": 49, "Ġor": 50, "Ġthey": 51, "Ġwe": 52, "Ġbut": 53, "Ġwho": 54, "Ġmore": 55, "Ġhad": 56, "Ġbeen": 57, "Ġwere": 58, "Ġabout": 59, ",\"": 60, "Ġwhich": 61, "Ġup": 62, "Ġits": 63, "Ġcan": 64, "Ġone": 65, "Ġout": 66, "Ġalso": 67, "Ġ$": 68, "Ġher": 69, "Ġall": 70, "Ġafter": 71, ".\"": 72, "/": 73, "Ġwould": 74, "'t": 75, "Ġyear": 76, "Ġwhen": 77, "Ġfirst": 78, "Ġshe": 79, "Ġtwo": 80, "Ġover": 81, "Ġpeople": 82, "ĠA": 83, "Ġour": 84, "ĠIt": 85, "Ġtime": 86, "Ġthan": 87, "Ġinto": 88, "Ġthere": 89, "t": 90, "ĠHe": 91, "Ġnew": 92, "ĠâĢĶ": 93, "Ġlast": 94, "Ġjust": 95, "ĠIn": 96, "Ġother": 97, "Ġso": 98, "Ġwhat": 99, "I": 100, "Ġlike": 101, "a": 102, "Ġsome": 103, "S": 104, "Ã«": 105, "Ġthem": 106, "Ġyears": 107, "'": 108, "Ġdo": 109, "Ġyour": 110, "Ġ-": 111, "Ġ1": 112, "\"": 113, "Ġif": 114, "Ġcould": 115, "?": 116, "Ġno": 117, "i": 118, "m": 119, "Ġget": 120, "ĠU": 121, "Ġnow": 122, "Ġhim": 123, "Ġback": 124, "ĠBut": 125, "ĠâĢĵ": 126, "Ġmy": 127, "Ġ'": 128, "Ġonly": 129, "Ġthree": 130, ";": 131, "Ġ2": 132, "The": 133, "1": 134, "Ġpercent": 135, "Ġagainst": 136, "Ġbefore": 137, ...

**this really make me confused .
hope for your help,and i will appreciate it!**


Is RoBERTa's pair of sequences tokenizer correct with double </s>

In RoBERTa's build_input_with_special_tokens, the comment says

```
A RoBERTa sequence has the following format:
single sequence: <s> X </s>
pair of sequences: <s> A </s></s> B </s>
```

I find the double `</s></s>` very peculiar. Can you please verify that it should not be `</s><s>` (as a normal XML tag).

Thank you for the amazing work here.


Bert TPU fine-tuning works on Colab but not in GCP

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name): SQUAD
* [ ] my own task or dataset: (give details)

## To Reproduce

I'm trying to fine-tune a BERT model on TPU. It works in Colab but fails when I switch to a paid TPU on GCP.

Steps to reproduce the behavior:
Jupyter notebook code is as follows:

```
[1] model = TFBertModel.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
# works
[2] cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(
    tpu='[My TPU]',
    zone='us-central1-a',
    project='[My Project]'
)
tf.config.experimental_connect_to_cluster(cluster_resolver)
tf.tpu.experimental.initialize_tpu_system(cluster_resolver)
tpu_strategy = tf.distribute.experimental.TPUStrategy(cluster_resolver)
#Also works. Got a bunch of startup messages from the TPU - all good.

[3] with tpu_strategy.scope():
    model = TFBertModel.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
#Generates the error below (long). Same line works in Colab.
```

---------------------------------------------------------------------------
```
NotFoundError                             Traceback (most recent call last)
<ipython-input-14-2cfc1a238903> in <module>
      1 with tpu_strategy.scope():
----> 2     model = TFBertModel.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

~/.local/lib/python3.5/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    309             return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file, allow_missing_keys=True)
    310
--> 311         ret = model(model.dummy_inputs, training=False)  # build the network with dummy inputs
    312
    313         assert os.path.isfile(resolved_archive_file), "Error retrieving file {}".format(resolved_archive_file)

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~/.local/lib/python3.5/site-packages/transformers/modeling_tf_bert.py in call(self, inputs, **kwargs)
    688
    689     def call(self, inputs, **kwargs):
--> 690         outputs = self.bert(inputs, **kwargs)
    691         return outputs
    692

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~/.local/lib/python3.5/site-packages/transformers/modeling_tf_bert.py in call(self, inputs, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, training)
    548
    549         embedding_output = self.embeddings([input_ids, position_ids, token_type_ids, inputs_embeds], training=training)
--> 550         encoder_outputs = self.encoder([embedding_output, extended_attention_mask, head_mask], training=training)
    551
    552         sequence_output = encoder_outputs[0]

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~/.local/lib/python3.5/site-packages/transformers/modeling_tf_bert.py in call(self, inputs, training)
    365                 all_hidden_states = all_hidden_states + (hidden_states,)
    366
--> 367             layer_outputs = layer_module([hidden_states, attention_mask, head_mask[i]], training=training)
    368             hidden_states = layer_outputs[0]
    369

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~/.local/lib/python3.5/site-packages/transformers/modeling_tf_bert.py in call(self, inputs, training)
    341         hidden_states, attention_mask, head_mask = inputs
    342
--> 343         attention_outputs = self.attention([hidden_states, attention_mask, head_mask], training=training)
    344         attention_output = attention_outputs[0]
    345         intermediate_output = self.intermediate(attention_output)

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~/.local/lib/python3.5/site-packages/transformers/modeling_tf_bert.py in call(self, inputs, training)
    290         input_tensor, attention_mask, head_mask = inputs
    291
--> 292         self_outputs = self.self_attention([input_tensor, attention_mask, head_mask], training=training)
    293         attention_output = self.dense_output([self_outputs[0], input_tensor], training=training)
    294         outputs = (attention_output,) + self_outputs[1:]  # add attentions if we output them

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~/.local/lib/python3.5/site-packages/transformers/modeling_tf_bert.py in call(self, inputs, training)
    222
    223         batch_size = shape_list(hidden_states)[0]
--> 224         mixed_query_layer = self.query(hidden_states)
    225         mixed_key_layer = self.key(hidden_states)
    226         mixed_value_layer = self.value(hidden_states)

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/keras/layers/core.py in call(self, inputs)
   1142         outputs = gen_math_ops.mat_mul(inputs, self.kernel)
   1143     if self.use_bias:
-> 1144       outputs = nn.bias_add(outputs, self.bias)
   1145     if self.activation is not None:
   1146       return self.activation(outputs)  # pylint: disable=not-callable

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/nn_ops.py in bias_add(value, bias, data_format, name)
   2756     else:
   2757       return gen_nn_ops.bias_add(
-> 2758           value, bias, data_format=data_format, name=name)
   2759
   2760

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/gen_nn_ops.py in bias_add(value, bias, data_format, name)
    675       try:
    676         return bias_add_eager_fallback(
--> 677             value, bias, data_format=data_format, name=name, ctx=_ctx)
    678       except _core._SymbolicException:
    679         pass  # Add nodes to the TensorFlow graph.

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/gen_nn_ops.py in bias_add_eager_fallback(value, bias, data_format, name, ctx)
    703     data_format = "NHWC"
    704   data_format = _execute.make_str(data_format, "data_format")
--> 705   _attr_T, _inputs_T = _execute.args_to_matching_eager([value, bias], ctx)
    706   (value, bias) = _inputs_T
    707   _inputs_flat = [value, bias]

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/eager/execute.py in args_to_matching_eager(l, ctx, default_dtype)
    265         dtype = ret[-1].dtype
    266   else:
--> 267     ret = [ops.convert_to_tensor(t, dtype, ctx=ctx) for t in l]
    268
    269   # TODO(slebedev): consider removing this as it leaks a Keras concept.

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/eager/execute.py in <listcomp>(.0)
    265         dtype = ret[-1].dtype
    266   else:
--> 267     ret = [ops.convert_to_tensor(t, dtype, ctx=ctx) for t in l]
    268
    269   # TODO(slebedev): consider removing this as it leaks a Keras concept.

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/framework/ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types)
   1312
   1313     if ret is None:
-> 1314       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
   1315
   1316     if ret is NotImplemented:

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/distribute/values.py in _tensor_conversion_mirrored(var, dtype, name, as_ref)
   1174 # allowing instances of the class to be used as tensors.
   1175 def _tensor_conversion_mirrored(var, dtype=None, name=None, as_ref=False):
-> 1176   return var._dense_var_to_tensor(dtype=dtype, name=name, as_ref=as_ref)  # pylint: disable=protected-access
   1177
   1178

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/distribute/values.py in _dense_var_to_tensor(self, dtype, name, as_ref)
    908     if _enclosing_tpu_context() is None:
    909       return super(TPUVariableMixin, self)._dense_var_to_tensor(
--> 910           dtype=dtype, name=name, as_ref=as_ref)
    911     # pylint: enable=protected-access
    912     elif dtype is not None and dtype != self.dtype:

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/distribute/values.py in _dense_var_to_tensor(self, dtype, name, as_ref)
   1164     assert not as_ref
   1165     return ops.convert_to_tensor(
-> 1166         self.get(), dtype=dtype, name=name, as_ref=as_ref)
   1167
   1168   def _clone_with_new_values(self, new_values):

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/distribute/values.py in get(self, device)
    835   def get(self, device=None):
    836     if (_enclosing_tpu_context() is None) or (device is not None):
--> 837       return super(TPUVariableMixin, self).get(device=device)
    838     else:
    839       raise NotImplementedError(

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/distribute/values.py in get(self, device)
    320         device = distribute_lib.get_update_device()
    321         if device is None:
--> 322           return self._get_cross_replica()
    323     device = device_util.canonicalize(device)
    324     return self._device_map.select_for_device(self._values, device)

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/distribute/values.py in _get_cross_replica(self)
   1136     replica_id = self._device_map.replica_for_device(device)
   1137     if replica_id is None:
-> 1138       return array_ops.identity(self.primary)
   1139     return array_ops.identity(self._values[replica_id])
   1140

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/util/dispatch.py in wrapper(*args, **kwargs)
    178     """Call target, and fall back on dispatchers if there is a TypeError."""
    179     try:
--> 180       return target(*args, **kwargs)
    181     except (TypeError, ValueError):
    182       # Note: convert_to_eager_tensor currently raises a ValueError, not a

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/array_ops.py in identity(input, name)
    265     # variables. Variables have correct handle data when graph building.
    266     input = ops.convert_to_tensor(input)
--> 267   ret = gen_array_ops.identity(input, name=name)
    268   # Propagate handle data for happier shape inference for resource variables.
    269   if hasattr(input, "_handle_data"):

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/gen_array_ops.py in identity(input, name)
   3824         pass  # Add nodes to the TensorFlow graph.
   3825     except _core._NotOkStatusException as e:
-> 3826       _ops.raise_from_not_ok_status(e, name)
   3827   # Add nodes to the TensorFlow graph.
   3828   _, _, _op, _outputs = _op_def_library._apply_op_helper(

/usr/local/lib/python3.5/dist-packages/tensorflow_core/python/framework/ops.py in raise_from_not_ok_status(e, name)
   6604   message = e.message + (" name: " + name if name is not None else "")
   6605   # pylint: disable=protected-access
-> 6606   six.raise_from(core._status_to_exception(e.code, message), None)
   6607   # pylint: enable=protected-access
   6608

/usr/local/lib/python3.5/dist-packages/six.py in raise_from(value, from_value)

NotFoundError: '_MklMatMul' is neither a type of a primitive operation nor a name of a function registered in binary running on n-aa2fcfb7-w-0. One possible root cause is the client and server binaries are not built with the same version. Please make sure the operation or function is registered in the binary running in this process. [Op:Identity]
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Would like the model to start...
<!-- A clear and concise description of what you expected to happen. -->

## Environment
GCP AI Notebook.  https://console.cloud.google.com/ai-platform/notebooks

* OS:
* Python version: 3.5
* PyTorch version: Tensorflow 2.1.0
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU ? Using TPU
* Distributed or parallel setup ? Distributed
* Any other relevant information:

## Additional context
Lots of config detail in the code above.

<!-- Add any other context about the problem here. -->


Why isn't BERT doing wordpiece tokenization?

My code is
```
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = '''why isn't my card working'''
encoded = tokenizer.encode(text, add_special_tokens=False)
text_tokenized = tokenizer.decode(encoded, clean_up_tokenization_spaces=False)
print(text_tokenized)
```
and the output result (the tokenization) is
```
why isn ' t my card working
```

But this isn't the wordpiece nokenization BERT should be using. E.g., `working` should be tokenized as `work ##ing`. Is there anything wrong with my code? And will the fact we are not using wordpiece tokenization decrease BERT's performance?


[run_lm_finetuning] Train from scratch

Ability to train a model from scratch, rather than finetune a pretrained one.


Add lower bound to tqdm for tqdm.auto

- It appears that `tqdm` only introduced `tqdm.auto` in 4.27.
- See https://github.com/tqdm/tqdm/releases/tag/v4.27.0.
- Without a lower bound I received an error when importing `transformers` in an environment where I already had `tqdm` installed.
- `transformers` version:
```
$ pip list | grep transformers
transformers                  2.3.0
```
- repro:
```
$ pip install tqdm==4.23
$ ipython
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import transformers
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-279c49635b32> in <module>()
----> 1 import transformers

~/anaconda3/envs/allennlp/lib/python3.6/site-packages/transformers/__init__.py in <module>()
     18
     19 # Files and general utilities
---> 20 from .file_utils import (TRANSFORMERS_CACHE, PYTORCH_TRANSFORMERS_CACHE, PYTORCH_PRETRAINED_BERT_CACHE,
     21                          cached_path, add_start_docstrings, add_end_docstrings,
     22                          WEIGHTS_NAME, TF2_WEIGHTS_NAME, TF_WEIGHTS_NAME, CONFIG_NAME, MODEL_CARD_NAME,

~/anaconda3/envs/allennlp/lib/python3.6/site-packages/transformers/file_utils.py in <module>()
     22 from botocore.exceptions import ClientError
     23 import requests
---> 24 from tqdm.auto import tqdm
     25 from contextlib import contextmanager
     26 from . import __version__

ModuleNotFoundError: No module named 'tqdm.auto'
```


Finetuning ALBERT using examples/run_lm_finetuning.py

## 🚀 Feature

The current run_lm_finetuning.py script seems to not have ALBERT added. We should be able to finetune ALBERT in the same way we do to other models in your library.


Bert perform way worse than simple LSTM+Glove

Hi, I am doing a very straightforward entity classification task, but Bert is not giving a good result. I am wondering if there is something wrong with my code.

My task is give a sentence and an entity boundary in that sentence, I predict entity type.

Here is my code to prepare input data. The basic idea is I have a sentence batch `batch_data`, then  I use tokenizer to encode that sentence batch to `s_encoded`. Then to pad the encoded ids so that each encoded id list can have same length, I pad them with 0's according the longest sentence in that batch (`max_len`).  Because I need to extract feature for the entity, so I use `entity_idx` to keep track of the entity boundary after it's tokenized.

Later, I just use `s_tensor` and `attn_mask_list` as input for BERT, and use `entity_idx` list to extract BERT feature for the entity.

Is there anything wrong with this part? Thank you!

```
s_bat = [x.tokens for x in batch_data]
s_encoded = tokenizer.batch_encode_plus([' '.join(x) for x in s_bat], add_special_tokens=True)
max_len = max([len(x) for x in s_encoded['input_ids']])

start_ids = [x.mention_start for x in batch_data]
ids_list = []
entity_idx = []
attn_mask_list = []
for i in range(len(s_bat)):
    ids = s_encoded['input_ids'][i]
    ids.extend([0 for _ in range(max_len-len(ids))])
    attn = s_encoded['attention_mask'][i]
    attn.extend([0 for _ in range(max_len-len(attn))])
    ids_list.append(ids)
    attn_mask_list.append(attn)

    m_start_idx = len(self.tokenizer.encode(' '.join(s_bat[i][:start_ids[i]]))) if start_ids[i]>0 else 0
    entity_idx.append([1+m_start_idx, 1+m_start_idx+len(m_encoded['input_ids'][i])]) #consider start special token
s_tensor = torch.LongTensor(ids_list)
attn_mask_list = torch.Tensor(attn_mask_list)

```


question about tokenizer changes original sequence length

Hi, I am working on an entity classification task where I know the entity boundary and the context.
when I use tokenizer to encode the entire sequence, some token got split up to word pieces that will change the original length of the sequence.  I want to extract the states only for the entity, but since the seq length changed, does that mean I need to recalcuate the boundary? Is there a way to automatically do this? Thanks!


Optionally convert output of FeatureExtraction pipeline to list

Core: Pipeline

Internals of the library; Pipeline.

What is the purpose of .tolist in FeatureExtraction pipeline?
Why is it called? Is this because of some kind of compatibility issue?

If someone needs to use __call__ a lot, it only slows it down. I've tried subclassing FeatureExtractionPipeline, but it's very ugly since then I can't just use it with pipelines.pipeline.


Fix glue processor failing on tf datasets


Fix typo in examples/run_squad.py

Rul -> Run


Architectures for Dialogue

## ❓ Questions & Help
Hi 👋
I'm trying to build a dialogue system which should reply based on a history, memory (which is represented as a string) and a confidence if the memory content is correct and should be used.

Here two examples:

- history: _Hi_
memory: _name: Max_
confidence: _0.2_
=> expected output: _Hi, what's your name?_

- history: _Hi_
memory: _name: Max_
confidence: _0.9_
=> expected output: _Hi Max_

First of all, are there some best practices how to encode non textual input or inject such information into a model?

I already trained a Bert2Bert model which is not really performing very well. The generated response seems to be not conditioned on the encoder output.

Are there any recommendations what to try next?

Many thanks in advance for your hints!


Model upload and sharing - delete, update, rename....

## 🚀 Feature

It would be great to have the option of deleting, renaming and adding description to the community models. I saw that there are already some errors in the model names which can probably not be fixed because of this missing functionality.

We should have something like:
transformers-cli delete
transformers-cli rename
transformers-cli update

## Motivation

I think that without this options the list will get quite messy in the future. Also as mentioned already in other issues(#2281 and #2520) we should be able to add how we trained the model, what dataset was used etc.


why this implementation didn't apply residual and layer norm?

## ❓ Questions & Help

In ALBERT implementation code `modeling_albert.py`, i can't find applying skip-connection and layer normalizing after multi-head attention layer. I didn't read about this tequnique.
Is there a special reason about it?

One more, i saw the argument `inner_group_num` in `AlbertLayerGroup` class, is it introduced in original ABERT paper?

Thank you for good implementing.


Prediction on NER Tensorflow 2

Hi,
I tried running the implementation of NER on Tensorflow 2. I have a problem doing the prediction. Seems like the label to index are off. Here is some examples:

```
SOCCER B-ORG
- B-ORG
JAPAN B-MISC
GET B-ORG
LUCKY B-MISC
WIN B-ORG
, B-ORG
CHINA B-MISC
IN B-ORG
SURPRISE O
DEFEAT B-ORG
. B-ORG

Nadim B-MISC
Ladki B-PER

AL-AIN I-PER
, B-ORG
United I-PER
Arab I-MISC
Emirates I-MISC
1996-12-06 B-ORG

Japan I-PER
began B-ORG
the B-ORG
defence B-ORG
of B-ORG
their B-ORG
Asian O
Cup I-ORG
title B-ORG
with B-ORG
a B-ORG
lucky B-ORG
2-1 B-ORG
win B-ORG
against B-ORG
Syria I-PER
in B-ORG
a B-ORG
Group B-ORG
C I-ORG
championship B-ORG
match B-ORG
on B-ORG
Friday B-ORG
. B-ORG
```

Anyone has the same problem? Thanks!


solve the exception: [AttributeError: 'bool' object has no attribute 'mean']

modified method simple_accuracy(),
before:
it's (preds == labels).mean()
This will cause an exception[AttributeError: 'bool' object has no attribute 'mean'],
then after update:
change to accuracy_score(labels,preds),
use this method accuracy_score() in package sklearn.metrics


Fix BasicTokenizer to respect `never_split` parameters

`never_split` was not being passed to `_split_on_punc`, causing special tokens to be split apart. Failing test (in first commit) demonstrates the problem.


Quantized model not preserved when imported using from_pretrained()

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
When I import the saved quantized model using `from_pretrained()`, the model's size is inflated to the pre-quantized version.  The model also takes a significant performance hit, both accuracy and time, from the original quantized model.

The tasks I am working on is:
This is the official Pytorch notebook from: https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb

## To Reproduce

Steps to reproduce the behavior:

1.  Run through the notebook fully.
2.  Load the quantized model with from_pretrained()
3.  Run `3.1 Check the model size` and `3.2 Evaluate the inference accuracy and time`

You will see that the size is of the pre-quantized model (>400 MB), and the accuracy AND time take a huge hit from the original quantized model.

## Expected behavior

1.  The quantized model can be loaded in at its original size of <200 MB
2.  The quantized model preserves its accuracy when loaded in
3.  The quantized model preserves its time to run when loaded in

## Environment

Colab


Fix output name

Output variable name `all_hidden_states` found in README is inconsistent with documentation's `hidden_states`: https://huggingface.co/transformers/model_doc/bert.html#bertmodel


CTRL tokenizer has no special tokens to indicate EOS

## 🐛 Bug

The `generate` method from `PreTrainedModel` by default uses index 0 as EOS. This is a problem with CTRL, because its tokenizer has the word `the` mapped to this id.

Actually the CTRL has no special tokens besides UNK:

```
tokenizer = CTRLTokenizer.from_pretrained('ctrl')
tokenizer.special_tokens_map
# {'unk_token': '<unk>'}

tokenizer.convert_ids_to_tokens([0])
# ['the']
```

I believe the CTRL tokenizer should have some special token to use as EOS or PAD, as the other models do.


Model not learning when using albert-base-v2 -- ALBERT

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): AlbertForSequenceClassification

Language I am using the model on (English, Chinese....): English

The problem arise when using:
When I use `albert-base-v2` instead of `albert-base-v1` for the model and tokenizer, the model does not learn during training.

The tasks I am working on is:
https://colab.research.google.com/drive/1Y4o3jh3ZH70tl6mCd76vz_IxX23biCPP

## To Reproduce

Steps to reproduce the behavior:

1. Open the colab notebook I have referenced above
2. Change the word Bert to Albert in necessary places
3. Run 4.3. Training Loop, you will see that the model does not learn

## Expected behavior

I would expect the model to learn on the given task.

## Environment

Colab

## Additional context

I can not pinpoint exactly what the problem is, but it almost seems like the data that is being fed to the model is not understood.  If I use the AlbertTokenizer for a BertForSequenceClassification model, which the BERT model would obviously not understand, the same behavior is exhibited.


fix #2549

closes #2549
proposed solution for unsupported operand type error in tokenizer.batch_encode_plus