internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
Logic issue with evaluating cased models in `run_squad.py`
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details) `run_squad.py` with cased models
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) squad
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Finetune a cased model with `--do_train` and `--do_eval` (the latter is optional)
2. Use `--do_eval` to make predictions.

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

At evaluation time, the tokenizer should also be cased, but because it's loading from a path and not using a model name, the `from_pretrained` method in `BertTokenizer` fails to identify casing information, and the `BasicTokenizer` defaults to uncased (`do_lower_case=True`).

## Environment

(most of this is probably not relevant anyway)

* OS: Ubuntu 16.04
* Python version: 3.6.8
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): master
* Using GPU ? Yes
* Distributed of parallel setup ? DataParallel
* Any other relevant information:

## Additional context


One solution is to add `do_lower_case=args.do_lower_case` in the kwargs here:
https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_squad.py#L501


Discussion:
Thanks for the bug report, will look into it.
Looks good to me, do you want to push a PR to fix this as you proposed @qipeng?

Resolving patch:
diff --git a/examples/run_glue.py b/examples/run_glue.py
--- a/examples/run_glue.py
+++ b/examples/run_glue.py
@@ -467,7 +467,7 @@ def main():
 
         # Load a trained model and vocabulary that you have fine-tuned
         model = model_class.from_pretrained(args.output_dir)
-        tokenizer = tokenizer_class.from_pretrained(args.output_dir)
+        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
         model.to(args.device)
 
 
diff --git a/examples/run_squad.py b/examples/run_squad.py
--- a/examples/run_squad.py
+++ b/examples/run_squad.py
@@ -481,7 +481,7 @@ def main():
 
 
     # Save the trained model and the tokenizer
-    if args.do_train and args.local_rank == -1 or torch.distributed.get_rank() == 0:
+    if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
         # Create output directory if needed
         if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]:
             os.makedirs(args.output_dir)
@@ -498,7 +498,7 @@ def main():
 
         # Load a trained model and vocabulary that you have fine-tuned
         model = model_class.from_pretrained(args.output_dir)
-        tokenizer = tokenizer_class.from_pretrained(args.output_dir)
+        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
         model.to(args.device)
 
 


Test patch:
Repository: huggingface/transformers

Issue:
No parameter which is presented in documentation
Documentation web page: https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2Model
See `Inputs -> attention_mask`.

There is actually no parameter `attention_mask` in `GPT2Model.forward` method (see https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_gpt2.py#L473)

Of course trying to provide `attention_mask` parameter to model raises an exception:
> TypeError: forward() got an unexpected keyword argument 'attention_mask'

Please either add parameter `attention_mask` to `GPT2Model.forward` or delete it from documentation.

Same for https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2LMHeadModel
and for
https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2DoubleHeadsModel

I've created #1116 in case you want just delete it from documentation.


Discussion:

Resolving patch:
diff --git a/pytorch_transformers/modeling_gpt2.py b/pytorch_transformers/modeling_gpt2.py
--- a/pytorch_transformers/modeling_gpt2.py
+++ b/pytorch_transformers/modeling_gpt2.py
@@ -408,10 +408,6 @@ def init_weights(self, module):
             list of ``torch.FloatTensor`` (one for each layer):
             that contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model
             (see `past` output below). Can be used to speed up sequential decoding.
-        **attention_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, sequence_length)``:
-            Mask to avoid performing attention on padding token indices.
-            Mask values selected in ``[0, 1]``:
-            ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
         **head_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(num_heads,)`` or ``(num_layers, num_heads)``:
             Mask to nullify selected heads of the self-attention modules.
             Mask values selected in ``[0, 1]``:
@@ -642,10 +638,6 @@ class GPT2DoubleHeadsModel(GPT2PreTrainedModel):
             list of ``torch.FloatTensor`` (one for each layer):
             that contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model
             (see `past` output below). Can be used to speed up sequential decoding.
-        **attention_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, num_choices, sequence_length)``:
-            Mask to avoid performing attention on padding token indices.
-            Mask values selected in ``[0, 1]``:
-            ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
         **head_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(num_heads,)`` or ``(num_layers, num_heads)``:
             Mask to nullify selected heads of the self-attention modules.
             Mask values selected in ``[0, 1]``:


Test patch:
Repository: huggingface/transformers

Issue:
Wrong parameter name in documentation
Documentation web page: https://huggingface.co/pytorch-transformers/model_doc/gpt2.html#pytorch_transformers.GPT2DoubleHeadsModel

See `Inputs -> multiple_choice_labels`. There is actually no such parameter in `GPT2DoubleHeadsModel.forward` method. It was renamed to `mc_labels`. Also it is presented two times in documentation which seems to be just a copy past error.

Please change `multiple_choice_labels` to `mc_labels` and delete second occurrence of this parameter in documentation.

Created #1118 to fix documentation. Also you may squash it with fix of #1115


Discussion:

Resolving patch:
diff --git a/pytorch_transformers/modeling_gpt2.py b/pytorch_transformers/modeling_gpt2.py
--- a/pytorch_transformers/modeling_gpt2.py
+++ b/pytorch_transformers/modeling_gpt2.py
@@ -656,14 +656,11 @@ class GPT2DoubleHeadsModel(GPT2PreTrainedModel):
             Indices are selected in ``[-1, 0, ..., config.vocab_size]``
             All labels set to ``-1`` are ignored (masked), the loss is only
             computed for labels in ``[0, ..., config.vocab_size]``
-        **multiple_choice_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size)``:
+        **mc_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size)``:
             Labels for computing the multiple choice classification loss.
             Indices should be in ``[0, ..., num_choices]`` where `num_choices` is the size of the second dimension
             of the input tensors. (see `input_ids` above)
 
-            `multiple_choice_labels`: optional multiple choice labels: ``torch.LongTensor`` of shape [batch_size]
-                with indices selected in [0, ..., num_choices].
-
     Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs:
         **lm_loss**: (`optional`, returned when ``lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``:
             Language modeling loss.


Test patch:
Repository: huggingface/transformers

Issue:
Tons of warnings on use of TransfoXLModel. masked_fill_ input dtype torch.uint8 should be changed to torch.bool
## 🕑 Usage of deprecated behaviour

Using example from documentation web page: https://huggingface.co/pytorch-transformers/model_doc/transformerxl.html#pytorch_transformers.TransfoXLModel
```
import torch
from pytorch_transformers import *

tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLModel.from_pretrained('transfo-xl-wt103')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states, mems = outputs[:2]
```

Get tons of same warnings:
> /pytorch/aten/src/ATen/native/LegacyDefinitions.cpp:14: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead.
![image](https://user-images.githubusercontent.com/9883873/63766644-4b180c80-c8d4-11e9-890c-9da640b5c0e8.png)

Created #1120 to fix it.


Discussion:

Resolving patch:
diff --git a/pytorch_transformers/modeling_transfo_xl.py b/pytorch_transformers/modeling_transfo_xl.py
--- a/pytorch_transformers/modeling_transfo_xl.py
+++ b/pytorch_transformers/modeling_transfo_xl.py
@@ -1142,10 +1142,10 @@ def _forward(self, dec_inp, mems=None, head_mask=None):
             else:
                 mask_shift_len = qlen
             dec_attn_mask = (torch.triu(all_ones, 1+mlen)
-                    + torch.tril(all_ones, -mask_shift_len)).byte()[:, :, None] # -1
+                    + torch.tril(all_ones, -mask_shift_len)).bool()[:, :, None] # -1
         else:
             dec_attn_mask = torch.triu(
-                word_emb.new_ones(qlen, klen), diagonal=1+mlen).byte()[:,:,None]
+                word_emb.new_ones(qlen, klen), diagonal=1+mlen).bool()[:,:,None]
 
         hids = []
         attentions = []


Test patch:
Repository: huggingface/transformers

Issue:
Documentation error in GPT2Tokenizer
Documentation page:
https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2Tokenizer
Code place:
https://github.com/huggingface/transformers/blob/d7d36181fdefdabadc53adf51bed4a2680f5880a/transformers/tokenization_gpt2.py#L112-L113

This phrase:
> Otherwise, this tokenizer encode and decode method will not conserve the absence of a space at the beginning of a string: tokenizer.decode(tokenizer.encode(“Hello”)) = ” Hello”

is **NOT** correct.

Actually:
> tokenizer.decode(tokenizer.encode(“Hello”)) = ”Hello”

Try this example:
```
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
print("'" + tokenizer.decode(tokenizer.encode("Hello")) + "'")
```
Output:
```
'Hello'
```


Discussion:
According to the code (https://github.com/huggingface/transformers/blob/master/transformers/tokenization_gpt2.py#L183) the docstring is incomplete. 

The `add_prefix_space` parameter should be passed to `tokenizer.tokenize()` as well as `tokenizer.encode()` or `tokenizer.decode()`.

As well, the docstring should actually say:

```
Otherwise, this tokenizer encode and decode method will not conserve a space at the beginning of a string: tokenizer.decode(tokenizer.encode(“ Hello”)) = ”Hello”
```

According to #1380, the Roberta/GPT2 tokenizer expects sequences to start with a space.  Simply prepending a space to the input sequence doesn't give the same result because `GPT2Tokenizer` only overrides the internally-used `_tokenizer()` method and `PreTrainedTokenizer`'s `tokenize()` method (which is called by the user) does it's own preprocessing.

Resolving patch:
diff --git a/transformers/tokenization_gpt2.py b/transformers/tokenization_gpt2.py
--- a/transformers/tokenization_gpt2.py
+++ b/transformers/tokenization_gpt2.py
@@ -107,10 +107,10 @@ class GPT2Tokenizer(PreTrainedTokenizer):
     """
     GPT-2 BPE tokenizer. Peculiarities:
         - Byte-level Byte-Pair-Encoding
-        - Requires a space to start the input string => the encoding methods should be called with the
+        - Requires a space to start the input string => the encoding and tokenize methods should be called with the
           ``add_prefix_space`` flag set to ``True``.
-          Otherwise, this tokenizer ``encode`` and ``decode`` method will not conserve
-          the absence of a space at the beginning of a string: `tokenizer.decode(tokenizer.encode("Hello")) = " Hello"`
+          Otherwise, this tokenizer's ``encode``, ``decode``, and ``tokenize`` methods will not conserve
+          the spaces at the beginning of a string: `tokenizer.decode(tokenizer.encode(" Hello")) = "Hello"`
     """
     vocab_files_names = VOCAB_FILES_NAMES
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP


Test patch:
Repository: huggingface/transformers

Issue:
TFBertModel ValueError: Tried to convert 'dims' to a tensor and failed. Error: Cannot convert a partially known TensorShape to a Tensor
The code below was fine when transformers 2.1.1

but after I update to transformers 2.2.0
```
model = TFBertForSequenceClassification.from_pretrained('bert-base-chinese', num_labels=5)
model.summary()

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

model_fit = model.fit(train_input_ids, train_label, 
                      batch_size=8, epochs=1, 
                    validation_data=(validation_input_ids, validation_label)
                   )
```
```
ValueError: Tried to convert 'dims' to a tensor and failed. 
Error: Cannot convert a partially known TensorShape to a Tensor: (None, 512)
```


Discussion:
> The code below was fine when transformers 2.1.1
> 
> but after I update to transformers 2.2.0
> 
> ```
> model = TFBertForSequenceClassification.from_pretrained('bert-base-chinese', num_labels=5)
> model.summary()
> 
> optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
> loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
> metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
> model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
> 
> model_fit = model.fit(train_input_ids, train_label, 
>                       batch_size=8, epochs=1, 
>                     validation_data=(validation_input_ids, validation_label)
>                    )
> ```
> 
> ```
> ValueError: Tried to convert 'dims' to a tensor and failed. 
> Error: Cannot convert a partially known TensorShape to a Tensor: (None, 512)
> ```

The problem seem to cause by  input's shape.

For transformers 2.2.0
I fix it myself by modifying file "transformers/modeling_tf_albert.py"
On line 648, I change:
input_shape = input_ids.shape
Into:
input_shape = tf.shape(input_ids)
Then the problem fixed.

Feel free to leave a comment if it work for you.

Resolving patch:
diff --git a/transformers/modeling_tf_albert.py b/transformers/modeling_tf_albert.py
--- a/transformers/modeling_tf_albert.py
+++ b/transformers/modeling_tf_albert.py
@@ -645,7 +645,7 @@ def call(self, inputs, attention_mask=None, token_type_ids=None, position_ids=No
         if input_ids is not None and inputs_embeds is not None:
             raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
         elif input_ids is not None:
-            input_shape = input_ids.shape
+            input_shape = tf.shape(input_ids)
         elif inputs_embeds is not None:
             input_shape = inputs_embeds.shape[:-1]
         else:


Test patch:
Repository: huggingface/transformers

Issue:
import Error from official example caused by fastprogress
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): ALL

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

V0.2.1 of fastprogress released a couple of days ago seems to cause errors in run_tf_ner.py in the official example.

Traceback (most recent call last):
  File "run_tf_ner.py", line 12, in <module>
    from fastprogress import master_bar, progress_bar
ImportError: cannot import name 'master_bar' from 'fastprogress' (/usr/local/lib/python3.7/dist-packages/fastprogress/__init__.py)

users need to either downgrade: pip3 install fastprogress==0.1.22
or change the code: 
` 
from fastprogress.fastprogress import master_bar, progress_bar
`


## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/examples/run_tf_ner.py b/examples/run_tf_ner.py
--- a/examples/run_tf_ner.py
+++ b/examples/run_tf_ner.py
@@ -9,7 +9,6 @@
 import numpy as np
 import tensorflow as tf
 from absl import app, flags, logging
-from fastprogress import master_bar, progress_bar
 from seqeval import metrics
 
 from transformers import (
@@ -29,6 +28,12 @@
 from utils_ner import convert_examples_to_features, get_labels, read_examples_from_file
 
 
+try:
+    from fastprogress import master_bar, progress_bar
+except ImportError:
+    from fastprogress.fastprogress import master_bar, progress_bar
+
+
 ALL_MODELS = sum(
     (tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, RobertaConfig, DistilBertConfig)), ()
 )


Test patch:
Repository: huggingface/transformers

Issue:
Wrong logic in `batch_encode_plus()`
# 🐛 Bug

I'm trying to use `batch_encode_plus()` with already tokenized ID as input.

So I have a `list of list of int` as input. For example :

```python
x = [[0, 83, 313, 11, 10551, 2278, 16, 2183, 1958, 804, 7, 916, 11, 13933, 982, 4, 286, 68, 5046, 6, 37, 40, 3627, 231, 2697, 9, 1958, 11, 41, 36027, 27894, 1001, 21466, 424, 2233, 4, 2], [0, 20, 291, 212, 13989, 191, 3772, 42, 983, 4, 815, 34, 1714, 8617, 187, 63, 17692, 11, 8008, 4, 993, 864, 549, 1492, 2624, 5391, 9686, 8, 12291, 240, 7, 464, 4, 2]]
```

When I call `batch_encode_plus()` with this input, I receive `AssertionError` :

```
tokenizer.batch_encode_plus(x)
# >>> line 1130, in batch_encode_plus
#                    assert len(ids_or_pair_ids) == 2
```

---

After reviewing the code, it seems there is an error in the logic. Here :

https://github.com/huggingface/transformers/blob/b370cc7e99c5b8c7436154d4694c33b461ea0f08/src/transformers/tokenization_utils.py#L1128-L1137

since I gave only tokenized ID and not a tuple of tokenized ID (not a pair), the logic should bring me to the `else` clause. But instead, I enter the `if` clause, leading to the assertion error.

**It's currently not possible to use `batch_encode_plus()` with already tokenized text / IDs** if the input is not a pair.

---

I think this line :

https://github.com/huggingface/transformers/blob/b370cc7e99c5b8c7436154d4694c33b461ea0f08/src/transformers/tokenization_utils.py#L1129

Should be changed to this :

```python
if isinstance(ids_or_pair_ids, (list, tuple)) and len(ids_or_pair_ids) == 2:
```



Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -1126,8 +1126,7 @@ def get_input_ids(text):
 
         input_ids = []
         for ids_or_pair_ids in batch_text_or_text_pairs:
-            if isinstance(ids_or_pair_ids, (list, tuple)):
-                assert len(ids_or_pair_ids) == 2
+            if isinstance(ids_or_pair_ids, (list, tuple)) and len(ids_or_pair_ids) == 2:
                 ids, pair_ids = ids_or_pair_ids
             else:
                 ids, pair_ids = ids_or_pair_ids, None


Test patch:
Repository: huggingface/transformers

Issue:
TFAlbertMainLayer cannot be imported from the transformers library.
Unlike the TFBertMainLayer class which can be imported from transformers, TFAlbertMainLayer cannot be imported.

Locally I made the changes to init.py:

Import TFAlbertMainLayer from modeling_tf_albert 

Similar to tensorflow version, pytorch version should also be implemented.


Discussion:
This seems like a valid point, what do you think @LysandreJik?

Resolving patch:
diff --git a/src/transformers/__init__.py b/src/transformers/__init__.py
--- a/src/transformers/__init__.py
+++ b/src/transformers/__init__.py
@@ -434,6 +434,7 @@
 
     from .modeling_tf_albert import (
         TFAlbertPreTrainedModel,
+        TFAlbertMainLayer,
         TFAlbertModel,
         TFAlbertForMaskedLM,
         TFAlbertForSequenceClassification,


Test patch:
Repository: huggingface/transformers

Issue:
An Error report about pipeline
# 🐛 Bug

## Information

This may be an easy question, but it has been bothering me all day.

When I run the code: 
nlp = pipeline("question-answering")

It always tells me: 
Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-distilled-squad-modelcard.json' to download model card file.
Creating an empty model card.

If I ignore it and continue to run the rest of the code: 
nlp({
    'question': 'What is the name of the repository ?',
    'context': 'Pipeline have been included in the huggingface/transformers repository'
})

The error will appear:
KeyError: 'token_type_ids'


Discussion:
I have this same issue, but have no problems running:

nlp = pipeline("question-answering")


Note: To install the library, I had to install tokenizers version 0.6.0 separately, git clone the transformers repo and edit the setup.py file before installing as per @dafraile's answer for issue: https://github.com/huggingface/transformers/issues/2831

Update: This error was fixed when I installed tokenizers==0.5.2
I sadly have this issue too with the newest transformers 2.6.0 version.

Tokenizers is at version 0.5.2. But newest version of tokenizers sadly also doesn't work.

And solutions to fix this issue?

Resolving patch:
diff --git a/examples/utils_multiple_choice.py b/examples/utils_multiple_choice.py
--- a/examples/utils_multiple_choice.py
+++ b/examples/utils_multiple_choice.py
@@ -320,7 +320,9 @@ def convert_examples_to_features(
             else:
                 text_b = example.question + " " + ending
 
-            inputs = tokenizer.encode_plus(text_a, text_b, add_special_tokens=True, max_length=max_length,)
+            inputs = tokenizer.encode_plus(
+                text_a, text_b, add_special_tokens=True, max_length=max_length, return_token_type_ids=True
+            )
             if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                 logger.info(
                     "Attention! you are cropping tokens (swag task is ok). "
diff --git a/src/transformers/data/processors/squad.py b/src/transformers/data/processors/squad.py
--- a/src/transformers/data/processors/squad.py
+++ b/src/transformers/data/processors/squad.py
@@ -139,6 +139,7 @@ def squad_convert_example_to_features(example, max_seq_length, doc_stride, max_q
             pad_to_max_length=True,
             stride=max_seq_length - doc_stride - len(truncated_query) - sequence_pair_added_tokens,
             truncation_strategy="only_second" if tokenizer.padding_side == "right" else "only_first",
+            return_token_type_ids=True,
         )
 
         paragraph_len = min(


Test patch: