internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
Deprecation warning due to invalid escape sequences in Python 3.7
# 🐛 Bug

## To reproduce

Deprecation warnings are raised due to invalid escape sequences. This can be fixed by using raw strings or escaping the literals.

Steps to reproduce the behavior:

```
 find . -iname '*.py' | grep -v example | xargs -P 4 -I{} python3.8 -Wall -m py_compile {} 
./src/transformers/tokenization_transfo_xl.py:123: DeprecationWarning: invalid escape sequence \:
  self.punctuation_symbols = '!"#$%&()*+,-./\:;<=>?@[\\]^_`{|}~'  # noqa: W605
./src/transformers/tokenization_transfo_xl.py:150: DeprecationWarning: invalid escape sequence \s
  look_ahead_to_match_all_except_space = "(?=[^\s])"  # noqa: W605
```

## Expected behavior

No warnings

## Environment info
     
- `transformers` version: master branch
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:



Discussion:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/src/transformers/tokenization_transfo_xl.py b/src/transformers/tokenization_transfo_xl.py
--- a/src/transformers/tokenization_transfo_xl.py
+++ b/src/transformers/tokenization_transfo_xl.py
@@ -113,7 +113,7 @@ def __init__(
         self.delimiter = delimiter
         self.vocab_file = vocab_file
         self.never_split = never_split
-        self.punctuation_symbols = '!"#$%&()*+,-./\:;<=>?@[\\]^_`{|}~'  # noqa: W605
+        self.punctuation_symbols = '!"#$%&()*+,-./\\:;<=>?@[\\]^_`{|}~'
         self.punction_without_space_before_pattern = re.compile(r"[^\s][{}]".format(self.punctuation_symbols))
         self.punctuation_with_space_around_pattern = self._compile_space_around_punctuation_pattern()
 
@@ -140,7 +140,7 @@ def __init__(
 
     def _compile_space_around_punctuation_pattern(self):
         look_ahead_for_special_token = "(?=[{}])".format(self.punctuation_symbols)
-        look_ahead_to_match_all_except_space = "(?=[^\s])"  # noqa: W605
+        look_ahead_to_match_all_except_space = r"(?=[^\s])"
         return re.compile(r"" + look_ahead_for_special_token + look_ahead_to_match_all_except_space)
 
     def count_file(self, path, verbose=False, add_eos=False):


Test patch:
Repository: huggingface/transformers

Issue:
Perform evaluation on HANS with Trainer (like GLUE example)
Current [HANS](https://github.com/huggingface/transformers/tree/master/examples/adversarial) evaluation implementation is carried out in the old way. It'd be good to do it in the same manner as other examples are implemented now with the Trainer class. 



Discussion:
Maybe @sgugger would be interested in taking a look at this
There is an evaluation issue that I faced with HANS evaluation. Issue [here](https://github.com/huggingface/transformers/issues/4766). I tried to run in the exact same manner as listed in `examples/adversarial`. But the results obtained seem skewed for some reason. If @sgugger can point some possible reason out, I can work on it and send a PR. Thanks.

Resolving patch:
diff --git a/examples/adversarial/run_hans.py b/examples/adversarial/run_hans.py
--- a/examples/adversarial/run_hans.py
+++ b/examples/adversarial/run_hans.py
@@ -216,6 +216,7 @@ def main():
         output_eval_file = os.path.join(training_args.output_dir, "hans_predictions.txt")
         if trainer.is_world_master():
             with open(output_eval_file, "w") as writer:
+                writer.write("pairID,gold_label\n")
                 for pid, pred in zip(pair_ids, preds):
                     writer.write("ex" + str(pid) + "," + label_list[int(pred)] + "\n")
 


Test patch:
Repository: huggingface/transformers

Issue:
Better hyperparameter tensorboard logging in Trainer.
# 🚀 Feature request

`Trainer` should write `train_batch_size` and `eval_batch_size` to tensorboard. Currently only `per_device_` batch sizes are logged as hyperparamters, which means unless you know how many gpus you trained on (which is also not logged), you can't know the actual batch sizes used in training.

## Your contribution
 
Submitted a PR: [Adds train_batch_size, eval_batch_size, and n_gpu to to_sanitized_dict output for logging.](https://github.com/huggingface/transformers/pull/5331#issue-440857548)



Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -327,7 +327,7 @@ def _setup_wandb(self):
             logger.info(
                 'Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"'
             )
-            wandb.init(project=os.getenv("WANDB_PROJECT", "huggingface"), config=vars(self.args))
+            wandb.init(project=os.getenv("WANDB_PROJECT", "huggingface"), config=self.args.to_sanitized_dict())
             # keep track of model topology and gradients, unsupported on TPU
             if not is_torch_tpu_available() and os.getenv("WANDB_WATCH") != "false":
                 wandb.watch(
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -211,7 +211,10 @@ def to_sanitized_dict(self) -> Dict[str, Any]:
         Sanitized serialization to use with TensorBoard’s hparams
         """
         d = dataclasses.asdict(self)
+        d = {**d, **{"train_batch_size": self.train_batch_size, "eval_batch_size": self.eval_batch_size}}
+
         valid_types = [bool, int, float, str]
         if is_torch_available():
             valid_types.append(torch.Tensor)
+
         return {k: v if type(v) in valid_types else str(v) for k, v in d.items()}


Test patch:
Repository: huggingface/transformers

Issue:
Unexpected behavior encoding token_type_ids in GPT models
# 🐛 Bug

## Information

When `token_type_ids` are passed into the `GPT2Model` and subclasses, they're encoded using the `nn.Embedding` lookup table as the vocabulary ([line](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_gpt2.py#L477)). Because the token type ids output by `encode_plus()` are `{0,1}`, this seems to create unexpected behavior by using the same embedding vectors to represent both word-tokens 0 and 1 in the vocabulary and the very distinct token type ids.

## Expected Behavior

Either:
1. instead of returning `token_type_ids` consisting of indices 0 and 1, extend the vocabulary by 2 and use those two ids as the `token_type_ids` in `encode_plus()`, or
2. addition of a separate `nn.Embedding` matrix for token_type_ids [here](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_gpt2.py#L352) (such as [in BERTModel](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_bert.py#L153))
3. discourage or throw a warning when `token_type_ids` are passed to GPTModel instances



Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_gpt2.py b/src/transformers/tokenization_gpt2.py
--- a/src/transformers/tokenization_gpt2.py
+++ b/src/transformers/tokenization_gpt2.py
@@ -137,6 +137,7 @@ class GPT2Tokenizer(PreTrainedTokenizer):
     vocab_files_names = VOCAB_FILES_NAMES
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
     max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["attention_mask"]
 
     def __init__(
         self,
@@ -330,6 +331,7 @@ class GPT2TokenizerFast(PreTrainedTokenizerFast):
     vocab_files_names = VOCAB_FILES_NAMES
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
     max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["attention_mask"]
 
     def __init__(
         self,
diff --git a/src/transformers/tokenization_openai.py b/src/transformers/tokenization_openai.py
--- a/src/transformers/tokenization_openai.py
+++ b/src/transformers/tokenization_openai.py
@@ -96,6 +96,7 @@ class OpenAIGPTTokenizer(PreTrainedTokenizer):
     vocab_files_names = VOCAB_FILES_NAMES
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
     max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["attention_mask"]
 
     def __init__(self, vocab_file, merges_file, unk_token="<unk>", **kwargs):
         super().__init__(unk_token=unk_token, **kwargs)
@@ -261,6 +262,7 @@ class OpenAIGPTTokenizerFast(PreTrainedTokenizerFast):
     vocab_files_names = VOCAB_FILES_NAMES
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
     max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["attention_mask"]
 
     def __init__(self, vocab_file, merges_file, unk_token="<unk>", **kwargs):
         kwargs.setdefault("unk_token", unk_token)


Test patch:
Repository: huggingface/transformers

Issue:
incorrect typehint for PreTrainedTokenizer.convert_ids_to_tokens() return value
# 🐛 Bug

## Information

The return value of `tokenization_utils.PreTrainedTokenizer.convert_ids_to_tokens()` is declared as `Union[int, List[int]]` while the method clearly returns a `Union[str, List[str]]` object type.

Would you like me to open a PR? ;)

Thanks for the great work!
Andreas

PS: Excuse me for skipping my system specs since this is a rather small issue and I'm just a bit annoyed by PyCharm's inspection warning on this ;)


Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -649,7 +649,7 @@ def get_special_tokens_mask(
 
     def convert_ids_to_tokens(
         self, ids: Union[int, List[int]], skip_special_tokens: bool = False
-    ) -> Union[int, List[int]]:
+    ) -> Union[str, List[str]]:
         """ Converts a single index or a sequence of indices (integers) in a token "
             (resp.) a sequence of tokens (str), using the vocabulary and added tokens.
 


Test patch:
Repository: huggingface/transformers

Issue:
TextGenerationPipeline breaks when used with device=0
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): model-agnostic (breaks with GPT2 and XLNet)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
[x] my own modified scripts: (give details below)

The tasks I am working on is:
[x] my own task or dataset: plain old language generation

## To reproduce

Steps to reproduce the behavior:

```
#!/usr/bin/env python3
import random
from transformers import pipeline, XLNetLMHeadModel
import torch
import time
random.seed(0)
torch.manual_seed(0)

generator = pipeline("text-generation", model="xlnet-base-cased", tokenizer="xlnet-base-cased", device=0)
output_to_check = generator("Today is a beautiful day and I, ", offset=offset, do_sample=True, top_k=50, max_len=100)
```

## Expected behavior

What should happen : text generation
What actually happens : 

```
Traceback (most recent call last):
  File "/home/teven/dev_transformers/perso/transformers/generation_script.py", line 15, in <module>
    output_to_check = generator("Today is a beautiful day and I, ", offset=offset, do_sample=True, top_k=50, max_len=100)
  File "/home/teven/dev_transformers/perso/transformers/src/transformers/pipelines.py", line 692, in __call__
    generated_sequence = generated_sequence.numpy().tolist()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
```

Just missing a conversion before the `.numpy()` call

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-62-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No



Discussion:
If that's not the case, we should make sure that the pipelines run on GPU in the GPU CI. (fast and slow), to catch things like this.

Resolving patch:
diff --git a/src/transformers/pipelines.py b/src/transformers/pipelines.py
--- a/src/transformers/pipelines.py
+++ b/src/transformers/pipelines.py
@@ -689,6 +689,8 @@ def __call__(
 
             result = []
             for generated_sequence in output_sequences:
+                if self.framework == "pt" and generated_sequence is not None:
+                    generated_sequence = generated_sequence.cpu()
                 generated_sequence = generated_sequence.numpy().tolist()
                 record = {}
                 if return_tensors:


Test patch:
Repository: huggingface/transformers

Issue:
T5 has no module  ```torch_xla``` when using T5 fine-tuned on SQUADv2
# 🐛 Bug

## Information

I get this error:
``` ModuleNotFoundError: No module named 'torch_xla'```
Full error message:

```
      2 
      3 tokenizer = AutoTokenizer.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
----> 4 model = AutoModelWithLMHead.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
      5 
      6 def get_answer(question, context):

1 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    796 
    797         if hasattr(config, "xla_device") and config.xla_device:
--> 798             import torch_xla.core.xla_model as xm
    799 
    800             model = xm.send_cpu_data_to_device(model, xm.xla_device())

ModuleNotFoundError: No module named 'torch_xla'
```

## To reproduce
```python
from transformers import AutoModelWithLMHead, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mrm8488/t5-base-finetuned-squadv2")
model = AutoModelWithLMHead.from_pretrained("mrm8488/t5-base-finetuned-squadv2")

def get_answer(question, context):
  input_text = "question: %s  context: %s 
" % (question, context)
  features = tokenizer([input_text], return_tensors='pt')

  output = model.generate(input_ids=features['input_ids'], 
               attention_mask=features['attention_mask'])

  return tokenizer.decode(output[0])

context = "Manuel have created RuPERTa-base with the support of HF-Transformers and Google"
question = "Who has supported Manuel?"

get_answer(question, context)
```
I used this example code a few weeks ago and had no problem...


Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_utils.py b/src/transformers/modeling_utils.py
--- a/src/transformers/modeling_utils.py
+++ b/src/transformers/modeling_utils.py
@@ -34,6 +34,7 @@
     cached_path,
     hf_bucket_url,
     is_remote_url,
+    is_torch_tpu_available,
 )
 from .generation_utils import GenerationMixin
 
@@ -794,7 +795,7 @@ def load(module: nn.Module, prefix=""):
             }
             return model, loading_info
 
-        if hasattr(config, "xla_device") and config.xla_device:
+        if hasattr(config, "xla_device") and config.xla_device and is_torch_tpu_available():
             import torch_xla.core.xla_model as xm
 
             model = xm.send_cpu_data_to_device(model, xm.xla_device())


Test patch:
Repository: huggingface/transformers

Issue:
tokenizer_bert.py not call _clean_text?
for transformers/src/transformers/tokenization_bert.py, there is a function called _clean_text.
But seems this function is not be called at all?

In google bert(https://github.com/google-research/bert/blob/master/tokenization.py) there exists a same function and that function has been called at the beginning of the tokenization.


Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_bert.py b/src/transformers/tokenization_bert.py
--- a/src/transformers/tokenization_bert.py
+++ b/src/transformers/tokenization_bert.py
@@ -398,6 +398,7 @@ def tokenize(self, text, never_split=None):
         """
         # union() returns a new set by concatenating the two sets.
         never_split = self.never_split.union(set(never_split)) if never_split else self.never_split
+        text = self._clean_text(text)
 
         # This was added on November 1st, 2018 for the multilingual and Chinese
         # models. This is also applied to the English models now, but it doesn't


Test patch:
diff --git a/tests/test_tokenization_bert.py b/tests/test_tokenization_bert.py
--- a/tests/test_tokenization_bert.py
+++ b/tests/test_tokenization_bert.py
@@ -222,6 +222,17 @@ def test_is_punctuation(self):
         self.assertFalse(_is_punctuation("A"))
         self.assertFalse(_is_punctuation(" "))
 
+    def test_clean_text(self):
+        tokenizer = self.get_tokenizer()
+        rust_tokenizer = self.get_rust_tokenizer()
+
+        # Example taken from the issue https://github.com/huggingface/tokenizers/issues/340
+        self.assertListEqual([tokenizer.tokenize(t) for t in ["Test", "\xad", "test"]], [["[UNK]"], [], ["[UNK]"]])
+
+        self.assertListEqual(
+            [rust_tokenizer.tokenize(t) for t in ["Test", "\xad", "test"]], [["[UNK]"], [], ["[UNK]"]]
+        )
+
     @slow
     def test_sequence_builders(self):
         tokenizer = self.tokenizer_class.from_pretrained("bert-base-uncased")
Repository: huggingface/transformers

Issue:
Transformer-XL: no mems are return when using 'return_tuple'
# 🐛 Bug

## Information
The forward pass of the `TransfoXLLMHeadModel` returns no `mems` when using `return_tuple=True`.

Model I am using: Transformer-XL

Language I am using the model on: English

The problem arises when using:
* [x] my own modified scripts: (give details below)


## To reproduce

```Python
from transformers import TransfoXLLMHeadModel, TransfoXLTokenizer

model = TransfoXLLMHeadModel.from_pretrained("transfo-xl-wt103")
model.train()
tokenizer = TransfoXLTokenizer.from_pretrained("transfo-xl-wt103")

encoded = tokenizer("Max is walking the dog in the streets", return_tensors='pt')
outputs = model(input_ids=encoded['input_ids'], mems=None, labels=encoded['input_ids'], return_tuple=True)
loss, _, mems = outputs

print(loss.size())
print(len(mems))      # should be 18 due to the 18 layers
```
Output:
```
Traceback (most recent call last):
  File "user/script.py", line 10, in <module>
    loss, _, mems = outputs
ValueError: not enough values to unpack (expected 3, got 2)
```


## Expected behavior
Output:
```
torch.Size([1, 7])
18
```

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No



Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_transfo_xl.py b/src/transformers/modeling_transfo_xl.py
--- a/src/transformers/modeling_transfo_xl.py
+++ b/src/transformers/modeling_transfo_xl.py
@@ -1045,14 +1045,13 @@ def forward(
 
         last_hidden = transformer_outputs[0]
         pred_hid = last_hidden[:, -tgt_len:]
-        outputs = transformer_outputs[1:]
 
         softmax_output = self.crit(pred_hid, labels)
         prediction_scores = softmax_output.view(bsz, tgt_len, -1) if labels is None else ()
         loss = softmax_output.view(bsz, tgt_len - 1) if labels is not None else None
 
         if return_tuple:
-            output = (prediction_scores,) + outputs[1:]
+            output = (prediction_scores,) + transformer_outputs[1:]
             return ((loss,) + output) if loss is not None else output
 
         return TransfoXLLMHeadModelOutput(


Test patch:
Repository: huggingface/transformers

Issue:
i dont know what Tranier`s Dataset is.
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
![image](https://user-images.githubusercontent.com/32102558/88669501-8ccdba80-d116-11ea-8b29-8dda1a61ce42.png)
![image](https://user-images.githubusercontent.com/32102558/88669549-9bb46d00-d116-11ea-840c-ff661b0a2b0d.png)
 i thought its my customer dataset goes wrong, i dont konw what dataset it should return. the Trainer receive what dataset.
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Discussion:

Resolving patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -5,6 +5,7 @@
 from torch.nn.utils.rnn import pad_sequence
 
 from ..tokenization_utils import PreTrainedTokenizer
+from ..tokenization_utils_base import BatchEncoding
 
 
 InputDataClass = NewType("InputDataClass", Any)
@@ -33,7 +34,7 @@ def default_data_collator(features: List[InputDataClass]) -> Dict[str, torch.Ten
     # have the same attributes.
     # So we will look at the first element as a proxy for what attributes exist
     # on the whole batch.
-    if not isinstance(features[0], dict):
+    if not isinstance(features[0], (dict, BatchEncoding)):
         features = [vars(f) for f in features]
 
     first = features[0]
@@ -78,7 +79,7 @@ class DataCollatorForLanguageModeling:
     mlm_probability: float = 0.15
 
     def __call__(self, examples: List[Union[torch.Tensor, Dict[str, torch.Tensor]]]) -> Dict[str, torch.Tensor]:
-        if isinstance(examples[0], dict):
+        if isinstance(examples[0], (dict, BatchEncoding)):
             examples = [e["input_ids"] for e in examples]
         batch = self._tensorize_batch(examples)
         if self.mlm:
@@ -151,7 +152,7 @@ class DataCollatorForPermutationLanguageModeling:
     max_span_length: int = 5  # maximum length of a span of masked tokens
 
     def __call__(self, examples: List[Union[torch.Tensor, Dict[str, torch.Tensor]]]) -> Dict[str, torch.Tensor]:
-        if isinstance(examples[0], dict):
+        if isinstance(examples[0], (dict, BatchEncoding)):
             examples = [e["input_ids"] for e in examples]
         batch = self._tensorize_batch(examples)
         inputs, perm_mask, target_mapping, labels = self.mask_tokens(batch)


Test patch: