internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
Error when fine tuning GPT2 on GPU 
An error occur when training with `run_language_modeling.py` file (Ubuntu 18, pytorch with cuda compiled), while the error does not occur on Macbook without the GPU

```
  File "train.py", line 283, in <module>
    main()
  File "train.py", line 247, in main
    trainer.train(model_path=model_path)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 518, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py", line 1017, in __iter__
    for obj in iterable:
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/data/data_collator.py", line 90, in __call__
    labels[labels == self.tokenizer.pad_token_id] = -100
TypeError: eq() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (NoneType)
 * (Number other)
      didn't match because some of the arguments have invalid types: (NoneType)
```

Command and arguments used to run:
```bash
python train.py \
    --output_dir ckpts/prova \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file data.train \
    --num_train_epochs 1 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --save_steps 200
```

Any idea on how to solve this?


Discussion:

Resolving patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -87,7 +87,8 @@ def __call__(self, examples: List[Union[torch.Tensor, Dict[str, torch.Tensor]]])
             return {"input_ids": inputs, "labels": labels}
         else:
             labels = batch.clone().detach()
-            labels[labels == self.tokenizer.pad_token_id] = -100
+            if self.tokenizer.pad_token_id is not None:
+                labels[labels == self.tokenizer.pad_token_id] = -100
             return {"input_ids": batch, "labels": labels}
 
     def _tensorize_batch(self, examples: List[torch.Tensor]) -> torch.Tensor:


Test patch:
Repository: huggingface/transformers

Issue:
Hashing a tokenizer using the 🤗 nlp lib is not deterministic
In the `nlp` library it is common to use a tokenizer on a dataset.
The library takes care of caching the results, so that if you run the tokenization twice, it will reuse the previous results.
To make the caching work, we compute a hash of the tokenizer.

However the `unique_no_split_tokens` attribute of tokenizers is not deterministic, and it makes the hashing return different hashes for the same tokenizer over different sessions.

`unique_no_split_tokens` can be a list like `['[CLS]', '[MASK]', '[PAD]', '[SEP]', '[UNK]']` for example. But it happens that re-loading a tokenizer in another session shuffles the tokens in the list.

For example this code doesn't always return the same output over different sessions:

```python
from transformers import AutoTokenizer

model_name = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(model_name)

print(tokenizer.unique_no_split_tokens)
```

Reproduce on google colab: https://colab.research.google.com/drive/1nyskaLavcTCkXibZBlYX71bkG476uSzz?usp=sharing


Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -207,10 +207,10 @@ def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_to
 
         # Make sure we don't split on any special tokens (even they were already in the vocab before e.g. for Albert)
         if special_tokens:
-            self.unique_no_split_tokens = list(set(self.unique_no_split_tokens).union(set(new_tokens)))
+            self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(new_tokens)))
         else:
             # Or on the newly added tokens
-            self.unique_no_split_tokens = list(set(self.unique_no_split_tokens).union(set(tokens_to_add)))
+            self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(tokens_to_add)))
 
         return len(tokens_to_add)
 


Test patch:
Repository: huggingface/transformers

Issue:
Huggingface create_optimizer method not working
## Environment info
     
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.6
- PyTorch version (GPU?): 1.5.0+cpu (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger
@jplu 

## Information

Model I am using (Bert, XLNet ...): Roberta

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run this code:
```
import tensorflow as tf
from transformers import RobertaConfig, TFRobertaForMaskedLM, create_optimizer
config = RobertaConfig()  
optimizer,lr = create_optimizer(1e-4,1000000,10000,0.1,1e-6,0.01)
training_loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model = TFRobertaForMaskedLM(config)
model.compile(optimizer=optimizer, loss=training_loss)
input = tf.random.uniform(shape=[1,25], maxval=100, dtype=tf.int32)
hist = model.fit(input, input, epochs=1, steps_per_epoch=1,verbose=0)

```
2. I am getting an error:

> TypeError: apply_gradients() got an unexpected keyword argument 'experimental_aggregate_gradients'


## Expected behavior
optimizer should be created


Discussion:
It seems that keras is passing `experimental_aggregate_gradients` to `apply_gradients`, but the *transformers* TF2 optimizer does not have this argument (see https://github.com/huggingface/transformers/blob/master/src/transformers/optimization_tf.py#L224).

One workaround right now is to set `optimizer._HAS_AGGREGATE_GRAD = False`, which prevents keras from passing this argument.
Thanks for the analysis @volker42maru. @jplu when you're back from vacation, we should fix this optimizer to accept this argument.
Hello!

I was aware of this, and this was on purpose to make the trainer compliant with all the TF 2.X versions. Now that the trainer is fixed to v 2.2 min, I will modify accordingly the method. Thanks @volker42maru to raise this and make me remember I had to update this.

Resolving patch:
diff --git a/src/transformers/optimization_tf.py b/src/transformers/optimization_tf.py
--- a/src/transformers/optimization_tf.py
+++ b/src/transformers/optimization_tf.py
@@ -221,9 +221,9 @@ def _decay_weights_op(self, var, learning_rate, apply_state):
             )
         return tf.no_op()
 
-    def apply_gradients(self, grads_and_vars, name=None):
+    def apply_gradients(self, grads_and_vars, name=None, **kwargs):
         grads, tvars = list(zip(*grads_and_vars))
-        return super(AdamWeightDecay, self).apply_gradients(zip(grads, tvars), name=name,)
+        return super(AdamWeightDecay, self).apply_gradients(zip(grads, tvars), name=name, **kwargs)
 
     def _get_lr(self, var_device, var_dtype, apply_state):
         """Retrieves the learning rate with the given state."""


Test patch:
Repository: huggingface/transformers

Issue:
Some weights of AlbertModel were not initialized ['albert.embeddings.position_ids'] 
Hello!

There seems to be a problem with the current code to load a pre-trained Albert model. This warning appears in any configuration of the Albert model:
`Some weights of AlbertModel were not initialized from the model checkpoint at albert-base-v2 and are newly initialized: ['albert.embeddings.position_ids']`
`You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.`

I found this happens only when I install it from the source. Models load correctly (without warning) when installing the library with pip.


Discussion:
Hello! Which checkpoint are you trying to load? Is it one of your checkpoints or is it one of the checkpoints hosted on the modelhub?
This is probably because of this PR: https://github.com/huggingface/transformers/pull/5773 , but should not pose a real problem. I guess we just have to add the position ids as "allowed" non-initialized weights.
@LysandreJik I used model hub checkpoints. I runned the following lines with pytorch:

`from transformers import AlbertForPreTraining`
`model = AlbertForPreTraining.from_pretrained('albert-base-v2')`


Resolving patch:
diff --git a/src/transformers/modeling_albert.py b/src/transformers/modeling_albert.py
--- a/src/transformers/modeling_albert.py
+++ b/src/transformers/modeling_albert.py
@@ -403,6 +403,7 @@ class AlbertPreTrainedModel(PreTrainedModel):
 
     config_class = AlbertConfig
     base_model_prefix = "albert"
+    authorized_missing_keys = [r"position_ids"]
 
     def _init_weights(self, module):
         """ Initialize the weights.


Test patch:
Repository: huggingface/transformers

Issue:
Resuming training from a checkpoint on Windows does not resume at the correct global_step
### Who can help
Trainer: @sgugger


## Information

I've noticed that resuming training from a checkpoint on Windows fails to start at the right global step (though learning rate, loss, etc continue from where the training left off). The same doesn't happen under macOS.

## Solution?

I think the issue is in Trainer.train, specifically this line:
`self.global_step = int(model_path.split("-")[-1].split("/")[0])`

And it seems to work as expected by swapping `"/"` with `os.sep`




Discussion:
Seems wrong indeed. Would you mind fixing with a PR?
I'm embarrassed to say I'm not super familiar with git or open source collaboration in general, so even though it seems super trivial, I'm worried I'll mess something up. 
I can fix this but it'll be next week as I'm off tonight and tying a few other loose ends.
Thank you! I'm in no personal rush – I got my aborted training to resume. I just wanted someone on the project to know about the problem/solution I found. 

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -644,7 +644,7 @@ def train(self, model_path: Optional[str] = None, trial: Union["optuna.Trial", D
         if model_path is not None:
             # set global_step to global_step of last saved checkpoint from model path
             try:
-                self.global_step = int(model_path.split("-")[-1].split("/")[0])
+                self.global_step = int(model_path.split("-")[-1].split(os.path.sep)[0])
                 epochs_trained = self.global_step // (len(train_dataloader) // self.args.gradient_accumulation_steps)
                 steps_trained_in_current_epoch = self.global_step % (
                     len(train_dataloader) // self.args.gradient_accumulation_steps


Test patch:
Repository: huggingface/transformers

Issue:
How to use the reformer for question answering?
In the code example in the documentation the tokenizer gets passed just a single string. Doesn't it need a context and a question? Also, how exactly to get the answer?


Discussion:

Resolving patch:
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -303,14 +303,15 @@ def _prepare_output_docstrings(output_type, config_class):
         >>> tokenizer = {tokenizer_class}.from_pretrained('{checkpoint}')
         >>> model = {model_class}.from_pretrained('{checkpoint}', return_dict=True)
 
-        >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
+        >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
+        >>> inputs = tokenizer(question, text, return_tensors='pt')
         >>> start_positions = torch.tensor([1])
         >>> end_positions = torch.tensor([3])
 
         >>> outputs = model(**inputs, start_positions=start_positions, end_positions=end_positions)
         >>> loss = outputs.loss
-        >>> start_scores = outputs.start_scores
-        >>> end_scores = outputs.end_scores
+        >>> start_scores = outputs.start_logits
+        >>> end_scores = outputs.end_logits
 """
 
 PT_SEQUENCE_CLASSIFICATION_SAMPLE = r"""


Test patch:
Repository: huggingface/transformers

Issue:
PyTorch (with GPU) Trainer leaks CPU memory on Google Colab
## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
Trainer: @sgugger

## Information

Model I am using `RoBERTaForMaskedLM`:
`
config = RobertaConfig(
    vocab_size=32_000,
    max_position_embeddings=256+2,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)
`
The problem arises when using:
* [ ] [my own modified scripts](https://colab.research.google.com/drive/1bVv6V9IIhTNXvWbpUvQyPyRpK3LOgFfz?usp=sharing): I am trying to follow the official blog tutorial on training a language model from scratch. I have made a few changes from the official script (using Marathi OSCAR corpus, changed model config, and vocabulary size, and am fetching the sentences in the dataset on the fly

The tasks I am working on is:
* [ ] my own task or dataset: Masked Language Modelling with RoBERTa on Marathi Oscar Corpus

## To reproduce

Steps to reproduce the behavior:

1. Run this [colab notebook](https://colab.research.google.com/drive/1bVv6V9IIhTNXvWbpUvQyPyRpK3LOgFfz?usp=sharing).

## Expected behavior
The RAM consumption starts rising towards the end of the first epoch, ultimately crashing the entire session due to full memory consumption (12.72GB of Colab RAM)



Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -1024,7 +1024,7 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
         else:
             loss.backward()
 
-        return loss
+        return loss.detach()
 
     def is_local_master(self) -> bool:
         """


Test patch:
Repository: huggingface/transformers

Issue:
Longformer global attention mask, 2 or 1? 
This example in the LongFormer documentation (https://huggingface.co/transformers/model_doc/longformer.html )
```
attention_mask[:, [1, 4, 21,]] = 2  # Set global attention based on the task. For example,
...                                     # classification: the 
 token
...                                     # QA: question tokens
...                                     # LM: potentially on the beginning of sentences and paragraphs
```
seems to mislead the user to put a 2 as the global attention mask instead of a 1 this is described in the actual documentation. I'm guessing this is a mistake because (a) the example is directly copied from the AllenAI (https://github.com/allenai/longformer#how-to-use), and (b) the source code actually adds a +1 to the attention mask in the function `_merge_to_attention_mask` (https://huggingface.co/transformers/_modules/transformers/modeling_longformer.html#LongformerModel.forward). 

Could the documentation please be fixed or clarified? Thanks! 

Also what is the 1, 4, 21 in that example? 

### Who can help
 Longformer/Reformer: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): LongFormer 



Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_longformer.py b/src/transformers/modeling_longformer.py
--- a/src/transformers/modeling_longformer.py
+++ b/src/transformers/modeling_longformer.py
@@ -795,6 +795,7 @@ class LongformerPreTrainedModel(PreTrainedModel):
 
     config_class = LongformerConfig
     base_model_prefix = "longformer"
+    authorized_missing_keys = [r"position_ids"]
 
     def _init_weights(self, module):
         """ Initialize the weights """
@@ -1019,11 +1020,13 @@ def forward(
 
             >>> # Attention mask values -- 0: no attention, 1: local attention, 2: global attention
             >>> attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
-            >>> attention_mask[:, [1, 4, 21,]] = 2  # Set global attention based on the task. For example,
+            >>> global_attention_mask = torch.zeros(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to global attention to be deactivated for all tokens
+            >>> global_attention_mask[:, [1, 4, 21,]] = 1  # Set global attention to random tokens for the sake of this example
+            ...                                     # Usually, set global attention based on the task. For example,
             ...                                     # classification: the 
 token
             ...                                     # QA: question tokens
             ...                                     # LM: potentially on the beginning of sentences and paragraphs
-            >>> outputs = model(input_ids, attention_mask=attention_mask)
+            >>> outputs = model(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask)
             >>> sequence_output = outputs.last_hidden_state
             >>> pooled_output = outputs.pooler_output
         """


Test patch:
Repository: huggingface/transformers

Issue:
non-interactive transformers-cli upload?
# 🚀 Feature request

Currently, `transformers-cli upload` works only interactively due to its prompt:
`Proceed? [Y/n]`

After running the updated model conversion, I would like to be able to do:
```
cd data
transformers-cli upload fsmt-wmt19-ru-en
transformers-cli upload fsmt-wmt19-en-ru
transformers-cli upload fsmt-wmt19-de-en
transformers-cli upload fsmt-wmt19-en-de
cd -
```
But this won't work:

Would it be possible to add a `-y` override?

Alternatively, would it be possible to give it all dirs to upload in one command?
```
transformers-cli upload fsmt-wmt19-ru-en fsmt-wmt19-en-ru fsmt-wmt19-de-en fsmt-wmt19-en-de
```

## Motivation

I have been re-uploading 4 x 1.1GB models on a relatively slow connection, and I have to be around to hit Y for each one of them, which is very counter-productive, as I have to go back and re-check whether each upload has been completed. I can probably code some shell expect script to feed it automatically, but this defeats the purpose.

Thank you!



Discussion:
FWIW, I found a workaround (god bless Stackoverflow):

```
cd data
yes Y | transformers-cli upload fsmt-wmt19-ru-en
yes Y | transformers-cli upload fsmt-wmt19-en-ru
yes Y | transformers-cli upload fsmt-wmt19-de-en
yes Y | transformers-cli upload fsmt-wmt19-en-de
cd -
```
Ah, nice find :)

I think a `-y` flag would be reasonable if you want to open a PR
Will do. Thank you.

Resolving patch:
diff --git a/src/transformers/commands/user.py b/src/transformers/commands/user.py
--- a/src/transformers/commands/user.py
+++ b/src/transformers/commands/user.py
@@ -40,6 +40,7 @@ def register_subcommand(parser: ArgumentParser):
         upload_parser.add_argument(
             "--filename", type=str, default=None, help="Optional: override individual object filename on S3."
         )
+        upload_parser.add_argument("-y", "--yes", action="store_true", help="Optional: answer Yes to the prompt")
         upload_parser.set_defaults(func=lambda args: UploadCommand(args))
 
 
@@ -221,10 +222,11 @@ def run(self):
                 )
             )
 
-        choice = input("Proceed? [Y/n] ").lower()
-        if not (choice == "" or choice == "y" or choice == "yes"):
-            print("Abort")
-            exit()
+        if not self.args.yes:
+            choice = input("Proceed? [Y/n] ").lower()
+            if not (choice == "" or choice == "y" or choice == "yes"):
+                print("Abort")
+                exit()
         print(ANSI.bold("Uploading... This might take a while if files are large"))
         for filepath, filename in files:
             try:


Test patch:
Repository: huggingface/transformers

Issue:
[example/glue] run_glue compute metrics fail for bart like models
This PR #7126 introduced multiple predictions for trainer. This breaks the `compute_metrics_fn` of `run_glue.py`  for `bart` like models which return multiple predictions.

For `BartForSequenceClassfication` `p.predictions`  is a `tuple`, so following code fails
https://github.com/huggingface/transformers/blob/1d90d0f386af2af52017d51c421e71a51ec94de0/examples/text-classification/run_glue.py#L154

@sgugger 



Discussion:

Resolving patch:
diff --git a/examples/text-classification/run_glue.py b/examples/text-classification/run_glue.py
--- a/examples/text-classification/run_glue.py
+++ b/examples/text-classification/run_glue.py
@@ -150,10 +150,11 @@ def main():
 
     def build_compute_metrics_fn(task_name: str) -> Callable[[EvalPrediction], Dict]:
         def compute_metrics_fn(p: EvalPrediction):
+            preds = p.predictions[0] if type(p.predictions) == tuple else p.predictions
             if output_mode == "classification":
-                preds = np.argmax(p.predictions, axis=1)
-            elif output_mode == "regression":
-                preds = np.squeeze(p.predictions)
+                preds = np.argmax(preds, axis=1)
+            else:  # regression
+                preds = np.squeeze(preds)
             return glue_compute_metrics(task_name, preds, p.label_ids)
 
         return compute_metrics_fn


Test patch: