internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
BART summarization, tokenizer not working
@patil-suraj 
When I am running pytorch/summarization, the logs are as below:
```
Adding AddedToken(content='
', single_word=False, lstrip=False, rstrip=False, normalized=True) to the vocabulary
Adding AddedToken(content='
', single_word=False, lstrip=False, rstrip=False, normalized=True) to the vocabulary
Adding AddedToken(content='<pad>', single_word=False, lstrip=False, rstrip=False, normalized=True) to the vocabulary
Adding AddedToken(content='<mask>', single_word=False, lstrip=True, rstrip=False, normalized=True) to the vocabulary
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
https://huggingface.co/facebook/bart-large-cnn/resolve/main/pytorch_model.bin not found in cache or force_download set to True, downloading to /home/sdiaoaa/.cache/huggingface/transformers/tmpama_iuh8
```
But the embedding did not fit for the added tokens.
The embedding is still [50264] but the token_ides ranges from [0, 50269]


Discussion:
Did you resize the embeddings after adding the new tokens ? To resize the embedding 
```python3
model.resize_token_embeddings(len(tokenizer))
```
> Did you resize the embeddings after adding the new tokens ? To resize the embedding
> 
> ```python
> model.resize_token_embeddings(len(tokenizer))
> ```

Thanks so much for your reply!
The issue has been solved by adding that line.
Just curious, is it a common practice to add 'resize_token_emb' function?  because previously, I have not seen this happens and was wondering why it is not included in the official run_summarization code.

Thanks!
Yes, the embeddings need to be resized after adding new tokens.

And yes, you are right, the embedding should be resized in the example.

Resolving patch:
diff --git a/examples/pytorch/summarization/run_summarization.py b/examples/pytorch/summarization/run_summarization.py
--- a/examples/pytorch/summarization/run_summarization.py
+++ b/examples/pytorch/summarization/run_summarization.py
@@ -353,6 +353,8 @@ def main():
         use_auth_token=True if model_args.use_auth_token else None,
     )
 
+    model.resize_token_embeddings(len(tokenizer))
+
     if model.config.decoder_start_token_id is None:
         raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined")
 
diff --git a/examples/pytorch/translation/run_translation.py b/examples/pytorch/translation/run_translation.py
--- a/examples/pytorch/translation/run_translation.py
+++ b/examples/pytorch/translation/run_translation.py
@@ -337,6 +337,8 @@ def main():
         use_auth_token=True if model_args.use_auth_token else None,
     )
 
+    model.resize_token_embeddings(len(tokenizer))
+
     # Set decoder_start_token_id
     if model.config.decoder_start_token_id is None and isinstance(tokenizer, (MBartTokenizer, MBartTokenizerFast)):
         if isinstance(tokenizer, MBartTokenizer):


Test patch:
Repository: huggingface/transformers

Issue:
[fairscale] rng states saving fails in an extended multi-gpu test
@sgugger, following up on the RNG states PR
```
RUN_SLOW=1 pytest tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_fully_sharded_ddp_fp16 -sv
```
fails on multi-gpu scheduled CI:

```
File "/__w/transformers/transformers/examples/pytorch/translation/run_translation.py", line 589, in <module>
  main()
File "/__w/transformers/transformers/examples/pytorch/translation/run_translation.py", line 522, in main
  train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/__w/transformers/transformers/src/transformers/trainer.py", line 1316, in train
  self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
File "/__w/transformers/transformers/src/transformers/trainer.py", line 1397, in _maybe_log_save_evaluate
  self._save_checkpoint(model, trial, metrics=metrics)
File "/__w/transformers/transformers/src/transformers/trainer.py", line 1534, in _save_checkpoint
stderr: Saving model checkpoint to /tmp/tmpb6tfhvqb/checkpoint-1
  torch.save(rng_states, os.path.join(output_dir, f"rng_state_{local_rank}.pth"))
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 369, in save
  with _open_file_like(f, 'wb') as opened_file:
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 230, in _open_file_like
  return _open_file(name_or_buffer, mode)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 211, in __init__
  super(_open_file, self).__init__(open(name, mode))
stderr: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpb6tfhvqb/checkpoint-1/rng_state_1.pth'
```

https://github.com/huggingface/transformers/runs/2514006341?check_suite_focus=true

Oddly enough it works fine for me if I try it locally.




Discussion:
Found the root: the "checkpoint-1" folder is only created by process 0 when calling "model.save_pretrained", so if process 1 is faster and arrives at `_save_checkpoint` first, it will not see this folder (there is almost no saving on process 1, so it goes through this method quickly).

Will send a fix shortly.

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -1527,6 +1527,9 @@ def _save_checkpoint(self, model, trial, metrics=None):
         if is_torch_tpu_available():
             rng_states["xla"] = xm.get_rng_state()
 
+        # A process can arrive here before the process 0 has a chance to save the model, in which case output_dir may
+        # not yet exist.
+        os.makedirs(output_dir, exist_ok=True)
         local_rank = xm.get_local_ordinal() if is_torch_tpu_available() else self.args.local_rank
         if local_rank == -1:
             torch.save(rng_states, os.path.join(output_dir, "rng_state.pth"))


Test patch:
Repository: huggingface/transformers

Issue:
[Doc] Something wrong in description of 'DistilBertForSequenceClassification' in doc
I think the description of input_ids (one of the parameters) of [DistilBertForSequenceClassification](https://huggingface.co/transformers/model_doc/distilbert.html#distilbertforsequenceclassification) is not correct.
I think the input_ids should be `torch.LongTensor of shape (batch_size, sequence_length)`, rather than `torch.LongTensor of shape (batch_size, num_choices)`.



Discussion:
That's correct, thanks for spotting! Could you open a PR to fix this?

Thanks!

Resolving patch:
diff --git a/src/transformers/models/distilbert/modeling_distilbert.py b/src/transformers/models/distilbert/modeling_distilbert.py
--- a/src/transformers/models/distilbert/modeling_distilbert.py
+++ b/src/transformers/models/distilbert/modeling_distilbert.py
@@ -588,7 +588,7 @@ def __init__(self, config):
 
         self.init_weights()
 
-    @add_start_docstrings_to_model_forward(DISTILBERT_INPUTS_DOCSTRING.format("batch_size, num_choices"))
+    @add_start_docstrings_to_model_forward(DISTILBERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
     @add_code_sample_docstrings(
         tokenizer_class=_TOKENIZER_FOR_DOC,
         checkpoint=_CHECKPOINT_FOR_DOC,


Test patch:
Repository: huggingface/transformers

Issue:
get_length_grouped_indices() uses slow list concat
Hi,

get_length_grouped_indices() in LengthGroupedSampler and DistributedLengthGroupedSampler is prohibitively slow for large number of megabatches (in my case takes hours for ~270k megabatches with 100 items each) due to slow list concatenation with sum(megabatches, []). 

Concatenating the lists with sum() may be repeatedly reallocating memory with each successive concatenation (similar to performance issues with string concatenation).

[item for sublist in megabatches for item in sublist] approach appears to significantly improve speed for large megabatch number, especially for megabatches with larger number of items.

For example:

    # 50,000 megabatches with 3 items each:
    megabatches = [[1,2,3] for _ in range(50_000)]

    %timeit [item for sublist in megabatches for item in sublist];
    3.72 ms ± 75.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    %timeit sum(megabatches, []);
    7.66 s ± 31.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

------------------------------------------
    # 100,000 megabatches with 3 items each:
    megabatches = [[1,2,3] for _ in range(100_000)]

    %timeit [item for sublist in megabatches for item in sublist];
    8.03 ms ± 14.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    %timeit sum(megabatches, []);
    29.6 s ± 36.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

------------------------------------------
    # 100,000 megabatches with 100 items each:
    megabatches = [list(range(100)) for _ in range(100_000)]

    %timeit [item for sublist in megabatches for item in sublist];
    208 ms ± 44.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

    %timeit -r1 -n1 sum(megabatches, []);
    44min 3s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)


Thank you for your wonderful work and consideration of this edit. @sgugger


Discussion:
Thanks a lot for looking at this optimization. It does look like a nice speedup! Do you want to open a PR with the suggested changes since you're the one who designed it?

Resolving patch:
diff --git a/src/transformers/trainer_pt_utils.py b/src/transformers/trainer_pt_utils.py
--- a/src/transformers/trainer_pt_utils.py
+++ b/src/transformers/trainer_pt_utils.py
@@ -495,7 +495,7 @@ def get_length_grouped_indices(lengths, batch_size, mega_batch_mult=None, genera
     # Switch to put the longest element in first position
     megabatches[0][0], megabatches[max_idx][0] = megabatches[max_idx][0], megabatches[0][0]
 
-    return sum(megabatches, [])
+    return [i for megabatch in megabatches for i in megabatch]
 
 
 class LengthGroupedSampler(Sampler):


Test patch:
Repository: huggingface/transformers

Issue:
[docs] XLnet reference link bug in description of past_index Parameter of TrainingArguments 
XLnet reference link bug in description of past_index Parameter of TrainingArguments 
link to the doc: https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments

**current description:** Some models like TransformerXL or **:doc`XLNet <../model_doc/xlnet>`** can make use of the past hidden states for their predictions. If this argument is set to a positive int, the Trainer will use the corresponding output (usually index 2) as the past state and feed it to the model at the next training step under the keyword argument mems.

**expected description:** Some models like TransformerXL or **XLNet** can make use of the past hidden states for their predictions. If this argument is set to a positive int, the Trainer will use the corresponding output (usually index 2) as the past state and feed it to the model at the next training step under the keyword argument mems.

## Environment info
Not required
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

### Who can help
@sgugger
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.



Discussion:
Thanks for flagging! Should be fixed by the PR linked above!

Resolving patch:
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -202,7 +202,7 @@ class TrainingArguments:
             Number of subprocesses to use for data loading (PyTorch only). 0 means that the data will be loaded in the
             main process.
         past_index (:obj:`int`, `optional`, defaults to -1):
-            Some models like :doc:`TransformerXL <../model_doc/transformerxl>` or :doc`XLNet <../model_doc/xlnet>` can
+            Some models like :doc:`TransformerXL <../model_doc/transformerxl>` or :doc:`XLNet <../model_doc/xlnet>` can
             make use of the past hidden states for their predictions. If this argument is set to a positive int, the
             ``Trainer`` will use the corresponding output (usually index 2) as the past state and feed it to the model
             at the next training step under the keyword argument ``mems``.


Test patch:
Repository: huggingface/transformers

Issue:
[AutomaticSpeechRecognitionPipeline] CUDA support
## Environment info

- `transformers` version: 4.6.0
- Platform: Linux-4.15.0-106-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
- pipelines: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): Wav2Vec2

The problem arises when using:
* [ X ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
* [ X ]  Automatic Speech Recognition 

## To reproduce

Steps to reproduce the behavior:

1. Instantiate AutomaticSpeechRecognitionPipeline with device set to GPU
2. Run pipeline inference on example  audio input

```
import transformers
model = transformers.Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-large-xlsr-53-spanish")
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/wav2vec2-large-xlsr-53-spanish")
feature_extractor = transformers.AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-large-xlsr-53-spanish")
pl = transformers.AutomaticSpeechRecognitionPipeline(feature_extractor=feature_extractor, model=model, tokenizer=tokenizer, framework='pt',device=0)

pl('waveform.wav')
```

The snippet above results in the following error:

`RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same`

## Expected behavior

Inputs should be converted to CUDA tensor. 
I believe this is happening because the feature extractor doesn't preserve the device.
I'm able to solve the issue if I add 
`processed = self.ensure_tensor_on_device(**processed)`
after https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/automatic_speech_recognition.py#L136-L138

If this solution is acceptable, I'm happy to open a PR.


Discussion:

Resolving patch:
diff --git a/src/transformers/pipelines/automatic_speech_recognition.py b/src/transformers/pipelines/automatic_speech_recognition.py
--- a/src/transformers/pipelines/automatic_speech_recognition.py
+++ b/src/transformers/pipelines/automatic_speech_recognition.py
@@ -136,6 +136,7 @@ def __call__(
         processed = self.feature_extractor(
             inputs, sampling_rate=self.feature_extractor.sampling_rate, return_tensors="pt"
         )
+        processed = self.ensure_tensor_on_device(**processed)
 
         name = self.model.__class__.__name__
         if name.endswith("ForConditionalGeneration"):


Test patch:
Repository: huggingface/transformers

Issue:
[skipped test] to fix
https://github.com/huggingface/transformers/pull/12059 skipped failing: `tests/test_modeling_tapas.py::TapasUtilitiesTest::test_reduce_sum_vectorized`

This issue is to track its resolution so that it won't be forgotten.

@LysandreJik 


Discussion:

Resolving patch:
diff --git a/src/transformers/models/tapas/modeling_tapas.py b/src/transformers/models/tapas/modeling_tapas.py
--- a/src/transformers/models/tapas/modeling_tapas.py
+++ b/src/transformers/models/tapas/modeling_tapas.py
@@ -1697,9 +1697,9 @@ def _segment_reduce(values, index, segment_reduce_fn, name):
 
     segment_means = scatter(
         src=flat_values,
-        index=flat_index.indices.type(torch.long),
+        index=flat_index.indices.long(),
         dim=0,
-        dim_size=flat_index.num_segments,
+        dim_size=int(flat_index.num_segments),
         reduce=segment_reduce_fn,
     )
 


Test patch:
diff --git a/tests/test_modeling_tapas.py b/tests/test_modeling_tapas.py
--- a/tests/test_modeling_tapas.py
+++ b/tests/test_modeling_tapas.py
@@ -1044,7 +1044,6 @@ def test_reduce_max(self):
         # We use np.testing.assert_array_equal rather than Tensorflow's assertAllEqual
         np.testing.assert_array_equal(maximum.numpy(), [2, 3])
 
-    @unittest.skip("Fix me I'm failing on CI")
     def test_reduce_sum_vectorized(self):
         values = torch.as_tensor([[1.0, 2.0, 3.0], [2.0, 3.0, 4.0], [3.0, 4.0, 5.0]])
         index = IndexMap(indices=torch.as_tensor([0, 0, 1]), num_segments=2, batch_dims=0)
Repository: huggingface/transformers

Issue:
[Examples][Flax] AttributeError: 'DataTrainingArguments' object has no attribute 'test_file'
## Description

While running run_summarization_flax.py with local files, currently we have only two DataTrainingArguments, one for training and another for validation file still we are validating test_file which is producing an error
```
Traceback (most recent call last):
  File "transformers/examples/flax/summarization/run_summarization_flax.py", line 808, in <module>
    main()
  File "transformers/examples/flax/summarization/run_summarization_flax.py", line 352, in main
    if data_args.test_file is not None:
AttributeError: 'DataTrainingArguments' object has no attribute 'test_file'
```

## Environment info
- `transformers` version: 4.9.0(master branch)
- Platform: TPU VM
- Python version: 3.9
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:



### Who can help
@sgugger, @patil-suraj

### Possible Fix:
Either we can add test_file argument or remove test file validation section https://github.com/huggingface/transformers/blob/7d6285a921a23c06169e2d90c94faa0d92d00d78/examples/flax/summarization/run_summarization_flax.py#L352-L354




Discussion:

Resolving patch:
diff --git a/examples/flax/summarization/run_summarization_flax.py b/examples/flax/summarization/run_summarization_flax.py
--- a/examples/flax/summarization/run_summarization_flax.py
+++ b/examples/flax/summarization/run_summarization_flax.py
@@ -135,6 +135,10 @@ class DataTrainingArguments:
         default=None,
         metadata={"help": "An optional input evaluation data file to evaluate the perplexity on (a text file)."},
     )
+    test_file: Optional[str] = field(
+        default=None,
+        metadata={"help": "An optional input predict data file to do prediction on (a text file)."},
+    )
     max_source_length: Optional[int] = field(
         default=1024,
         metadata={


Test patch:
Repository: huggingface/transformers

Issue:
[Flax]Not able to Run Hugging Face GPT2 model for jax on TPU's
 Hi,Trying to do FlaxGPT2ForMultipleChoice.I am trying to run GPT2 using hugging face for TPU's.It is showing tuple out of index.But when i run it in CPU there is no such error.Even when i run a simple model without any code of mine it is also behaving the same in TPU.


![image](https://user-images.githubusercontent.com/58116635/124922271-f5a3ff00-e016-11eb-8679-3258fc3727e9.png)


This below example is for a basic model 


![image](https://user-images.githubusercontent.com/58116635/124922511-313ec900-e017-11eb-9d2c-7473fe8b21af.png)
To Reproduce
This is the colab notebook:
https://colab.research.google.com/drive/1h8CeTM5NUpHeS1oGHONX1YbwtGHf3nyU?usp=sharing



Discussion:
@patrickvonplaten 

Hey @vivekvkashyap,
instead of putting a screenshot here - could you maybe copy paste the link to your google colab instead so that we can reproduce the error?

Thank you!
@patrickvonplaten i have added the  colab notebook in the section To Reproduce

@patil-suraj

Similar to #12578 - we are working on it :-)


Resolving patch:
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -1715,10 +1715,10 @@ def is_tensor(x):
             return True
 
     if is_flax_available():
-        import jaxlib.xla_extension as jax_xla
+        import jax.numpy as jnp
         from jax.core import Tracer
 
-        if isinstance(x, (jax_xla.DeviceArray, Tracer)):
+        if isinstance(x, (jnp.ndarray, Tracer)):
             return True
 
     return isinstance(x, np.ndarray)


Test patch:
Repository: huggingface/transformers

Issue:
`tokenizer.special_tokens_map` has stringified list for "additional_special_tokens" value.
## Environment info

- `transformers` version: 4.8.2
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik maybe?

## Information

Model I am using (Bert, XLNet ...): XLMRoberta

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `>>> from transformers import XLMRobertaTokenizer`
2. `>>> m = XLMRobertaTokenizer.from_pretrained('xlm-roberta-base')`
3. `>>> m.add_special_tokens({"additional_special_tokens": ["<space>"]})`
4. `>>> m.special_tokens_map['additional_special_tokens'] == "['<space>']" # True`

## Expected behavior

The value should be a list containing the special characters. :-) The work-around is to use the `additional_special_tokens` attribute directly. 


Discussion:
That seems like an issue indeed! Pinging @SaulLu 

Resolving patch:
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -1192,7 +1192,11 @@ def special_tokens_map(self) -> Dict[str, Union[str, List[str]]]:
         for attr in self.SPECIAL_TOKENS_ATTRIBUTES:
             attr_value = getattr(self, "_" + attr)
             if attr_value:
-                set_attr[attr] = str(attr_value)
+                set_attr[attr] = (
+                    type(attr_value)(str(attr_value_sub) for attr_value_sub in attr_value)
+                    if isinstance(attr_value, (list, tuple))
+                    else str(attr_value)
+                )
         return set_attr
 
     @property


Test patch:
diff --git a/tests/test_tokenization_common.py b/tests/test_tokenization_common.py
--- a/tests/test_tokenization_common.py
+++ b/tests/test_tokenization_common.py
@@ -2462,6 +2462,10 @@ def test_add_tokens(self):
                 self.assertEqual(
                     tokenizer_r.add_special_tokens({"additional_special_tokens": ["<testtoken3>", "<testtoken4>"]}), 2
                 )
+                self.assertIn("<testtoken3>", tokenizer_r.special_tokens_map["additional_special_tokens"])
+                self.assertIsInstance(tokenizer_r.special_tokens_map["additional_special_tokens"], list)
+                self.assertGreaterEqual(len(tokenizer_r.special_tokens_map["additional_special_tokens"]), 2)
+
                 self.assertEqual(len(tokenizer_r), vocab_size + 8)
 
     def test_offsets_mapping(self):