internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
#7858 breaks IterableDataset with __len__ in Trainer
https://github.com/huggingface/transformers/blob/08f534d2da47875a4b7eb1c125cfa7f0f3b79642/src/transformers/trainer.py#L381-L382

This used to be (before #7858)
```python
if isinstance(self.train_dataset, torch.utils.data.IterableDataset):
```

I am using IterableDataset with __len__ in Trainer. This change makes it return a sampler and results in an error later. `ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.RandomSampler object at 0x7fa32c57b340>`

Maybe change to this?
```python
if (isinstance(self.train_dataset, torch.utils.data.IterableDataset) or
        not isinstance(self.train_dataset, collections.abc.Sized)):
```

@j-rossi-nl @sgugger 



Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -374,7 +374,9 @@ def _remove_unused_columns(self, dataset: "datasets.Dataset", description: Optio
         dataset.set_format(type=dataset.format["type"], columns=columns)
 
     def _get_train_sampler(self) -> Optional[torch.utils.data.sampler.Sampler]:
-        if not isinstance(self.train_dataset, collections.abc.Sized):
+        if isinstance(self.train_dataset, torch.utils.data.IterableDataset) or not isinstance(
+            self.train_dataset, collections.abc.Sized
+        ):
             return None
         elif is_torch_tpu_available():
             return get_tpu_sampler(self.train_dataset)


Test patch:
Repository: huggingface/transformers

Issue:
ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler'
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): 
- Using GPU in script?: no
- Using distributed or parallel set-up in script?:no

### Who can help

 Trainer: @sgugger
 



## Information

This import is not compatible with PyTorch 1.4.0 

The problem arises when using:
* [ *] the official example scripts: (give details below) 

The tasks I am working on is:
* [ *] an official GLUE/SQUaD task: (give the name) 
## To reproduce

Steps to reproduce the behavior:

```python
>>> from transformers import PreTrainedTokenizer, is_tf_available, is_torch_available
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yuchenlin/anaconda3/envs/mcqa/lib/python3.7/site-packages/transformers/__init__.py", line 611, in <module>
    from .trainer import Trainer
  File "/home/yuchenlin/anaconda3/envs/mcqa/lib/python3.7/site-packages/transformers/trainer.py", line 69, in <module>
    from .trainer_pt_utils import (
  File "/home/yuchenlin/anaconda3/envs/mcqa/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 26, in <module>
    from torch.optim.lr_scheduler import SAVE_STATE_WARNING
ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler' (/home/yuchenlin/anaconda3/envs/mcqa/lib/python3.7/site-packages/torch/optim/lr_scheduler.py)
```
 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->



Discussion:
Oh I didn't check when they added this. Do you know if PyTorch 1.4.0 is the last version without it? Will add a fix this morning.

Resolving patch:
diff --git a/src/transformers/trainer_pt_utils.py b/src/transformers/trainer_pt_utils.py
--- a/src/transformers/trainer_pt_utils.py
+++ b/src/transformers/trainer_pt_utils.py
@@ -23,7 +23,7 @@
 
 import numpy as np
 import torch
-from torch.optim.lr_scheduler import SAVE_STATE_WARNING
+from packaging import version
 from torch.utils.data.distributed import DistributedSampler
 from torch.utils.data.sampler import RandomSampler, Sampler
 
@@ -34,6 +34,11 @@
 if is_torch_tpu_available():
     import torch_xla.core.xla_model as xm
 
+if version.parse(torch.__version__) <= version.parse("1.4.1"):
+    SAVE_STATE_WARNING = ""
+else:
+    from torch.optim.lr_scheduler import SAVE_STATE_WARNING
+
 logger = logging.get_logger(__name__)
 
 


Test patch:
Repository: huggingface/transformers

Issue:
DataCollatorForWholeWordMask error persists after fix
After quick fix #8379
Tried same flow of #8378
**Gives error:**
```
/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in __call__(self, examples)
    321         for e in examples:
    322             ref_tokens = []
--> 323             for id in e["input_ids"].tolist():
    324                 token = self.tokenizer._convert_id_to_token(id)
    325                 ref_tokens.append(token)

AttributeError: 'list' object has no attribute 'tolist'
```

When I tried with _DataCollatorForLanguageModeling_, trainer works as usual.


Discussion:
The problem can be fixed  by changing `e["input_ids"].tolist()` to `e["input_ids"]`  

Resolving patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -206,6 +206,10 @@ def _collate_batch(examples, tokenizer):
     return result
 
 
+def tolist(x: Union[List[Any], torch.Tensor]):
+    return x.tolist() if isinstance(x, torch.Tensor) else x
+
+
 @dataclass
 class DataCollatorForLanguageModeling:
     """
@@ -320,13 +324,13 @@ def __call__(
         mask_labels = []
         for e in examples:
             ref_tokens = []
-            for id in e["input_ids"].tolist():
+            for id in tolist(e["input_ids"]):
                 token = self.tokenizer._convert_id_to_token(id)
                 ref_tokens.append(token)
 
             # For Chinese tokens, we need extra inf to mark sub-word, e.g [喜,欢]-> [喜，##欢]
             if "chinese_ref" in e:
-                ref_pos = e["chinese_ref"].tolist()
+                ref_pos = tolist(e["chinese_ref"])
                 len_seq = e["input_ids"].size(0)
                 for i in range(len_seq):
                     if i in ref_pos:


Test patch:
Repository: huggingface/transformers

Issue:
Trainer QA Model Label Names
In the Trainer Class, the default label names for a QA model is ["start_positions, end_positions"]. You can see this defined on the line I linked here. Is this supposed to be the case or should it be a list with 2 strings? The tutorials for using Trainer on the transformers documentation show setting up a custom dataset with encoding having two separate keys for the start and end positions. Should we be setting the label_names parameter in our training args or is this a bug? If this is not a bug, could the documentation for setting up a custom QA dataset be updated so others don't get confused as well.
https://github.com/huggingface/transformers/blob/77a257fc210a56f1fd0d75166ecd654cf58111f3/src/transformers/trainer.py#L331


Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -328,7 +328,7 @@ def __init__(
         self.hp_search_backend = None
         self.use_tune_checkpoints = False
         default_label_names = (
-            ["start_positions, end_positions"]
+            ["start_positions", "end_positions"]
             if type(self.model) in MODEL_FOR_QUESTION_ANSWERING_MAPPING.values()
             else ["labels"]
         )


Test patch:
Repository: huggingface/transformers

Issue:
comment correction in modeling_bart.py
Hello, in here 
https://github.com/huggingface/transformers/blob/07708793f20ec3a949ccab32cc4fe0c7272dcc4c/src/transformers/modeling_bart.py#L588

I think the comment should be 
```
# Convert to Bart output format: (BS, seq_len, model_dim) -> (seq_len, BS, model_dim) 
```
Before transpose , shape of x and encoder_hidden_states are both (BS, seq_len, model_dim) to me.
Thanks.





Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_bart.py b/src/transformers/modeling_bart.py
--- a/src/transformers/modeling_bart.py
+++ b/src/transformers/modeling_bart.py
@@ -585,7 +585,7 @@ def forward(
 
         x = F.dropout(x, p=self.dropout, training=self.training)
 
-        # Convert to Bart output format: (seq_len, BS, model_dim) -> (BS, seq_len, model_dim)
+        # Convert to Bart output format: (BS, seq_len, model_dim) ->  (seq_len, BS, model_dim)
         x = x.transpose(0, 1)
         encoder_hidden_states = encoder_hidden_states.transpose(0, 1)
 
diff --git a/src/transformers/modeling_tf_bart.py b/src/transformers/modeling_tf_bart.py
--- a/src/transformers/modeling_tf_bart.py
+++ b/src/transformers/modeling_tf_bart.py
@@ -570,7 +570,7 @@ def call(
             x = self.layernorm_embedding(x + positions)
         x = tf.nn.dropout(x, rate=self.dropout if training else 0)
 
-        # Convert to Bart output format: (seq_len, BS, model_dim) -> (BS, seq_len, model_dim)
+        # Convert to Bart output format: (BS, seq_len, model_dim) ->  (seq_len, BS, model_dim)
         x = tf.transpose(x, perm=(1, 0, 2))
         assert len(shape_list(encoder_hidden_states)) == 3, "encoder_hidden_states must be a 3D tensor"
         encoder_hidden_states = tf.transpose(encoder_hidden_states, perm=(1, 0, 2))


Test patch:
Repository: huggingface/transformers

Issue:
Inconsistent PreTrainedTokenizerBase.pad  argument default value & docstring
The docstring states the argument `padding` has a default of `False` but its default is `True`

docstring: 
https://github.com/huggingface/transformers/blob/d5b3e56de5376aa85ef46e7f0325139d9e299a41/src/transformers/tokenization_utils_base.py#L2469-L2470
arg:
https://github.com/huggingface/transformers/blob/d5b3e56de5376aa85ef46e7f0325139d9e299a41/src/transformers/tokenization_utils_base.py#L2431-L2472

This causes issues when using `DataCollatorForLanguageModeling` with an already padded dataset as it resets the attention mask.


Discussion:
Seems to have been added in this commit: 
https://github.com/huggingface/transformers/commit/f3065abdb8805f5beaed9ff1e92ce874e655f5c9#diff-85b29486a884f445b1014a26fecfb189141f2e6b09f4ae701ee758a754fddcc1R2146-R2168
As part of merge https://github.com/huggingface/transformers/pull/6110
Hi, indeed! The docs should be changed to reflect the method signature. Do you want to open a PR?

Resolving patch:
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -2604,7 +2604,7 @@ def pad(
 
                 Instead of :obj:`List[int]` you can have tensors (numpy arrays, PyTorch tensors or TensorFlow tensors),
                 see the note above for the return type.
-            padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`False`):
+            padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`):
                  Select a strategy to pad the returned sequences (according to the model's padding side and padding
                  index) among:
 


Test patch:
Repository: huggingface/transformers

Issue:
[trainer] add distributed_env to TrainingArguments
As discussed in https://github.com/huggingface/transformers/pull/8823 it's not simple to check whether the downstream code is running under distributed mode or not (currently requires checking `self.args.local_rank != -1` which is far from obvious.

So we were discussing about adding a flag like `distributed_env` so that the downstream code could do a much simpler intuitive check.

I'm not sure whether we need just True/False for ddp or whether we also need to have another flag if we are under DP as well?

@sgugger 


Discussion:

Resolving patch:
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -455,6 +455,27 @@ def n_gpu(self):
         """
         return self._setup_devices[1]
 
+    @property
+    @torch_required
+    def parallel_mode(self):
+        """
+        The current mode used for parallelism if multiple GPUs/TPU cores are available. One of:
+
+        - :obj:`ParallelMode.NOT_PARALLEL`: no parallelism (CPU or one GPU).
+        - :obj:`ParallelMode.NOT_DISTRIBUTED`: several GPUs in one single process (uses :obj:`torch.nn.DataParallel`).
+        - :obj:`ParallelMode.DISTRIBUTED`: several GPUs, each ahving its own process (uses
+          :obj:`torch.nn.DistributedDataParallel`).
+        - :obj:`ParallelMode.TPU`: several TPU cores.
+        """
+        if is_torch_tpu_available():
+            return ParallelMode.TPU
+        elif self.local_rank != -1:
+            return ParallelMode.DISTRIBUTED
+        elif self.n_gpu > 1:
+            return ParallelMode.NOT_DISTRIBUTED
+        else:
+            return ParallelMode.NOT_PARALLEL
+
     def to_dict(self):
         """
         Serializes this instance while replace `Enum` by their values (for JSON serialization support).
@@ -483,3 +504,10 @@ def to_sanitized_dict(self) -> Dict[str, Any]:
             valid_types.append(torch.Tensor)
 
         return {k: v if type(v) in valid_types else str(v) for k, v in d.items()}
+
+
+class ParallelMode(Enum):
+    NOT_PARALLEL = "not_parallel"
+    NOT_DISTRIBUTED = "not_distributed"
+    DISTRIBUTED = "distributed"
+    TPU = "tpu"


Test patch:
Repository: huggingface/transformers

Issue:
AttributeError: 'NoneType' object has no attribute 'from_pretrained'
This code was working yesterday but doesn't work today:

```py
from transformers import AutoTokenizer
AutoTokenizer("Helsinki-NLP/opus-mt-en-fr")
```


Discussion:
Same here a couple of hours ago
1. Hi, could you please provide the information related to your environment? 

2. When you say it was working yesterday but was working before, do you mean to say you've upgraded to version v4.0.0 released yesterday? If this is so, you may be obtaining the following error message: `AttributeError: 'NoneType' object has no attribute 'from_pretrained'`. This would be because you do not have `sentencepiece` installed.

3. Are you sure this worked previously? This should never have worked, as `AutoTokenizer` cannot be initialized like this, but has to be instantiated from the `from_pretrained` method:

```py
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr")
```

which works on v4.0.0 and on `master`, as long as you have SentencePiece installed.

Resolving patch:
diff --git a/src/transformers/models/auto/tokenization_auto.py b/src/transformers/models/auto/tokenization_auto.py
--- a/src/transformers/models/auto/tokenization_auto.py
+++ b/src/transformers/models/auto/tokenization_auto.py
@@ -369,7 +369,13 @@ def from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs):
             if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
                 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
             else:
-                return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
+                if tokenizer_class_py is not None:
+                    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
+                else:
+                    raise ValueError(
+                        "This tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed "
+                        "in order to use this tokenizer."
+                    )
 
         raise ValueError(
             "Unrecognized configuration class {} to build an AutoTokenizer.\n"


Test patch:
Repository: huggingface/transformers

Issue:
AttributeError: 'Trainer' object has no attribute 'is_world_master'
## Environment info
     
- `transformers` version: 4.0.0
- Platform: Linux-4.15.0-96-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.8
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
- Ray version: 1.0.1.post1

### Who can help
@sgugger — Would you be able to offer any insight?

## Information

Model I am using (Bert, XLNet ...): BertForSequenceClassification

The problem arises when using:
* [ ] the official example scripts:
* [ x] my own modified scripts:

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task:
* [ x] my own task or dataset:

## To reproduce
The error occurs when I try to run a hyperparameter search for my finetuning step using Ray Tune. I am able to successfully finetune the BertForSequenceClassification model normally — the error only arises when running hyperparameter search.

```
config = BertConfig.from_pretrained(pretrained_model_path, num_labels=num_labels, finetuning_task ='text-classification')

def model_init():
    return BertForSequenceClassification.from_pretrained(pretrained_model_path, config=config)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    model_init=model_init
)

trainer.hyperparameter_search(
    direction="minimize", 
    backend="ray",
    n_trials=20,
    keep_checkpoints_num=1,
    resources_per_trial = {'gpu':1, 'cpu':1}
)
```

## Expected behavior

I am trying to run Ray Tune from Huggingface as per these instructions: https://huggingface.co/blog/ray-tune

If anyone has any insight as to what could be causing this error, it would be greatly appreciated, thank you!



Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -543,7 +543,7 @@ def _tune_save_checkpoint(self):
             self.args.output_dir = checkpoint_dir
             output_dir = os.path.join(self.args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{self.state.global_step}")
             self.save_model(output_dir)
-            if self.is_world_master():
+            if self.is_world_process_zero():
                 self.state.save_to_json(os.path.join(output_dir, "trainer_state.json"))
                 torch.save(self.optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
                 torch.save(self.lr_scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))


Test patch:
Repository: huggingface/transformers

Issue:
"run_mlm_wwm.py", line 284 AttributeError: 'DataTrainingArguments' object has no attribute 'valid_ref_file'
Hi! There is a tiny typo in the code "transformers/examples/language-modeling/run_mlm_wwm.py" at line 284. It should be:
`if data_args.validation_ref_file is not None:` since at line 103 in `DataTrainingArguments` it defined as `validation_ref_file:`


Discussion:
Hi! Do you want to submit a PR?
Sure, I'll be happy to do it. I need permission... `remote: Permission to huggingface/transformers.git denied to NatLun137.`
Hmmm I think you just tried to push on `huggingface/transformers`? You should fork the repo, apply your changes there and then open a PR here. I see you created your fork already, how did you open a PR then? Did you use the GitHub UI?

Resolving patch:
diff --git a/examples/language-modeling/run_mlm_wwm.py b/examples/language-modeling/run_mlm_wwm.py
--- a/examples/language-modeling/run_mlm_wwm.py
+++ b/examples/language-modeling/run_mlm_wwm.py
@@ -281,7 +281,7 @@ def tokenize_function(examples):
     # Add the chinese references if provided
     if data_args.train_ref_file is not None:
         tokenized_datasets["train"] = add_chinese_references(tokenized_datasets["train"], data_args.train_ref_file)
-    if data_args.valid_ref_file is not None:
+    if data_args.validation_ref_file is not None:
         tokenized_datasets["validation"] = add_chinese_references(
             tokenized_datasets["validation"], data_args.validation_ref_file
         )


Test patch:
Repository: huggingface/transformers

Issue:
BatchEncoding.to accepted types too restrictive
## Environment info
     
- `transformers` version: 4.1.1
- Platform: Linux-4.14.81.bm.15-amd64-x86_64-with-debian-9.11
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help

## Information

In `BatchEncoding.to`, the only accepted class types are `str` and `torch.device`. I think some libraries like pytorch-lightning call `.to` with the integer value for the GPU number, and HF complains about this when it is perfectly valid:

```
>>> x = torch.zeros(1)
>>> x.to(0)
tensor([0.], device='cuda:0')
```

## Expected behavior

Also allow int values in BatchEncoding.to


Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -781,7 +781,7 @@ def to(self, device: Union[str, "torch.device"]) -> "BatchEncoding":
         # This check catches things like APEX blindly calling "to" on all inputs to a module
         # Otherwise it passes the casts down and casts the LongTensor containing the token idxs
         # into a HalfTensor
-        if isinstance(device, str) or isinstance(device, torch.device):
+        if isinstance(device, str) or isinstance(device, torch.device) or isinstance(device, int):
             self.data = {k: v.to(device=device) for k, v in self.data.items()}
         else:
             logger.warning(


Test patch:
Repository: huggingface/transformers

Issue:
Improve coverage of the documentation
Currently, some public classes are not documented anywhere because we didn't create the corresponding doc pages. Those missing pages are:

- Benchmark classes
- Bert Japanese
- Data collators

If someone feels like working on one of those, please tag yourself with a comment on this issue. Once the objects are properly documented, they can be removed from the `SHOULD_BE_DOCUMENTED` constant in [this file](https://github.com/huggingface/transformers/blob/1310e1a758edc8e89ec363db76863c771fbeb1de/utils/check_repo.py#L374).



Discussion:

Resolving patch:
diff --git a/utils/check_repo.py b/utils/check_repo.py
--- a/utils/check_repo.py
+++ b/utils/check_repo.py
@@ -382,8 +382,6 @@ def find_all_documented_objects():
     "BertJapaneseTokenizer",
     "CharacterTokenizer",
     "MecabTokenizer",
-    # Bertweet
-    "BertweetTokenizer",
     # Herbert
     "HerbertTokenizer",
     "HerbertTokenizerFast",


Test patch: