internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
[docs] TFRobertaModel example: last_hidden_states -> last_hidden_state
This is a documentation error on the currently deployed version of https://huggingface.co/transformers/

### Who can help

 examples/distillation: @VictorSanh
 documentation: @sgugger

## Information

Model I am using: TFRoberta

The problem arises when using:
* [x] the official example scripts

## To reproduce

Steps to reproduce the behavior:

1. View the code example at https://huggingface.co/transformers/model_doc/roberta.html#tfrobertamodel
2. `last_hidden_states = outputs.last_hidden_states` should be `last_hidden_states = outputs.last_hidden_state`

The current incorrect spelling will yield an error. I apologize that I was not able to find that line in the repo, otherwise I would submit a PR.



Discussion:
looks like this is what you're looking for: https://github.com/huggingface/transformers/blob/ae333d04b29a25be1a70eaccd6260c294c243c5b/src/transformers/file_utils.py#L842-L855
Hey @ck37,

Thanks for your issue! Yes, this typo should be corrected -> it would be great if you could open a PR :-) 

Resolving patch:
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -851,7 +851,7 @@ def _prepare_output_docstrings(output_type, config_class):
         >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
         >>> outputs = model(inputs)
 
-        >>> last_hidden_states = outputs.last_hidden_states
+        >>> last_hidden_states = outputs.last_hidden_state
 """
 
 TF_MULTIPLE_CHOICE_SAMPLE = r"""


Test patch:
Repository: huggingface/transformers

Issue:
Improve coverage of the documentation
Currently, some public classes are not documented anywhere because we didn't create the corresponding doc pages. Those missing pages are:

- Benchmark classes
- Bert Japanese
- Data collators

If someone feels like working on one of those, please tag yourself with a comment on this issue. Once the objects are properly documented, they can be removed from the `SHOULD_BE_DOCUMENTED` constant in [this file](https://github.com/huggingface/transformers/blob/1310e1a758edc8e89ec363db76863c771fbeb1de/utils/check_repo.py#L374).



Discussion:
I added docs for Bertweet [here](https://github.com/huggingface/transformers/pull/9379), first contribution, let me know if there is anything missing

Resolving patch:
diff --git a/utils/check_repo.py b/utils/check_repo.py
--- a/utils/check_repo.py
+++ b/utils/check_repo.py
@@ -402,8 +402,6 @@ def find_all_documented_objects():
     "BertJapaneseTokenizer",
     "CharacterTokenizer",
     "MecabTokenizer",
-    # Phoebus
-    "PhobertTokenizer",
     # Benchmarks
     "PyTorchBenchmark",
     "PyTorchBenchmarkArguments",


Test patch:
Repository: huggingface/transformers

Issue:
Improve coverage of the documentation
Currently, some public classes are not documented anywhere because we didn't create the corresponding doc pages. Those missing pages are:

- Benchmark classes
- Bert Japanese
- Data collators

If someone feels like working on one of those, please tag yourself with a comment on this issue. Once the objects are properly documented, they can be removed from the `SHOULD_BE_DOCUMENTED` constant in [this file](https://github.com/huggingface/transformers/blob/1310e1a758edc8e89ec363db76863c771fbeb1de/utils/check_repo.py#L374).



Discussion:
I added docs for Bertweet [here](https://github.com/huggingface/transformers/pull/9379), first contribution, let me know if there is anything missing

Resolving patch:
diff --git a/utils/check_repo.py b/utils/check_repo.py
--- a/utils/check_repo.py
+++ b/utils/check_repo.py
@@ -386,9 +386,6 @@ def find_all_documented_objects():
     "BertJapaneseTokenizer",
     "CharacterTokenizer",
     "MecabTokenizer",
-    # Herbert
-    "HerbertTokenizer",
-    "HerbertTokenizerFast",
     # Phoebus
     "PhobertTokenizer",
     # Benchmarks


Test patch:
Repository: huggingface/transformers

Issue:
Dynamic padding + truncation in classification script
# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

@VictorSanh
@sgugger 

It seems that it is not possible to use dynamic padding along with truncation in classification script, because when `tokenizer` gets `max_length = None` it just skips truncation. 
https://github.com/huggingface/transformers/blob/063d8d27f4e1d089dc76f22e378b86b219167e3b/examples/text-classification/run_glue.py#L290

On the other hand, in language modeling script it works.
https://github.com/huggingface/transformers/blob/063d8d27f4e1d089dc76f22e378b86b219167e3b/examples/language-modeling/run_mlm.py#L311



Discussion:
Yes, the `max_length` should be passed the same way. Would like to open a PR to fix `run_glue.py`?
Okay, I will do that.

Resolving patch:
diff --git a/examples/text-classification/run_glue.py b/examples/text-classification/run_glue.py
--- a/examples/text-classification/run_glue.py
+++ b/examples/text-classification/run_glue.py
@@ -283,11 +283,9 @@ def main():
     # Padding strategy
     if data_args.pad_to_max_length:
         padding = "max_length"
-        max_length = data_args.max_seq_length
     else:
         # We will pad later, dynamically at batch creation, to the max sequence length in each batch
         padding = False
-        max_length = None
 
     # Some models have set the order of the labels to use, so let's make sure we do use it.
     label_to_id = None
@@ -314,7 +312,7 @@ def preprocess_function(examples):
         args = (
             (examples[sentence1_key],) if sentence2_key is None else (examples[sentence1_key], examples[sentence2_key])
         )
-        result = tokenizer(*args, padding=padding, max_length=max_length, truncation=True)
+        result = tokenizer(*args, padding=padding, max_length=data_args.max_seq_length, truncation=True)
 
         # Map labels to IDs (not necessary for GLUE tasks)
         if label_to_id is not None and "label" in examples:


Test patch:
Repository: huggingface/transformers

Issue:
Missing None verification in the CLM language modeling example
Here: https://github.com/huggingface/transformers/blob/1682804ebd504d3381523116773583a52f35afd1/examples/language-modeling/run_clm.py#L230, data_args.train_file can be None (as it is checked some lines above). Therefore, there should be a check to see if it is the case or not.



Discussion:
HI @Aunsiels you are right, thanks! Feel free to open PR to fix it :)

If `train_file` is `None` it should use `validation_file` to get the extension

Resolving patch:
diff --git a/examples/language-modeling/run_clm.py b/examples/language-modeling/run_clm.py
--- a/examples/language-modeling/run_clm.py
+++ b/examples/language-modeling/run_clm.py
@@ -227,7 +227,11 @@ def main():
             data_files["train"] = data_args.train_file
         if data_args.validation_file is not None:
             data_files["validation"] = data_args.validation_file
-        extension = data_args.train_file.split(".")[-1]
+        extension = (
+            data_args.train_file.split(".")[-1]
+            if data_args.train_file is not None
+            else data_args.validation_file.split(".")[-1]
+        )
         if extension == "txt":
             extension = "text"
         datasets = load_dataset(extension, data_files=data_files)


Test patch:
Repository: huggingface/transformers

Issue:
Tokenizer Batch decoding  of predictions obtained from model.generate in t5
How to do batch decoding of sequences obtained from model.generate in t5? Is there a function available for batch decoding in tokenizer `tokenizer.batch_decode_plus` similar to batch enocding `tokenizer.batch_encode_plus`?


Discussion:
There is `batch_decode`, yes, the docs are [here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=batch_decode#transformers.tokenization_utils_base.PreTrainedTokenizerBase.batch_decode).

@sgugger I wonder if we shouldn't make the docs of this method more prominent? The "Utilities for tokenizer" page mentions: "Most of those are only useful if you are studying the code of the tokenizers in the library.", but `batch_decode` and `decode` are only found here, and are very important methods of the tokenization pipeline.
We should add them to the `PreTrainedTokenizer` and `PreTrainedTokenizerFast` documentation. Or did you want to add them to all models?


Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -101,7 +101,7 @@ def _is_start_of_word(text):
     return bool(_is_control(first_char) | _is_punctuation(first_char) | _is_whitespace(first_char))
 
 
-@add_end_docstrings(INIT_TOKENIZER_DOCSTRING, """    .. automethod:: __call__""")
+@add_end_docstrings(INIT_TOKENIZER_DOCSTRING)
 class PreTrainedTokenizer(PreTrainedTokenizerBase):
     """
     Base class for all slow tokenizers.
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -2056,7 +2056,7 @@ def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bo
                 Whether or not to add the special tokens associated with the corresponding model.
             kwargs (additional keyword arguments, `optional`):
                 Will be passed to the underlying model specific encode method. See details in
-                :meth:`~transformers.PreTrainedTokenizer.__call__`
+                :meth:`~transformers.PreTrainedTokenizerBase.__call__`
 
         Returns:
             :obj:`List[str]`: The list of tokens.
diff --git a/src/transformers/tokenization_utils_fast.py b/src/transformers/tokenization_utils_fast.py
--- a/src/transformers/tokenization_utils_fast.py
+++ b/src/transformers/tokenization_utils_fast.py
@@ -56,12 +56,7 @@
 ADDED_TOKENS_FILE = "added_tokens.json"
 
 
-@add_end_docstrings(
-    INIT_TOKENIZER_DOCSTRING,
-    """
-    .. automethod:: __call__
-    """,
-)
+@add_end_docstrings(INIT_TOKENIZER_DOCSTRING)
 class PreTrainedTokenizerFast(PreTrainedTokenizerBase):
     """
     Base class for all fast tokenizers (wrapping HuggingFace tokenizers library).


Test patch:
Repository: huggingface/transformers

Issue:
BertGenerationTokenizer provides an unexpected value for BertGenerationModel
- `transformers` version: 4.2.2
- PyTorch version (GPU?): 1.7.0+cu101
- tokenizers: @n1t0, @LysandreJik

## Information

in both models BertGenerationEncoder,  BertGenerationDecoder, there's no need for `token_type_ids` however the BertGenerationTokenizer provides it, this issue will be raised if you want to input the tokenizer results directly with `**`,

and if it meant to be like this, and the user should be aware of this behaviour, I think a change should be in the documentation.

Note: Another issue with BertGenerationTokenizer is the necessity of sentencepiece module, do you prefer that it should for the user to install it separately or it should be included in transformers dependencies.



Discussion:
Hi @sadakmed!

You're right, there's no need for token type IDs in this tokenizer. The workaround for this is to remove `token_type_ids` from the model input names, as it is done in the DistilBERT tokenizer:

https://github.com/huggingface/transformers/blob/cdd86592317e7db3bab75555c3837fabc74e3429/src/transformers/models/distilbert/tokenization_distilbert.py#L71

Do you want to open a PR to fix this?

Regarding the necessity of sentencepiece module, yes it is necessary. It was previously in the transformers dependencies and we removed it because it was causing compilation issues on some hardware. The error should be straightforward and mention a `sentencepiece` installation is necessary in order to use that tokenizer, so no problem there.

Resolving patch:
diff --git a/src/transformers/models/bert_generation/tokenization_bert_generation.py b/src/transformers/models/bert_generation/tokenization_bert_generation.py
--- a/src/transformers/models/bert_generation/tokenization_bert_generation.py
+++ b/src/transformers/models/bert_generation/tokenization_bert_generation.py
@@ -58,6 +58,7 @@ class BertGenerationTokenizer(PreTrainedTokenizer):
     pretrained_vocab_files_map = {"vocab_file": {"bert_for_seq_generation": tokenizer_url}}
     max_model_input_sizes = {"bert_for_seq_generation": 512}
     prefix_tokens: List[int] = []
+    model_input_names = ["input_ids", "attention_mask"]
 
     def __init__(
         self,


Test patch:
Repository: huggingface/transformers

Issue:
convert_tokens_to_string documentation bug
The [documentation](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.convert_tokens_to_string) states that convert_tokens_to_string would convert _a sequence of token ids in a single string._ 

That is actually not correct as it converts a sequence of tokens. The method that converts a sequence of token ids is the decode method.

Documentation: @sgugger




convert_tokens_to_string documentation bug
The [documentation](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.convert_tokens_to_string) states that convert_tokens_to_string would convert _a sequence of token ids in a single string._ 

That is actually not correct as it converts a sequence of tokens. The method that converts a sequence of token ids is the decode method.

Documentation: @sgugger






Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -3054,12 +3054,13 @@ def _pad(
 
     def convert_tokens_to_string(self, tokens: List[str]) -> str:
         """
-        Converts a sequence of token ids in a single string. The most simple way to do it is ``" ".join(tokens)`` but
-        we often want to remove sub-word tokenization artifacts at the same time
+        Converts a sequence of tokens in a single string. The most simple way to do it is ``" ".join(tokens)`` but
+        we often want to remove sub-word tokenization artifacts at the same time.
 
         Args:
             tokens (:obj:`List[str]`): The token to join in a string.
-        Return: The joined tokens.
+        Returns:
+            :obj:`str`: The joined tokens.
         """
         raise NotImplementedError
 


Test patch:
Repository: huggingface/transformers

Issue:
Typo in deberta_v2/__init__.py
https://github.com/huggingface/transformers/blob/c503a1c15ec1b11e69a3eaaf06edfa87c05a2849/src/transformers/models/deberta_v2/__init__.py#L31
Should be '' DEBERTA_V2_PRETRAINED_MODEL_ARCHIVE_LIST ''.


Discussion:
That's correct! Do you want to open a PR to fix it?

Resolving patch:
diff --git a/src/transformers/models/deberta_v2/__init__.py b/src/transformers/models/deberta_v2/__init__.py
--- a/src/transformers/models/deberta_v2/__init__.py
+++ b/src/transformers/models/deberta_v2/__init__.py
@@ -28,7 +28,7 @@
 
 if is_torch_available():
     _import_structure["modeling_deberta_v2"] = [
-        "DEBERTA_PRETRAINED_MODEL_ARCHIVE_LIST",
+        "DEBERTA_V2_PRETRAINED_MODEL_ARCHIVE_LIST",
         "DebertaV2ForSequenceClassification",
         "DebertaV2Model",
         "DebertaV2ForMaskedLM",


Test patch:
Repository: huggingface/transformers

Issue:
Dead link to optuna.create_study under hyperparamter_search in Trainer
I noticed the hyperlink to the documentation of optuna's create_study under ```kwargs``` in the ```hyperparameter_search``` method of Trainer is outdated. 

https://huggingface.co/transformers/main_classes/trainer.html

### Who can help
Documentation: @sgugger

New URL (I'm guessing): https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html
Old (currently used): https://optuna.readthedocs.io/en/stable/reference/alias_generated/optuna.create_study.html#optuna.create_study





Discussion:
Thanks for flagging! Do you want to make a PR to fix it?

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -1323,7 +1323,7 @@ def hyperparameter_search(
                 more information see:
 
                 - the documentation of `optuna.create_study
-                  <https://optuna.readthedocs.io/en/stable/reference/alias_generated/optuna.create_study.html#optuna.create_study>`__
+                  <https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html>`__
                 - the documentation of `tune.run
                   <https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run>`__
 


Test patch:
Repository: huggingface/transformers

Issue:
GPTNeo: importing model with padded vocab size should truncate wte
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: latest from `master`

### Who can help


@LysandreJik

## Information

Model I am using (Bert, XLNet ...): GPTNeo

Script: https://github.com/huggingface/transformers/blob/master/src/transformers/models/gpt_neo/convert_gpt_neo_mesh_tf_to_pytorch.py

Some GPTNeo models are trained with a vocab size greater than the actual used vocab size (i.e 50304 in config when the actual vocab size is 50257) where all tokens after the first i.e 50257 are unused. These models cannot currently be converted using the script because there is no way to cut the extra embeddings out of wte.


Discussion:

Resolving patch:
diff --git a/src/transformers/models/gpt_neo/modeling_gpt_neo.py b/src/transformers/models/gpt_neo/modeling_gpt_neo.py
--- a/src/transformers/models/gpt_neo/modeling_gpt_neo.py
+++ b/src/transformers/models/gpt_neo/modeling_gpt_neo.py
@@ -112,6 +112,10 @@ def load_tf_weights_in_gpt_neo(model, config, gpt_neo_checkpoint_path):
         if name[-1] == "w" and name[-2] in ["out_proj", "k_proj", "q_proj", "v_proj", "c_proj", "c_fc"]:
             array = array.transpose()
 
+        if name == ["wte"]:
+            # if vocab is padded, then trim off the padding embeddings
+            array = array[: config.vocab_size]
+
         try:
             assert (
                 pointer.shape == array.shape


Test patch:
Repository: huggingface/transformers

Issue:
Bug in numpy_pad_and_concatenate
https://github.com/huggingface/transformers/blob/77b862847b8069d57c0849ca012f48414c427d8e/src/transformers/trainer_pt_utils.py#L71

I believe this should be 
`np.concatenate((array1, array2), axis=0)`
Bug in numpy_pad_and_concatenate
https://github.com/huggingface/transformers/blob/77b862847b8069d57c0849ca012f48414c427d8e/src/transformers/trainer_pt_utils.py#L71

I believe this should be 
`np.concatenate((array1, array2), axis=0)`


Discussion:
Indeed. Would you like to do a PR to fix this since you spotted the bug?
sure thing!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Indeed. Would you like to do a PR to fix this since you spotted the bug?
sure thing!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Resolving patch:
diff --git a/src/transformers/trainer_pt_utils.py b/src/transformers/trainer_pt_utils.py
--- a/src/transformers/trainer_pt_utils.py
+++ b/src/transformers/trainer_pt_utils.py
@@ -72,7 +72,7 @@ def torch_pad_and_concatenate(tensor1, tensor2, padding_index=-100):
 def numpy_pad_and_concatenate(array1, array2, padding_index=-100):
     """Concatenates `array1` and `array2` on first axis, applying padding on the second if necessary."""
     if len(array1.shape) == 1 or array1.shape[1] == array2.shape[1]:
-        return np.concatenate((array1, array2), dim=0)
+        return np.concatenate((array1, array2), axis=0)
 
     # Let's figure out the new shape
     new_shape = (array1.shape[0] + array2.shape[0], max(array1.shape[1], array2.shape[1])) + array1.shape[2:]


Test patch:
Repository: huggingface/transformers

Issue:
Trainer._remove_unused_columns() returns None
## Environment info
- `transformers` version: 4.6.0.dev0
- Platform: Linux-4.15.0-134-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1 (False)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger, @LysandreJik

## Information

`Trainer._remove_unused_columns()` returns None in case `args.remove_unused_columns` is `False`, instead of returning the given dataset.
Related to #11343.

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [x] the official example scripts: (give details below) run_mlm/glue/...
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Set `TrainingArguments.remove_unused_columns=False`
2. Train/eval/test your model using `Trainer`
3. The dataset would be None, and so the following exception would raise:

```
Traceback (most recent call last):
  ...
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/guyrosin/src/transformers/src/transformers/trainer.py", line 814, in num_examples
    self.num_examples(train_dataloader) if train_dataset_is_sized else total_train_batch_size * args.max_steps
  File "/home/guyrosin/src/transformers/src/transformers/trainer.py", line 814, in num_examples
self.num_examples(train_dataloader) if train_dataset_is_sized else total_train_batch_size * args.max_steps
  File "/home/guyrosin/src/transformers/src/transformers/trainer.py", line 814, in num_examples
    return len(dataloader.dataset)
TypeError: object of type 'NoneType' has no len()
```

## Expected behavior

`Trainer._remove_unused_columns()` should always return a dataset.


Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -482,7 +482,7 @@ def remove_callback(self, callback):
 
     def _remove_unused_columns(self, dataset: "datasets.Dataset", description: Optional[str] = None):
         if not self.args.remove_unused_columns:
-            return
+            return dataset
         if self._signature_columns is None:
             # Inspect model forward signature to keep only the arguments it accepts.
             signature = inspect.signature(self.model.forward)


Test patch: