internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
Possibly wrong API documentation for BigBirdTokenizerFast 
- `transformers` version: v.4.9.0

### Who can help

Documentation: @sgugger 

## Information

At this URL: 
https://huggingface.co/transformers/model_doc/bigbird.html#transformers.BigBirdTokenizerFast

The doc says,

`bos_token (str, optional, defaults to "[CLS]")`
and
`eos_token (str, optional, defaults to "[SEP]")`

but the actual code is:

```
    def __init__(
        self,
        vocab_file=None,
        tokenizer_file=None,
        unk_token="<unk>",
        bos_token="
",
        eos_token="
",
        pad_token="<pad>",
```
May be the API document need a fix? the explanation seems already clear though, only default value of those params are wrong.



Discussion:

Resolving patch:
diff --git a/src/transformers/models/big_bird/tokenization_big_bird_fast.py b/src/transformers/models/big_bird/tokenization_big_bird_fast.py
--- a/src/transformers/models/big_bird/tokenization_big_bird_fast.py
+++ b/src/transformers/models/big_bird/tokenization_big_bird_fast.py
@@ -67,14 +67,14 @@ class BigBirdTokenizerFast(PreTrainedTokenizerFast):
         vocab_file (:obj:`str`):
             `SentencePiece <https://github.com/google/sentencepiece>`__ file (generally has a `.spm` extension) that
             contains the vocabulary necessary to instantiate a tokenizer.
-        bos_token (:obj:`str`, `optional`, defaults to :obj:`"[CLS]"`):
+        bos_token (:obj:`str`, `optional`, defaults to :obj:`"
"`):
             The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
 
             .. note::
 
                When building a sequence using special tokens, this is not the token that is used for the beginning of
                sequence. The token used is the :obj:`cls_token`.
-        eos_token (:obj:`str`, `optional`, defaults to :obj:`"[SEP]"`):
+        eos_token (:obj:`str`, `optional`, defaults to :obj:`"
"`):
             The end of sequence token. .. note:: When building a sequence using special tokens, this is not the token
             that is used for the end of sequence. The token used is the :obj:`sep_token`.
         unk_token (:obj:`str`, `optional`, defaults to :obj:`"<unk>"`):


Test patch:
Repository: huggingface/transformers

Issue:
Feature request: Show command line argument defaults
# 🚀 Feature request

When running with `--help`, show the default values for command line arguments.

## Motivation

There are dozens of command line arguments. When I'm trying to figure out how to run a script, I often want to know what value is being used when I don't specify it. But running with `--help` doesn't show the default values unless it's explicitly written in the description (which is only for three of them for the example script I'm using).

For example, `--evaluation_strategy`

```
  --evaluation_strategy {no,steps,epoch}
                        The evaluation strategy to use.
```

This ends up being a bit of a frustrating user experience. The two ways I currently use to find the value are:

1. Run the script again without `--help` and log all the arguments (done in the examples). This shows the assigned value, which will be the default if not passed. However, it doesn't show the description of what it does.

2. Go to the documentation. This will show the default value and a more thorough description, but requires opening a web browser and Googling to find the right page.

In other Python projects, I use the `argparse.ArgumentDefaultsHelpFormatter`, which automatically displays default values in the `--help` message along with their descriptions.

```python
parser = argparse.ArgumentParser(
    formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
```

I wonder whether the Huggingface arguments could support the same feature?

Many thanks for considering this!


Discussion:

Resolving patch:
diff --git a/src/transformers/hf_argparser.py b/src/transformers/hf_argparser.py
--- a/src/transformers/hf_argparser.py
+++ b/src/transformers/hf_argparser.py
@@ -16,7 +16,7 @@
 import json
 import re
 import sys
-from argparse import ArgumentParser, ArgumentTypeError
+from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, ArgumentTypeError
 from enum import Enum
 from pathlib import Path
 from typing import Any, Iterable, List, NewType, Optional, Tuple, Union
@@ -59,6 +59,9 @@ def __init__(self, dataclass_types: Union[DataClassType, Iterable[DataClassType]
             kwargs:
                 (Optional) Passed to `argparse.ArgumentParser()` in the regular way.
         """
+        # To make the default appear when using --help
+        if "formatter_class" not in kwargs:
+            kwargs["formatter_class"] = ArgumentDefaultsHelpFormatter
         super().__init__(**kwargs)
         if dataclasses.is_dataclass(dataclass_types):
             dataclass_types = [dataclass_types]


Test patch:
Repository: huggingface/transformers

Issue:
`Trainer.train(resume_from_checkpoint=False)` is causing an exception
Since `resume_from_checkpoint` can be `str` and `bool` it should be possible to pass `False` to it.
But when `resume_from_checkpoint` is `False` it causes an exception here:

https://github.com/huggingface/transformers/blob/3d4b3bc3fd77e0e48e2364464ea90379f13bcf37/src/transformers/trainer.py#L1049-L1050

```text
E       TypeError: expected str, bytes or os.PathLike object, not bool
```

The most simple solution would be to do this at the beginning of the `train` function:

```python
resume_from_checkpoint = None if not resume_from_checkpoint else resume_from_checkpoint
```

If wanted I can provide a PR.


Discussion:
That seems like the right fix indeed. Please go ahead with a PR, thanks! :-)

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -1005,6 +1005,7 @@ def train(
             kwargs:
                 Additional keyword arguments used to hide deprecated arguments
         """
+        resume_from_checkpoint = None if not resume_from_checkpoint else resume_from_checkpoint
 
         # memory metrics - must set up as early as possible
         self._memory_tracker.start()


Test patch:
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -827,6 +827,20 @@ def test_resume_training_with_randomness(self):
         self.assertAlmostEqual(a, a1, delta=1e-8)
         self.assertAlmostEqual(b, b1, delta=1e-8)
 
+    # regression for this issue: https://github.com/huggingface/transformers/issues/12970
+    def test_training_with_resume_from_checkpoint_flase(self):
+        train_dataset = RegressionDataset(length=128)
+        eval_dataset = RegressionDataset()
+
+        config = RegressionModelConfig(a=0, b=2)
+        model = RegressionRandomPreTrainedModel(config)
+
+        tmp_dir = self.get_auto_remove_tmp_dir()
+        args = RegressionTrainingArguments(tmp_dir, save_steps=5, learning_rate=0.1)
+        trainer = Trainer(model, args, train_dataset=train_dataset, eval_dataset=eval_dataset)
+
+        trainer.train(resume_from_checkpoint=False)
+
     @require_torch_up_to_2_gpus
     def test_resume_training_with_gradient_accumulation(self):
         # This test will fail for more than 2 GPUs since the batch size will get bigger and with the number of
Repository: huggingface/transformers

Issue:
Deprecate Mirror
The mirror option for downloading is now deprecated due to the switching to Git LFS and removal of Tuna.

https://github.com/tuna/issues/issues/937


Discussion:
(i am not a maintainer, please wait for maintainer review/approval before merging!) @sgugger @LysandreJik 
> (i am not a maintainer, please wait for maintainer review/approval before merging!) @sgugger @LysandreJik 

Oh I'll be more cautious next time (though I'm sure this one is okay)
> Thanks for the PR and the concept of deprecating this `mirror` argument sounds good but:
> 
> - we use proper deprecation warnings for this
> 
> - we never do breaking changes between minor releases, so the argument should still behave in the same way for now with the deprecation warning, and we should only change its behavior or remove it for the next major version.
> 
> 
> 
> I think the easiest is to revert this merge and start fresh on a new PR.

Hi @sgugger 

I was in a hurry (gonna be on a trip soon). Sorry for rushing for it!

It's not an actual breaking change because the HF mirrors have already been removed from these servers. They are already broken now. The warning is a way to explain why it breaks.

I'll fix it in a new PR. Gonna throw a ValueError with no deprecation warning.

Resolving patch:
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -1257,11 +1257,8 @@ def hf_bucket_url(
         filename = f"{subfolder}/{filename}"
 
     if mirror:
-        logger.warning(
-            "Deprecated: The mirroring is deprecated."
-            "Tuna and BFSU mirrors are no longer available."
-            "We will soon provide a self-hosted alternative."
-        )
+        if mirror in ["tuna", "bfsu"]:
+            raise ValueError("The Tuna and BFSU mirrors are no longer available. Try removing the mirror argument.")
         legacy_format = "/" not in model_id
         if legacy_format:
             return f"{mirror}/{model_id}-{filename}"


Test patch:
Repository: huggingface/transformers

Issue:
sentencepiece version need upgrade
## Environment info

- `transformers` version: 4.11.0.dev0
- Platform: Debian Gnu/Linux
- Python version: 3.9
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help

- setup: @LysandreJik may be ?

## Information

sentencepiece==0.1.91 does not exist anymore

pip install fails:

```
ERROR: Could not find a version that satisfies the requirement sentencepiece==0.1.91
ERROR: No matching distribution found for sentencepiece==0.1.91
```

## To reproduce

Steps to reproduce the behavior:

1. create a new venv
2. pip install -e .


## Expected behavior

- Need to bump to the latest but minor version change 0.1.96 for transformer's install to work



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -134,7 +134,7 @@
     "sacremoses",
     "sagemaker>=2.31.0",
     "scikit-learn",
-    "sentencepiece==0.1.91",
+    "sentencepiece>=0.1.91,!=0.1.92",
     "soundfile",
     "sphinx-copybutton",
     "sphinx-markdown-tables",
diff --git a/src/transformers/dependency_versions_table.py b/src/transformers/dependency_versions_table.py
--- a/src/transformers/dependency_versions_table.py
+++ b/src/transformers/dependency_versions_table.py
@@ -52,7 +52,7 @@
     "sacremoses": "sacremoses",
     "sagemaker": "sagemaker>=2.31.0",
     "scikit-learn": "scikit-learn",
-    "sentencepiece": "sentencepiece==0.1.91",
+    "sentencepiece": "sentencepiece>=0.1.91,!=0.1.92",
     "soundfile": "soundfile",
     "sphinx-copybutton": "sphinx-copybutton",
     "sphinx-markdown-tables": "sphinx-markdown-tables",


Test patch:
Repository: hartwork/jawanndenn

Issue:
Error 500 when pasting JSON with single surrogate characters
Example:
```json
{
  "equal_width": false,
  "title": "Which fruit do *you* like?",
  "options": [
    "Apple",
    "Banana",
    "Orange",
    "Papaya", "\ud800"
  ]
}
```
Error 500 when pasting JSON with single surrogate characters
Example:
```json
{
  "equal_width": false,
  "title": "Which fruit do *you* like?",
  "options": [
    "Apple",
    "Banana",
    "Orange",
    "Papaya", "\ud800"
  ]
}
```


Discussion:

Resolving patch:
diff --git a/jawanndenn/views.py b/jawanndenn/views.py
--- a/jawanndenn/views.py
+++ b/jawanndenn/views.py
@@ -1,10 +1,9 @@
 # Copyright (C) 2019 Sebastian Pipping <sebastian@pipping.org>
 # Licensed under GNU Affero GPL v3 or later
 
-import json  # lgtm [py/import-and-import-from]
 from functools import wraps
-from json import JSONDecodeError
 
+import rapidjson as json  # lgtm [py/import-and-import-from]
 from django.conf import settings
 from django.db import transaction
 from django.http import (Http404, HttpResponseBadRequest, HttpResponseNotFound,
@@ -18,6 +17,7 @@
 from jawanndenn.markup import safe_html
 from jawanndenn.models import Ballot, Poll, Vote
 from jawanndenn.serializers import PollConfigSerializer
+from rapidjson import JSONDecodeError
 from rest_framework.exceptions import ValidationError
 
 _staticfile_finders = [import_string(cls_string)()


Test patch:
diff --git a/jawanndenn/tests/test_views.py b/jawanndenn/tests/test_views.py
--- a/jawanndenn/tests/test_views.py
+++ b/jawanndenn/tests/test_views.py
@@ -1,9 +1,9 @@
 # Copyright (C) 2019 Sebastian Pipping <sebastian@pipping.org>
 # Licensed under GNU Affero GPL v3 or later
 
-import json
 from http import HTTPStatus
 
+import rapidjson as json
 from django.test import TestCase
 from django.urls import reverse
 from django.utils import timezone
Repository: hartwork/jawanndenn

Issue:
missing rapid-json dependency for pip
installing via pip3 does not also install the rapid-json dependency.
missing rapid-json dependency for pip
installing via pip3 does not also install the rapid-json dependency.


Discussion:
I confirm as a bug.

```console
# cd "$(mktemp -d)"
# virtualenv --python=python3.9 venv
# source venv/bin/activate
# pip install jawanndenn
# jawanndenn
Traceback (most recent call last):
[..]
  File "/tmp/tmp.SKmozabD1B/venv/lib/python3.9/site-packages/jawanndenn/views.py", line 6, in <module>
    import rapidjson as json  # lgtm [py/import-and-import-from]
ModuleNotFoundError: No module named 'rapidjson'
```
I confirm as a bug.

```console
# cd "$(mktemp -d)"
# virtualenv --python=python3.9 venv
# source venv/bin/activate
# pip install jawanndenn
# jawanndenn
Traceback (most recent call last):
[..]
  File "/tmp/tmp.SKmozabD1B/venv/lib/python3.9/site-packages/jawanndenn/views.py", line 6, in <module>
    import rapidjson as json  # lgtm [py/import-and-import-from]
ModuleNotFoundError: No module named 'rapidjson'
```

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -52,6 +52,8 @@ def _collect_package_data(top_directory):
             'djangorestframework>=3.11.0',
             'gunicorn>=20.0.4',
             'gunicorn-color>=0.1.0',
+            'python-dateutil>=2.8.1',
+            'python-rapidjson>=1.0',
         ],
         extras_require=_extras_require,
         tests_require=_tests_require,


Test patch:
Repository: Azure/azure-linux-extensions

Issue:
[CustomScript] error message misleading
if you leave the `commandToExecute` field empty in configuration, you get an error like:

```
Enable failed: Failed with error: CommandToExecute is empty or invalid, Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.OSTCExtensions.CustomScriptForLinux-1.5.2.0/customscript.py", line 79, in main
    enable(hutil)
  File "/var/lib/waagent/Microsoft.OSTCExtensions.CustomScriptForLinux-1.5.2.0/customscript.py", line 113, in enable
    start_daemon(hutil)
  File "/var/lib/waagent/Microsoft.OSTCExtensions.CustomScriptForLinux-1.5.2.0/customscript.py", line 263, in start_daemon
    raise ValueError(error_msg)
ValueError: CommandToExecute is empty or invalid
```

notice the difference between `CommandToExecute` and `commandToExecute`. there should be consistency.



Discussion:

Resolving patch:
diff --git a/CustomScript/customscript.py b/CustomScript/customscript.py
--- a/CustomScript/customscript.py
+++ b/CustomScript/customscript.py
@@ -254,7 +254,7 @@ def start_daemon(hutil):
         hutil.do_exit(0, 'Enable', 'transitioning', '0',
                       'Launching the script...')
     else:
-        error_msg = "CommandToExecute is empty or invalid"
+        error_msg = "commandToExecute is empty or invalid"
         hutil.error(error_msg)
         waagent.AddExtensionEvent(name=ExtensionShortName,
                                   op=RunScriptOp,
@@ -292,7 +292,7 @@ def daemon(hutil):
     if args:
         ScriptUtil.run_command(hutil, args, prepare_download_dir(hutil.get_seq_no()), 'Daemon', ExtensionShortName, Version)
     else:
-        error_msg = "CommandToExecute is empty or invalid."
+        error_msg = "commandToExecute is empty or invalid."
         hutil.error(error_msg)
         waagent.AddExtensionEvent(name=ExtensionShortName,
                                   op=RunScriptOp,


Test patch:
Repository: Azure/azure-linux-extensions

Issue:
[LAD] Use 'storageAccountEndPoint' not 'endpoint' 
On a VM deployed on MoonMake using portal on 8/15/2016 using 2.3.9007 LAD, LAD couldn't find the proper endpoint of storage. It turned out that LAD code is searching key = 'endpoint', while the actual key is 'storageAccountEndPoint'.

{"storageAccountName":"xxx","storageAccountKey":"key*********",
"storageAccountEndPoint":"https://core.chinacloudapi.cn"}

[root@MyCentosVM5 Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9007]#



Discussion:

Resolving patch:
diff --git a/Diagnostic/diagnostic.py b/Diagnostic/diagnostic.py
--- a/Diagnostic/diagnostic.py
+++ b/Diagnostic/diagnostic.py
@@ -396,6 +396,18 @@ def setEventVolume(mdsdCfg,ladCfg):
             
     XmlUtil.setXmlValue(mdsdCfg,"Management","eventVolume",eventVolume)
         
+def getStorageAccountEndPoint(account):
+    endpoint = readPrivateConfig('storageAccountEndPoint')
+    if endpoint:
+        parts = endpoint.split('//', 1)
+        if len(parts) > 1:
+            endpoint = parts[0]+'//'+account+".table."+parts[1]
+        else:
+            endpoint = 'https://'+account+".table."+parts[0]
+    else:
+        endpoint = 'https://'+account+'.table.core.windows.net'
+    return endpoint
+
 def configSettings():
     '''
     Generates XML cfg file for mdsd, from JSON config settings (public & private).
@@ -458,10 +470,7 @@ def configSettings():
     key = readPrivateConfig('storageAccountKey')
     if not key:
         return False, "Empty storageAccountKey"
-    endpoint = readPrivateConfig('endpoint')
-    if not endpoint:
-        endpoint = 'table.core.windows.net'
-    endpoint = 'https://'+account+"."+endpoint
+    endpoint = getStorageAccountEndPoint(account)
 
     createAccountSettings(mdsdCfg,account,key,endpoint,aikey)
 


Test patch:
Repository: Azure/azure-linux-extensions

Issue:
Azure Monitor Agent Linux failing to install 1.27.2
A recent change created an issue with the agent on linux. This should be:
`if "auth" in public_settings.get("proxy") and public_settings.get("proxy").get("auth") == True:` and not `if "auth" in public_settings.get("proxy") and public_settings.get("proxy").get("auth") == true:`

Here is the error:

```
2023/07/06 16:45:20 ERROR:    if "auth" in public_settings.get("proxy") and public_settings.get("proxy").get("auth") == true:
2023/07/06 16:45:20 ERROR:NameError: name 'true' is not defined
```

https://github.com/Azure/azure-linux-extensions/blob/ac28d63cfa7991559563ee9fa228f527079240c2/AzureMonitorAgent/agent.py#L692C96-L692C101

Introduced by: https://github.com/Azure/azure-linux-extensions/pull/1732


Discussion:

Resolving patch:
diff --git a/AzureMonitorAgent/agent.py b/AzureMonitorAgent/agent.py
--- a/AzureMonitorAgent/agent.py
+++ b/AzureMonitorAgent/agent.py
@@ -689,7 +689,7 @@ def handle_mcs_config(public_settings, protected_settings, default_configs):
         else:
             log_and_exit("Enable", MissingorInvalidParameterErrorCode, 'Parameter "address" is required in proxy public setting')
 
-        if "auth" in public_settings.get("proxy") and public_settings.get("proxy").get("auth") == true:
+        if "auth" in public_settings.get("proxy") and public_settings.get("proxy").get("auth") == True:
             if protected_settings is not None and "proxy" in protected_settings and "username" in protected_settings.get("proxy") and "password" in protected_settings.get("proxy"):
                 default_configs["MDSD_PROXY_USERNAME"] = protected_settings.get("proxy").get("username")
                 default_configs["MDSD_PROXY_PASSWORD"] = protected_settings.get("proxy").get("password")


Test patch:
Repository: RDFLib/rdflib

Issue:
_PythonToXSD maps long to xsd:long which is wrong
In the [current version of terms.py](https://github.com/RDFLib/rdflib/blob/4c7c76c4e728e42d5be26802ccd04bc8df02745e/rdflib/term.py#L936) there's a mapping from python long to xsd:long. While xsd:long's have a maximum size of 9223372036854775807 according to http://www.w3.org/TR/xmlschema-2/#long that doesn't hold true for python longs, which can grow arbitrarily.

We should rather map python longs to xsd:integer: http://www.w3.org/TR/xmlschema-2/#integer

Patch upcoming.

Also see #217 for problems in the other direction.



Discussion:

Resolving patch:
diff --git a/rdflib/term.py b/rdflib/term.py
--- a/rdflib/term.py
+++ b/rdflib/term.py
@@ -933,7 +933,7 @@ def _castPythonToLiteral(obj):
     (float     , (None, URIRef(_XSD_PFX+'double'))),
     (bool      , (lambda i:str(i).lower(), URIRef(_XSD_PFX+'boolean'))),
     (int       , (None, URIRef(_XSD_PFX+'integer'))),
-    (long      , (None, URIRef(_XSD_PFX+'long'))),
+    (long      , (None, URIRef(_XSD_PFX+'integer'))),
     (Decimal   , (None, URIRef(_XSD_PFX+'decimal'))),
     (datetime  , (lambda i:i.isoformat(), URIRef(_XSD_PFX+'dateTime'))),
     (date      , (lambda i:i.isoformat(), URIRef(_XSD_PFX+'date'))),


Test patch: