internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: spectrochempy/spectrochempy

Issue:
Automate building of docs for new release and dev version.
Author: @fernandezc (Christian Fernandez )
Date: 2020-05-28
Redmine Issue: 74, https://redmine.spectrochempy.fr/issues/74

---

None




Discussion:

Resolving patch:
diff --git a/spectrochempy/core/readers/readopus.py b/spectrochempy/core/readers/readopus.py
--- a/spectrochempy/core/readers/readopus.py
+++ b/spectrochempy/core/readers/readopus.py
@@ -22,10 +22,7 @@
 from brukeropusreader import read_file
 from warnings import warn
 from datetime import datetime, timezone, timedelta
-
-
-
-
+from numpy import linspace
 
 # ----------------------------------------------------------------------------------------------------------------------
 # third party imports
@@ -37,6 +34,8 @@
 from spectrochempy.core.dataset.nddataset import NDDataset
 from spectrochempy.core.dataset.ndcoord import Coord
 from spectrochempy.utils import readfilename
+
+
 # ======================================================================================================================
 # Public functions
 # ======================================================================================================================
@@ -90,7 +89,7 @@ def read_opus(dataset=None, **kwargs):
                          filetypes=['Bruker files (*.*)',
                                     'all files (*)'],
                          dictionary=False)
-    #todo: see how to use regular expression in Qt filters
+    # todo: see how to use regular expression in Qt filters
 
     if not files:
         # there is no files, return nothing
@@ -109,13 +108,18 @@ def read_opus(dataset=None, **kwargs):
             warn("opus file {} could not be read".format(file))
             continue
 
+        npt = opus_data['AB Data Parameter']['NPT']
+        fxv = opus_data['AB Data Parameter']['FXV']
+        lxv = opus_data['AB Data Parameter']['LXV']
+        xdata = linspace(fxv, lxv, npt)
+
         if not xaxis:
-            xaxis = Coord(opus_data.get_range("AB"), title='Wavenumbers', units='cm^-1')
+            xaxis = Coord(x=xdata, title='Wavenumbers', units='cm^-1')
 
-        elif (opus_data.get_range("AB") != xaxis.data).any():
+        elif (xdata != xaxis.data).any():
             raise ValueError("spectra have incompatible dimensions (xaxis)")
 
-        intensities.append(opus_data["AB"])
+        intensities.append(opus_data["AB"][:npt])
         names.append(opus_data["Sample"]['SNM'])
         acqdate = opus_data["AB Data Parameter"]["DAT"]
         acqtime = opus_data["AB Data Parameter"]["TIM"]


Test patch:
Repository: spectrochempy/spectrochempy

Issue:
Fix unit display
Since the recent version of Pint, we get problems in HTML display both in docs and in jupyter notebooks. 
This is linked with the "H" format that strangely return non html string. Try some example from the Pint doc
shows the same problem in notebook.  


Discussion:

Resolving patch:
diff --git a/spectrochempy/units/units.py b/spectrochempy/units/units.py
--- a/spectrochempy/units/units.py
+++ b/spectrochempy/units/units.py
@@ -39,7 +39,7 @@ def _pretty_fmt_exponent(num):
     """
     # work badly for decimals as superscript dot do not exist in unicode
     # (as far as we know)
-    ret = '{0:n}'.format(num).replace('-', '⁻').replace('.', u"\u2027")
+    ret = '{0:n}'.format(num).replace('-', '⁻').replace('.', "\u22C5")
     for n in range(10):
         ret = ret.replace(str(n), _PRETTY_EXPONENTS[n])
     return ret
@@ -85,7 +85,14 @@ def _pretty_fmt_exponent(num):
 formatting._FORMATS.update(formats)
 formatting._KNOWN_TYPES = frozenset(list(formatting._FORMATS.keys()) + ['~'])
 
-setattr(Quantity, '_repr_html_', lambda cls: cls.__format__('~H'))
+def _repr_html_(cls):
+    p = cls.__format__('~H')
+    # attempt to solve a display problem in notebook (recent version of pint
+    # have a strange way to handle HTML. For me it doen't work
+    p = p.replace(r'\[','').replace(r'\]','').replace(r'\ ',' ')
+    return p
+
+setattr(Quantity, '_repr_html_', _repr_html_)
 setattr(Quantity, '_repr_latex_', lambda cls: "$" + cls.__format__('~L') + "$")
 
 # TODO: work on this latex format
@@ -130,7 +137,7 @@ def __format__(self, spec):
         units = self._units
 
     if "H" in spec:
-        # HTML / Jupyter Notebook
+        # HTML / Jupyter Notebook (
         return r"\[" + format(units, spec).replace(" ", r"\ ") + r"\]"
 
     return '%s' % (format(units, spec))


Test patch:
Repository: spectrochempy/spectrochempy

Issue:
[FEATURE]: scp.write(): set confirm=False as default
### Is your feature request related to a problem?

`write()` opens a dialog box (through a call to `check_filename_to_save()` in utils/file.py , unless `confirm=False` is passed (and `scp.NO_DIALOG` is normally set to `False` except when testing or building the doc).   

### Describe the solution you'd like

When a filename is passed, `write()` should behave as in pandas or numpy, including overwriting the file if it exists. A dialog box should pop up only if no filename is provided. By setting `confirm=False` as the default, one has always the possibility to set it to True if unsure of the path/filename and/or use a test, to warn the enduser e.g.

```
if filename_exists:
   warn(f'{filename} exists and will be overwitten. Please confirm')
   datase.write('dataset.csv', confirm=True)
else
   datase.write('dataset.csv')
```



### Describe alternatives you've considered

 When a filename is passed, a warning+dialog box could be raised by default only if the filename already exists AND confirm=True ? , but having a default behaviour close to pandas or numpy is probably better ?   

### Additional context

_No response_


Discussion:

Resolving patch:
diff --git a/spectrochempy/utils/file.py b/spectrochempy/utils/file.py
--- a/spectrochempy/utils/file.py
+++ b/spectrochempy/utils/file.py
@@ -595,6 +595,7 @@ def check_filename_to_save(
 ):
 
     from spectrochempy import NO_DIALOG
+    from spectrochempy.core import info_
 
     NODIAL = NO_DIALOG or "DOC_BUILDING" in environ
 
@@ -603,17 +604,28 @@ def check_filename_to_save(
 
     if not filename or save_as or filename.exists():
 
+        from spectrochempy.core import save_dialog
+
         # no filename provided
         if filename is None or (NODIAL and pathclean(filename).is_dir()):
             filename = dataset.name
             filename = filename + kwargs.get("suffix", ".scp")
+            caption = "Save as ..."
+            open_diag = True
+
+        # existing filename provided
+        elif filename.exists():
+            if confirm:
+                caption = "File exists. Confirm overwrite"
+                open_diag = True
+            else:
+                info_(f"A file {filename} was present and has been overwritten.")
+                open_diag = False
 
-        if not NODIAL and confirm:
-
-            from spectrochempy.core import save_dialog
+        if not NODIAL and open_diag:
 
             filename = save_dialog(
-                caption=kwargs.pop("caption", "Save as ..."),
+                caption=kwargs.pop("caption", caption),
                 filename=filename,
                 filters=kwargs.pop("filetypes", ["All file types (*.*)"]),
                 **kwargs,


Test patch:
Repository: spectrochempy/spectrochempy

Issue:
[Bug]: The documentation does not correctly indicate the last release information
### What happened?

should show `whatsnew` correctly

### What did you expect to happen?

_No response_

### Minimal Complete Verifiable Example

_No response_

### Relevant log output

_No response_

### Anything else we need to know?

_No response_

### Environment

N.A.


Discussion:

Resolving patch:
diff --git a/.ci/update_version_and_release_notes.py b/.ci/update_version_and_release_notes.py
--- a/.ci/update_version_and_release_notes.py
+++ b/.ci/update_version_and_release_notes.py
@@ -165,7 +165,7 @@ def make_zenodo(version):
 
 def make_release_note_index(revision):
 
-    # remove old rev files
+    # remove old last revision files
     files = WN.glob("v*.dev*.rst")
     for file in files:
         file.unlink()
@@ -207,6 +207,8 @@ def make_release_note_index(revision):
         # void changelog (keep only section titles)
         (WN / "changelog.rst").write_text(
             """
+:orphan:
+
 What's new in revision {{ revision }}
 ---------------------------------------------------------------------------------------
 


Test patch:
Repository: spectrochempy/spectrochempy

Issue:
[Bug]: show_versions() not working.
### What happened?

```
In [3]: scp.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:10:28) [Clang 15.0.7 ]
python-bits: 64
OS: Darwin
OS-release: 21.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
LOCALE: ('en_US', 'ISO8859-1')

 ERROR | FileNotFoundError: [Errno 2] No such file or directory: '/Users/christian/micromamba/envs/scpy-base/lib/python3.11/site-packages/requirements/requirements_dev.txt'
```


### What did you expect to happen?

_No response_

### Minimal Complete Verifiable Example

_No response_

### Relevant log output

_No response_

### Anything else we need to know?

_No response_

### Environment

latest and previous release.


Discussion:

Resolving patch:
diff --git a/docs/userguide/analysis/baseline.py b/docs/userguide/analysis/baseline.py
--- a/docs/userguide/analysis/baseline.py
+++ b/docs/userguide/analysis/baseline.py
@@ -8,7 +8,7 @@
 #       extension: .py
 #       format_name: percent
 #       format_version: '1.3'
-#       jupytext_version: 1.14.5
+#       jupytext_version: 1.15.2
 #   kernelspec:
 #     display_name: Python 3 (ipykernel)
 #     language: python
@@ -22,7 +22,7 @@
 #     name: python
 #     nbconvert_exporter: python
 #     pygments_lexer: ipython3
-#     version: 3.10.11
+#     version: 3.11.5
 # ---
 
 # %% [markdown]
diff --git a/spectrochempy/utils/print_versions.py b/spectrochempy/utils/print_versions.py
--- a/spectrochempy/utils/print_versions.py
+++ b/spectrochempy/utils/print_versions.py
@@ -92,7 +92,18 @@ def show_versions(file=sys.stdout):
 
     # dependencies
     deps = []
-    reqs = Path(__file__).parent.parent.parent / "requirements" / "requirements_dev.txt"
+
+    try:
+        # dev installation
+        reqs = (
+            Path(__file__).parent.parent.parent
+            / "requirements"
+            / "requirements_dev.txt"
+        )
+        print(" -- dev installation --")
+    except FileNotFoundError:
+        reqs = Path(__file__).parent.parent.parent / "requirements" / "requirements.txt"
+
     reqs = reqs.read_text().split("\n")
     for req in reqs:
         req = req.strip()


Test patch:
Repository: huggingface/dataset-viewer

Issue:
regression: fallback if streaming fails is disabled
Causes https://github.com/huggingface/datasets/issues/3185 for example: the fallback should have loaded the dataset in normal mode.


Discussion:

Resolving patch:
diff --git a/src/datasets_preview_backend/config.py b/src/datasets_preview_backend/config.py
--- a/src/datasets_preview_backend/config.py
+++ b/src/datasets_preview_backend/config.py
@@ -12,6 +12,7 @@
     DEFAULT_LOG_LEVEL,
     DEFAULT_MAX_AGE_LONG_SECONDS,
     DEFAULT_MAX_AGE_SHORT_SECONDS,
+    DEFAULT_MAX_SIZE_FALLBACK,
     DEFAULT_MONGO_CACHE_DATABASE,
     DEFAULT_MONGO_QUEUE_DATABASE,
     DEFAULT_MONGO_URL,
@@ -50,6 +51,7 @@
 os.environ["HF_SCRIPTS_VERSION"] = DATASETS_REVISION
 
 # for tests - to be removed
+MAX_SIZE_FALLBACK = get_int_value(os.environ, "MAX_SIZE_FALLBACK", DEFAULT_MAX_SIZE_FALLBACK)
 ROWS_MAX_BYTES = get_int_value(d=os.environ, key="ROWS_MAX_BYTES", default=DEFAULT_ROWS_MAX_BYTES)
 ROWS_MAX_NUMBER = get_int_value(d=os.environ, key="ROWS_MAX_NUMBER", default=DEFAULT_ROWS_MAX_NUMBER)
 ROWS_MIN_NUMBER = get_int_value(d=os.environ, key="ROWS_MIN_NUMBER", default=DEFAULT_ROWS_MIN_NUMBER)
diff --git a/src/datasets_preview_backend/models/row.py b/src/datasets_preview_backend/models/row.py
--- a/src/datasets_preview_backend/models/row.py
+++ b/src/datasets_preview_backend/models/row.py
@@ -28,7 +28,7 @@ def get_rows(
         dataset_name,
         name=config_name,
         split=split_name,
-        streaming=True,
+        streaming=streaming,
         download_mode=DownloadMode.FORCE_REDOWNLOAD,
         use_auth_token=hf_token,
     )


Test patch:
diff --git a/tests/models/test_split.py b/tests/models/test_split.py
--- a/tests/models/test_split.py
+++ b/tests/models/test_split.py
@@ -1,4 +1,4 @@
-from datasets_preview_backend.config import HF_TOKEN, ROWS_MAX_NUMBER
+from datasets_preview_backend.config import HF_TOKEN, MAX_SIZE_FALLBACK, ROWS_MAX_NUMBER
 from datasets_preview_backend.models.split import get_split
 
 # TODO: test fallback
@@ -24,4 +24,21 @@ def test_gated() -> None:
     assert split["rows_response"]["rows"][0]["row"]["year"] == "1855"
 
 
+def test_fallback() -> None:
+    # https://github.com/huggingface/datasets/issues/3185
+    dataset_name = "samsum"
+    config_name = "samsum"
+    split_name = "train"
+    split = get_split(
+        dataset_name,
+        config_name,
+        split_name,
+        HF_TOKEN,
+        rows_max_number=ROWS_MAX_NUMBER,
+        max_size_fallback=MAX_SIZE_FALLBACK,
+    )
+
+    assert len(split["rows_response"]["rows"]) == ROWS_MAX_NUMBER
+
+
 # TODO: test the truncation
Repository: huggingface/dataset-viewer

Issue:
Use main instead of master to load the datasets
The main branch in https://github.com/huggingface/datasets is now `main`, not `master` anymore. Note that it's backward compatible, so no need to hurry


Discussion:

Resolving patch:
diff --git a/services/worker/src/worker/constants.py b/services/worker/src/worker/constants.py
--- a/services/worker/src/worker/constants.py
+++ b/services/worker/src/worker/constants.py
@@ -2,7 +2,7 @@
 
 DEFAULT_ASSETS_BASE_URL: str = "assets"
 DEFAULT_ASSETS_DIRECTORY: None = None
-DEFAULT_DATASETS_REVISION: str = "master"
+DEFAULT_DATASETS_REVISION: str = "main"
 DEFAULT_HF_TOKEN: Optional[str] = None
 DEFAULT_LOG_LEVEL: str = "INFO"
 DEFAULT_MAX_JOB_RETRIES: int = 3


Test patch:
Repository: huggingface/dataset-viewer

Issue:
Install a stale bot



Discussion:

Resolving patch:
diff --git a/tools/stale.py b/tools/stale.py
new file mode 100644
--- /dev/null
+++ b/tools/stale.py
@@ -0,0 +1,63 @@
+# Copyright 2021 The HuggingFace Team, the AllenNLP library authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Script to close stale issue. Taken in part from the AllenNLP repository.
+https://github.com/allenai/allennlp.
+Copied from https://github.com/huggingface/transformers
+"""
+from datetime import datetime as dt
+import os
+
+from github import Github
+
+
+LABELS_TO_EXEMPT = [
+    "good first issue",
+    "feature request",
+]
+
+
+def main():
+    g = Github(os.environ["GITHUB_TOKEN"])
+    repo = g.get_repo("huggingface/datasets-server")
+    open_issues = repo.get_issues(state="open")
+
+    for issue in open_issues:
+        comments = sorted([comment for comment in issue.get_comments()], key=lambda i: i.created_at, reverse=True)
+        last_comment = comments[0] if len(comments) > 0 else None
+        if (
+            last_comment is not None and last_comment.user.login == "github-actions[bot]"
+            and (dt.utcnow() - issue.updated_at).days > 7
+            and (dt.utcnow() - issue.created_at).days >= 30
+            and not any(label.name.lower() in LABELS_TO_EXEMPT for label in issue.get_labels())
+        ):
+            # print(f"Would close issue {issue.number} since it has been 7 days of inactivity since bot mention.")
+            issue.edit(state="closed")
+        elif (
+            (dt.utcnow() - issue.updated_at).days > 23
+            and (dt.utcnow() - issue.created_at).days >= 30
+            and not any(label.name.lower() in LABELS_TO_EXEMPT for label in issue.get_labels())
+        ):
+            # print(f"Would add stale comment to {issue.number}")
+            issue.create_comment(
+                "This issue has been automatically marked as stale because it has not had "
+                "recent activity. If you think this still needs to be addressed "
+                "please comment on this thread.\n\nPlease note that issues that do not follow the "
+                "[contributing guidelines](https://github.com/huggingface/datasets-server/blob/main/CONTRIBUTING.md) "
+                "are likely to be ignored."
+            )
+
+
+if __name__ == "__main__":
+    main()


Test patch:
Repository: huggingface/dataset-viewer

Issue:
No specific error when dataset tries to import a non-installed module
When a dataset script tries to import a module/library that is not installed, there is no informative error message.

See:
- #1067 
- #1068

Related to:
- #976


Discussion:

Resolving patch:
diff --git a/services/worker/src/worker/job_runners/config_names.py b/services/worker/src/worker/job_runners/config_names.py
--- a/services/worker/src/worker/job_runners/config_names.py
+++ b/services/worker/src/worker/job_runners/config_names.py
@@ -13,7 +13,7 @@
 from worker.job_runner import CompleteJobResult, JobRunnerError, ParameterMissingError
 from worker.job_runners._datasets_based_job_runner import DatasetsBasedJobRunner
 
-ConfigNamesJobRunnerErrorCode = Literal["EmptyDatasetError", "ConfigNamesError"]
+ConfigNamesJobRunnerErrorCode = Literal["EmptyDatasetError", "DatasetModuleNotInstalledError", "ConfigNamesError"]
 
 
 class ConfigNamesJobRunnerError(JobRunnerError):
@@ -39,6 +39,13 @@ def __init__(self, message: str, cause: Optional[BaseException] = None):
         super().__init__(message, HTTPStatus.INTERNAL_SERVER_ERROR, "EmptyDatasetError", cause, True)
 
 
+class DatasetModuleNotInstalledError(ConfigNamesJobRunnerError):
+    """Raised when the dataset tries to import a module that is not installed."""
+
+    def __init__(self, message: str, cause: Optional[BaseException] = None):
+        super().__init__(message, HTTPStatus.INTERNAL_SERVER_ERROR, "DatasetModuleNotInstalledError", cause, True)
+
+
 class ConfigNamesError(ConfigNamesJobRunnerError):
     """Raised when the config names could not be fetched."""
 
@@ -77,6 +84,8 @@ def compute_config_names_response(
     Raises the following errors:
         - [`~job_runners.config_names.EmptyDatasetError`]
           The dataset is empty.
+        - [`~job_runners.config_names.DatasetModuleNotInstalledError`]
+          The dataset tries to import a module that is not installed.
         - [`~job_runners.config_names.ConfigNamesError`]
           If the list of configs could not be obtained using the datasets library.
     </Tip>
@@ -91,6 +100,10 @@ def compute_config_names_response(
         ]
     except _EmptyDatasetError as err:
         raise EmptyDatasetError("The dataset is empty.", cause=err) from err
+    except ImportError as err:
+        raise DatasetModuleNotInstalledError(
+            "The dataset tries to import a module that is not installed.", cause=err
+        ) from err
     except Exception as err:
         raise ConfigNamesError("Cannot get the config names for the dataset.", cause=err) from err
     return ConfigNamesResponse(config_names=config_name_items)


Test patch:
Repository: huggingface/dataset-viewer

Issue:
Better handling of k8s SIGTERM when stopping workers
We've observed recently that worker pods can take more than 1h to get killed by kubernetes (see internal [slack thread](https://huggingface.slack.com/archives/C04L6P8KNQ5/p1697641144036899) for example).
According to the k8s docs, at one point it sends a TERM and then after a grace period a SIGKILL on all the running processes.
If this keeps happening we need to improve the way we handle those to kill the pods faster.

One idea is to catch SIGTERM in the worker executor (main python process of workers) that would stop the worker loop (its subprocess where the actual code runs).


Discussion:
previous reference in Slack, by @rtrompier : https://huggingface.slack.com/archives/C04L6P8KNQ5/p1691002860789569?thread_ts=1690989017.253219&cid=C04L6P8KNQ5 (internal)


closely related to https://github.com/huggingface/datasets-server/issues/1606

Resolving patch:
diff --git a/services/worker/src/worker/executor.py b/services/worker/src/worker/executor.py
--- a/services/worker/src/worker/executor.py
+++ b/services/worker/src/worker/executor.py
@@ -3,6 +3,7 @@
 import asyncio
 import logging
 import os
+import signal
 import sys
 from collections.abc import Callable
 from datetime import datetime, timedelta
@@ -89,6 +90,7 @@ def custom_exception_handler(loop: asyncio.AbstractEventLoop, context: dict[str,
                 loop.stop()
 
         loop = asyncio.get_event_loop()
+        loop.add_signal_handler(signal.SIGTERM, self.stop, worker_loop_executor)
         loop.set_exception_handler(custom_exception_handler)
         logging.info("Starting heartbeat.")
         loop.create_task(every(self.heartbeat, seconds=self.heartbeat_interval_seconds))
@@ -117,6 +119,9 @@ def custom_exception_handler(loop: asyncio.AbstractEventLoop, context: dict[str,
         if exceptions:
             raise RuntimeError(f"Some async tasks failed: {exceptions}")
 
+    def stop(self, worker_loop_executor: OutputExecutor) -> None:
+        worker_loop_executor.stop()
+
     def get_state(self) -> Optional[WorkerState]:
         worker_state_file_path = self.state_file_path
         if not os.path.exists(worker_state_file_path):


Test patch: