internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/dataset-viewer

Issue:
Give insights to the user when we raise TooBigRows in /rows, /search, or /filter
https://huggingface.co/datasets/danielz01/xView2/discussions/2#655dc3ccde919d41622b51bd

Currently, we only show

```
Error code:   UnexpectedApiError
```

We should say that the rows contents are too heavy and that we don't show them for that reason (> 400MB in that case)


Discussion:

Resolving patch:
diff --git a/libs/libapi/src/libapi/exceptions.py b/libs/libapi/src/libapi/exceptions.py
--- a/libs/libapi/src/libapi/exceptions.py
+++ b/libs/libapi/src/libapi/exceptions.py
@@ -116,6 +116,15 @@ def __init__(self, message: str, cause: Optional[BaseException] = None):
         super().__init__(message, HTTPStatus.BAD_REQUEST, "SearchFeatureNotAvailableError", cause, True)
 
 
+class TooBigContentError(ApiError):
+    """The content size in bytes is bigger than the supported value."""
+
+    def __init__(self, message: str, cause: Optional[BaseException] = None):
+        super().__init__(
+            message, HTTPStatus.INTERNAL_SERVER_ERROR, "TransformRowsProcessingError", cause=cause, disclose_cause=True
+        )
+
+
 class TransformRowsProcessingError(ApiError):
     """There was an error when transforming rows to list."""
 
diff --git a/services/rows/src/rows/routes/rows.py b/services/rows/src/rows/routes/rows.py
--- a/services/rows/src/rows/routes/rows.py
+++ b/services/rows/src/rows/routes/rows.py
@@ -6,7 +6,7 @@
 
 from fsspec.implementations.http import HTTPFileSystem
 from libapi.authentication import auth_check
-from libapi.exceptions import ApiError, UnexpectedApiError
+from libapi.exceptions import ApiError, TooBigContentError, UnexpectedApiError
 from libapi.request import (
     get_request_parameter,
     get_request_parameter_length,
@@ -20,7 +20,7 @@
     get_json_ok_response,
     try_backfill_dataset_then_raise,
 )
-from libcommon.parquet_utils import Indexer
+from libcommon.parquet_utils import Indexer, TooBigRows
 from libcommon.processing_graph import ProcessingGraph
 from libcommon.prometheus import StepProfiler
 from libcommon.simple_cache import CachedArtifactError, CachedArtifactNotFoundError
@@ -96,7 +96,10 @@ async def rows_endpoint(request: Request) -> Response:
                         )
                         revision = rows_index.revision
                     with StepProfiler(method="rows_endpoint", step="query the rows"):
-                        pa_table = rows_index.query(offset=offset, length=length)
+                        try:
+                            pa_table = rows_index.query(offset=offset, length=length)
+                        except TooBigRows as err:
+                            raise TooBigContentError(str(err)) from None
                     with StepProfiler(method="rows_endpoint", step="transform to a list"):
                         response = await create_response(
                             dataset=dataset,


Test patch:
Repository: huggingface/dataset-viewer

Issue:
Don't ignore webhooks when a dataset changes visibility
See:
- https://github.com/huggingface/moon-landing/issues/8779
- https://github.com/huggingface/moon-landing/pull/8825

A webhook is sent when a dataset is toggle between public and private.

Currently, we ignore them due to 

https://github.com/huggingface/datasets-server/blob/66c1e089e204ab33195b957e1b99b0da6a4dd2de/services/api/src/api/routes/webhook.py#L84-L89

Instead, I think we should delete+update the dataset


Discussion:
It's currently a webhook with

```
    "event": {
        "action": "create",
        "scope": "repo"
    },
```

and it will soon be:

```
    "event": {
        "action": "update",
        "scope": "repo.config"
    },
```

let's wait for https://github.com/huggingface/moon-landing/pull/8825, then update our code
note for datasets-server, if they have a "global" webhook, they are not impacted by the current bug.


Resolving patch:
diff --git a/services/api/src/api/routes/webhook.py b/services/api/src/api/routes/webhook.py
--- a/services/api/src/api/routes/webhook.py
+++ b/services/api/src/api/routes/webhook.py
@@ -71,7 +71,7 @@ def process_payload(
     hf_timeout_seconds: Optional[float] = None,
     storage_clients: Optional[list[StorageClient]] = None,
 ) -> None:
-    if payload["repo"]["type"] != "dataset" or payload["scope"] not in ("repo", "repo.content"):
+    if payload["repo"]["type"] != "dataset" or payload["scope"] not in ("repo", "repo.content", "repo.config"):
         # ^ it filters out the webhook calls for non-dataset repos and discussions in dataset repos
         return
     dataset = payload["repo"]["name"]
@@ -81,7 +81,11 @@ def process_payload(
     if event == "remove":
         delete_dataset(dataset=dataset, storage_clients=storage_clients)
     elif event in ["add", "update", "move"]:
-        if event == "update" and get_current_revision(dataset) == payload["repo"]["headSha"]:
+        if (
+            event == "update"
+            and get_current_revision(dataset) == payload["repo"]["headSha"]
+            and not payload["scope"] == "repo.config"
+        ):
             # ^ it filters out the webhook calls when the refs/convert/parquet branch is updated
             logging.warning(
                 f"Webhook revision for {dataset} is the same as the current revision in the db - skipping update."


Test patch:
Repository: huggingface/dataset-viewer

Issue:
Retry jobs that finish with `ClientConnection` error?
Maybe here: https://github.com/huggingface/datasets-server/blob/f311a9212aaa91dd0373e5c2d4f5da9b6bdabcb5/chart/env/prod.yaml#L209

Internal conversation on Slack: https://huggingface.slack.com/archives/C0311GZ7R6K/p1698224875005729

Anyway: I'm wondering if we can have the error now that the dataset scripts are disabled by default.


Discussion:
we still have 103 occurrences of the error in the cache

https://github.com/huggingface/datasets-server/issues/1443#issuecomment-1929905798

```
{ _id: 'ConnectionError', count: 103 }
```

The errors are defined here:

https://github.com/huggingface/datasets-server/blob/c22f563f350a36148a1deaaf12531d4da991a677/libs/libcommon/src/libcommon/constants.py#L38-L44

Resolving patch:
diff --git a/libs/libcommon/src/libcommon/constants.py b/libs/libcommon/src/libcommon/constants.py
--- a/libs/libcommon/src/libcommon/constants.py
+++ b/libs/libcommon/src/libcommon/constants.py
@@ -36,6 +36,7 @@
 PARQUET_REVISION = "refs/convert/parquet"
 
 ERROR_CODES_TO_RETRY = {
+    "ConnectionError",
     "CreateCommitError",
     "ExternalServerError",
     "JobManagerCrashedError",


Test patch:
Repository: huggingface/dataset-viewer

Issue:
Replace `DatasetModuleNotInstalledError` errors with `DatasetWithScriptNotSupportedError`
We should never have a `DatasetModuleNotInstalledError` error, because we should return a `DatasetWithScriptNotSupportedError` error before

See https://github.com/huggingface/datasets-server/issues/1067#issuecomment-1924305954


Discussion:
Impact:

```
db.cachedResponsesBlue.countDocuments({error_code: "DatasetModuleNotInstalledError"})
989
db.cachedResponsesBlue.countDocuments({error_code: "DatasetWithScriptNotSupportedError"})
23593
```



Resolving patch:
diff --git a/services/worker/src/worker/job_runners/dataset/config_names.py b/services/worker/src/worker/job_runners/dataset/config_names.py
--- a/services/worker/src/worker/job_runners/dataset/config_names.py
+++ b/services/worker/src/worker/job_runners/dataset/config_names.py
@@ -75,18 +75,21 @@ def compute_config_names_response(
         ]
     except _EmptyDatasetError as err:
         raise EmptyDatasetError("The dataset is empty.", cause=err) from err
-    except ImportError as err:
-        raise DatasetModuleNotInstalledError(
-            "The dataset tries to import a module that is not installed.", cause=err
-        ) from err
-    except Exception as err:
-        if isinstance(err, ValueError) and "trust_remote_code" in str(err):
+    except ValueError as err:
+        if "trust_remote_code" in str(err):
             raise DatasetWithScriptNotSupportedError(
                 "The dataset viewer doesn't support this dataset because it runs "
                 "arbitrary python code. Please open a discussion in the discussion tab "
                 "if you think this is an error and tag @lhoestq and @severo."
             ) from err
         raise ConfigNamesError("Cannot get the config names for the dataset.", cause=err) from err
+    except ImportError as err:
+        # this should only happen if the dataset is in the allow list, which should soon disappear
+        raise DatasetModuleNotInstalledError(
+            "The dataset tries to import a module that is not installed.", cause=err
+        ) from err
+    except Exception as err:
+        raise ConfigNamesError("Cannot get the config names for the dataset.", cause=err) from err
 
     number_of_configs = len(config_name_items)
     if number_of_configs > max_number:


Test patch:
Repository: huggingface/dataset-viewer

Issue:
Upgrade ruff?
My VSCode plugin says:

```
Ruff >=0.2.1 required, but found 0.1.3 at /home/slesage/hf/datasets-server/services/worker/.venv/bin/ruff
```


Discussion:

Resolving patch:
diff --git a/services/search/src/search/config.py b/services/search/src/search/config.py
--- a/services/search/src/search/config.py
+++ b/services/search/src/search/config.py
@@ -18,9 +18,7 @@
 )
 
 DUCKDB_INDEX_CACHE_DIRECTORY = None
-DUCKDB_INDEX_TARGET_REVISION = (
-    "refs/convert/parquet"
-)  # TODO: Change to refs/convert/duckdb once all split-duckdb-index-010 entries have been computed
+DUCKDB_INDEX_TARGET_REVISION = "refs/convert/parquet"  # TODO: Change to refs/convert/duckdb once all split-duckdb-index-010 entries have been computed
 
 
 @dataclass(frozen=True)
diff --git a/services/worker/src/worker/config.py b/services/worker/src/worker/config.py
--- a/services/worker/src/worker/config.py
+++ b/services/worker/src/worker/config.py
@@ -267,8 +267,8 @@ def from_env(cls) -> "ConfigNamesConfig":
 DUCKDB_INDEX_URL_TEMPLATE = "/datasets/%s/resolve/%s/%s"
 DUCKDB_INDEX_EXTENSIONS_DIRECTORY: Optional[str] = None
 DUCKDB_INDEX_CLI_PATH = (
-    "/src/services/worker/duckdb"
-)  # TODO: Remove once all split-duckdb-index-010 entries have been computed
+    "/src/services/worker/duckdb"  # TODO: Remove once all split-duckdb-index-010 entries have been computed
+)
 
 
 @dataclass(frozen=True)
diff --git a/services/worker/src/worker/job_runners/config/parquet_and_info.py b/services/worker/src/worker/job_runners/config/parquet_and_info.py
--- a/services/worker/src/worker/job_runners/config/parquet_and_info.py
+++ b/services/worker/src/worker/job_runners/config/parquet_and_info.py
@@ -703,7 +703,7 @@ def write_table(self, pa_table: pa.Table, row_group_size: Optional[int] = None)
                 super().write_table(pa_table, row_group_size=row_group_size)
 
         def limited_generator(
-            generator: Callable[..., Generator[T, None, None]]
+            generator: Callable[..., Generator[T, None, None]],
         ) -> Callable[..., Generator[T, None, None]]:
             """Stop the underlying generator once we reach the maximum dataset size"""
 


Test patch:
diff --git a/services/worker/tests/fixtures/hub.py b/services/worker/tests/fixtures/hub.py
--- a/services/worker/tests/fixtures/hub.py
+++ b/services/worker/tests/fixtures/hub.py
@@ -948,7 +948,7 @@ def hub_responses_descriptive_statistics_string_text(
 
 @pytest.fixture
 def hub_responses_descriptive_statistics_parquet_builder(
-    hub_public_descriptive_statistics_parquet_builder: str
+    hub_public_descriptive_statistics_parquet_builder: str,
 ) -> HubDatasetTest:
     return {
         "name": hub_public_descriptive_statistics_parquet_builder,
Repository: huggingface/dataset-viewer

Issue:
/search and /filter are currently broken
Any search or filter is currently failing.

https://datasets-server.huggingface.co/search?dataset=gsarti%2Fflores_101&config=afr&split=devtest&offset=0&length=100&query=a
https://datasets-server.huggingface.co/filter?dataset=gsarti%2Fflores_101&config=afr&split=devtest&offset=0&length=100&where=id%3E%3D409+and+id%3C511

Logs:

```
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckdb-index'
```

when doing `disk_stat = os.statvfs(path)` here: 

https://github.com/huggingface/dataset-viewer/blob/a489c0b6ad4e5a78b2670679abbfab93f9be5742/libs/libapi/src/libapi/duckdb.py#L78


Discussion:
Maybe I'm wrong, but it seems it raises an uncaught error if the directory does not exist, which is the case when we create the machine... 

Resolving patch:
diff --git a/libs/libapi/src/libapi/duckdb.py b/libs/libapi/src/libapi/duckdb.py
--- a/libs/libapi/src/libapi/duckdb.py
+++ b/libs/libapi/src/libapi/duckdb.py
@@ -75,7 +75,12 @@ def get_download_folder(
 
 
 def check_available_disk_space(path: StrPath, required_space: int) -> None:
-    disk_stat = os.statvfs(path)
+    try:
+        disk_stat = os.statvfs(path)
+    except FileNotFoundError:
+        # The path does not exist, we create it and
+        init_dir(path)
+        disk_stat = os.statvfs(path)
     # Calculate free space in bytes
     free_space = disk_stat.f_bavail * disk_stat.f_frsize
     logging.debug(f"{free_space} available space, needed {required_space}")


Test patch:
diff --git a/libs/libapi/tests/test_duckdb.py b/libs/libapi/tests/test_duckdb.py
--- a/libs/libapi/tests/test_duckdb.py
+++ b/libs/libapi/tests/test_duckdb.py
@@ -5,7 +5,7 @@
 import pytest
 from pytest import TempPathFactory
 
-from libapi.duckdb import get_index_file_location_and_download_if_missing
+from libapi.duckdb import check_available_disk_space, get_index_file_location_and_download_if_missing
 
 
 @pytest.mark.parametrize("partial_index", [False, True])
@@ -57,3 +57,11 @@ def download_index_file(
         args, kwargs = download_mock.call_args
         assert not args
         assert kwargs["repo_file_location"] == expected_repo_file_location
+
+
+@pytest.mark.parametrize("subpath", [None, "does_not_exist"])
+def test_check_available_disk_space(tmp_path_factory: TempPathFactory, subpath: Optional[str]) -> None:
+    path = tmp_path_factory.mktemp("test_check_available_disk_space")
+    if subpath:
+        path = path / subpath
+    check_available_disk_space(path=path, required_space=1)
Repository: huggingface/dataset-viewer

Issue:
Truncate all the logs
We sometimes have very big logs (one row > 5MB). It's not useful at all and triggers warnings from infra. When we setup the logs configuration, we could try to set a maximum length

https://github.com/huggingface/dataset-viewer/blob/95527c2f1f0b8f077ed9ec74d3c75e45dbc1d00a/libs/libcommon/src/libcommon/log.py#L7-L9




Discussion:

Resolving patch:
diff --git a/libs/libcommon/src/libcommon/log.py b/libs/libcommon/src/libcommon/log.py
--- a/libs/libcommon/src/libcommon/log.py
+++ b/libs/libcommon/src/libcommon/log.py
@@ -5,5 +5,5 @@
 
 
 def init_logging(level: int = logging.INFO) -> None:
-    logging.basicConfig(level=level, format="%(levelname)s: %(asctime)s - %(name)s - %(message)s")
+    logging.basicConfig(level=level, format="%(levelname)s: %(asctime)s - %(name)s - %(message).5000s")
     logging.debug(f"Log level set to: {logging.getLevelName(logging.getLogger().getEffectiveLevel())}")


Test patch:
Repository: apluslms/a-plus

Issue:
Numbers should be safe characters in file names
If the attachment for the "Exercise with attachment" is named for example "2.zip", the name for that file will be just ".zip" in the database because numbers are not considered as safe characters:

https://github.com/Aalto-LeTech/a-plus/blob/a86bf/lib/helpers.py#L62



Discussion:

Resolving patch:
diff --git a/lib/helpers.py b/lib/helpers.py
--- a/lib/helpers.py
+++ b/lib/helpers.py
@@ -59,7 +59,7 @@ def has_same_domain(url1, url2):
     return uri1.netloc == uri2.netloc
 
 
-FILENAME_CHARS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ._-"
+FILENAME_CHARS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ._-0123456789"
 
 def safe_file_name(name):
     safename = "".join(c for c in name if c in FILENAME_CHARS)


Test patch:
Repository: apluslms/a-plus

Issue:
Different submission count between course "exercise results" page and assignment submission view
I spotted this in Aalto's production A+ (I'm a student enrolled on O1).

When I view the "exercise results" page, for one assignment I see the following (note the "0/10" submissions):

![Screenshot 2019-11-30 at 19 29 42](https://user-images.githubusercontent.com/6674046/69904228-cba34900-13ac-11ea-8b03-1c05dddb85e4.png)

When I follow the link to the chapter, and the view the assignment, it states I've used one submission (which is currently ungraded).
 
![Screenshot 2019-11-30 at 19 29 55](https://user-images.githubusercontent.com/6674046/69904238-e7a6ea80-13ac-11ea-8e30-17e91ea93eff.png)

It seems to only count graded submissions (this particular assignment is human marked).

This may be the intended behaviour, but it felt wrong that they were not consistent and it caused me a brief moment of intense "did I forget to submit the big assignment before the deadline" panic.


Discussion:
This commit is related to this issue:
https://github.com/apluslms/a-plus/commit/0a9fd39a92bad5cb4fd92fba4b6506b5e6f67d5f
Due to that change, only submissions in the ready status are counted. Submissions to Rubyric exercises tend to remain a long time in the waiting status since they are manually graded.

@alalav1 can start working on this.

Resolving patch:
diff --git a/exercise/cache/points.py b/exercise/cache/points.py
--- a/exercise/cache/points.py
+++ b/exercise/cache/points.py
@@ -87,7 +87,7 @@ def r_augment(children):
                 entry = tree[-1]
                 ready = submission.status == Submission.STATUS.READY
                 unofficial = submission.status == Submission.STATUS.UNOFFICIAL
-                if ready:
+                if ready or submission.status in (Submission.STATUS.WAITING, Submission.STATUS.INITIALIZED):
                     entry['submission_count'] += 1
                 entry['submissions'].append({
                     'id': submission.id,


Test patch:
Repository: apluslms/a-plus

Issue:
The exercise results page does not show that a module is open for reading while the exercises are still closed.
The UI leaves the impression that the module is completely closed (between the reading opening time and the opening time). The "opens" badge in the module box also only shows the normal open time. The UI should show clearly that a module is open for reading. The collapsing box should be opened by default for read-open modules.


Discussion:
@alalav1 you can work on this.

Resolving patch:
diff --git a/exercise/templatetags/exercise.py b/exercise/templatetags/exercise.py
--- a/exercise/templatetags/exercise.py
+++ b/exercise/templatetags/exercise.py
@@ -223,8 +223,11 @@ def min_group_size(context):
 
 @register.simple_tag(takes_context=True)
 def module_accessible(context, entry):
-    t = entry.get('reading_opening_time', entry.get('opening_time'))
-    return _is_accessible(context, entry, t)
+    t = entry.get('reading_opening_time')
+    if t:
+        return _is_accessible(context, entry, t)
+    else:
+        return exercise_accessible(context, entry)
 
 
 @register.simple_tag(takes_context=True)
@@ -293,4 +296,3 @@ def get_format_info(format):
 @register.simple_tag
 def get_format_info_list(formats):
     return [get_format_info(format) for format in formats.split()]
-


Test patch:
Repository: apluslms/a-plus

Issue:
auth: hide maintenance login from login page
The maintenance login form shouldn't be visible by default, when there are other authentication boxes (Shibboleth or social).

So, hide the login div behind a button, a link or a collapse.


Discussion:

Resolving patch:
diff --git a/aplus/settings.py b/aplus/settings.py
--- a/aplus/settings.py
+++ b/aplus/settings.py
@@ -32,13 +32,10 @@
 
 WELCOME_TEXT = 'Welcome to A+ <small>modern learning environment</small>'
 SHIBBOLETH_TITLE_TEXT = 'Aalto University users'
-SHIBBOLETH_BODY_TEXT = 'Log in with Aalto University user account by clicking the button below. Programme students and faculty must login here.'
-SHIBBOLETH_BUTTON_TEXT = 'Aalto Login'
+SHIBBOLETH_BODY_TEXT = 'Log in with Aalto University user account by clicking the button below. Programme students and faculty must log in here.'
+SHIBBOLETH_BUTTON_TEXT = 'Aalto login'
 MOOC_TITLE_TEXT = 'Users external to Aalto'
-MOOC_BODY_TEXT = 'Some of our courses are open for everyone. Login with your user account from one of the following services.'
-LOGIN_TITLE_TEXT = ''
-LOGIN_BODY_TEXT = ''
-LOGIN_BUTTON_TEXT = 'Maintenance login'
+MOOC_BODY_TEXT = 'Some of our courses are open for everyone. Log in with your user account from one of the following services.'
 INTERNAL_USER_LABEL = 'Aalto'
 EXTERNAL_USER_LABEL = 'MOOC'
 
@@ -48,9 +45,6 @@
 SHIBBOLETH_BUTTON_TEXT_FI = 'Aalto-kirjautuminen'
 MOOC_TITLE_TEXT_FI = 'Käyttäjät Aallon ulkopuolelta'
 MOOC_BODY_TEXT_FI = 'Osa kursseistamme on avoinna kaikille. Kirjaudu sisään jonkin seuraavan palvelun käyttäjätunnuksellasi.'
-LOGIN_TITLE_TEXT_FI = ''
-LOGIN_BODY_TEXT_FI = ''
-LOGIN_BUTTON_TEXT_FI = 'Ylläpidon kirjautuminen'
 
 TRACKING_HTML = ''
 
diff --git a/userprofile/views.py b/userprofile/views.py
--- a/userprofile/views.py
+++ b/userprofile/views.py
@@ -26,6 +26,7 @@ class CustomLoginView(LoginView):
     extra_context = {
         'shibboleth_login': 'shibboleth_login' in settings.INSTALLED_APPS,
         'mooc_login': 'social_django' in settings.INSTALLED_APPS,
+        'brand_name': settings_text('BRAND_NAME'),
     }
 
     def get_context_data(self, **kwargs):
@@ -37,9 +38,6 @@ def get_context_data(self, **kwargs):
         # in the class level, but there is a request when this method is called
         # (self.request).
         context.update({
-            'login_title_text': settings_text('LOGIN_TITLE_TEXT'),
-            'login_body_text': settings_text('LOGIN_BODY_TEXT'),
-            'login_button_text': settings_text('LOGIN_BUTTON_TEXT'),
             'shibboleth_title_text': settings_text('SHIBBOLETH_TITLE_TEXT'),
             'shibboleth_body_text': settings_text('SHIBBOLETH_BODY_TEXT'),
             'shibboleth_button_text': settings_text('SHIBBOLETH_BUTTON_TEXT'),


Test patch: