Repository: huggingface/dataset-viewer

Issue:
"Errors to retry" only retried once a day
It's strange: `JobManagerCrashedError` is listed in the errors to retry, but I witnessed a dataset where it was not retried (`failed_runs: 0`, no jobs).

See discussion opened here: https://huggingface.co/datasets/Samuel-Martineau/kryptik/discussions/1

<img width="1485" alt="Capture d’écran 2024-02-12 à 12 53 36" src="https://github.com/huggingface/datasets-server/assets/1676121/5df7efeb-593c-43e0-b920-7c4918805756">



Discussion:
OK, it's a very small proportion of the cases: only the ones from today. They will be retried in the daily backfill. 

```
db.cachedResponsesBlue.countDocuments({error_code: "JobManagerCrashedError", "details.copied_from_artifact": {$exists: false}})
4800
db.cachedResponsesBlue.countDocuments({error_code: "JobManagerCrashedError", "details.copied_from_artifact": {$exists: false}, failed_runs: 0})
32
db.cachedResponsesBlue.find({error_code: "JobManagerCrashedError", "details.copied_from_artifact": {$exists: false}, failed_runs: 0}).sort({updated_at: 1}).limit(1)
...
  updated_at: 2024-02-11T23:15:54.489Z }
```
The errors to retry are:

```
ERROR_CODES_TO_RETRY = {
    "ConnectionError",
    "CreateCommitError",
    "ExternalServerError",
    "JobManagerCrashedError",
    "LockedDatasetTimeoutError",
    "StreamingRowsError",
}
```

They are retried only in the daily backfill.

I think we should check in `finish_job()`, to retry them (create a new job) immediately if needed. Otherwise, it's unclear for the user.
The original dataset is not public anymore. Here was my comment in the discussion @Samuel-Martineau.

> For some reason, we're not able to generate the Parquet files from your dataset, and the worker crashes.
> 
> I opened an issue on a related bug (https://github.com/huggingface/datasets-server/issues/2439): we should retry earlier in case of crash. But anyway, I did it manually on your dataset and it still crashes. Maybe @albertvillanova @lhoestq have more clues about why the worker is crashing, and what could be done to fix it?
Hi, thanks for your quick response. I deleted the dataset, as I only needed it to test a few things about Hugging Face. If you believe it would help if it were back online, I can do that tonight.
No, not needed. Feel free to open a discussion if you encounter another issue with your datasets.
Another example: https://huggingface.co/datasets/CohereForAI/aya_collection/discussions/7
On the other hand, it helps in case of temporary failure of the Hub: if we retry immediately until reaching the maximum number of retries, we will always stay with an error.
With the current implementation, we wait one day between two retries. The conditions that allowed the failure might have changed, and we might end up with a success that time.
A (lot too complex) alternative would be to allow the user to re-run the missing steps from the UI, as in GitHub's CI with failed jobs.
Another alternative would be to create a new cron job, run every 10m or hour for example, that would only target these error codes and try to backfill the affected datasets. It would concern only a fraction of the datasets, and thus should only take a fraction of the 6h required for the daily backfill

Resolving patch:
diff --git a/jobs/cache_maintenance/src/cache_maintenance/backfill.py b/jobs/cache_maintenance/src/cache_maintenance/backfill.py
--- a/jobs/cache_maintenance/src/cache_maintenance/backfill.py
+++ b/jobs/cache_maintenance/src/cache_maintenance/backfill.py
@@ -7,7 +7,7 @@
 
 from libcommon.dtos import Priority
 from libcommon.operations import OperationsStatistics, backfill_dataset
-from libcommon.simple_cache import get_all_datasets
+from libcommon.simple_cache import get_all_datasets, get_datasets_with_retryable_errors
 from libcommon.storage_client import StorageClient
 
 
@@ -35,7 +35,7 @@ def get_log(self) -> str:
         )
 
 
-def backfill_cache(
+def backfill_all_datasets(
     hf_endpoint: str,
     blocked_datasets: list[str],
     hf_token: Optional[str] = None,
@@ -43,11 +43,44 @@ def backfill_cache(
 ) -> None:
     logging.info("backfill datasets in the database and delete non-supported ones")
     datasets_in_database = get_all_datasets()
-    logging.info(f"analyzing {len(datasets_in_database)} datasets in the database")
-    statistics = BackfillStatistics(num_total_datasets=len(datasets_in_database))
+    backfill_datasets(
+        dataset_names=datasets_in_database,
+        hf_endpoint=hf_endpoint,
+        blocked_datasets=blocked_datasets,
+        hf_token=hf_token,
+        storage_clients=storage_clients,
+    )
+
+
+def backfill_retryable_errors(
+    hf_endpoint: str,
+    blocked_datasets: list[str],
+    hf_token: Optional[str] = None,
+    storage_clients: Optional[list[StorageClient]] = None,
+) -> None:
+    logging.info("backfill datasets that have a retryable error")
+    dataset_names = get_datasets_with_retryable_errors()
+    backfill_datasets(
+        dataset_names=dataset_names,
+        hf_endpoint=hf_endpoint,
+        blocked_datasets=blocked_datasets,
+        hf_token=hf_token,
+        storage_clients=storage_clients,
+    )
+
+
+def backfill_datasets(
+    dataset_names: set[str],
+    hf_endpoint: str,
+    blocked_datasets: list[str],
+    hf_token: Optional[str] = None,
+    storage_clients: Optional[list[StorageClient]] = None,
+) -> None:
+    logging.info(f"analyzing {len(dataset_names)} datasets in the database")
+    statistics = BackfillStatistics(num_total_datasets=len(dataset_names))
     log_batch = 100
 
-    for dataset in datasets_in_database:
+    for dataset in dataset_names:
         try:
             statistics.add(
                 BackfillStatistics(
diff --git a/jobs/cache_maintenance/src/cache_maintenance/main.py b/jobs/cache_maintenance/src/cache_maintenance/main.py
--- a/jobs/cache_maintenance/src/cache_maintenance/main.py
+++ b/jobs/cache_maintenance/src/cache_maintenance/main.py
@@ -10,7 +10,7 @@
 from libcommon.storage import init_dir
 from libcommon.storage_client import StorageClient
 
-from cache_maintenance.backfill import backfill_cache
+from cache_maintenance.backfill import backfill_all_datasets, backfill_retryable_errors
 from cache_maintenance.cache_metrics import collect_cache_metrics
 from cache_maintenance.clean_directory import clean_directory
 from cache_maintenance.config import JobConfig
@@ -36,7 +36,7 @@ def run_job() -> None:
         ) as queue_resource,
     ):
         start_time = datetime.now()
-        if action == "backfill":
+        if action in ("backfill", "backfill-retryable-errors"):
             if not cache_resource.is_available():
                 logging.warning(
                     "The connection to the cache database could not be established. The action is skipped."
@@ -61,12 +61,20 @@ def run_job() -> None:
                 s3_config=job_config.s3,
                 # no need to specify a url_signer
             )
-            backfill_cache(
-                hf_endpoint=job_config.common.hf_endpoint,
-                hf_token=job_config.common.hf_token,
-                blocked_datasets=job_config.common.blocked_datasets,
-                storage_clients=[cached_assets_storage_client, assets_storage_client],
-            )
+            if action == "backfill":
+                backfill_all_datasets(
+                    hf_endpoint=job_config.common.hf_endpoint,
+                    hf_token=job_config.common.hf_token,
+                    blocked_datasets=job_config.common.blocked_datasets,
+                    storage_clients=[cached_assets_storage_client, assets_storage_client],
+                )
+            else:
+                backfill_retryable_errors(
+                    hf_endpoint=job_config.common.hf_endpoint,
+                    hf_token=job_config.common.hf_token,
+                    blocked_datasets=job_config.common.blocked_datasets,
+                    storage_clients=[cached_assets_storage_client, assets_storage_client],
+                )
         elif action == "clean-directory":
             directory_path = init_dir(directory=job_config.directory_cleaning.cache_directory)
             folder_pattern = f"{directory_path}/{job_config.directory_cleaning.subfolder_pattern}"
diff --git a/libs/libcommon/src/libcommon/simple_cache.py b/libs/libcommon/src/libcommon/simple_cache.py
--- a/libs/libcommon/src/libcommon/simple_cache.py
+++ b/libs/libcommon/src/libcommon/simple_cache.py
@@ -29,6 +29,7 @@
     CACHE_COLLECTION_RESPONSES,
     CACHE_METRICS_COLLECTION,
     CACHE_MONGOENGINE_ALIAS,
+    ERROR_CODES_TO_RETRY,
 )
 from libcommon.dtos import JobParams
 from libcommon.utils import get_datetime
@@ -577,6 +578,10 @@ def get_all_datasets() -> set[str]:
     return set(CachedResponseDocument.objects().distinct("dataset"))
 
 
+def get_datasets_with_retryable_errors() -> set[str]:
+    return set(CachedResponseDocument.objects(error_code__in=list(ERROR_CODES_TO_RETRY)).distinct("dataset"))
+
+
 def is_successful_response(kind: str, dataset: str, config: Optional[str] = None, split: Optional[str] = None) -> bool:
     """
     Check if the response is successful.


Test patch: