Repository: huggingface/dataset-viewer

Issue:
The workers fail with `mongoengine.errors.FieldDoesNotExist: The fields "{'force'}" do not exist on the document "Job"`
```
INFO: 2023-05-13 10:50:18,007 - root - Worker loop started
INFO: 2023-05-13 10:50:18,023 - root - Starting heartbeat.
ERROR: 2023-05-13 10:50:18,115 - asyncio - Task exception was never retrieved
future: <Task finished name='Task-2' coro=<every() done, defined at /src/services/worker/src/worker/executor.py:26> exception=FieldDoesNotExist('The fields "{\'force\'}" do not exist on the document "Job"')>
Traceback (most recent call last):
  File "/src/services/worker/src/worker/executor.py", line 30, in every
    out = func(*args, **kwargs)
  File "/src/services/worker/src/worker/executor.py", line 117, in kill_zombies
    zombies = queue.get_zombies(max_seconds_without_heartbeat=self.max_seconds_without_heartbeat_for_zombies)
  File "/src/libs/libcommon/src/libcommon/queue.py", line 622, in get_zombies
    zombies = [
  File "/src/libs/libcommon/src/libcommon/queue.py", line 622, in <listcomp>
    zombies = [
  File "/src/services/worker/.venv/lib/python3.9/site-packages/mongoengine/queryset/queryset.py", line 110, in _iter_results
    self._populate_cache()
  File "/src/services/worker/.venv/lib/python3.9/site-packages/mongoengine/queryset/queryset.py", line 129, in _populate_cache
    self._result_cache.append(next(self))
  File "/src/services/worker/.venv/lib/python3.9/site-packages/mongoengine/queryset/base.py", line 1599, in __next__
    doc = self._document._from_son(
  File "/src/services/worker/.venv/lib/python3.9/site-packages/mongoengine/base/document.py", line 836, in _from_son
    obj = cls(__auto_convert=False, _created=created, **data)
  File "/src/services/worker/.venv/lib/python3.9/site-packages/mongoengine/base/document.py", line 99, in __init__
    raise FieldDoesNotExist(msg)
mongoengine.errors.FieldDoesNotExist: The fields "{'force'}" do not exist on the document "Job"
Traceback (most recent call last):
  File "/src/services/worker/src/worker/main.py", line 61, in <module>
    worker_executor.start()
  File "/src/services/worker/src/worker/executor.py", line 89, in start
    loop.run_until_complete(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 645, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.
```

<img width="622" alt="Capture d’écran 2023-05-13 à 12 52 06" src="https://github.com/huggingface/datasets-server/assets/1676121/3e4d7e1a-dd02-476d-b671-57ce03d28d7b">



Discussion:
Maybe due to 298 jobs that still contain the "force" field in the database? They were created yesterday between 2023-05-12T16:45:25.465+00:00 and 2023-05-12T16:46:21.671+00:00, ie. during the deployment of the last version. Possibly we have to resync to get rid of them. Trying that.

Edit: the migration has already been done, it will do nothing more if run again.

I manually deleted the index: `type_1_dataset_1_config_1_split_1_status_1_force_1_priority_1`
Note also that the jobs seem to be increasing only:

<img width="363" alt="Capture d’écran 2023-05-13 à 12 56 34" src="https://github.com/huggingface/datasets-server/assets/1676121/2ab2baed-d59f-4c1b-b349-a9f7678166aa">

I manually deleted the three last entries in the maintenance collection so that the migrations will be run again on the next sync


<img width="404" alt="Capture d’écran 2023-05-13 à 13 01 50" src="https://github.com/huggingface/datasets-server/assets/1676121/a5091d29-d4c9-4ef2-9116-e0ba3053b4dd">

Sync'ed again, the migrations have been applied:

```
INFO: 2023-05-13 11:03:52,593 - root - Start migrations
INFO: 2023-05-13 11:03:52,657 - root - 25 migrations have already been applied. They will be skipped.
INFO: 2023-05-13 11:03:52,657 - root - Migrate 20230511100600: add to the migrations collection
INFO: 2023-05-13 11:03:52,663 - root - Migrate 20230511100600: apply
INFO: 2023-05-13 11:03:52,663 - root - Removing 'force' field.
INFO: 2023-05-13 11:03:53,753 - root - Migrate 20230511100600: validate
INFO: 2023-05-13 11:03:53,753 - root - Ensure that a random selection of cached results don't have 'force' field
INFO: 2023-05-13 11:03:53,814 - root - Migrate 20230511100600: done
INFO: 2023-05-13 11:03:53,814 - root - Migrate 20230511100700: add to the migrations collection
INFO: 2023-05-13 11:03:53,819 - root - Migrate 20230511100700: apply
INFO: 2023-05-13 11:03:53,819 - root - Delete indexes that contain the force field.
INFO: 2023-05-13 11:03:53,821 - root - Migrate 20230511100700: validate
INFO: 2023-05-13 11:03:53,821 - root - Check that the indexes do not exist anymore
INFO: 2023-05-13 11:03:53,823 - root - Migrate 20230511100700: done
INFO: 2023-05-13 11:03:53,823 - root - Migrate 20230511110700: add to the migrations collection
INFO: 2023-05-13 11:03:53,828 - root - Migrate 20230511110700: apply
INFO: 2023-05-13 11:03:53,828 - root - Delete jobs with status skipped.
INFO: 2023-05-13 11:03:53,951 - root - Migrate 20230511110700: validate
INFO: 2023-05-13 11:03:53,951 - root - Ensure that a random selection of cached results don't have the status {status}
INFO: 2023-05-13 11:03:53,994 - root - Migrate 20230511110700: done
INFO: 2023-05-13 11:03:53,994 - root - All migrations have been applied
```

The jobs collection does not contain jobs with the 'force' field anymore.
OK, it was only part of the problem. The main issue seems to be that the loop breaks, and thus the pod crashes, on every exception (even those that we expect)! It seems like an `except` clause has been changed

Resolving patch:
diff --git a/services/worker/src/worker/job_manager.py b/services/worker/src/worker/job_manager.py
--- a/services/worker/src/worker/job_manager.py
+++ b/services/worker/src/worker/job_manager.py
@@ -115,17 +115,25 @@ def run(self) -> Literal[Status.SUCCESS, Status.ERROR]:
         except Exception:
             self.exception(f"error while computing {self}")
             result = Status.ERROR
-        self.backfill()
+        revision = self.get_dataset_git_revision(allow_raise=False)
+        if revision is not None:
+            self.backfill(revision=revision)
         return result
 
-    def get_dataset_git_revision(self) -> Optional[str]:
+    def get_dataset_git_revision(self, allow_raise: bool = True) -> Optional[str]:
         """Get the git revision of the dataset repository."""
         if self._dataset_git_revision is None:
-            self._dataset_git_revision = get_dataset_git_revision(
-                dataset=self.job_params["dataset"],
-                hf_endpoint=self.common_config.hf_endpoint,
-                hf_token=self.common_config.hf_token,
-            )
+            try:
+                self._dataset_git_revision = get_dataset_git_revision(
+                    dataset=self.job_params["dataset"],
+                    hf_endpoint=self.common_config.hf_endpoint,
+                    hf_token=self.common_config.hf_token,
+                )
+            except Exception as e:
+                if allow_raise:
+                    raise e
+                else:
+                    return None
         return self._dataset_git_revision
 
     def raise_if_parallel_response_exists(self, parallel_cache_kind: str, parallel_job_version: int) -> None:
@@ -210,12 +218,12 @@ def process(
             self.debug(f"response for job_info={self.job_info} had an error, cache updated")
             return False
 
-    def backfill(self) -> None:
+    def backfill(self, revision: str) -> None:
         """Evaluate the state of the dataset and backfill the cache if necessary."""
         DatasetState(
             dataset=self.job_params["dataset"],
             processing_graph=self.processing_graph,
-            revision=self.get_dataset_git_revision(),
+            revision=revision,
             error_codes_to_retry=ERROR_CODES_TO_RETRY,
             priority=self.priority,
         ).backfill()
@@ -230,7 +238,7 @@ def set_crashed(self, message: str, cause: Optional[BaseException] = None) -> No
             error_code=error.code,
             details=dict(error.as_response_with_cause()),
             job_runner_version=self.job_runner.get_job_runner_version(),
-            dataset_git_revision=self.get_dataset_git_revision(),
+            dataset_git_revision=self.get_dataset_git_revision(allow_raise=False),
         )
         logging.debug(
             f"response for dataset={self.job_params['dataset']} job_info={self.job_info} had an error (crashed), cache"
@@ -247,7 +255,7 @@ def set_exceeded_maximum_duration(self, message: str, cause: Optional[BaseExcept
             error_code=error.code,
             details=dict(error.as_response_with_cause()),
             job_runner_version=self.job_runner.get_job_runner_version(),
-            dataset_git_revision=self.get_dataset_git_revision(),
+            dataset_git_revision=self.get_dataset_git_revision(allow_raise=False),
         )
         logging.debug(
             f"response for dataset={self.job_params['dataset']} job_info={self.job_info} had an error (exceeded"


Test patch: