Repository: MatterMiners/tardis

Issue:
Missing resource_status results is crashing Prometheus plugin
Dear all,

we are currently migrating our C/T to the newest image (the old image is from September 2022 with a custom auditor plugin) and are seeing some crashes related to the Prometheus monitoring plugin. We are running the following image:

```
docker images --digests
REPOSITORY                                      TAG       DIGEST                                                                    IMAGE ID       CREATED        SIZE
matterminers/cobald-tardis                      latest    sha256:7b6fc72444eb7f25d8b17d6e957311fb4d7d5e3abed70aed4875e373aafcbafc   d2ca28594b2b   6 weeks ago    1.03GB
```

The crash happens when a drone changes to RequestState:

```
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]: cobald.runtime.tardis.plugins.prometheusmonitoring: 2023-07-10 08:03:24 Drone: {'site_name': 'NEMO', 'machine_type': 'tardis_c40m100', 'obs_machine_meta_data_translation_mapping': {'Cores': 1, 'Memory': 1000, 'Disk': 1000}, 'remote_resource_uuid':
 None, 'created': datetime.datetime(2023, 7, 10, 8, 3, 23, 922620), 'updated': datetime.datetime(2023, 7, 10, 8, 3, 24, 469225), 'drone_uuid': 'nemo-34fd16c8b9'} has changed state to RequestState
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]: cobald.runtime.runner.asyncio: 2023-07-10 08:03:24 runner aborted: <cobald.daemon.runners.asyncio_runner.AsyncioRunner object at 0x7f51391150d0>
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]: Traceback (most recent call last):
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/cobald/daemon/runners/base_runner.py", line 68, in run
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     await self.manage_payloads()
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/cobald/daemon/runners/asyncio_runner.py", line 54, in manage_payloads
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     await self._payload_failure
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/cobald/daemon/runners/asyncio_runner.py", line 40, in _monitor_payload
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     result = await payload()
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/tardis/resources/drone.py", line 120, in run
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     await self.set_state(RequestState())
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/tardis/resources/drone.py", line 143, in set_state
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     await self.notify_plugins()
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/tardis/resources/drone.py", line 153, in notify_plugins
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     await plugin.notify(self.state, self.resource_attributes)
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/tardis/plugins/prometheusmonitoring.py", line 67, in notify
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     new_status = resource_attributes.resource_status
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:   File "/usr/local/lib/python3.8/site-packages/tardis/utilities/attributedict.py", line 17, in __getattr__
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]:     raise AttributeError(
Jul 10 10:03:24 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2129502]: AttributeError: resource_status is not a valid attribute. Dict contains {'site_name': 'NEMO', 'machine_type': 'tardis_c40m100', 'obs_machine_meta_data_translation_mapping': {'Cores': 1, 'Memory': 1000, 'Disk': 1000}, 'remote_resource_uuid': None, 
'created': datetime.datetime(2023, 7, 10, 8, 3, 23, 922620), 'updated': datetime.datetime(2023, 7, 10, 8, 3, 24, 469225), 'drone_uuid': 'nemo-34fd16c8b9'}.
```

As the error message indicates, the attribute `resource_status` is missing in the `resource_attributes` dict. This dict is accessed  in the `notify` method of the Prometheus plugin (line 67): 

https://github.com/MatterMiners/tardis/blob/0f76db0b3ccb3645fcf971fa82e15c33d4f51258/tardis/plugins/prometheusmonitoring.py#L47-L73

After the service restarts C/T and some time, we hover see that the `recource_status` is now present in the `resource_attributes` dict:

```
Jul 10 10:03:38 monopol.bfg.privat docker-COBalD-Tardis-atlhei[2130347]: cobald.runtime.tardis.plugins.sqliteregistry: 2023-07-10 08:03:38 Drone: {'site_name': 'NEMO', 'machine_type': 'tardis_c40m100', 'obs_machine_meta_data_translation_mapping': {'Cores': 1, 'Memory': 1000, 'Disk': 1000}, 'remote_resource_uuid': 16960754, 'created': datetime.datetime(2023, 7, 10, 8, 3, 38, 592070), 'updated': datetime.datetime(2023, 7, 10, 8, 3, 38, 592190), 'drone_uuid': 'nemo-34fd16c8b9', 'resource_status': <ResourceStatus.Booting: 1>} has changed state to BootingState
```

I can work on a fix, but I would need to know how we want the Prometheus plugin to behave. I currently have two ideas in mind:

- just skip the prometheus update if the `resource_status` attribute is missing
- set the `new_status` variable to a default value (e.g. `BootingState`) when the `resource_status` attribute is missing

But maybe you have also a different fix in mind.


Comments:

Patch:
diff --git a/tardis/resources/drone.py b/tardis/resources/drone.py
index 4d0322b..a5b3c13 100644
--- a/tardis/resources/drone.py
+++ b/tardis/resources/drone.py
@@ -3,9 +3,9 @@ from typing import List, Union, Optional, Type
 from tardis.agents.batchsystemagent import BatchSystemAgent
 from tardis.agents.siteagent import SiteAgent
 from tardis.interfaces.plugin import Plugin
+from tardis.interfaces.siteadapter import ResourceStatus
 from tardis.interfaces.state import State
-from .dronestates import RequestState
-from .dronestates import DownState
+from .dronestates import DownState, RequestState
 from ..plugins.sqliteregistry import SqliteRegistry
 from ..utilities.attributedict import AttributeDict
 from ..utilities.utils import load_states
@@ -32,8 +32,8 @@ class Drone(Pool):
         remote_resource_uuid=None,
         drone_uuid=None,
         state: Optional[State] = None,
-        created: float = None,
-        updated: float = None,
+        created: Optional[float] = None,
+        updated: Optional[float] = None,
     ):
         self._site_agent = site_agent
         self._batch_system_agent = batch_system_agent
@@ -117,6 +117,12 @@ class Drone(Pool):
             # to be notified on the first state change. As calling the
             # ``set_state`` coroutine is not possible in the constructor, we
             # initiate the first state change here
+            #
+            # In addition, all necessary attributes in `resource_attributes`
+            # `AttributeDict` need to be present and have meaningful defaults.
+            # `resource_status` should be set to `ResourceStatus.Booting` on
+            # newly created drones by default.
+            self.resource_attributes.resource_status = ResourceStatus.Booting
             await self.set_state(RequestState())
         while True:
             current_state = self.state


Test patch:
diff --git a/tests/resources_t/test_drone.py b/tests/resources_t/test_drone.py
index f071036..1d38efe 100644
--- a/tests/resources_t/test_drone.py
+++ b/tests/resources_t/test_drone.py
@@ -1,9 +1,10 @@
 from ..utilities.utilities import async_return, run_async, set_awaitable_return_value
 
 from tardis.interfaces.plugin import Plugin
+from tardis.interfaces.siteadapter import ResourceStatus
 from tardis.interfaces.state import State
 from tardis.resources.drone import Drone
-from tardis.resources.dronestates import DrainState, DownState
+from tardis.resources.dronestates import DrainState, DownState, RequestState
 from tardis.plugins.sqliteregistry import SqliteRegistry
 from tardis.utilities.attributedict import AttributeDict
 
@@ -112,6 +113,24 @@ class TestDrone(TestCase):
     def test_site_agent(self):
         self.assertEqual(self.drone.site_agent, self.mock_site_agent)
 
+    @patch("tardis.resources.drone.RequestState", spec=RequestState)
+    def test_first_run_of_initialized_drone(self, mocked_request_state):
+        # self.drone.run() is executed in an endless loop, therefore an
+        # EndOfLoop exception is thrown, which then can be caught. Throwing
+        # StopIteration does not work in a while loop
+        class EndOfLoop(Exception):
+            pass
+
+        mocked_request_state.return_value.run.side_effect = EndOfLoop
+        with self.assertRaises(EndOfLoop):
+            run_async(self.drone.run)
+
+        # Actual test code
+        self.assertIsInstance(self.drone.state, RequestState)
+        self.assertEqual(
+            self.drone.resource_attributes.resource_status, ResourceStatus.Booting
+        )
+
     @patch("tardis.resources.drone.asyncio.sleep")
     def test_run(self, mocked_asyncio_sleep):
         mocked_asyncio_sleep.side_effect = async_return