Repository: DataBiosphere/toil

Issue:
ToilMetrics object has no attirbute 'nodeExporterProc'
I ran cactus on a toil leader node with the following command:
```
cactus --nodeTypes m6a.4xlarge,m6a.2xlarge,m5.4xlarge --minNodes 0,0,0 --maxNodes 4,8,4 --consCores 4 --provisioner aws --batchSystem mesos --metrics aws:us-east-1:cactus-examples-1  evolverMammals.txt output.hal
```

At the end (I think it was the end), I got this error:

```
2023-01-24T02:17:39+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2023-01-24T02:17:39+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
Traceback (most recent call last):
  File "/venv/bin/cactus", line 8, in <module>
    sys.exit(main())
  File "/venv/lib/python3.9/site-packages/cactus/progressive/cactus_progressive.py", line 400, in main
    hal_id = toil.start(Job.wrapJobFn(progressive_workflow, options, config_node, mc_tree, og_map, input_seq_id_map))
  File "/usr/local/lib/python3.9/dist-packages/toil/common.py", line 1017, in start
    return self._runMainLoop(rootJobDescription)
  File "/usr/local/lib/python3.9/dist-packages/toil/common.py", line 1456, in _runMainLoop
    return Leader(config=self.config,
  File "/usr/local/lib/python3.9/dist-packages/toil/leader.py", line 308, in run
    self.toilMetrics.shutdown()
  File "/usr/local/lib/python3.9/dist-packages/toil/common.py", line 1666, in shutdown
    if self.nodeExporterProc:
AttributeError: 'ToilMetrics' object has no attribute 'nodeExporterProc'
```

Any idea how I can fix this?  Btw, if I remove the --metrics option I don't get the error.

Thanks very much.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1267)
┆Issue Number: TOIL-1267



Discussion:
It looks like at metrics startup we only set `nodeExporterProc` to anything at all, even None, if `--provisioner` isn't passed. Then at shutdown we check if it is None or not to tear it down, but if it is completely unset we will crash like this.

So we need to set it to None if the provisioner is in use (and the provisioning logic is responsible for managing that container). Otherwise we will always crash when you use `--metrics` and `--provisioner` together.

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -1544,9 +1544,10 @@ def __init__(self, bus: MessageBus, provisioner: Optional["AbstractProvisioner"]
 
         # On single machine, launch a node exporter instance to monitor CPU/RAM usage.
         # On AWS this is handled by the EC2 init script
+        self.nodeExporterProc: Optional[subprocess.Popen[bytes]] = None
         if not provisioner:
             try:
-                self.nodeExporterProc: Optional[subprocess.Popen[bytes]] = subprocess.Popen(
+                self.nodeExporterProc = subprocess.Popen(
                     ["docker", "run",
                      "--rm",
                      "--net=host",
@@ -1561,9 +1562,9 @@ def __init__(self, bus: MessageBus, provisioner: Optional["AbstractProvisioner"]
                      "^/(sys|proc|dev|host|etc)($|/)"])
             except subprocess.CalledProcessError:
                 logger.warning("Couldn't start node exporter, won't get RAM and CPU usage for dashboard.")
-                self.nodeExporterProc = None
             except KeyboardInterrupt:
-                self.nodeExporterProc.terminate()  # type: ignore[union-attr]
+                if self.nodeExporterProc is not None:
+                    self.nodeExporterProc.terminate()
 
         # When messages come in on the message bus, call our methods.
         # TODO: Just annotate the methods with some kind of @listener and get
@@ -1673,10 +1674,14 @@ def logCompletedJob(self, m: JobCompletedMessage) -> None:
         self.log("completed_job %s" % m.job_type)
 
     def shutdown(self) -> None:
-        if self.mtailProc:
+        if self.mtailProc is not None:
+            logger.debug('Stopping mtail')
             self.mtailProc.kill()
-        if self.nodeExporterProc:
+            logger.debug('Stopped mtail')
+        if self.nodeExporterProc is not None:
+            logger.debug('Stopping node exporter')
             self.nodeExporterProc.kill()
+            logger.debug('Stopped node exporter')
         self._listeners = []
 
 


Test patch: