Repository: DataBiosphere/toil

Issue:
RuntimeError: Cannot store message argument of type <enum 'BatchJobExitReason'>: BatchJobExitReason.MEMLIMIT
Hello,

I'm running Cactus on a LSF farm, and a bunch of jobs consumed more memory than they had requested, and got killed by LSF ("MEMLIMIT")
```
[2023-01-20T12:09:50+0000] [Thread-2  ] [I] [toil.batchSystems.lsf] [job ID 9986865, Command singularity exec /software/treeoflife/shpc/0.1.16/container_base/quay.io/comparative-genomics-toolkit/cactus/v2.4.0/quay.io-comparative-genomics-toolkit-cactus-v2.4.0-sha256:8c677dfccd0dfd4b6e645775a5d64dfffe86345753edaeba74a31821cebca74c.sif _toil_worker run_lastz file:/nfs/users/nfs_m/mm49/nfs/scratch123/cactus/runtime/lsf_js kind-run_lastz/instance-q9shyj5b --context gASVzgAAAAAAAACMIXRvaWwuYmF0Y2hTeXN0ZW1zLmNsZWFudXBfc3VwcG9ydJSMFFdvcmtlckNsZWFudXBDb250ZXh0lJOUKYGUfZSMEXdvcmtlckNsZWFudXBJbmZvlIwldG9pbC5iYXRjaFN5c3RlbXMuYWJzdHJhY3RCYXRjaFN5c3RlbZSMEVdvcmtlckNsZWFudXBJbmZvlJOUKE5OjCRhNmY4ZTU2Yi01NzU5LTQ5ODUtODE4YS1iMDM5MjZlMThjNTSUjAZhbHdheXOUdJSBlHNiLg==] Max Memory Used: 309 Mbytes
[2023-01-20T12:09:50+0000] [Thread-2  ] [E] [toil.batchSystems.lsf] bjobs detected job failed with:
exit code: 130
exit reason: TERM_MEMLIMIT: job killed after reaching LSF memory usage limit
for job: 9986865
```

This resulted in Cactus/toil failing with this error:
```
Traceback (most recent call last):
  File "/home/cactus/cactus_env/bin/cactus", line 8, in <module>   
    sys.exit(main())
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/progressive/cactus_progressive.py", line 400, in main
    hal_id = toil.start(Job.wrapJobFn(progressive_workflow, options, config_node, mc_tree, og_map, input_seq_id_map))
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1017, in start
    return self._runMainLoop(rootJobDescription)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1461, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/leader.py", line 292, in run
    self.innerLoop()
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/leader.py", line 789, in innerLoop
    self._gatherUpdatedJobs(updatedJobTuple)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/leader.py", line 746, in _gatherUpdatedJobs
    self._messages.publish(JobCompletedMessage(updatedJob.get_job_kind(), updatedJob.jobStoreID, exitStatus))
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/bus.py", line 610, in publish
    self._bus.publish(message)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/bus.py", line 296, in publish
    self._deliver(message)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/bus.py", line 329, in _deliver
    self._pubsub.sendMessage(topic, message=message)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/pubsub/core/publisher.py", line 216, in sendMessage
    topicObj.publish(**msgData)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/pubsub/core/topicobj.py", line 452, in publish
    self.__sendMessage(msgData, topicObj, msgDataSubset)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/pubsub/core/topicobj.py", line 482, in __sendMessage
    listener(data, self, allData)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/pubsub/core/listener.py", line 237, in __call__
    cb(**kwargs)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/bus.py", line 404, in handler
    stream.write(message_to_bytes(message))
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/bus.py", line 213, in message_to_bytes
    raise RuntimeError(f"Cannot store message argument of type {item_type}: {item}")
RuntimeError: Cannot store message argument of type <enum 'BatchJobExitReason'>: BatchJobExitReason.MEMLIMIT
Command exited with non-zero status 1
```

Job ID 9986865 was not the only one to fail with error 130 at the same time, but it's the last one in the log before the stack-trace. Here is its report:
```
Job <9986865>, Job Name <toil_job_255>, User <mm49>, Project <default>, Status
                     <EXIT>, Queue <normal>, Command <singularity exec /softwar
                     e/treeoflife/shpc/0.1.16/container_base/quay.io/comparativ
                     e-genomics-toolkit/cactus/v2.4.0/quay.io-comparative-genom
                     ics-toolkit-cactus-v2.4.0-sha256:8c677dfccd0dfd4b6e645775a
                     5d64dfffe86345753edaeba74a31821cebca74c.sif _toil_worker r
                     un_lastz file:/nfs/users/nfs_m/mm49/nfs/scratch123/cactus/
                     runtime/lsf_js kind-run_lastz/instance-q9shyj5b --context
                     gASVzgAAAAAAAACMIXRvaWwuYmF0Y2hTeXN0ZW1zLmNsZWFudXBfc3VwcG
                     9ydJSMFFdvcmtlckNsZWFudXBDb250ZXh0lJOUKYGUfZSMEXdvcmtlckNs
                     ZWFudXBJbmZvlIwldG9pbC5iYXRjaFN5c3RlbXMuYWJzdHJhY3RCYXRjaF
                     N5c3RlbZSMEVdvcmtlckNsZWFudXBJbmZvlJOUKE5OjCRhNmY4ZTU2Yi01
                     NzU5LTQ5ODUtODE4YS1iMDM5MjZlMThjNTSUjAZhbHdheXOUdJSBlHNiLg
                     ==>, Share group charged </WTSI/tol/tol-ops/team328-grp/mm
                     49>
Fri Jan 20 12:08:40: Submitted from host <farm5-os0000001>, CWD </lustre/scratc
                     h123/tol/teams/tolit/users/mm49/cactus/runtime>, Specified
                      CWD </lustre/scratch123/tol/teams/tolit/users/mm49/cactus
                     /runtime/.>, Output File </tmp/toil_a6f8e56b-5759-4985-818
                     a-b03926e18c54.255.%J.out.log>, Error File </tmp/toil_a6f8
                     e56b-5759-4985-818a-b03926e18c54.255.%J.err.log>;
Fri Jan 20 12:08:41: Dispatched 1 Task(s) on Host(s) <node-14-12>, Allocated 1
                     Slot(s) on Host(s) <node-14-12>, Effective RES_REQ <select
                     [(mem>279.00) && (type == any )] order[r15s:pg] rusage[mem
                     =279.00] span[hosts=1] affinity[thread(1)*1] >;
Fri Jan 20 12:09:14: Completed <exit>; TERM_MEMLIMIT: job killed after reaching
                      LSF memory usage limit.

 EXCEPTION STATUS:  underrun

Accounting information about this job:
     Share group charged </WTSI/tol/tol-ops/team328-grp/mm49>
     CPU_T     WAIT     TURNAROUND   STATUS     HOG_FACTOR    MEM    SWAP
     14.46        1             34     exit         0.4253   309M      0M
     CPU_PEAK     CPU_EFFICIENCY      MEM_EFFICIENCY
      0.44                43.75%             110.75%
```

I'm using the official Docker image of Cactus v2.4.0, converted to Singularity. This is toil 5.8.0, according to their changelog.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1264)
┆Issue Number: TOIL-1264



Discussion:
I think the problem is here:

https://github.com/DataBiosphere/toil/blob/332eeed9bdd9e6a017a795d1982eba28bf890f89/src/toil/bus.py#L206-L207

`BatchJobExitReason.MEMLIMIT` *is* an `int`, but `type()` on it is a type that is a subclass of int. So we need to match subclasses and not just on the final type.

Resolving patch:
diff --git a/src/toil/batchSystems/abstractBatchSystem.py b/src/toil/batchSystems/abstractBatchSystem.py
--- a/src/toil/batchSystems/abstractBatchSystem.py
+++ b/src/toil/batchSystems/abstractBatchSystem.py
@@ -43,7 +43,7 @@
 # Value to use as exitStatus in UpdatedBatchJobInfo.exitStatus when status is not available.
 EXIT_STATUS_UNAVAILABLE_VALUE = 255
 
-class BatchJobExitReason(enum.Enum):
+class BatchJobExitReason(enum.IntEnum):
     FINISHED: int = 1  # Successfully finished.
     FAILED: int = 2  # Job finished, but failed.
     LOST: int = 3  # Preemptable failure (job's executing host went away).
diff --git a/src/toil/bus.py b/src/toil/bus.py
--- a/src/toil/bus.py
+++ b/src/toil/bus.py
@@ -203,14 +203,15 @@ def message_to_bytes(message: NamedTuple) -> bytes:
     """
     parts = []
     for item in message:
-        item_type = type(item)
-        if item_type in [int, float, bool] or item is None:
-            parts.append(str(item).encode('utf-8'))
-        elif item_type == str:
+        if isinstance(item, (int, float, bool)) or item is None:
+            # This also handles e.g. values from an IntEnum, where the type extends int.
+            # They might replace __str__() but we hope they use a compatible __format__()
+            parts.append(f"{item}".encode('utf-8'))
+        elif isinstance(item, str):
             parts.append(item.encode('unicode_escape'))
         else:
             # We haven't implemented this type yet.
-            raise RuntimeError(f"Cannot store message argument of type {item_type}: {item}")
+            raise RuntimeError(f"Cannot store message argument of type {type(item)}: {item}")
     return b'\t'.join(parts)
 
 


Test patch:
diff --git a/src/toil/test/src/busTest.py b/src/toil/test/src/busTest.py
--- a/src/toil/test/src/busTest.py
+++ b/src/toil/test/src/busTest.py
@@ -15,13 +15,39 @@
 import logging
 from threading import Thread, current_thread
 
-from toil.bus import JobIssuedMessage, MessageBus
-from toil.test import ToilTest
+from toil.batchSystems.abstractBatchSystem import BatchJobExitReason
+from toil.bus import JobCompletedMessage, JobIssuedMessage, MessageBus, replay_message_bus
+from toil.test import ToilTest, get_temp_file
 
 logger = logging.getLogger(__name__)
 
 class MessageBusTest(ToilTest):
-    def test_cross_thread_messaging(self):
+    
+    def test_enum_ints_in_file(self) -> None:
+        """
+        Make sure writing bus messages to files works with enums.
+        """
+        bus_file = get_temp_file()
+
+        bus = MessageBus()
+        # Connect the handler and hold the result to protect it from GC
+        handler_to_keep_alive = bus.connect_output_file(bus_file)
+        logger.debug("Connected bus to file %s", bus_file)
+        bus.publish(JobCompletedMessage("blah", "blah", BatchJobExitReason.MEMLIMIT))
+        # Make sure stuff goes away in the right order
+        del handler_to_keep_alive
+        del bus
+        
+        for line in open(bus_file):
+            logger.debug("Bus line: %s", line)
+
+        replay = replay_message_bus(bus_file)
+        assert len(replay) > 0
+
+    def test_cross_thread_messaging(self) -> None:
+        """
+        Make sure message bus works across threads.
+        """
 
         main_thread = current_thread()