Repository: DataBiosphere/toil

Issue:
Add back `addBatchSystemFactory` feature for workflow-supplied batch systems
I'm trying to update `toil-container` for recent versions of Toil. It seems that in maybe version `5.0.0` the `addBatchSystemFactory` function was removed. It's easy enough to recreate, but does this suggest there is a better way to add to the batch systems registry?

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-902)
┆Issue Number: TOIL-902



Discussion:
Specifically it was this commit: https://github.com/DataBiosphere/toil/commit/e8dac0218829cefd3a95ba4e83110446b3eb7798#diff-9bb3555241fd8ef74577b7acfc10c32c02f124c5dc6858ed93217b7fec37a1cb
Did you see https://github.com/DataBiosphere/toil/pull/3314/files#diff-9bb3555241fd8ef74577b7acfc10c32c02f124c5dc6858ed93217b7fec37a1cbR61 ?
Yes so the commit above is from that PR, but I can't find any explanation behind why this API was deleted or if there is a new alternative.
It looks like we never really documented `addBatchSystemFactory` or the batch system plugin system; there was no docstring on the function explaining what it was supposed to do, let alone that user code was supposed to use it. Nobody who was active on the project in 2016 when #1392 was added was still around in 2020 when @DailyDreaming simplified that function away.

If you're actually using it @multimeric we can add it back, with a docstring and a test to make it clear that this is actually a public-facing feature and not a stub bit of code that we forgot to call from anywhere.

I think the workaround would be to go get `toil.batchSystems.registry.BATCH_SYSTEM_FACTORY_REGISTRY` and `toil.batchSystems.registry.BATCH_SYSTEMS` and add your batch system into those by mutating them yourself. But those are even less of a public API than the old way was.
It's not me that's using it, but rather https://github.com/papaemmelab/toil_container. But yes I think it might be helpful to have a public API function for adding batch systems rather than just mutating the data structures, just so that we can target it with unit tests to ensure that it's stable.

Resolving patch:
diff --git a/src/toil/batchSystems/registry.py b/src/toil/batchSystems/registry.py
--- a/src/toil/batchSystems/registry.py
+++ b/src/toil/batchSystems/registry.py
@@ -13,6 +13,12 @@
 # limitations under the License.
 
 
+from typing import Callable, Type, TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from toil.batchSystems.abstractBatchSystem import AbstractBatchSystem
+
+
 def gridengine_batch_system_factory():
     from toil.batchSystems.gridengine import GridEngineBatchSystem
     return GridEngineBatchSystem
@@ -71,3 +77,10 @@ def kubernetes_batch_system_factory():
 }
 BATCH_SYSTEMS = list(BATCH_SYSTEM_FACTORY_REGISTRY.keys())
 DEFAULT_BATCH_SYSTEM = 'single_machine'
+
+def addBatchSystemFactory(key: str, batchSystemFactory: Callable[[], Type['AbstractBatchSystem']]):
+    """
+    Adds a batch system to the registry for workflow-supplied batch systems.
+    """
+    BATCH_SYSTEMS.append(key)
+    BATCH_SYSTEM_FACTORY_REGISTRY[key] = batchSystemFactory
\ No newline at end of file


Test patch:
diff --git a/src/toil/test/batchSystems/batchSystemTest.py b/src/toil/test/batchSystems/batchSystemTest.py
--- a/src/toil/test/batchSystems/batchSystemTest.py
+++ b/src/toil/test/batchSystems/batchSystemTest.py
@@ -34,6 +34,10 @@
 # protected by annotations.
 from toil.batchSystems.mesos.test import MesosTestSupport
 from toil.batchSystems.parasol import ParasolBatchSystem
+from toil.batchSystems.registry import (BATCH_SYSTEM_FACTORY_REGISTRY,
+                                        BATCH_SYSTEMS,
+                                        single_machine_batch_system_factory,
+                                        addBatchSystemFactory)
 from toil.test.batchSystems.parasolTestSupport import ParasolTestSupport
 from toil.batchSystems.singleMachine import SingleMachineBatchSystem
 from toil.common import Config, Toil
@@ -307,6 +311,14 @@ def _waitForJobsToStart(self, numJobs, tries=20):
                 time.sleep(1)
             return runningIDs
 
+        def testAddBatchSystemFactory(self):
+            def test_batch_system_factory():
+                return SingleMachineBatchSystem
+
+            addBatchSystemFactory('testBatchSystem', test_batch_system_factory)
+            assert ('testBatchSystem', test_batch_system_factory) in BATCH_SYSTEM_FACTORY_REGISTRY.items()
+            assert 'testBatchSystem' in BATCH_SYSTEMS
+
     class AbstractBatchSystemJobTest(ToilTest, metaclass=ABCMeta):
         """
         An abstract base class for batch system tests that use a full Toil workflow rather