Repository: DataBiosphere/toil

Issue:
Support the `arv:UsePreemptible` hint in CWL to allow CWL tasks on spot instances
Is it possible to specify in a multi-stage workflow, consisted of multiple tasks (e.g. alignment, sort vcf, index vcf, etc), specific task to be using on-demand (non-preemptable) nodes, while others can use preemptable. This is because for very long running workflow, there are some steps that will for sure take 15+ hours to finish and because it is not using spot instances, no matter how many retries, it'll very likely be killed if using spot. But I don't want the entire workflow to be using on-demand instances, as there are many other smaller tasks fully suitable for using spot.

From the arguments (`defaultPreemptable`, `nodeTypes`, `preemptableCompensation`), they only control at the entire job level, and not to the individual task. It was mentioned in [documentation](https://toil.readthedocs.io/en/releases-3.6.x/running/cloud.html) "While individual jobs can each explicitly specify whether or not they should be run on preemptable nodes via the boolean preemptable resource requirement, ...", was this referring to specify `preemptable` in `ResourceRequirement` in the CWL? This would result in the error `invalid field preemptable, expected one of: 'class', 'coresMin', 'coresMax', 'ramMin', 'ramMax', 'tmpdirMin', 'tmpdirMax', 'outdirMin', 'outdirMax'`, as `preemptable` isn't one of the options.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1235)
┆friendlyId: TOIL-1235



Discussion:
Toil has support for this internally, but it's possible it isn't bound to the CWL front-end.

@mr-c, where in a CWL workflow should something like a hint that a task should run on a spot-market, preemptable node be placed?
> @mr-c, where in a CWL workflow should something like a hint that a task should run on a spot-market, preemptable node be placed?

`hints`/`reqs` can be added in many places: the CWL input object, workflow level, step level, sub-workflow (and its steps, etc.), in a `CommandLineTool` ; or via an external `overrides` file with `--overrides`.

As for the syntax of such a hint, I would suggest reusing `http://arvados.org/cwl#UsePreemptible` (a.k.a. `arv:UsePreemptible`) if you only need a boolean Yes/No.

https://github.com/arvados/arvados/blob/f293f0f41c5de9623239fe2b1eb6c4ef83ae00a0/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.0.yml#L389

If users need more control than that, then we can develop a custom requirement for toil.
So in CWL this would look like:

```
$namespaces:
  arv: "http://arvados.org/cwl#"

...

hints:
  arv:UsePreemptible:
    usePreemptible: true
```

But the `arv` can be anything, so we also need to support:

```
$namespaces:
  foo: "http://arvados.org/cwl#"

...

hints:
  foo:UsePreemptible:
    usePreemptible: true
```

And we need to handle `false` values as well, probably allowing them to override a default of preemptible.

I think cwltool does most of the parsing and namespace stuff for us, and we already have logic to look at extensions like this for GPUs, so this shouldn't be too hard to add.


The extension docs are at https://doc.arvados.org/user/cwl/cwl-extensions.html
> I think cwltool does most of the parsing and namespace stuff for us, and we already have logic to look at extensions like this for GPUs, so this shouldn't be too hard to add.

Correct!

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1866,6 +1866,7 @@ def __init__(
         memory: Union[int, str, None] = "1GiB",
         disk: Union[int, str, None] = "1MiB",
         accelerators: Optional[List[AcceleratorRequirement]] = None,
+        preemptible: Optional[bool] = None,
         tool_id: Optional[str] = None,
         parent_name: Optional[str] = None,
         subjob_name: Optional[str] = None,
@@ -1910,6 +1911,7 @@ def __init__(
             memory=memory,
             disk=disk,
             accelerators=accelerators,
+            preemptible=preemptible,
             unitName=unit_name,
             displayName=display_name,
             local=local,
@@ -2182,7 +2184,7 @@ def __init__(
 
         accelerators: Optional[List[AcceleratorRequirement]] = None
         if req.get("cudaDeviceCount", 0) > 0:
-            # There's a CUDARequirement
+            # There's a CUDARequirement, which cwltool processed for us
             # TODO: How is cwltool deciding what value to use between min and max?
             accelerators = [
                 {
@@ -2192,6 +2194,42 @@ def __init__(
                 }
             ]
 
+        # cwltool doesn't handle http://arvados.org/cwl#UsePreemptible as part
+        # of its resource logic so we have to do it manually.
+        #
+        # Note that according to
+        # https://github.com/arvados/arvados/blob/48a0d575e6de34bcda91c489e4aa98df291a8cca/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.1.yml#L345
+        # this can only be a literal boolean! cwltool doesn't want to evaluate
+        # expressions in the value for us like it does for CUDARequirement
+        # which has a schema which allows for CWL expressions:
+        # https://github.com/common-workflow-language/cwltool/blob/1573509eea2faa3cd1dc959224e52ff1d796d3eb/cwltool/extensions.yml#L221
+        #
+        # By default we have default preemptibility.
+        preemptible: Optional[bool] = None
+        preemptible_req, _ = tool.get_requirement("http://arvados.org/cwl#UsePreemptible")
+        if preemptible_req:
+            if "usePreemptible" not in preemptible_req:
+                # If we have a requirement it has to have the value
+                raise ValidationException(
+                    f"Unacceptable syntax for http://arvados.org/cwl#UsePreemptible: "
+                    f"expected key usePreemptible but got: {preemptible_req}"
+                )
+            parsed_value = preemptible_req["usePreemptible"]
+            if isinstance(parsed_value, str) and ("$(" in parsed_value or "${" in parsed_value):
+                # Looks like they tried to use an expression
+                 raise ValidationException(
+                    f"Unacceptable value for usePreemptible in http://arvados.org/cwl#UsePreemptible: "
+                    f"expected true or false but got what appears to be an expression: {repr(parsed_value)}. "
+                    f"Note that expressions are not allowed here by Arvados's schema."
+                )
+            if not isinstance(parsed_value, bool):
+                # If we have a value it has to be a bool flag
+                raise ValidationException(
+                    f"Unacceptable value for usePreemptible in http://arvados.org/cwl#UsePreemptible: "
+                    f"expected true or false but got: {repr(parsed_value)}"
+                )
+            preemptible = parsed_value
+
         super().__init__(
             cores=req["cores"],
             memory=int(req["ram"] * (2**20)),
@@ -2200,6 +2238,7 @@ def __init__(
                 + (cast(int, req["outdirSize"]) * (2**20))
             ),
             accelerators=accelerators,
+            preemptible=preemptible,
             tool_id=self.cwltool.tool["id"],
             parent_name=parent_name,
             local=isinstance(tool, cwltool.command_line_tool.ExpressionTool),


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -66,6 +66,8 @@
     AbstractAWSAutoscaleTest
 from toil.test.provisioners.clusterTest import AbstractClusterTest
 
+from schema_salad.exceptions import ValidationException
+
 log = logging.getLogger(__name__)
 CONFORMANCE_TEST_TIMEOUT = 3600
 
@@ -519,6 +521,53 @@ def test_streamable(self):
         with open(out[out_name]["location"][len("file://") :]) as f:
             self.assertEqual(f.read().strip(), "When is s4 coming out?")
 
+    def test_preemptible(self):
+        """
+        Tests that the http://arvados.org/cwl#UsePreemptible extension is supported.
+        """
+        cwlfile = "src/toil/test/cwl/preemptible.cwl"
+        jobfile = "src/toil/test/cwl/empty.json"
+        out_name = "output"
+        from toil.cwl import cwltoil
+
+        st = StringIO()
+        args = [
+            "--outdir",
+            self.outDir,
+            os.path.join(self.rootDir, cwlfile),
+            os.path.join(self.rootDir, jobfile),
+        ]
+        cwltoil.main(args, stdout=st)
+        out = json.loads(st.getvalue())
+        out[out_name].pop("http://commonwl.org/cwltool#generation", None)
+        out[out_name].pop("nameext", None)
+        out[out_name].pop("nameroot", None)
+        with open(out[out_name]["location"][len("file://") :]) as f:
+            self.assertEqual(f.read().strip(), "hello")
+
+    def test_preemptible_expression(self):
+        """
+        Tests that the http://arvados.org/cwl#UsePreemptible extension is validated.
+        """
+        cwlfile = "src/toil/test/cwl/preemptible_expression.cwl"
+        jobfile = "src/toil/test/cwl/preemptible_expression.json"
+        from toil.cwl import cwltoil
+
+        st = StringIO()
+        args = [
+            "--outdir",
+            self.outDir,
+            os.path.join(self.rootDir, cwlfile),
+            os.path.join(self.rootDir, jobfile),
+        ]
+        try:
+            cwltoil.main(args, stdout=st)
+            raise RuntimeError("Did not raise correct exception")
+        except ValidationException as e:
+            # Make sure we chastise the user appropriately.
+            assert "expressions are not allowed" in str(e)
+        
+
     @staticmethod
     def _expected_seqtk_output(outDir):
         loc = "file://" + os.path.join(outDir, "out")
diff --git a/src/toil/test/cwl/preemptible.cwl b/src/toil/test/cwl/preemptible.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/preemptible.cwl
@@ -0,0 +1,21 @@
+$namespaces:
+  arv: "http://arvados.org/cwl#"
+
+class: CommandLineTool
+doc: "Be preemptible"
+cwlVersion: v1.0
+
+inputs: []
+
+outputs:
+  output:
+    type: File
+    outputBinding:
+      glob: output.txt
+
+hints:
+  arv:UsePreemptible:
+    usePreemptible: true
+
+baseCommand: ["echo", "hello"]
+stdout: output.txt
diff --git a/src/toil/test/cwl/preemptible_expression.cwl b/src/toil/test/cwl/preemptible_expression.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/preemptible_expression.cwl
@@ -0,0 +1,28 @@
+$namespaces:
+  arv: "http://arvados.org/cwl#"
+
+class: CommandLineTool
+doc: "Be preemptible"
+cwlVersion: v1.0
+
+inputs:
+  - id: preemptible
+    type: boolean
+
+outputs:
+  output:
+    type: File
+    outputBinding:
+      glob: output.txt
+
+requirements:
+  InlineJavascriptRequirement: {}
+
+hints:
+  arv:UsePreemptible:
+    # This isn't allowed; this always has to be a real boolean and can't be a cwl:Expression. See
+    # https://github.com/arvados/arvados/blob/48a0d575e6de34bcda91c489e4aa98df291a8cca/sdk/cwl/arvados_cwl/arv-cwl-schema-v1.1.yml#L345
+    usePreemptible: $(inputs.preemptible)
+
+baseCommand: ["echo", "hello"]
+stdout: output.txt
diff --git a/src/toil/test/cwl/preemptible_expression.json b/src/toil/test/cwl/preemptible_expression.json
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/preemptible_expression.json
@@ -0,0 +1 @@
+{"preemptible": true}