Repository: AgnostiqHQ/covalent (python)

Issue:
Sublattices without explicitly specified executors don't inherit the parent lattice's executor.
### Description

When users do not specify an executor in a lattice decorator, the lattice's executor is intended to be automatically set as follows:
- If the lattice is top-level, use the default executor from the Covalent config.
- If the lattice is a sublattice, use its parent lattice's executor.

Consider the following simple lattice consisting of a single sublattice. The top-level lattice's executor is set to `"dask"`, whereas the sublattice's executor is unspecified.
```python
@ct.electron
def task(x):
    return x*x

@ct.electron
@ct.lattice
def subworkflow(x):
    return task(x)

@ct.lattice(executor="dask")
def workflow(x):
    return subworkflow(x)

dispatch_id = ct.dispatch(workflow)(5)
result_object = ct.get_result(dispatch_id, wait=True)
sub_result_object = result_object.lattice.transport_graph.get_node_value(0, "sublattice_result")
```

Since the sublattice does not specify an executor, it should inherit its parent lattice's executor. However, this is not what happens; the sublattice's executor is set to `"local"`, the default executor:

```python
print(result_object.lattice.metadata["executor"])
print(sub_result_object.lattice.metadata["executor"])
```

```
dask
local
```


### Acceptance criteria

- [ ] If a sublattice does not specify an executor, it should inherit its parent lattice's executor.
- [ ] Explicit executor settings for a sublattice shall be honored.
- [ ] If a top-level lattice (not a sublattice) does not specify an executor, it should use the Covalent default executor -- `"local"` if Covalent is started with the `--no-cluster` option and `"dask"` otherwise.
- [ ] Add tests to verify executor settings for lattices and sublattices in all three of the above scenarios.


Some suggestions to start debugging:
- The [lattice decorator](https://github.com/AgnostiqHQ/covalent/blob/develop/covalent/_workflow/lattice.py#L307)
- The server's [internal logic for running sublattices](https://github.com/AgnostiqHQ/covalent/blob/develop/covalent_dispatcher/_core/execution.py#L191); see also `_build_sublattice_graph`.
- The related issue #1284 for automatically setting electron executors.



Pull request:
Fix for 1291
- [x] I have added the tests to cover my changes.
- [x] I have updated the documentation and CHANGELOG accordingly.
- [x] I have read the CONTRIBUTING document.
Closes #1291.


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 220762a6..389ebe8d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [UNRELEASED]
 
+### Added 
+
+- Sublattices inherit the parent lattice executor.
+
 ### Docs
 
 - Added DNN tutorial
diff --git a/covalent/_workflow/electron.py b/covalent/_workflow/electron.py
index 0e075c69..dcb06309 100644
--- a/covalent/_workflow/electron.py
+++ b/covalent/_workflow/electron.py
@@ -320,14 +320,19 @@ class Electron:
             return output.get_deserialized()
 
         # Setting metadata for default values according to lattice's metadata
-        # If metadata is default, then set it to lattice's default
         for k in self.metadata:
-            if (
-                k not in consumable_constraints
-                and k in DEFAULT_METADATA_VALUES
-                and self.get_metadata(k) is DEFAULT_METADATA_VALUES[k]
-            ):
-                self.set_metadata(k, active_lattice.get_metadata(k))
+            if k != "executor":
+                if (
+                    k not in consumable_constraints
+                    and k in DEFAULT_METADATA_VALUES
+                    and self.get_metadata(k) == DEFAULT_METADATA_VALUES[k]
+                ):
+                    self.set_metadata(k, active_lattice.get_metadata(k))
+            else:
+                if self.get_metadata(k) != "":
+                    self.set_metadata(k, self.get_metadata(k))
+                else:
+                    self.set_metadata(k, active_lattice.get_metadata(k))
 
         # Add a node to the transport graph of the active lattice
         self.node_id = active_lattice.transport_graph.add_node(
@@ -520,9 +525,7 @@ def electron(
     _func: Optional[Callable] = None,
     *,
     backend: Optional[str] = None,
-    executor: Optional[
-        Union[List[Union[str, "BaseExecutor"]], Union[str, "BaseExecutor"]]
-    ] = DEFAULT_METADATA_VALUES["executor"],
+    executor: Optional[Union[List[Union[str, "BaseExecutor"]], Union[str, "BaseExecutor"]]] = "",
     # Add custom metadata fields here
     files: List[FileTransfer] = [],
     deps_bash: Union[DepsBash, List, str] = DEFAULT_METADATA_VALUES["deps"].get("bash", []),


Test patch:
diff --git a/tests/covalent_tests/executor/sublattice_to_lattice.py b/tests/covalent_tests/executor/sublattice_to_lattice.py
new file mode 100644
index 00000000..c87e1892
--- /dev/null
+++ b/tests/covalent_tests/executor/sublattice_to_lattice.py
@@ -0,0 +1,157 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+"""Integration tests to check that sublattices inherit the parent lattice executor."""
+
+import json
+
+import covalent as ct
+from covalent._workflow.lattice import Lattice as LatticeClass
+
+
+def test_sublattices_have_default_executor():
+    """start the covalent server with dask, and don't specify executor to main lattice. check if everything has dask."""
+    import covalent as ct
+
+    @ct.electron
+    def add(a, b):
+        return a + b
+
+    @ct.electron
+    def sub(a, b):
+        return a - b
+
+    @ct.electron
+    @ct.lattice
+    def mul(a, b):
+        res1 = add(a, b)
+        res2 = sub(a, b)
+        return res1, res2
+
+    @ct.electron
+    @ct.lattice
+    def div(a, b):
+        a = add(a, b)
+        b = sub(a, b)
+        return a / b
+
+    @ct.lattice
+    def solution(a, b):
+        res1, res2 = mul(a, b)
+        sol = div(res1, res2)
+        return sol
+
+    # Dispatch the workflow
+    solution.build_graph(3, 2)
+    data = solution.transport_graph.serialize_to_json()
+    data = json.loads(data)
+    solution = LatticeClass.deserialize_from_json(solution.serialize_to_json())
+
+    for i in data["nodes"]:
+        if "sublattice" in i["name"]:
+            assert i["metadata"]["executor"] == "dask"
+
+
+def test_sublattices_have_lattice_executor():
+    """start the covalent server with dask, and change the executor to local. Now, even the sublattices executor should be local."""
+    import covalent as ct
+
+    @ct.electron
+    def add(a, b):
+        return a + b
+
+    @ct.electron
+    def sub(a, b):
+        return a - b
+
+    @ct.electron
+    @ct.lattice
+    def mul(a, b):
+        res1 = add(a, b)
+        res2 = sub(a, b)
+        return res1, res2
+
+    @ct.electron
+    @ct.lattice
+    def div(a, b):
+        a = add(a, b)
+        b = sub(a, b)
+        return a / b
+
+    @ct.lattice(executor="local")
+    def solution(a, b):
+        res1, res2 = mul(a, b)
+        sol = div(res1, res2)
+        return sol
+
+    # Dispatch the workflow
+    solution.build_graph(3, 2)
+    data = solution.transport_graph.serialize_to_json()
+    data = json.loads(data)
+    solution = LatticeClass.deserialize_from_json(solution.serialize_to_json())
+
+    for i in data["nodes"]:
+        if "sublattice" in i["name"]:
+            assert i["metadata"]["executor"] == solution.metadata["executor"]
+
+
+def test_sublattices_precede_lattice_executor():
+    """Start covalent server with dask, Change a sublattice executor and now it shouldn't take the lattice executor."""
+    import covalent as ct
+
+    @ct.electron
+    def add(a, b):
+        return a + b
+
+    @ct.electron
+    def sub(a, b):
+        return a - b
+
+    @ct.electron
+    @ct.lattice
+    def mul(a, b):
+        res1 = add(a, b)
+        res2 = sub(a, b)
+        return res1, res2
+
+    @ct.electron(executor="dask")
+    @ct.lattice
+    def div(a, b):
+        a = add(a, b)
+        b = sub(a, b)
+        return a / b
+
+    @ct.lattice(executor="local")
+    def solution(a, b):
+        res1, res2 = mul(a, b)
+        sol = div(res1, res2)
+        return sol
+
+    # Dispatch the workflow
+    solution.build_graph(3, 2)
+    data = solution.transport_graph.serialize_to_json()
+    data = json.loads(data)
+    solution = LatticeClass.deserialize_from_json(solution.serialize_to_json())
+
+    for i in data["nodes"]:
+        if ("div" in i["name"]) and ("sublattice" in i["name"]):
+            assert i["metadata"]["executor"] == "dask"
+        elif "sublattice" in i["name"]:
+            assert i["metadata"]["executor"] == solution.metadata["executor"]
diff --git a/tests/covalent_tests/workflow/sublattice_executor_test.py b/tests/covalent_tests/workflow/sublattice_executor_test.py
new file mode 100644
index 00000000..a8a3f2ce
--- /dev/null
+++ b/tests/covalent_tests/workflow/sublattice_executor_test.py
@@ -0,0 +1,156 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+"""Unit tests to check that sublattices inherit the parent lattice executor."""
+
+import json
+
+import covalent as ct
+
+
+def test_sublattices_have_default_executor():
+    """start the covalent server with dask, and don't specify executor to main lattice.
+    This is to check if everything has dask, including sublattices."""
+    import covalent as ct
+
+    @ct.electron
+    def add(a, b):
+        return a + b
+
+    @ct.electron
+    def sub(a, b):
+        return a - b
+
+    @ct.electron
+    @ct.lattice
+    def mul(a, b):
+        res1 = add(a, b)
+        res2 = sub(a, b)
+        return res1, res2
+
+    @ct.electron
+    @ct.lattice
+    def div(a, b):
+        a = add(a, b)
+        b = sub(a, b)
+        return a / b
+
+    @ct.lattice
+    def solution(a, b):
+        res1, res2 = mul(a, b)
+        sol = div(res1, res2)
+        return sol
+
+    # Dispatch the workflow
+    solution.build_graph(3, 2)
+    data = solution.transport_graph.serialize_to_json()
+    data = json.loads(data)
+
+    for i in data["nodes"]:
+        if "sublattice" in i["name"]:
+            assert i["metadata"]["executor"] == "dask"
+
+
+def test_sublattices_have_lattice_executor():
+    """start the covalent server with dask, and change the executor to local.
+    This checks that sublattices executor should be local."""
+    import covalent as ct
+
+    @ct.electron
+    def add(a, b):
+        return a + b
+
+    @ct.electron
+    def sub(a, b):
+        return a - b
+
+    @ct.electron
+    @ct.lattice
+    def mul(a, b):
+        res1 = add(a, b)
+        res2 = sub(a, b)
+        return res1, res2
+
+    @ct.electron
+    @ct.lattice
+    def div(a, b):
+        a = add(a, b)
+        b = sub(a, b)
+        return a / b
+
+    @ct.lattice(executor="local")
+    def solution(a, b):
+        res1, res2 = mul(a, b)
+        sol = div(res1, res2)
+        return sol
+
+    # Dispatch the workflow
+    solution.build_graph(3, 2)
+    data = solution.transport_graph.serialize_to_json()
+    data = json.loads(data)
+
+    for i in data["nodes"]:
+        if "sublattice" in i["name"]:
+            assert i["metadata"]["executor"] == "local"
+
+
+def test_sublattices_precede_lattice_executor():
+    """Start covalent server with dask, Change a sublattice executor.
+    Now it shouldn't take the lattice executor."""
+    import covalent as ct
+
+    @ct.electron
+    def add(a, b):
+        return a + b
+
+    @ct.electron
+    def sub(a, b):
+        return a - b
+
+    @ct.electron
+    @ct.lattice
+    def mul(a, b):
+        res1 = add(a, b)
+        res2 = sub(a, b)
+        return res1, res2
+
+    @ct.electron(executor="dask")
+    @ct.lattice
+    def div(a, b):
+        a = add(a, b)
+        b = sub(a, b)
+        return a / b
+
+    @ct.lattice(executor="local")
+    def solution(a, b):
+        res1, res2 = mul(a, b)
+        sol = div(res1, res2)
+        return sol
+
+    # Dispatch the workflow
+    solution.build_graph(3, 2)
+    data = solution.transport_graph.serialize_to_json()
+    data = json.loads(data)
+
+    for i in data["nodes"]:
+        if ("div" in i["name"]) and ("sublattice" in i["name"]):
+            assert i["metadata"]["executor"] == "dask"
+        elif "sublattice" in i["name"]:
+            assert i["metadata"]["executor"] == "local"