Repository: DataBiosphere/toil

Issue:
flatten() implemented as WDL.
Description of WDL function to be added:

```
## Array[X] flatten(Array[Array[X]])

Given an array of arrays, the `flatten` function concatenates all the
member arrays in the order to appearance to give the result. It does not
deduplicate the elements. Arrays nested more deeply than 2 must be
flattened twice (or more) to get down to an unnested `Array[X]`.
```
https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#arrayx-flattenarrayarrayx

 - [ ] Add the following WDL function to https://github.com/DataBiosphere/toil/blob/master/src/toil/wdl/wdl_functions.py
 - [ ] Add a test that runs a wdl/json with toil and then checks against cromwell to verify that toil and cromwell have the same behavior
       See: https://github.com/DailyDreaming/wdl-conformance-tests/tree/master/tests/standard_library

For example:

```wdl
Array[Array[Int]] ai2D = [[1, 2, 3], [1], [21, 22]]
Array[Int] ai = flatten(ai2D)   # [1, 2, 3, 1, 21, 22]

Array[Array[File]] af2D = [["/tmp/X.txt"], ["/tmp/Y.txt", "/tmp/Z.txt"], []]
Array[File] af = flatten(af2D)   # ["/tmp/X.txt", "/tmp/Y.txt", "/tmp/Z.txt"]

Array[Array[Pair[Float,String]]] aap2D = [[(0.1, "mouse")], [(3, "cat"), (15, "dog")]]

Array[Pair[Float,String]] ap = flatten(aap2D) # [(0.1, "mouse"), (3, "cat"), (15, "dog")]
```

The last example (`aap2D`) is useful because `Map[X, Y]` can be coerced to `Array[Pair[X, Y]]`.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-645)
┆Epic: WDL Standard Library Implementation
┆Issue Number: TOIL-645



Discussion:

Resolving patch:
diff --git a/src/toil/wdl/wdl_functions.py b/src/toil/wdl/wdl_functions.py
--- a/src/toil/wdl/wdl_functions.py
+++ b/src/toil/wdl/wdl_functions.py
@@ -976,3 +976,22 @@ def collect_by_key(in_array: List[WDLPair]) -> dict:
         map.setdefault(pair.left, []).append(pair.right)
 
     return map
+
+
+def flatten(in_array: List[list]) -> list:
+    """
+    Given an array of arrays, the `flatten` function concatenates all the member
+    arrays in the order to appearance to give the result. It does not deduplicate
+    the elements.
+
+    WDL syntax: Array[X] flatten(Array[Array[X]])
+    """
+    assert isinstance(in_array, list), f'flatten() requires "{in_array}" to be a list!  Not: {type(in_array)}'
+
+    arr = []
+
+    for element in in_array:
+        assert isinstance(element, list), f'flatten() requires all collections to be a list!  Not: {type(element)}'
+        arr.extend(element)
+
+    return arr
diff --git a/src/toil/wdl/wdl_synthesis.py b/src/toil/wdl/wdl_synthesis.py
--- a/src/toil/wdl/wdl_synthesis.py
+++ b/src/toil/wdl/wdl_synthesis.py
@@ -146,6 +146,7 @@ def write_modules(self):
                     from toil.wdl.wdl_functions import as_map
                     from toil.wdl.wdl_functions import keys
                     from toil.wdl.wdl_functions import collect_by_key
+                    from toil.wdl.wdl_functions import flatten
                     import fnmatch
                     import textwrap
                     import subprocess


Test patch:
diff --git a/src/toil/test/wdl/builtinTest.py b/src/toil/test/wdl/builtinTest.py
--- a/src/toil/test/wdl/builtinTest.py
+++ b/src/toil/test/wdl/builtinTest.py
@@ -498,6 +498,9 @@ def test_collect_by_key(self):
         # NOTE: this result is expected according to the spec but differs from Cromwell.
         self.check_function('collect_by_key', cases=['as_input'], expected_result='{"a":[1,3],"b":[2]}')
 
+    def test_flatten(self):
+        self.check_function('flatten', cases=['as_input'], expected_result='[1,2,3,1,21,22]')
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/src/toil/test/wdl/standard_library/flatten.json b/src/toil/test/wdl/standard_library/flatten.json
new file mode 100644
--- /dev/null
+++ b/src/toil/test/wdl/standard_library/flatten.json
@@ -0,0 +1,7 @@
+{
+  "flattenWorkflow.in_array": [
+    [1, 2, 3],
+    [1],
+    [21, 22]
+  ]
+}
diff --git a/src/toil/test/wdl/standard_library/flatten_as_input.wdl b/src/toil/test/wdl/standard_library/flatten_as_input.wdl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/wdl/standard_library/flatten_as_input.wdl
@@ -0,0 +1,18 @@
+workflow flattenWorkflow {
+  Array[Array[Int]] in_array
+
+  call copy_output {input: in_array=flatten(in_array)}
+}
+
+task copy_output {
+
+  Array[Int] in_array
+
+  command {
+    cp ${write_json(in_array)} output.txt
+  }
+
+  output {
+    File the_output = 'output.txt'
+  }
+}