Repository: DataBiosphere/toil

Issue:
collect_by_key() implemented as WDL.
Description of WDL function to be added:

```
## Map[X,Array[Y]] collect_by_key(Array[Pair[X,Y]])

Given an Array consisting of Pairs, the `collect_by_key` function returns a Map in which the left elements of the Pairs are the keys and the right elements the values. The left element of the Pairs passed to `collect_by_key` must be a primitive type. The values will be placed in an Array to allow for multiple Pairs to produce the same key. The order of the keys in the Map is the same as the order in the Array based on their first occurence. The order of the elements in the resulting Arrays is the same as their occurence in the given Array of Pairs.
```

 - [ ] Add the following WDL function to https://github.com/DataBiosphere/toil/blob/master/src/toil/wdl/wdl_functions.py
 - [ ] Add a test that runs a wdl/json with toil and then checks against cromwell to verify that toil and cromwell have the same behavior
       See: https://github.com/DailyDreaming/wdl-conformance-tests/tree/master/tests/standard_library

For example:

```wdl
Array[Pair[String,Int]] x = [("a", 1), ("b", 2), ("a", 3)]
Array[Pair[String,Pair[File,File]]] y = [("a", ("a_1.bam", "a_1.bai")), ("b", ("b.bam", "b.bai")), ("a", ("a_2.bam", "a_2.bai"))]

Map[String,Array[Int]] xmap = collect_by_key(x) # {"a": [1, 3], "b": [2]}
Map[String,Array[Pair[File,File]]] ymap = collect_by_key(y) # {"a": [("a_1.bam", "a_1.bai"), ("a_2.bam", "a_2.bai")], "b": [("b.bam", "b.bai")]}
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-643)
┆Epic: WDL Standard Library Implementation
┆Issue Number: TOIL-643



Discussion:

Resolving patch:
diff --git a/src/toil/wdl/wdl_functions.py b/src/toil/wdl/wdl_functions.py
--- a/src/toil/wdl/wdl_functions.py
+++ b/src/toil/wdl/wdl_functions.py
@@ -957,3 +957,22 @@ def keys(in_map: dict) -> list:
     """
 
     return list(in_map.keys())
+
+
+def collect_by_key(in_array: List[WDLPair]) -> dict:
+    """
+    Given an Array consisting of Pairs, the `collect_by_key` function returns a Map
+    in which the left elements of the Pairs are the keys and the right elements the
+    values.
+
+    WDL syntax: Map[X,Array[Y]] collect_by_key(Array[Pair[X,Y]])
+    """
+    if not isinstance(in_array, list):
+        raise WDLRuntimeError(f'as_map() requires "{in_array}" to be a list!  Not: {type(in_array)}')
+
+    map = {}
+
+    for pair in in_array:
+        map.setdefault(pair.left, []).append(pair.right)
+
+    return map
diff --git a/src/toil/wdl/wdl_synthesis.py b/src/toil/wdl/wdl_synthesis.py
--- a/src/toil/wdl/wdl_synthesis.py
+++ b/src/toil/wdl/wdl_synthesis.py
@@ -145,6 +145,7 @@ def write_modules(self):
                     from toil.wdl.wdl_functions import as_pairs
                     from toil.wdl.wdl_functions import as_map
                     from toil.wdl.wdl_functions import keys
+                    from toil.wdl.wdl_functions import collect_by_key
                     import fnmatch
                     import textwrap
                     import subprocess


Test patch:
diff --git a/src/toil/test/wdl/builtinTest.py b/src/toil/test/wdl/builtinTest.py
--- a/src/toil/test/wdl/builtinTest.py
+++ b/src/toil/test/wdl/builtinTest.py
@@ -494,6 +494,10 @@ def test_as_map(self):
     def test_keys(self):
         self.check_function('keys', cases=['as_input'], expected_result='["a","b","c"]')
 
+    def test_collect_by_key(self):
+        # NOTE: this result is expected according to the spec but differs from Cromwell.
+        self.check_function('collect_by_key', cases=['as_input'], expected_result='{"a":[1,3],"b":[2]}')
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/src/toil/test/wdl/standard_library/collect_by_key.json b/src/toil/test/wdl/standard_library/collect_by_key.json
new file mode 100644
--- /dev/null
+++ b/src/toil/test/wdl/standard_library/collect_by_key.json
@@ -0,0 +1 @@
+{}
diff --git a/src/toil/test/wdl/standard_library/collect_by_key_as_input.wdl b/src/toil/test/wdl/standard_library/collect_by_key_as_input.wdl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/wdl/standard_library/collect_by_key_as_input.wdl
@@ -0,0 +1,23 @@
+version development
+
+workflow collectByKeyWorkflow {
+  input {
+    Array[Pair[String, Int]] in_array = [("a", 1), ("b", 2), ("a", 3)]
+  }
+
+  call copy_output {input: in_map=collect_by_key(in_array)}
+}
+
+task copy_output {
+  input {
+    Map[String, Array[Int]] in_map
+  }
+
+  command <<<
+    cp ~{write_json(in_map)} output.txt
+  >>>
+
+  output {
+    File the_output = 'output.txt'
+  }
+}