Repository: bids-standard/pybids

Issue:
GroupBy on Nodes, Filter on Edges
We need to figure out the exact semantics of Filter. It should ideally be something pretty straightforward from a Pandas perspective.

Something like:

```YAML
{
  "Source": ...
  "Destination": ...
  "Filter": {"contrast": ["contrast1", "contrast2"], "group": ["group1", "group2"]}
}
```

This would be read that we keep rows corresponding to `contrast in ["contrast1", "contrast2"] and group in ["group1", "group2"]`. So OR within filter, AND between filters.


Discussion:
In practice, the filtering is already handled by additional keyword arguments to `run()`. We could easily change that behavior so it's passed in as a single `filters` argument though (I have no preference). The actual filtering isn't done in pandas, it's done at the level of collections and contrasts via `bids.utils.matches_entities()`, so the semantics are exactly as you suggest. Only thing we *might* want to add support for at some point (would be easy to do by updating `matches_entities`) would be regexes.
One thing to note is that currently pybids doesn't itself directly use the `Filter` or `GroupBy` values from the node definitions (though it stores them); it expects the user to pass them in to the `run()` call. I think we had agreed that the way to handle this would be to make the defaults for `group_by` reflect what was specified in the node definition (though this isn't currently implemented yet). For `filters` it's a bit trickier though, because when you call `run()`, the node has no way of knowing what edge to look at, and I believe we agreed that the filters should be stored on the edges.

One way to handle this would be to let the caller pass a `BIDSStatsModelsEdge` object as an argument to `run()`, and then take filters from that. But this seems kind of silly seeing as it's literally just the filters that we need. So my inclination is to leave things as is and just expect the user to pass the filters directly.

If we still think we want some kind of "strict" execution mode, where every node is pre-executed exactly per the JSON, the easiest way to do that would be to add a `run()` method at graph level that pre-runs all nodes and stores up the outputs for retrieval. That could return a dictionary with node names as keys and lists of `BIDSStatsModelsNodeOutput` as values.

Resolving patch:
diff --git a/bids/modeling/statsmodels.py b/bids/modeling/statsmodels.py
--- a/bids/modeling/statsmodels.py
+++ b/bids/modeling/statsmodels.py
@@ -50,7 +50,7 @@ def validate_model(model):
 
 
 BIDSStatsModelsEdge = namedtuple('BIDSStatsModelsEdge',
-                                 ('source', 'destination', 'group_by'))
+                                 ('source', 'destination', 'filter'))
 
 
 ContrastInfo = namedtuple('ContrastInfo', ('name', 'conditions', 'weights',
@@ -118,20 +118,14 @@ def _load_edges(model, nodes):
         if not edges or model.get('pipeline', False):
             node_vals = list(nodes.values())
             for i in range(1, len(node_vals)):
-                # by default, we loop over contrast and the level of the
-                # receiving node.
-                group_by = ['contrast']
-                if node_vals[i].level != 'dataset':
-                    group_by.append(node_vals[i].level)
                 edges.append({
                     'source': node_vals[i-1].name,
                     'destination': node_vals[i].name,
-                    'group_by': group_by
                 })
 
         for edge in edges:
             src_node, dst_node = nodes[edge['source']], nodes[edge['destination']]
-            edge = BIDSStatsModelsEdge(src_node, dst_node, edge['group_by'])
+            edge = BIDSStatsModelsEdge(src_node, dst_node, edge.get('filter', {}))
             src_node.add_child(edge)
             dst_node.add_parent(edge)
 
@@ -228,11 +222,19 @@ def __init__(self, level, name, transformations=None, model=None,
         self.transformations = transformations or []
         self.contrasts = contrasts or []
         self.dummy_contrasts = dummy_contrasts
-        self.group_by = group_by or []
         self._collections = []
         self._group_data = []
         self.children = []
         self.parents = []
+        if group_by is None:
+            group_by = []
+            # Loop over contrasts after first level
+            if self.level != "run":
+                group_by.append("contrast")
+            # Loop over node level of this node
+            if self.level != "dataset":
+                group_by.append(self.level)
+        self.group_by = group_by
 
     @staticmethod
     def _build_groups(objects, group_by):
@@ -374,8 +376,7 @@ def run(self, inputs=None, group_by=None, force_dense=True,
 
         inputs = inputs or []
         collections = self._collections
-        group_by = group_by or self.group_by
-        group_by = listify(group_by)
+        group_by = listify(group_by or self.group_by)
 
         # Filter inputs and collections if needed
         if filters:


Test patch:
diff --git a/bids/tests/data/ds005/models/ds-005_type-test_model.json b/bids/tests/data/ds005/models/ds-005_type-test_model.json
--- a/bids/tests/data/ds005/models/ds-005_type-test_model.json
+++ b/bids/tests/data/ds005/models/ds-005_type-test_model.json
@@ -38,6 +38,10 @@
         {
             "Name": "participant",
             "Level": "Subject",
+            "GroupBy": [
+                "subject",
+                "contrast"
+            ],
             "Model": {
                 "X": [
                     1
@@ -50,6 +54,9 @@
         {
             "Name": "by-group",
             "Level": "Dataset",
+            "GroupBy": [
+                "sex"
+            ],
             "Model": {
                 "X": [
                     1
@@ -62,6 +69,7 @@
         {
             "Name": "group-diff",
             "Level": "Dataset",
+	    "GroupBy": [],
             "Model": {
                 "X": [
                     1,
@@ -76,23 +84,15 @@
     "Edges": [
         {
             "Source": "run",
-            "Destination": "participant",
-            "GroupBy": [
-                "subject",
-                "contrast"
-            ]
+            "Destination": "participant"
         },
         {
             "Source": "participant",
-            "Destination": "by-group",
-            "GroupBy": [
-                "sex"
-            ]
+            "Destination": "by-group"
         },
         {
             "Source": "participant",
-            "Destination": "group-diff",
-            "GroupBy": []
+            "Destination": "group-diff"
         }
     ]
-}
\ No newline at end of file
+}