Repository: PaulSchweizer/flowpipe

Issue:
Utility to distribute data into subgraphs
**Is your feature request related to a problem? Please describe.**
I often have subgraphs with several nodes that will take the same input from outside the subgraph (e.g. several nodes taking the same file from a file-reading node). This is not well supported as of now, all options I came up with have drawbacks. 

**Describe the solution you'd like**
I think a way to group inputs would be useful. Such a feature could be used to clearly set up several inputs to take the same information. Connecting all these inputs to the same output can then be done transparently in a single connection operation.

I think that such a grouping might be implemented as a `GroupingPlug`, which inherits from `InputPlug`, providing methods to
* be connected up to an `OutputPlug`, thereby connecting all grouped Inputs with that output
* Add a new `InputPlug` to the group, connecting it with the `OutputPlug` if a connection already exists.
* Disconnect form the output, thereby disconnecting all inputs in the group

**Describe alternatives you've considered**
Right now, there are three options to connecting that up: 

1. Connect the outside nodes directly to the inputs of the nodes within the subgraph. This is not great, since I define the subgraphs to abstract away the internal structure as much as possible.
2. Promote all the input plugs to the subgraph. This is awkward, because the subgraph then has several inputs for the same data.
3. Introduce an extra node that trivially pipes through the data. This way, we can have a single input to the subgraph, however, the extra node makes the graph unnecessarily complicated and, when caching outputs, increases the memory footprint.

**Additional context**
I think such a feature would be useful in all graphs that fan out widely. Right after instantiating a node, one could add the inputs this node shares with other nodes to the group, keeping that behavior neatly in one place. Still, the connection code would be much less boilerplate, since I could just connect `my_file_reader_node.outputs["file"] >> file_consumers`.
By being a plug itself, my suggested implementation above could provide the functionality I'm after: Group all inputs for the same outside data inside a subgraph and expose the bundle as a single unit.

Some illustrations:
Options 1 or 2 of the alternatives yield such a behavior:
![flowpipe_1](https://user-images.githubusercontent.com/23141072/97970271-8e67e800-1dc1-11eb-901d-e250d14e04e9.png)

Option 3 adds a node to the graph, that doesn't do very much and caches the data redundantly, if the output cashing is not disabled:
![flowpipe_2](https://user-images.githubusercontent.com/23141072/97970276-9162d880-1dc1-11eb-83ad-0ef46c15e7e9.png)

The requested feature would provide some utility for the circle in this diagram:
![flowpipe_3](https://user-images.githubusercontent.com/23141072/97970280-932c9c00-1dc1-11eb-8d9d-f8ff6a65475d.png)



Comments:

Patch:
diff --git a/flowpipe/__init__.py b/flowpipe/__init__.py
index a14309f..a48b2ba 100644
--- a/flowpipe/__init__.py
+++ b/flowpipe/__init__.py
@@ -1,4 +1,5 @@
 """Flow-based programming with python."""
 from .graph import Graph
 from .node import INode, Node
-from .plug import InputPlug, OutputPlug, SubInputPlug, SubOutputPlug
+from .plug import (InputPlug, InputPlugGroup, OutputPlug, SubInputPlug,
+                   SubOutputPlug)
diff --git a/flowpipe/graph.py b/flowpipe/graph.py
index 6a27fbc..fb3f3ca 100644
--- a/flowpipe/graph.py
+++ b/flowpipe/graph.py
@@ -19,7 +19,6 @@ try:
 except ImportError:
     from ordereddict import OrderedDict
 
-
 log = logging.getLogger(__name__)
 
 
@@ -32,6 +31,7 @@ class Graph(object):
         self.nodes = nodes or []
         self.inputs = {}
         self.outputs = {}
+        self.input_groups = {}
 
     def __unicode__(self):
         """Display the Graph."""
diff --git a/flowpipe/plug.py b/flowpipe/plug.py
index 0912195..fc3ef4a 100644
--- a/flowpipe/plug.py
+++ b/flowpipe/plug.py
@@ -1,9 +1,12 @@
 """Plugs are ins and outs for Nodes through which they exchange data."""
 from __future__ import print_function
-from abc import abstractmethod
+
 import sys
 import warnings
+from abc import abstractmethod
+
 from .utilities import get_hash
+
 __all__ = ['OutputPlug', 'InputPlug']
 
 try:
@@ -106,6 +109,10 @@ class IPlug(object):
 
     def disconnect(self, plug):
         """Break the connection to the given Plug."""
+        if isinstance(plug, InputPlugGroup):
+            for plug_ in plug:
+                self.disconnect(plug_)
+            return
         if plug in self.connections:
             self.connections.pop(self.connections.index(plug))
             self.is_dirty = True
@@ -133,7 +140,7 @@ class OutputPlug(IPlug):
             name (str): The name of the Plug.
             node (INode): The Node holding the Plug.
         """
-        self.accepted_plugs = (InputPlug,)
+        self.accepted_plugs = (InputPlug, InputPlugGroup)
         super(OutputPlug, self).__init__(name, node)
         if not isinstance(self, SubPlug):
             self.node.outputs[self.name] = self
@@ -160,6 +167,11 @@ class OutputPlug(IPlug):
         if not isinstance(plug, self.accepted_plugs):
             raise TypeError("Cannot connect {0} to {1}".format(
                 type(self), type(plug)))
+        if isinstance(plug, InputPlugGroup):
+            for plug_ in plug:
+                self.connect(plug_)
+            return
+
         if self.node.graph.accepts_connection(self, plug):
             for connection in plug.connections:
                 plug.disconnect(connection)
@@ -390,3 +402,44 @@ class SubOutputPlug(SubPlug, OutputPlug):
             'value': self.value,
             'connections': connections
         }
+
+
+class InputPlugGroup(object):
+    """Group plugs inside a group into one entry point on the graph."""
+
+    def __init__(self, name, graph, plugs=None):
+        """Initialize the group and assigning it to the `Graph.input_groups`.
+
+        Can be connected to an OutputPlug.
+        Args:
+            name (str): The name of the InputPlugGroup.
+            graph (Graph): The Graph holding the PlugGroup.
+            plugs (list of InputPlug): The plugs in this group.
+        """
+        self.name = name
+        self.graph = graph
+        self.plugs = plugs or []
+        self.graph.input_groups[self.name] = self
+
+    def connect(self, plug):
+        """Connect all plugs in this group to the given plug."""
+        for input_plug in self.plugs:
+            plug.connect(input_plug)
+
+    def disconnect(self, plug):
+        """Disconnect all plugs in this group from the given plug."""
+        for input_plug in self.plugs:
+            plug.disconnect(input_plug)
+
+    def __iter__(self):
+        """Convenience to iterate over the plugs in this group."""
+        for plug in self.plugs:
+            yield plug
+
+    def __rshift__(self, other):
+        """Syntactic sugar for the connect() method."""
+        self.connect(other)
+
+    def __lshift__(self, other):
+        """Syntactic sugar for the disconnect() method."""
+        self.disconnect(other)


Test patch:
diff --git a/tests/test_inputpluggroup.py b/tests/test_inputpluggroup.py
new file mode 100644
index 0000000..f5011bc
--- /dev/null
+++ b/tests/test_inputpluggroup.py
@@ -0,0 +1,126 @@
+import pytest
+from flowpipe import Graph, InputPlugGroup, Node
+
+
+@Node(outputs=["out"])
+def DemoNode(in_):
+    """
+    +-----------+
+    | DemoNode  |
+    |-----------|
+    o in_<>     |
+    |     out<> o
+    +-----------+
+    """
+    return {"out": in_}
+
+
+@pytest.fixture
+def demo_graph_fixture():
+    """
+    +---main----+          +---sub----+
+    |     A     |          |    C1    |
+    |-----------|          |----------|
+    o in_<>     |     +--->o in_<>    |
+    |     out<> o-----+    |    out<> o
+    +-----------+     |    +----------+
+                      |    +---sub----+
+                      |    |    C2    |
+                      |    |----------|
+                      +--->o in_<>    |
+                           |    out<> o
+                           +----------+
+    """
+    # Sub graph
+    sub = Graph("sub")
+    c1 = DemoNode(graph=sub, name="C1")
+    c2 = DemoNode(graph=sub, name="C2")
+
+    # Main graph
+    main = Graph("main")
+    DemoNode(graph=main, name="A")
+
+    # Group inputs in the sub graph
+    InputPlugGroup("graph_in", sub, [
+        c1.inputs["in_"],
+        c2.inputs["in_"],
+    ])
+    return sub, main
+
+
+def test_connect_groupinput_to_output(demo_graph_fixture):
+    sub, main = demo_graph_fixture
+    sub.input_groups["graph_in"].connect(main["A"].outputs["out"])
+
+    assert main["A"].outputs["out"] in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] in sub["C2"].inputs["in_"].connections
+
+    sub.input_groups["graph_in"].disconnect(main["A"].outputs["out"])
+
+    assert main["A"].outputs["out"] not in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] not in sub["C2"].inputs["in_"].connections
+
+
+def test_connect_output_to_groupinput(demo_graph_fixture):
+    sub, main = demo_graph_fixture
+    main["A"].outputs["out"].connect(sub.input_groups["graph_in"])
+
+    assert main["A"].outputs["out"] in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] in sub["C2"].inputs["in_"].connections
+
+    main["A"].outputs["out"].disconnect(sub.input_groups["graph_in"])
+
+    assert main["A"].outputs["out"] not in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] not in sub["C2"].inputs["in_"].connections
+
+
+def test_rshift_connect_groupinput_to_output(demo_graph_fixture):
+    sub, main = demo_graph_fixture
+    sub.input_groups["graph_in"] >> main["A"].outputs["out"]
+
+    assert main["A"].outputs["out"] in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] in sub["C2"].inputs["in_"].connections
+
+    sub.input_groups["graph_in"] << main["A"].outputs["out"]
+
+    assert main["A"].outputs["out"] not in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] not in sub["C2"].inputs["in_"].connections
+
+
+def test_rshift_connect_output_to_groupinput(demo_graph_fixture):
+    sub, main = demo_graph_fixture
+    main["A"].outputs["out"] >> sub.input_groups["graph_in"]
+
+    assert main["A"].outputs["out"] in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] in sub["C2"].inputs["in_"].connections
+
+    main["A"].outputs["out"] << sub.input_groups["graph_in"]
+
+    assert main["A"].outputs["out"] not in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"] not in sub["C2"].inputs["in_"].connections
+
+
+def test_connect_groupinput_to_suboutput(demo_graph_fixture):
+    sub, main = demo_graph_fixture
+    sub.input_groups["graph_in"].connect(main["A"].outputs["out"]["1"])
+
+    assert main["A"].outputs["out"]["1"] in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"]["1"] in sub["C2"].inputs["in_"].connections
+
+    sub.input_groups["graph_in"].disconnect(main["A"].outputs["out"]["1"])
+
+    assert main["A"].outputs["out"]["1"] not in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"]["1"] not in sub["C2"].inputs["in_"].connections
+
+
+def test_connect_suboutput_to_groupinput(demo_graph_fixture):
+    sub, main = demo_graph_fixture
+    main["A"].outputs["out"]["1"].connect(sub.input_groups["graph_in"])
+
+    assert main["A"].outputs["out"]["1"] in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"]["1"] in sub["C2"].inputs["in_"].connections
+
+    main["A"].outputs["out"]["1"].disconnect(sub.input_groups["graph_in"])
+
+    assert main["A"].outputs["out"]["1"] not in sub["C1"].inputs["in_"].connections
+    assert main["A"].outputs["out"]["1"] not in sub["C2"].inputs["in_"].connections