Repository: pybel/pybel
Pull request #400 (2020-01-05 11:54:30)

Add umbrella node-link exporter
This PR defines and adds an exporter for the the umbrella Node-Link JSON format. It is similar to node-link but uses full BEL terms as nodes.

Given a BEL statement describing that ``X`` phosphorylates ``Y`` like ``act(p(X)) -> p(Y, pmod(Ph))``, PyBEL usually stores the ``act()`` information about ``X`` as part of the relationship. In Umbrella mode, this stays as part of the node. Note that this generates additional nodes in the network for each of the "modified" versions of the node. For example, ``act(p(X))`` will be represented as individual node instead of ``p(X)``, as in the standard node-link JSON exporter.

A user might want to use this exporter in the following scenarios:
- Represent transitivity in activities like in ``p(X, pmod(Ph)) -> act(p(X)) -> p(Y, pmod(Ph)) -> act(p(Y))``
  with four nodes that are more ammenable to simulatons (e.g., boolean networks, petri nets).
- Visualizing networks that in similar way to the legacy BEL [Cytoscape plugin](https://apps.cytoscape.org/apps/belnavigator) from the BEL Framework (warning: now defunct) using tools like Cytoscape.

codecov[bot]: # [Codecov](https://codecov.io/gh/pybel/pybel/pull/400?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@305b99b`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `45.83%`.

[![Impacted file tree graph](https://codecov.io/gh/pybel/pybel/pull/400/graphs/tree.svg?width=650&token=J7joRTRygG&height=150&src=pr)](https://codecov.io/gh/pybel/pybel/pull/400?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master     #400   +/-   ##
=========================================
  Coverage          ?   87.04%           
=========================================
  Files             ?      144           
  Lines             ?     7023           
  Branches          ?      942           
=========================================
  Hits              ?     6113           
  Misses            ?      697           
  Partials          ?      213
```


| [Impacted Files](https://codecov.io/gh/pybel/pybel/pull/400?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/pybel/io/nodelink.py](https://codecov.io/gh/pybel/pybel/pull/400/diff?src=pr&el=tree#diff-c3JjL3B5YmVsL2lvL25vZGVsaW5rLnB5) | `80.51% <35.71%> (ø)` | |
| [src/pybel/io/extras.py](https://codecov.io/gh/pybel/pybel/pull/400/diff?src=pr&el=tree#diff-c3JjL3B5YmVsL2lvL2V4dHJhcy5weQ==) | `84.61% <60%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pybel/pybel/pull/400?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pybel/pybel/pull/400?src=pr&el=footer). Last update [305b99b...ff813e8](https://codecov.io/gh/pybel/pybel/pull/400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ddomingof: @cthoyt I added the tests and the suggested changes. Thanks for the recommendations!
I have a doubt with the naming of the new JSON exporter submodule. I called it for now custom_json but it probably should be canonicalize_json or something like that since it is more implicit what it does. What do you think?
ddomingof: By the way, why are the tests now failing? They should be fine.
cthoyt: I don't like using "canonical" anywhere. It needs to be clear that this type of export is only for backwards compatibility and needs to explain why someone would want to include edge information in the nodes themselves. I think this is unsafe because of the huge combinatorics potential of modifications of any given nodes, especially before normalization of identifiers.

In the graphml example, I renamed this option to "umbrella". Maybe you would like `to_umbrella_nodelink_json()`, etc. since it keeps all information in one place 
ddomingof: Fine, but does "umbrella" means something? I really don't like the name.
cthoyt: Umbrella is the thing that keeps you dry in the rain, but it can also be used to mean "has all of the stuff inside" which is sort of what is going on with this schema
ddomingof: Changes made
ddomingof: @cthoyt how am I supposed to fix this?
"""docstring of pybel.io.umbrella_nodelink:3:Inline literal start-string without end-string.""

cthoyt: I fixed the documentation syntax and updated the text... Please check why the unit tests are failing
ddomingof: @cthoyt the problem was the order of the link dictionaries in Python 3.5 (it was not maintained). In python 3.7 is maintained and it was not creating a problem. I removed that part of the tests and separate them into two different ones (one for edges and one for nodes). 
cthoyt: *Or rather after travis passes
ddomingof: Thanks to you, do you plan to make the release today? I want to release the new version of PathMe with this feature
cthoyt: @ddomingof yeah will make a new release after I merge

Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 2cc22513..34a894df 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -6,7 +6,11 @@ The format is based on `Keep a Changelog <http://keepachangelog.com/>`_
 and this project adheres to `Semantic Versioning <http://semver.org/>`_
 
 `Unreleased <https://github.com/pybel/pybel/compare/v0.14.2...HEAD>`_
-----------------------------------------------------------------------
+---------------------------------------------------------------------
+Added
+~~~~~
+- Umbrella node-link JSON exporter(https://github.com/pybel/pybel/pull/400)
+- GraphML exporter with umbrella nodes (https://github.com/pybel/pybel/pull/400)
 
 `0.14.2 <https://github.com/pybel/pybel/compare/v0.14.1...v0.14.2>`_ - 2019-11-26
 ---------------------------------------------------------------------------------
diff --git a/docs/source/reference/io.rst b/docs/source/reference/io.rst
index 118a6ad7..2129269b 100644
--- a/docs/source/reference/io.rst
+++ b/docs/source/reference/io.rst
@@ -59,6 +59,15 @@ Node-Link JSON
 .. autofunction:: pybel.from_nodelink_file
 .. autofunction:: pybel.to_nodelink_file
 
+Export
+------
+Umbrella Node-Link JSON
+~~~~~~~~~~~~~~~~~~~~~~~
+.. automodule:: pybel.io.umbrella_nodelink
+
+.. autofunction:: pybel.to_umbrella_nodelink
+.. autofunction:: pybel.to_umbrella_nodelink_file
+
 JSON Graph Interchange Format
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. automodule:: pybel.io.jgif
@@ -68,18 +77,25 @@ JSON Graph Interchange Format
 
 .. autofunction:: pybel.from_cbn_jgif
 
-Export
-------
-.. automodule:: pybel.io.extras
+GraphML
+~~~~~~~
+.. automodule:: pybel.io.graphml
 
 .. autofunction:: pybel.to_graphml
+
+Miscellaneous
+~~~~~~~~~~~~~
+.. automodule:: pybel.io.extras
+
 .. autofunction:: pybel.to_csv
 .. autofunction:: pybel.to_sif
 .. autofunction:: pybel.to_gsea
 .. autofunction:: pybel.to_tsv
 
-Database
---------
+Databases
+---------
+SQL Databases
+~~~~~~~~~~~~~
 .. automodule:: pybel.manager.database_io
 
 .. autofunction:: pybel.from_database
diff --git a/src/pybel/canonicalize.py b/src/pybel/canonicalize.py
index acb623b0..3115044e 100644
--- a/src/pybel/canonicalize.py
+++ b/src/pybel/canonicalize.py
@@ -25,6 +25,7 @@ __all__ = [
     'to_bel_script',
     'to_bel_script_lines',
     'edge_to_bel',
+    'edge_to_tuple',
     'calculate_canonical_name',
 ]
 
diff --git a/src/pybel/io/__init__.py b/src/pybel/io/__init__.py
index c1a794bc..aed8febe 100644
--- a/src/pybel/io/__init__.py
+++ b/src/pybel/io/__init__.py
@@ -8,8 +8,9 @@ ability of other existing software is excluded due the immaturity of the BEL to
 """
 
 from .cx import from_cx, from_cx_file, from_cx_gz, from_cx_jsons, to_cx, to_cx_file, to_cx_gz, to_cx_jsons
-from .extras import to_csv, to_graphml, to_gsea, to_sif
+from .extras import to_csv, to_gsea, to_sif
 from .gpickle import from_bytes, from_pickle, to_bytes, to_pickle
+from .graphml import to_graphml
 from .indra import from_biopax, from_indra_pickle, from_indra_statements, to_indra_statements
 from .jgif import (
     from_cbn_jgif, from_jgif, from_jgif_file, from_jgif_gz, from_jgif_jsons, post_jgif, to_jgif,
@@ -22,4 +23,5 @@ from .nodelink import (
     to_nodelink_file, to_nodelink_gz, to_nodelink_jsons,
 )
 from .tsv import to_edgelist, to_tsv
+from .umbrella_nodelink import to_umbrella_nodelink, to_umbrella_nodelink_file
 from .web import from_web, to_web
diff --git a/src/pybel/io/extras.py b/src/pybel/io/extras.py
index 0bf7a1f6..d57a9a6e 100644
--- a/src/pybel/io/extras.py
+++ b/src/pybel/io/extras.py
@@ -3,45 +3,20 @@
 """This module contains IO functions for outputting BEL graphs to lossy formats, such as GraphML and CSV."""
 
 import json
-from typing import BinaryIO, Optional, TextIO, Union
+from typing import Optional, TextIO, Union
 
-import networkx as nx
 from networkx.utils import open_file
 
-from ..constants import RELATION
 from ..dsl import CentralDogma
 from ..struct import BELGraph
 
 __all__ = [
-    'to_graphml',
     'to_csv',
     'to_sif',
     'to_gsea',
 ]
 
 
-def to_graphml(graph: BELGraph, path: Union[str, BinaryIO]) -> None:
-    """Write this graph to GraphML XML file using :func:`networkx.write_graphml`.
-
-    The .graphml file extension is suggested so Cytoscape can recognize it.
-    """
-    rv = nx.MultiDiGraph()
-
-    for node in graph:
-        rv.add_node(node.as_bel(), function=node.function)
-
-    for u, v, key, edge_data in graph.edges(data=True, keys=True):
-        rv.add_edge(
-            u.as_bel(),
-            v.as_bel(),
-            interaction=edge_data[RELATION],
-            bel=graph.edge_to_bel(u, v, edge_data),
-            key=key,
-        )
-
-    nx.write_graphml(rv, path)
-
-
 @open_file(1, mode='w')
 def to_csv(graph: BELGraph, path: Union[str, TextIO], sep: Optional[str] = None) -> None:
     """Write the graph as a tab-separated edge list.
diff --git a/src/pybel/io/graphml.py b/src/pybel/io/graphml.py
new file mode 100644
index 00000000..dc1b5b28
--- /dev/null
+++ b/src/pybel/io/graphml.py
@@ -0,0 +1,80 @@
+# -*- coding: utf-8 -*-
+
+"""Conversion functions for BEL graphs with `GraphML <https://en.wikipedia.org/wiki/GraphML>`_."""
+
+from typing import BinaryIO, Optional, Union
+
+import networkx as nx
+
+from ..canonicalize import edge_to_tuple
+from ..constants import RELATION
+from ..struct import BELGraph
+
+__all__ = [
+    'to_graphml',
+]
+
+
+def to_graphml(graph: BELGraph, path: Union[str, BinaryIO], schema: Optional[str] = None) -> None:
+    """Write a graph to a GraphML XML file using :func:`networkx.write_graphml`.
+
+    :param graph: BEL Graph
+    :param path: Path to the new exported file
+    :param schema: Type of export. Currently supported: "simple" and "umbrella".
+
+    The .graphml file extension is suggested so Cytoscape can recognize it.
+    By default, this function exports using the PyBEL schema of including modifier information into the edges.
+    As an alternative, this function can also distinguish between
+    """
+    if schema is None or schema == 'simple':
+        rv = _to_graphml_simple(graph)
+    elif schema == 'umbrella':
+        rv = _to_graphml_umbrella(graph)
+    else:
+        raise ValueError('Unhandled schema: {}'.format(schema))
+
+    nx.write_graphml(rv, path)
+
+
+def _to_graphml_simple(graph: BELGraph) -> nx.MultiDiGraph:
+    """Convert a BEL graph to a simple graph.
+
+    :param graph: A BEL graph
+    """
+    rv = nx.MultiDiGraph()
+
+    for node in graph:
+        rv.add_node(node.as_bel(), function=node.function)
+
+    for u, v, key, edge_data in graph.edges(data=True, keys=True):
+        u_key, v_key = u.as_bel(), v.as_bel()
+        rv.add_edge(
+            u_key,
+            v_key,
+            key=key,
+            interaction=edge_data[RELATION],
+            bel=graph.edge_to_bel(u, v, edge_data),
+        )
+
+    return rv
+
+
+def _to_graphml_umbrella(graph: BELGraph) -> nx.MultiDiGraph:
+    """Convert a BEL graph to a new graph the nodes as original BEL terms strings.
+
+    :param graph: A BEL graph
+    """
+    rv = nx.MultiDiGraph()
+
+    for u, v, key, edge_data in graph.edges(data=True, keys=True):
+        u_key, _, v_key = edge_to_tuple(u, v, edge_data)
+
+        rv.add_edge(
+            u_key,
+            v_key,
+            key=key,
+            relation=edge_data[RELATION],
+            bel=graph.edge_to_bel(u, v, edge_data),
+        )
+
+    return rv
diff --git a/src/pybel/io/nodelink.py b/src/pybel/io/nodelink.py
index c318d03c..18e9801d 100644
--- a/src/pybel/io/nodelink.py
+++ b/src/pybel/io/nodelink.py
@@ -30,7 +30,10 @@ __all__ = [
 
 
 def to_nodelink(graph: BELGraph) -> Mapping[str, Any]:
-    """Convert this graph to a node-link JSON object."""
+    """Convert this graph to a node-link JSON object.
+
+    :param graph: BEL Graph
+    """
     graph_json_dict = _to_nodelink_json_helper(graph)
 
     # Convert annotation list definitions (which are sets) to canonicalized/sorted lists
@@ -96,6 +99,8 @@ def from_nodelink_jsons(graph_json_str: str, check_version: bool = True) -> BELG
 def _to_nodelink_json_helper(graph: BELGraph) -> Mapping[str, Any]:
     """Convert a BEL graph to a node-link format.
 
+    :param graph: BEL Graph
+
     Adapted from :func:`networkx.readwrite.json_graph.node_link_data`
     """
     nodes = sorted(graph, key=methodcaller('as_bel'))
diff --git a/src/pybel/io/umbrella_nodelink.py b/src/pybel/io/umbrella_nodelink.py
new file mode 100644
index 00000000..0b4c6eb5
--- /dev/null
+++ b/src/pybel/io/umbrella_nodelink.py
@@ -0,0 +1,90 @@
+# -*- coding: utf-8 -*-
+
+"""The Umbrella Node-Link JSON format is similar to node-link but uses full BEL terms as nodes.
+
+Given a BEL statement describing that ``X`` phosphorylates ``Y`` like ``act(p(X)) -> p(Y, pmod(Ph))``,
+PyBEL usually stores the ``act()`` information about ``X`` as part of the relationship. In Umbrella mode,
+this stays as part of the node.
+
+Note that this generates additional nodes in the network for each of the "modified" versions of
+the node. For example, ``act(p(X))`` will be represented as individual node instead of
+``p(X)``, as in the standard node-link JSON exporter.
+
+A user might want to use this exporter in the following scenarios:
+
+- Represent transitivity in activities like in ``p(X, pmod(Ph)) -> act(p(X)) -> p(Y, pmod(Ph)) -> act(p(Y))``
+  with four nodes that are more ammenable to simulatons (e.g., boolean networks, petri nets).
+- Visualizing networks that in similar way to the legacy BEL
+  `Cytoscape plugin <https://apps.cytoscape.org/apps/belnavigator>`_ from the BEL Framework (warning: now defunct)
+  using tools like Cytoscape.
+"""
+
+import json
+from itertools import chain, count
+from typing import Any, Mapping, TextIO, Union
+
+from networkx.utils import open_file
+
+from ..canonicalize import _decanonicalize_edge_node, edge_to_tuple
+from ..constants import GRAPH_ANNOTATION_LIST, OBJECT, SUBJECT
+from ..struct import BELGraph
+
+__all__ = [
+    'to_umbrella_nodelink',
+    'to_umbrella_nodelink_file',
+]
+
+
+def to_umbrella_nodelink(graph: BELGraph) -> Mapping[str, Any]:
+    """Convert this graph to an umbrella node-link JSON object.
+
+    :param graph: A BEL graph
+    """
+    nodes = set()
+
+    for u, v, data in graph.edges(data=True):
+        u_key, _, v_key = edge_to_tuple(u, v, data)
+        nodes.add(u_key)
+        nodes.add(v_key)
+
+    nodes = sorted(list(nodes))
+    mapping = dict(zip(nodes, count()))
+
+    graph_json_dict = {
+        'directed': True,
+        'multigraph': True,
+        'graph': graph.graph.copy(),
+        'nodes': nodes,
+        'links': [
+            dict(
+                chain(
+                    data.copy().items(),
+                    [
+                        ('source', mapping[_decanonicalize_edge_node(u, data, node_position=SUBJECT)]),
+                        ('target', mapping[_decanonicalize_edge_node(v, data, node_position=OBJECT)]),
+                        ('key', key),
+                    ],
+                ),
+            )
+            for u, v, key, data in graph.edges(keys=True, data=True)
+        ],
+    }
+
+    # Convert annotation list definitions (which are sets) to canonicalized/sorted lists
+    graph_json_dict['graph'][GRAPH_ANNOTATION_LIST] = {
+        keyword: list(sorted(values))
+        for keyword, values in graph_json_dict['graph'].get(GRAPH_ANNOTATION_LIST, {}).items()
+    }
+
+    return graph_json_dict
+
+
+@open_file(1, mode='w')
+def to_umbrella_nodelink_file(graph: BELGraph, path: Union[str, TextIO], **kwargs) -> None:
+    """Write this graph as an umbrella node-link JSON to a file.
+
+    :param graph: A BEL graph
+    :param path: A path or file-like
+    """
+    graph_json_dict = to_umbrella_nodelink(graph)
+    json.dump(graph_json_dict, path, ensure_ascii=False, **kwargs)


Test patch:
diff --git a/tests/test_io/test_umbrella_nodelink.py b/tests/test_io/test_umbrella_nodelink.py
new file mode 100644
index 00000000..25c7e0ad
--- /dev/null
+++ b/tests/test_io/test_umbrella_nodelink.py
@@ -0,0 +1,38 @@
+# -*- coding: utf-8 -*-
+
+"""Tests for the umbrella node-link JSON exporter."""
+
+import unittest
+
+from pybel.io.umbrella_nodelink import to_umbrella_nodelink
+from tests.test_io.test_cx.examples import example_graph
+
+
+class TestUmbrellaNodeLinkExporter(unittest.TestCase):
+    """Tests for the umbrella node-link JSON exporter."""
+
+    def test_exporter_new_nodes(self):
+        """Test new nodes created."""
+        # Check original number of nodes and edges in the example BEL Graph
+        self.assertEqual(29, example_graph.number_of_nodes())
+        self.assertEqual(29, example_graph.number_of_edges())
+
+        custom_json_dict = to_umbrella_nodelink(example_graph)
+
+        self.assertEqual(32, len(custom_json_dict['nodes']))
+
+        # 3 new nodes are created:
+        self.assertIn("act(p(hgnc:MAPK1), ma(kin))", custom_json_dict['nodes'])
+        self.assertIn("act(p(hgnc:PTK2, pmod(Ph, Tyr, 925)), ma(kin))", custom_json_dict['nodes'])
+        self.assertIn('act(p(fus(hgnc:BCR, "?", hgnc:ABL1, "?")), ma(kin))', custom_json_dict['nodes'])
+
+    def test_exporter_edges(self):
+        """Test no new edges created."""
+        # Check original number of nodes and edges in the example BEL Graph
+        self.assertEqual(29, example_graph.number_of_nodes())
+        self.assertEqual(29, example_graph.number_of_edges())
+
+        custom_json_dict = to_umbrella_nodelink(example_graph)
+
+        # Number of edges is maintained
+        self.assertEqual(29, len(custom_json_dict['links']))