Introducing Hiveplotlib. Better Network Visualization in Python... | by Gary Koplik | Towards Data Science

Introducing hiveplotlib— a new, open-source Python package for generating Hive Plots. Originally developed by Martin Krzywinski, Hive Plots generate well-defined figures that allow for interpretable, visual explorations of network data.
The hiveplotlib repository is visible to all on Gitlab, with documentation including further explanation of Hive Plots, examples (both toy data and real data), and full documentation of the code.
Hiveplotlib can be installed via pip:
$ pip install hiveplotlib
Currently, we only support matplotlib-based visualization, but we plan to extend hiveplotlib for use with interactive Python viz packages like bokeh, plotly, and holoviews.
hiveplotlib scales nicely to large network datasets. As a demonstration on synthetic data, let’s build a Hive Plot similar to the “o” in the hiveplotlib logo:
from hiveplotlib import hive_plot_n_axes, Nodefrom hiveplotlib.viz import hive_plot_viz_mplimport numpy as npimport matplotlib.pyplot as pltnum_nodes = 20num_edges = 80np.random.seed(0)# build node datanodes = []for i in range(num_nodes):    temp_node = Node(unique_id=i,                     data={'a': np.random.uniform(),                           'b': np.random.uniform(),                           'c': np.random.uniform()})    nodes.append(temp_node)# give the nodes simple int IDsnode_ids = np.arange(num_nodes)# build random edgesedges = np.random.choice(np.arange(num_nodes),                         size=num_edges*2).reshape(-1, 2)# construct HivePlot instance, evenly spacing out nodes over 3 axeshp = hive_plot_n_axes(node_list=nodes, edges=edges,                      axes_assignments=[                          node_ids[:num_nodes//3],                          node_ids[num_nodes//3:2*num_nodes//3],                          node_ids[2*num_nodes//3:]                      ], sorting_variables=["a", "b", "c"],                      axes_names=["A", "B", "C"],                      vmins=[0, 0, 0], vmaxes=[1, 1, 1],                      orient_angle=-30)# change the line kwargs for edges in plothp.add_edge_kwargs(axis_id_1="A", axis_id_2="B",                   c=f"C0", lw=3, alpha=1, zorder=1.5)hp.add_edge_kwargs(axis_id_1="B", axis_id_2="C",                   c=f"C2", lw=3, alpha=1, zorder=1.5)hp.add_edge_kwargs(axis_id_1="A", axis_id_2="C",                   c=f"C1", lw=3, alpha=1, zorder=1.5)fig, ax = hive_plot_viz_mpl(hive_plot=hp)plt.show()
This code (minus the import statements) runs on a laptop in ~200 ms. If we scale up by changing only the above code to num_nodes = 10000 and num_edges = 10000, the runtime goes up to ~1 second. 100,000 nodes and edges runs in ~10 seconds.
In order to use hiveplotlib, we need to wrangle two sources of data, nodes and edges.
Setting up a node in our framework simply requires a dictionary of data and a unique_id to go along with it. This will play particularly nicely with any json output from a database query (for example, queries from a Mongo database through pymongo). pandas dataframes can also easily be converted into this format with df.to_dict(orient="records").
Edges are stored as an (n, 2) numpy.ndarray, where the edges (if directed) move from the first column to the second column.
To demonstrate the code in action with real data, let’s look at a small but popular example: Zachary’s Karate Club.
From 1970–1972, Wayne W. Zachary observed a karate club split into two factions, those that supported the Club President, referred to as “John A,” and those that supported one of the instructors, referred to as “Mr. Hi.” Eventually, the two factions formally split into two clubs.
This frequently-used dataset contains 34 club members (nodes) and a record of who socialized with whom outside of the class (edges) right before the formal split of the club.
Let’s start by putting all our imports for the below work in one place:
from hiveplotlib import Axis, Node, HivePlotfrom hiveplotlib.viz import axes_viz_mpl, node_viz_mpl, edge_viz_mplfrom matplotlib.lines import Line2Dimport matplotlib.pyplot as pltimport networkx as nximport numpy as np# if you're in a jupyter notebook%matplotlib inline
Grabbing the Karate Club dataset is convenient through networkx:
G = nx.karate_club_graph()
The visualizations of this network — in the original paper, networkx, and even Wikipedia — are always done with a circular graph, so let’s use that as a starting point:
# color the nodes by factioncolor = []for node in G.nodes():    if G.nodes.data()[node]['club'] == "Mr. Hi":        color.append("C0")    else:        color.append("C1")fig, ax = plt.subplots(figsize=(10, 10))plt.axis("equal")nx.draw_circular(G, with_labels=True, node_color=color, ax=ax, node_size=1000)ax.set_title("Zachary's Karate Club\nCircular Network Plot", fontsize=20)# legendjohn_a_legend = Line2D([], [], markerfacecolor="C1", markeredgecolor='C1',                       marker='o', linestyle='None', markersize=10)mr_hi_legend = Line2D([], [], markerfacecolor="C0", markeredgecolor='C0',                       marker='o', linestyle='None', markersize=10)ax.legend([mr_hi_legend, john_a_legend],          ["Mr. Hi", "John A."],          loc='upper left', bbox_to_anchor=(1, 1), title="Faction")plt.show()
One clear and unsurprising conclusion from this graph is that Mr. Hi (node 0) and John A. (node 33) were popular, but it’s hard to conclude much else.
Look at the above figure for roughly 10 seconds, then ask yourself the following questions:
How socially separated are the two factions? How long does it take to confirm there exists a connection between blue and orange?
Are the connections between the two groups from people who are generally more social?
To answer the first question, we could certainly be more careful in ordering these nodes in our plot, partitioning orange from blue, but the second question would still be difficult.
Hive Plots allow us to carefully choose both the axes on which to place nodes and how to align the nodes on those axes.
There is thus a lot of necessary declaration, but with the payoff of a far more interpretable network visualization.
To answer our above questions, we will structure our Hive Plot in the following way:
We will construct a total of 4 axes — 2 axes for the John A. faction, and 2 axes for the Mr. Hi Faction. This use of repeat axes allows us to see the intra-faction behavior in a well-defined way in our resulting visualization.
We will look at 3 sets of edges — edges within the John A. faction, edges within the Mr. Hi faction, and edges between the two factions. This will give us a clear answer to our first question above.
We will sort one axis for each faction by node degree. This allows us to nicely answer our second question above, but more on this later.
Let’s first calculate degree for all of our nodes, and simultaneously build the necessary data structures for hiveplotlib:
edges = np.array(G.edges)# pull out degree information from nodes for later usenode_ids, degrees = np.unique(edges, return_counts=True)nodes = []for node_id, degree in zip(node_ids, degrees):    # store the index number as a way to align the nodes on axes    G.nodes.data()[node_id]['loc'] = node_id    # also store the degree of each node as another way to    #  align nodes on axes    G.nodes.data()[node_id]['degree'] = degree    temp_node = Node(unique_id=node_id,                     data=G.nodes.data()[node_id])    nodes.append(temp_node)
Next, the hiveplotlib component. Let’s build out a HivePlot() instance:
karate_hp = HivePlot()### nodes ###karate_hp.add_nodes(nodes)### axes ###axis0 = Axis(axis_id="hi_id", start=1, end=5, angle=-30,             long_name="Mr. Hi Faction\n(Sorted by ID)")axis1 = Axis(axis_id="hi_degree", start=1, end=5, angle=30,             long_name="Mr. Hi Faction\n(Sorted by Degree)")axis2 = Axis(axis_id="john_degree", start=1, end=5, angle=180 - 30,             long_name="John A. Faction\n(Sorted by Degree)")axis3 = Axis(axis_id="john_id", start=1, end=5, angle=180 + 30,             long_name="John A. Faction\n(Sorted by ID)")axes = [axis0, axis1, axis2, axis3]karate_hp.add_axes(axes)### node assignments #### partition the nodes into "Mr. Hi" nodes and "John A." nodeshi_nodes = [node.unique_id for node in nodes            if node.data['club'] == "Mr. Hi"]john_a_nodes = [node.unique_id for node in nodes                if node.data['club'] == "Officer"]# assign nodes and sorting procedure to position nodes on axiskarate_hp.place_nodes_on_axis(axis_id="hi_id", unique_ids=hi_nodes,                              sorting_feature_to_use="loc",                              vmin=0, vmax=33)karate_hp.place_nodes_on_axis(axis_id="hi_degree",                               unique_ids=hi_nodes,                              sorting_feature_to_use="degree",                              vmin=0, vmax=17)karate_hp.place_nodes_on_axis(axis_id="john_degree",                              unique_ids=john_a_nodes,                              sorting_feature_to_use="degree",                              vmin=0, vmax=17)karate_hp.place_nodes_on_axis(axis_id="john_id",                              unique_ids=john_a_nodes,                              sorting_feature_to_use="loc",                              vmin=0, vmax=33)### edges ###karate_hp.connect_axes(edges=edges, axis_id_1="hi_degree",                       axis_id_2="hi_id", c="C0")karate_hp.connect_axes(edges=edges, axis_id_1="john_degree",                       axis_id_2="john_id", c="C1")karate_hp.connect_axes(edges=edges, axis_id_1="hi_degree",                       axis_id_2="john_degree", c="C2")
As an extension of the hiveplotlib visualization, we will also pull out the John A. and Mr. Hi node placements to plot them in different colors in the final figure.
# pull out the location of the John A. and Mr. Hi nodes#  for visual emphasis laterjohn_a_degree_locations = \    karate_hp.axes["john_degree"].node_placementsjohn_a_node = \    john_a_degree_locations\        .loc[john_a_degree_locations.loc[:, 'unique_id'] == 33,             ['x', 'y']].values.flatten()mr_hi_degree_locations = \    karate_hp.axes["hi_degree"].node_placementsmr_hi_node = \    mr_hi_degree_locations\        .loc[mr_hi_degree_locations.loc[:, 'unique_id'] == 0,             ['x', 'y']].values.flatten()
We’re now ready to plot:
# plot axesfig, ax = axes_viz_mpl(karate_hp,                       axes_labels_buffer=1.4)# plot nodesnode_viz_mpl(karate_hp,             fig=fig, ax=ax, s=80, c="black")# plot edgesedge_viz_mpl(hive_plot=karate_hp, fig=fig, ax=ax, alpha=0.7,             zorder=-1)    ax.set_title("Zachary’s Karate Club\nHive Plot", fontsize=20, y=0.9)
We’ll also add in a the highlighting nodes for Mr. Hi and John A. and a custom legend for reference, all using standard matplotlib:
# highlight Mr. Hi and John. A on the degree axesax.scatter(john_a_node[0], john_a_node[1],           facecolor="red", edgecolor="black", s=150, lw=2)ax.scatter(mr_hi_node[0], mr_hi_node[1],           facecolor="yellow", edgecolor="black", s=150, lw=2)### legend #### edgescustom_lines = [Line2D([0], [0], color=f'C{i}', lw=3, linestyle='-')                for i in range(3)]# John A. and Mr. Hi nodesjohn_a_legend = Line2D([], [], markerfacecolor="red",                        markeredgecolor='black',                       marker='o', linestyle='None', markersize=10)custom_lines.append(john_a_legend)mr_hi_legend = Line2D([], [], markerfacecolor="yellow",                      markeredgecolor='black',                      marker='o', linestyle='None', markersize=10)custom_lines.append(mr_hi_legend)ax.legend(custom_lines, ["Within Mr. Hi Faction",                         "Within John A. Faction",                         "Between Factions",                         "John A.",                         "Mr. Hi"],          loc='upper left', bbox_to_anchor=(0.37, 0.35),          title="Social Connections")plt.show()
Let’s revisit our questions from earlier:
How socially separated are the two factions? How long does it take to confirm there exists a connection between blue and orange?
From this figure, there appear to be far more intra-faction connections than inter-faction connections, but we can clearly see inter-faction connections in green.
Are the connections between the two groups from people who are more generally social?
There does not appear to be a particularly strong correlation between inter-faction connections and general sociability. Otherwise, there would be green connections only between nodes high on each of the degree axes.
The setup costs are of course higher to generate this Hive Plot visualization than the circular layout — we had to make the axes and sorting decisions.
As a reward, however, we can generate unambiguous visualizations that can serve as first steps in answering genuine research questions.
We’re excited to continue development on this project going forward. Some of our planned next steps include, but are not limited to:
Extending the package to work with interactive Python visualization packages.
Extending sorting procedures (scale by monotonic functions, sorting on categorical data).
Manipulation of individual edge weights in the resulting visualization.
“Binned Hive Plots” — drawing connections between binnings of nodes on edges rather than individual node-to-node edges.
Thanks to Geometric Data Analytics for supporting the development and open-sourcing of this project.
We’d also like to thank Rodrigo Garcia-Herrera for his work on pyveplot, which we referenced as a starting point for our structural design. We also translated some of his utility methods for use in this repository.
Our documentation can be found at: https://geomdata.gitlab.io/hiveplotlib/index.html
The Hiveplotlib Gitlab repository: https://gitlab.com/geomdata/hiveplotlib
PyPI: https://pypi.org/project/hiveplotlib/
Another excellent resource on Hive Plots: http://www.hiveplot.com/
If you’re looking for network data to play with, the Stanford Large Network Dataset Collection is an excellent resource: https://snap.stanford.edu/data/
Krzywinski M, Birol I, Jones S, Marra M (2011). Hive Plots — Rational Approach to Visualizing Networks. Briefings in Bioinformatics (early access 9 December 2011, doi: 10.1093/bib/bbr069).
Zachary W. (1977). An information flow model for conflict and fission in small groups. Journal of Anthropological Research, 33, 452–473.