Repository: networkx/networkx

Issue:
Mistake in computation of average neighbor degree for directed networks (migrated from Trac #639)
Original ticket https://networkx.lanl.gov/trac/ticket/639
Reported 2011-09-27 by Oleguer Sagarra <oleguer.sagarra, assigned to @hagberg.

There is a problem in the algorithm to compute the negihbor degree (and average neighbor degree) for directed networks. The thing is that as networkx is based on a dict of dicts one may follow the neighbors of a node "forward" but not "backwards", meaning that with the functions such as G.in_degree(G[n]) one lists the incoming degree of the neighbors of the node "n", neighbors meaning destination of directed edges starting at node n. For all this, the normalization factor used is wrong. Additionally, it has some bugs with division over zero values.

These features need to be better redefined for various reasons:

a) With directed networks one can compute 4 types of features divided into two subgroups:
       a.1) For the neighbors of a node n (by neighbors meaning ending points of directed edges starting at node n) one can compute the Incoming our outgoing degree of a node.
       a.2) For all predecessors of node n (meaning all sources of edges finishing in node n) one can compute the very same thing.

b) Additionally, accounting for the average connectivity (or average neighbor degree as a function of degree of a node) things get messy, because then one may compute 8 different features divided into two groups, providing different information:
        b.1) Consider all the neighbors (defined earlier) of node n. Consider all the nodes n in the net. For each node n compute the (in-degree, out-degree) of their neighbors, then average over all n nodes of (in-degree, out-degree) 'k'.
       b.2) The same for predecessors of degree n.

The current code can be adapted to do so, using the function G.reverse() to overcome the difficulty in quickly computing the predecessors of a node while keeping the code structure. But the documentation need to be rewritten and the normalization of the averages redone.

Reference :  Serrano, M.A., Maguitman, A., Boguna, M., Fortunato, S. & Vespignani, A. Decoding the structure of the WWW: facts versus sampling biases. Main 10 (2005).at http://arxiv.org/abs/cs/0511035

Simple example showing what I mean:
Case 1: Error in normalization division by 0

In [1]: import networkx as nx

In [2]: G=nx.DiGraph()

In [3]: G.add_edges_from(([1,2],[1,3],[1,4]))
## In [6]: nx.average_in_degree_connectivity(G)

ZeroDivisionError                         Traceback (most recent call last)
/Users/ugalss/<ipython-input-6-e2e455d286f8> in <module>()
----> 1 nx.average_in_degree_connectivity(G)

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/networkx-1.5-py2.7.egg/networkx/algorithms/neighbor_degree.pyc in average_in_degree_connectivity(G, nodes, weighted)
    205         raise nx.NetworkXError("Not defined for undirected graphs.")
    206     degree_method = G.in_degree
--> 207     return _avg_deg_conn(G, degree_method, nodes=nodes, weighted=weighted)
    208 average_in_degree_connectivity.__doc__=average_degree_connectivity.**doc**
    209 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/networkx-1.5-py2.7.egg/networkx/algorithms/neighbor_degree.pyc in _avg_deg_conn(G, degree_method, nodes, weighted)
    141         dc[k]=avg
    142         if avg > 0:
--> 143             dc[k]/=dnorm[k]
    144     return dc
    145 

ZeroDivisionError: float division by zero

Case 2: Wrong computation of neighbor degree (incoming):

In [7]: G.add_edges_from(([2,3],[3,4],[4,2]))
In [8]: nx.average_neighbor_in_degree(G)
Out[8]: {1: 6.0, 2: 1.0, 3: 1.0, 4: 1.0} # This is wrong. The average in-degree should be 2 for neighbors of node 1.

I attach a new version of neighbor_degree.py with proposed (and commented) changes...



Discussion:
Attachment in Trac by Oleguer Sagarra <oleguer.sagarra, 2011-09-27: [neighbor_degree.py](https://networkx.lanl.gov/trac/attachment/ticket/639/neighbor_degree.py)

Comment in Trac by Oleguer Sagarra <oleguer.sagarra, 2011-09-27

I can add the feature to compute the different pairs (in-out, out-out, out-in, in-in) but then the structure of calling the functions should be adapted...

Comment in Trac by @hagberg, 2011-09-27

Comment in Trac by trac user ugalss, 2011-09-28

Well, I corrected the normalization issues, I added the option to compute the predecessor (mean) degree of each node.

Now is just a matter of adapting the k-nearest neighbors and predecessors to be computed using the 4 possible pairs and further test beign done, although I checked and it seems to work all right.

The coding is not quite elegant, but it works. Although the option G.reverse() might take a while for heavy networks (have not checked).

See attachment neighbor&precedessor_degree.py

Comment in Trac by trac user ugalss, 2011-09-30

Added examples to the documentation. Just need to apply the dictionary solution used in #640 to the neighbor /predecessor connectivity for all possible pairs.

Attachment in Trac by trac user ugalss, 2011-09-30: [neighbor&predecessor_degree.py](https://networkx.lanl.gov/trac/attachment/ticket/639/neighbor&predecessor_degree.py)

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-02

In [f77d9a37a602713db4f493da3a20d39e89b86549/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="f77d9a37a602713db4f493da3a20d39e89b86549"
Move mixing.py and neighbor_degree.py and tests into assortativity module.
Addresses #640 and #639
'''

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-02

In [d2e03fca6b9e8e06cefb0c6925da8946d30c98cb/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="d2e03fca6b9e8e06cefb0c6925da8946d30c98cb"
Refactor assortativity into parts (pairs, mixing, correlation,...)
Addresses #639 and #640
'''

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-02

In [4274d6966b89917efd983ad88c105447e2efacb1/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="4274d6966b89917efd983ad88c105447e2efacb1"
Refactor tests for assortativity. Addresses #640 and #639
'''

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-02

In [cc05a4fac66798418a8e45cd1ec360eed2756b8c/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="cc05a4fac66798418a8e45cd1ec360eed2756b8c"
Update docs.  Simplify nbr_connectivity.
Addresses #639
'''

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-02

In [1c9a21261f54a5ba029cebfdc25422679aecbed6/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="1c9a21261f54a5ba029cebfdc25422679aecbed6"
Allow all options for average neighbor degree (combinations of in and out degree).  Addresses #639
'''

Comment in Trac by @hagberg, 2011-10-02

Replying to [comment:17 Aric Hagberg <aric.hagberg@…>]:

> In [1c9a21261f54a5ba029cebfdc25422679aecbed6/networkx]:
> '''
> #!CommitTicketReference repository="networkx" revision="1c9a21261f54a5ba029cebfdc25422679aecbed6"
> Allow all options for average neighbor degree (combinations of in and out degree).  Addresses #639
> '''

Is this the right approach here?  Needs tests and also definitions in the docs.

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-02

In [273ef2a637c15ebd7069d05d545a8103ffe5e9d7/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="273ef2a637c15ebd7069d05d545a8103ffe5e9d7"
Update documentation in neighbor degree.
Addresses #639
'''

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-02

In [0bf086792199676f116aefbf5ba9661831e70387/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="0bf086792199676f116aefbf5ba9661831e70387"
Allow in, out, and in+out degree in neighbor connectivity. Addresses #639
'''

Comment in Trac by @hagberg, 2011-10-02

Replying to [comment:20 Aric Hagberg <aric.hagberg@…>]:

> In [0bf086792199676f116aefbf5ba9661831e70387/networkx]:
> '''
> #!CommitTicketReference repository="networkx" revision="0bf086792199676f116aefbf5ba9661831e70387"
> Allow in, out, and in+out degree in neighbor connectivity. Addresses #639
> '''

Here is another approach to the various degree combinations using "in", "out" and "in+out".
Is this better?  Are the definitions right?  Needs updated documentation and tests.

Comment in Trac by Oleguer Sagarra <oleguer.sagarra, 2011-10-03

Hi Aric, just began working.. let me take a look during the week and I'll get back to you on the weekend! Anyhow, I think the approach is ok but need to be corrected again for normalization issues. Also, I don't seem to see how did you implement the predecessor connectivity proposed (see first post, there are 4 sets of different data to be computed, grouped in two groups: neighbors and predecessors) but need to take a deeper look.

Will get back to you.

Comment in Trac by Aric Hagberg <aric.hagberg, 2011-10-07

In [9bc6654ed10644abea0e607618b34ef5432020e5/networkx]:
'''
#!CommitTicketReference repository="networkx" revision="9bc6654ed10644abea0e607618b34ef5432020e5"
Fix == bug.
Addresses #639
'''

Comment in Trac by @hagberg, 2011-10-10

Comment in Trac by trac user ugalss, 2011-11-01

Sorry for the delay. All seems to be ok now. For the predecessors calculation one just needs to reverse the net prior to running the function.

Comment in Trac by @hagberg, 2011-11-06

Actually I think that was a bug.  I made a fix in #650 that I believe now handles the predecessors part correctly.

Comment in Trac by trac user ugalss, 2011-11-06

All good now then :) It seems to work allright.

Comment in Trac by @hagberg, 2011-11-14

Comment in Trac by trac user ugalss, 2012-01-24

Detected a minor bug. Just need to add 2 lines of code to solve it.
I attach the new file here for version 1.7

Comment in Trac by @hagberg, 2012-01-28

Can you add a comment about the bug and a test that shows the bug/fix?

Comment in Trac by unknown, 2012-01-29

When calling the function "average_degree_connectivity" using the option [ source="in"], the function fails due to the call of the function _avg_deg_conn that goes through the successors of each node i but in the weighted option computes the edges of the directed links connecting i with its predecessors, and not the other way around. It is enough to add 2 lines of code in _Avg_deg_conn as done in the attached file.

######### Here I add an example of the bug  ###################

'''
In [16]: import networkx as nx
In [17]: G=nx.DiGraph()
In [18]: G.add_edge(1,3,weight=3) # node 1 connects to 3, but 3 does not connect to one
In [19]: G.add_edge(1,2,weight=3) # node 1 connects to 2, but 2 does not connect to one.

## In [21]: nx.assortativity.connectivity.average_degree_connectivity(G,source='in',target='out',weight='weight')

KeyError                                  Traceback (most recent call last)
/Users/ugalss/<ipython-input-21-1a44745de4fa> in <module>()
----> 1 nx.assortativity.connectivity.average_degree_connectivity(G,source='in',target='out',weight='weight')

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/networkx/algorithms/assortativity/connectivity.pyc in average_degree_connectivity(G, source, target, nodes, weight)
    112             neighbors=G.successors
    113     return _avg_deg_conn(G, neighbors, source_degree, target_degree,
--> 114                          nodes=nodes, weight=weight)
    115 
    116 k_nearest_neighbors=average_degree_connectivity

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/networkx/algorithms/assortativity/connectivity.pyc in _avg_deg_conn(G, neighbors, source_degree, target_degree, nodes, weight)
     23         else: # weight nbr degree by weight of (n,nbr) edge
     24             s = float(sum((G[n][nbr].get(weight,1)*d 
---> 25                            for nbr,d in nbrdeg.items())))
     26         dnorm[k] += source_degree(n, weight=weight)
     27         dsum[k] += s

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/networkx/algorithms/assortativity/connectivity.pyc in <genexpr>((nbr, d))
     23         else: # weight nbr degree by weight of (n,nbr) edge
     24             s = float(sum((G[n][nbr].get(weight,1)*d 
---> 25                            for nbr,d in nbrdeg.items())))
     26         dnorm[k] += source_degree(n, weight=weight)
     27         dsum[k] += s

KeyError: 1
'''
(It fails because it fetchs the link between 3 and 1, whereas it should check the link between 1 and 3).

Comment in Trac by @hagberg, 2012-04-08

I'm not sure that is the correct fix.  Can you provide some test cases for directed weighted and unweighted graphs?

Comment in Trac by unknown, 2012-04-09

It does not seem to work with negative weights, but for the rest it seems ok:

In [19]: G=nx.DiGraph()

In [20]: G.add_edge(1,2,weight=1)

In [21]: G.add_edge(1,3,weight=2)

In [22]: G.add_edge(1,4,weight=3)

In [23]: G.add_edge(4,2,weight=3000)

In [24]: nx.assortativity.connectivity.average_degree_connectivity(G,source='in',target='out',weight=None) 
Out[24]: {0: 0.0, 1: 3.0, 2: 2.0} # Ok (uweighted)

In [25]: nx.assortativity.connectivity.average_degree_connectivity(G,source='in',target='out',weight='weight')
Out[25]: {0: 0.0, 1: 3.0, 2: 1.0006664445184938} # Ok (weighted)

Comment in Trac by @hagberg, 2012-04-15

It still fails for the example above.
'''
In [11]: G=nx.DiGraph()

In [12]: G.add_edge(1,2,weight=3) 

In [13]: G.add_edge(1,3,weight=3) 

## In [14]: nx.average_degree_connectivity(G,source='in',target='out',weight='weight')

KeyError                                  Traceback (most recent call last)

/networkx-master/<ipython console> in <module>()

/home/aric/Software/networkx/networkx-master/networkx/algorithms/assortativity/connectivity.py in average_degree_connectivity(G, source, target, nodes, weight)
    112             neighbors=G.successors
    113     return _avg_deg_conn(G, neighbors, source_degree, target_degree,
--> 114                          nodes=nodes, weight=weight)
    115 
    116 k_nearest_neighbors=average_degree_connectivity

/networkx-master/networkx/algorithms/assortativity/connectivity.py in _avg_deg_conn(G, neighbors, source_degree, target_degree, nodes, weight)
     23         else: # weight nbr degree by weight of (n,nbr) edge
     24             s = float(sum((G[n][nbr].get(weight,1)*d 
---> 25                            for nbr,d in nbrdeg.items())))
     26         dnorm[k] += source_degree(n, weight=weight)
     27         dsum[k] += s

/networkx-master/networkx/algorithms/assortativity/connectivity.py in <genexpr>((nbr, d))
     23         else: # weight nbr degree by weight of (n,nbr) edge
     24             s = float(sum((G[n][nbr].get(weight,1)*d 
---> 25                            for nbr,d in nbrdeg.items())))
     26         dnorm[k] += source_degree(n, weight=weight)
     27         dsum[k] += s

KeyError: 1

In [15]: nx.average_degree_connectivity(G,source='in',target='out')
Out[15]: {0: 0.0, 1: 2.0} # is this correct?
'''

Attachment in Trac by trac user Oleguer Sagarra, 2012-04-16: [connectivity.py](https://networkx.lanl.gov/trac/attachment/ticket/639/connectivity.py)

Comment in Trac by trac user Oleguer Sagarra, 2012-04-16

The problem is that the G.adjacency_iter iterator goes through all edges in a forward manner, hence in non symmetric networks some links are missing. I added a fix to the problem, but I suggest that this routine should be re-written using the function G.predecessors() if its performance is too slow (I don't know which way is faster).

I uploaded the file connectivity.py.

Comment in Trac by @hagberg, 2012-05-27

  I'm not sure that is the correct fix.  And for sure we don't want to pass on a keyerror in the inner loop.  

Can you provide some tests cases that show this bug and fix?

Comment in Trac by @hagberg, 2012-06-01

Comment in Trac by @hagberg, 2012-06-12


Resolving patch:
diff --git a/networkx/algorithms/assortativity/connectivity.py b/networkx/algorithms/assortativity/connectivity.py
--- a/networkx/algorithms/assortativity/connectivity.py
+++ b/networkx/algorithms/assortativity/connectivity.py
@@ -21,8 +21,15 @@ def _avg_deg_conn(G, neighbors, source_degree, target_degree,
         if weight is None:
             s = float(sum(nbrdeg.values()))
         else: # weight nbr degree by weight of (n,nbr) edge
-            s = float(sum((G[n][nbr].get(weight,1)*d 
-                           for nbr,d in nbrdeg.items())))
+            if neighbors == G.neighbors:
+                s = float(sum((G[n][nbr].get(weight,1)*d 
+                               for nbr,d in nbrdeg.items())))
+            elif neighbors == G.successors:
+                s = float(sum((G[n][nbr].get(weight,1)*d 
+                               for nbr,d in nbrdeg.items())))
+            elif neighbors == G.predecessors:
+                s = float(sum((G[nbr][n].get(weight,1)*d 
+                               for nbr,d in nbrdeg.items())))
         dnorm[k] += source_degree(n, weight=weight)
         dsum[k] += s
         


Test patch:
diff --git a/networkx/algorithms/assortativity/tests/test_connectivity.py b/networkx/algorithms/assortativity/tests/test_connectivity.py
--- a/networkx/algorithms/assortativity/tests/test_connectivity.py
+++ b/networkx/algorithms/assortativity/tests/test_connectivity.py
@@ -108,3 +108,14 @@ def test_zero_deg(self):
         assert_equal(c,{0:0,3:1})
 
 
+    def test_in_out_weight(self):
+        from itertools import permutations
+        G=nx.DiGraph()
+        G.add_edge(1,2,weight=1)
+        G.add_edge(1,3,weight=1)
+        G.add_edge(3,1,weight=1)
+        for s,t in permutations(['in','out','in+out'],2):
+            c = nx.average_degree_connectivity(G, source=s, target=t)
+            cw = nx.average_degree_connectivity(G,source=s, target=t, 
+                                                weight='weight')
+            assert_equal(c,cw)