Repository: networkx/networkx

Issue:
strongly_connected_components too slow: quadratic run time for some graphs
nx.strongly_connected_components(dg) is prohibitively slow for certain graphs.

For example, the code below shows approximately quadratic run time for a very simple non-cyclic DAG having one start node linking to each of N intermediate nodes each linking to an end node.

SCC should run in O(num_nodes + num_edges) using Tarjan's algorithm.

I ran into this in a practical problem (with a much more complex graph) where SCC computation using networkx just seemed to hang for some graphs.

Tested with Python 3.5.2, networkx 2.0 (installed with pip3 install networkx on Jan 13, 2018).

Here's the test code:
```
import time
import networkx as nx
for count in range(10000, 100000, 10000):
    # Create the graph
    dg = nx.DiGraph()
    dg.add_node(0)  # Start node
    dg.add_node(1)  # End node
    # Add intermediate nodes and their edges
    for i in range(2, count):
        dg.add_node(i)  # Add intermediate node
        dg.add_edge(0, i)  # From start node to intermediate
        dg.add_edge(i, 1)  # From intermediate to end node
    # Time finding strongly connected components
    t = time.time()
    ret = tuple(nx.strongly_connected_components(dg))
    print("with {} nodes: {} secs".format(count, time.time() - t))
```


Discussion:
Yes, there must be something wrong in the implementation of `strongly_connected_components`. 

The recursive version of Tarjan's algorithm `strongly_connected_components_recursive` and `kosaraju_strongly_connected_components` both show (using your code) linear scaling in num_nodes + num_edges.  

Where is the bug?  Thanks for the simple example that shows the issue.
Hi, I happened to see this post, and did some profiling. I might be the case that the implementation is correct, just inefficient. Not sure if this will help anyone, but here's the profiling result.
![tarjan](https://user-images.githubusercontent.com/32769797/35210630-20b70272-ffa7-11e7-9599-7b707706089e.PNG)


Hello,
@blueTaxi , thank you for pointing to the exact lines.
@dschult , @hagberg  please check my  [PR](https://github.com/networkx/networkx/pull/2841)

The problem was in the loop that you may see in the message above.
We iterating over and over again by the elements that already have been detected as SCC,
so removing them from the list of nodes improves performance.

I've used simple performance test provided by @tatuylonen 
Also looks like there are big fluctuations in my performance testing - because I did it on my personal laptop.



Performance results `without patch` vs `with patch`
```
Old Performance results:
10000
with 10000 nodes: 2.124323844909668 secs
with 10000 nodes: 0.06797289848327637 secs
iterative / recursive: 31.25251228520619
20000
with 20000 nodes: 8.607456922531128 secs
with 20000 nodes: 0.08330178260803223 secs
iterative / recursive: 103.328604179248
30000
with 30000 nodes: 19.59886121749878 secs
with 30000 nodes: 0.12491989135742188 secs
iterative / recursive: 156.8914366201484
40000
with 40000 nodes: 35.78781795501709 secs
with 40000 nodes: 0.30605173110961914 secs
iterative / recursive: 116.93388508297278
50000
with 50000 nodes: 57.72680306434631 secs
with 50000 nodes: 0.24113011360168457 secs
iterative / recursive: 239.40105282620755


New performance results:
10000
with 10000 nodes: 0.2716481685638428 secs
with 10000 nodes: 0.042136192321777344 secs
iterative / recursive: 6.446908313152117
20000
with 20000 nodes: 0.5712282657623291 secs
with 20000 nodes: 0.08850312232971191 secs
iterative / recursive: 6.45432896292924
30000
with 30000 nodes: 1.0197288990020752 secs
with 30000 nodes: 0.14125633239746094 secs
iterative / recursive: 7.218996003186649
40000
with 40000 nodes: 1.5178308486938477 secs
with 40000 nodes: 0.26900696754455566 secs
iterative / recursive: 5.64234771518492
50000
with 50000 nodes: 2.1965038776397705 secs
with 50000 nodes: 0.34764623641967773 secs
iterative / recursive: 6.318215609813638

```










I think those good results may be special for the specific graphs being tested. Can you describe why this would move the algorithm from quadratic time to linear time?  When I try the new version for random graphs, I get quadratic dependence.

I think the quadratic nature comes from rechecking all the ```w``` in ```v_nbrs``` for being in ```preorder``` each time we come back to ```v``` being on the top of the queue. So, for each neighbor we have to pass through all the previously explored neighbors. 

I tried to create a version that stores the ```v_nbrs``` on the queue with ```v``` but wasn't able to get it working in the time I had. ```v_nbrs``` is used in more than one place and sometimes needs to use ```G[v]```

Thanks everybody for looking at this!
Thank you for pointing on the `queue` I realized that my patch actually not fix the problem. Will try to implement your approach with `queue`.


@ArseniyAntonov  
Maybe the best performance would include a combination of your approach with removing nodes in addition to storing ```v_nbrs``` as a set and popping nodes off it as preorder is checked. G'Luck -- :)
@dschult 
I've added queue and check it on the random graph. 
Here are my results:
for the graph specified in this issue:

```
10000
with 10000 nodes: 0.17738914489746094 secs
with 10000 nodes: 0.04557490348815918 secs
iterative / recursive: 3.8922549763281107
20000
with 20000 nodes: 0.4013180732727051 secs
with 20000 nodes: 0.0905609130859375 secs
iterative / recursive: 4.431471145745577
30000
with 30000 nodes: 0.6294271945953369 secs
with 30000 nodes: 0.1387310028076172 secs
iterative / recursive: 4.537033408950299
40000
with 40000 nodes: 0.9015460014343262 secs
with 40000 nodes: 0.2698240280151367 secs
iterative / recursive: 3.3412369093524568
50000
with 50000 nodes: 1.1479790210723877 secs
with 50000 nodes: 0.35440802574157715 secs
iterative / recursive: 3.239145103078046
60000
with 60000 nodes: 1.2027249336242676 secs
with 60000 nodes: 0.42836713790893555 secs
iterative / recursive: 2.807696546227477
70000
with 70000 nodes: 1.4035720825195312 secs
with 70000 nodes: 0.5212817192077637 secs
iterative / recursive: 2.6925403880509364
80000
with 80000 nodes: 2.561509847640991 secs
with 80000 nodes: 0.5745389461517334 secs
iterative / recursive: 4.458374605930556
90000
with 90000 nodes: 2.9348678588867188 secs
with 90000 nodes: 0.7854900360107422 secs
iterative / recursive: 3.736352753488756
```

For the random graph with 30000 nodes:
`nx.gn_graph(30000)`
```
Attempt 0
with 30000 nodes: 1.2159347534179688e-05 secs
with 30000 nodes: 6.9141387939453125e-06 secs
iterative / recursive: 1.7586206896551724
Attempt 1
with 30000 nodes: 3.0994415283203125e-06 secs
with 30000 nodes: 2.86102294921875e-06 secs
iterative / recursive: 1.0833333333333333
Attempt 2
with 30000 nodes: 2.86102294921875e-06 secs
with 30000 nodes: 3.0994415283203125e-06 secs
iterative / recursive: 0.9230769230769231
Attempt 3
with 30000 nodes: 3.0994415283203125e-06 secs
with 30000 nodes: 1.9073486328125e-06 secs
iterative / recursive: 1.625
```

Please check it also on your side (I did not find any performance tests in networkx)
@dschult is anybody wants to review?
Thanks for the gentle nudge! :)
I looked more at this last week and almost got to a resolution.

I am leaning toward removing the part that copies the graph and removes the scc nodes. The storing of ```nbrs``` seems to be the change that removes quadratic dependence. 

@ArseniyAntonov What do you think about that approach? 

Resolving patch:
diff --git a/networkx/algorithms/components/strongly_connected.py b/networkx/algorithms/components/strongly_connected.py
--- a/networkx/algorithms/components/strongly_connected.py
+++ b/networkx/algorithms/components/strongly_connected.py
@@ -80,6 +80,7 @@ def strongly_connected_components(G):
        Information Processing Letters 49(1): 9-14, (1994)..
 
     """
+    nbrs = {}
     preorder = {}
     lowlink = {}
     scc_found = {}
@@ -94,7 +95,9 @@ def strongly_connected_components(G):
                     i = i + 1
                     preorder[v] = i
                 done = 1
-                v_nbrs = G[v]
+                if v not in nbrs:
+                    nbrs[v] = iter(G[v])
+                v_nbrs = nbrs[v]
                 for w in v_nbrs:
                     if w not in preorder:
                         queue.append(w)
@@ -102,7 +105,7 @@ def strongly_connected_components(G):
                         break
                 if done == 1:
                     lowlink[v] = preorder[v]
-                    for w in v_nbrs:
+                    for w in G[v]:
                         if w not in scc_found:
                             if preorder[w] > preorder[v]:
                                 lowlink[v] = min([lowlink[v], lowlink[w]])


Test patch:
diff --git a/networkx/algorithms/components/tests/test_strongly_connected.py b/networkx/algorithms/components/tests/test_strongly_connected.py
--- a/networkx/algorithms/components/tests/test_strongly_connected.py
+++ b/networkx/algorithms/components/tests/test_strongly_connected.py
@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+import time
 from nose.tools import *
 import networkx as nx
 from networkx import NetworkXNotImplemented
@@ -154,3 +155,29 @@ def test_connected_raise(self):
         assert_raises(NetworkXNotImplemented, nx.condensation, G)
         # deprecated
         assert_raises(NetworkXNotImplemented, nx.strongly_connected_component_subgraphs, G)
+
+#    Commented out due to variability on Travis-CI hardware/operating systems
+#    def test_linear_time(self):
+#        # See Issue #2831
+#        count = 100  # base case
+#        dg = nx.DiGraph()
+#        dg.add_nodes_from([0, 1])
+#        for i in range(2, count):
+#            dg.add_node(i)
+#            dg.add_edge(i, 1)
+#            dg.add_edge(0, i)
+#        t = time.time()
+#        ret = tuple(nx.strongly_connected_components(dg))
+#        dt = time.time() - t
+#
+#        count = 200
+#        dg = nx.DiGraph()
+#        dg.add_nodes_from([0, 1])
+#        for i in range(2, count):
+#            dg.add_node(i)
+#            dg.add_edge(i, 1)
+#            dg.add_edge(0, i)
+#        t = time.time()
+#        ret = tuple(nx.strongly_connected_components(dg))
+#        dt2 = time.time() - t
+#        assert_less(dt2, dt * 2.3)  # should be 2 times longer for this graph