Repository: apache/airflow

Issue:
Tree view is slow for recombining DAGs
The Tree View has a very cool approach to representing graphs that aren't actually trees by letting users collapse and uncollapse leaves that appear in multiple places. Unfortunately, this approach comes with an extreme computational burden since before it can display the tree, it must build a data object containing _every possible path from the root to every single leaf_. When DAGs recombine (meaning if it the edges weren't directed, it would be a cyclic graph), the number of possible paths explodes. For example, my machine totally freezes when trying to load Tree View for this relatively simple graph:

``` python
import airflow
from airflow.operators import DummyOperator
import datetime

dag = airflow.DAG(
    dag_id='break_the_tree',
    default_args=dict(owner='airflow' , start_date=datetime.datetime.now()))

group, prev_group = None, None

for i in range(5):
    ops = [DummyOperator(task_id='dummy{}{}'.format(i, j), dag=dag) for j in range(10)]
    if prev_group:
        prev_group.set_downstream(ops)
    group = DummyOperator(task_id='group{}'.format(i), dag=dag)
    group.set_upstream(ops)
    prev_group = group
```

![screen shot 2015-08-24 at 6 33 37 pm](https://cloud.githubusercontent.com/assets/153965/9454271/ad310714-4a8e-11e5-92b9-aff6ebc62539.png)

However, if we are willing to abandon the collapse/uncollapse functionality, then we could implement a much more efficient traversal algorithm. I did, and now I can generate this Tree View for the above graph almost instantly:
![screen shot 2015-08-25 at 7 50 53 am](https://cloud.githubusercontent.com/assets/153965/9465845/26e90402-4afe-11e5-8dd0-3b14080c936d.png)

I really like the collapse/expand functionality, but I think it's much more important to be able to see the tree than worrying that I'm going to freeze my server by clicking the wrong button. Maybe there's a way to have a checkbox on the Tree View page to turn on the fancy collapse functionality, if the user wants it? I could submit a PR that checks an argument `expand_collapse` and runs the efficient search when it's `False` and the old way if it's `True`, but I don't know how to build the UI piece.



Discussion:
What about adding a url parameter `nocollapse=true` to force your alternate view? Maybe compute the size of the tree and fallback on this view if the tree goes over a limit?


Resolving patch:
diff --git a/airflow/www/app.py b/airflow/www/app.py
--- a/airflow/www/app.py
+++ b/airflow/www/app.py
@@ -12,6 +12,7 @@
 import os
 import socket
 import sys
+import time
 
 from flask._compat import PY2
 from flask import (
@@ -1035,8 +1036,19 @@ def tree(self):
 
         expanded = []
 
-        def recurse_nodes(task):
-            children = [recurse_nodes(t) for t in task.upstream_list]
+        # The default recursion traces every path so that tree view has full
+        # expand/collapse functionality. After 5,000 nodes we stop and fall
+        # back on a quick DFS search for performance. See PR #320.
+        node_count = [0]
+        node_limit = 5000 / len(dag.roots)
+
+        def recurse_nodes(task, visited):
+            visited.add(task)
+            node_count[0] += 1
+
+            children = [
+                recurse_nodes(t, visited) for t in task.upstream_list
+                if node_count[0] < node_limit or t not in visited]
 
             # D3 tree uses children vs _children to define what is
             # expanded or not. The following block makes it such that
@@ -1072,10 +1084,10 @@ def recurse_nodes(task):
             data = {
                 'name': 'root',
                 'instances': [],
-                'children': [recurse_nodes(t) for t in dag.roots]
+                'children': [recurse_nodes(t, set()) for t in dag.roots]
             }
         elif len(dag.roots) == 1:
-            data = recurse_nodes(dag.roots[0])
+            data = recurse_nodes(dag.roots[0], set())
         else:
             flash("No tasks found.", "error")
             data = []


Test patch: