Repository: dask/distributed

Issue:
Gather information from running frames
This adds two features:

1.  Captures statistical profiling over time for all worker threads
2.  Adds a `call_stack` method to the client

### Call stack

```python
In [1]: from dask.distributed import Client
c;
In [2]: client = Client()

In [3]: import dask.array as da

In [4]: x = da.random.random(100000, chunks=(1000,)).persist()

In [5]: def f(x):
   ...:     time.sleep(0.1)
   ...:     

In [6]: def g(x):
   ...:     f(x)
   ...:     return x + 1
   ...: 

In [7]: import time

In [12]: y = x.map_blocks(g).map_blocks(g).persist()

In [13]: L = client.call_stack(y)
In [15]: list(L)
Out[15]: 
['tcp://127.0.0.1:40435',
 'tcp://127.0.0.1:38442',
 'tcp://127.0.0.1:39592',
 'tcp://127.0.0.1:41992']

In [16]: list(L['tcp://127.0.0.1:40435'])
Out[16]: ["('g-e33d515806f427a08fc14397bfb96bb2', 42)"]

In [17]: print(''.join(L['tcp://127.0.0.1:40435']["('g-e33d515806f427a08fc14397bfb96bb2', 42)"]))
  File "/home/mrocklin/Software/anaconda/lib/python3.6/threading.py", line 884, in _bootstrap
	self._bootstrap_inner()
  File "/home/mrocklin/Software/anaconda/lib/python3.6/threading.py", line 916, in _bootstrap_inner
	self.run()
  File "/home/mrocklin/Software/anaconda/lib/python3.6/threading.py", line 864, in run
	self._target(*self._args, **self._kwargs)
  File "/home/mrocklin/workspace/distributed/distributed/threadpoolexecutor.py", line 45, in _worker
	task.run()
  File "/home/mrocklin/workspace/distributed/distributed/_concurrent_futures_thread.py", line 64, in run
	result = self.fn(*self.args, **self.kwargs)
  File "/home/mrocklin/workspace/distributed/distributed/worker.py", line 709, in apply_function
	result = function(*args, **kwargs)
  File "/home/mrocklin/workspace/distributed/distributed/worker.py", line 639, in execute_task
	return func(*map(execute_task, args))
  File "<ipython-input-6-b4c4ee315192>", line 2, in g
	  File "<ipython-input-5-0e89f8acd5c2>", line 2, in f
```

### Profiling

Every 10ms we now check on what all of the worker threads are doing.  We store this information in a tree structure that captures both counts and relations between calls in the stack.

```python
In [18]: d = client.run(lambda dask_worker: dask_worker.profile_recent)

In [19]: list(d)
Out[19]: 
['tcp://127.0.0.1:38442',
 'tcp://127.0.0.1:39592',
 'tcp://127.0.0.1:40435',
 'tcp://127.0.0.1:41992']

In [20]: d['tcp://127.0.0.1:38442']
Out[20]: 
{'children': {'_apply_random;/home/mrocklin/workspace/dask/dask/array/random.py;355': {'children': {},
   'count': 1,
   'description': '  File "/home/mrocklin/workspace/dask/dask/array/random.py", line 357, in _apply_random\n\tstate = np.random.RandomState(state_data)\n'},
  'execute_task;/home/mrocklin/workspace/distributed/distributed/worker.py;628': {'children': {'execute_task;/home/mrocklin/workspace/distributed/distributed/worker.py;628': {'children': {'g;<ipython-input-6-b4c4ee315192>;1': {'children': {'f;<ipython-input-5-0e89f8acd5c2>;1': {'children': {},
         'count': 502,
         'description': '  File "<ipython-input-5-0e89f8acd5c2>", line 2, in f\n\t'}},
       'count': 502,
       'description': '  File "<ipython-input-6-b4c4ee315192>", line 2, in g\n\t'}},
     'count': 502,
     'description': '  File "/home/mrocklin/workspace/distributed/distributed/worker.py", line 639, in execute_task\n\treturn func(*map(execute_task, args))\n'},
    'g;<ipython-input-6-b4c4ee315192>;1': {'children': {'f;<ipython-input-5-0e89f8acd5c2>;1': {'children': {},
       'count': 501,
       'description': '  File "<ipython-input-5-0e89f8acd5c2>", line 2, in f\n\t'}},
     'count': 501,
     'description': '  File "<ipython-input-6-b4c4ee315192>", line 2, in g\n\t'}},
   'count': 1003,
   'description': '  File "/home/mrocklin/workspace/distributed/distributed/worker.py", line 639, in execute_task\n\treturn func(*map(execute_task, args))\n'}},
 'count': 1004,
 'description': 'root'}
```

### Future

We should figure out nice ways to format and expose this information.


Discussion:

Resolving patch:
diff --git a/distributed/executor.py b/distributed/executor.py
--- a/distributed/executor.py
+++ b/distributed/executor.py
@@ -250,7 +250,7 @@ class Executor(object):
     distributed.scheduler.Scheduler: Internal scheduler
     """
     def __init__(self, address=None, start=True, loop=None, timeout=3,
-                 set_as_default=False):
+                 set_as_default=True):
         self.futures = dict()
         self.refcount = defaultdict(lambda: 0)
         self._should_close_loop = loop is None and start


Test patch:
diff --git a/distributed/tests/test_executor.py b/distributed/tests/test_executor.py
--- a/distributed/tests/test_executor.py
+++ b/distributed/tests/test_executor.py
@@ -3005,6 +3005,9 @@ def test_default_get(loop):
         e.shutdown()
         assert _globals['get'] is pre_get
 
+        with Executor(('127.0.0.1', s['port']), loop=loop) as e:
+            assert _globals['get'] == e.get
+
 
 @gen_cluster(executor=True)
 def test_get_stacks_processing(e, s, a, b):