Repository: dask/distributed

Issue:
Profile code raising AttributeError
I've noticed occasional errors with distributed 1.18.1 relating to the profiling code.
![screen shot 2017-10-05 at 22 56 53](https://user-images.githubusercontent.com/3025875/31256246-323ff11c-aa29-11e7-9e6b-5801162066ee.png)

AttributeError: 'NoneType' object has no attribute 'f_code'

Looking through the profiling code (relevant excerpt below), I couldn't immediately see how this was possible, 
```
    if frame.f_back is not None and (stop is None or not frame.f_back.f_code.co_filename.endswith(stop)):
        state = process(frame.f_back, frame, state, stop=stop)
```
unless frame.f_back can return a different result on the different occasions it is inspected.

Unfortunately I haven't yet managed to see this error when running under a debugger.

In case it is relevant: Python 2.7 on Windows.




Discussion:
I am now getting this frequently. Not sure what I changed in my dockers to trigger this.

radiant-dask-worker         | tornado.application - ERROR - Exception in callback <bound method Worker.trigger_profile of <Worker: tcp://172.19.0.5:45231, running, stored: 0, running: 3/4, ready: 0, comm: 0, waiting: 0>>
radiant-dask-worker         | Traceback (most recent call last):
radiant-dask-worker         |   File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 1026, in _run
radiant-dask-worker         |     return self.callback()
radiant-dask-worker         |   File "/usr/local/lib/python2.7/dist-packages/distributed/worker.py", line 2234, in trigger_profile
radiant-dask-worker         |     profile.process(frame, None, self.profile_keys[key], stop='_concurrent_futures_thread.py')
radiant-dask-worker         |   File "/usr/local/lib/python2.7/dist-packages/distributed/profile.py", line 88, in process
radiant-dask-worker         |     state = process(frame.f_back, frame, state, stop=stop)
radiant-dask-worker         |   File "/usr/local/lib/python2.7/dist-packages/distributed/profile.py", line 86, in process
radiant-dask-worker         |     ident = identifier(frame)
radiant-dask-worker         |   File "/usr/local/lib/python2.7/dist-packages/distributed/profile.py", line 42, in identifier
radiant-dask-worker         |     return ';'.join((frame.f_code.co_name,
radiant-dask-worker         | AttributeError: 'NoneType' object has no attribute 'f_code'

If anyone is able to provide a reproducible case for this, that would be very helpful.

Resolving patch:
diff --git a/distributed/profile.py b/distributed/profile.py
--- a/distributed/profile.py
+++ b/distributed/profile.py
@@ -39,9 +39,12 @@ def identifier(frame):
 
     Strings are cheaper to use as indexes into dicts than tuples or dicts
     """
-    return ';'.join((frame.f_code.co_name,
-                     frame.f_code.co_filename,
-                     str(frame.f_code.co_firstlineno)))
+    if frame is None:
+        return 'None'
+    else:
+        return ';'.join((frame.f_code.co_name,
+                         frame.f_code.co_filename,
+                         str(frame.f_code.co_firstlineno)))
 
 
 def repr_frame(frame):
@@ -83,10 +86,11 @@ def process(frame, child, state, stop=None):
      'description': 'root',
      'children': {'...'}}
     """
-    ident = identifier(frame)
     if frame.f_back is not None and (stop is None or not frame.f_back.f_code.co_filename.endswith(stop)):
         state = process(frame.f_back, frame, state, stop=stop)
 
+    ident = identifier(frame)
+
     try:
         d = state['children'][ident]
     except KeyError:


Test patch:
diff --git a/distributed/tests/test_profile.py b/distributed/tests/test_profile.py
--- a/distributed/tests/test_profile.py
+++ b/distributed/tests/test_profile.py
@@ -3,7 +3,8 @@
 from toolz import first
 from threading import Thread
 
-from distributed.profile import process, merge, create, call_stack
+from distributed.profile import (process, merge, create, call_stack,
+        identifier)
 from distributed.compatibility import get_thread_identity
 
 
@@ -106,3 +107,9 @@ def test_call_stack():
     assert isinstance(L, list)
     assert all(isinstance(s, str) for s in L)
     assert 'test_call_stack' in str(L[-1])
+
+
+def test_identifier():
+    frame = sys._current_frames()[get_thread_identity()]
+    assert identifier(frame) == identifier(frame)
+    assert identifier(None) == identifier(None)