Repository: dask/dask

Issue:
ResourceProfiler raises NoSuchProcess exception
I have hit this bug reliably on 0.11 and 0.14.1.

I am doing the following, for a moderately complicated graph, on a 16 core machine. The profiling code is taken straight from the dask documentation.

```python
from dask.diagnostics import Profiler, ResourceProfiler, CacheProfiler, visualize
from cachey import nbytes

with Profiler() as prof, ResourceProfiler(dt=0.25) as rprof, CacheProfiler(metric=nbytes) as cprof:
    x.compute()
```

It reliably gives the following error:

```pytb
Process _Tracker-50:
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=69516)
Traceback (most recent call last):
  File "/home/pwaller/.local/lib/python3.5/site-packages/psutil/_pslinux.py", line 1332, in wrapper
    return fun(self, *args, **kwargs)
  File "/home/pwaller/.local/lib/python3.5/site-packages/psutil/_pslinux.py", line 1528, in memory_info
    with open_binary("%s/%s/statm" % (self._procfs_path, self.pid)) as f:
  File "/home/pwaller/.local/lib/python3.5/site-packages/psutil/_pslinux.py", line 190, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/69516/statm'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/pwaller/.local/lib/python3.5/site-packages/dask/diagnostics/profile.py", line 231, in run
    mem = sum(p.memory_info().rss for p in ps) / 1e6
  File "/home/pwaller/.local/lib/python3.5/site-packages/dask/diagnostics/profile.py", line 231, in <genexpr>
    mem = sum(p.memory_info().rss for p in ps) / 1e6
  File "/home/pwaller/.local/lib/python3.5/site-packages/psutil/_common.py", line 313, in wrapper
    return fun(self)
  File "/home/pwaller/.local/lib/python3.5/site-packages/psutil/__init__.py", line 1108, in memory_info
    return self._proc.memory_info()
  File "/home/pwaller/.local/lib/python3.5/site-packages/psutil/_pslinux.py", line 1338, in wrapper
    raise NoSuchProcess(self.pid, self._name)
```

I assume the problem here is that the multiprocessing code is waiting on the process before `memory_info()` is called, so the process has gone away by the time the profiler asks for the memory usage.


Discussion:
Thanks for the bug report. Does this error happen early on in the computation, or towards the end? I can see how we could have a race condition in shutdown of the process tracker, but not in startup, just want to make sure that's what you're seeing.
Judging from the time taken for the error to appear, it's at the end, but it's difficult to be certain. It seems likely that it's a race at the end, given the nature of the bug.

Resolving patch:
diff --git a/dask/diagnostics/profile.py b/dask/diagnostics/profile.py
--- a/dask/diagnostics/profile.py
+++ b/dask/diagnostics/profile.py
@@ -228,9 +228,18 @@ def run(self):
                 ps = self._update_pids(pid)
                 while not self.child_conn.poll():
                     tic = default_timer()
-                    mem = sum(p.memory_info().rss for p in ps) / 1e6
-                    cpu = sum(p.cpu_percent() for p in ps)
-                    data.append((tic, mem, cpu))
+                    mem = cpu = 0
+                    for p in ps:
+                        try:
+                            mem2 = p.memory_info().rss
+                            cpu2 = p.cpu_percent()
+                        except Exception: # could be a few different exceptions
+                            pass
+                        else:
+                            # Only increment if both were successful
+                            mem += mem2
+                            cpu += cpu2
+                    data.append((tic, mem / 1e6, cpu))
                     sleep(self.dt)
             elif msg == 'send_data':
                 self.child_conn.send(data)


Test patch: