Repository: Parsl/parsl

Issue:
monitor() KeyError
I've had the following printed to the console during a run (that otherwise appears to be running OK).

My guess is that a process exited between being detected as a child and information being asked of it, so the relevant `/proc/<pid>` entry disappeared.

I suspect (though I haven't checked) that this has killed off future resource monitoring for this task.

```
Process Process-5:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/_common.py", line 340, in wrapper
    ret = cache[fun]
KeyError: <function Process.memory_info at 0x7f99ef24d378>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/_pslinux.py", line 1390, in wrapper
    return fun(self, *args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/_pslinux.py", line 1594, in memory_info
    with open_binary("%s/%s/statm" % (self._procfs_path, self.pid)) as f:
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/_pslinux.py", line 187, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/14862/statm'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/python/miniconda3-4.5.4/envs/lsst-scipipe-fcd27eb/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/lsst/software/stack/python/miniconda3-4.5.4/envs/lsst-scipipe-fcd27eb/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/benc/parsl/src/parsl/parsl/monitoring/monitoring.py", line 471, in monitor
    for k, v in child.as_dict(attrs=summable_values).items():
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/__init__.py", line 528, in as_dict
    ret = meth()
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/__init__.py", line 1087, in memory_percent
    metrics = fun()
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/_common.py", line 342, in wrapper
    ret = cache[fun] = fun(self)
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/__init__.py", line 1049, in memory_info
    return self._proc.memory_info()
  File "/opt/lsst/software/stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/python_psutil/5.4.3+2/lib/python/psutil-5.4.3-py3.6-linux-x86_64.egg/psutil/_pslinux.py", line 1401, in wrapper
    raise NoSuchProcess(self.pid, self._name)
psutil._exceptions.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=14862)
```


Discussion:
As discussion on slack, should change the wrapper as follows.
```
try:
    assemble message and send 
except e:
    log that exception;   
sleep;  
```



Resolving patch:
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -482,10 +482,11 @@ def monitor(pid, task_id, monitoring_hub_url, run_id, sleep_dur=10):
                     logging.exception("Exception reading IO counters for child {k}. Recorded IO usage may be incomplete".format(k=k), exc_info=True)
                     d['psutil_process_disk_write'] += 0
                     d['psutil_process_disk_read'] += 0
-
-        finally:
             logging.debug("sending message")
             radio.send(MessageType.TASK_INFO, task_id, d)
-            logging.debug("sleeping")
-            time.sleep(sleep_dur)
             first_msg = False
+        except Exception:
+            logging.exception("Exception getting the resource usage. Not sending usage to Hub", exc_info=True)
+
+        logging.debug("sleeping")
+        time.sleep(sleep_dur)


Test patch: