Repository: Parsl/parsl

Issue:
monitoring does not always record task_time_running and hostname fields (on short jobs?)
I see missing hostname and task_time_running for the first task, `create_empty_repo`, which runs extremely fast.

```
sqlite> select run_id, task_func_name, hostname, task_time_submitted, task_time_running, task_time_returned from task;
37cb5ae7-f4f9-40a7-86ec-9457760344f7|create_empty_repo||2019-06-04 04:13:31.476801||2019-06-04 04:13:31.553611
37cb5ae7-f4f9-40a7-86ec-9457760344f7|install_transmission_curves|cori06|2019-06-04 04:13:31.554481|2019-06-04 04:13:31.644297|2019-06-04 04:13:36.155113
37cb5ae7-f4f9-40a7-86ec-9457760344f7|import_ci_hsc|cori06|2019-06-04 04:13:31.560238|2019-06-04 04:13:31.650255|2019-06-04 04:18:30.889769
37cb5ae7-f4f9-40a7-86ec-9457760344f7|pccd_show|cori06|2019-06-04 04:18:30.891500|2019-06-04 04:18:30.989590|2019-06-04 04:18:49.810345
37cb5ae7-f4f9-40a7-86ec-9457760344f7|pccd_process|cori06|2019-06-04 04:18:30.898264|2019-06-04 04:18:31.031377|2019-06-04 05:26:59.966683
37cb5ae7-f4f9-40a7-86ec-9457760344f7|makeDiscreteSkyMap|cori06|2019-06-04 05:26:59.968017|2019-06-04 05:27:00.054919|2019-06-04 05:27:08.146285
```


Discussion:
The two fields are from the first message from the monitoring wrapper of task. I guess the task is too short - so there is not enough time to run `psutil` and send message to hub.
I also am seeing this issue.  Don't know whether it is related or not, but I also see situations where SLURM (at NERSC) claims a job has been running for 5-10 minutes, but there is not yet any confirming information in the monitoring DB.
@benclifford Not sure what the ideal fix here is. How about adding mechanism to guarantee the monitoring wrapper sends out the first message?
I think making the code always send start and end messages would be a good thing 
@TomGlanzman Not quite sure this is related. I think this issue only happens to the tasks that are very short (like a no-op). If the DB does not receive that message, then there might be some other issues though. Could you please post the runinfo dir?
Make a new issue for the slurm related problem - that probably isn't this one.

Resolving patch:
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -280,17 +280,23 @@ def monitor_wrapper(f,
         Wrap the Parsl app with a function that will call the monitor function and point it at the correct pid when the task begins.
         """
         def wrapped(*args, **kwargs):
+            command_q = Queue(maxsize=10)
             p = Process(target=monitor,
                         args=(os.getpid(),
                               task_id,
                               monitoring_hub_url,
                               run_id,
+                              command_q,
                               logging_level,
                               sleep_dur),
                         name="Monitor-Wrapper-{}".format(task_id))
             p.start()
             try:
-                return f(*args, **kwargs)
+                try:
+                    return f(*args, **kwargs)
+                finally:
+                    command_q.put("Finished")
+                    p.join()
             finally:
                 # There's a chance of zombification if the workers are killed by some signals
                 p.terminate()
@@ -437,6 +443,7 @@ def monitor(pid,
             task_id,
             monitoring_hub_url,
             run_id,
+            command_q,
             logging_level=logging.INFO,
             sleep_dur=10):
     """Internal
@@ -447,6 +454,7 @@ def monitor(pid,
 
     import logging
     import time
+    from multiprocessing import Queue
 
     format_string = "%(asctime)s.%(msecs)03d %(name)s:%(lineno)d [%(levelname)s]  %(message)s"
     logging.basicConfig(filename='{logbase}/monitor.{task_id}.{pid}.log'.format(
@@ -525,5 +533,13 @@ def monitor(pid,
         except Exception:
             logging.exception("Exception getting the resource usage. Not sending usage to Hub", exc_info=True)
 
+        try:
+            msg = command_q.get(block=False)
+            if msg == "Finished":
+                logging.info("Received task finished message. Ending the monitoring loop now.")
+                break
+        except Queue.empty:
+            logging.debug("Have not received any message.")
+
         logging.debug("sleeping")
         time.sleep(sleep_dur)


Test patch: