Repository: Parsl/parsl

Issue:
investigate task-end-to-task-end delays in Work Queue
**Describe the bug**
Performance is poor in the work queue executor when running sequential tasks: the time from one task ending to the next task starting is around 1 second. I'm not sure if this is inside the parsl wq executor code, or inside core wq.

**To Reproduce**
submit two tasks, one which depends on the other. observe a 1 second delay between task 1 completing on the worker and task 2 starting on the worker.

**Expected behavior**
This could be much faster.

**Environment**
my laptop

cc @khk-globus 


Discussion:
looks like most of the time in the pytest test suite is being spent in this line: `t = q.wait(1)` waiting for a result (that will never come, because the only result we're expecting has already been processed by an earlier iteration) for one second before submitting the next task.

Things I tried:

* Make 1 smaller - that parameter is an int.
and that parameter can only go down to 1 (second) by the looks of it - setting the timeout to 0 seems to make the call wait up to 1 second. So I can't try a smaller value here.

* Look at q.stats.tasks_with_results - this seems to be 0 always, so I can't poll for task readiness faster. I haven't investigated further what is going on there.

cc @dthain 
link to that line:

https://github.com/Parsl/parsl/blob/99701e819f7a85a40194b201cb6a4baf06e91d84/parsl/executors/workqueue/executor.py#L965
In htex I've found this "poll one thing for 1s, poll another thing for 1s" behaviour awkward to deal with in one thread. I'm not sure with WQ the two different activities in that thread (submitting a task, and processing results) could happen in two different threads - are there issues accessing work queue from multiple threads?
@colinthomas-z80 please take a look at this.  Yes, it appears to be a classic example of the dual blocking wait problem.

I can reproduce the issue with the wq executor. I put together a similar test with plain work queue and didn't observe the issue. I will do some digging into the executor retrieval loop. 

Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -959,43 +959,42 @@ def _work_queue_submit_wait(*,
 
         # If the queue is not empty wait on the WorkQueue queue for a task
         task_found = True
-        if not q.empty():
-            while task_found and not should_stop.value:
-                # Obtain the task from the queue
-                t = q.wait(1)
-                if t is None:
-                    task_found = False
-                    continue
-                # When a task is found:
-                executor_task_id = t.tag
-                logger.debug("Completed Work Queue task {}, executor task {}".format(t.id, t.tag))
-                result_file = result_file_of_task_id.pop(t.tag)
-
-                # A tasks completes 'succesfully' if it has result file.
-                # The check whether this file can load a serialized Python object
-                # happens later in the collector thread of the executor process.
-                logger.debug("Looking for result in {}".format(result_file))
-                if os.path.exists(result_file):
-                    logger.debug("Found result in {}".format(result_file))
-                    collector_queue.put_nowait(WqTaskToParsl(id=executor_task_id,
-                                                             result_received=True,
-                                                             result_file=result_file,
-                                                             reason=None,
-                                                             status=t.return_status))
-                # If a result file could not be generated, explain the
-                # failure according to work queue error codes.
-                else:
-                    reason = _explain_work_queue_result(t)
-                    logger.debug("Did not find result in {}".format(result_file))
-                    logger.debug("Wrapper Script status: {}\nWorkQueue Status: {}"
-                                 .format(t.return_status, t.result))
-                    logger.debug("Task with executor id {} / Work Queue id {} failed because:\n{}"
-                                 .format(executor_task_id, t.id, reason))
-                    collector_queue.put_nowait(WqTaskToParsl(id=executor_task_id,
-                                                             result_received=False,
-                                                             result_file=None,
-                                                             reason=reason,
-                                                             status=t.return_status))
+        while not q.empty() and task_found and not should_stop.value:
+            # Obtain the task from the queue
+            t = q.wait(1)
+            if t is None:
+                task_found = False
+                continue
+            # When a task is found:
+            executor_task_id = t.tag
+            logger.debug("Completed Work Queue task {}, executor task {}".format(t.id, t.tag))
+            result_file = result_file_of_task_id.pop(t.tag)
+
+            # A tasks completes 'succesfully' if it has result file.
+            # The check whether this file can load a serialized Python object
+            # happens later in the collector thread of the executor process.
+            logger.debug("Looking for result in {}".format(result_file))
+            if os.path.exists(result_file):
+                logger.debug("Found result in {}".format(result_file))
+                collector_queue.put_nowait(WqTaskToParsl(id=executor_task_id,
+                                                         result_received=True,
+                                                         result_file=result_file,
+                                                         reason=None,
+                                                         status=t.return_status))
+            # If a result file could not be generated, explain the
+            # failure according to work queue error codes.
+            else:
+                reason = _explain_work_queue_result(t)
+                logger.debug("Did not find result in {}".format(result_file))
+                logger.debug("Wrapper Script status: {}\nWorkQueue Status: {}"
+                             .format(t.return_status, t.result))
+                logger.debug("Task with executor id {} / Work Queue id {} failed because:\n{}"
+                             .format(executor_task_id, t.id, reason))
+                collector_queue.put_nowait(WqTaskToParsl(id=executor_task_id,
+                                                         result_received=False,
+                                                         result_file=None,
+                                                         reason=reason,
+                                                         status=t.return_status))
     logger.debug("Exiting WorkQueue Monitoring Process")
     return 0
 


Test patch: