anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: parsl/parsl
Issue: Duplicate log lines when using IPP

The following code fragment emits log lines to STDERR with the default format style even though no logging is requested. I suspect that `ipyparallel` is doing something dirty.

    from parsl import DataFlowKernel
    from parsl.configs.local import localIPP as config

    dfk = DataFlowKernel(config=config)
    dfk.cleanup()

The above code with the minor change of using threads will not emit the log lines.


    from parsl import DataFlowKernel
    from parsl.configs.local import localThreads as config

    dfk = DataFlowKernel(config=config)
    dfk.cleanup()


Please help test by running this with the latest parsl code. 

Changed files:
parsl/dataflow/strategy.py

diff --git a/parsl/dataflow/strategy.py b/parsl/dataflow/strategy.py
index c9910eb6..47b77a71 100644
--- a/parsl/dataflow/strategy.py
+++ b/parsl/dataflow/strategy.py
@@ -116,6 +116,8 @@ class Strategy(object):
         self.strategies = {None: self._strategy_noop, 'simple': self._strategy_simple}
 
         self.strategize = self.strategies[self.config.strategy]
+        self.logger_flag = False
+        self.prior_loghandlers = set(logging.getLogger().handlers)
 
         logger.debug("Scaling strategy: {0}".format(self.config.strategy))
 
@@ -129,6 +131,20 @@ class Strategy(object):
             - kind (Not used)
         """
 
+    def unset_logging(self):
+        """ Mute newly added handlers to the root level, right after calling executor.status
+        """
+        if self.logger_flag is True:
+            return
+
+        root_logger = logging.getLogger()
+
+        for hndlr in root_logger.handlers:
+            if hndlr not in self.prior_loghandlers:
+                hndlr.setLevel(logging.ERROR)
+
+        self.logger_flag = True
+
     def _strategy_simple(self, tasks, *args, kind=None, **kwargs):
         """Peek at the DFK and the executors specified.
 
@@ -143,9 +159,6 @@ class Strategy(object):
         KWargs:
             - kind (Not used)
         """
-        # Add logic to check executors
-        # for task in tasks :
-        #    if self.dfk.tasks[task]:
 
         for label, executor in self.dfk.executors.items():
             if not executor.scaling_enabled:
@@ -155,6 +168,7 @@ class Strategy(object):
             active_tasks = executor.executor.outstanding
 
             status = executor.status()
+            self.unset_logging()
 
             # FIXME we need to handle case where provider does not define these
             # FIXME probably more of this logic should be moved to the provider
Repository: parsl/parsl
Issue: WorkQueue outstanding() race condition vs new task submission

**Describe the bug**
One of the users that I am supporting has encountered this:


2022-02-23 14:42:14 parsl.dataflow.flow_control:114 MainProcess(112424) FlowControl-Thread [ERROR]  Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/flow_control.py", line 112, in make_callback
    self.callback(tasks=self._event_buffer, kind=kind)
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/task_status_poller.py", line 104, in poll
    self._strategy.strategize(self._poll_items, tasks)
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/strategy.py", line 140, in _strategy_simple
    self._general_strategy(status_list, tasks, strategy_type='simple')
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/strategy.py", line 174, in _general_strategy
    active_tasks = executor.outstanding
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/executors/workqueue/executor.py", line 633, in outstanding
    for fut in self.tasks.values():
RuntimeError: dictionary changed size during iteration
```

This happens repeatedly during execution under high job submission load, and looks like a race condition on both flow control and job submission trying to access the work queue executor's `self.tasks` structure.

Later on, the run does not actually successfully execute any jobs and work queue workers time out after a while. I am unclear if this is due to this error or something else.

**To Reproduce**
High job submission load with WQ.

**Expected behavior**
No exceptions.

**Distributed Environment**
desc parsl branch

Changed files:
parsl/executors/workqueue/executor.py

diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
index 79c1ff4d..78646c0c 100644
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -623,9 +623,10 @@ class WorkQueueExecutor(BlockProviderExecutor, putils.RepresentationMixin):
         implemented and probably could be replaced with a counter.
         """
         outstanding = 0
-        for fut in self.tasks.values():
-            if not fut.done():
-                outstanding += 1
+        with self.tasks_lock:
+            for fut in self.tasks.values():
+                if not fut.done():
+                    outstanding += 1
         logger.debug(f"Counted {outstanding} outstanding tasks")
         return outstanding
Repository: parsl/parsl
Issue: Workqueue Executor should subclass RepresentationMixin

**Describe the bug**
Most classes used in parsl configuration subclass RepresentationMixin so that they print out as a string similar to the form they are constructed with. Workqueue executor doesn't do this, so it prints out like this:

  executors=[<parsl.executors.workqueue.executor.WorkQueueExecutor object at 0x2aaae356b7c0>, ThreadPoolExecutor(
        label='submit-node', 
        managed=True, 
        max_threads=1, 
        storage_access=None, 
        thread_name_prefix='', 
        working_dir=None
    )], 

**Expected behavior**
WorkqueueExecutor objects should print out in config logs with all their parameters.

**Environment**
lsst branch


Changed files:
parsl/executors/workqueue/executor.py

diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
index b85b3faa..d9149d2f 100644
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -74,7 +74,7 @@ WqTaskToParsl = namedtuple('WqTaskToParsl', 'id result_received result reason st
 ParslFileToWq = namedtuple('ParslFileToWq', 'parsl_name stage cache')
 
 
-class WorkQueueExecutor(NoStatusHandlingExecutor):
+class WorkQueueExecutor(NoStatusHandlingExecutor, putils.RepresentationMixin):
     """Executor to use Work Queue batch system
 
     The WorkQueueExecutor system utilizes the Work Queue framework to
@@ -235,7 +235,7 @@ class WorkQueueExecutor(NoStatusHandlingExecutor):
         self.use_cache = use_cache
         self.working_dir = working_dir
         self.registered_files = set()  # type: Set[str]
-        self.full = full_debug
+        self.full_debug = full_debug
         self.source = True if pack else source
         self.pack = pack
         self.extra_pkgs = extra_pkgs or []
@@ -285,7 +285,7 @@ class WorkQueueExecutor(NoStatusHandlingExecutor):
                                  "launch_cmd": self.launch_cmd,
                                  "data_dir": self.function_data_dir,
                                  "collector_queue": self.collector_queue,
-                                 "full": self.full,
+                                 "full": self.full_debug,
                                  "shared_fs": self.shared_fs,
                                  "autolabel": self.autolabel,
                                  "autolabel_window": self.autolabel_window,
Repository: parsl/parsl
Issue: Parsl internal bugs silently squashed then hangs

@benclifford reported an alarming bug in which a error in the parsl code does not throw an exception but is silently squashed. This is followed by the code hanging and not making progress. To make the situation even worse the strategy thread continues to run, giving the impression that parsl is doing something. 

My suspicion is the the exception silently kills some thread, and that goes unreported.

The string formatting bug added here (in memoization code) -> https://github.com/Parsl/parsl/tree/pr238-related-example reproduces this issue with a test script that hits this memoization code. 

Some recommendations:
1. Better exception handling
2. Logs to show thread-ids so that we can see when threads have gone missing

Changed files:
parsl/__init__.py

diff --git a/parsl/__init__.py b/parsl/__init__.py
index 475173df..b6f94431 100644
--- a/parsl/__init__.py
+++ b/parsl/__init__.py
@@ -71,6 +71,12 @@ def set_stream_logger(name='parsl', level=logging.DEBUG, format_string=None):
     handler.setFormatter(formatter)
     logger.addHandler(handler)
 
+    # Concurrent.futures errors are also of interest, as exceptions
+    # which propagate out of the top of a callback are logged this way
+    # and then discarded. (see #240)
+    futures_logger = logging.getLogger("concurrent.futures")
+    futures_logger.addHandler(handler)
+
 
 def set_file_logger(filename, name='parsl', level=logging.DEBUG, format_string=None):
     """Add a stream log handler.
@@ -95,6 +101,11 @@ def set_file_logger(filename, name='parsl', level=logging.DEBUG, format_string=N
     handler.setFormatter(formatter)
     logger.addHandler(handler)
 
+    # see note in set_stream_logger for notes about logging
+    # concurrent.futures
+    futures_logger = logging.getLogger("concurrent.futures")
+    futures_logger.addHandler(handler)
+
 
 class NullHandler(logging.Handler):
     """Setup default logging to /dev/null since this is library."""