internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Parsl/parsl

Issue:
Provider base and cluster provider to be added to reference
**Is your feature request related to a problem? Please describe.**

In the last set of doc updates where we trimmed some of the developer docs we've moved to relying more on the references to point someone to. It looks like the [provider base](https://github.com/Parsl/parsl/blob/master/parsl/providers/provider_base.py) class and [cluster provider](https://github.com/Parsl/parsl/blob/master/parsl/providers/cluster_provider.py) are missing from there. 

**Describe the solution you'd like**
Update docs to add these to the reference.

Provider base and cluster provider to be added to reference
**Is your feature request related to a problem? Please describe.**

In the last set of doc updates where we trimmed some of the developer docs we've moved to relying more on the references to point someone to. It looks like the [provider base](https://github.com/Parsl/parsl/blob/master/parsl/providers/provider_base.py) class and [cluster provider](https://github.com/Parsl/parsl/blob/master/parsl/providers/cluster_provider.py) are missing from there. 

**Describe the solution you'd like**
Update docs to add these to the reference.



Discussion:

Resolving patch:
diff --git a/parsl/providers/cluster_provider.py b/parsl/providers/cluster_provider.py
--- a/parsl/providers/cluster_provider.py
+++ b/parsl/providers/cluster_provider.py
@@ -14,18 +14,18 @@ class ClusterProvider(ExecutionProvider):
 
     Parameters
     ----------
-    label : str
+      label : str
         Label for this provider.
-    channel : Channel
+      channel : Channel
         Channel for accessing this provider. Possible channels include
         :class:`~parsl.channels.LocalChannel` (the default),
         :class:`~parsl.channels.SSHChannel`, or
         :class:`~parsl.channels.SSHInteractiveLoginChannel`.
-    walltime : str
+      walltime : str
         Walltime requested per block in HH:MM:SS.
-    launcher : str
+      launcher : str
         FIXME
-    cmd_timeout : int
+      cmd_timeout : int
         Timeout for commands made to the scheduler in seconds
 
     .. code:: python


Test patch:
Repository: Parsl/parsl

Issue:
garbage collection vs usage tracking shutdown race condition
**Describe the bug**
I've seen this exception at least once in CI local tests. I think it comes from job completion garbage collection racing with DFK shutdown usage reporting, in examining the task dictionary.

```
parsl/tests/test_staging/test_elaborate_noop_file.py:60: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
parsl/dataflow/dflow.py:1051: in cleanup
    self.usage_tracker.send_message()
parsl/dataflow/usage_tracking/usage.py:224: in send_message
    message = self.construct_end_message()
parsl/dataflow/usage_tracking/usage.py:182: in construct_end_message
    app_fails = len([t for t in self.dfk.tasks if
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.0 = <dict_keyiterator object at 0x7f86107ca040>
>   app_fails = len([t for t in self.dfk.tasks if
                     self.dfk.tasks[t]['status'] in FINAL_FAILURE_STATES])
E   RuntimeError: dictionary changed size during iteration

```


**To Reproduce**
non-deterministic in CI

**Environment**
CI, near master 02d3b93ad4fd97b2411c000cf0b8820c76e5dfef



Discussion:
From a quick look at that code, I think task garbage collection should be removing tasks in final failure states anyway, so that the code block raising the exception, which counts failed tasks, would always count 0 tasks.

I haven't verified if that is the case.

If it is, either failed task count should be dropped from usage tracking, or this code should be re-written to count without the `dfk.tasks` table.

I just did a quick check with latest, and failed tasks are still getting reported in the cleanup phase:

```
2021-03-02 14:56:57.029 parsl.dataflow.dflow:1030 [INFO]  DFK cleanup initiated
2021-03-02 14:56:57.029 parsl.dataflow.dflow:920 [INFO]  Summary of tasks in DFK:
2021-03-02 14:56:57.029 parsl.dataflow.dflow:934 [INFO]  Tasks in state States.failed: 10
2021-03-02 14:56:57.029 parsl.dataflow.dflow:940 [INFO]  End of summary
```
That value comes from:

>         keytasks[States.failed] = self.tasks_failed_count

not from the self.tasks table, though, I think.

(and if that is the case, this should be an easy fix)

Resolving patch:
diff --git a/parsl/dataflow/usage_tracking/usage.py b/parsl/dataflow/usage_tracking/usage.py
--- a/parsl/dataflow/usage_tracking/usage.py
+++ b/parsl/dataflow/usage_tracking/usage.py
@@ -10,7 +10,6 @@
 import platform
 import multiprocessing as mp
 
-from parsl.dataflow.states import FINAL_FAILURE_STATES
 from parsl.version import VERSION as PARSL_VERSION
 
 logger = logging.getLogger(__name__)
@@ -179,8 +178,7 @@ def construct_end_message(self):
 
         site_count = len([x for x in self.dfk.config.executors if x.managed])
 
-        app_fails = len([t for t in self.dfk.tasks if
-                         self.dfk.tasks[t]['status'] in FINAL_FAILURE_STATES])
+        app_fails = self.dfk.tasks_failed_count + self.dfk.tasks_dep_fail_count
 
         message = {'uuid': self.uuid,
                    'end': time.time(),


Test patch:
Repository: Parsl/parsl

Issue:
Workqueue Executor should subclass RepresentationMixin
**Describe the bug**
Most classes used in parsl configuration subclass RepresentationMixin so that they print out as a string similar to the form they are constructed with. Workqueue executor doesn't do this, so it prints out like this:

  executors=[<parsl.executors.workqueue.executor.WorkQueueExecutor object at 0x2aaae356b7c0>, ThreadPoolExecutor(
        label='submit-node', 
        managed=True, 
        max_threads=1, 
        storage_access=None, 
        thread_name_prefix='', 
        working_dir=None
    )], 

**Expected behavior**
WorkqueueExecutor objects should print out in config logs with all their parameters.

**Environment**
lsst branch



Discussion:

Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -74,7 +74,7 @@
 ParslFileToWq = namedtuple('ParslFileToWq', 'parsl_name stage cache')
 
 
-class WorkQueueExecutor(NoStatusHandlingExecutor):
+class WorkQueueExecutor(NoStatusHandlingExecutor, putils.RepresentationMixin):
     """Executor to use Work Queue batch system
 
     The WorkQueueExecutor system utilizes the Work Queue framework to
@@ -235,7 +235,7 @@ def __init__(self,
         self.use_cache = use_cache
         self.working_dir = working_dir
         self.registered_files = set()  # type: Set[str]
-        self.full = full_debug
+        self.full_debug = full_debug
         self.source = True if pack else source
         self.pack = pack
         self.extra_pkgs = extra_pkgs or []
@@ -285,7 +285,7 @@ def start(self):
                                  "launch_cmd": self.launch_cmd,
                                  "data_dir": self.function_data_dir,
                                  "collector_queue": self.collector_queue,
-                                 "full": self.full,
+                                 "full": self.full_debug,
                                  "shared_fs": self.shared_fs,
                                  "autolabel": self.autolabel,
                                  "autolabel_window": self.autolabel_window,


Test patch:
Repository: Parsl/parsl

Issue:
grid engine provider hangs when qstat output is large
**Describe the bug**

From Quentin Le Boulc'h;

```
I still have the issue with the qstat command from Parsl which is hitting the 60s timeout. When I have 1000 jobs running every qstat fails, I can see many processes accumulating and Parsl get completely stucked.
I checked I can reproduce the issue by myself running the subprocess command you are using. With several hundreds of jobs running the proc.wait() is waiting forever. However a manual qstat returns the output in less than a second. And after killing the proc.wait()  I can check that the stdout is actually here so the command has succeeded.
I can also reproduce the issue with a different command than `qstat`, like a simple script returning several hundreds of lines.
Could you check if there is some issue with your subprocess command?
See https://docs.python.org/3/library/subprocess.html#popen-objects for instance: “This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.”
```


**Expected behavior**
This should not hang

**Environment**
in2p3, parsl desc branch


Discussion:

Resolving patch:
diff --git a/parsl/channels/local/local.py b/parsl/channels/local/local.py
--- a/parsl/channels/local/local.py
+++ b/parsl/channels/local/local.py
@@ -68,9 +68,7 @@ def execute_wait(self, cmd, walltime=None, envs={}):
                 shell=True,
                 preexec_fn=os.setpgrp
             )
-            proc.wait(timeout=walltime)
-            stdout = proc.stdout.read()
-            stderr = proc.stderr.read()
+            (stdout, stderr) = proc.communicate(timeout=walltime)
             retcode = proc.returncode
 
         except Exception as e:


Test patch:
Repository: Parsl/parsl

Issue:
WorkQueue outstanding() race condition vs new task submission
**Describe the bug**
One of the users that I am supporting has encountered this:


2022-02-23 14:42:14 parsl.dataflow.flow_control:114 MainProcess(112424) FlowControl-Thread [ERROR]  Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/flow_control.py", line 112, in make_callback
    self.callback(tasks=self._event_buffer, kind=kind)
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/task_status_poller.py", line 104, in poll
    self._strategy.strategize(self._poll_items, tasks)
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/strategy.py", line 140, in _strategy_simple
    self._general_strategy(status_list, tasks, strategy_type='simple')
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/dataflow/strategy.py", line 174, in _general_strategy
    active_tasks = executor.outstanding
  File "REDACTED/wq_env/lib/python3.8/site-packages/parsl/executors/workqueue/executor.py", line 633, in outstanding
    for fut in self.tasks.values():
RuntimeError: dictionary changed size during iteration
```

This happens repeatedly during execution under high job submission load, and looks like a race condition on both flow control and job submission trying to access the work queue executor's `self.tasks` structure.

Later on, the run does not actually successfully execute any jobs and work queue workers time out after a while. I am unclear if this is due to this error or something else.

**To Reproduce**
High job submission load with WQ.

**Expected behavior**
No exceptions.

**Distributed Environment**
desc parsl branch


Discussion:

Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -623,9 +623,10 @@ def outstanding(self) -> int:
         implemented and probably could be replaced with a counter.
         """
         outstanding = 0
-        for fut in self.tasks.values():
-            if not fut.done():
-                outstanding += 1
+        with self.tasks_lock:
+            for fut in self.tasks.values():
+                if not fut.done():
+                    outstanding += 1
         logger.debug(f"Counted {outstanding} outstanding tasks")
         return outstanding
 


Test patch:
Repository: Parsl/parsl

Issue:
Remove parsl container bits
This issue is to remind us to remove Parsl container support and update the docs as soon as the funcX executor is integrated-- we should switch to recommending container support through it.


Discussion:
Is this the same as removing IPP container support in PR #1558 ?
I do not think this is related to the IPP container support question. The issue here is that providers do not support containers well enough and were designed for the case where the managers/pilots were launched in containers simply by being invoked via say `docker run ...`. We can launch bash app's in containers if the command requested says `<container_flavor> run <some_command>`, but if we want python apps to run inside containers, we'll have to launch managers/workers inside containers. This latter situation is not well supported.
Now that bad docs are removed, let's come back to this post 1.0

Resolving patch:
diff --git a/docker/app1/app1.py b/docker/app1/app1.py
deleted file mode 100644
--- a/docker/app1/app1.py
+++ /dev/null
@@ -1,4 +0,0 @@
-
-def predict(list_items):
-    """Returns the double of the items"""
-    return [i*2 for i in list_items]
diff --git a/docker/app2/app2.py b/docker/app2/app2.py
deleted file mode 100644
--- a/docker/app2/app2.py
+++ /dev/null
@@ -1,4 +0,0 @@
-
-def predict(list_items):
-    """Returns items+10"""
-    return [i+10 for i in list_items]


Test patch:
Repository: Parsl/parsl

Issue:
Monitoring visualization show negative CPU usage with retries
**Describe the bug**
Currently, the logged CPU time in monitoring db is cumulative. This works for most of the cases, except when a task is failed and retried. As a result, multiple figures in the visualization will show a sawtooth.

**To Reproduce**
Steps to reproduce the behavior, for e.g:
1. Setup Parsl master with Python 3.6 on cluster
2. Run all the tests and open the visualization
3. See error

**Expected behavior**
Each retried task should be on a new row.

**Possible solution**
Add a new column (e.g., `retries`) to the table as a new primary key.
Fix visualization based on that



Discussion:
PR #1808 switches resource utilisation to be keyed by try identity, not by task identity. That should address the database side of this issue, although probably not visualization of the information in parsl-visualize.

Resolving patch:
diff --git a/parsl/monitoring/visualization/plots/default/workflow_resource_plots.py b/parsl/monitoring/visualization/plots/default/workflow_resource_plots.py
--- a/parsl/monitoring/visualization/plots/default/workflow_resource_plots.py
+++ b/parsl/monitoring/visualization/plots/default/workflow_resource_plots.py
@@ -152,6 +152,7 @@ def resource_efficiency(resource, node, label='CPU'):
         end = resource['epoch_time'].max()
         resource['relative_time'] = resource['epoch_time'] - start
         node['relative_time'] = node['epoch_time'] - start
+        resource['key'] = resource['task_id'].astype(str) + "-" + resource['try_id'].astype(str)
 
         task_plot = [0] * (end - start + 1)
         if label == 'CPU':
@@ -160,8 +161,8 @@ def resource_efficiency(resource, node, label='CPU'):
             total = node['total_memory'].sum() / 1024 / 1024 / 1024
 
         resource['total_cpu_time'] = resource['psutil_process_time_user'] + resource['psutil_process_time_system']
-        for task_id in resource['task_id'].unique():
-            tmp = resource[resource['task_id'] == task_id]
+        for key in resource['key'].unique():
+            tmp = resource[resource['key'] == key]
             tmp['last_timestamp'] = tmp['relative_time'].shift(1)
             if label == 'CPU':
                 tmp['last_cputime'] = tmp['total_cpu_time'].shift(1)


Test patch:
Repository: Parsl/parsl

Issue:
task garbage collection doesn't remove failed tasks with checkpointing turned on
**Describe the bug**
Tasks that fail rather than complete successfully are not garbage collected if checkpointing is turned on.

@tomglanzman reports this behaviour, which I think is a bug - all tasks should be forgotten once parsl can make no further progress.




Discussion:

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -167,6 +167,7 @@ def __init__(self, config=Config()):
         self.checkpointed_tasks = 0
         self._checkpoint_timer = None
         self.checkpoint_mode = config.checkpoint_mode
+        self.checkpointable_tasks = []
 
         # the flow control keeps track of executors and provider task states;
         # must be set before executors are added since add_executors calls
@@ -434,20 +435,21 @@ def handle_app_update(self, task_record: TaskRecord, future: AppFuture) -> None:
         self.memoizer.update_memo(task_record, future)
 
         # Cover all checkpointing cases here:
-        # Do we need to checkpoint the task now,
-        # keep it around for one of the other checkpoint
-        # methods, or can we wipe it immediately?
+        # Do we need to checkpoint now, or queue for later,
+        # or do nothing?
         if self.checkpoint_mode == 'task_exit':
             self.checkpoint(tasks=[task_record])
         elif self.checkpoint_mode == 'manual' or \
                 self.checkpoint_mode == 'periodic' or \
                 self.checkpoint_mode == 'dfk_exit':
-            pass
+            with self.checkpoint_lock:
+                self.checkpointable_tasks.append(task_record)
         elif self.checkpoint_mode is None:
-            self.wipe_task(task_id)
+            pass
         else:
             raise RuntimeError(f"Invalid checkpoint mode {self.checkpoint_mode}")
 
+        self.wipe_task(task_id)
         return
 
     def _complete_task(self, task_record: TaskRecord, new_state: States, result: Any) -> None:
@@ -1159,11 +1161,11 @@ def checkpoint(self, tasks: Optional[Sequence[TaskRecord]] = None) -> str:
             run under RUNDIR/checkpoints/{tasks.pkl, dfk.pkl}
         """
         with self.checkpoint_lock:
-            checkpoint_queue = None
             if tasks:
                 checkpoint_queue = tasks
             else:
-                checkpoint_queue = list(self.tasks.values())
+                checkpoint_queue = self.checkpointable_tasks
+                self.checkpointable_tasks = []
 
             checkpoint_dir = '{0}/checkpoint'.format(self.run_dir)
             checkpoint_dfk = checkpoint_dir + '/dfk.pkl'
@@ -1191,7 +1193,6 @@ def checkpoint(self, tasks: Optional[Sequence[TaskRecord]] = None) -> str:
 
                     if app_fu.done() and app_fu.exception() is None:
                         hashsum = task_record['hashsum']
-                        self.wipe_task(task_id)
                         if not hashsum:
                             continue
                         t = {'hash': hashsum,


Test patch:
Repository: Parsl/parsl

Issue:
Towards type-checking dfk.__init__
On the benc-mypy branch, I ended up with dfk.__init__ being type-checked. This is important as a step towards the whole DFK class being type checked, which is important as being one of the core classes of parsl.

This requires some prerequisite untangling, which I have separated out into several patches, and which I will merge as separate PRs.

These are detailed below.

Some untangling comes from the vagueness of when/which executors should be scaled:
- [x] #2537 removes the `executor.managed` property, and with it the vague notion of managed vs unmanaged executors
- [x] #2545 executor.scaling_enabled is a further flag that indicates whether scaling should happen. In line with #2539 this should go away.
- [x] #2544 #2543 #2542 several explicit type annotations are needed to help `mypy` infer the full types of attributes initialised to `None` - this is basically a technical limitation of python static type checking.
- [x] #2551 the DFK code is vague about whether app argument lists are lists or tuples, and some code moves between both of those formats. There is already some code to treat app arguments as Sequences, which accomodates both, and some more tidyup needs to happen here.
- [x] #2541 checkpoint configuration for timers needs some tidying up to make it type check, and at the same time I will take the opportunity to make it raise a configuration error rather than fairly quietly continue if it cannot parse a duration, as a usability enhancement
- [ ] finally typeguard can be enabled on dfk.__init__ and type annotations can be added to dfk.__init__. This turns on typechecking at runtime for dfk.__init__, as well as propagating types of the various DFK attributes throughout parsl, whenever an already type-checked function makes reference to a DFK.



Discussion:

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -12,6 +12,7 @@
 import sys
 import datetime
 from getpass import getuser
+from typeguard import typechecked
 from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
 from uuid import uuid4
 from socket import gethostname
@@ -64,7 +65,8 @@ class DataFlowKernel(object):
 
     """
 
-    def __init__(self, config=Config()):
+    @typechecked
+    def __init__(self, config: Config = Config()) -> None:
         """Initialize the DataFlowKernel.
 
         Parameters
@@ -77,10 +79,6 @@ def __init__(self, config=Config()):
         # this will be used to check cleanup only happens once
         self.cleanup_called = False
 
-        if isinstance(config, dict):
-            raise ConfigurationError(
-                    'Expected `Config` class, received dictionary. For help, '
-                    'see http://parsl.readthedocs.io/en/stable/stubs/parsl.config.Config.html')
         self._config = config
         self.run_dir = make_rundir(config.run_dir)
 


Test patch:
Repository: Parsl/parsl

Issue:
Parsl + Work Queue coprocess mode does not work on non-shared fs
**Describe the bug**
When running with WorkQueueExecutor(coprocess=True), tasks fail with this error:

```
parsl.executors.workqueue.errors.WorkQueueTaskFailure: ('work queue result: The result file was not transfered from the worker.\nThis usually means that there is a problem with the python setup,\nor the wrapper that executes the function.\nTrace:\n{"Result": "[Errno 2] No such file or directory: \'/tmp/bxc/runinfo/004/WorkQueueExecutor/function_data/0009/result\'", "StatusCode": 500}', FileNotFoundError(2, 'No such file or directory'))
```

It looks like staging of at least the result file (and possibly staging to workers of function definitions) isn't happening here. (This staging happens with non-coprocess mode, always)

**To Reproduce**
Move to a non-shared directory (for example /tmp on an HPC batch system) and try to run some tasks on worker nodes.

**Expected behavior**
This should work
**Environment**
parsl master 0f5bf133097f0b2985410b4809b1292ed1231c83, 

```
work_queue_worker --version
work_queue_worker version 7.5.5 FINAL (released 2023-05-12 12:54:33 +0000)
	Built by conda@7cbc6a798682 on 2023-05-12 12:54:33 +0000
```

running submit side on a perlmutter login node, workers running in slurm



Discussion:

Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -860,7 +860,11 @@ def _work_queue_submit_wait(*,
                     logger.debug("Sending executor task {} with command: {}".format(task.id, command_str))
                     t = wq.Task(command_str)
                 else:
-                    t = wq.RemoteTask("run_parsl_task", "parsl_coprocess", task.map_file, task.function_file, task.result_file)
+                    t = wq.RemoteTask("run_parsl_task",
+                                      "parsl_coprocess",
+                                      os.path.basename(task.map_file),
+                                      os.path.basename(task.function_file),
+                                      os.path.basename(task.result_file))
                     t.specify_exec_method("direct")
                     logger.debug("Sending executor task {} to coprocess".format(task.id))
 


Test patch: