internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Parsl/parsl

Issue:
Support monitoring with multiple managers / node
In many HTC workflows, it is often useful to run in non-exclusive mode (you will be provisioned resources more quickly if you're willing to share the node with other jobs). In this case, you may have a node with multiple managers on it. Monitoring does not support this because it expects the hostname, run_id, and reg_time to be unique.
```
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: node.hostname, node.run_id, node.reg_time
[SQL: INSERT INTO node (run_id, hostname, cpu_count, total_memory, active, worker_count, python_v, reg_time) VALUES (?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: (('1932d003-db9c-4280-a6ec-e514036f5cc1', 'd12chas544.crc.nd.edu', 12, 134291927040, 1, 24, '3.6.5', '2019-09-12 12:56:43.000000'), ('1932d003-db9c-4280-a6ec-e514036f5cc1', 'd12chas544.crc.nd.edu', 12, 134291927040, 1, 24, '3.6.5', '2019-09-12 12:56:43.000000'), ('1932d003-db9c-4280-a6ec-e514036f5cc1', 'd12chas544.crc.nd.edu', 12, 134291927040, 1, 24, '3.6.5', '2019-09-12 12:56:43.000000'))]
```


Discussion:
Note this is a requirement for the parsl coffea backend. cc @lgray 
cc @klannon

Resolving patch:
diff --git a/parsl/monitoring/db_manager.py b/parsl/monitoring/db_manager.py
--- a/parsl/monitoring/db_manager.py
+++ b/parsl/monitoring/db_manager.py
@@ -143,6 +143,7 @@ class Task(Base):
 
     class Node(Base):
         __tablename__ = NODE
+        id = Column('id', Integer, nullable=False, primary_key=True, autoincrement=True)
         run_id = Column('run_id', Text, nullable=False)
         hostname = Column('hostname', Text, nullable=False)
         cpu_count = Column('cpu_count', Integer, nullable=False)
@@ -151,9 +152,6 @@ class Node(Base):
         worker_count = Column('worker_count', Integer, nullable=False)
         python_v = Column('python_v', Text, nullable=False)
         reg_time = Column('reg_time', DateTime, nullable=False)
-        __table_args__ = (
-            PrimaryKeyConstraint('hostname', 'run_id', 'reg_time'),
-        )
 
     class Resource(Base):
         __tablename__ = RESOURCE
diff --git a/parsl/monitoring/visualization/models.py b/parsl/monitoring/visualization/models.py
--- a/parsl/monitoring/visualization/models.py
+++ b/parsl/monitoring/visualization/models.py
@@ -27,6 +27,7 @@ class Workflow(db.Model):
 
 class Node(db.Model):
     __tablename__ = NODE
+    id = db.Column('id', db.Integer, nullable=False, primary_key=True, autoincrement=True)
     run_id = db.Column('run_id', db.Text, nullable=False)
     hostname = db.Column('hostname', db.Text, nullable=False)
     cpu_count = db.Column('cpu_count', db.Integer, nullable=False)
@@ -35,9 +36,6 @@ class Node(db.Model):
     worker_count = db.Column('worker_count', db.Integer, nullable=False)
     python_v = db.Column('python_v', db.Text, nullable=False)
     reg_time = db.Column('reg_time', db.DateTime, nullable=False)
-    __table_args__ = (
-        db.PrimaryKeyConstraint('hostname', 'run_id', 'reg_time'),
-    )
 
 
 # TODO: expand to full set of info


Test patch:
Repository: Parsl/parsl

Issue:
Configurable max_idletime on simple strategy
currently it is hard coded to 120s. It would be great to make this parameter configurable.


Discussion:

Resolving patch:
diff --git a/parsl/config.py b/parsl/config.py
--- a/parsl/config.py
+++ b/parsl/config.py
@@ -47,6 +47,8 @@ class Config(RepresentationMixin):
     strategy : str, optional
         Strategy to use for scaling resources according to workflow needs. Can be 'simple' or `None`. If `None`, dynamic
         scaling will be disabled. Default is 'simple'.
+    max_idletime : float, optional
+        The maximum idle time for an executor in the 'simple' strategy. Default is 120.0 seconds.
     usage_tracking : bool, optional
         Set this field to True to opt-in to Parsl's usage tracking system. Parsl only collects minimal, non personally-identifiable,
         information used for reporting to our funding agencies. Default is False.
@@ -64,6 +66,7 @@ def __init__(self,
                  retries: int = 0,
                  run_dir: str = 'runinfo',
                  strategy: Optional[str] = 'simple',
+                 max_idletime: float = 120.0,
                  monitoring: Optional[MonitoringHub] = None,
                  usage_tracking: bool = False):
         if executors is None:
@@ -89,6 +92,7 @@ def __init__(self,
         self.retries = retries
         self.run_dir = run_dir
         self.strategy = strategy
+        self.max_idletime = max_idletime
         self.usage_tracking = usage_tracking
         self.monitoring = monitoring
 
diff --git a/parsl/dataflow/strategy.py b/parsl/dataflow/strategy.py
--- a/parsl/dataflow/strategy.py
+++ b/parsl/dataflow/strategy.py
@@ -111,7 +111,7 @@ def __init__(self, dfk):
         self.dfk = dfk
         self.config = dfk.config
         self.executors = {}
-        self.max_idletime = 60 * 2  # 2 minutes
+        self.max_idletime = self.dfk.config.max_idletime
 
         for e in self.dfk.config.executors:
             self.executors[e.label] = {'idle_since': None, 'config': e.label}


Test patch:
Repository: Parsl/parsl

Issue:
Fix IndexError when using CondorProvider with strategy enabled
```
2019-09-20 15:32:16.535 parsl.dataflow.flow_control:142 [ERROR]  Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
  File "/afs/crc.nd.edu/user/a/awoodard/.local/lib/python3.6/site-packages/parsl/dataflow/flow_control.py", line 140, in make_callback
    self.callback(tasks=self._event_buffer, kind=kind)
  File "/afs/crc.nd.edu/user/a/awoodard/.local/lib/python3.6/site-packages/parsl/dataflow/strategy.py", line 177, in _strategy_simple
    status = executor.status()
  File "/afs/crc.nd.edu/user/a/awoodard/.local/lib/python3.6/site-packages/parsl/executors/high_throughput/executor.py", line 620, in status
    status = self.provider.status(list(self.blocks.values()))
  File "/afs/crc.nd.edu/user/a/awoodard/.local/lib/python3.6/site-packages/parsl/providers/condor/condor.py", line 160, in status
    self._status()
  File "/afs/crc.nd.edu/user/a/awoodard/.local/lib/python3.6/site-packages/parsl/providers/condor/condor.py", line 142, in _status
    job_id = parts[0]
IndexError: list index out of range
```


Discussion:
Can you paste in the output of
`condor_q {0} -af:jr JobStatus`
where {0} is replaced by a few job IDs?

Resolving patch:
diff --git a/parsl/providers/condor/condor.py b/parsl/providers/condor/condor.py
--- a/parsl/providers/condor/condor.py
+++ b/parsl/providers/condor/condor.py
@@ -143,8 +143,8 @@ def _status(self):
         34524643.0 1
         """
 
-        for line in stdout.strip().split('\n'):
-            parts = line.split()
+        for line in stdout.splitlines():
+            parts = line.strip().split()
             job_id = parts[0]
             status = translate_table.get(parts[1], 'UNKNOWN')
             self.resources[job_id]['status'] = status


Test patch:
Repository: Parsl/parsl

Issue:
RepresentationMixin breaks on classes with no default parameters
```
from parsl.utils import RepresentationMixin

class A(RepresentationMixin):

   def __init__(self, q):
     self.q = q

x = A(q=4)
print(x)
```

gives:

```
$ python b.py 
Traceback (most recent call last):
  File "b.py", line 10, in <module>
    print(x)
  File "/home/benc/parsl/src/parsl/parsl/utils.py", line 193, in __repr__
    defaults = dict(zip(reversed(argspec.args), reversed(argspec.defaults)))
TypeError: 'NoneType' object is not reversible
```

Changing `__init__` to:

```
   def __init__(self, q=3):
```

fixes this.

At a guess, argspec.defaults is None rather than an empty sequence in the breaking case.


Discussion:

Resolving patch:
diff --git a/parsl/utils.py b/parsl/utils.py
--- a/parsl/utils.py
+++ b/parsl/utils.py
@@ -189,7 +189,7 @@ def __repr__(self):
             init = init.__wrapped__
 
         argspec = inspect.getfullargspec(init)
-        if len(argspec.args) > 1:
+        if len(argspec.args) > 1 and argspec.defaults is not None:
             defaults = dict(zip(reversed(argspec.args), reversed(argspec.defaults)))
         else:
             defaults = {}
@@ -199,7 +199,10 @@ def __repr__(self):
                 template = 'class {} uses {} in the constructor, but does not define it as an attribute'
                 raise AttributeError(template.format(self.__class__.__name__, arg))
 
-        args = [getattr(self, a) for a in argspec.args[1:-len(defaults)]]
+        if len(defaults) != 0:
+            args = [getattr(self, a) for a in argspec.args[1:-len(defaults)]]
+        else:
+            args = [getattr(self, a) for a in argspec.args[1:]]
         kwargs = {key: getattr(self, key) for key in defaults}
 
         def assemble_multiline(args, kwargs):


Test patch:
Repository: Parsl/parsl

Issue:
Incorrect pgid identification in LocalProvider kills too many processes
This code in LocalProvider probably should have a space added to give `-o pgid= {}`. This should be two parameters, one '-o pgid=` meaning output only pgids, and give that single column an empty heading; and secondly the process ID to inspect.

```
                cmd = "kill -- -$(ps -o pgid={} | grep -o '[0-9]*')".format(self.resources[job]['remote_pid'])
```

The code as it is probably doesn't shut down the expected process - it might sometimes; it might kill other processes or process groups, as the resulting ps output contains a list of many PGIDs, some of which might be valid PIDs to kill; or it might not kill anything. This might be related to bad shutdown behaviour described in #674 

I have not tested this but I encountered similar behaviour reviewing  #1297 


Discussion:
I've encountered this in production - when used with an SSHChannel to `localhost`, it kills all processes owned by my user (including my entire desktop environment) when a workflow shuts down.

More generally, this means LocalProvider + SSHChannel is unusably destructive when used against a host where a user is expecting any other processes to be running.

Resolving patch:
diff --git a/parsl/providers/local/local.py b/parsl/providers/local/local.py
--- a/parsl/providers/local/local.py
+++ b/parsl/providers/local/local.py
@@ -238,7 +238,7 @@ def cancel(self, job_ids):
                 self.resources[job]['status'] = 'CANCELLED'
 
             elif self.resources[job]['remote_pid']:
-                cmd = "kill -- -$(ps -o pgid={} | grep -o '[0-9]*')".format(self.resources[job]['remote_pid'])
+                cmd = "kill -- -$(ps -o pgid= {} | grep -o '[0-9]*')".format(self.resources[job]['remote_pid'])
                 retcode, stdout, stderr = self.channel.execute_wait(cmd, self.cmd_timeout)
                 if retcode != 0:
                     logger.warning("Failed to kill PID: {} and child processes on {}".format(self.resources[job]['remote_pid'],


Test patch:
Repository: Parsl/parsl

Issue:
slurm mandatory `partition` and non-mandatory `qos` are not always mandatory and non-mandatory. 
See Parsl/libsubmit#66


Discussion:
@tomglanzman has encountered this now (see comment on the corresponding libsubmit issue)

Resolving patch:
diff --git a/parsl/providers/slurm/slurm.py b/parsl/providers/slurm/slurm.py
--- a/parsl/providers/slurm/slurm.py
+++ b/parsl/providers/slurm/slurm.py
@@ -42,7 +42,7 @@ class SlurmProvider(ClusterProvider, RepresentationMixin):
     Parameters
     ----------
     partition : str
-        Slurm partition to request blocks from.
+        Slurm partition to request blocks from. If none, no partition slurm directive will be specified.
     channel : Channel
         Channel for accessing this provider. Possible channels include
         :class:`~parsl.channels.LocalChannel` (the default),
@@ -82,7 +82,7 @@ class SlurmProvider(ClusterProvider, RepresentationMixin):
 
     @typeguard.typechecked
     def __init__(self,
-                 partition: str,
+                 partition: Optional[str],
                  channel: Channel = LocalChannel(),
                  nodes_per_block: int = 1,
                  cores_per_node: Optional[int] = None,
@@ -118,6 +118,8 @@ def __init__(self,
         self.scheduler_options = scheduler_options + '\n'
         if exclusive:
             self.scheduler_options += "#SBATCH --exclusive\n"
+        if partition:
+            self.scheduler_options += "#SBATCH --partition={}\n".format(partition)
         self.worker_init = worker_init + '\n'
 
     def _status(self):
@@ -197,7 +199,6 @@ def submit(self, command, tasks_per_node, job_name="parsl.slurm"):
         job_config["walltime"] = wtime_to_minutes(self.walltime)
         job_config["scheduler_options"] = scheduler_options
         job_config["worker_init"] = worker_init
-        job_config["partition"] = self.partition
         job_config["user_script"] = command
 
         # Wrap the command
diff --git a/parsl/providers/slurm/template.py b/parsl/providers/slurm/template.py
--- a/parsl/providers/slurm/template.py
+++ b/parsl/providers/slurm/template.py
@@ -4,7 +4,6 @@
 #SBATCH --output=${submit_script_dir}/${jobname}.submit.stdout
 #SBATCH --error=${submit_script_dir}/${jobname}.submit.stderr
 #SBATCH --nodes=${nodes}
-#SBATCH --partition=${partition}
 #SBATCH --time=${walltime}
 #SBATCH --ntasks-per-node=${tasks_per_node}
 ${scheduler_options}


Test patch:
Repository: Parsl/parsl

Issue:
GridEngine provider mis-parses wall times, treating minutes as seconds
**Describe the bug**
The GridEngine system expects walltime in seconds, but parsl provides `10`, which leads to difficult-to-recognize errors relating to walltime errors.

**To Reproduce**
Steps to reproduce the behavior, for e.g:
1. Setup Parsl 0.9.0 with Python 3.6 on cluster
2. Run a sleep job with GridEngine provider
3. Watch the `qstat`, where you will see jobs disappear

**Expected behavior**
10 minute default walltime

**Environment**
 - OS: CentOS 7
 - Python version 3.6
 - Parsl version 0.9

**Distributed Environment**
- Where are you running the Parsl script from ? Login node

- Where do you need the workers to run ?Worker nodes on same network as login node


Discussion:
The default is provided as a literal `walltime="00:10:00"` in `grid_engine.py` and then converted to minutes by `job_config["walltime"] = wtime_to_minutes(self.walltime)`.

A quick google suggests that SGE expects `HH:MM:SS` format times - although I guess it is accepting a single integer in this case too.

I think this is a bug in the handling of any walltime specification, not only the default value, as an explicitly specified walltime will also go through `wtime_to_minutes`.


Resolving patch:
diff --git a/parsl/providers/grid_engine/grid_engine.py b/parsl/providers/grid_engine/grid_engine.py
--- a/parsl/providers/grid_engine/grid_engine.py
+++ b/parsl/providers/grid_engine/grid_engine.py
@@ -7,7 +7,7 @@
 from parsl.providers.grid_engine.template import template_string
 from parsl.launchers import SingleNodeLauncher
 from parsl.providers.provider_base import JobState, JobStatus
-from parsl.utils import RepresentationMixin, wtime_to_minutes
+from parsl.utils import RepresentationMixin
 
 logger = logging.getLogger(__name__)
 
@@ -99,7 +99,7 @@ def get_configs(self, command, tasks_per_node):
         job_config = {}
         job_config["submit_script_dir"] = self.channel.script_dir
         job_config["nodes"] = self.nodes_per_block
-        job_config["walltime"] = wtime_to_minutes(self.walltime)
+        job_config["walltime"] = self.walltime
         job_config["scheduler_options"] = self.scheduler_options
         job_config["worker_init"] = self.worker_init
         job_config["user_script"] = command


Test patch:
Repository: Parsl/parsl

Issue:
Monitoring db stdout/stderr should not store file mode in filename
**Is your feature request related to a problem? Please describe.**
The monitoring db stores a file mode on the end of stdout/stderr, like this:
`single_frame_driver.400461.R01.stdout;a+`  where the actual filename is ``single_frame_driver.400461.R01.stdout`

The stdout/stderr fields should store the actual name used, not an encoded data structure.

**Describe the solution you'd like**
Drop the file mode entirely from monitoring db.

**Describe alternatives you've considered**
An additional column in the DB for each of stdout and stderr to store the file mode.

**Additional context**
cc @TomGlanzman who raised this with me in the context of lsst, @mjwilde who added file modes to parsl, and @ZhuozhaoLi who implemented this particular formatting of modes in the monitoring db.


Discussion:

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -202,19 +202,17 @@ def _create_task_log_info(self, task_id, fail_mode):
         stdout_spec = self.tasks[task_id]['kwargs'].get('stdout', None)
         stderr_spec = self.tasks[task_id]['kwargs'].get('stderr', None)
         try:
-            stdout_name, stdout_mode = get_std_fname_mode('stdout', stdout_spec)
+            stdout_name, _ = get_std_fname_mode('stdout', stdout_spec)
         except Exception as e:
             logger.warning("Incorrect stdout format {} for Task {}".format(stdout_spec, task_id))
-            stdout_name, stdout_mode = str(e), None
+            stdout_name = str(e)
         try:
-            stderr_name, stderr_mode = get_std_fname_mode('stderr', stderr_spec)
+            stderr_name, _ = get_std_fname_mode('stderr', stderr_spec)
         except Exception as e:
             logger.warning("Incorrect stderr format {} for Task {}".format(stderr_spec, task_id))
-            stderr_name, stderr_mode = str(e), None
-        stdout_spec = ";".join((stdout_name, stdout_mode)) if stdout_mode else stdout_name
-        stderr_spec = ";".join((stderr_name, stderr_mode)) if stderr_mode else stderr_name
-        task_log_info['task_stdout'] = stdout_spec
-        task_log_info['task_stderr'] = stderr_spec
+            stderr_name = str(e)
+        task_log_info['task_stdout'] = stdout_name
+        task_log_info['task_stderr'] = stderr_name
         task_log_info['task_fail_history'] = None
         if self.tasks[task_id]['fail_history'] is not None:
             task_log_info['task_fail_history'] = ",".join(self.tasks[task_id]['fail_history'])


Test patch:
Repository: Parsl/parsl

Issue:
checkpointing vs parsl.AUTO_LOGNAME
`parsl.AUTO_LOG_NAME` is resolved into a dynamically generated path before memoization happens. This means that an app invoked with `parsl.AUTO_LOG_NAME` will not be memoizable/checkpointable, as the arguments differ each time.

Maybe this resolution to a concrete path should be ignored by checkpointing: if an explicit stdout is specified, then it is reasonable to assume that the stdout/stderr really is an output; but if it is an automatic log name, it is debugging-style output and should not affect checkpointing.


Discussion:
PR #1568 allows stdout/stderr to be ignored for memoization purposes. I've been using this in production for a few weeks and it works ok. This issue can be closed if/when #1568 is merged.
@benclifford I just got burned by this. Since there is no conceivable situation where a user would want `parsl.AUTO_LOGNAME` to make an app un-memoizable, what do you think about us adding `stdout` and `stderr` to `ignore_for_cache` automatically if `parsl.AUTO_LOGNAME` is set? Otherwise this is likely to result in significant time wasted for people who haven't studied the docs closely to debug the unexpected behavior.
>  what do you think about us adding stdout and stderr to ignore_for_cache automatically if parsl.AUTO_LOGNAME is set? 

@annawoodard i think that's probably ok, in the sense of "deal with my stdout sensibly for me" rather than specifically "generate a filename"

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -713,6 +713,8 @@ def submit(self, func, app_args, executors='all', fn_hash=None, cache=False, ign
         for kw in ['stdout', 'stderr']:
             if kw in app_kwargs:
                 if app_kwargs[kw] == parsl.AUTO_LOGNAME:
+                    if kw not in ignore_for_cache:
+                        ignore_for_cache += [kw]
                     app_kwargs[kw] = os.path.join(
                                 self.run_dir,
                                 'task_logs',


Test patch:
Repository: Parsl/parsl

Issue:
Allow higher buffer threshold size for serializing objects
At the moment, an app which returns a 1x2 numpy array will succeed, but one that returns a 100x2 numpy array will fail, because its size exceeds the default `buffer_threshold` for [serialize_object](https://github.com/Parsl/parsl/blob/master/parsl/executors/high_throughput/process_worker_pool.py#L509) above which a `MemoryView` object (which can't be pickled) is returned by that function. Eventually we will port back serialization improvements implemented in funcX, but as a stopgap until then I think we should increase the threshold to a more reasonable value.

Here's a notebook showing the problem: [https://gist.github.com/annawoodard/03a9be6bde85480fd85634c29dbfa7a0](https://gist.github.com/annawoodard/03a9be6bde85480fd85634c29dbfa7a0)

This is related to #1480.


Discussion:

Resolving patch:
diff --git a/parsl/executors/high_throughput/process_worker_pool.py b/parsl/executors/high_throughput/process_worker_pool.py
--- a/parsl/executors/high_throughput/process_worker_pool.py
+++ b/parsl/executors/high_throughput/process_worker_pool.py
@@ -506,7 +506,7 @@ def worker(worker_id, pool_id, pool_size, task_queue, result_queue, worker_queue
 
         try:
             result = execute_task(req['buffer'])
-            serialized_result = serialize_object(result)
+            serialized_result = serialize_object(result, buffer_threshold=1e6)
         except Exception as e:
             logger.info('Caught an exception: {}'.format(e))
             result_package = {'task_id': tid, 'exception': serialize_object(RemoteExceptionWrapper(*sys.exc_info()))}


Test patch: