internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Parsl/parsl

Issue:
Monitoring should record tasks when they are created, not when they start running
**Is your feature request related to a problem? Please describe.**
It is unclear from (for example) Gantt plots when tasks are created by app invocation. Monitoring at the moment only records a task once it is passed to an executor, ignoring that the task may not be passed to an executor for some time (for example, because it is waiting on dependencies).

**Describe the solution you'd like**
Monitoring should record a task as soon as it is known to the DFK (pretty much when an app invocation happens)


Discussion:
I actually modified the code to achieve this for the Biology GDC pipeline so I can see the DAG first. I can make a PR for this.

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -779,6 +779,10 @@ def submit(self, func, app_args, executors='all', fn_hash=None, cache=False, ign
         task_def['status'] = States.pending
         logger.debug("Task {} set to pending state with AppFuture: {}".format(task_id, task_def['app_fu']))
 
+        if self.monitoring is not None:
+            task_log_info = self._create_task_log_info(task_id)
+            self.monitoring.send(MessageType.TASK_INFO, task_log_info)
+
         # at this point add callbacks to all dependencies to do a launch_if_ready
         # call whenever a dependency completes.
 
diff --git a/parsl/monitoring/db_manager.py b/parsl/monitoring/db_manager.py
--- a/parsl/monitoring/db_manager.py
+++ b/parsl/monitoring/db_manager.py
@@ -326,7 +326,8 @@ def start(self, priority_queue, node_queue, resource_queue):
                                           'tasks_completed_count'],
                                  messages=update_messages)
                     self._update(table=TASK,
-                                 columns=['task_time_returned',
+                                 columns=['task_time_submitted',
+                                          'task_time_returned',
                                           'run_id', 'task_id',
                                           'task_fail_count',
                                           'task_fail_history'],


Test patch:
Repository: Parsl/parsl

Issue:
Scheduler options not honored in Torque
**Describe the bug**
If a `scheduler_option` that conflicts with a default request in the standard template is made via the `TorqueProvider`, it's currently ignored. It turns out that Torque honors the first line rather than the last in case of a conflicting option. 

For eg:
```
#PBS -l nodes=1:ppn=1
#PBS -l nodes=1:ppn=16:xk   # <--- from scheduler_options, will be ignored.
```



Discussion:

Resolving patch:
diff --git a/parsl/providers/torque/template.py b/parsl/providers/torque/template.py
--- a/parsl/providers/torque/template.py
+++ b/parsl/providers/torque/template.py
@@ -1,5 +1,6 @@
 template_string = '''#!/bin/bash
 
+${scheduler_options}
 #PBS -S /bin/bash
 #PBS -N ${jobname}
 #PBS -m n
@@ -7,7 +8,6 @@
 #PBS -l nodes=${nodes_per_block}:ppn=${tasks_per_node}
 #PBS -o ${submit_script_dir}/${jobname}.submit.stdout
 #PBS -e ${submit_script_dir}/${jobname}.submit.stderr
-${scheduler_options}
 
 ${worker_init}
 
diff --git a/parsl/providers/torque/torque.py b/parsl/providers/torque/torque.py
--- a/parsl/providers/torque/torque.py
+++ b/parsl/providers/torque/torque.py
@@ -58,6 +58,7 @@ class TorqueProvider(ClusterProvider, RepresentationMixin):
         Walltime requested per block in HH:MM:SS.
     scheduler_options : str
         String to prepend to the #PBS blocks in the submit script to the scheduler.
+        WARNING: scheduler_options should only be given #PBS strings, and should not have trailing newlines.
     worker_init : str
         Command to be run before starting a worker, such as 'module load Anaconda; source activate env'.
     launcher : Launcher


Test patch:
Repository: Parsl/parsl

Issue:
Support older versions of Bash
Slurm submission scripts written by Parsl 0.7 use the Bash builtin command 'wait' with the -n option. I believe this option was introduced in a relatively recent version of Bash, perhaps 5.0 Many of the systems at my organization run Redhat Enterprise Linux 7, which uses Bash-4.2. This version of bash does not recognize the -n option to wait, and the script errors out.

Please consider implementing the slurm submission script without using wait -n to improve compatibility with older versions of bash, which I imagine are still in common use.


Discussion:
Hi @jadamstephens, could you try switching to the latest parsl version (v1.0.0)? In the latest version I do not see `wait` being used with the `SlurmProvider`+`SrunLauncher` combination. If you still have issues, please share a tarball of your runinfo directory. 
A quick skim doesn't show me `wait -n` in 0.7.0 either, though, so I'm unclear where the -n is coming from for you. Can you attach a full submission script that has this in it?
Hi,

I wrongly assumed that I was using 0.7 based on some instructions that are probably now outdated. It looks like I did install 1.0.

Anyway, [this line](https://github.com/Parsl/parsl/blob/f11417a0255ed290fd0d78ffa1bc52cfe7a06301/parsl/launchers/launchers.py#L103) in launchers/launcher.py seems to be the source of the wait command in the slurm submission script.

I'm encountering this issue when running [example.py](https://github.com/Parsl/dynamic_launcher) using the command:

```
python3 example.py --fileconfig=blake --count=10 --example=tasks
```

(I modified cori.py to use on a machine here named blake.)


OK. That line isn't in 1.0.0, but was introduced into master in PR #1724. So I guess you've actually installed the latest master, not 1.0.0 (although it likely claims to be 1.0.0).
tagging @hategan because that's some code he's been working on recently
You're right. I did clone master. I checked out and installed the 1.0.0 tag and now it's working. Even so, since the change is on master, my feature request stands :)

Resolving patch:
diff --git a/parsl/launchers/launchers.py b/parsl/launchers/launchers.py
--- a/parsl/launchers/launchers.py
+++ b/parsl/launchers/launchers.py
@@ -88,6 +88,7 @@ def __call__(self, command, tasks_per_node, nodes_per_block):
 [[ "{debug}" == "1" ]] && echo "Found cores : $CORES"
 WORKERCOUNT={task_blocks}
 FAILONANY={fail_on_any}
+PIDS=""
 
 CMD() {{
 {command}
@@ -95,12 +96,13 @@ def __call__(self, command, tasks_per_node, nodes_per_block):
 for COUNT in $(seq 1 1 $WORKERCOUNT); do
     [[ "{debug}" == "1" ]] && echo "Launching worker: $COUNT"
     CMD $COUNT &
+    PIDS="$PIDS $!"
 done
 
 ALLFAILED=1
 ANYFAILED=0
-for COUNT in $(seq 1 1 $WORKERCOUNT); do
-    wait -n
+for PID in $PIDS ; do
+    wait $PID
     if [ "$?" != "0" ]; then
         ANYFAILED=1
     else


Test patch:
Repository: Parsl/parsl

Issue:
Add account option to slurm provider
See Parsl/libsubmit#68


Discussion:

Resolving patch:
diff --git a/parsl/providers/slurm/slurm.py b/parsl/providers/slurm/slurm.py
--- a/parsl/providers/slurm/slurm.py
+++ b/parsl/providers/slurm/slurm.py
@@ -43,6 +43,9 @@ class SlurmProvider(ClusterProvider, RepresentationMixin):
     ----------
     partition : str
         Slurm partition to request blocks from. If none, no partition slurm directive will be specified.
+    account : str
+        Slurm account to which to charge resources used by the job. If none, the job will use the
+        user's default account.
     channel : Channel
         Channel for accessing this provider. Possible channels include
         :class:`~parsl.channels.LocalChannel` (the default),
@@ -77,12 +80,13 @@ class SlurmProvider(ClusterProvider, RepresentationMixin):
         :class:`~parsl.launchers.SingleNodeLauncher` (the default),
         :class:`~parsl.launchers.SrunLauncher`, or
         :class:`~parsl.launchers.AprunLauncher`
-     move_files : Optional[Bool]: should files be moved? by default, Parsl will try to move files.
+    move_files : Optional[Bool]: should files be moved? by default, Parsl will try to move files.
     """
 
     @typeguard.typechecked
     def __init__(self,
                  partition: Optional[str],
+                 account: Optional[str] = None,
                  channel: Channel = LocalChannel(),
                  nodes_per_block: int = 1,
                  cores_per_node: Optional[int] = None,
@@ -115,11 +119,14 @@ def __init__(self,
         self.mem_per_node = mem_per_node
         self.exclusive = exclusive
         self.move_files = move_files
+        self.account = account
         self.scheduler_options = scheduler_options + '\n'
         if exclusive:
             self.scheduler_options += "#SBATCH --exclusive\n"
         if partition:
             self.scheduler_options += "#SBATCH --partition={}\n".format(partition)
+        if account:
+            self.scheduler_options += "#SBATCH --account={}\n".format(account)
         self.worker_init = worker_init + '\n'
 
     def _status(self):


Test patch:
Repository: Parsl/parsl

Issue:
Broken link to online documentation
In the [Quickstart Guide](https://parsl.readthedocs.io/en/stable/quickstart.html#installation-using-conda), the third option in the Tutorial list, "Read through the online tutorial documentation," contains a broken link: <https://parsl.readthedocs.io/en/stable/parsl-introduction> leads to rtd's "Maze Not Found" error page.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -40,9 +40,9 @@
     'sphinx.ext.napoleon'
 ]
 
-url = 'https://raw.githubusercontent.com/Parsl/parsl-tutorial/master/parsl-introduction.ipynb'
+url = 'https://raw.githubusercontent.com/Parsl/parsl-tutorial/master/1-parsl-introduction.ipynb'
 r = requests.get(url)
-with open(os.path.join(os.path.dirname(__file__), 'parsl-introduction.ipynb'), 'wb') as f:
+with open(os.path.join(os.path.dirname(__file__), '1-parsl-introduction.ipynb'), 'wb') as f:
     f.write(r.content)
 
 nbsphinx_execute = 'never'


Test patch:
Repository: Parsl/parsl

Issue:
Support passing custom arguments to WQ worker
We should support passing custom arguments to the WQ worker through the WQ executor.

An argument could be made we should just reproduce any needed arguments as executor keyword args (so all the documentation would be in one place), but for now I favor just allowing us to pass args directly.


Discussion:

Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -169,6 +169,9 @@ class WorkQueueExecutor(NoStatusHandlingExecutor):
         init_command: str
             Command line to run before executing a task in a worker.
             Default is ''.
+
+        worker_options: str
+            Extra options passed to work_queue_worker. Default is ''.
     """
 
     @typeguard.typechecked
@@ -191,6 +194,7 @@ def __init__(self,
                  autolabel_window: int = 1,
                  autocategory: bool = False,
                  init_command: str = "",
+                 worker_options: str = "",
                  full_debug: bool = True):
         NoStatusHandlingExecutor.__init__(self)
         self._provider = provider
@@ -224,6 +228,7 @@ def __init__(self,
         self.autocategory = autocategory
         self.should_stop = multiprocessing.Value(c_bool, False)
         self.cached_envs = {}  # type: Dict[int, str]
+        self.worker_options = worker_options
 
         if not self.address:
             self.address = socket.gethostname()
@@ -380,6 +385,8 @@ def _construct_worker_command(self):
         worker_command = 'work_queue_worker'
         if self.project_password_file:
             worker_command += ' --password {}'.format(self.project_password_file)
+        if self.worker_options:
+            worker_command += ' {}'.format(self.worker_options)
         if self.project_name:
             worker_command += ' -M {}'.format(self.project_name)
         else:


Test patch:
Repository: Parsl/parsl

Issue:
Reduce polling logs from task_status_poller
**Describe the bug**

The `task_status_poller` wakes every 5s, and writes out logs like this : 
```
2020-08-06 13:23:52.412 parsl.dataflow.task_status_poller:62 [DEBUG]  Polling
2020-08-06 13:23:52.412 parsl.dataflow.task_status_poller:26 [DEBUG]  Polling htex_local
2020-08-06 13:23:52.412 parsl.providers.local.local:76 [DEBUG]  Checking status of: ['665352']
2020-08-06 13:23:52.435 parsl.providers.local.local:76 [DEBUG]  Checking status of: ['665352']
```

This is basically useless information and can be removed.

**To Reproduce**
Steps to reproduce the behavior, for e.g:
1. Setup Parsl master locally 
2. Run `https://github.com/Parsl/parsl/blob/htex_scaling_down_%23172/parsl/tests/manual_tests/test_scale_down_zz.py`
3. Wait 5 mins
4. Check runinfo dir for the parsl.log


Discussion:

Resolving patch:
diff --git a/parsl/dataflow/task_status_poller.py b/parsl/dataflow/task_status_poller.py
--- a/parsl/dataflow/task_status_poller.py
+++ b/parsl/dataflow/task_status_poller.py
@@ -25,7 +25,6 @@ def _should_poll(self, now: float):
 
     def poll(self, now: float):
         if self._should_poll(now):
-            logger.debug("Polling {}".format(self._executor.label))
             self._status = self._executor.status()
             self._last_poll_time = now
 
@@ -62,7 +61,6 @@ def __init__(self, dfk: "parsl.dataflow.dflow.DataFlowKernel"):
         self._error_handler = JobErrorHandler()
 
     def poll(self, tasks=None, kind=None):
-        logger.debug("Polling")
         self._update_state()
         self._error_handler.run(self._poll_items)
         self._strategy.strategize(self._poll_items, tasks)
diff --git a/parsl/providers/local/local.py b/parsl/providers/local/local.py
--- a/parsl/providers/local/local.py
+++ b/parsl/providers/local/local.py
@@ -73,7 +73,6 @@ def status(self, job_ids):
 
         '''
 
-        logger.debug("Checking status of: {0}".format(job_ids))
         for job_id in self.resources:
             # This job dict should really be a class on its own
             job_dict = self.resources[job_id]


Test patch:
Repository: Parsl/parsl

Issue:
Support disabling of WQ resource monitor
At the moment, disabling the resource monitor requires setting `autolabel=False` and `full_debug=False`. Disabling the resource monitor should just require setting `autolabel=False` so that other debugging can remain enabled.


Discussion:

Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -712,7 +712,7 @@ def _work_queue_submit_wait(task_queue=multiprocessing.Queue(),
     if wq_log_dir is not None:
         wq_master_log = os.path.join(wq_log_dir, "master_log")
         wq_trans_log = os.path.join(wq_log_dir, "transaction_log")
-        if full:
+        if full and autolabel:
             wq_resource_log = os.path.join(wq_log_dir, "resource_logs")
             q.enable_monitoring_full(dirname=wq_resource_log)
         q.specify_log(wq_master_log)


Test patch:
Repository: Parsl/parsl

Issue:
PR 1727 broke storage of task_hashsum in monitoring
**Describe the bug**
After PR 1727, the monitoring `task` no longer has the task_hashsum field populated.

**To Reproduce**

Run this code on parsl 1.0.0 and on origin/master da2e36d13129db4e567ca40a63c55363aa92f8d7. In the first case, the output of sqlite will show a couple of task_hashsum hashes. In the second case it will not. I used `git bisect` to discover that ed946a92561aa278998b6cd480edf9e44e6e6a13 is the first bad commit in the history of master.

```
rm monitoring.db ; python memo.py "select task_hashsum from task;" ; sqlite3 monitoring.db "select task_hashsum from task;"
```

```
benc@parsl-dev-3-7-8367:~/parsl/src/parsl$ cat memo.py 
import parsl
from parsl.utils import get_all_checkpoints

from parsl.tests.configs.htex_local_alternate import config
parsl.set_stream_logger()

config.checkpoint_files = get_all_checkpoints()

parsl.load(config)

@parsl.python_app(cache=True)
def a():
    import time
    time.sleep(3)

@parsl.python_app
def b():
    import time
    time.sleep(3)

@parsl.python_app(cache=True)
def c():
    import time
    time.sleep(3)

f1 = a()
f2 = b()
f3 = c()
f1.result()
f2.result()
f3.result()
```

**Expected behavior**
The task_hashsum field should be populated in master as it was in 1.0.0

**Environment**
my laptop



Discussion:

Resolving patch:
diff --git a/parsl/monitoring/db_manager.py b/parsl/monitoring/db_manager.py
--- a/parsl/monitoring/db_manager.py
+++ b/parsl/monitoring/db_manager.py
@@ -383,7 +383,8 @@ def start(self, priority_queue, node_queue, resource_queue) -> None:
                                  columns=['task_time_invoked',
                                           'task_time_returned',
                                           'run_id', 'task_id',
-                                          'task_fail_count'],
+                                          'task_fail_count',
+                                          'task_hashsum'],
                                  messages=task_info_update_messages)
                 logger.debug("Inserting {} task_info_all_messages into status table".format(len(task_info_all_messages)))
 


Test patch:
Repository: Parsl/parsl

Issue:
Support memoization of tuples
We already support lists, so implementing an `id_for_memo` for tuple types is straightforward. 



Discussion:

Resolving patch:
diff --git a/parsl/dataflow/memoization.py b/parsl/dataflow/memoization.py
--- a/parsl/dataflow/memoization.py
+++ b/parsl/dataflow/memoization.py
@@ -62,6 +62,19 @@ def id_for_memo_list(denormalized_list, output_ref=False):
     return serialize(normalized_list)
 
 
+@id_for_memo.register(tuple)
+def id_for_memo_tuple(denormalized_tuple, output_ref=False):
+    if type(denormalized_tuple) != tuple:
+        raise ValueError("id_for_memo_tuple cannot work on subclasses of tuple")
+
+    normalized_list = []
+
+    for e in denormalized_tuple:
+        normalized_list.append(id_for_memo(e, output_ref=output_ref))
+
+    return serialize(normalized_list)
+
+
 @id_for_memo.register(dict)
 def id_for_memo_dict(denormalized_dict, output_ref=False):
     """This normalises the keys and values of the supplied dictionary.


Test patch:
Repository: Parsl/parsl

Issue:
WQ category is always the same
**Describe the bug**

The WQ category is always being set according to the function's `__qualname__` [here](https://github.com/Parsl/parsl/blob/master/parsl/executors/workqueue/executor.py#L409), and the function is being wrapped [here](https://github.com/Parsl/parsl/blob/master/parsl/app/errors.py#L154). For some reason, the qualname is not being updated correctly, so the category is always `wrap_error.<locals>.wrapper`:

```
1600806505778540 2352716 CATEGORY wrap_error.<locals>.wrapper MAX {"cores":[18,"cores"],"memory":[2464,"MB"],"disk":[2,"MB"]}
1600806505778556 2352716 CATEGORY wrap_error.<locals>.wrapper MIN {"cores":[3,"cores"],"memory":[100,"MB"],"disk":[2,"MB"]}
1600806505778565 2352716 CATEGORY wrap_error.<locals>.wrapper FIRST MAX_THROUGHPUT {"cores":[3,"cores"],"memory":[100,"MB"],"disk":[2,"MB"]}
```
According to the [docs](https://docs.python.org/3/library/functools.html#functools.wraps), `functools.wraps` should update the qualname as well as the name. But this patch fixes the issue, does anyone object to just setting according to name?

```diff
(gf) annawoodard@daenerys:/cephfs/users/annawoodard/parsl/parsl$ git diff executors/workqueue/
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
index 52aa5c2..accae4e 100644
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -407,6 +407,7 @@ class WorkQueueExecutor(NoStatusHandlingExecutor):
         # Create message to put into the message queue
         logger.debug("Placing task {} on message queue".format(task_id))
-        category = func.__qualname__ if self.autocategory else 'parsl-default'
+        category = func.__name__ if self.autocategory else 'parsl-default'
         self.task_queue.put_nowait(ParslTaskToWq(task_id,
                                                  category,
                                                  cores,
```


Discussion:
`__name__` is what is used else where (for example, monitoring) to distinguish between apps, so it would be more consistent to use `__name__`.


Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -406,7 +406,7 @@ def submit(self, func, resource_specification, *args, **kwargs):
 
         # Create message to put into the message queue
         logger.debug("Placing task {} on message queue".format(task_id))
-        category = func.__qualname__ if self.autocategory else 'parsl-default'
+        category = func.__name__ if self.autocategory else 'parsl-default'
         self.task_queue.put_nowait(ParslTaskToWq(task_id,
                                                  category,
                                                  cores,


Test patch:
Repository: Parsl/parsl

Issue:
bash wrapper does not close log file, resulting in accumulation of open files
**Describe the bug**
This code in the bash remote wrapper:
```
    set_file_logger(filename='{0}/bashexec.{1}.log'.format(logbase, t), name=logname, level=logging.DEBUG, format_string=format_string)
```
opens a new log file per app. But it does not ever close that log file, so a worker running many bash apps will accumulate many open files.

This log file should be closed at the end of each bash app execution.

**To Reproduce**
Run two bash apps in one worker with a long delay. use `lsof` to see which files are open for that worker.

**Expected behavior**
log file should be closed at end of bash app execution



Discussion:

Resolving patch:
diff --git a/parsl/app/bash.py b/parsl/app/bash.py
--- a/parsl/app/bash.py
+++ b/parsl/app/bash.py
@@ -12,26 +12,10 @@ def remote_side_bash_executor(func, *args, **kwargs):
     command-line to run, and then run that command-line using bash.
     """
     import os
-    import time
     import subprocess
-    import logging
     import parsl.app.errors as pe
-    from parsl import set_file_logger
     from parsl.utils import get_std_fname_mode
 
-    logbase = "/tmp"
-    format_string = "%(asctime)s.%(msecs)03d %(name)s:%(lineno)d [%(levelname)s]  %(message)s"
-
-    # make this name unique per invocation so that each invocation can
-    # log to its own file. It would be better to include the task_id here
-    # but that is awkward to wire through at the moment as apps do not
-    # have access to that execution context.
-    t = time.time()
-
-    logname = __name__ + "." + str(t)
-    logger = logging.getLogger(logname)
-    set_file_logger(filename='{0}/bashexec.{1}.log'.format(logbase, t), name=logname, level=logging.DEBUG, format_string=format_string)
-
     func_name = func.__name__
 
     executable = None
@@ -53,11 +37,8 @@ def remote_side_bash_executor(func, *args, **kwargs):
     except IndexError as e:
         raise pe.AppBadFormatting("App formatting failed for app '{}' with IndexError: {}".format(func_name, e))
     except Exception as e:
-        logger.error("Caught exception during formatting of app '{}': {}".format(func_name, e))
         raise e
 
-    logger.debug("Executable: %s", executable)
-
     # Updating stdout, stderr if values passed at call time.
 
     def open_std_fd(fdname):


Test patch: