Repository: Parsl/parsl

Issue:
slurm provider can lose track of blocks if squeue polling isn't fast enough, leading to parsl hang
**Describe the bug**
The slurm provider can lose track of slurm jobs/blocks in certain circumstances. It will not report any status change for those blocks, and so if a block has finished but the current recorded status is that the block is running, then the block will be reported as running forever.
This can result in parsl hanging forever, reporting one block available, as it believes it has the ability to make progress when it does not, like this:

```
2020-03-03 06:57:54.444 parsl.dataflow.strategy:199 [DEBUG]  Executor worker-nodes has 109 active tasks, 1/0 running/pending blocks, and 0 connected workers
...
2020-03-03 08:48:29 parsl.dataflow.strategy:199 [DEBUG]  Executor worker-nodes has 109 active tasks, 1/0 running/pending blocks, and 0 connected workers
```

There is a race condition between the provider polling for jobs with `squeue` and slurm moving jobs from `C` (completed) state to not being known/listed.

If that move from `C` to non-existent happens before `slurm._status()` successfully polls that job to discover completion, then further polls with `squeue` will exit with non-zero unix exit code, and this will be ignored silently by `slurm.py` as a slurm failure, forever, rather than as a job completion.

It looks like this may have been introduced around 37250848b8bbf6bdbed3b3ec2f54d5482b79ba5f

Here is an example of such a poll:
```
bxc@cori02:~> squeue --job 28509629
slurm_load_jobs error: Invalid job id specified
bxc@cori02:~> echo $?
1
```

Jobs which are executing will die with `ManagerLost` and cause progress, but jobs which are queued for that executor but waiting to be executed will sit, waiting, forever.

**To Reproduce**
This needs a delay in polling long enough for the completed job to disappear from the slurm queue.

**Expected behavior**
parsl should never hang without making progress.

at the very least, failures in executing slurm commands should be logged as WARNINGs rather than silently ignored.

**Environment**
cori. `lsst-dm-202002` branch of parsl.


Discussion:
There's a bit more subtlety here. Here is some information about behaviour on `cori`:

Right after a job finishes and no longer appears in `squeue` output, `squeue` behaves like this:
```
> date; squeue --job=28556060 ; echo $?
Thu Mar  5 07:05:34 PST 2020
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
0
```
This is the behaviour the slurm provider is expecting.

But after some time, the behaviour changes to:
```
> date; squeue --job=28556060 ; echo $?
Thu Mar  5 07:13:55 PST 2020
slurm_load_jobs error: Invalid job id specified
1
```
which is when the problem described in this issue occurs.
I'm getting this error everywhere in my log. 

Squeue can take some time to return at SDSC, and will be unresponsive some other times, I usually put a retrying error wrapper around it when I put it in scripts.

I'm attaching my log

[parsl.log](https://github.com/Parsl/parsl/files/7359548/parsl.log)


Resolving patch:
diff --git a/parsl/providers/slurm/slurm.py b/parsl/providers/slurm/slurm.py
--- a/parsl/providers/slurm/slurm.py
+++ b/parsl/providers/slurm/slurm.py
@@ -138,15 +138,21 @@ def _status(self):
         Returns:
               [status...] : Status list of all jobs
         '''
-        job_id_list = ','.join(self.resources.keys())
+        job_id_list = ','.join(
+            [jid for jid, job in self.resources.keys() if not job['status'].terminal]
+        )
+        if not job_id_list:
+            logger.debug('No active jobs, skipping status update')
+            return
+
         cmd = "squeue --job {0}".format(job_id_list)
-        logger.debug("Executing sqeueue")
+        logger.debug("Executing %s", cmd)
         retcode, stdout, stderr = self.execute_wait(cmd)
-        logger.debug("sqeueue returned")
+        logger.debug("sqeueue returned %s %s", stdout, stderr)
 
         # Execute_wait failed. Do no update
         if retcode != 0:
-            logger.warning("squeue failed with non-zero exit code {} - see https://github.com/Parsl/parsl/issues/1588".format(retcode))
+            logger.warning("squeue failed with non-zero exit code {}".format(retcode))
             return
 
         jobs_missing = list(self.resources.keys())


Test patch: