Repository: Parsl/parsl

Issue:
[PBSPro] max_blocks limit is not obeyed when submitting as array-jobs 
**Describe the bug**
Parsl keeps launching new blocks beyond the `max_blocks` limit when array-jobs mode is enabled using Parsl config `scheduler_options` parameter (e.g. #PBS -J 1-10).

**To Reproduce**
Enable Parsl monitoring and checkpointing
Enable HighThroughputExecutor and PBSProProvider
Add `#PBS -J 1-10` option to `scheduler_options`
Set `max_blocks` limit to 3

**Expected behavior**
No more than 3 blocks should be launched.

**Actual behavior**
Parsl keeps on launching new blocks. Following polling log can be seen
`2020-04-23 14:41:33.575 parsl.dataflow.strategy:205 [DEBUG]  Executor htex_array_jobs has 2739 active tasks, 0/4 running/pending blocks, and 60 connected workers`

This means Parsl does not consider partially activated blocks (array-jobs with some jobs Running others in Queue status out of total array jobs in that block) when making a launch decision.
It seems that conditional check is done here [1] but I couldn't find a place where the `provisioned_blocks` variable is updated. Could you shed some light on how this is updated?

**Environment**
- OS: RHEL6.1
- Python version: 3.7 (Anaconda 4.8.2)
- Parsl version: master branch commit: a30ce173cf8593a34b81d5a9cdd646dcf63fa798

**Distributed Environment**
- PBS Pro in NSCC's ASPIRE1

[1] https://github.com/Parsl/parsl/blob/master/parsl/providers/pbspro/pbspro.py#L109


Discussion:
@annawoodard 
I don't think that provisioned_blocks line has any effect and that code is unused/irrelevant. dataflow/strategy.py should never request more than max_blocks and the provider does not need to enforce that. There is some history here https://github.com/Parsl/parsl/issues/921

Parsl shouldn't be requesting more than max_blocks even if they are in pending state.

So there might be another problem here. Please can you attach a parsl.log showing this behaviour?

cc @hategan @yadudoc who have been working on this scaling behaviour recently.
Also, @ravihansa3000 can you check/confirm that if you remove `#PBS -J 1-10` but change nothing else, does the problem still appear?
I tested without job-array PBS directive and did not observe the original issue.

Thanks for the pointer, now I realize that launch decision is made in the strategy based on the job status provided by the provider. I went through the execution flow in the code and noticed that there is no entry for batch type jobs (- 'B'). Would you be able to add that?

https://github.com/Parsl/parsl/blob/master/parsl/providers/torque/torque.py#L15


@ravihansa3000 please can you attach a parsl.log file for a run showing the bad behaviour?

Resolving patch:
diff --git a/parsl/providers/torque/torque.py b/parsl/providers/torque/torque.py
--- a/parsl/providers/torque/torque.py
+++ b/parsl/providers/torque/torque.py
@@ -13,6 +13,7 @@
 
 # From the man pages for qstat for PBS/Torque systems
 translate_table = {
+    'B': JobState.RUNNING,  # This state is returned for running array jobs
     'R': JobState.RUNNING,
     'C': JobState.COMPLETED,  # Completed after having run
     'E': JobState.COMPLETED,  # Exiting after having run


Test patch: