Repository: Parsl/parsl

Issue:
Strategy htex_auto_scale removing blocks far too slow
**Describe the bug**
The "htex_auto_scale" strategy is very slow to remove blocks in case there are more slots than tasks. It removes one block per 5 seconds at most.

**To Reproduce**
Steps to reproduce the behavior, for e.g:
1. Run the test script
2. Check parsl.log and wait for the first handful of tasks are completed
3. The logs shows that it will need around a minute after the idle time out to remove the blocks, even though all tasks but one have been finished. Most of the blocks are idle.

**Expected behavior**
All idle blocks are removed as soon as possible. This becomes really important if you are dealing with, say 500 blocks and most of them are idle.

**Environment**
Parsl 1.2.0, Python 3.8


**Test script**
```python
import parsl
from parsl.providers import LocalProvider
from parsl.channels import LocalChannel
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl.app.app import python_app


local_htex = Config(
    executors=[
        HighThroughputExecutor(
            label="htex_Local",
            worker_debug=True,
            max_workers=1,
            provider=LocalProvider(
                channel=LocalChannel(),
                init_blocks=2,
                max_blocks=20,
            ),
        )
    ],
    strategy="htex_auto_scale",
    max_idletime=3,
)
parsl.load(local_htex)


@python_app
def long():
    import time
    time.sleep(10000)


@python_app
def short():
    import time
    time.sleep(1)


a = long()
b = [short() for i in range(20)]
a.result()
```


Discussion:

Resolving patch:
diff --git a/parsl/dataflow/strategy.py b/parsl/dataflow/strategy.py
--- a/parsl/dataflow/strategy.py
+++ b/parsl/dataflow/strategy.py
@@ -242,8 +242,8 @@ def _general_strategy(self, status_list, tasks, *, strategy_type):
                 # Case 2b
                 else:
                     # logger.debug("Strategy: Case.2b")
-                    excess = math.ceil((active_tasks * parallelism) - active_slots)
-                    excess_blocks = math.ceil(float(excess) / (tasks_per_node * nodes_per_block))
+                    excess_slots = math.ceil((active_tasks * parallelism) - active_slots)
+                    excess_blocks = math.ceil(float(excess_slots) / (tasks_per_node * nodes_per_block))
                     excess_blocks = min(excess_blocks, max_blocks - active_blocks)
                     logger.debug("Requesting {} more blocks".format(excess_blocks))
                     exec_status.scale_out(excess_blocks)
@@ -262,7 +262,10 @@ def _general_strategy(self, status_list, tasks, *, strategy_type):
                     logger.debug("More slots than tasks")
                     if isinstance(executor, HighThroughputExecutor):
                         if active_blocks > min_blocks:
-                            exec_status.scale_in(1, force=False, max_idletime=self.max_idletime)
+                            excess_slots = math.ceil(active_slots - (active_tasks * parallelism))
+                            excess_blocks = math.ceil(float(excess_slots) / (tasks_per_node * nodes_per_block))
+                            excess_blocks = min(excess_blocks, active_blocks - min_blocks)
+                            exec_status.scale_in(excess_blocks, force=False, max_idletime=self.max_idletime)
 
                 elif strategy_type == 'simple':
                     # skip for simple strategy


Test patch: