Repository: Parsl/parsl

Issue:
[htex auto scale] RuntimeError: dictionary changed size during iteration
**Describe the bug**
A RuntimeError exception is thrown (stack trace below) while running a workflow based on Parsl.

```
2020-09-10 00:15:14 parsl.dataflow.dflow:509 [INFO]  Task 0 launched on executor medium
2020-09-10 00:15:14 workflow:68 [INFO]  Waiting for validation task to complete...
2020-09-10 00:15:17 parsl.providers.torque.torque:128 [DEBUG]  coerced job_id 1838438.wlm01 -> 1838438.wlm01
2020-09-10 00:15:17 parsl.dataflow.strategy:332 [DEBUG]  Executor medium has 0 active tasks, 0/1 running/pending blocks, and 0 connected workers
2020-09-10 00:15:17 parsl.dataflow.flow_control:114 [ERROR]  Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
  File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/flow_control.py", line 112, in make_callback
    self.callback(tasks=self._event_buffer, kind=kind)
  File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 66, in poll
    self._strategy.strategize(self._poll_items, tasks)
  File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/strategy.py", line 302, in _strategy_htex_auto_scale
    for label, executor in self.dfk.executors.items():
RuntimeError: dictionary changed size during iteration
2020-09-10 00:15:22 parsl.providers.torque.torque:128 [DEBUG]  coerced job_id 1838438.wlm01 -> 1838438.wlm01
```

This is similar to the issue reported in #1606 which occurred intermittently when using `strategy='simple'`
The same fix should be done for `htex_auto_scale` which was introduced in #1563

**To Reproduce**
1. Setup Parsl with master branch commit: 5445f90
2. Enable checkpointing and monitoring
3. Set `strategy='htex_auto_scale'` in Parsl config
4. Execute a simple workflow (multiple bash_apps and python_apps chained together)

**Expected behavior**
No RuntimeError exceptions.

**Environment**
OS: RHEL6.1
Python version: 3.7 (Anaconda 4.8.2)
Parsl version: master branch commit: 5445f90

**Distributed Environment**
PBS Pro in NSCC's ASPIRE1

[htex auto scale] RuntimeError: dictionary changed size during iteration
**Describe the bug**
A RuntimeError exception is thrown (stack trace below) while running a workflow based on Parsl.

```
2020-09-10 00:15:14 parsl.dataflow.dflow:509 [INFO]  Task 0 launched on executor medium
2020-09-10 00:15:14 workflow:68 [INFO]  Waiting for validation task to complete...
2020-09-10 00:15:17 parsl.providers.torque.torque:128 [DEBUG]  coerced job_id 1838438.wlm01 -> 1838438.wlm01
2020-09-10 00:15:17 parsl.dataflow.strategy:332 [DEBUG]  Executor medium has 0 active tasks, 0/1 running/pending blocks, and 0 connected workers
2020-09-10 00:15:17 parsl.dataflow.flow_control:114 [ERROR]  Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
  File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/flow_control.py", line 112, in make_callback
    self.callback(tasks=self._event_buffer, kind=kind)
  File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/task_status_poller.py", line 66, in poll
    self._strategy.strategize(self._poll_items, tasks)
  File "/home/akila/Documents/NUS_CSI/Parsl/parsl.git/parsl/dataflow/strategy.py", line 302, in _strategy_htex_auto_scale
    for label, executor in self.dfk.executors.items():
RuntimeError: dictionary changed size during iteration
2020-09-10 00:15:22 parsl.providers.torque.torque:128 [DEBUG]  coerced job_id 1838438.wlm01 -> 1838438.wlm01
```

This is similar to the issue reported in #1606 which occurred intermittently when using `strategy='simple'`
The same fix should be done for `htex_auto_scale` which was introduced in #1563

**To Reproduce**
1. Setup Parsl with master branch commit: 5445f90
2. Enable checkpointing and monitoring
3. Set `strategy='htex_auto_scale'` in Parsl config
4. Execute a simple workflow (multiple bash_apps and python_apps chained together)

**Expected behavior**
No RuntimeError exceptions.

**Environment**
OS: RHEL6.1
Python version: 3.7 (Anaconda 4.8.2)
Parsl version: master branch commit: 5445f90

**Distributed Environment**
PBS Pro in NSCC's ASPIRE1



Discussion:
@benclifford perhaps we should add a test case to uncover this race condition
ideally i think those two strategies should share more code because they don't do very much different from each other.

@benclifford perhaps we should add a test case to uncover this race condition
ideally i think those two strategies should share more code because they don't do very much different from each other.


Resolving patch:
diff --git a/parsl/dataflow/strategy.py b/parsl/dataflow/strategy.py
--- a/parsl/dataflow/strategy.py
+++ b/parsl/dataflow/strategy.py
@@ -155,127 +155,13 @@ def unset_logging(self):
         self.logger_flag = True
 
     def _strategy_simple(self, status_list, tasks, *args, kind=None, **kwargs):
-        """Peek at the DFK and the executors specified.
+        self._general_strategy(status_list, tasks, *args, strategy_type='simple', kind=None, **kwargs)
 
-        We assume here that tasks are not held in a runnable
-        state, and that all tasks from an app would be sent to
-        a single specific executor, i.e tasks cannot be specified
-        to go to one of more executors.
+    def _strategy_htex_auto_scale(self, status_list, tasks, *args, kind=None, **kwargs):
+        self._general_strategy(status_list, tasks, *args, strategy_type='htex', kind=None, **kwargs)
 
-        Args:
-            - tasks (task_ids): Not used here.
-
-        KWargs:
-            - kind (Not used)
-        """
-
-        for exec_status in status_list:
-            executor = exec_status.executor
-            label = executor.label
-            if not executor.scaling_enabled:
-                continue
-
-            # Tasks that are either pending completion
-            active_tasks = executor.outstanding
-
-            status = exec_status.status
-            # Dict[object, JobStatus]: job_id -> status
-            self.unset_logging()
-
-            # FIXME we need to handle case where provider does not define these
-            # FIXME probably more of this logic should be moved to the provider
-            min_blocks = executor.provider.min_blocks
-            max_blocks = executor.provider.max_blocks
-            if isinstance(executor, HighThroughputExecutor):
-                tasks_per_node = executor.workers_per_node
-            elif isinstance(executor, ExtremeScaleExecutor):
-                tasks_per_node = executor.ranks_per_node
-
-            nodes_per_block = executor.provider.nodes_per_block
-            parallelism = executor.provider.parallelism
-
-            running = sum([1 for x in status.values() if x.state == JobState.RUNNING])
-            pending = sum([1 for x in status.values() if x.state == JobState.PENDING])
-            active_blocks = running + pending
-            active_slots = active_blocks * tasks_per_node * nodes_per_block
-
-            if hasattr(executor, 'connected_workers'):
-                logger.debug('Executor {} has {} active tasks, {}/{} running/pending blocks, and {} connected workers'.format(
-                    label, active_tasks, running, pending, executor.connected_workers))
-            else:
-                logger.debug('Executor {} has {} active tasks and {}/{} running/pending blocks'.format(
-                    label, active_tasks, running, pending))
-
-            # reset kill timer if executor has active tasks
-            if active_tasks > 0 and self.executors[executor.label]['idle_since']:
-                self.executors[executor.label]['idle_since'] = None
-
-            # Case 1
-            # No tasks.
-            if active_tasks == 0:
-                # Case 1a
-                # Fewer blocks that min_blocks
-                if active_blocks <= min_blocks:
-                    # Ignore
-                    # logger.debug("Strategy: Case.1a")
-                    pass
-
-                # Case 1b
-                # More blocks than min_blocks. Scale down
-                else:
-                    # We want to make sure that max_idletime is reached
-                    # before killing off resources
-                    if not self.executors[executor.label]['idle_since']:
-                        logger.debug("Executor {} has 0 active tasks; starting kill timer (if idle time exceeds {}s, resources will be removed)".format(
-                            label, self.max_idletime)
-                        )
-                        self.executors[executor.label]['idle_since'] = time.time()
-
-                    idle_since = self.executors[executor.label]['idle_since']
-                    if (time.time() - idle_since) > self.max_idletime:
-                        # We have resources idle for the max duration,
-                        # we have to scale_in now.
-                        logger.debug("Idle time has reached {}s for executor {}; removing resources".format(
-                            self.max_idletime, label)
-                        )
-                        exec_status.scale_in(active_blocks - min_blocks)
-
-                    else:
-                        pass
-                        # logger.debug("Strategy: Case.1b. Waiting for timer : {0}".format(idle_since))
-
-            # Case 2
-            # More tasks than the available slots.
-            elif (float(active_slots) / active_tasks) < parallelism:
-                # Case 2a
-                # We have the max blocks possible
-                if active_blocks >= max_blocks:
-                    # Ignore since we already have the max nodes
-                    # logger.debug("Strategy: Case.2a")
-                    pass
+    def _general_strategy(self, status_list, tasks, *args, strategy_type='simple', kind=None, **kwargs):
 
-                # Case 2b
-                else:
-                    # logger.debug("Strategy: Case.2b")
-                    excess = math.ceil((active_tasks * parallelism) - active_slots)
-                    excess_blocks = math.ceil(float(excess) / (tasks_per_node * nodes_per_block))
-                    excess_blocks = min(excess_blocks, max_blocks - active_blocks)
-                    logger.debug("Requesting {} more blocks".format(excess_blocks))
-                    exec_status.scale_out(excess_blocks)
-
-            elif active_slots == 0 and active_tasks > 0:
-                # Case 4
-                # Check if slots are being lost quickly ?
-                logger.debug("Requesting single slot")
-                if active_blocks < max_blocks:
-                    exec_status.scale_out(1)
-            # Case 3
-            # tasks ~ slots
-            else:
-                # logger.debug("Strategy: Case 3")
-                pass
-
-    def _strategy_htex_auto_scale(self, tasks, *args, kind=None, **kwargs):
         """ HTEX specific auto scaling strategy
 
         This strategy works only for HTEX. This strategy will scale up by
@@ -296,15 +182,16 @@ def _strategy_htex_auto_scale(self, tasks, *args, kind=None, **kwargs):
         KWargs:
             - kind (Not used)
         """
-
-        for label, executor in self.dfk.executors.items():
+        for exec_status in status_list:
+            executor = exec_status.executor
+            label = executor.label
             if not executor.scaling_enabled:
                 continue
 
             # Tasks that are either pending completion
             active_tasks = executor.outstanding
 
-            status = executor.status()
+            status = exec_status.status
             self.unset_logging()
 
             # FIXME we need to handle case where provider does not define these
@@ -364,7 +251,7 @@ def _strategy_htex_auto_scale(self, tasks, *args, kind=None, **kwargs):
                         logger.debug("Idle time has reached {}s for executor {}; removing resources".format(
                             self.max_idletime, label)
                         )
-                        executor.scale_in(active_blocks - min_blocks)
+                        exec_status.scale_in(active_blocks - min_blocks)
 
                     else:
                         pass
@@ -387,21 +274,28 @@ def _strategy_htex_auto_scale(self, tasks, *args, kind=None, **kwargs):
                     excess_blocks = math.ceil(float(excess) / (tasks_per_node * nodes_per_block))
                     excess_blocks = min(excess_blocks, max_blocks - active_blocks)
                     logger.debug("Requesting {} more blocks".format(excess_blocks))
-                    executor.scale_out(excess_blocks)
+                    exec_status.scale_out(excess_blocks)
 
             elif active_slots == 0 and active_tasks > 0:
                 # Case 4
                 logger.debug("Requesting single slot")
                 if active_blocks < max_blocks:
-                    executor.scale_out(1)
+                    exec_status.scale_out(1)
 
             # Case 4
             # More slots than tasks
             elif active_slots > 0 and active_slots > active_tasks:
-                logger.debug("More slots than tasks")
-                if isinstance(executor, HighThroughputExecutor):
-                    if active_blocks > min_blocks:
-                        executor.scale_in(1, force=False, max_idletime=self.max_idletime)
+                if strategy_type == 'htex':
+                    # Scale down for htex
+                    logger.debug("More slots than tasks")
+                    if isinstance(executor, HighThroughputExecutor):
+                        if active_blocks > min_blocks:
+                            exec_status.scale_in(1, force=False, max_idletime=self.max_idletime)
+
+                elif strategy_type == 'simple':
+                    # skip for simple strategy
+                    pass
+
             # Case 3
             # tasks ~ slots
             else:
diff --git a/parsl/dataflow/task_status_poller.py b/parsl/dataflow/task_status_poller.py
--- a/parsl/dataflow/task_status_poller.py
+++ b/parsl/dataflow/task_status_poller.py
@@ -61,8 +61,11 @@ def status(self) -> Dict[object, JobStatus]:
     def executor(self) -> ParslExecutor:
         return self._executor
 
-    def scale_in(self, n):
-        ids = self._executor.scale_in(n)
+    def scale_in(self, n, force=True, max_idletime=None):
+        if force and not max_idletime:
+            ids = self._executor.scale_in(n)
+        else:
+            ids = self._executor.scale_in(n, force=force, max_idletime=max_idletime)
         if ids is not None:
             new_status = {}
             for id in ids:


Test patch: