Repository: cylc/cylc-flow
Pull request #6643: host-select: fix compatibility with force-condemned hosts
Author: login: oliver-sanders
Labels: name: bug
description: Something is wrong :(
Created: 2025-02-28T15:26:54Z | Merged: 2025-03-20T12:05:17Z

Supersedes https://github.com/cylc/cylc-flow/pull/6623

Fix an issue with host-select which prevented us from condemning hosts in "force mode".

Also, better link into the "auto restart" docs which can be found here: https://cylc.github.io/cylc-doc/stable/html/plugins/main-loop/built-in/cylc.flow.main_loop.auto_restart.html#module-cylc.flow.main_loop.auto_restart

I am **not** going to go applying terminology changes to this long-standing feature as part of this bugfix (out of scope).

**Check List**

- [x] I have read `CONTRIBUTING.md` and added my name as a Code Contributor.
- [x] Contains logically grouped changes (else tidy your branch by rebase).
- [x] Does not contain off-topic changes (use other PRs for other changes).
- [x] Applied any dependency changes to both `setup.cfg` (and `conda-environment.yml` if present).
- [x] Tests are included (or explain why tests are not needed).
- [x] Changelog entry included if this is a change that can affect users
- [x] [Cylc-Doc](https://github.com/cylc/cylc-doc) pull request opened if required at cylc/cylc-doc/pull/XXXX.
- [x] If this is a bug fix, PR should be raised against the relevant `?.?.x` branch.


Diff:
diff --git a/changes.d/6623.fix.md b/changes.d/6623.fix.md
new file mode 100644
index 00000000000..b99142b12c0
--- /dev/null
+++ b/changes.d/6623.fix.md
@@ -0,0 +1,3 @@
+Auto restart: The option to condemn a host in "force" mode (that tells
+workflows running on a server to shutdown as opposed to migrate) hasn't worked
+with the host-selection mechanism since Cylc 8.0.0. This has now been fixed.
diff --git a/cylc/flow/cfgspec/globalcfg.py b/cylc/flow/cfgspec/globalcfg.py
index 7ca7372e442..629f07db67d 100644
--- a/cylc/flow/cfgspec/globalcfg.py
+++ b/cylc/flow/cfgspec/globalcfg.py
@@ -826,16 +826,52 @@ def default_for(
                    range.
             ''')
             Conf('condemned', VDR.V_ABSOLUTE_HOST_LIST, desc=f'''
-                These hosts will not be used to run jobs.
+                List run hosts that workflows should *not* run on.
 
-                If workflows are already running on
-                condemned hosts, Cylc will shut them down and
-                restart them on different hosts.
+                These hosts will be subtracted from the
+                `available <global.cylc[scheduler][run hosts]>` hosts
+                preventing new workflows from starting on the "condemned" host.
+
+                Any workflows running on these hosts will either migrate
+                to another host, or shut down according to
+                :py:mod:`the configuration <cylc.flow.main_loop.auto_restart>`.
+
+                This feature requires ``auto restart`` to be listed
+                in `global.cylc[scheduler][main loop]plugins`.
+
+                For more information, see the
+                :py:mod:`auto restart <cylc.flow.main_loop.auto_restart>`
+                plugin.
+
+                .. rubric:: Example:
+
+                .. code-block:: cylc
+
+                   [scheduler]
+                       [[main loop]]
+                           # activate the "auto restart" plugin
+                           plugins = auto restart
+                       [[run hosts]]
+                           # there are three hosts in the "pool"
+                           available = host1, host2, host3
+
+                           # however two have been taken out:
+                           # * workflows running on "host1" will attempt to
+                           #   restart on "host3"
+                           # * workflows running on "host2" will shutdown
+                           condemned = host1, host2!
 
                 .. seealso::
 
+                   :py:mod:`cylc.flow.main_loop.auto_restart`
                    :ref:`auto-stop-restart`
 
+                .. versionchanged:: 8.4.2
+
+                   The "force mode" (activated by a "!" suffix) caused issues
+                   at workflow startup for Cylc versions between 8.0.0 and
+                   8.4.1 inclusive.
+
                 .. versionchanged:: 8.0.0
 
                    {REPLACES}``[suite servers]condemned hosts``.
@@ -1336,7 +1372,7 @@ def default_for(
                 The means by which task progress messages are reported back to
                 the running workflow.
 
-                ..rubric:: Options:
+                .. rubric:: Options:
 
                 zmq
                    Direct client-server TCP communication via network ports
diff --git a/cylc/flow/host_select.py b/cylc/flow/host_select.py
index 69e32c68a71..cf940864b90 100644
--- a/cylc/flow/host_select.py
+++ b/cylc/flow/host_select.py
@@ -128,6 +128,13 @@ def select_workflow_host(cached=True):
     # be returned with the up-to-date configuration.
     global_config = glbl_cfg(cached=cached)
 
+    # condemned hosts may be suffixed with an "!" to activate "force mode"
+    blacklist = []
+    for host in global_config.get(['scheduler', 'run hosts', 'condemned'], []):
+        if host.endswith('!'):
+            host = host[:-1]
+        blacklist.append(host)
+
     return select_host(
         # list of workflow hosts
         global_config.get([
@@ -138,9 +145,7 @@ def select_workflow_host(cached=True):
             'scheduler', 'run hosts', 'ranking'
         ]),
         # list of condemned hosts
-        blacklist=global_config.get(
-            ['scheduler', 'run hosts', 'condemned']
-        ),
+        blacklist=blacklist,
         blacklist_name='condemned host'
     )
 
diff --git a/tests/functional/restart/43-auto-restart-force-override-normal.t b/tests/functional/restart/43-auto-restart-force-override-normal.t
index b61d08c68cb..35edc57d1f9 100644
--- a/tests/functional/restart/43-auto-restart-force-override-normal.t
+++ b/tests/functional/restart/43-auto-restart-force-override-normal.t
@@ -50,7 +50,10 @@ create_test_global_config '' "
 ${BASE_GLOBAL_CONFIG}
 [scheduler]
     [[run hosts]]
-        available = ${CYLC_TEST_HOST_1}
+        available = ${CYLC_TEST_HOST_1}, ${CYLC_TEST_HOST_2}
+        # ensure the workflow can start if a host is condemned
+        # in force mode see #6623
+        condemned = ${CYLC_TEST_HOST_2}!
 "
 
 set_test_number 8


Reviews:
user: login: oliver-sanders
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T15:27:22Z
user: login: hjoliver
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-17T04:18:09Z
user: login: oliver-sanders
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T11:30:32Z
user: login: oliver-sanders
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T11:30:38Z
user: login: hjoliver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T23:14:47Z
user: login: hjoliver
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-20T05:52:40Z
user: login: wxtim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T09:01:20Z
user: login: oliver-sanders
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T09:06:57Z
user: login: wxtim
body: - [x] Run test locally
- [x] Read code

I've left one comment, which you should consider. 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T09:11:14Z
user: login: wxtim
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-20T09:11:39Z

Review comments:
diff_hunk: @@ -128,6 +128,13 @@ def select_workflow_host(cached=True):
     # be returned with the up-to-date configuration.
     global_config = glbl_cfg(cached=cached)
 
+    # condemned hosts may be suffixed with an "!" to activate "force mode"
+    blacklist = []
+    for host in global_config.get(['scheduler', 'run hosts', 'condemned'], []):
+        if host.endswith('!'):
+            host = host[:-1]
path: cylc/flow/host_select.py
user: login: oliver-sanders
body: Note, this line is covered by the test amended in this PR (confirm by running it on master), however, that test is not run in CI due to it's shared filesystem setup.
created_at: 2025-02-28T15:27:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,3 @@
+Auto restart: The option to condemn a host in "force" mode (that tells
+workflows running on a server to shutdown as opposed to migrate) hasn't worked
path: changes.d/6623.fix.md
user: login: hjoliver
body: ```suggestion
Auto restart: The option to tell 
workflows running on a server to shutdown as opposed to migrate hasn't worked
```
created_at: 2025-03-17T04:15:38Z
author_association: MEMBER
diff_hunk: @@ -826,16 +826,52 @@ def default_for(
                    range.
             ''')
             Conf('condemned', VDR.V_ABSOLUTE_HOST_LIST, desc=f'''
-                These hosts will not be used to run jobs.
+                List run hosts that workflows should *not* run on.
 
-                If workflows are already running on
-                condemned hosts, Cylc will shut them down and
-                restart them on different hosts.
+                These hosts will be subtracted from the
+                `available <global.cylc[scheduler][run hosts]>` hosts
+                preventing new workflows from starting on the "condemned" host.
+
+                Any workflows running on these hosts will either migrate
+                to another host, or shutdown according to
path: cylc/flow/cfgspec/globalcfg.py
user: login: hjoliver
body: ```suggestion
                to another host, or shut down according to
```
created_at: 2025-03-17T04:16:02Z
author_association: MEMBER
diff_hunk: @@ -826,16 +826,52 @@ def default_for(
                    range.
             ''')
             Conf('condemned', VDR.V_ABSOLUTE_HOST_LIST, desc=f'''
-                These hosts will not be used to run jobs.
+                List run hosts that workflows should *not* run on.
 
-                If workflows are already running on
-                condemned hosts, Cylc will shut them down and
-                restart them on different hosts.
+                These hosts will be subtracted from the
+                `available <global.cylc[scheduler][run hosts]>` hosts
+                preventing new workflows from starting on the "condemned" host.
+
+                Any workflows running on these hosts will either migrate
+                to another host, or shutdown according to
+                :py:mod:`the configuration <cylc.flow.main_loop.auto_restart>`.
+
+                This feature requires ``auto restart`` to be listed
+                in `global.cylc[scheduler][main loop]plugins`.
+
+                For more information, see the
+                :py:mod:`auto restart <cylc.flow.main_loop.auto_restart>`
+                plugin.
+
+                .. rubric:: Example:
+
+                .. code-block:: cylc
+
+                   [scheduler]
+                       [[main loop]]
+                           # activate the "auto restart" plugin
+                           plugins = auto restart
+                       [[run hosts]]
+                           # there are three hosts in the "pool"
+                           available = host1, host2, host3
+
+                           # however two have been taken out:
+                           # * workflows running on "host1" will attempt to
+                           #   restart on "host3"
+                           # * workflows running on "host2" will shutdown
+                           condemned = host1, host2!
 
                 .. seealso::
 
+                   :py:mod:`cylc.flow.main_loop.auto_restart`
                    :ref:`auto-stop-restart`
 
+                .. versionchanged:: 8.4.2
+
+                   The "force mode" (activated by a "!" suffix) caused issues
path: cylc/flow/cfgspec/globalcfg.py
user: login: hjoliver
body: ```suggestion
                   The stop (as opposed to migrate) option (activated by a "!" suffix)
                   caused issues
```
created_at: 2025-03-17T04:17:46Z
author_association: MEMBER
diff_hunk: @@ -826,16 +826,52 @@ def default_for(
                    range.
             ''')
             Conf('condemned', VDR.V_ABSOLUTE_HOST_LIST, desc=f'''
-                These hosts will not be used to run jobs.
+                List run hosts that workflows should *not* run on.
 
-                If workflows are already running on
-                condemned hosts, Cylc will shut them down and
-                restart them on different hosts.
+                These hosts will be subtracted from the
+                `available <global.cylc[scheduler][run hosts]>` hosts
+                preventing new workflows from starting on the "condemned" host.
+
+                Any workflows running on these hosts will either migrate
+                to another host, or shutdown according to
+                :py:mod:`the configuration <cylc.flow.main_loop.auto_restart>`.
+
+                This feature requires ``auto restart`` to be listed
+                in `global.cylc[scheduler][main loop]plugins`.
+
+                For more information, see the
+                :py:mod:`auto restart <cylc.flow.main_loop.auto_restart>`
+                plugin.
+
+                .. rubric:: Example:
+
+                .. code-block:: cylc
+
+                   [scheduler]
+                       [[main loop]]
+                           # activate the "auto restart" plugin
+                           plugins = auto restart
+                       [[run hosts]]
+                           # there are three hosts in the "pool"
+                           available = host1, host2, host3
+
+                           # however two have been taken out:
+                           # * workflows running on "host1" will attempt to
+                           #   restart on "host3"
+                           # * workflows running on "host2" will shutdown
+                           condemned = host1, host2!
 
                 .. seealso::
 
+                   :py:mod:`cylc.flow.main_loop.auto_restart`
                    :ref:`auto-stop-restart`
 
+                .. versionchanged:: 8.4.2
+
+                   The "force mode" (activated by a "!" suffix) caused issues
path: cylc/flow/cfgspec/globalcfg.py
user: login: oliver-sanders
body: This PR will continue to use the longstanding "force mode" terms documented in the plugins section:

https://cylc.github.io/cylc-doc/stable/html/plugins/main-loop/built-in/cylc.flow.main_loop.auto_restart.html#module-cylc.flow.main_loop.auto_restart
created_at: 2025-03-18T11:30:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,3 @@
+Auto restart: The option to condemn a host in "force" mode (that tells
+workflows running on a server to shutdown as opposed to migrate) hasn't worked
path: changes.d/6623.fix.md
user: login: oliver-sanders
body: This PR will continue to use the longstanding "force mode" terms documented in the plugins section:

https://cylc.github.io/cylc-doc/stable/html/plugins/main-loop/built-in/cylc.flow.main_loop.auto_restart.html#module-cylc.flow.main_loop.auto_restart
created_at: 2025-03-18T11:30:38Z
author_association: MEMBER
diff_hunk: @@ -826,16 +826,52 @@ def default_for(
                    range.
             ''')
             Conf('condemned', VDR.V_ABSOLUTE_HOST_LIST, desc=f'''
-                These hosts will not be used to run jobs.
+                List run hosts that workflows should *not* run on.
 
-                If workflows are already running on
-                condemned hosts, Cylc will shut them down and
-                restart them on different hosts.
+                These hosts will be subtracted from the
+                `available <global.cylc[scheduler][run hosts]>` hosts
+                preventing new workflows from starting on the "condemned" host.
+
+                Any workflows running on these hosts will either migrate
+                to another host, or shutdown according to
+                :py:mod:`the configuration <cylc.flow.main_loop.auto_restart>`.
+
+                This feature requires ``auto restart`` to be listed
+                in `global.cylc[scheduler][main loop]plugins`.
+
+                For more information, see the
+                :py:mod:`auto restart <cylc.flow.main_loop.auto_restart>`
+                plugin.
+
+                .. rubric:: Example:
+
+                .. code-block:: cylc
+
+                   [scheduler]
+                       [[main loop]]
+                           # activate the "auto restart" plugin
+                           plugins = auto restart
+                       [[run hosts]]
+                           # there are three hosts in the "pool"
+                           available = host1, host2, host3
+
+                           # however two have been taken out:
+                           # * workflows running on "host1" will attempt to
+                           #   restart on "host3"
+                           # * workflows running on "host2" will shutdown
+                           condemned = host1, host2!
 
                 .. seealso::
 
+                   :py:mod:`cylc.flow.main_loop.auto_restart`
                    :ref:`auto-stop-restart`
 
+                .. versionchanged:: 8.4.2
+
+                   The "force mode" (activated by a "!" suffix) caused issues
path: cylc/flow/cfgspec/globalcfg.py
user: login: hjoliver
body: > This PR will continue to use the longstanding "force mode" terms documented in the plugins section:

Well, it will so long as your reviewers agree to that 😜 
created_at: 2025-03-18T23:14:46Z
author_association: MEMBER
diff_hunk: @@ -128,6 +128,13 @@ def select_workflow_host(cached=True):
     # be returned with the up-to-date configuration.
     global_config = glbl_cfg(cached=cached)
 
+    # condemned hosts may be suffixed with an "!" to activate "force mode"
+    blacklist = []
path: cylc/flow/host_select.py
user: login: wxtim
body: prefer `condemn_list` or `condemned_hosts`?
created_at: 2025-03-20T09:01:20Z
author_association: MEMBER
diff_hunk: @@ -128,6 +128,13 @@ def select_workflow_host(cached=True):
     # be returned with the up-to-date configuration.
     global_config = glbl_cfg(cached=cached)
 
+    # condemned hosts may be suffixed with an "!" to activate "force mode"
+    blacklist = []
path: cylc/flow/host_select.py
user: login: oliver-sanders
body: The name `blacklist` is used elsewhere in the file, so using the same name here for consistency. To explain:

The `select_host` method below is abstract (as we may have other uses for this beyond Cylc server selection) so uses the generic name `blacklist`.

This is the concrete method for Cylc server selection. It populates the `blacklist` argument of `select_host` from condemned hosts found in the configuration.
created_at: 2025-03-20T09:06:57Z
author_association: MEMBER

PR comments:
user: login: hjoliver
created_at: 2025-03-17T04:14:19Z
body: > I am not going to go applying terminology changes to this long-standing feature as part of this bugfix (out of scope).

I presume you mean the "force condemned" term? That needs to be dropped, which you more or less agreed to on the original PR - why have you changed your mind? 

For readers who didn't follow the original discussion, "force condemned" is just bad terminology, given the universally accepted meaning of the word condemned: the subject (the host, here) has been sentenced to some *future* dire fate (e.g. host shutdown). 

Once a host is condemned we need to drain it, either by telling its workflows to migrate or to stop. But it is 100% condemned either way, *before draining*, so "force condemned" doesn't make sense.  What's being forced, if anything, is the draining, not the condemnation.

There's really no need to use a specific term for this anyway since there is no associated config item. We can just document that  there are two ways of draining a condemned host - migrate and stop - and why we might want to use the stop option.

author_association: MEMBER
user: login: hjoliver
created_at: 2025-03-17T04:22:34Z
body: > There's really no need to use a specific term for this anyway since there is no associated config item. 

See my review tweaks above - I only had to nuke "force" in two rather unimportant places - the change log and one version-changed note, which kinda confirms that the existing concepts of "migrate" and "stop" are perfectly sufficient for this.

(I'm fine with mentioning that it used to be called "force  condemned", but the fact that it's been broken since 8.0 suggests that no one will really care).
author_association: MEMBER
user: login: oliver-sanders
created_at: 2025-03-18T11:29:23Z
body: > I presume you mean the "force condemned" term? That needs to be dropped, which you more or less agreed to on the original PR - why have you changed your mind?

Slight misunderstanding.

In the last PR I used the term "force-condemned" which "irked" you, fine, removed. This term hadn't been used before and I have stripped it completely from this PR. Consequently, there is no need for terminology changes, as the long standing terms "force mode" and "normal mode" are absolutely fine.
author_association: MEMBER
user: login: hjoliver
created_at: 2025-03-18T23:10:59Z
body: I'm somewhat happier with that, but just to note you muddied the waters here by literally using "force-condemned hosts" in the title of *this* PR! - `host-select: fix compatibility with force-condemned hosts`

Anyhow, I don't really understand the objection to changing to more sensible descriptive terminology here. It should be this:

- **normal mode -> migrate mode** (workflows told to migrate)
  - (what's "normal" about auto-migration?)
- **force mode -> stop mode** (workflows told to stop)
  - (why is this more "forced" than the other mode?)
  - (because all workflows will stop if asked to (with `--now --now` at least) but won't necessarily migrate if asked to? But that's just normal stop behaviour so it still isn't a strong reason to call it forced IMO)

The plugin documentation needs updating anyway, as it isn't very clear: it doesn't even state that "normal mode" will not stop workflows that fail to migrate, or that "stop mode" actually tells the workflows to stop (and what kind of stop or kill?), and that we likely need to use "normal mode" first followed by "stop mode" to clean up.

(I'm keen to sort this out after discovering recently that I had misunderstood exactly what these "modes" do myself, for ages. I guess I should have read the fine print more closely, but descriptive terminology is better).
author_association: MEMBER
user: login: oliver-sanders
created_at: 2025-03-19T09:44:28Z
body: @hjoliver, if you still have qualms with the pre-existing terminology, please raise a PR to master after this bugfix.
author_association: MEMBER
user: login: hjoliver
created_at: 2025-03-20T05:34:26Z
body: My opinion - this is not exactly a time-critical fix (it's been broken for years and evidently no one noticed) and we often fix closely related bits, terminology or otherwise, when other changes directly touch them - as here.

However, it seems you're not going to budge, and this isn't important enough to keep arguing forever, so fine, I'll punt it to a follow-up PR!
author_association: MEMBER