Repository: daos-stack/daos
Pull request #16261: DAOS-15947 docs: Update admin guide dmg system command syntax
Author: login: tanabarr
Labels: name: control-plane
description: work on the management infrastructure of the DAOS Control Plane
name: priority
description: Ticket has high priority (automatically managed)
name: doc-only
description: Changes only affect documentation, not code
Created: 2025-04-15T13:44:15Z | Merged: 2025-04-28T13:44:56Z

Documentation updates relating to:
- New commands `dmg system drain|reintegrate`
- New command syntax for multiple --ranks
  `dmg pool drain|reintegrate|exclude –ranks 1-9,11-19`
- New command flag `dmg storage format --replace`

Doc-only: true

### Steps for the author:

* [ ] Commit message follows the [guidelines](https://daosio.atlassian.net/wiki/spaces/DC/pages/11133911069/Commit+Comments).
* [ ] Appropriate [Features or Test-tag](https://daosio.atlassian.net/wiki/spaces/DC/pages/10984259629/Test+Tags) pragmas were used.
* [ ] Appropriate [Functional Test Stages](https://daosio.atlassian.net/wiki/spaces/DC/pages/12147556353/CI+Functional+Test+Stages) were run.
* [ ] At least two positive code reviews including at least one code owner from each category referenced in the PR.
* [ ] Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

#### After all prior steps are complete:
* [ ] Gatekeeper requested (daos-gatekeeper added as a reviewer).



Diff:
diff --git a/docs/admin/administration.md b/docs/admin/administration.md
index ad5778a9eae..81da19765e7 100644
--- a/docs/admin/administration.md
+++ b/docs/admin/administration.md
@@ -1029,6 +1029,23 @@ DAOS I/O Engines will be started, and all DAOS pools will have been removed.
     Then restart DAOS Servers and format.
 
 
+### Storage Format Replace
+
+If storage metadata for a rank is lost, for example after losing PMem contents after NVDIMM failure,
+storage for that rank will need to be formatted and rank metadata regenerated. If other hardware on
+the storage server has not changed the old rank can be "reused" by formatting using the
+`dmg storage format --replace` option.
+
+An examples workflow would be:
+- `daos_server` is running and PMem NVDIMM fails causing an engine to enter excluded state.
+- `daos_server` is stopped, storage server powered down, faulty PMem NVDIMM is replaced.
+- After powering up storage server, `daos_server scm prepare` command is used to repair PMem.
+- Storage server is rebooted after running `daos_server scm prepare` and command is run again.
+- Now PMem is intact, clear with `wipefs -a /dev/pmemX` where "X" refers to the repaired PMem ID.
+- `daos_server` can be started again. On start-up repaired engine prompts for "SCM format required".
+- Running `dmg storage format` now will create new "Joined" rank, instead run with `--replace` flag.
+- Formatted engine will join using the existing (old) rank which is mapped to the engine's hardware.
+
 ### System Erase
 
 To erase the DAOS sorage configuration, the `dmg system erase`
diff --git a/docs/admin/pool_operations.md b/docs/admin/pool_operations.md
index 5c72d1198d9..216fdd1fe8b 100644
--- a/docs/admin/pool_operations.md
+++ b/docs/admin/pool_operations.md
@@ -1219,20 +1219,30 @@ surviving engine.
 
 ### Manual Exclusion
 
-An operator can exclude one or more engines or targets from a specific DAOS pool
-using the rank the target resides, as well as the target idx on that rank.
-If a target idx list is not provided, all targets on the engine rank will be excluded.
+An operator can exclude one or more engines or targets from a specific DAOS pool using the rank(s)
+where the target(s) reside, as well as the target index(es) on the rank(s). If a target idx list is
+not provided, all selected targets on the engine rank(s) will be excluded.
 
-To exclude a target from a pool:
+To exclude multiple targets on a single rank from a pool:
 
 ```bash
-$ dmg pool exclude --rank=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+$ dmg pool exclude --ranks=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
 
-The pool target exclude command accepts 2 parameters:
+To exclude multiple targets on multiple ranks from a pool:
 
-* The engine rank of the target(s) to be excluded.
-* The target indices of the targets to be excluded from that engine rank (optional).
+```bash
+$ dmg pool exclude --ranks=${rank_range_list} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+```
+
+The pool target exclude command accepts 4 parameters:
+
+* The label or UUID of the pool that the targets will be excluded.
+* The engine rank(s) of the target(s) to be excluded. This can be a single integer or a list of
+  comma-separated ranges e.g. "1" or "1-9,10,12-19".
+* The target indices of the targets to be excluded from each specified engine rank (optional).
+* The force flag which can be used to override RF check as documented below (optional, use with
+  caution).
 
 Upon successful manual exclusion, the self-healing mechanism will be triggered
 to restore redundancy on the remaining engines.
@@ -1244,6 +1254,36 @@ to restore redundancy on the remaining engines.
     the --force option. Please note that forcing the operation may result in data loss,
     and it is strongly recommended to verify the RF status before proceeding.
 
+#### System Exclude
+
+To exclude ranks or hosts from all pools that they belong to, the 'dmg system exclude'
+command can be used. The command takes either a host-set or rank-set. Requesting a
+host-set excludes all ranks on selected hosts.
+
+To exclude a set of hosts from all pools:
+
+```Bash
+$ dmg system exclude --rank-hosts foo-[001-100]
+```
+
+To exclude a set of ranks from all pools:
+
+```Bash
+$ dmg system exclude --ranks 1-100
+```
+
+Selected ranks will be set to `AdminExcluded` state in system membership (as shown in output of
+`dmg system query` command). This will exclude all rank targets from all pools and render the ranks
+unusable in pools until `dmg system clear-exclude --ranks=${rank_range_list}` is run to return the
+ranks to the previous (normally `Excluded`) state. Then the engine can be restarted to rejoin the
+system and will enter the `Joined` state and ready to be reintegrated into pools.
+
+Engine state can be queryed via the system membership using:
+
+```bash
+$ dmg system query
+```
+
 ### Drain
 
 Alternatively, when an operator would like to remove one or more engines or
@@ -1258,16 +1298,28 @@ data would not be integrated into a rebuild and would be lost.
 Drain operation is not allowed if there are other ongoing rebuild operations, otherwise
 it will return -DER_BUSY.
 
-To drain a target from a pool:
+An operator can drain one or more engines or targets from a specific DAOS pool using the rank(s)
+where the target(s) reside, as well as the target index(es) on the rank(s). If a target idx list is
+not provided, all selected targets on the engine rank(s) will be drained.
+
+To drain multiple targets on a single rank from a pool:
 
 ```bash
-$ dmg pool drain --rank=${rank} --target-idx=${idx1},${idx2},${idx3} $DAOS_POOL
+$ dmg pool drain --ranks=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
 
-The pool target drain command accepts 2 parameters:
+To drain multiple targets on multiple ranks from a pool:
 
-* The engine rank of the target(s) to be drained.
-* The target indices of the targets to be drained from that engine rank (optional).
+```bash
+$ dmg pool drain --ranks=${rank_range_list} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+```
+
+The pool target drain command accepts 3 parameters:
+
+* The label or UUID of the pool that the targets to be drained.
+* The engine rank(s) of the target(s) to be drained. This can be a single integer or a list of
+  comma-separated ranges e.g. "1" or "1-9,10,12-19".
+* The target indices of the targets to be drained from each specified engine rank (optional).
 
 #### System Drain
 
@@ -1296,15 +1348,28 @@ supplying a target idx list, or reintegrate an entire engine rank by omitting th
 Reintegrate operation is not allowed if there are other ongoing rebuild operations,
 otherwise it will return -DER_BUSY.
 
+An operator can reintegrate one or more engines or targets from a specific DAOS pool using the
+rank(s) where the target(s) reside, as well as the target index(es) on the rank(s). If a target idx
+list is not provided, all selected targets on the engine rank(s) will be reintegrated.
+
+To reintegrate multiple targets on a single rank from a pool:
+
+```bash
+$ dmg pool reintegrate --ranks=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
-$ dmg pool reintegrate $DAOS_POOL --rank=${rank} --target-idx=${idx1},${idx2},${idx3}
+
+To reintegrate multiple targets on multiple ranks from a pool:
+
+```bash
+$ dmg pool reintegrate --ranks=${rank_range_list} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
 
-The pool reintegrate command accepts 3 parameters:
+The pool target reintegrate command accepts 3 parameters:
 
 * The label or UUID of the pool that the targets will be reintegrated into.
-* The engine rank of the affected targets.
-* The target indices of the targets to be reintegrated on that engine rank (optional).
+* The engine rank(s) of the target(s) to be reintegrated. This can be a single integer or a list of
+  comma-separated ranges e.g. "1" or "1-9,10,12-19".
+* The target indices of the targets to be reintegrated from each specified engine rank (optional).
 
 When rebuild is triggered it will list the operations and their related engines/targets
 by their engine rank and target index.
@@ -1320,7 +1385,7 @@ Target (rank 5 idx 1) is down.
 These should be the same values used when reintegrating the targets.
 
 ```
-$ dmg pool reintegrate $DAOS_POOL --rank=5 --target-idx=0,1
+$ dmg pool reintegrate $DAOS_POOL --ranks=5 --target-idx=0,1
 ```
 
 !!! warning
@@ -1328,9 +1393,26 @@ $ dmg pool reintegrate $DAOS_POOL --rank=5 --target-idx=0,1
     pool, there is currently no way to list the targets that have actually
     been disabled. As a result, it is recommended for now to try to reintegrate
     all engine ranks one after the other via `for i in seq $NR_RANKs; do dmg
-    pool reintegrate --rank=$i; done`. This limitation will be addressed in the
+    pool reintegrate --ranks=$i; done`. This limitation will be addressed in the
     next release.
 
+#### System Reintegrate
+
+To reintegrate ranks or hosts from all pools that they belong to, the 'dmg system reintegrate'
+command can be used. The command takes either a host-set or rank-set. Selecting a host-set reintegrates all ranks on selected hosts.
+
+To reintegrate a set of hosts from all pools:
+
+```Bash
+$ dmg system reintegrate --rank-hosts foo-[001-100]
+```
+
+To reintegrate a set of ranks from all pools:
+
+```Bash
+$ dmg system reintegrate --ranks 1-100
+```
+
 ## Pool Extension
 
 ### Addition & Space Rebalancing


Reviews:
user: login: knard38
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:16:57Z
user: login: kjacque
body: @knard38's comments about the copy/paste type errors need to be fixed for landing.

As far as my comments: The punctuation typo is something I'd like fixed for grammar. The other two are suggestions I wouldn't block on. Take or leave them as you like.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T18:32:59Z
user: login: tanabarr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:30:19Z
user: login: tanabarr
body: Review comments from kjacque and knard38 addressed, @johannlombardi @Michael-Hennecke do you have any further change requests?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T11:35:25Z
user: login: knard38
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T12:04:42Z
user: login: kjacque
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T17:10:30Z

Review comments:
diff_hunk: @@ -1258,16 +1291,28 @@ data would not be integrated into a rebuild and would be lost.
 Drain operation is not allowed if there are other ongoing rebuild operations, otherwise
 it will return -DER_BUSY.
 
-To drain a target from a pool:
+An operator can drain one or more engines or targets from a specific DAOS pool using the rank(s)
+where the target(s) reside, as well as the target index(es) on the rank(s). If a target idx list is
+not provided, all selected targets on the engine rank(s) will be drained.
+
+To drain multiple targets on a single rank from a pool:
+
+```bash
+$ dmg pool drain --ranks=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+```
+
+To drain multiple targets on multiple ranks from a pool:
 
 ```bash
-$ dmg pool drain --rank=${rank} --target-idx=${idx1},${idx2},${idx3} $DAOS_POOL
+$ dmg pool drain --ranks=${rank_range_list} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
 
-The pool target drain command accepts 2 parameters:
+The pool target reintegrate command accepts 3 parameters:
 
-* The engine rank of the target(s) to be drained.
-* The target indices of the targets to be drained from that engine rank (optional).
+* The label or UUID of the pool that the targets will be reintegrated into.
path: docs/admin/pool_operations.md
user: login: knard38
body: ```suggestion
* The label or UUID of the pool that the targets to be drained.
```
created_at: 2025-04-15T15:05:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1258,16 +1291,28 @@ data would not be integrated into a rebuild and would be lost.
 Drain operation is not allowed if there are other ongoing rebuild operations, otherwise
 it will return -DER_BUSY.
 
-To drain a target from a pool:
+An operator can drain one or more engines or targets from a specific DAOS pool using the rank(s)
+where the target(s) reside, as well as the target index(es) on the rank(s). If a target idx list is
+not provided, all selected targets on the engine rank(s) will be drained.
+
+To drain multiple targets on a single rank from a pool:
+
+```bash
+$ dmg pool drain --ranks=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+```
+
+To drain multiple targets on multiple ranks from a pool:
 
 ```bash
-$ dmg pool drain --rank=${rank} --target-idx=${idx1},${idx2},${idx3} $DAOS_POOL
+$ dmg pool drain --ranks=${rank_range_list} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
 
-The pool target drain command accepts 2 parameters:
+The pool target reintegrate command accepts 3 parameters:
path: docs/admin/pool_operations.md
user: login: knard38
body: ```suggestion
The pool target drain command accepts 3 parameters:
```
created_at: 2025-04-15T15:05:41Z
author_association: CONTRIBUTOR
path: docs/admin/pool_operations.md
user: login: knard38
body: at line 1286 of this file the sentence
`or target until after the rebuild is complete.`
should probably replace by
`or target until the rebuild is complete.`
created_at: 2025-04-15T15:07:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1029,6 +1029,13 @@ DAOS I/O Engines will be started, and all DAOS pools will have been removed.
     Then restart DAOS Servers and format.
 
 
+### Storage Format Replace
+
+If storage metadata for a rank is lost, for example after losing PMem contents after NVDIMM failure.
+Storage for that rank will need to be formatted and rank metadata regenerated. If other hardware on
+the storage server has not changed the old rank can be "reused" by formatting using the
+`dmg storage format --replace` option.
+
path: docs/admin/administration.md
user: login: knard38
body: A small example of usage could be helpful
created_at: 2025-04-15T15:08:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1219,20 +1219,30 @@ surviving engine.
 
 ### Manual Exclusion
 
-An operator can exclude one or more engines or targets from a specific DAOS pool
-using the rank the target resides, as well as the target idx on that rank.
-If a target idx list is not provided, all targets on the engine rank will be excluded.
+An operator can exclude one or more engines or targets from a specific DAOS pool using the rank(s)
+where the target(s) reside, as well as the target index(es) on the rank(s). If a target idx list is
+not provided, all selected targets on the engine rank(s) will be excluded.
 
-To exclude a target from a pool:
+To exclude multiple targets on a single rank from a pool:
 
 ```bash
-$ dmg pool exclude --rank=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+$ dmg pool exclude --ranks=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
 
-The pool target exclude command accepts 2 parameters:
+To exclude multiple targets on multiple ranks from a pool:
 
-* The engine rank of the target(s) to be excluded.
-* The target indices of the targets to be excluded from that engine rank (optional).
+```bash
+$ dmg pool exclude --ranks=${rank_range_list} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+```
+
+The pool target reintegrate command accepts 4 parameters:
path: docs/admin/pool_operations.md
user: login: knard38
body: ```suggestion
The pool target exclude command accepts 4 parameters:
```
created_at: 2025-04-15T15:10:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1219,20 +1219,30 @@ surviving engine.
 
 ### Manual Exclusion
 
-An operator can exclude one or more engines or targets from a specific DAOS pool
-using the rank the target resides, as well as the target idx on that rank.
-If a target idx list is not provided, all targets on the engine rank will be excluded.
+An operator can exclude one or more engines or targets from a specific DAOS pool using the rank(s)
+where the target(s) reside, as well as the target index(es) on the rank(s). If a target idx list is
+not provided, all selected targets on the engine rank(s) will be excluded.
 
-To exclude a target from a pool:
+To exclude multiple targets on a single rank from a pool:
 
 ```bash
-$ dmg pool exclude --rank=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+$ dmg pool exclude --ranks=${rank} --target-idx=${idx1},${idx2},${idx3} <pool_label>
 ```
 
-The pool target exclude command accepts 2 parameters:
+To exclude multiple targets on multiple ranks from a pool:
 
-* The engine rank of the target(s) to be excluded.
-* The target indices of the targets to be excluded from that engine rank (optional).
+```bash
+$ dmg pool exclude --ranks=${rank_range_list} --target-idx=${idx1},${idx2},${idx3} <pool_label>
+```
+
+The pool target reintegrate command accepts 4 parameters:
+
+* The label or UUID of the pool that the targets will be reintegrated into.
path: docs/admin/pool_operations.md
user: login: knard38
body: ```suggestion
* The label or UUID of the pool that the targets will be excluded.
```
created_at: 2025-04-15T15:11:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1244,6 +1254,29 @@ to restore redundancy on the remaining engines.
     the --force option. Please note that forcing the operation may result in data loss,
     and it is strongly recommended to verify the RF status before proceeding.
 
+#### System Exclude
+
+To exclude ranks or hosts from all pools that they belong to, the 'dmg system exclude'
+command can be used. The command takes either a host-set or rank-set:
+
+To exclude a set of hosts from all pools (excludes all ranks on selected hosts):
+
+```Bash
+$ dmg system exclude --rank-hosts foo-[001-100]
+```
+
+To exclude a set of ranks from all pools:
+
+```Bash
+$ dmg system exclude --ranks 1-100
+```
+
+Selected ranks will be set to `AdminExcluded` state in system membership (as shown in output of
+`dmg system query` command). This will exclude all rank targets from all pools and render the ranks
+unusable in pools until `dmg system clear-exclude --ranks=${rank_range_list}` is run to return the
+ranks to the previous (normally `Excluded`) state. Then the engine can be restarted to rejoin the
+system and will enter the `Joined` state and ready to be reintegrated into pools.
path: docs/admin/pool_operations.md
user: login: knard38
body: It could be indicated that the engine will rejoin automatically the system and also indicate how much time it took in general.  Could be helpful to add an example of dmg command usage for checking if the engines have enter the `Joined` state.
created_at: 2025-04-15T15:13:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1029,6 +1029,13 @@ DAOS I/O Engines will be started, and all DAOS pools will have been removed.
     Then restart DAOS Servers and format.
 
 
+### Storage Format Replace
+
+If storage metadata for a rank is lost, for example after losing PMem contents after NVDIMM failure.
+Storage for that rank will need to be formatted and rank metadata regenerated. If other hardware on
path: docs/admin/administration.md
user: login: kjacque
body: minor copyedit

```suggestion
If storage metadata for a rank is lost, for example after losing PMem contents after NVDIMM failure,
storage for that rank will need to be formatted and rank metadata regenerated. If other hardware on
```
created_at: 2025-04-16T16:35:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1244,6 +1254,29 @@ to restore redundancy on the remaining engines.
     the --force option. Please note that forcing the operation may result in data loss,
     and it is strongly recommended to verify the RF status before proceeding.
 
+#### System Exclude
+
+To exclude ranks or hosts from all pools that they belong to, the 'dmg system exclude'
+command can be used. The command takes either a host-set or rank-set:
+
+To exclude a set of hosts from all pools (excludes all ranks on selected hosts):
path: docs/admin/pool_operations.md
user: login: kjacque
body: minor clarity suggestion
```suggestion
To exclude ranks or hosts from all pools that they belong to, the 'dmg system exclude'
command can be used. The command takes either a host-set or rank-set. Requesting a
host-set excludes all ranks on selected hosts.

To exclude a set of hosts from all pools:
```
created_at: 2025-04-16T18:22:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1320,17 +1378,34 @@ Target (rank 5 idx 1) is down.
 These should be the same values used when reintegrating the targets.
 
 ```
-$ dmg pool reintegrate $DAOS_POOL --rank=5 --target-idx=0,1
+$ dmg pool reintegrate $DAOS_POOL --ranks=5 --target-idx=0,1
 ```
 
 !!! warning
     While dmg pool query and list show how many targets are disabled for each
     pool, there is currently no way to list the targets that have actually
     been disabled. As a result, it is recommended for now to try to reintegrate
     all engine ranks one after the other via `for i in seq $NR_RANKs; do dmg
-    pool reintegrate --rank=$i; done`. This limitation will be addressed in the
+    pool reintegrate --ranks=$i; done`. This limitation will be addressed in the
     next release.
 
+#### System Reintegrate
+
+To reintegrate ranks or hosts from all pools that they belong to, the 'dmg system reintegrate'
+command can be used. The command takes either a host-set or rank-set:
+
+To reintegrate a set of hosts from all pools (reintegrates all ranks on selected hosts):
path: docs/admin/pool_operations.md
user: login: kjacque
body: Suggestion similar to the exclude case:
```suggestion
command can be used. The command takes either a host-set or rank-set. Selecting a host-set reintegrates all ranks on selected hosts.

To reintegrate a set of hosts from all pools:
```
created_at: 2025-04-16T18:25:32Z
author_association: CONTRIBUTOR
path: docs/admin/pool_operations.md
user: login: tanabarr
body: I think "after" actually reads better.
created_at: 2025-04-17T23:30:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1029,6 +1029,13 @@ DAOS I/O Engines will be started, and all DAOS pools will have been removed.
     Then restart DAOS Servers and format.
 
 
+### Storage Format Replace
+
+If storage metadata for a rank is lost, for example after losing PMem contents after NVDIMM failure.
+Storage for that rank will need to be formatted and rank metadata regenerated. If other hardware on
+the storage server has not changed the old rank can be "reused" by formatting using the
+`dmg storage format --replace` option.
+
path: docs/admin/administration.md
user: login: tanabarr
body: done
created_at: 2025-04-18T11:19:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1244,6 +1254,29 @@ to restore redundancy on the remaining engines.
     the --force option. Please note that forcing the operation may result in data loss,
     and it is strongly recommended to verify the RF status before proceeding.
 
+#### System Exclude
+
+To exclude ranks or hosts from all pools that they belong to, the 'dmg system exclude'
+command can be used. The command takes either a host-set or rank-set:
+
+To exclude a set of hosts from all pools (excludes all ranks on selected hosts):
+
+```Bash
+$ dmg system exclude --rank-hosts foo-[001-100]
+```
+
+To exclude a set of ranks from all pools:
+
+```Bash
+$ dmg system exclude --ranks 1-100
+```
+
+Selected ranks will be set to `AdminExcluded` state in system membership (as shown in output of
+`dmg system query` command). This will exclude all rank targets from all pools and render the ranks
+unusable in pools until `dmg system clear-exclude --ranks=${rank_range_list}` is run to return the
+ranks to the previous (normally `Excluded`) state. Then the engine can be restarted to rejoin the
+system and will enter the `Joined` state and ready to be reintegrated into pools.
path: docs/admin/pool_operations.md
user: login: tanabarr
body: time is not easy to determine as it depends on the size of the cluster and lots of other factors. I can add the syntax for system query. Done
created_at: 2025-04-18T11:22:27Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-15T13:44:35Z
body: Ticket title is 'Issues over pool lifetime when rank MD is reformatted'
Status is 'Resolved'
Labels: 'ALCF,LRZ,google-cloud-daos,usability'
Job should run at elevated priority (1)
Errors are Unknown component
https://daosio.atlassian.net/browse/DAOS-15947
<!-- thollander/actions-comment-pull-request "jira_query_message" -->
author_association: NONE
user: login: tanabarr
created_at: 2025-04-22T11:20:41Z
body: Gatekeeper requested, documentation change only.
author_association: CONTRIBUTOR