Repository: minio/minio (language: go)

Issue:
Removal of decommissioned pool fails
Hello Friends,

i have been using MinIO on a bare metal Environment for quite some time now, and want to decommission my old pool. I followed the documentation strictly, but run into the same problem on my testing environment every time.
The decommission completes, and when i remove the old pool from the deployment and restart the new server it mentions it cannot initialize the old backend.

## Expected Behavior
As stated in the documentation:
After completing a decommission and removing the old pool from deployment, MinIO service should restart and work again.

## Current Behavior
Decommission succeeds and after removal of the deployment it cannot restart with the remaining Server Pool, as it demands configuration of the old decommissioned Pool
`ERROR Unable to initialize backend: pool(1st) = https://minio1.example.com:9000/mnt/disk{1...8}/data is not specified, please specify on server command line`

## Steps to Reproduce (for bugs)
Create a MinIO distributed Setup and following /etc/default/minio:
```
# Remote volumes to be used for MinIO server.
MINIO_VOLUMES="https://minio1.example.com:9000/mnt/disk{1...8}/data"
# Use if you want to run MinIO on a custom port.
MINIO_OPTS="--console-address :9001"
# Root user for the server.
MINIO_ROOT_USER=minio-admin
# Root secret for the server.
MINIO_ROOT_PASSWORD=change-me
MINIO_SERVER_URL="https://minio.example.com"
```
after uploading Files on the Server, you get following output for `mc admin decommission status` 
```
┌─────┬─────────────────────────────────────────────────────────┬─────────────────────────────────┬────────┐
│ ID  │ Pools                                                   │ Capacity                        │ Status │
│ 1st │ https://minio1.example.com:9000/mnt/disk{1...8}/data    │ 8.0 GiB (used) / 80 GiB (total) │ Active │
└─────┴─────────────────────────────────────────────────────────┴─────────────────────────────────┴────────┘
```

extend the Setup with another MinIO Server:
`MINIO_VOLUMES="https://minio1.example.com:9000/mnt/disk{1...8}/data https://minio2.example.com:9000/mnt/disk{1...8}/data"`
you get following output for `mc admin decommission status example` 
```
┌─────┬─────────────────────────────────────────────────────────┬─────────────────────────────────┬────────┐
│ ID  │ Pools                                                   │ Capacity                        │ Status │
│ 1st │ https://minio1.example.com:9000/mnt/disk{1...8}/data    │ 8.0 GiB (used) / 80 GiB (total) │ Active │
│ 2nd │ https://minio2.example.com:9000/mnt/disk{1...8}/data    │ 701 MiB (used) / 80 GiB (total) │ Active │
└─────┴─────────────────────────────────────────────────────────┴─────────────────────────────────┴────────┘
```
now trigger decommission of Pool 1:
```
~ mc admin decommission start example https://minio1.example.com:9000/mnt/disk\{1...8\}/data
Decommission started successfully for `https://minio1.example.com:9000/mnt/disk{1...8}/data`.
~ mc admin decommission status example                                                         
┌─────┬─────────────────────────────────────────────────────────┬─────────────────────────────────┬──────────┐
│ ID  │ Pools                                                   │ Capacity                        │ Status   │
│ 1st │ https://minio1.example.com:9000/mnt/disk{1...8}/data    │ 8.0 GiB (used) / 80 GiB (total) │ Draining │
│ 2nd │ https://minio2.example.com:9000/mnt/disk{1...8}/data    │ 970 MiB (used) / 80 GiB (total) │ Active   │
└─────┴─────────────────────────────────────────────────────────┴─────────────────────────────────┴──────────┘
~ mc admin decommission status example https://minio1.example.com:9000/mnt/disk\{1...8\}/data
Decommissioning rate at 34 MiB/sec [6.3 GiB/80 GiB]
~ mc admin decommission status example https://minio1.example.com:9000/mnt/disk\{1...8\}/data
Decommission of pool https://minio1.example.com:9000/mnt/disk{1...8}/data is complete, you may now remove it from server command line
~ mc admin decommission status example                                                          
┌─────┬─────────────────────────────────────────────────────────┬─────────────────────────────────┬──────────┐
│ ID  │ Pools                                                   │ Capacity                        │ Status   │
│ 1st │ https://minio1.example.com:9000/mnt/disk{1...8}/data    │ 709 MiB (used) / 80 GiB (total) │ Complete │
│ 2nd │ https://minio2.example.com:9000/mnt/disk{1...8}/data    │ 8.0 GiB (used) / 80 GiB (total) │ Active   │
└─────┴─────────────────────────────────────────────────────────┴─────────────────────────────────┴──────────┘
```
now remove the old pool from server deployment:
```
# Remote volumes to be used for MinIO server.
MINIO_VOLUMES="https://minio2.example.com:9000/mnt/disk{1...8}/data"
# Use if you want to run MinIO on a custom port.
MINIO_OPTS="--console-address :9001"
# Root user for the server.
MINIO_ROOT_USER=minio-admin
# Root secret for the server.
MINIO_ROOT_PASSWORD=change-me
MINIO_SERVER_URL="https://minio.example.com"
```
`systemctl restart minio.service` now results into a crashing minio with following `journalctl -u miinio.service`
```
Apr 15 09:46:38 minio2 systemd[1]: Starting MinIO...
Apr 15 09:46:38 minio2 systemd[1]: Started MinIO.
Apr 15 09:46:38 minio2 minio[1865]: ERROR Unable to initialize backend: pool(1st) = https://minio1.example.com:9000/mnt/disk{1...8}/data is not specified, please specify on server command line
Apr 15 09:46:38 minio2 systemd[1]: minio.service: Main process exited, code=exited, status=1/FAILURE
Apr 15 09:46:38 minio2 systemd[1]: minio.service: Failed with result 'exit-code'.
Apr 15 09:46:38 minio2 systemd[1]: minio.service: Scheduled restart job, restart counter is at 5.
Apr 15 09:46:38 minio2 systemd[1]: Stopped MinIO.
Apr 15 09:46:38 minio2 systemd[1]: minio.service: Start request repeated too quickly.
Apr 15 09:46:38 minio2 systemd[1]: minio.service: Failed with result 'exit-code'.
Apr 15 09:46:38 minio2 systemd[1]: Failed to start MinIO.
```
## Context
I want to migrate all data From my old Server to new one. 

## Your Environment
* Version used (`minio --version`): RELEASE.2022-04-12T06-55-35Z
* Server setup and configuration: 2 VM with 8 extra Drives each, Ubuntu 20.04.
* Operating System and version (`uname -a`): `Linux minio2 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux`



Discussion / hints:
> Apr 15 09:46:38 minio2 minio[1865]: ERROR Unable to initialize backend: pool(1st) = https://minio1.example.com:9000/mnt/disk{1...8}/data is not specified, please specify on server command line

This looks like the `pool.bin` still references the older pool as not `Complete` fully @HannesDampft you should share the server logs here on what happened with your decommission process.

Until then we won't look at this issue.

Fix patch:
diff --git a/cmd/erasure-server-pool-decom.go b/cmd/erasure-server-pool-decom.go
index 2ff0644c3060a..a28185eae2628 100644
--- a/cmd/erasure-server-pool-decom.go
+++ b/cmd/erasure-server-pool-decom.go
@@ -900,7 +900,7 @@ func (z *erasureServerPools) DecommissionCancel(ctx context.Context, idx int) (e
 	defer z.poolMetaMutex.Unlock()
 
 	if z.poolMeta.DecommissionCancel(idx) {
-		z.decommissionCancelers[idx]() // cancel any active thread.
+		defer z.decommissionCancelers[idx]() // cancel any active thread.
 		if err = z.poolMeta.save(ctx, z.serverPools); err != nil {
 			return err
 		}
@@ -922,7 +922,7 @@ func (z *erasureServerPools) DecommissionFailed(ctx context.Context, idx int) (e
 	defer z.poolMetaMutex.Unlock()
 
 	if z.poolMeta.DecommissionFailed(idx) {
-		z.decommissionCancelers[idx]() // cancel any active thread.
+		defer z.decommissionCancelers[idx]() // cancel any active thread.
 		if err = z.poolMeta.save(ctx, z.serverPools); err != nil {
 			return err
 		}
@@ -944,7 +944,7 @@ func (z *erasureServerPools) CompleteDecommission(ctx context.Context, idx int)
 	defer z.poolMetaMutex.Unlock()
 
 	if z.poolMeta.DecommissionComplete(idx) {
-		z.decommissionCancelers[idx]() // cancel any active thread.
+		defer z.decommissionCancelers[idx]() // cancel any active thread.
 		if err = z.poolMeta.save(ctx, z.serverPools); err != nil {
 			return err
 		}


Test patch: