Repository: minio/minio (language: go)

Issue:
Rebalance never releases the lock
Hi,
After execution of `mc admin rebalance start`, our MinIO Deployment can't decommission any pools. Even after `mc admin rebalance stop` or after rebalance mark pool from which it moved data as `Completed`

## Expected Behavior

Rebalance can be stopped, or after moving all data release lock, so I can execute decommission.

## Current Behavior

Rebalance got stuck, and execution `mc admin rebalance stop` returns `success` but does not stop after moving all data. Two pools with the least amount of data at the time of execution `mc admin rebalance start` report `None` as rebalance status.

## Possible Solution

It is possible that this issue can be solved by actions described in [issue #19179](https://github.com/minio/minio/issues/19179), but it looks like a way dangerous solution

## Steps to Reproduce (for bugs)

1. Add a new pool to deployment.
2. Execute `mc admin rebalance start`.
3. Wait till it finishes and release the lock.

`mc admin rebalance status`

```txt
Per-pool usage:
┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│ Pool-0 │ Pool-1 │ Pool-2 │ Pool-3 │ Pool-4 │ Pool-5 │ Pool-6 │ Pool-7 │ Pool-8 │
│ 15.68% │ 29.08% │ 28.81% │ 28.87% │ 29.04% │ 28.62% │ 28.84% │ 34.52% │ 11.44% │
└────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘
Summary:
Data: 24 TiB (5178071 objects, 5178079 versions)
Time: 117h43m52.383579294s (0s to completion)
```

`mc admin rebalance status --json`

```json
{"ID":"73Nj6txfv3rWPPRCieFxgL","stoppedAt":"2024-06-24T14:50:49.727837335Z","pools":[{"id":0,"status":"None","used":0.15681572383998796,"progress":{"objects":0,"versions":0,"bytes":0,"bucket":"","object":"","elapsed":0,"eta":0}},{"id":1,"status":"Completed","used":0.29086223474269113,"progress":{"objects":863845,"versions":863845,"bytes":4489464067996,"bucket":"","object":"","elapsed":384548959118361,"eta":0}},{"id":2,"status":"Completed","used":0.2880991389168018,"progress":{"objects":861563,"versions":861564,"bytes":4476320868232,"bucket":"","object":"","elapsed":384317907296420,"eta":0}},{"id":3,"status":"Completed","used":0.2887188148361973,"progress":{"objects":860038,"versions":860040,"bytes":4394931259097,"bucket":"","object":"","elapsed":373384961476932,"eta":0}},{"id":4,"status":"Completed","used":0.2904467819755646,"progress":{"objects":862736,"versions":862736,"bytes":4548684156731,"bucket":"","object":"","elapsed":398713964110767,"eta":0}},{"id":5,"status":"Completed","used":0.28622529910934746,"progress":{"objects":868107,"versions":868109,"bytes":4498917224451,"bucket":"","object":"","elapsed":423832383579294,"eta":0}},{"id":6,"status":"Completed","used":0.28835005893056187,"progress":{"objects":861782,"versions":861785,"bytes":4440767925423,"bucket":"","object":"","elapsed":380046410064450,"eta":0}},{"id":7,"status":"Completed","used":0.34528082629050644,"progress":{"objects":0,"versions":0,"bytes":0,"bucket":"","object":"","elapsed":13972320292,"eta":0}},{"id":8,"status":"None","used":0.11441227779999562,"progress":{"objects":0,"versions":0,"bytes":0,"bucket":"","object":"","elapsed":0,"eta":0}}]}
```

Logs related to `rebalance` for one week look like this:

```
2024-06-25 18:07:28.928	
{"deploymentid":"27a2a1c5-5ef0-42e9-b596-386a413e5bec","level":"EVENT","time":"2024-06-25T15:07:28.914432516Z","api":{"name":"SYSTEM.rebalance","args":{}},"message":"Pool 6 rebalancing is done"}

2024-06-20 21:44:07.943	
{"deploymentid":"27a2a1c5-5ef0-42e9-b596-386a413e5bec","level":"ERROR","time":"2024-06-20T18:44:07.942884362Z","api":{"name":"SYSTEM.rebalance","args":{}},"error":{"message":"Storage resources are insufficient for the read operation {REDACTED}(cmd.InsufficientReadQuorum)","source":["internal/logger/logger.go:268:logger.LogIf()","cmd/logging.go:34:cmd.rebalanceLogIf()","cmd/erasure-server-pool-rebalance.go:700:cmd.(*erasureServerPools).rebalanceBucket.func2()"],"variables":{"objectLocation":{"name":" {REDACTED}","poolId":1,"setId":1,"drives":["http://{REDACTED}:9000/data/1","http://{REDACTED}:9000/data/2","http://{REDACTED}:9000/data/3","http://{REDACTED}:9000/data/4","http://{REDACTED}:9000/data/5","http://{REDACTED}:9000/data/6","http://{REDACTED}:9000/data/7","http://{REDACTED}:9000/data/8","http://{REDACTED}:9000/data/9","http://{REDACTED}:9000/data/10","http://{REDACTED}:9000/data/11","http://{REDACTED}:9000/data/12"]}}}}
	
2024-06-20 20:23:36.586	
{"deploymentid":"27a2a1c5-5ef0-42e9-b596-386a413e5bec","level":"EVENT","time":"2024-06-20T17:23:36.586028284Z","api":{"name":"SYSTEM.rebalance","args":{}},"message":"Pool 6 rebalancing is started"}
```

## Context

Can't decommission any pools in the MinIO Deployment.

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* MinIO image used: `quay.io/minio/minio:RELEASE.2024-06-11T03-13-30Z`
* Operating System and version: Red Hat Enterprise Linux release 8.10 (Ootpa) Kernel 4.18.0-553.el8_10.x86_64
* Server setup and configuration: Kubernetes 1.30
Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: minio
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: minio
    app.kubernetes.io/version: 0.0.60
    argocd.argoproj.io/instance: minio
    helm.sh/chart: minio-0.1.5
  name: minio
  namespace: minio
spec:
  progressDeadlineSeconds: 600
  replicas: 15
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/component: server
      app.kubernetes.io/instance: minio
      app.kubernetes.io/name: minio
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        checksum/wait-etcd: 89bdaf02aad1b552641af6ced3c9abc331b6bb6deaa8bbaf2d2e61e779018786
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: server
        app.kubernetes.io/instance: minio
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: minio
        app.kubernetes.io/version: 0.0.60
        helm.sh/chart: minio-0.1.5
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                      - {REDACTED}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: server
                  app.kubernetes.io/instance: minio
                  app.kubernetes.io/name: minio
              namespaces:
                - minio
              topologyKey: kubernetes.io/hostname
      containers:
        - command:
            - minio
            - server
            - '--address'
            - '0.0.0.0:9000'
            - '--console-address'
            - '0.0.0.0:9001'
            - '--json'
            - 'http://{REDACTED}:9000/data/{1...12}'
            - 'http://{REDACTED}:9000/data/{1...12}'
            - 'http://{REDACTED}:9000/data/{1...12}'
            - 'http://{REDACTED}:9000/data/{1...12}'
            - 'http://{REDACTED}:9000/data/{1...12}'
            - 'http://{REDACTED}:9000/data/{1...12}'
            - 'http://{REDACTED}:9000/data/{1...12}'
            - 'http://minio-p1-s{1...4}:9000/data/{1...12}'
            - 'http://minio-p2-s{1...4}:9000/data/{1...12}'
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: POD_UID
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.uid
            - name: MINIO_IDENTITY_OPENID_ENABLE
              value: 'on'
            - name: MINIO_IDENTITY_OPENID_CONFIG_URL
              value: >-
                https://{REDACTED}/.well-known/openid-configuration
            - name: MINIO_IDENTITY_OPENID_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: clientId
                  name: minio-oidc-credentials
            - name: MINIO_IDENTITY_OPENID_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: clientSecret
                  name: minio-oidc-credentials
            - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
              value: 'https://minio.{REDACTED}/oauth_callback'
            - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
              value: policy
            - name: MINIO_IDENTITY_OPENID_SCOPES
              value: 'openid,profile,email'
            - name: MINIO_PROMETHEUS_AUTH_TYPE
              value: public
            - name: MINIO_PROMETHEUS_JOB_ID
              value: minio
            - name: MINIO_PROMETHEUS_URL
              value: 'http://prometheus-stack-kube-prom-prometheus.monitoring.svc.cluster.local:9090'
            - name: MINIO_SERVER_URL
              value: 'https://minio-api.{REDACTED}'
            - name: MINIO_BROWSER_REDIRECT_URL
              value: 'https://minio.{REDACTED}'
            - name: MINIO_CALLHOME_ENABLE
              value: 'off'
            - name: MINIO_CALLHOME_FREQUENCY
              value: 24h
            - name: MINIO_ROOT_USER
              valueFrom:
                secretKeyRef:
                  key: MINIO_ROOT_USER
                  name: minio-credentials
            - name: MINIO_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: MINIO_ROOT_PASSWORD
                  name: minio-credentials
            - name: MINIO_KMS_KES_ENDPOINT
              value: 'https://minio-kes-server.minio.svc.cluster.local:7373'
            - name: MINIO_KMS_KES_CERT_FILE
              value: /opt/kes/certs/kes-server-minio.crt
            - name: MINIO_KMS_KES_KEY_FILE
              value: /opt/kes/certs/kes-server-minio.key
            - name: MINIO_KMS_KES_CAPATH
              value: /root/.minio/certs/CAs/kes-server.crt
            - name: MINIO_KMS_KES_KEY_NAME
              value: minio-main
            - name: MINIO_AUDIT_WEBHOOK_ENABLE_loki
              value: 'on'
            - name: MINIO_AUDIT_WEBHOOK_ENDPOINT_loki
              value: http://loki.monitoring.svc.cluster.local:8080/v1/push
            - name: MINIO_AUDIT_WEBHOOK_QUEUE_SIZE_loki
              value: '128'
            - name: MINIO_ETCD_ENDPOINTS
              value: https://minio-etcd-headless.minio.svc.cluster.local:2379
            - name: MINIO_ETCD_CLIENT_CERT
              value: /run/secrets/etcd/tls.crt
            - name: MINIO_ETCD_CLIENT_CERT_KEY
              value: /run/secrets/etcd/tls.key
          image: 'quay.io/minio/minio:RELEASE.2024-06-11T03-13-30Z'
          imagePullPolicy: Always
          name: minio
          ports:
            - containerPort: 9000
              hostPort: 9000
              name: server
              protocol: TCP
            - containerPort: 9001
              hostPort: 9001
              name: console
              protocol: TCP
          resources:
            limits:
              cpu: '16'
              ephemeral-storage: 8Gi
              memory: 96Gi
            requests:
              cpu: '6'
              ephemeral-storage: 8Gi
              memory: 64Gi
          securityContext:
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /data/1
              name: data-1
            - mountPath: /data/2
              name: data-2
            - mountPath: /data/3
              name: data-3
            - mountPath: /data/4
              name: data-4
            - mountPath: /data/5
              name: data-5
            - mountPath: /data/6
              name: data-6
            - mountPath: /data/7
              name: data-7
            - mountPath: /data/8
              name: data-8
            - mountPath: /data/9
              name: data-9
            - mountPath: /data/10
              name: data-10
            - mountPath: /data/11
              name: data-11
            - mountPath: /data/12
              name: data-12
            - mountPath: /tmp
              name: tmp
            - mountPath: /etc/ssl/certs/ca-certificates.crt
              name: ca-cert-pem
              readOnly: true
            - mountPath: /root/.minio/certs/CAs/kes-server.crt
              name: kes-server-keys
              readOnly: true
              subPath: kes-server.crt
            - mountPath: /opt/kes/certs/kes-server-minio.key
              name: kes-server-keys
              readOnly: true
              subPath: kes-server-minio.key
            - mountPath: /opt/kes/certs/kes-server-minio.crt
              name: kes-server-keys
              readOnly: true
              subPath: kes-server-minio.crt
            - mountPath: /run/secrets/etcd
              name: etcd-client-tls
              readOnly: true
            - mountPath: /root/.minio/certs/CAs/etcd-ca.crt
              name: etcd-client-tls
              readOnly: true
              subPath: ca.crt
      dnsPolicy: ClusterFirstWithHostNet
      enableServiceLinks: false
      hostNetwork: true
      initContainers:
        - command:
            - dockerize
            - '-timeout'
            - 240s
            - '-wait'
            - 'tcp://minio-kes-server.minio.svc.cluster.local:7373'
            - echo
            - KES Server start detected
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: POD_UID
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.uid
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
          image: '{REDACTED}/dockerize:latest'
          imagePullPolicy: Always
          name: wait-kes-server
          resources:
            limits:
              cpu: 100m
              ephemeral-storage: 64Mi
              memory: 64Gi
            requests:
              cpu: 100m
              ephemeral-storage: 64Mi
              memory: 64Gi
          securityContext:
            readOnlyRootFilesystem: true
            runAsGroup: 65534
            runAsUser: 65534
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        - command:
            - python
            - /mnt/wait_etcd.py
          image: '{REDACTED}/python-k8s:latest'
          imagePullPolicy: Always
          name: wait-etcd
          resources:
            limits:
              cpu: 100m
              ephemeral-storage: 32Mi
              memory: 128Mi
            requests:
              cpu: 100m
              ephemeral-storage: 32Mi
              memory: 128Mi
          securityContext:
            readOnlyRootFilesystem: true
            runAsGroup: 65534
            runAsUser: 65534
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /mnt/wait_etcd.py
              name: wait-etcd
              readOnly: true
              subPath: wait_etcd.py
            - mountPath: /run/secrets/etcd
              name: etcd-client-tls
              readOnly: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: minio
      serviceAccountName: minio
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/component: server
              app.kubernetes.io/instance: minio
              app.kubernetes.io/name: minio
          maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
      volumes:
        - hostPath:
            path: /data/disk-1/minio
            type: DirectoryOrCreate
          name: data-1
        - hostPath:
            path: /data/disk-2/minio
            type: DirectoryOrCreate
          name: data-2
        - hostPath:
            path: /data/disk-3/minio
            type: DirectoryOrCreate
          name: data-3
        - hostPath:
            path: /data/disk-4/minio
            type: DirectoryOrCreate
          name: data-4
        - hostPath:
            path: /data/disk-5/minio
            type: DirectoryOrCreate
          name: data-5
        - hostPath:
            path: /data/disk-6/minio
            type: DirectoryOrCreate
          name: data-6
        - hostPath:
            path: /data/disk-7/minio
            type: DirectoryOrCreate
          name: data-7
        - hostPath:
            path: /data/disk-8/minio
            type: DirectoryOrCreate
          name: data-8
        - hostPath:
            path: /data/disk-9/minio
            type: DirectoryOrCreate
          name: data-9
        - hostPath:
            path: /data/disk-10/minio
            type: DirectoryOrCreate
          name: data-10
        - hostPath:
            path: /data/disk-11/minio
            type: DirectoryOrCreate
          name: data-11
        - hostPath:
            path: /data/disk-12/minio
            type: DirectoryOrCreate
          name: data-12
        - emptyDir:
            medium: Memory
            sizeLimit: 64Mi
          name: tmp
        - hostPath:
            path: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
            type: File
          name: ca-cert-pem
        - name: kes-server-keys
          secret:
            defaultMode: 420
            items:
              - key: kes-server-minio.key
                mode: 256
                path: kes-server-minio.key
              - key: kes-server-minio.crt
                mode: 292
                path: kes-server-minio.crt
              - key: kes-server.crt
                mode: 292
                path: kes-server.crt
            secretName: minio-kes-server-keys
        - name: etcd-client-tls
          secret:
            defaultMode: 420
            secretName: minio-etcd-client-tls
        - configMap:
            defaultMode: 292
            name: minio-server-etcd-wait
          name: wait-etcd
```



Discussion / hints:
> ```
> Per-pool usage:
> ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
> │ Pool-0 │ Pool-1 │ Pool-2 │ Pool-3 │ Pool-4 │ Pool-5 │ Pool-6 │ Pool-7 │ Pool-8 │
> │ 15.68% │ 29.08% │ 28.81% │ 28.87% │ 29.04% │ 28.62% │ 28.84% │ 34.52% │ 11.44% │
> └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘
> ```

Yet another non-sensical use of rebalance. 
Hi [Harshavardhana](https://github.com/harshavardhana),
Thank you for your reply.
I disagree that the mentioned rebalance usage is non-sensical. From our experience, after we added Pool-7, we experienced massive performance degradation because our data engineers worked mostly with new data.
We only performed a rebalance to decrease performance degradation time when we added Pool-8.
Here is the initial status right after we add Pool-8:
```
Per-pool usage:
┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│ Pool-0 │ Pool-1 │ Pool-2 │ Pool-3 │ Pool-4 │ Pool-5 │ Pool-6 │ Pool-7 │ Pool-8 │
│ 15.86% │ 38.77% │ 38.57% │ 38.57% │ 38.89% │ 38.79% │ 38.67% │ 28.77% │ 0.72%  │
└────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘
```
In our configuration, Pool-7 and Pool-8 have new, faster disks. That is why we decided to move data to newer disks. We didn't expect that rebalance would move any data to Pool-7 because it is already loaded two to three times more than any of the other pools. Pool-0 to Pool-6 have 29TiB in their configuration, while Pool-7 and Pool-8 have 131TiB. I don't know why `rebalance` moved any data to Pool-7.

The issue is that MinIO is not stopping rebalance, not via command `mc admin rebalance stop`, not after it is completed moving data. Is there any way to stop rebalance without deleting meta files directly on the disks?
> I disagree that the mentioned rebalance usage is non-sensical. From our experience, after we added Pool-7, we experienced massive performance degradation because our data engineers worked mostly with new data

Yes, that tells me you have a terrible scaling model; you haven't followed anything we documented. 

- Having a nonsensical expansion of single node EC sets into multi-pool 
- Then, on top of that, try to rebalance because you are losing on IOPs on a single pool
- then realizing that you need HA, now you moved into a 4-node model

I don't even know where to begin. 

> The issue is that MinIO is not stopping rebalance, not via command `mc admin rebalance stop`, not after it is completed moving data. Is there any way to stop rebalance without deleting meta files directly on the disks?

We can fix that bug, but it doesn't make your setup kosher by any means. It is just a bunch of incorrectly followed practices, not knowing what you want from a deployment. 

Rebalance is never necessary unless you have poorly scaled your cluster. 
Hi [Harshavardhana](https://github.com/harshavardhana),
Thank you for your reply.
Sadly, single-node pools are what we inherited when our team began to work on this project. That is why we are adding new pools. From a business perspective, they do not care that pools are created with one server but they care about performance.

What is the approximate time when the bug could be fixed? I mentioned [issue](https://github.com/minio/minio/issues/19179), which reported about this issue in March.

What else can we do except delete files directly from drives to unlock our deployment?

If, after adding a new pool, we see in our monitoring system that newly added servers are under a much higher load than other servers. What else should we do, except rebalance and wait, to fix performance degradation?

> they do not care that pools are created with one server but they care about performance

That is a direct contradiction. Multiple pools are not conducive to the best performance.

We will look at the issue, but we cannot provide any timeline. I suggest you look into our [subscriptions](https://min.io/pricing) for an architecture/performance review.

Fix patch:
diff --git a/cmd/admin-handlers-pools.go b/cmd/admin-handlers-pools.go
index cd965582c0df7..5b3ac615143be 100644
--- a/cmd/admin-handlers-pools.go
+++ b/cmd/admin-handlers-pools.go
@@ -374,6 +374,7 @@ func (a adminAPIHandlers) RebalanceStop(w http.ResponseWriter, r *http.Request)
 	globalNotificationSys.StopRebalance(r.Context())
 	writeSuccessResponseHeadersOnly(w)
 	adminLogIf(ctx, pools.saveRebalanceStats(GlobalContext, 0, rebalSaveStoppedAt))
+	globalNotificationSys.LoadRebalanceMeta(ctx, false)
 }
 
 func proxyDecommissionRequest(ctx context.Context, defaultEndPoint Endpoint, w http.ResponseWriter, r *http.Request) (proxy bool) {
diff --git a/cmd/erasure-server-pool-rebalance.go b/cmd/erasure-server-pool-rebalance.go
index cce7de9bf575e..5f4c8033347f6 100644
--- a/cmd/erasure-server-pool-rebalance.go
+++ b/cmd/erasure-server-pool-rebalance.go
@@ -350,8 +350,15 @@ func (z *erasureServerPools) IsRebalanceStarted() bool {
 	z.rebalMu.RLock()
 	defer z.rebalMu.RUnlock()
 
-	if r := z.rebalMeta; r != nil {
-		if r.StoppedAt.IsZero() {
+	r := z.rebalMeta
+	if r == nil {
+		return false
+	}
+	if !r.StoppedAt.IsZero() {
+		return false
+	}
+	for _, ps := range r.PoolStats {
+		if ps.Participating && ps.Info.Status != rebalCompleted {
 			return true
 		}
 	}


Test patch: