Repository: minio/minio (language: go)

Issue:
Rebalance causes server not initialized crash
Hi,

I just added 2 pools on my cluster and started a rebalance but now one of my original node crashes at startup.

## Expected Behavior
The node should start, or I should be able to take manual steps to fix it.

## Current Behavior
```
sidecar 2023/06/14 09:28:20 sidecar.go:48: Starting Sidecar
minio 
minio API: SYSTEM()
minio Time: 09:28:21 UTC 06/14/2023
minio Error: Server not initialized, please try again (*errors.errorString)
minio        6: internal/logger/logger.go:258:logger.LogIf()
minio        5: internal/logger/logger.go:385:logger.CriticalIf()
minio        4: cmd/bucket-object-lock.go:47:cmd.(*BucketObjectLockSys).Get()
minio        3: cmd/erasure-server-pool-rebalance.go:439:cmd.(*erasureServerPools).rebalanceBucket()
minio        2: cmd/erasure-server-pool-rebalance.go:397:cmd.(*erasureServerPools).rebalanceBuckets()
minio        1: cmd/erasure-server-pool-rebalance.go:814:cmd.(*erasureServerPools).StartRebalance.func1()
minio panic: (struct {}) 0x654e0a8
minio 
minio goroutine 382 [running]:
minio github.com/minio/minio/internal/logger.CriticalIf(...)
minio     github.com/minio/minio/internal/logger/logger.go:386
minio github.com/minio/minio/cmd.(*BucketObjectLockSys).Get(0x4f65b80?, {0xc000169200, 0x11})
minio     github.com/minio/minio/cmd/bucket-object-lock.go:47 +0x16c
minio github.com/minio/minio/cmd.(*erasureServerPools).rebalanceBucket(0xc0020524d0, {0x4f65b80, 0xc002d08f80}, {0xc000169200, 0x11}, 0x0)
minio     github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:439 +0x156
minio github.com/minio/minio/cmd.(*erasureServerPools).rebalanceBuckets(0xc0020524d0, {0x4f65b80?, 0xc002d08f80}, 0x0)
minio     github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:397 +0x245
minio github.com/minio/minio/cmd.(*erasureServerPools).StartRebalance.func1(0x6749336244523363?)
minio     github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:814 +0x6a
minio created by github.com/minio/minio/cmd.(*erasureServerPools).StartRebalance
minio     github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:812 +0x2ab
```

## Steps to Reproduce (for bugs)
1. Expand the cluster of 1 pool to 3 pools.
2. Run `mcli admin rebalance start`.
3. 1 node is crashing.

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Regression
I don't know.

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`): `RELEASE.2023-06-09T07-32-12Z`
* Server setup and configuration: k8s `v1.26.5` with the minio operator `v5.0.5`


Discussion / hints:
Because it is still balancing (this balancing information has been saved) but you restarted the node because there is a problem with the boot order causing this problem. From my point of view I can't think of anything that would solve your current problem, but you could delete a file with the rebalance information for each node, it's in `.minio.sys/rebalance.bin` , I'm not sure what that would entail. @hrenard @harshavardhana @klauspost 
@jiuker Yes, but it should never crash, so there is a check missing somewhere.
Okay, I just did a `mc admin rebalance stop` and the server properly started. But when I do a `mc admin rebalance start`, the same server (pool 0, node 0) crashes instantly.
```
panic: runtime error: integer divide by zero

goroutine 238530 [running]:
github.com/minio/minio/cmd.(*rebalanceStats).update(...)
	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:66
github.com/minio/minio/cmd.(*erasureServerPools).updatePoolStats(_, _, {_, _}, {{0xc00e18b1d0, 0x11}, {0xc00cabab90, 0x44}, {0x0, 0x0}, ...})
	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:186 +0x1f4
github.com/minio/minio/cmd.(*erasureServerPools).rebalanceBucket.func2({{0xc00cabab90, 0x44}, {0xc008fc9b00, 0x21c, 0x21c}, 0xc01b7bb200, 0x0})
	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:544 +0xad8
created by github.com/minio/minio/cmd.(*erasureServerPools).rebalanceBucket.func3.1
	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:631 +0x13e
```
If I understand correctly there are 2 issues : the crash at rebalance start and the boot ordering.
> Okay, I just did a `mc admin rebalance stop` and the server properly started. But when I do a `mc admin rebalance start`, the same server (pool 0, node 0) crashes instantly.
> 
> ```
> panic: runtime error: integer divide by zero
> 
> goroutine 238530 [running]:
> github.com/minio/minio/cmd.(*rebalanceStats).update(...)
> 	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:66
> github.com/minio/minio/cmd.(*erasureServerPools).updatePoolStats(_, _, {_, _}, {{0xc00e18b1d0, 0x11}, {0xc00cabab90, 0x44}, {0x0, 0x0}, ...})
> 	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:186 +0x1f4
> github.com/minio/minio/cmd.(*erasureServerPools).rebalanceBucket.func2({{0xc00cabab90, 0x44}, {0xc008fc9b00, 0x21c, 0x21c}, 0xc01b7bb200, 0x0})
> 	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:544 +0xad8
> created by github.com/minio/minio/cmd.(*erasureServerPools).rebalanceBucket.func3.1
> 	github.com/minio/minio/cmd/erasure-server-pool-rebalance.go:631 +0x13e
> ```
> 
> If I understand correctly there are 2 issues : the crash at rebalance start and the boot ordering.

And it will restart normal? @hrenard 
Yes, when the rebalance is stopped.

Fix patch:
diff --git a/cmd/erasure-server-pool.go b/cmd/erasure-server-pool.go
index ef00f910271e6..eec8034fb3c57 100644
--- a/cmd/erasure-server-pool.go
+++ b/cmd/erasure-server-pool.go
@@ -141,6 +141,10 @@ func newErasureServerPools(ctx context.Context, endpointServerPools EndpointServ
 	}
 
 	z.decommissionCancelers = make([]context.CancelFunc, len(z.serverPools))
+
+	// initialize the object layer.
+	setObjectLayer(z)
+
 	r := rand.New(rand.NewSource(time.Now().UnixNano()))
 	for {
 		err := z.Init(ctx) // Initializes all pools.
@@ -159,6 +163,7 @@ func newErasureServerPools(ctx context.Context, endpointServerPools EndpointServ
 	globalLocalDrivesMu.Lock()
 	globalLocalDrives = localDrives
 	defer globalLocalDrivesMu.Unlock()
+
 	return z, nil
 }
 
diff --git a/cmd/server-main.go b/cmd/server-main.go
index 71fc97c5ec2a4..6c83c1f4163f4 100644
--- a/cmd/server-main.go
+++ b/cmd/server-main.go
@@ -413,12 +413,9 @@ func bootstrapTrace(msg string) {
 	})
 }
 
-func initServer(ctx context.Context, newObject ObjectLayer) error {
+func initServerConfig(ctx context.Context, newObject ObjectLayer) error {
 	t1 := time.Now()
 
-	// Once the config is fully loaded, initialize the new object layer.
-	setObjectLayer(newObject)
-
 	r := rand.New(rand.NewSource(time.Now().UnixNano()))
 
 	for {
@@ -633,6 +630,12 @@ func serverMain(ctx *cli.Context) {
 		}
 	}
 
+	if !globalDisableFreezeOnBoot {
+		// Freeze the services until the bucket notification subsystem gets initialized.
+		bootstrapTrace("freezeServices")
+		freezeServices()
+	}
+
 	bootstrapTrace("newObjectLayer")
 	newObject, err := newObjectLayer(GlobalContext, globalEndpoints)
 	if err != nil {
@@ -645,31 +648,22 @@ func serverMain(ctx *cli.Context) {
 	bootstrapTrace("newSharedLock")
 	globalLeaderLock = newSharedLock(GlobalContext, newObject, "leader.lock")
 
-	// Enable background operations for erasure coding
+	// Enable background operations on
+	//
+	// - Disk auto healing
+	// - MRF (most recently failed) healing
+	// - Background expiration routine for lifecycle policies
 	bootstrapTrace("initAutoHeal")
 	initAutoHeal(GlobalContext, newObject)
+
 	bootstrapTrace("initHealMRF")
 	initHealMRF(GlobalContext, newObject)
+
 	bootstrapTrace("initBackgroundExpiry")
 	initBackgroundExpiry(GlobalContext, newObject)
 
-	if !globalCLIContext.StrictS3Compat {
-		logger.Info(color.RedBold("WARNING: Strict AWS S3 compatible incoming PUT, POST content payload validation is turned off, caution is advised do not use in production"))
-	}
-
-	if globalActiveCred.Equal(auth.DefaultCredentials) {
-		msg := fmt.Sprintf("WARNING: Detected default credentials '%s', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables",
-			globalActiveCred)
-		logger.Info(color.RedBold(msg))
-	}
-
-	if !globalDisableFreezeOnBoot {
-		// Freeze the services until the bucket notification subsystem gets initialized.
-		freezeServices()
-	}
-
-	bootstrapTrace("initializing the server")
-	if err = initServer(GlobalContext, newObject); err != nil {
+	bootstrapTrace("initServerConfig")
+	if err = initServerConfig(GlobalContext, newObject); err != nil {
 		var cerr config.Err
 		// For any config error, we don't need to drop into safe-mode
 		// instead its a user error and should be fixed by user.
@@ -685,12 +679,22 @@ func serverMain(ctx *cli.Context) {
 		logger.LogIf(GlobalContext, err)
 	}
 
+	if !globalCLIContext.StrictS3Compat {
+		logger.Info(color.RedBold("WARNING: Strict AWS S3 compatible incoming PUT, POST content payload validation is turned off, caution is advised do not use in production"))
+	}
+
+	if globalActiveCred.Equal(auth.DefaultCredentials) {
+		msg := fmt.Sprintf("WARNING: Detected default credentials '%s', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables",
+			globalActiveCred)
+		logger.Info(color.RedBold(msg))
+	}
+
 	// Initialize users credentials and policies in background right after config has initialized.
 	go func() {
 		bootstrapTrace("globalIAMSys.Init")
 		globalIAMSys.Init(GlobalContext, newObject, globalEtcdClient, globalRefreshIAMInterval)
 
-		// Initialize
+		// Initialize Console UI
 		if globalBrowserEnabled {
 			bootstrapTrace("initConsoleServer")
 			srv, err := initConsoleServer()
@@ -721,7 +725,6 @@ func serverMain(ctx *cli.Context) {
 
 	go func() {
 		if !globalDisableFreezeOnBoot {
-			bootstrapTrace("freezeServices")
 			defer unfreezeServices()
 			defer bootstrapTrace("unfreezeServices")
 			t := time.AfterFunc(5*time.Minute, func() {


Test patch: