Repository: minio/minio (language: go)

Issue:
minio via Docker in `gateway nas` mode fails to generate events after container is stopped and restarted 
<!--- Provide a general summary of the issue in the Title above -->
I'm running minio via the official Docker image and docker compose. The system is running in `gateway nas` mode. I have `etcd` configured and I'm using `rabbitmq` locally for events and setting it all up via env vars.

When I configure a bucket event that includes a prefix it works as expected. e.g.:

```
mc event add alias/bucket arn:minio:sqs::_:amqp --prefix "simple-test/"
```

If I stop the container (i.e. `docker-compose down` or hitting `cntl+c` when running in the foreground) and start it again (i.e. `docker-compose up`), events stop getting emitted by minio. If I remove and re-add the same event configuration, events resume working. e.g.:

```
mc event remove alias/bucket arn:minio:sqs::_:amqp --prefix "simple-test/"
mc event add alias/bucket arn:minio:sqs::_:amqp --prefix "simple-test/"
```

I am ensuring that minio does not start until after the auth provider and rabbitmq are fully up and ready.

I'm not sure if something isn't be persisted properly in `gateway nas` mode? The event configs are there the second time the container is started, but events are not emitted. 

## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->
I'd expect minio to load all event configurations and resume emitting events on container stop/start. 

## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
When the container starts, any previously configured events do not emit events. If you manipulate event config in a running container it works.

## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->
As a workaround, you can simply remove and re-add all the events, but this is complex and brittle. 

## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->

1. Start minio in `gateway nas` mode via Docker. Configure `MINIO_NOTIFY_AMQP_URL` and related env vars using a rabbitmq instance 
2. Add a bucket event configuration (e.g. `mc event add alias/bucket arn:minio:sqs::_:amqp --prefix "simple-test/"`
3. Write an object and see the event get generated
4. Stop the container
5. Start the container
6. Write an object and see NO event is seen in the queue

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
I'm using minio in nas mode to expose file shares programmatically via the S3 API. Events are used to do additional processing on certain objects. The container needs to be occasionally restarted or upgraded and event configurations are then effectively lost because events stop.

## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`): `RELEASE.2021-11-09T03-21-45Z`
* Server setup and configuration: 

  Docker container, running `RELEASE.2021-11-09T03-21-45Z` with a slight modification of an additional script during the entrypoint to wait for auth and rabbitmq servers. Auth setup via OIDC. And etcd enabled. A slightly redacted docker-compose file is how it's run:

```
version: '3.7'

services:
  minio:
    build:
      context: ./
      dockerfile: ./datastore/Dockerfile
      args:                                                                      
        AUTH_ENDPOINT: ${AUTH_SERVICE_ENDPOINT}           
    volumes:
      - ${NAS_ROOT}:/container/vol
      - ${CONFIG_DIR}/data/events:/container/events
    expose:
      - "9000"
    user: "${UID}:${GID}"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
      MINIO_ETCD_ENDPOINTS: http://etcd-0:2379
      MINIO_IDENTITY_OPENID_CONFIG_URL: ${AUTH_SERVICE_ENDPOINT}/.well-known/openid-configuration
      MINIO_IDENTITY_OPENID_CLIENT_ID: MyServer
      MINIO_IDENTITY_OPENID_SCOPES: "openid,email,profile,service"
      MINIO_IDENTITY_OPENID_CLAIM_NAME: "nickname"
      MINIO_NOTIFY_AMQP_ENABLE: "on"
      MINIO_NOTIFY_AMQP_URL: "amqp://${RABBITMQ_USER}:${RABBITMQ_PASS}@rabbitmq:5672"
      MINIO_NOTIFY_AMQP_EXCHANGE: "service"
      MINIO_NOTIFY_AMQP_EXCHANGE_TYPE: "direct"
      MINIO_NOTIFY_AMQP_ROUTING_KEY: "bucket_notifications"
      MINIO_NOTIFY_AMQP_MANDATORY: "on"
      MINIO_NOTIFY_AMQP_DURABLE: "on"
      MINIO_NOTIFY_AMQP_NO_WAIT: "off"
      MINIO_NOTIFY_AMQP_AUTO_DELETED: "off"
      MINIO_NOTIFY_AMQP_DELIVERY_MODE: 2
      MINIO_NOTIFY_AMQP_QUEUE_DIR: '/container/events'
      MINIO_NOTIFY_AMQP_QUEUE_LIMIT: '100000'
      MINIO_NOTIFY_AMQP_COMMENT: Notification queue for bucket events 
    command: gateway nas /container/vol --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 15s
      timeout: 5s
      retries: 3
    depends_on:
      - reverse-proxy
      - rabbitmq
      - etcd-0
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=web"
      - "traefik.http.routers.minio.rule=Host(`${DOMAIN}`) && (PathPrefix(`/`) || PathPrefix(`/minio`))"
      - "traefik.http.routers.minio.priority=1"
      - "traefik.http.routers.minio.service=minio-store"
      - "traefik.http.services.minio-store.loadbalancer.server.port=9000"
      - "traefik.http.middlewares.ui-redirectregex.redirectregex.regex=^${EXTERNAL_HOSTNAME}/$$"
      - "traefik.http.middlewares.ui-redirectregex.redirectregex.replacement=${EXTERNAL_HOSTNAME}/ui"
      - "traefik.http.routers.minio.middlewares=ui-redirectregex@docker"
    networks:
      - internal
      - web
    restart: always
    deploy:
      mode: replicated
      replicas: 1

  etcd-0:
    image: gcr.io/etcd-development/etcd:v3.3.9
    hostname: etcd-0
    command:
      - etcd
      - --name=etcd-0
      - --data-dir=/etcd-data
      - --advertise-client-urls=http://etcd-0:2379
      - --listen-client-urls=http://0.0.0.0:2379
      - --initial-advertise-peer-urls=http://etcd-0:2380
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-cluster=etcd-0=http://etcd-0:2380
      - --initial-cluster-state=new
      - --initial-cluster-token=tkn
    volumes:
      - etcd-0vol:/etcd-data
    ports:
      - "2379:2379"
      - "2380:2380"
    deploy:
      replicas: 1
    restart: always
    networks:
      - internal
```



* Operating System and version (`uname -a`): `Linux a2b39d4efa96 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux`



Discussion / hints:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 15 days if no further activity occurs. Thank you for your contributions.
Some additional observations - still doing more testing, but it appears like I am seeing this behavior if a minio server is left running "for a while" in this configuration. Not sure yet what exactly "a while" means so trying to reproduce.

Again, bucket events stop being set to rabbitmq. Remove/add the event prefix and it all starts working again. 

Fix patch:
diff --git a/cmd/bootstrap-peer-server.go b/cmd/bootstrap-peer-server.go
index 8d0e7e7018daf..83eb09db83f90 100644
--- a/cmd/bootstrap-peer-server.go
+++ b/cmd/bootstrap-peer-server.go
@@ -205,11 +205,11 @@ func verifyServerSystemConfig(ctx context.Context, endpointServerPools EndpointS
 	for onlineServers < len(clnts)/2 {
 		for _, clnt := range clnts {
 			if err := clnt.Verify(ctx, srcCfg); err != nil {
-				if isNetworkError(err) {
-					offlineEndpoints = append(offlineEndpoints, clnt.String())
-					continue
+				if !isNetworkError(err) {
+					logger.Info(fmt.Errorf("%s has incorrect configuration: %w", clnt.String(), err).Error())
 				}
-				return fmt.Errorf("%s as has incorrect configuration: %w", clnt.String(), err)
+				offlineEndpoints = append(offlineEndpoints, clnt.String())
+				continue
 			}
 			onlineServers++
 		}
diff --git a/cmd/bucket-metadata-sys.go b/cmd/bucket-metadata-sys.go
index 79bcc3bfadeeb..82dbea459a603 100644
--- a/cmd/bucket-metadata-sys.go
+++ b/cmd/bucket-metadata-sys.go
@@ -434,6 +434,7 @@ func (sys *BucketMetadataSys) concurrentLoad(ctx context.Context, buckets []Buck
 			_, _ = objAPI.HealBucket(ctx, buckets[index].Name, madmin.HealOpts{
 				// Ensure heal opts for bucket metadata be deep healed all the time.
 				ScanMode: madmin.HealDeepScan,
+				Recreate: true,
 			})
 			meta, err := loadBucketMetadata(ctx, objAPI, buckets[index].Name)
 			if err != nil {
diff --git a/cmd/gateway-main.go b/cmd/gateway-main.go
index bb572624297ba..df8a032598557 100644
--- a/cmd/gateway-main.go
+++ b/cmd/gateway-main.go
@@ -298,15 +298,17 @@ func StartGateway(ctx *cli.Context, gw Gateway) {
 	globalObjectAPI = newObject
 	globalObjLayerMutex.Unlock()
 
+	go globalIAMSys.Init(GlobalContext, newObject, globalEtcdClient, globalRefreshIAMInterval)
+
 	if gatewayName == NASBackendGateway {
 		buckets, err := newObject.ListBuckets(GlobalContext)
 		if err != nil {
 			logger.Fatal(err, "Unable to list buckets")
 		}
-		logger.FatalIf(globalNotificationSys.Init(GlobalContext, buckets, newObject), "Unable to initialize notification system")
-	}
+		logger.FatalIf(globalBucketMetadataSys.Init(GlobalContext, buckets, newObject), "Unable to initialize bucket metadata")
 
-	go globalIAMSys.Init(GlobalContext, newObject, globalEtcdClient, globalNotificationSys, globalRefreshIAMInterval)
+		logger.FatalIf(globalNotificationSys.Init(GlobalContext, newObject), "Unable to initialize notification system")
+	}
 
 	if globalCacheConfig.Enabled {
 		// initialize the new disk cache objects.
diff --git a/cmd/iam.go b/cmd/iam.go
index 6041d2f1dd6f2..4669a505a1a40 100644
--- a/cmd/iam.go
+++ b/cmd/iam.go
@@ -66,7 +66,6 @@ type IAMSys struct {
 	sync.Mutex
 
 	iamRefreshInterval time.Duration
-	notificationSys    *NotificationSys
 
 	usersSysType UsersSysType
 
@@ -198,12 +197,11 @@ func (sys *IAMSys) Load(ctx context.Context) error {
 }
 
 // Init - initializes config system by reading entries from config/iam
-func (sys *IAMSys) Init(ctx context.Context, objAPI ObjectLayer, etcdClient *etcd.Client, nSys *NotificationSys, iamRefreshInterval time.Duration) {
+func (sys *IAMSys) Init(ctx context.Context, objAPI ObjectLayer, etcdClient *etcd.Client, iamRefreshInterval time.Duration) {
 	sys.Lock()
 	defer sys.Unlock()
 
 	sys.iamRefreshInterval = iamRefreshInterval
-	sys.notificationSys = nSys
 
 	// Initialize IAM store
 	sys.initStore(objAPI, etcdClient)
@@ -466,7 +464,7 @@ func (sys *IAMSys) DeletePolicy(ctx context.Context, policyName string, notifyPe
 	}
 
 	// Notify all other MinIO peers to delete policy
-	for _, nerr := range sys.notificationSys.DeletePolicy(policyName) {
+	for _, nerr := range globalNotificationSys.DeletePolicy(policyName) {
 		if nerr.Err != nil {
 			logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 			logger.LogIf(ctx, nerr.Err)
@@ -524,7 +522,7 @@ func (sys *IAMSys) SetPolicy(ctx context.Context, policyName string, p iampolicy
 
 	if !sys.HasWatcher() {
 		// Notify all other MinIO peers to reload policy
-		for _, nerr := range sys.notificationSys.LoadPolicy(policyName) {
+		for _, nerr := range globalNotificationSys.LoadPolicy(policyName) {
 			if nerr.Err != nil {
 				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 				logger.LogIf(ctx, nerr.Err)
@@ -546,7 +544,7 @@ func (sys *IAMSys) DeleteUser(ctx context.Context, accessKey string, notifyPeers
 
 	// Notify all other MinIO peers to delete user.
 	if notifyPeers && !sys.HasWatcher() {
-		for _, nerr := range sys.notificationSys.DeleteUser(accessKey) {
+		for _, nerr := range globalNotificationSys.DeleteUser(accessKey) {
 			if nerr.Err != nil {
 				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 				logger.LogIf(ctx, nerr.Err)
@@ -572,7 +570,7 @@ func (sys *IAMSys) CurrentPolicies(policyName string) string {
 func (sys *IAMSys) notifyForUser(ctx context.Context, accessKey string, isTemp bool) {
 	// Notify all other MinIO peers to reload user.
 	if !sys.HasWatcher() {
-		for _, nerr := range sys.notificationSys.LoadUser(accessKey, isTemp) {
+		for _, nerr := range globalNotificationSys.LoadUser(accessKey, isTemp) {
 			if nerr.Err != nil {
 				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 				logger.LogIf(ctx, nerr.Err)
@@ -735,7 +733,7 @@ func (sys *IAMSys) SetUserStatus(ctx context.Context, accessKey string, status m
 func (sys *IAMSys) notifyForServiceAccount(ctx context.Context, accessKey string) {
 	// Notify all other Minio peers to reload the service account
 	if !sys.HasWatcher() {
-		for _, nerr := range sys.notificationSys.LoadServiceAccount(accessKey) {
+		for _, nerr := range globalNotificationSys.LoadServiceAccount(accessKey) {
 			if nerr.Err != nil {
 				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 				logger.LogIf(ctx, nerr.Err)
@@ -940,7 +938,7 @@ func (sys *IAMSys) DeleteServiceAccount(ctx context.Context, accessKey string, n
 	}
 
 	if notifyPeers && !sys.HasWatcher() {
-		for _, nerr := range sys.notificationSys.DeleteServiceAccount(accessKey) {
+		for _, nerr := range globalNotificationSys.DeleteServiceAccount(accessKey) {
 			if nerr.Err != nil {
 				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 				logger.LogIf(ctx, nerr.Err)
@@ -1149,7 +1147,7 @@ func (sys *IAMSys) GetUser(ctx context.Context, accessKey string) (cred auth.Cre
 // Notify all other MinIO peers to load group.
 func (sys *IAMSys) notifyForGroup(ctx context.Context, group string) {
 	if !sys.HasWatcher() {
-		for _, nerr := range sys.notificationSys.LoadGroup(group) {
+		for _, nerr := range globalNotificationSys.LoadGroup(group) {
 			if nerr.Err != nil {
 				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 				logger.LogIf(ctx, nerr.Err)
@@ -1256,7 +1254,7 @@ func (sys *IAMSys) PolicyDBSet(ctx context.Context, name, policy string, isGroup
 
 	// Notify all other MinIO peers to reload policy
 	if !sys.HasWatcher() {
-		for _, nerr := range sys.notificationSys.LoadPolicyMapping(name, isGroup) {
+		for _, nerr := range globalNotificationSys.LoadPolicyMapping(name, isGroup) {
 			if nerr.Err != nil {
 				logger.GetReqInfo(ctx).SetTags("peerAddress", nerr.Host.String())
 				logger.LogIf(ctx, nerr.Err)
diff --git a/cmd/notification.go b/cmd/notification.go
index b620a223e4ee4..c7648d81066ab 100644
--- a/cmd/notification.go
+++ b/cmd/notification.go
@@ -674,7 +674,7 @@ func (sys *NotificationSys) set(bucket BucketInfo, meta BucketMetadata) {
 }
 
 // Init - initializes notification system from notification.xml and listenxl.meta of all buckets.
-func (sys *NotificationSys) Init(ctx context.Context, buckets []BucketInfo, objAPI ObjectLayer) error {
+func (sys *NotificationSys) Init(ctx context.Context, objAPI ObjectLayer) error {
 	if objAPI == nil {
 		return errServerNotInitialized
 	}
diff --git a/cmd/server-main.go b/cmd/server-main.go
index 007d355c1387e..7d2148fd9e8e0 100644
--- a/cmd/server-main.go
+++ b/cmd/server-main.go
@@ -34,7 +34,6 @@ import (
 	"time"
 
 	"github.com/minio/cli"
-	"github.com/minio/madmin-go"
 	"github.com/minio/minio/internal/auth"
 	"github.com/minio/minio/internal/bucket/bandwidth"
 	"github.com/minio/minio/internal/color"
@@ -43,7 +42,6 @@ import (
 	xhttp "github.com/minio/minio/internal/http"
 	"github.com/minio/minio/internal/logger"
 	"github.com/minio/minio/internal/rest"
-	"github.com/minio/minio/internal/sync/errgroup"
 	"github.com/minio/pkg/certs"
 	"github.com/minio/pkg/env"
 )
@@ -300,7 +298,7 @@ func configRetriableErrors(err error) bool {
 		errors.Is(err, os.ErrDeadlineExceeded)
 }
 
-func initServer(ctx context.Context, newObject ObjectLayer) ([]BucketInfo, error) {
+func initServer(ctx context.Context, newObject ObjectLayer) error {
 	// Once the config is fully loaded, initialize the new object layer.
 	setObjectLayer(newObject)
 
@@ -316,7 +314,7 @@ func initServer(ctx context.Context, newObject ObjectLayer) ([]BucketInfo, error
 		select {
 		case <-ctx.Done():
 			// Retry was canceled successfully.
-			return nil, fmt.Errorf("Initializing sub-systems stopped gracefully %w", ctx.Err())
+			return fmt.Errorf("Initializing sub-systems stopped gracefully %w", ctx.Err())
 		default:
 		}
 
@@ -348,16 +346,14 @@ func initServer(ctx context.Context, newObject ObjectLayer) ([]BucketInfo, error
 		if err = handleEncryptedConfigBackend(newObject); err == nil {
 			// Upon success migrating the config, initialize all sub-systems
 			// if all sub-systems initialized successfully return right away
-			var buckets []BucketInfo
-			buckets, err = initConfigSubsystem(lkctx.Context(), newObject)
-			if err == nil {
+			if err = initConfigSubsystem(lkctx.Context(), newObject); err == nil {
 				txnLk.Unlock(lkctx.Cancel)
 				// All successful return.
 				if globalIsDistErasure {
 					// These messages only meant primarily for distributed setup, so only log during distributed setup.
 					logger.Info("All MinIO sub-systems initialized successfully")
 				}
-				return buckets, nil
+				return nil
 			}
 		}
 
@@ -371,58 +367,28 @@ func initServer(ctx context.Context, newObject ObjectLayer) ([]BucketInfo, error
 		}
 
 		// Any other unhandled return right here.
-		return nil, fmt.Errorf("Unable to initialize sub-systems: %w", err)
+		return fmt.Errorf("Unable to initialize sub-systems: %w", err)
 	}
 }
 
-func initConfigSubsystem(ctx context.Context, newObject ObjectLayer) ([]BucketInfo, error) {
+func initConfigSubsystem(ctx context.Context, newObject ObjectLayer) error {
 	// %w is used by all error returns here to make sure
 	// we wrap the underlying error, make sure when you
 	// are modifying this code that you do so, if and when
 	// you want to add extra context to your error. This
 	// ensures top level retry works accordingly.
-	// List buckets to heal, and be re-used for loading configs.
-
-	buckets, err := newObject.ListBuckets(ctx)
-	if err != nil {
-		return nil, fmt.Errorf("Unable to list buckets to heal: %w", err)
-	}
-
-	if globalIsErasure {
-		if len(buckets) > 0 {
-			if len(buckets) == 1 {
-				logger.Info(fmt.Sprintf("Verifying if %d bucket is consistent across drives...", len(buckets)))
-			} else {
-				logger.Info(fmt.Sprintf("Verifying if %d buckets are consistent across drives...", len(buckets)))
-			}
-		}
-
-		// Limit to no more than 50 concurrent buckets.
-		g := errgroup.WithNErrs(len(buckets)).WithConcurrency(50)
-		for index := range buckets {
-			index := index
-			g.Go(func() error {
-				_, herr := newObject.HealBucket(ctx, buckets[index].Name, madmin.HealOpts{Recreate: true})
-				return herr
-			}, index)
-		}
-		for _, err := range g.Wait() {
-			if err != nil {
-				return nil, fmt.Errorf("Unable to list buckets to heal: %w", err)
-			}
-		}
-	}
 
 	// Initialize config system.
-	if err = globalConfigSys.Init(newObject); err != nil {
+	if err := globalConfigSys.Init(newObject); err != nil {
 		if configRetriableErrors(err) {
-			return nil, fmt.Errorf("Unable to initialize config system: %w", err)
+			return fmt.Errorf("Unable to initialize config system: %w", err)
 		}
+
 		// Any other config errors we simply print a message and proceed forward.
 		logger.LogIf(ctx, fmt.Errorf("Unable to initialize config, some features may be missing %w", err))
 	}
 
-	return buckets, nil
+	return nil
 }
 
 // serverMain handler called for 'minio server' command.
@@ -516,18 +482,9 @@ func serverMain(ctx *cli.Context) {
 	setHTTPServer(httpServer)
 
 	if globalIsDistErasure && globalEndpoints.FirstLocal() {
-		for {
-			// Additionally in distributed setup, validate the setup and configuration.
-			err := verifyServerSystemConfig(GlobalContext, globalEndpoints)
-			if err == nil || errors.Is(err, context.Canceled) {
-				break
-			}
-			logger.LogIf(GlobalContext, err, "Unable to initialize distributed setup, retrying.. after 5 seconds")
-			select {
-			case <-GlobalContext.Done():
-				return
-			case <-time.After(500 * time.Millisecond):
-			}
+		// Additionally in distributed setup, validate the setup and configuration.
+		if err := verifyServerSystemConfig(GlobalContext, globalEndpoints); err != nil {
+			logger.Fatal(err, "Unable to start the server")
 		}
 	}
 
@@ -546,8 +503,17 @@ func serverMain(ctx *cli.Context) {
 
 	initBackgroundExpiry(GlobalContext, newObject)
 
-	buckets, err := initServer(GlobalContext, newObject)
-	if err != nil {
+	if globalActiveCred.Equal(auth.DefaultCredentials) {
+		msg := fmt.Sprintf("WARNING: Detected default credentials '%s', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables",
+			globalActiveCred)
+		logStartupMessage(color.RedBold(msg))
+	}
+
+	if !globalCLIContext.StrictS3Compat {
+		logStartupMessage(color.RedBold("WARNING: Strict AWS S3 compatible incoming PUT, POST content payload validation is turned off, caution is advised do not use in production"))
+	}
+
+	if err = initServer(GlobalContext, newObject); err != nil {
 		var cerr config.Err
 		// For any config error, we don't need to drop into safe-mode
 		// instead its a user error and should be fixed by user.
@@ -576,67 +542,67 @@ func serverMain(ctx *cli.Context) {
 		}()
 	}
 
-	// Populate existing buckets to the etcd backend
-	if globalDNSConfig != nil {
-		// Background this operation.
-		go initFederatorBackend(buckets, newObject)
-	}
-
-	// Initialize bucket metadata sub-system.
-	globalBucketMetadataSys.Init(GlobalContext, buckets, newObject)
+	// Initialize users credentials and policies in background right after config has initialized.
+	go globalIAMSys.Init(GlobalContext, newObject, globalEtcdClient, globalRefreshIAMInterval)
 
-	// Initialize bucket notification sub-system.
-	globalNotificationSys.Init(GlobalContext, buckets, newObject)
+	// Background all other operations such as initializing bucket metadata etc.
+	go func() {
+		// Initialize transition tier configuration manager
+		if globalIsErasure {
+			initBackgroundReplication(GlobalContext, newObject)
+			initBackgroundTransition(GlobalContext, newObject)
+
+			go func() {
+				if err := globalTierConfigMgr.Init(GlobalContext, newObject); err != nil {
+					logger.LogIf(GlobalContext, err)
+				}
 
-	// Initialize site replication manager.
-	globalSiteReplicationSys.Init(GlobalContext, newObject)
+				globalTierJournal, err = initTierDeletionJournal(GlobalContext)
+				if err != nil {
+					logger.FatalIf(err, "Unable to initialize remote tier pending deletes journal")
+				}
+			}()
+		}
 
-	// Initialize quota manager.
-	globalBucketQuotaSys.Init(newObject)
+		// Initialize site replication manager.
+		globalSiteReplicationSys.Init(GlobalContext, newObject)
 
-	// Initialize users credentials and policies in background right after config has initialized.
-	go globalIAMSys.Init(GlobalContext, newObject, globalEtcdClient, globalNotificationSys, globalRefreshIAMInterval)
+		// Initialize quota manager.
+		globalBucketQuotaSys.Init(newObject)
 
-	initDataScanner(GlobalContext, newObject)
+		initDataScanner(GlobalContext, newObject)
 
-	// Initialize transition tier configuration manager
-	if globalIsErasure {
-		initBackgroundReplication(GlobalContext, newObject)
-		initBackgroundTransition(GlobalContext, newObject)
-
-		go func() {
-			if err := globalTierConfigMgr.Init(GlobalContext, newObject); err != nil {
-				logger.LogIf(GlobalContext, err)
-			}
+		// List buckets to heal, and be re-used for loading configs.
+		buckets, err := newObject.ListBuckets(GlobalContext)
+		if err != nil {
+			logger.LogIf(GlobalContext, fmt.Errorf("Unable to list buckets to heal: %w", err))
+		}
 
-			globalTierJournal, err = initTierDeletionJournal(GlobalContext)
-			if err != nil {
-				logger.FatalIf(err, "Unable to initialize remote tier pending deletes journal")
-			}
-		}()
-	}
+		// Populate existing buckets to the etcd backend
+		if globalDNSConfig != nil {
+			// Background this operation.
+			go initFederatorBackend(buckets, newObject)
+		}
 
-	// initialize the new disk cache objects.
-	if globalCacheConfig.Enabled {
-		logStartupMessage(color.Yellow("WARNING: Disk caching is deprecated for single/multi drive MinIO setups. Please migrate to using MinIO S3 gateway instead of disk caching"))
-		var cacheAPI CacheObjectLayer
-		cacheAPI, err = newServerCacheObjects(GlobalContext, globalCacheConfig)
-		logger.FatalIf(err, "Unable to initialize disk caching")
+		// Initialize bucket metadata sub-system.
+		globalBucketMetadataSys.Init(GlobalContext, buckets, newObject)
 
-		setCacheObjectLayer(cacheAPI)
-	}
+		// Initialize bucket notification sub-system.
+		globalNotificationSys.Init(GlobalContext, newObject)
 
-	// Prints the formatted startup message, if err is not nil then it prints additional information as well.
-	go printStartupMessage(getAPIEndpoints(), err)
+		// initialize the new disk cache objects.
+		if globalCacheConfig.Enabled {
+			logStartupMessage(color.Yellow("WARNING: Disk caching is deprecated for single/multi drive MinIO setups. Please migrate to using MinIO S3 gateway instead of disk caching"))
+			var cacheAPI CacheObjectLayer
+			cacheAPI, err = newServerCacheObjects(GlobalContext, globalCacheConfig)
+			logger.FatalIf(err, "Unable to initialize disk caching")
 
-	if globalActiveCred.Equal(auth.DefaultCredentials) {
-		msg := fmt.Sprintf("WARNING: Detected default credentials '%s', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables", globalActiveCred)
-		logStartupMessage(color.RedBold(msg))
-	}
+			setCacheObjectLayer(cacheAPI)
+		}
 
-	if !globalCLIContext.StrictS3Compat {
-		logStartupMessage(color.RedBold("WARNING: Strict AWS S3 compatible incoming PUT, POST content payload validation is turned off, caution is advised do not use in production"))
-	}
+		// Prints the formatted startup message, if err is not nil then it prints additional information as well.
+		printStartupMessage(getAPIEndpoints(), err)
+	}()
 
 	if serverDebugLog {
 		logger.Info("== DEBUG Mode enabled ==")
@@ -656,6 +622,7 @@ func serverMain(ctx *cli.Context) {
 		}
 		logger.Info("======")
 	}
+
 	<-globalOSSignalCh
 }
 


Test patch:
diff --git a/cmd/admin-handlers_test.go b/cmd/admin-handlers_test.go
index fa66f060e50e0..507f225aac1bb 100644
--- a/cmd/admin-handlers_test.go
+++ b/cmd/admin-handlers_test.go
@@ -78,7 +78,7 @@ func prepareAdminErasureTestBed(ctx context.Context) (*adminErasureTestBed, erro
 
 	initConfigSubsystem(ctx, objLayer)
 
-	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, globalNotificationSys, 2*time.Second)
+	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, 2*time.Second)
 
 	// Setup admin mgmt REST API handlers.
 	adminRouter := mux.NewRouter()
diff --git a/cmd/auth-handler_test.go b/cmd/auth-handler_test.go
index 174494832b248..24789855176e4 100644
--- a/cmd/auth-handler_test.go
+++ b/cmd/auth-handler_test.go
@@ -369,7 +369,7 @@ func TestIsReqAuthenticated(t *testing.T) {
 
 	initConfigSubsystem(ctx, objLayer)
 
-	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, globalNotificationSys, 2*time.Second)
+	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, 2*time.Second)
 
 	creds, err := auth.CreateCredentials("myuser", "mypassword")
 	if err != nil {
@@ -459,7 +459,7 @@ func TestValidateAdminSignature(t *testing.T) {
 
 	initConfigSubsystem(ctx, objLayer)
 
-	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, globalNotificationSys, 2*time.Second)
+	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, 2*time.Second)
 
 	creds, err := auth.CreateCredentials("admin", "mypassword")
 	if err != nil {
diff --git a/cmd/signature-v4-utils_test.go b/cmd/signature-v4-utils_test.go
index 602fe6aa1ff01..1cd00841af188 100644
--- a/cmd/signature-v4-utils_test.go
+++ b/cmd/signature-v4-utils_test.go
@@ -46,7 +46,7 @@ func TestCheckValid(t *testing.T) {
 
 	initConfigSubsystem(ctx, objLayer)
 
-	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, globalNotificationSys, 2*time.Second)
+	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, 2*time.Second)
 
 	req, err := newTestRequest(http.MethodGet, "http://example.com:9000/bucket/object", 0, nil)
 	if err != nil {
diff --git a/cmd/test-utils_test.go b/cmd/test-utils_test.go
index 3d5009c0811ac..4ab298998fd8f 100644
--- a/cmd/test-utils_test.go
+++ b/cmd/test-utils_test.go
@@ -361,7 +361,7 @@ func initTestServerWithBackend(ctx context.Context, t TestErrHandler, testServer
 
 	initConfigSubsystem(ctx, objLayer)
 
-	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, globalNotificationSys, 2*time.Second)
+	globalIAMSys.Init(ctx, objLayer, globalEtcdClient, 2*time.Second)
 
 	return testServer
 }
@@ -1522,7 +1522,7 @@ func initAPIHandlerTest(ctx context.Context, obj ObjectLayer, endpoints []string
 
 	initConfigSubsystem(ctx, obj)
 
-	globalIAMSys.Init(ctx, obj, globalEtcdClient, globalNotificationSys, 2*time.Second)
+	globalIAMSys.Init(ctx, obj, globalEtcdClient, 2*time.Second)
 
 	// get random bucket name.
 	bucketName := getRandomBucketName()
@@ -1807,7 +1807,7 @@ func ExecObjectLayerTest(t TestErrHandler, objTest objTestType) {
 			t.Fatal("Unexpected error", err)
 		}
 		initConfigSubsystem(ctx, objLayer)
-		globalIAMSys.Init(ctx, objLayer, globalEtcdClient, globalNotificationSys, 2*time.Second)
+		globalIAMSys.Init(ctx, objLayer, globalEtcdClient, 2*time.Second)
 
 		// Executing the object layer tests for single node setup.
 		objTest(objLayer, FSTestStr, t)
@@ -1832,7 +1832,7 @@ func ExecObjectLayerTest(t TestErrHandler, objTest objTestType) {
 		}
 		setObjectLayer(objLayer)
 		initConfigSubsystem(ctx, objLayer)
-		globalIAMSys.Init(ctx, objLayer, globalEtcdClient, globalNotificationSys, 2*time.Second)
+		globalIAMSys.Init(ctx, objLayer, globalEtcdClient, 2*time.Second)
 
 		// Executing the object layer tests for Erasure.
 		objTest(objLayer, ErasureTestStr, t)