Repository: minio/minio (language: go)

Issue:
restarted minio process show zero disks
<!--- Provide a general summary of the issue in the Title above -->

Hi, 

We've restart ( not a crash)  a minio service on one cluster node, and after 30 minutes we still have :
```
 ● REDACTED1:9000
   Uptime: 30 minutes
   Version: 2022-02-01T18:00:14Z
   Network: 4/4 OK
   Drives: 0/0 OK

●  REDACTED2:9000
   Uptime: 20 hours
   Version: 2022-02-01T18:00:14Z
   Network: 4/4 OK
   Drives: 4/4 OK

●  REDACTED3:9000
   Uptime: 20 hours
   Version: 2022-02-01T18:00:14Z
   Network: 4/4 OK
   Drives: 4/4 OK

●  REDACTED4:9000
   Uptime: 20 hours
   Version: 2022-02-01T18:00:14Z
   Network: 4/4 OK
   Drives: 4/4 OK
```

The node is up, but does not show the 4 configured drives.
There is no information in the console.

When we try to restart the node there is this line printed in the console :
`ERROR Server startup canceled upon user request: Initializing sub-systems stopped gracefully context canceled
`
When we inspect the process using lsof/strace, we see some disk i/o activity for a few seconds, reading the metadata of all buckets, then only network i/o with other nodes, no file stat or open.

## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

A functional minio node

## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->

The process is running, the node is up, respond to health checks, but no longer expose disks

## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->

1. kill a minio node part of a cluster with a term signal

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`):` minio version RELEASE.2022-02-01T18-00-14Z`
* Server setup and configuration: `minio server https://minio{1...4}.REDACTED.com:9000/mnt/disk{1...4}/data --console-address ":9001"`

* Operating System and version (`uname -a`): `Linux REDACTED 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06) x86_64 GNU/Linux
`


Discussion / hints:
@vixns what kind of disks are these? 
It is better to capture `mc admin info --json` output and share it here @vixns 
@harshavardhana  [config.json.gz](https://github.com/minio/minio/files/7997755/config.json.gz)
```

Disk /dev/sdd: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: HGST HUS726T4TAL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
```

from what I can tell the drives are not readable there will be message in the systemd logs on the node1 
Do not see any drives here

```    "servers": [
      {
        "state": "online",
        "endpoint": "REDACTED1:9000",
        "uptime": 4780,
        "version": "2022-02-01T18:00:14Z",
        "commitID": "067d21d0f23254ea9ced107ef1a408fed83964a5",
        "network": {
          "REDACTED1:9000": "online",
          "REDACTED2:9000": "online",
          "REDACTED3:9000": "online",
          "REDACTED4:9000": "online"
        },
        "poolNumber": 1,
        "mem_stats": {
          "Alloc": 68356664,
          "TotalAlloc": 4058499696,
          "Mallocs": 31786307,
          "Frees": 31649943,
          "HeapAlloc": 68356664
        }
      },
```


That's the problem. The drives are present and healthy
@vixns You had mentioned an upgrade via `mc` - we still would benefit from knowing what the previous version was.

For good measure, maybe `lsblk -f` so we can see those drives, and your `/etc/fstab` so we can confirm the mounting status.
@ravindk89 not sure about the exact previous version, it was quite recent, from late december or january

```
NAME        FSTYPE  FSVER            LABEL          UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
└─sda1      xfs                                     d08f9374-3477-471a-8010-b897e2c32181      2T    46% /mnt/disk1
sdb
└─sdb1      xfs                                     622fef97-04cf-4859-a2ff-4b413cf4553c      2T    46% /mnt/disk3
sdc
└─sdc1      xfs                                     4adacbb5-f7d4-4440-bedc-dc907cf8835f      2T    46% /mnt/disk2
sdd
└─sdd1      xfs                                     267f344e-82fc-49cf-afe5-9c7575d9a88b      2T    46% /mnt/disk4
```
The drives are mounted, when tracing the minio process start there is access to xl.meta files.
Did you mount via `/etc/fstab` or manually? Just to double check that. As @harshavardhana noted MinIO is not seeing any disks attached to that node. There should be a reason for this, we just need to identify it. 
@ravindk89 the disck are mounted via `/etc/fstab`
Can you provide us with the contents of `/etc/fstab` ? This is for verification purposes.
when launching using strace, we can see successfully files access 

```
strace.txt.16062:openat(AT_FDCWD, "/mnt/disk1/vixns/.minio.sys/config/iam/policydb/users/REDACTED1.json/xl.meta", O_RDONLY|O_DIRECT|O_NOATIME|O_CLOEXEC) = 44
strace.txt.16062:openat(AT_FDCWD, "/mnt/disk2/vixns/.minio.sys/config/iam/policydb/users/REDACTED2.json/xl.meta", O_RDONLY|O_DIRECT|O_NOATIME|O_CLOEXEC) = 32
strace.txt.16062:openat(AT_FDCWD, "/mnt/disk2/vixns/.minio.sys/config/iam/policydb/users/REDACTED3.json/xl.meta", O_RDONLY|O_DIRECT|O_NOATIME|O_CLOEXEC) = 32
strace.txt.16062:openat(AT_FDCWD, "/mnt/disk4/vixns/.minio.sys/config/iam/policydb/users/REDACTED4.json/xl.meta", O_RDONLY|O_DIRECT|O_NOATIME|O_CLOEXEC) = 54
strace.txt.16062:openat(AT_FDCWD, "/mnt/disk3/vixns/.minio.sys/config/iam/policydb/users/REDACTED5.json/xl.meta", O_RDONLY|O_DIRECT|O_NOATIME|O_CLOEXEC) = 54
```

fstab 
```
UUID=24a2b950-62ba-4568-ba6a-52d745caa559	/	xfs	defaults	0	1
/dev/nvme0n1p1	/boot/efi	vfat	defaults	0	1
UUID=ed86f11c-7c16-431e-b9d5-dcd9447cf471	swap	swap	defaults	0	0
UUID=d08f9374-3477-471a-8010-b897e2c32181	/mnt/disk1	xfs	defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k	0	0
UUID=4adacbb5-f7d4-4440-bedc-dc907cf8835f	/mnt/disk2	xfs	defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k	0	0
UUID=622fef97-04cf-4859-a2ff-4b413cf4553c	/mnt/disk3	xfs	defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k	0	0
UUID=267f344e-82fc-49cf-afe5-9c7575d9a88b	/mnt/disk4	xfs	defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k	0	0

```
I don't think these drives are allowing reading `format.json` 
The drives allow reading to the minio user.
```
 sudo -u minio cat /mnt/disk4/vixns/.minio.sys/config/iam/policydb/users/REDACTED.json/xl.meta | base64
WEwyIAEAAgDGAAABUoGoVmVyc2lvbnORgqRUeXBlAaVWMk9iat4AEaJJRMQQAAAAAAAAAAAAAAAA
[...]
```
@harshavardhana  Should I try to downgrade to version 2022-01-08T03-11-54Z ? Can I downgrade just this instance or is it required to have the same version on all cluster's nodes ?  
> @harshavardhana Should I try to downgrade to version 2022-01-08T03-11-54Z ? Can I downgrade just this instance or is it required to have the same version on all cluster's nodes ?

No you cannot do that. 
Can you share the server logs from that node? @vixns 
All cluster nodes must have the same version, and should be upgraded at the same time (no rolling restarts).

We generally would not recommend downgrading as a regular course of action.
also how is this setup? are you using some orchestrator here? 
No orchestrator, this setup is dead simple, 4 bare metal servers with 4 drives, a minimal debian 10 distribution with runit and minio, that's all,

I reviewed server logs, there is only basic system logs, ssh, sudo , cron, nothing else.
Couldn't this issue be a side effect of https://github.com/minio/minio/pull/14148 ? 
> No orchestrator, this setup is dead simple, 4 bare metal servers with 4 drives, a minimal debian 10 distribution with runit and minio, that's all,

Ah okay can you enable `export XTERM=screen`  and in your runit file and see if this shows up? @vixns 
> Couldn't this issue be a side effect of #14148 ?

No sir we have many environments locally there is no issue. 
```
kubectl exec -it mc /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@mc /]# mc admin info all-nodes
●  all-nodes-ss-0-0.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-1.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-2.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-3.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-4.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-5.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

1.2 GiB Used, 1 Bucket, 11,523 Objects
60 drives online, 0 drives offline
[root@mc /]# 
```

No issues here. 
I tried launching a minio on the same server / same 4 drives using a test folder, no issue.
And restarted one of the pods all come up fine

```
mc admin info all-nodes
●  all-nodes-ss-0-0.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-1.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 29 seconds 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-2.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-3.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-4.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-5.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

1.2 GiB Used, 1 Bucket, 11,523 Objects
60 drives online, 0 drives offline
```
Most probably you might have a terminal that is hanging due to a terminal issue - most probably related to https://github.com/minio/minio/issues/14235 
Either this or `format.json` is not readable on the previous drives, some kind of an XFS bug.  
Will you be able to provide access to me remotely to this server?
I see that there is no activity here. Closing this issue - please let us know if you want us to take a look further. 
@harshavardhana  The cluster was unstable, so, let's recklessly broke it (ノ°Д°）ノ︵ ┻━┻

I restart a second minio, then 0 drives on it,

Same for the third node.

Finally, after restarting the last node, the cluster were able to re-bootstrap, and everything is back to normal, except my frustration of not understanding the cause of this issue.

Thank you for your time and attention.
> @harshavardhana The cluster was unstable, so, let's recklessly broke it (ノ°Д°）ノ︵ ┻━┻
> 
> I restart a second minio, then 0 drives on it,
> 
> Same for the third node.
> 
> Finally, after restarting the last node, the cluster were able to re-bootstrap, and everything is back to normal, except my frustration of not understanding the cause of this issue.
> 
> Thank you for your time and attention.

This is will never happen in any situation, it is an impossible situation. If it happens I would love to look at it. 

Can you restart only one node again and see 0/0 drives?
Yes, same after restart one node, and back to normal after restarting all.
> Yes, same after restart one node, and back to normal after restarting all.

Okay will it be possible to provide remote access - it looks like you have some DNS issues here or we are doing some caching that is not working. 
Here is how I am testing right now

```
kubectl exec -it mc /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@mc /]# nfo^C
[root@mc /]# mc admin info all-nodes
●  all-nodes-ss-0-0.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: offline
   Drives: 0/10 OK 

●  all-nodes-ss-0-1.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 2 hours 
   Version: 2022-02-01T18:00:14Z
   Network: 5/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-2.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 5/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-3.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 5/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-4.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 5/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-5.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 5/6 OK 
   Drives: 10/10 OK 

1.2 GiB Used, 1 Bucket, 11,523 Objects
50 drives online, 10 drives offline
```

and then it came up fine

```
mc admin info all-nodes
●  all-nodes-ss-0-0.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 26 seconds 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-1.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 2 hours 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-2.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-3.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-4.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

●  all-nodes-ss-0-5.all-nodes-hl.ns-1.svc.cluster.local:9000
   Uptime: 1 day 
   Version: 2022-02-01T18:00:14Z
   Network: 6/6 OK 
   Drives: 10/10 OK 

1.2 GiB Used, 1 Bucket, 11,523 Objects
60 drives online, 0 drives offline
```

So it does look like you have something going on - on your servers. 
I don't think without putting some debug binary I don't think it is going to be easily able to tell what might be the issue in your environment. 
This is a production environment, so allowing remoting access outside may be difficult.
I could try to create a debug environment by using the `.minio.sys` folders only, not the buckets. 
Do you think it will work and can reproduce this issue ? 
You can provide `mc admin inspect alias/.minio.sys/format.json` 

> This is a production environment, so allowing remoting access outside may be difficult.

A production environment - yet you don't have a support subscription with us? @vixns - are you using this for commercial use? 
> You can provide `mc admin inspect alias/.minio.sys/format.json`
> 
> >

Is this still an issue? @vixns can you provide some details?
I did not try to reproduce yet, here is the required file [inspect.7e2f1e93.zip](https://github.com/minio/minio/files/8007763/inspect.7e2f1e93.zip)

> I did not try to reproduce yet, here is the required file [inspect.7e2f1e93.zip](https://github.com/minio/minio/files/8007763/inspect.7e2f1e93.zip)

Can you help us reproduce? Can you share the way you start MinIO do you run it as a script or part of systemd service?

There is literally no information here for me to look at 

```
Archive:  inspect.7e2f1e93.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
       0  Stored        0   0% 2022-02-05 02:01 00000000  REDACTED1:9000/
       0  Stored        0   0% 2022-02-05 02:01 00000000  REDACTED2:9000/
       0  Stored        0   0% 2022-02-05 02:01 00000000  REDACTED3:9000/
       0  Stored        0   0% 2022-02-05 02:01 00000000  REDACTED4:9000/
--------          -------  ---                            -------
       0                0   0%                            4 files
```

I don't think we can do anything to help you here - if you are going to manually hide information. 

Fix patch:
diff --git a/cmd/erasure-server-pool.go b/cmd/erasure-server-pool.go
index 4796c64f078d8..c128aa55665d3 100644
--- a/cmd/erasure-server-pool.go
+++ b/cmd/erasure-server-pool.go
@@ -1730,14 +1730,13 @@ func (z *erasureServerPools) Walk(ctx context.Context, bucket, prefix string, re
 // HealObjectFn closure function heals the object.
 type HealObjectFn func(bucket, object, versionID string) error
 
-func listAndHeal(ctx context.Context, bucket, prefix string, set *erasureObjects, healEntry func(metaCacheEntry) error, errCh chan<- error) {
+func listAndHeal(ctx context.Context, bucket, prefix string, set *erasureObjects, healEntry func(metaCacheEntry) error) error {
 	ctx, cancel := context.WithCancel(ctx)
 	defer cancel()
 
 	disks, _ := set.getOnlineDisksWithHealing()
 	if len(disks) == 0 {
-		errCh <- errors.New("listAndHeal: No non-healing disks found")
-		return
+		return errors.New("listAndHeal: No non-healing disks found")
 	}
 
 	// How to resolve partial results.
@@ -1763,8 +1762,8 @@ func listAndHeal(ctx context.Context, bucket, prefix string, set *erasureObjects
 		reportNotFound: false,
 		agreed: func(entry metaCacheEntry) {
 			if err := healEntry(entry); err != nil {
-				errCh <- err
-				return
+				logger.LogIf(ctx, err)
+				cancel()
 			}
 		},
 		partial: func(entries metaCacheEntries, nAgreed int, errs []error) {
@@ -1776,7 +1775,8 @@ func listAndHeal(ctx context.Context, bucket, prefix string, set *erasureObjects
 			}
 
 			if err := healEntry(*entry); err != nil {
-				errCh <- err
+				logger.LogIf(ctx, err)
+				cancel()
 				return
 			}
 		},
@@ -1784,13 +1784,13 @@ func listAndHeal(ctx context.Context, bucket, prefix string, set *erasureObjects
 	}
 
 	if err := listPathRaw(ctx, lopts); err != nil {
-		errCh <- fmt.Errorf("listPathRaw returned %w: opts(%#v)", err, lopts)
-		return
+		return fmt.Errorf("listPathRaw returned %w: opts(%#v)", err, lopts)
 	}
+
+	return nil
 }
 
 func (z *erasureServerPools) HealObjects(ctx context.Context, bucket, prefix string, opts madmin.HealOpts, healObjectFn HealObjectFn) error {
-	errCh := make(chan error)
 	healEntry := func(entry metaCacheEntry) error {
 		if entry.isDir() {
 			return nil
@@ -1829,34 +1829,33 @@ func (z *erasureServerPools) HealObjects(ctx context.Context, bucket, prefix str
 	ctx, cancel := context.WithCancel(ctx)
 	defer cancel()
 
-	go func() {
-		defer close(errCh)
-
+	var poolErrs [][]error
+	for idx, erasureSet := range z.serverPools {
+		if z.IsSuspended(idx) {
+			continue
+		}
+		errs := make([]error, len(erasureSet.sets))
 		var wg sync.WaitGroup
-		for idx, erasureSet := range z.serverPools {
-			if z.IsSuspended(idx) {
-				continue
-			}
-			for _, set := range erasureSet.sets {
-				wg.Add(1)
-				go func(set *erasureObjects) {
-					defer wg.Done()
+		for idx, set := range erasureSet.sets {
+			wg.Add(1)
+			go func(idx int, set *erasureObjects) {
+				defer wg.Done()
 
-					listAndHeal(ctx, bucket, prefix, set, healEntry, errCh)
-				}(set)
-			}
+				errs[idx] = listAndHeal(ctx, bucket, prefix, set, healEntry)
+			}(idx, set)
 		}
 		wg.Wait()
-	}()
-	var err error
-	for e := range errCh {
-		// Save first non-nil error.
-		if e != nil && err != nil {
-			err = e
-			cancel()
+		poolErrs = append(poolErrs, errs)
+	}
+	for _, errs := range poolErrs {
+		for _, err := range errs {
+			if err == nil {
+				continue
+			}
+			return err
 		}
 	}
-	return err
+	return nil
 }
 
 func (z *erasureServerPools) HealObject(ctx context.Context, bucket, object, versionID string, opts madmin.HealOpts) (madmin.HealResultItem, error) {


Test patch: