Repository: minio/minio (language: go)

Issue:
Insufficient number of disks online for ListBuckets
At startup the server is "Unable to list buckets to heal: Read failed. Insufficient number of disks online". `mc admin info` is all green. Interactions with a bucket works. But `mc ls myserver`  fails, and we can't log into the web console because of the same error.

## Expected Behavior
Should work properly because all drives are online and reachable.

## Current Behavior
- Can't list buckets.
- Can't use the web console.

## Steps to Reproduce (for bugs)
1. Upgrade from `RELEASE.2022-01-08T03-11-54Z` to `RELEASE.2022-07-30T05-21-40Z`. (Or else, I didn't try).
2. `mc ls cold`
3.  `mc admin info cold --json`
```
mc: <ERROR> Unable to list folder. We encountered an internal error, please try again.: cause(Read failed. Insufficient number of disks online)
```
```json
{
  "status": "success",
  "info": {
    "mode": "online",
    "domain": ["cold-objects.liiib.re", "minio-cold.svc.cluster.local"],
    "deploymentID": "6ccb138d-7f98-4197-9617-bf5671b0cb3f",
    "buckets": { "count": 91 },
    "objects": { "count": 674953 },
    "versions": { "count": 0 },
    "usage": { "size": 2365564670348 },
    "services": {
      "kms": { "status": "disabled" },
      "ldap": {},
      "audit": [{ "": { "status": "online" } }]
    },
    "backend": {
      "backendType": "Erasure",
      "onlineDisks": 10,
      "rrSCParity": 2,
      "standardSCParity": 5
    },
    "servers": [
      {
        "state": "online",
        "endpoint": "cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000",
        "uptime": 2355,
        "version": "2022-07-30T05:21:40Z",
        "commitID": "e6eab2091f632622fadbe3b25bb4732125b2d5df",
        "network": {
          "cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000": "online"
        },
        "drives": [
          {
            "endpoint": "https://cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000/export0",
            "path": "/export0",
            "state": "ok",
            "uuid": "863b28b0-9fb8-4bf5-b21a-ddfc942865ff",
            "totalspace": 1999422177280,
            "usedspace": 492754219008,
            "availspace": 1506667958272,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 3, "acc_time_ns": 68522615 },
                "ReadXL": { "count": 88, "acc_time_ns": 59321080 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 3,
                "CreateFile": 15,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 3,
                "ReadMultiple": 0,
                "ReadVersion": 717,
                "ReadXL": 4426,
                "RenameData": 29,
                "RenameFile": 0,
                "StatInfoFile": 0,
                "StatVol": 33,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193367130,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 0
          },
          {
            "endpoint": "https://cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000/export1",
            "path": "/export1",
            "state": "ok",
            "uuid": "d917f9a2-28a4-4eef-9ce7-16676f22c638",
            "totalspace": 1999422177280,
            "usedspace": 492770521088,
            "availspace": 1506651656192,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 3, "acc_time_ns": 49382338 },
                "ReadXL": { "count": 88, "acc_time_ns": 35144390 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 3,
                "CreateFile": 15,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 3,
                "ReadMultiple": 0,
                "ReadVersion": 717,
                "ReadXL": 4426,
                "RenameData": 29,
                "RenameFile": 0,
                "StatInfoFile": 0,
                "StatVol": 33,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193367034,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 5
          }
        ],
        "poolNumber": 1,
        "mem_stats": {
          "Alloc": 503876424,
          "TotalAlloc": 42245016928,
          "Mallocs": 514867790,
          "Frees": 512458640,
          "HeapAlloc": 503876424
        }
      },
      {
        "state": "online",
        "endpoint": "cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000",
        "uptime": 2277,
        "version": "2022-07-30T05:21:40Z",
        "commitID": "e6eab2091f632622fadbe3b25bb4732125b2d5df",
        "network": {
          "cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000": "online"
        },
        "drives": [
          {
            "endpoint": "https://cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000/export0",
            "path": "/export0",
            "state": "ok",
            "uuid": "7286fd5f-5236-4b08-86a3-9a8656b0910c",
            "totalspace": 1999422177280,
            "usedspace": 492790312960,
            "availspace": 1506631864320,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 2, "acc_time_ns": 32836705 },
                "ReadXL": { "count": 88, "acc_time_ns": 193056199 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 3,
                "CreateFile": 9,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 3,
                "ReadMultiple": 0,
                "ReadVersion": 716,
                "ReadXL": 4315,
                "RenameData": 20,
                "RenameFile": 1,
                "StatInfoFile": 0,
                "StatVol": 25,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193366872,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 1
          },
          {
            "endpoint": "https://cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000/export1",
            "path": "/export1",
            "state": "ok",
            "uuid": "b0a51fe5-487f-4106-9190-a990475fb7ab",
            "totalspace": 1999422177280,
            "usedspace": 492790165504,
            "availspace": 1506632011776,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 2, "acc_time_ns": 39966338 },
                "ReadXL": { "count": 88, "acc_time_ns": 63284321 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 3,
                "CreateFile": 9,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 2,
                "ReadMultiple": 0,
                "ReadVersion": 716,
                "ReadXL": 4315,
                "RenameData": 20,
                "RenameFile": 1,
                "StatInfoFile": 0,
                "StatVol": 25,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193366877,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 6
          }
        ],
        "poolNumber": 1,
        "mem_stats": {
          "Alloc": 555301360,
          "TotalAlloc": 9896781800,
          "Mallocs": 117514642,
          "Frees": 114365775,
          "HeapAlloc": 555301360
        }
      },
      {
        "state": "online",
        "endpoint": "cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000",
        "uptime": 2351,
        "version": "2022-07-30T05:21:40Z",
        "commitID": "e6eab2091f632622fadbe3b25bb4732125b2d5df",
        "network": {
          "cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000": "online"
        },
        "drives": [
          {
            "endpoint": "https://cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000/export0",
            "path": "/export0",
            "state": "ok",
            "uuid": "ad4f751c-55c3-4773-8cd9-3468698d2671",
            "totalspace": 1999422177280,
            "usedspace": 493674569728,
            "availspace": 1505747607552,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 6, "acc_time_ns": 93021219 },
                "ReadXL": { "count": 84, "acc_time_ns": 491778379 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 5,
                "CreateFile": 10,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 4,
                "ReadMultiple": 0,
                "ReadVersion": 740,
                "ReadXL": 4313,
                "RenameData": 27,
                "RenameFile": 1,
                "StatInfoFile": 0,
                "StatVol": 31,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193360921,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 2
          },
          {
            "endpoint": "https://cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000/export1",
            "path": "/export1",
            "state": "ok",
            "uuid": "b30d1c13-6297-4a02-9e28-9b402c7986c3",
            "totalspace": 1999422177280,
            "usedspace": 493668155392,
            "availspace": 1505754021888,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 6, "acc_time_ns": 89417586 },
                "ReadXL": { "count": 84, "acc_time_ns": 80697037 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 5,
                "CreateFile": 10,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 3,
                "ReadMultiple": 0,
                "ReadVersion": 740,
                "ReadXL": 4313,
                "RenameData": 27,
                "RenameFile": 1,
                "StatInfoFile": 0,
                "StatVol": 31,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193360929,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 7
          }
        ],
        "poolNumber": 1,
        "mem_stats": {
          "Alloc": 524474752,
          "TotalAlloc": 10185364440,
          "Mallocs": 121652554,
          "Frees": 118887339,
          "HeapAlloc": 524474752
        }
      },
      {
        "state": "online",
        "endpoint": "cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000",
        "uptime": 2150,
        "version": "2022-07-30T05:21:40Z",
        "commitID": "e6eab2091f632622fadbe3b25bb4732125b2d5df",
        "network": {
          "cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000": "online"
        },
        "drives": [
          {
            "endpoint": "https://cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000/export0",
            "path": "/export0",
            "state": "ok",
            "uuid": "28d6bb70-e581-414a-87c1-028dc2d08189",
            "totalspace": 1999422177280,
            "usedspace": 492859105280,
            "availspace": 1506563072000,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 5, "acc_time_ns": 65529520 },
                "ReadXL": { "count": 88, "acc_time_ns": 48888534 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 1,
                "CreateFile": 11,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 1,
                "ReadMultiple": 0,
                "ReadVersion": 697,
                "ReadXL": 4130,
                "RenameData": 23,
                "RenameFile": 0,
                "StatInfoFile": 0,
                "StatVol": 27,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193366447,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 3
          },
          {
            "endpoint": "https://cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000/export1",
            "path": "/export1",
            "state": "ok",
            "uuid": "2ad7fb8a-a96a-4cba-b084-aabcc23877da",
            "totalspace": 1999422177280,
            "usedspace": 492859097088,
            "availspace": 1506563080192,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 5, "acc_time_ns": 60499273 },
                "ReadXL": { "count": 88, "acc_time_ns": 114468842 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 1,
                "CreateFile": 11,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 1,
                "ReadMultiple": 0,
                "ReadVersion": 697,
                "ReadXL": 4130,
                "RenameData": 23,
                "RenameFile": 0,
                "StatInfoFile": 0,
                "StatVol": 27,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193366456,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 8
          }
        ],
        "poolNumber": 1,
        "mem_stats": {
          "Alloc": 350377936,
          "TotalAlloc": 9375408504,
          "Mallocs": 111676062,
          "Frees": 111149245,
          "HeapAlloc": 350377936
        }
      },
      {
        "state": "online",
        "endpoint": "cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000",
        "uptime": 2145,
        "version": "2022-07-30T05:21:40Z",
        "commitID": "e6eab2091f632622fadbe3b25bb4732125b2d5df",
        "network": {
          "cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000": "online",
          "cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000": "online"
        },
        "drives": [
          {
            "endpoint": "https://cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000/export0",
            "path": "/export0",
            "state": "ok",
            "uuid": "3ce93a7a-1fc5-46c8-82bf-7d199da935cd",
            "totalspace": 1999422177280,
            "usedspace": 492668305408,
            "availspace": 1506753871872,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 3, "acc_time_ns": 40520831 },
                "ReadXL": { "count": 91, "acc_time_ns": 130619459 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 2,
                "CreateFile": 14,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 2,
                "ReadMultiple": 0,
                "ReadVersion": 713,
                "ReadXL": 4116,
                "RenameData": 26,
                "RenameFile": 0,
                "StatInfoFile": 0,
                "StatVol": 30,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193366923,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 4
          },
          {
            "endpoint": "https://cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000/export1",
            "path": "/export1",
            "state": "ok",
            "uuid": "7067e1b6-84f7-4106-b6e5-efdaf661e470",
            "totalspace": 1999422177280,
            "usedspace": 492661469184,
            "availspace": 1506760708096,
            "metrics": {
              "lastMinute": {
                "ReadVersion": { "count": 3, "acc_time_ns": 46955906 },
                "ReadXL": { "count": 91, "acc_time_ns": 73400250 }
              },
              "apiCalls": {
                "AppendFile": 0,
                "CheckParts": 2,
                "CreateFile": 14,
                "Delete": 0,
                "DeleteVersion": 0,
                "DeleteVersions": 0,
                "DeleteVol": 0,
                "ListDir": 0,
                "ListVols": 0,
                "MakeVol": 0,
                "MakeVolBulk": 0,
                "ReadAll": 2,
                "ReadFile": 0,
                "ReadFileStream": 2,
                "ReadMultiple": 0,
                "ReadVersion": 713,
                "ReadXL": 4116,
                "RenameData": 26,
                "RenameFile": 0,
                "StatInfoFile": 0,
                "StatVol": 30,
                "UpdateMetadata": 0,
                "VerifyFile": 0,
                "WalkDir": 5,
                "WriteAll": 0,
                "WriteMetadata": 0
              }
            },
            "free_inodes": 193366935,
            "pool_index": 0,
            "set_index": 0,
            "disk_index": 9
          }
        ],
        "poolNumber": 1,
        "mem_stats": {
          "Alloc": 533376464,
          "TotalAlloc": 9463903224,
          "Mallocs": 111856634,
          "Frees": 109045018,
          "HeapAlloc": 533376464
        }
      }
    ]
  }
}
```

## Context
We just updated our cluster to the latest release (RELEASE.2022-07-30T05-21-40Z) from RELEASE.2022-01-08T03-11-54Z. We're using the latest version of the operator (v4.4.27) on k8s (v1.21).

On Slack @prakashsvmx asked for the disks permissions :
```
drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export0
drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export1

drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export0
drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export1

drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export0
drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export1

drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export0
drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export1

drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export0
drwxrwsr-x  94 root 1000 4096 Jul 22 09:39 export1
```

## Regression
It's possible.

## Your Environment
* Version used:  RELEASE.2022-07-30T05-21-40Z
* Server setup and configuration: k8s, 5 servers of 2 disks.
* Operating System and version: Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux



Discussion / hints:
> ```
> mc: <ERROR> Unable to list folder. We encountered an internal error, please try again.: cause(Read failed. Insufficient number of disks online)
> ```

Please share server logs @hrenard 
we have exact the same problem. We upgraded a 4 node ( 1 disk per node) cluster from `RELEASE.2022-03-24T00-43-44Z` to the latest release `RELEASE.2022-07-29T19-40-48Z` and are facing the same issue. Every disk is green in `mc admin info`.
I already discussed this issue on slack and there seem to be no configuration issue or something like that. Not suspecious in the server logs either.
```
Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 114
All MinIO sub-systems initialized successfully in 75.27889ms
Waiting for all MinIO IAM sub-system to be initialized.. lock acquired

API: SYSTEM()
Time: 11:55:05 UTC 08/01/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
Error: Unable to list buckets to heal: Read failed. Insufficient number of disks online (*fmt.wrapError)
       2: internal/logger/logger.go:259:logger.LogIf()
       1: cmd/server-main.go:612:cmd.serverMain.func4()
MinIO Object Storage Server
Status:         4 Online, 0 Offline.
API: http://10.244.1.83:9000/  http://127.0.0.1:9000/
Console: https://minio.k8s.*****.zone/

Documentation: https://docs.min.io/

 You are running an older version of MinIO released 9 hours ago
 Update: Run `mc admin update`
```
although everything looks ok:
```
# mc admin info sternsinger
●  s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000
   Uptime: 1 day
   Version: 2022-07-29T19:40:48Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000
   Uptime: 5 hours
   Version: 2022-07-29T19:40:48Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  s3-sternsinger-app-2.s3-sternsinger-app-svc.default.svc.cluster.local:9000
   Uptime: 1 day
   Version: 2022-07-29T19:40:48Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000
   Uptime: 1 day
   Version: 2022-07-29T19:40:48Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

Pools:
   1st, Erasure sets: 1, Disks per erasure set: 4

15 GiB Used, 3 Buckets, 6,012 Objects
4 drives online, 0 drives offline
```
a bit strange is that`mc admin trace -a` is writing a lot of lockR.lock events like:
```
2022-08-02T15:46:28.187 [403 Forbidden] lockR.Lock s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       717µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.033 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       847µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.111 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       905µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.186 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       721µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.646 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       1.046ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.695 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       1.018ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.924 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       1.143ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.646 [403 Forbidden] lockR.Lock s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       576µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.696 [403 Forbidden] lockR.Lock s3-sternsinger-app-2.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       321µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.924 [403 Forbidden] lockR.Lock s3-sternsinger-app-2.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       372µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.646 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       928µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.695 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       1.041ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:28.924 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       1.054ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:29.300 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       877µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:29.300 [403 Forbidden] lockR.Lock s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       654µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:29.299 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       886µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:29.360 [OS] os.Open s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000 /export/.minio.sys/tmp/.trash 69.852µs
2022-08-02T15:46:29.755 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       945µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:29.756 [403 Forbidden] lockR.Lock s3-sternsinger-app-2.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       406µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:29.756 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       948µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.245 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       940µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.246 [403 Forbidden] lockR.Lock s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       687µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.246 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       830µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.516 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       1.027ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.589 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       1.135ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.985 [403 Forbidden] lockR.Lock s3-sternsinger-app-1.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       1.207ms      ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.590 [403 Forbidden] lockR.Lock s3-sternsinger-app-2.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       517µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.986 [403 Forbidden] lockR.Lock s3-sternsinger-app-2.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       406µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.516 [403 Forbidden] lockR.Lock s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       340µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.516 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.54       750µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.589 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       752µs       ↑ 296 B ↓ 251 B
2022-08-02T15:46:30.986 [403 Forbidden] lockR.Lock s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000/minio/lock/v7/lock 10.244.3.56       653µs       ↑ 296 B ↓ 251 B
```

but `mc admin top locks <alias>` is just:
```
Time                  Type    Resource
5 hours               WRITE   .minio.sys/scanner/runDataScanner.lock
```
btw interaction on the bucket itself is working fine, so `mc ls <alias>` gives the error @hrenard reported but `mc ls <alias>/<bucket>` is working fine, and we can even access the files, upload etc.
if it helps, this is the link to the slack discussion: https://minio.slack.com/archives/C3NDUB8UA/p1659352543806919
Going to hide some unnecessary information posted here @obi12341 - when you say exact same issue do you see `mc ls alias` what do you see in the server logs? 


```
cold-pool-0-0 
cold-pool-0-0 API: SYSTEM()
cold-pool-0-0 Time: 16:33:56 UTC 08/02/2022
cold-pool-0-0 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-0 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-0        3: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-0        2: cmd/data-scanner.go:214:cmd.runDataScanner()
cold-pool-0-0        1: cmd/data-scanner.go:79:cmd.initDataScanner.func1()
cold-pool-0-4 
cold-pool-0-4 API: ListBuckets()
cold-pool-0-4 Time: 16:34:55 UTC 08/02/2022
cold-pool-0-4 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-4 RequestID: 17079458971C659C
cold-pool-0-4 RemoteHost: 10.244.30.1
cold-pool-0-4 Host: cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-4 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-4 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-4        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-4        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-4        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-4        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-4        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-2 
cold-pool-0-2 API: ListBuckets()
cold-pool-0-2 Time: 16:34:55 UTC 08/02/2022
cold-pool-0-2 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-2 RequestID: 1707945899DA0B8C
cold-pool-0-2 RemoteHost: 10.244.30.1
cold-pool-0-2 Host: cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-2 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-2 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-2        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-2        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-2        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-2        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-2        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-0 
cold-pool-0-0 API: ListBuckets()
cold-pool-0-0 Time: 16:34:56 UTC 08/02/2022
cold-pool-0-0 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-0 RequestID: 17079458A9445E99
cold-pool-0-0 RemoteHost: 10.244.30.1
cold-pool-0-0 Host: cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-0 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-0 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-0        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-0        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-1 
cold-pool-0-0        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-1 API: ListBuckets()
cold-pool-0-0        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-1 Time: 16:34:56 UTC 08/02/2022
cold-pool-0-0        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-1 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-1 RequestID: 17079458AB4B7B54
cold-pool-0-1 RemoteHost: 10.244.30.1
cold-pool-0-1 Host: cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-1 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-1 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-1        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-1        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-1        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-1        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-1        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-0 
cold-pool-0-0 API: SYSTEM()
cold-pool-0-0 Time: 16:34:56 UTC 08/02/2022
cold-pool-0-0 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-0 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-0        3: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-0        2: cmd/data-scanner.go:214:cmd.runDataScanner()
cold-pool-0-0        1: cmd/data-scanner.go:79:cmd.initDataScanner.func1()
cold-pool-0-3 
cold-pool-0-3 API: ListBuckets()
cold-pool-0-3 Time: 16:34:57 UTC 08/02/2022
cold-pool-0-3 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-3 RequestID: 17079458D7A7D528
cold-pool-0-3 RemoteHost: 10.244.30.1
cold-pool-0-3 Host: cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-3 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-3 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-3        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-3        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-3        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-3        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-3        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-4 
cold-pool-0-4 API: ListBuckets()
cold-pool-0-4 Time: 16:34:57 UTC 08/02/2022
cold-pool-0-4 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-4 RequestID: 17079458EFF9326E
cold-pool-0-4 RemoteHost: 10.244.30.1
cold-pool-0-4 Host: cold-pool-0-4.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-4 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-4 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-4        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-4        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-4        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-4        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-4        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-2 
cold-pool-0-2 API: ListBuckets()
cold-pool-0-2 Time: 16:34:57 UTC 08/02/2022
cold-pool-0-2 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-2 RequestID: 1707945903755666
cold-pool-0-0 
cold-pool-0-0 API: ListBuckets()
cold-pool-0-0 Time: 16:34:57 UTC 08/02/2022
cold-pool-0-0 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-0 RequestID: 1707945905D36F97
cold-pool-0-0 RemoteHost: 10.244.30.1
cold-pool-0-0 Host: cold-pool-0-0.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-0 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-2 RemoteHost: 10.244.30.1
cold-pool-0-2 Host: cold-pool-0-2.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-2 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-2 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-2        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-2        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-2        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-2        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-2        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-0 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-0        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-0        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-0        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-0        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-0        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-1 
cold-pool-0-1 API: ListBuckets()
cold-pool-0-1 Time: 16:34:58 UTC 08/02/2022
cold-pool-0-1 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-1 RequestID: 170794593DC8283B
cold-pool-0-1 RemoteHost: 10.244.30.1
cold-pool-0-1 Host: cold-pool-0-1.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-1 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-1 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-1        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-1        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-1        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-1        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-1        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
cold-pool-0-3 
cold-pool-0-3 API: ListBuckets()
cold-pool-0-3 Time: 16:34:59 UTC 08/02/2022
cold-pool-0-3 DeploymentID: 6ccb138d-7f98-4197-9617-bf5671b0cb3f
cold-pool-0-3 RequestID: 17079459580F0D2F
cold-pool-0-3 RemoteHost: 10.244.30.1
cold-pool-0-3 Host: cold-pool-0-3.cold-hl.minio-cold.svc.cluster.local:9000
cold-pool-0-3 UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-29T19-17-16Z
cold-pool-0-3 Error: Read failed. Insufficient number of disks online (*errors.errorString)
cold-pool-0-3        5: internal/logger/logger.go:259:logger.LogIf()
cold-pool-0-3        4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
cold-pool-0-3        3: cmd/api-errors.go:2160:cmd.toAPIError()
cold-pool-0-3        2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
cold-pool-0-3        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()

```
```
# mc ls alias <alias>
mc: <ERROR> Unable to list folder. We encountered an internal error, please try again.: cause(Read failed. Insufficient number of disks online)
```

Log:
```
API: ListBuckets()
Time: 16:35:27 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
RequestID: 1707945FEC94FC85
RemoteHost: 80.82.212.226
Host: s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000
UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-24T02-25-13Z
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       5: internal/logger/logger.go:259:logger.LogIf()
       4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
       3: cmd/api-errors.go:2160:cmd.toAPIError()
       2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()

API: ListBuckets()
Time: 16:35:29 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
RequestID: 17079460547BF345
RemoteHost: 80.82.212.226
Host: s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000
UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-24T02-25-13Z
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       5: internal/logger/logger.go:259:logger.LogIf()
       4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
       3: cmd/api-errors.go:2160:cmd.toAPIError()
       2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()

API: ListBuckets()
Time: 16:35:31 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
RequestID: 17079460EB31DE10
RemoteHost: 80.82.212.226
Host: s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000
UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-24T02-25-13Z
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       5: internal/logger/logger.go:259:logger.LogIf()
       4: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
       3: cmd/api-errors.go:2160:cmd.toAPIError()
       2: cmd/bucket-handlers.go:337:cmd.objectAPIHandlers.ListBucketsHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
```      
it is not specified to ListBuckets:
```
API: AccountInfo()
Time: 10:51:40 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
RequestID: 1707819D6EBD19D4
RemoteHost: 10.244.3.56
Host: s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000
UserAgent: MinIO (linux; amd64) madmin-go/0.0.1
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
       4: cmd/admin-handler-utils.go:233:cmd.toAdminAPIErrCode()
       3: cmd/admin-handler-utils.go:220:cmd.toAdminAPIErr()
       2: cmd/admin-handlers-users.go:1182:cmd.adminAPIHandlers.AccountInfoHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
```
Thats the log entry if you try to login in console
Thanks this helps @obi12341 
> API: AccountInfo()
> Time: 10:51:40 UTC 08/02/2022
> DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
> RequestID: 1707819D6EBD19D4
> RemoteHost: 10.244.3.56
> Host: s3-sternsinger-app-0.s3-sternsinger-app-svc.default.svc.cluster.local:9000
> UserAgent: MinIO (linux; amd64) madmin-go/0.0.1
> Error: Read failed. Insufficient number of disks online (*errors.errorString)
>        6: internal/logger/logger.go:259:logger.LogIf()
>        5: cmd/api-errors.go:2135:cmd.toAPIErrorCode()
>        4: cmd/admin-handler-utils.go:233:cmd.toAdminAPIErrCode()
>        3: cmd/admin-handler-utils.go:220:cmd.toAdminAPIErr()
>        2: cmd/admin-handlers-users.go:1182:cmd.adminAPIHandlers.AccountInfoHandler()
>        1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()

This call does internally ListBuckets()
ah makes sense, ok than probably just a ListBuckets() problem
are you both on k8s? @obi12341 @hrenard ?
`Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}`
This looks like some permission issue at the root of the mount, the new operator defaults to securityContext() which is causing this problem. 

https://github.com/minio/operator/blob/master/docs/upgrade.md#upgrading-from-minio-operator-v422-to-v423-

hmm i have seen this, but i am using the helm charts which have by default: https://github.com/minio/minio/blob/master/helm/minio/values.yaml#L238
All files are owned by uid/gid 1000
```
drwxrwsr-x    7 root 1000   4096 Jul 29 13:59 .
drwxr-xr-x    1 root root   4096 Aug  1 14:50 ..
drwxr-sr-x    9 1000 1000   4096 Aug  1 14:50 .minio.sys
drwxrws---    2 root 1000  16384 Mar 31 13:11 lost+found
drwxr-sr-x 1002 1000 1000  69632 Jul 29 13:36 ***bucket1***
drwxr-sr-x 2459 1000 1000 155648 Jul 29 14:45 ***bucket2***
drwxr-sr-x    3 1000 1000   4096 Apr  5 20:38 ***bucket3***
```
```
$ ls -l .minio.sys/
total 32
drwxr-sr-x 9 1000 1000 4096 Aug  1 09:27 buckets
drwxr-sr-x 5 1000 1000 4096 Jul 29 12:43 config
-rw-r--r-- 1 1000 1000  342 Mar 31 13:12 format.json
drwx--S--- 2 1000 1000 4096 Mar 31 13:12 ilm
drwxr-sr-x 2 1000 1000 4096 Mar 31 13:11 multipart
drwxr-sr-x 2 1000 1000 4096 Mar 31 13:35 pool.bin
drwxr-sr-x 3 1000 1000 4096 Aug  2 10:51 tmp
drwxr-sr-x 2 1000 1000 4096 Aug  1 14:50 tmp-old
bash-4.4$ ls -l .minio.sys/buckets/
total 12
drwxr-sr-x 4 1000 1000 4096 Jul 29 12:56 ***bucket1***
drwxr-sr-x 4 1000 1000 4096 Jul 29 17:42 ***bucket2***
drwxr-sr-x 4 1000 1000 4096 Apr  5 20:21 ***bucket3***
```
I have this in the stateful set :
```yaml
securityContext:
  fsGroup: 1000
  fsGroupChangePolicy: OnRootMismatch
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000
```
Should the pods run as root ?
Children are owned by `1000:1000` but the root folder (`/export`) is owned by `0:1000`.
I migrated the operator from `v4.4.3` to `v4.4.27`, so security context have not changed, I think. 
what is failing here is "readdir" on the root of the FS - now in-turn, why that is failing, is a bit strange, I looked if this is an upgrade on a local setup that works fine.
Interesting. K8s hasn't been upgraded, and I don't recall upgrading the CSI. Maybe some Debian packets, on the host, have been updated.
so if i get this right, if we change to:
```
      securityContext:
        runAsUser: 0
        runAsGroup: 0
        runAsNonRoot: false
        fsGroup: 0
```
it should work? Beside the fact that minio is running as a root process and probably decreasing security?
Thanks btw for looking into this :)
(I have another tenant, that I didn't upgrade that is still running fine after the operator upgrade).
> (I have another tenant, that I didn't upgrade that is still running fine after the operator upgrade).

If you didn't upgrade MinIO it won't change the securityContext - buckets listing is just a "readdir()" on the disk and returns an error

```
// listAllBuckets lists all buckets from all disks. It also
// returns the occurrence of each buckets in all disks
func listAllBuckets(ctx context.Context, storageDisks []StorageAPI, healBuckets map[string]VolInfo, readQuorum int) error {
        g := errgroup.WithNErrs(len(storageDisks))
        var mu sync.Mutex
        for index := range storageDisks {
                index := index
                g.Go(func() error {
                        if storageDisks[index] == nil {
                                // we ignore disk not found errors
                                return nil
                        }
                        volsInfo, err := storageDisks[index].ListVols(ctx)
                        if err != nil {
                                return err
                        }
```

The only place where ListBuckets will return an error to be calculated as per quorum requirements

So corresponding lower layer calls that would return an error is just this `readDir` can fail. 

```// List all the volumes from diskPath.
func listVols(dirPath string) ([]VolInfo, error) {
        if err := checkPathLength(dirPath); err != nil {
                return nil, err
        }
        entries, err := readDir(dirPath)
        if err != nil {
                return nil, errDiskNotFound
        }
```

So in essence root of your disks are return an error what is this error is hard to know since this is not logged. 

The only thing that changed in readDir() is to avoid "access time" 

```
commit 2f1ef02d35609581bc20617e9e1a0c5d24692f8a
Author: Anis Elleuch <vadmeste@users.noreply.github.com>
Date:   Thu Jul 28 20:40:48 2022 +0100

    Do not update directory access time (#15426)
    
    Most setups will have relatime it only updates the access time
    following a change in the directory.

diff --git a/cmd/os_unix.go b/cmd/os_unix.go
index 782702502..96f77f5ff 100644
--- a/cmd/os_unix.go
+++ b/cmd/os_unix.go
@@ -232,7 +232,7 @@ func readDirFn(dirPath string, fn func(name string, typ os.FileMode) error) erro
 // Return count entries at the directory dirPath and all entries
 // if count is set to -1
 func readDirWithOpts(dirPath string, opts readDirOpts) (entries []string, err error) {
-       f, err := Open(dirPath)
+       f, err := OpenFile(dirPath, readMode, 0)
        if err != nil {
                return nil, osErrToFileErr(err)
        }
```

> so if i get this right, if we change to:
> 
> ```
>       securityContext:
>         runAsUser: 0
>         runAsGroup: 0
>         runAsNonRoot: false
>         fsGroup: 0
> ```
> 
> it should work? Beside the fact that minio is running as a root process and probably decreasing security? Thanks btw for looking into this :)

That would be something to try. There is no security you get here your containers are already isolated - root inside container is not root on the host system. 
Would you be interested in some extra logs @obi12341 @hrenard ? I can give a new container that would help us see more details. 
Please use this 

```
minio/minio:RELEASE.2022-07-30T05-21-40Z.test.fdec67a59
```

and let me know. 
Indeed, your patch shouldn't change anything (https://cs.opensource.google/go/go/+/master:src/os/file.go;l=317).
Yes, I'll be glad to help you debug this. Although it's a production cluster. Did you just add logs ?
> Yes, I'll be glad to help you debug this. Although it's a production cluster. Did you just add logs ?

Yes just logs

```diff
diff --git a/cmd/xl-storage.go b/cmd/xl-storage.go
index f848b85ea..b8e5ab036 100644
--- a/cmd/xl-storage.go
+++ b/cmd/xl-storage.go
@@ -751,17 +751,18 @@ func (s *xlStorage) MakeVol(ctx context.Context, volume string) error {
 }
 
 // ListVols - list volumes.
-func (s *xlStorage) ListVols(context.Context) (volsInfo []VolInfo, err error) {
-       return listVols(s.diskPath)
+func (s *xlStorage) ListVols(ctx context.Context) (volsInfo []VolInfo, err error) {
+       return listVols(ctx, s.diskPath)
 }
 
 // List all the volumes from diskPath.
-func listVols(dirPath string) ([]VolInfo, error) {
+func listVols(ctx context.Context, dirPath string) ([]VolInfo, error) {
        if err := checkPathLength(dirPath); err != nil {
                return nil, err
        }
        entries, err := readDir(dirPath)
        if err != nil {
+               logger.LogIf(ctx, err)
                return nil, errDiskNotFound
        }
        volsInfo := make([]VolInfo, 0, len(entries))
```


get following problem:
Failed to pull image "quay.io/minio/minio:RELEASE.2022-07-30T05-21-40Z.test.fdec67a59"
is this located elsewhere?
Its not in `quay.io` its at `docker.io/` @obi12341 
```
API: SYSTEM()
Time: 18:37:16 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
Error: file access denied (cmd.StorageErr)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/xl-storage.go:765:cmd.listVols()
       4: cmd/xl-storage.go:755:cmd.(*xlStorage).ListVols()
       3: cmd/xl-storage-disk-id-check.go:283:cmd.(*xlStorageDiskIDCheck).ListVols()
       2: cmd/storage-rest-server.go:279:cmd.(*storageRESTServer).ListVolsHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
```
See what I meant? :-) 
Okay, one more container coming @obi12341 - this may be a umask issue. 
yeah i see, still a bit strange, everything is owned by 1000, no special configuration at the helm values.yaml. I am using as much as possible the default values. Nothing has changed at the file level expect deploying new helm chart version with newest image tag. Will try to set securityContext to 0 but still feels kind of unintended :P 
minio/minio:RELEASE.2022-07-30T05-21-40Z.test.6e77b7ea
Very weird. You want to try reverting your patch `OpenFile` to exclude it from a potential source of the issue ?
```
API: SYSTEM()
Time: 18:51:27 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
Error: file access denied (cmd.StorageErr)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/xl-storage.go:765:cmd.listVols()
       4: cmd/xl-storage.go:755:cmd.(*xlStorage).ListVols()
       3: cmd/xl-storage-disk-id-check.go:283:cmd.(*xlStorageDiskIDCheck).ListVols()
       2: cmd/erasure-healing.go:221:cmd.listAllBuckets.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

API: SYSTEM()
Time: 18:51:27 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       3: internal/logger/logger.go:259:logger.LogIf()
       2: cmd/data-scanner.go:214:cmd.runDataScanner()
       1: cmd/data-scanner.go:79:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 18:51:38 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
Error: file access denied (cmd.StorageErr)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/xl-storage.go:765:cmd.listVols()
       4: cmd/xl-storage.go:755:cmd.(*xlStorage).ListVols()
       3: cmd/xl-storage-disk-id-check.go:283:cmd.(*xlStorageDiskIDCheck).ListVols()
       2: cmd/storage-rest-server.go:279:cmd.(*storageRESTServer).ListVolsHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()

API: ListBuckets()
Time: 18:51:38 UTC 08/02/2022
DeploymentID: df056ba6-d022-4dca-86a0-b03302c51b60
RequestID: 17079BCE611F5F8F
RemoteHost: 80.82.212.226
Host: s3-sternsinger-app-3.s3-sternsinger-app-svc.default.svc.cluster.local:9000
UserAgent: MinIO (linux; amd64) minio-go/v7.0.31 mc/RELEASE.2022-07-24T02-25-13Z
Error: file access denied (cmd.StorageErr)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/xl-storage.go:765:cmd.listVols()
       4: cmd/xl-storage.go:755:cmd.(*xlStorage).ListVols()
       3: cmd/xl-storage-disk-id-check.go:283:cmd.(*xlStorageDiskIDCheck).ListVols()
       2: cmd/erasure-healing.go:221:cmd.listAllBuckets.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
```
sadly nothing changed
Okay so it is not umask it basically `OpenDir` is failing at the top level export - try changing the securityContex @obi12341 
> Very weird. You want to try reverting your patch `OpenFile` to exclude it from a potential source of the issue ?

No it doesn't seem like it - this simply looks like `root:1000` issue at the top level mount. 
hmm will do but that user is root of volume even with securityContext seems to be normal, checked it at a redis cluster:
`drwxrwsrwx    2 root     redis         4096 Aug  2 10:49 data`
Yeah, my other (working) tenant have the same permissions :
```
drwxrwsr-x 181 root 1000 8192 Jul 22 09:39 export
```
Did you change the version of golang ? Maybe the low level libs have changed... (Just throwing and idea)
```
securityContext:
  runAsUser: 0
  runAsGroup: 0
  fsGroup: 0
```
i can confirm this works. It is still a bit "meeh why didn't the other stuff worked" but yeah it is isolated in containers and probably not a big deal.

note: this is for a helm deployment, for operator set the stuff in here: https://github.com/minio/operator/blob/master/docs/upgrade.md#upgrading-from-minio-operator-v422-to-v423-

Fix patch:
diff --git a/cmd/os_unix.go b/cmd/os_unix.go
index 96f77f5ff4aad..b9e18e620c26a 100644
--- a/cmd/os_unix.go
+++ b/cmd/os_unix.go
@@ -232,9 +232,16 @@ func readDirFn(dirPath string, fn func(name string, typ os.FileMode) error) erro
 // Return count entries at the directory dirPath and all entries
 // if count is set to -1
 func readDirWithOpts(dirPath string, opts readDirOpts) (entries []string, err error) {
-	f, err := OpenFile(dirPath, readMode, 0)
+	f, err := OpenFile(dirPath, readMode, 0o666)
 	if err != nil {
-		return nil, osErrToFileErr(err)
+		if osIsPermission(err) {
+			f, err = Open(dirPath)
+			if err != nil {
+				return nil, osErrToFileErr(err)
+			}
+		} else {
+			return nil, osErrToFileErr(err)
+		}
 	}
 	defer f.Close()
 


Test patch: