Repository: minio/minio (language: go)

Issue:
Buckets info and cluster summary not changes
## NOTE
Buckets info and cluster summary not changes. Objects count and used space are the same even after few hours of big uploads. I now migrating date from one cluster to another. Before update to the current version information about object count and used space was changed every 3-5 minutes but now it shows same values for the all time after update. This values not changes in web console and also in metrics which sent to prometheus. It's interesting that only total used space metric which sending to prometheus are changes - objects count and buckets used space are the same. For now prometheus shows me 6.4 TB used and web console only 4.4. You can see it at screenshots attached.

## Expected Behavior
Information about used space and objects count should be changed according real data.

## Current Behavior
For in it shows values which was before update few days ago.

## Regression
It works fine before update in version RELEASE.2023-01-12T02-06-16Z (commit-id=7bc95c47a322971aff7d4d4c270dcf28a933e84b)

## Your Environment
* Version used (`minio --version`): RELEASE.2023-02-10T18-48-39Z (commit-id=d0f4cc89a5d0ca45e7ba00fbb7733c2fead3f44e)
* Server setup and configuration: 4 nodes with 4 disks on each
* Operating System and version (`uname -a`): Linux minio-cold-1 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux



Discussion / hints:
Screenshots.

![image](https://user-images.githubusercontent.com/13428863/219563547-f25f9530-a957-48dd-84a2-14b2ce2ed335.png)
![image](https://user-images.githubusercontent.com/13428863/219563573-037857bb-295a-4065-9247-b37691bf0d70.png)
![image](https://user-images.githubusercontent.com/13428863/219563602-319bd2a7-ab63-43ed-8142-7609632b10fe.png)
![image](https://user-images.githubusercontent.com/13428863/219563660-10d44d5a-15d3-4f56-8258-33fcc64f8f06.png)

Does this value change in `mc admin info` ? 
No, this command also show wrong info.

root@minio-cold-1:~# mc admin info minio-cold
●  minio-cold-1.bazissoft.ru:9001
   Uptime: 2 days
   Version: 2023-02-10T18:48:39Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio-cold-2.bazissoft.ru:9001
   Uptime: 2 days
   Version: 2023-02-10T18:48:39Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio-cold-3.bazissoft.ru:9001
   Uptime: 2 days
   Version: 2023-02-10T18:48:39Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio-cold-4.bazissoft.ru:9001
   Uptime: 1 hour
   Version: 2023-02-10T18:48:39Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

Pools:
   1st, Erasure sets: 1, Drives per erasure set: 16

4.8 TiB Used, 4 Buckets, 268,051 Objects
16 drives online, 0 drives offline

It asked me to add the --airgap parameter:

root@minio-cold-1:~# mc support profile minio-cold --duration 120 --dev --type goroutines
mc: <ERROR> Unable to reach http://localhost:9000, please use --airgap if there is no connectivity to SUBNET. Head "http://localhost:9000": dial tcp 127.0.0.1:9000: connect: connection refused.
root@minio-cold-1:~# mc support profile minio-cold --duration 120 --dev --type goroutines --airgap
mc: Profiling 'minio-cold' for 120 seconds... saved successfully at 'profile.zip'

Result in attachment.

[profile.zip](https://github.com/minio/minio/files/10764770/profile.zip)

Please post output from `mc admin scanner status minio-cold`.

Depending on the number of objects it can take a while to scan.
> Please post output from `mc admin scanner status minio-cold`.
> 
> Depending on the number of objects it can take a while to scan.

![image](https://user-images.githubusercontent.com/13428863/219597738-622e1c47-4d58-42e8-abf0-5513e9eeca78.png)

It was working fine before I've updated to the current version and it stops working at the same day. As I said values doesn't chages after update.
It's interesting that tools/test.txt doesn't exits. I've removed it few days ago.
@harshavardhana Stuck queueing  a heal:
```
goroutine 10588435 [select, 3565 minutes]:
github.com/minio/minio/cmd.(*healSequence).queueHealTask(0xc001326000, {{0xc007b8e5e0, 0x5}, {0xc005535b88, 0x8}, {0x0, 0x0}, 0x0, 0x0}, {0x2a0a87b, ...})
	github.com/minio/minio/cmd/admin-heal-ops.go:742 +0x606
github.com/minio/minio/cmd.(*folderScanner).scanFolder.func4({0xc000487180?, 0x10, 0x46cb01?}, {0x430da5?, 0xc00752e728?, 0x40?})
	github.com/minio/minio/cmd/data-scanner.go:776 +0x711
github.com/minio/minio/cmd.listPathRaw({0x4ebd680, 0xc00bef5600}, {{0xc0076c8700, 0x10, 0x10}, {0x0, 0x0, 0x0}, {0xc007b8e5e0, 0x5}, ...})
	github.com/minio/minio/cmd/metacache-set.go:1064 +0x732
github.com/minio/minio/cmd.(*folderScanner).scanFolder(0xc0001f1200, {0x4ebd680, 0xc0128e8240}, {{0xc007b8e0ea, 0x5}, 0x0, 0x1}, 0xc00752f2a8)
	github.com/minio/minio/cmd/data-scanner.go:722 +0x1b68
github.com/minio/minio/cmd.scanDataFolder({_, _}, _, _, {_, _}, {{{0xc007b8e0ea, 0x5}, 0x1ecb, {0x21e28e18, ...}, ...}, ...}, ...)
	github.com/minio/minio/cmd/data-scanner.go:368 +0x7f1
github.com/minio/minio/cmd.(*xlStorage).NSScanner(_, {_, _}, {{{0xc007b8e0ea, 0x5}, 0x1ecb, {0x21e28e18, 0xedb7de5ce, 0x6352ec0}, 0x0, ...}, ...}, ...)
	github.com/minio/minio/cmd/xl-storage.go:487 +0x6b9
github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).NSScanner(_, {_, _}, {{{0xc007b8e0ea, 0x5}, 0x1ecb, {0x21e28e18, 0xedb7de5ce, 0x6352ec0}, 0x0, ...}, ...}, ...)
	github.com/minio/minio/cmd/xl-storage-disk-id-check.go:184 +0x178
github.com/minio/minio/cmd.erasureObjects.nsScanner.func3(0xc001613f80?)
	github.com/minio/minio/cmd/erasure.go:507 +0x7de
created by github.com/minio/minio/cmd.erasureObjects.nsScanner
	github.com/minio/minio/cmd/erasure.go:458 +0x967
```
Likely regression from https://github.com/minio/minio/pull/16433 - the only recent change to this functionality.
> @harshavardhana Stuck queueing a heal:
> 
> ```
> goroutine 10588435 [select, 3565 minutes]:
> github.com/minio/minio/cmd.(*healSequence).queueHealTask(0xc001326000, {{0xc007b8e5e0, 0x5}, {0xc005535b88, 0x8}, {0x0, 0x0}, 0x0, 0x0}, {0x2a0a87b, ...})
> 	github.com/minio/minio/cmd/admin-heal-ops.go:742 +0x606
> github.com/minio/minio/cmd.(*folderScanner).scanFolder.func4({0xc000487180?, 0x10, 0x46cb01?}, {0x430da5?, 0xc00752e728?, 0x40?})
> 	github.com/minio/minio/cmd/data-scanner.go:776 +0x711
> github.com/minio/minio/cmd.listPathRaw({0x4ebd680, 0xc00bef5600}, {{0xc0076c8700, 0x10, 0x10}, {0x0, 0x0, 0x0}, {0xc007b8e5e0, 0x5}, ...})
> 	github.com/minio/minio/cmd/metacache-set.go:1064 +0x732
> github.com/minio/minio/cmd.(*folderScanner).scanFolder(0xc0001f1200, {0x4ebd680, 0xc0128e8240}, {{0xc007b8e0ea, 0x5}, 0x0, 0x1}, 0xc00752f2a8)
> 	github.com/minio/minio/cmd/data-scanner.go:722 +0x1b68
> github.com/minio/minio/cmd.scanDataFolder({_, _}, _, _, {_, _}, {{{0xc007b8e0ea, 0x5}, 0x1ecb, {0x21e28e18, ...}, ...}, ...}, ...)
> 	github.com/minio/minio/cmd/data-scanner.go:368 +0x7f1
> github.com/minio/minio/cmd.(*xlStorage).NSScanner(_, {_, _}, {{{0xc007b8e0ea, 0x5}, 0x1ecb, {0x21e28e18, 0xedb7de5ce, 0x6352ec0}, 0x0, ...}, ...}, ...)
> 	github.com/minio/minio/cmd/xl-storage.go:487 +0x6b9
> github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).NSScanner(_, {_, _}, {{{0xc007b8e0ea, 0x5}, 0x1ecb, {0x21e28e18, 0xedb7de5ce, 0x6352ec0}, 0x0, ...}, ...}, ...)
> 	github.com/minio/minio/cmd/xl-storage-disk-id-check.go:184 +0x178
> github.com/minio/minio/cmd.erasureObjects.nsScanner.func3(0xc001613f80?)
> 	github.com/minio/minio/cmd/erasure.go:507 +0x7de
> created by github.com/minio/minio/cmd.erasureObjects.nsScanner
> 	github.com/minio/minio/cmd/erasure.go:458 +0x967
> ```

This blocking is normal @klauspost - that PR only adds code for non-blocking behavior. 

Something else is waiting. 
@AlexZIX can you also share `mc support top locks alias/ --count=1000` and share it here? 
From what I can tell this is not regression instead there is some network layer issue.

```
goroutine 115695989 [select, 79 minutes]:
net/http.(*persistConn).roundTrip(0xc00e602240, 0xc0102b6f00)
        net/http/transport.go:2620 +0x974
net/http.(*Transport).roundTrip(0xc002048b40, 0xc00e753b00)
        net/http/transport.go:595 +0x7ba
net/http.(*Transport).RoundTrip(0x203002?, 0x4eab8c0?)
        net/http/roundtrip.go:17 +0x19
net/http.send(0xc00e753b00, {0x4eab8c0, 0xc002048b40}, {0x299f5c0?, 0xc00d51ed01?, 0x0?})
        net/http/client.go:251 +0x5f7
net/http.(*Client).send(0xc029d57080, 0xc00e753b00, {0x203002?, 0xc00892e4b0?, 0x0?})
        net/http/client.go:175 +0x9b
net/http.(*Client).do(0xc029d57080, 0xc00e753b00)
        net/http/client.go:715 +0x8fc
net/http.(*Client).Do(...)
        net/http/client.go:581
github.com/minio/minio/internal/rest.(*Client).Call(0xc027c0d600, {0x4ebd6b8, 0xc000054080}, {0x2a148b3, 0xb}, 0x0?, {0x7f5395d6f288, 0xc0021fa520}, 0xffffffffffffffff)
        github.com/minio/minio/internal/rest/client.go:252 +0x372
github.com/minio/minio/cmd.(*storageRESTClient).call(0xc005912460, {0x4ebd6b8, 0xc000054080}, {0x2a148b3, 0xb}, 0x43cec7?, {0x7f5395d6f288, 0xc0021fa520}, 0xc004c6fba0?)
        github.com/minio/minio/cmd/storage-rest-client.go:164 +0x1c5
github.com/minio/minio/cmd.(*storageRESTClient).CreateFile(0x0?, {0x4ebd6b8, 0xc000054080}, {0x2a1bce2, 0xe}, {0xc013690320, 0x50}, 0xc00932a8c0?, {0x4eab480, 0xc00dadb7b8})
        github.com/minio/minio/cmd/storage-rest-client.go:378 +0x3c6
github.com/minio/minio/cmd.newStreamingBitrotWriter.func1()
        github.com/minio/minio/cmd/bitrot-streaming.go:105 +0x195
created by github.com/minio/minio/cmd.newStreamingBitrotWriter
        github.com/minio/minio/cmd/bitrot-streaming.go:99 +0x3be
```

Like its taking 79 minutes of network roundtrip time. 

```goroutine 116448647 [IO wait, 56 minutes]:
internal/poll.runtime_pollWait(0x7f5391c36f68, 0x72)
        runtime/netpoll.go:305 +0x89
internal/poll.(*pollDesc).wait(0xc00ae0d600?, 0xc00c3d2000?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00ae0d600, {0xc00c3d2000, 0x8000, 0x8000})
        internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00ae0d600, {0xc00c3d2000?, 0x0?, 0x2?})
        net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00e48dd90, {0xc00c3d2000?, 0xc01a0aca20?, 0x0?})
        net/net.go:183 +0x45
net/http.(*persistConn).Read(0xc008410d80, {0xc00c3d2000?, 0xc0045e6ae0?, 0x68cd25?})
        net/http/transport.go:1929 +0x4e
bufio.(*Reader).fill(0xc00e917860)
        bufio/bufio.go:106 +0xff
bufio.(*Reader).ReadSlice(0xc00e917860, 0x0?)
        bufio/bufio.go:372 +0x2f
bufio.(*Reader).ReadLine(0xc00e917860)
        bufio/bufio.go:401 +0x27
net/textproto.(*Reader).readLineSlice(0xc0045e6c88)
        net/textproto/reader.go:56 +0x99
net/textproto.(*Reader).ReadLine(...)
        net/textproto/reader.go:37
net/http.ReadResponse(0xc00e917860, 0xc00ac85200)
        net/http/response.go:161 +0xa6
net/http.(*persistConn).readResponse(0xc008410d80, {{}, 0xc00ac85200, {0xc00ac85100}, 0xc00a88ab40, 0x0, 0xc00a88aa80, 0xc00a88aae0}, 0xc00f066a00)
        net/http/transport.go:2286 +0xda
net/http.(*persistConn).readLoop(0xc008410d80)
        net/http/transport.go:2108 +0x3aa
created by net/http.(*Transport).dialConn
        net/http/transport.go:1751 +0x173e
```

Another blocked for 1115 minutes
```
goroutine 68341257 [select, 1115 minutes]:
net/http.(*persistConn).roundTrip(0xc0056b5c20, 0xc00d531b80)
        net/http/transport.go:2620 +0x974
net/http.(*Transport).roundTrip(0xc002048b40, 0xc007eedd00)
        net/http/transport.go:595 +0x7ba
net/http.(*Transport).RoundTrip(0x203003?, 0x4eab8c0?)
        net/http/roundtrip.go:17 +0x19
net/http.send(0xc007eedd00, {0x4eab8c0, 0xc002048b40}, {0x299f5c0?, 0xc00915a501?, 0x0?})
        net/http/client.go:251 +0x5f7
net/http.(*Client).send(0xc007025c50, 0xc007eedd00, {0x203003?, 0xc00f557800?, 0x0?})
        net/http/client.go:175 +0x9b
net/http.(*Client).do(0xc007025c50, 0xc007eedd00)
        net/http/client.go:715 +0x8fc
net/http.(*Client).Do(...)
        net/http/client.go:581
github.com/minio/minio/internal/rest.(*Client).Call(0xc0013be400, {0x4ebd6b8, 0xc000054080}, {0x2a148b3, 0xb}, 0x6352ec0?, {0x7f5395d6f288, 0xc006c1efe0}, 0xffffffffffffffff)
        github.com/minio/minio/internal/rest/client.go:252 +0x372
github.com/minio/minio/cmd.(*storageRESTClient).call(0xc00141a500, {0x4ebd6b8, 0xc000054080}, {0x2a148b3, 0xb}, 0xc002a06ee0?, {0x7f5395d6f288, 0xc006c1efe0}, 0xc002a06ee0?)
        github.com/minio/minio/cmd/storage-rest-client.go:164 +0x1c5
github.com/minio/minio/cmd.(*storageRESTClient).CreateFile(0x2a0551b?, {0x4ebd6b8, 0xc000054080}, {0x2a1bce2, 0xe}, {0xc00955b2c0, 0x50}, 0xc001592120?, {0x4eab480, 0xc000698928})
        github.com/minio/minio/cmd/storage-rest-client.go:378 +0x3c6
github.com/minio/minio/cmd.newStreamingBitrotWriter.func1()
        github.com/minio/minio/cmd/bitrot-streaming.go:105 +0x195
created by github.com/minio/minio/cmd.newStreamingBitrotWriter
        github.com/minio/minio/cmd/bitrot-streaming.go:99 +0x3be
```

there are pollWait() in minutes - this doesn't make much sense. 

```
108 @ 0x43ce16 0x44cf5c 0x7265b4 0x71a0fa 0x700b59 0x6bcd77 0x6bc5fb 0x6be97c 0x1707052 0x170703a 0x22b9085 0x22bc106 0x1f81db5 0x46ec21
#       0x7265b3        net/http.(*persistConn).roundTrip+0x973                                 net/http/transport.go:2620
#       0x71a0f9        net/http.(*Transport).roundTrip+0x7b9                                   net/http/transport.go:595
#       0x700b58        net/http.(*Transport).RoundTrip+0x18                                    net/http/roundtrip.go:17
#       0x6bcd76        net/http.send+0x5f6                                                     net/http/client.go:251
#       0x6bc5fa        net/http.(*Client).send+0x9a                                            net/http/client.go:175
#       0x6be97b        net/http.(*Client).do+0x8fb                                             net/http/client.go:715
#       0x1707051       net/http.(*Client).Do+0x371                                             net/http/client.go:581
#       0x1707039       github.com/minio/minio/internal/rest.(*Client).Call+0x359               github.com/minio/minio/internal/rest/client.go:252
#       0x22b9084       github.com/minio/minio/cmd.(*storageRESTClient).call+0x1c4              github.com/minio/minio/cmd/storage-rest-client.go:164
#       0x22bc105       github.com/minio/minio/cmd.(*storageRESTClient).CreateFile+0x3c5        github.com/minio/minio/cmd/storage-rest-client.go:378
#       0x1f81db4       github.com/minio/minio/cmd.newStreamingBitrotWriter.func1+0x194         github.com/minio/minio/cmd/bitrot-streaming.go:105
```

```
44 @ 0x43ce16 0x44cf5c 0x4a3551 0x4a3d05 0x4a18da 0x113bd05 0x113bc29 0x2335e2f 0x23358c5 0x22fa145 0x1f81db5 0x46ec21
#       0x4a3550        io.(*pipe).read+0xb0                                                    io/pipe.go:57
#       0x4a3d04        io.(*PipeReader).Read+0x24                                              io/pipe.go:136
#       0x4a18d9        io.ReadAtLeast+0x99                                                     io/io.go:332
#       0x113bd04       io.ReadFull+0x104                                                       io/io.go:351
#       0x113bc28       github.com/minio/minio/internal/ioutil.CopyAligned+0x28                 github.com/minio/minio/internal/ioutil/ioutil.go:280
#       0x2335e2e       github.com/minio/minio/cmd.(*xlStorage).writeAllDirect+0x42e            github.com/minio/minio/cmd/xl-storage.go:1893
#       0x23358c4       github.com/minio/minio/cmd.(*xlStorage).CreateFile+0x344                github.com/minio/minio/cmd/xl-storage.go:1855
#       0x22fa144       github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).CreateFile+0x224     github.com/minio/minio/cmd/xl-storage-disk-id-check.go:348
#       0x1f81db4       github.com/minio/minio/cmd.newStreamingBitrotWriter.func1+0x194         github.com/minio/minio/cmd/bitrot-streaming.go:105

12 @ 0x43ce16 0x406d5b 0x406858 0x723625 0x726a1f 0x71a0fa 0x700b59 0x6bcd77 0x6bc5fb 0x6be97c 0x1707052 0x170703a 0x22b9085 0x22bc106 0x1f81db5 0x46ec21
#       0x723624        net/http.(*persistConn).mapRoundTripError+0x64                          net/http/transport.go:2015
#       0x726a1e        net/http.(*persistConn).roundTrip+0xdde                                 net/http/transport.go:2659
#       0x71a0f9        net/http.(*Transport).roundTrip+0x7b9                                   net/http/transport.go:595
#       0x700b58        net/http.(*Transport).RoundTrip+0x18                                    net/http/roundtrip.go:17
#       0x6bcd76        net/http.send+0x5f6                                                     net/http/client.go:251
#       0x6bc5fa        net/http.(*Client).send+0x9a                                            net/http/client.go:175
#       0x6be97b        net/http.(*Client).do+0x8fb                                             net/http/client.go:715
#       0x1707051       net/http.(*Client).Do+0x371                                             net/http/client.go:581
#       0x1707039       github.com/minio/minio/internal/rest.(*Client).Call+0x359               github.com/minio/minio/internal/rest/client.go:252
#       0x22b9084       github.com/minio/minio/cmd.(*storageRESTClient).call+0x1c4              github.com/minio/minio/cmd/storage-rest-client.go:164
#       0x22bc105       github.com/minio/minio/cmd.(*storageRESTClient).CreateFile+0x3c5        github.com/minio/minio/cmd/storage-rest-client.go:378
#       0x1f81db4       github.com/minio/minio/cmd.newStreamingBitrotWriter.func1+0x194         github.com/minio/minio/cmd/bitrot-streaming.go:105
```

There is actually 44 callers writing to disk and blocked - this is even local disk. 

Do you have really slow clients? @AlexZIX 

> mc support top locks alias/ --count=1000

Can't run this command because it shows:

mc: <ERROR>  Please register the cluster first by running 'mc license register minio-cold'

Also I haven't clients yet - just copying data from one minio cluster to another using rclone:

rclone sync minio-cold-old:wp3d-angstrem-archive/webplanner minio-cold:wp3d-angstrem-archive --bwlimit 20M --progress --transfers=10
But you're right - this copy process working for a few days and my old minio cluster sometimes are really slow because built from 12 hdd 5 TB each. Sometimes listing files in a bucket takes 3-5 minutes. It's the reason why I created new cluster with SAS disks and now migrating to it. Total data size which should be moved is 11 TB and this issue arose after 4.5 TB was copied.
@harshavardhana 

When queuing the tasks, with [nowait](
https://github.com/minio/minio/blob/d08e3cc89526a11c8a263d763cfd8e869860ca72/cmd/admin-heal-ops.go#L716) we do not set a task.respCh. 

However, we still [wait for it](https://github.com/minio/minio/blob/d08e3cc89526a11c8a263d763cfd8e869860ca72/cmd/admin-heal-ops.go#L742). That seems like a problem.

This explains the blocking here:

```
goroutine 57506867 [select, 1517 minutes]:
github.com/minio/minio/cmd.(*healSequence).queueHealTask(0xc001326000, {{0xc008b53c46, 0x15}, {0xc008b53c5c, 0x12}, {0x0, 0x0}, 0x1, 0xc00a252b88}, {0x2a0a87b, ...})
	github.com/minio/minio/cmd/admin-heal-ops.go:742 +0x606
github.com/minio/minio/cmd.healObject({0xc008b53c46, 0x15}, {0xc008b53c5c, 0x12}, {0x0, 0x0}, 0x1)
	github.com/minio/minio/cmd/global-heal.go:420 +0x1a5
created by github.com/minio/minio/cmd.erasureObjects.getObjectFileInfo
	github.com/minio/minio/cmd/erasure-object.go:683 +0xaa5
```

... which will leak a goroutine every time AFAICT.

Don't see right now, how that would explain the scanner being blocked.

AFAICT there aren't any healing tasks running. All ` *healRoutine).AddWorker` are waiting for input AFAICT.
> However, we still [wait for it](https://github.com/minio/minio/blob/d08e3cc89526a11c8a263d763cfd8e869860ca72/cmd/admin-heal-ops.go#L742). That seems like a problem.

We do not wait for it @klauspost we return right away in the `default: {` case right? if the task is blocked otherwise it's blocked on the task. 

There is no way the control flow should reach `h.respCh` and there should be no wait on the respCh. 

Fix patch:
diff --git a/cmd/admin-heal-ops.go b/cmd/admin-heal-ops.go
index 6cae8170c6761..b9402a5aa004e 100644
--- a/cmd/admin-heal-ops.go
+++ b/cmd/admin-heal-ops.go
@@ -407,9 +407,6 @@ type healSequence struct {
 	// bucket, and object on which heal seq. was initiated
 	bucket, object string
 
-	// A channel of entities with heal result
-	respCh chan healResult
-
 	// Report healing progress
 	reportProgress bool
 
@@ -472,7 +469,6 @@ func newHealSequence(ctx context.Context, bucket, objPrefix, clientAddr string,
 	clientToken := mustGetUUID()
 
 	return &healSequence{
-		respCh:         make(chan healResult),
 		bucket:         bucket,
 		object:         objPrefix,
 		reportProgress: true,
@@ -719,28 +715,30 @@ func (h *healSequence) queueHealTask(source healSource, healType madmin.HealItem
 			if serverDebugLog {
 				logger.Info("Task in the queue: %#v", task)
 			}
-		case <-h.ctx.Done():
-			return nil
 		default:
 			// task queue is full, no more workers, we shall move on and heal later.
 			return nil
 		}
-	} else {
-		// respCh must be set for guaranteed result
-		task.respCh = h.respCh
-		select {
-		case globalBackgroundHealRoutine.tasks <- task:
-			if serverDebugLog {
-				logger.Info("Task in the queue: %#v", task)
-			}
-		case <-h.ctx.Done():
-			return nil
+		// Don't wait for result
+		return nil
+	}
+
+	// respCh must be set to wait for result.
+	// We make it size 1, so a result can always be written
+	// even if we aren't listening.
+	task.respCh = make(chan healResult, 1)
+	select {
+	case globalBackgroundHealRoutine.tasks <- task:
+		if serverDebugLog {
+			logger.Info("Task in the queue: %#v", task)
 		}
+	case <-h.ctx.Done():
+		return nil
 	}
 
 	// task queued, now wait for the response.
 	select {
-	case res := <-h.respCh:
+	case res := <-task.respCh:
 		if !h.reportProgress {
 			if errors.Is(res.err, errSkipFile) { // this is only sent usually by nopHeal
 				return nil
diff --git a/cmd/global-heal.go b/cmd/global-heal.go
index 7b2160be067ff..cac79b2552a76 100644
--- a/cmd/global-heal.go
+++ b/cmd/global-heal.go
@@ -48,7 +48,6 @@ func newBgHealSequence() *healSequence {
 	}
 
 	return &healSequence{
-		respCh:      make(chan healResult),
 		startTime:   UTCNow(),
 		clientToken: bgHealingUUID,
 		// run-background heal with reserved bucket


Test patch: