Repository: minio/minio (language: go)

Issue:
Some of the files cannot be downloaded when using tiering and multiple pools
# Bug


<!--- Provide a general summary of the issue in the Title above -->
We have a cluster with 3 nodes and configured tering for the bucket. After the expansion by additional nodes, 2 days after the addition, some files become unavailable when downloading.

``` 
<ERROR> Failed to copy `http://localhost/test/4/apport/NEWS.gz`. unexpected EOF 
```
## Expected Behavior

<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->

## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->

1. initial pool configuration:
Version: 2023-11-01T18:37:25Z
- host minio-c1-node1/10.0.0.11 - s3 node
- host minio-c1-node2/10.0.0.21 - s3 node
- host minio-c1-node3/10.0.0.18 - s3 node
- host minio-c1-lb1/10.0.0.25 - haproxy balancer
All host have configurations:
* 2 CPU
* 4 GB mem
* System disk 24GB /
* Disk 16GB /mnt/disk1
* Disk 16GB /mnt/disk2
* Disk 16GB /mnt/disk3
* Disk 16GB /mnt/disk4

```
root@minio-c1-node1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           392M  1.1M  391M   1% /run
/dev/vda1        23G  4.3G   18G  20% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/vdc         16G  1.7G   15G  11% /mnt/disk4
/dev/vdd         16G  1.7G   15G  11% /mnt/disk1
/dev/vde         16G  1.7G   15G  11% /mnt/disk3
/dev/vdb         16G  1.7G   15G  11% /mnt/disk2
tmpfs           392M  4.0K  392M   1% /run/user/1000
```

Configuration  minio:
```
root@minio-c1-node1:~# cat /etc/default/minio
MINIO_VOLUMES="http://minio-c1-node{1...3}.mcs.local:9000/mnt/disk{1...4}"
MINIO_OPTS="--console-address :9001"
MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=redacted

```
Configuration for balancer with haproxy:
```
root@minio-c1-lb1:~# cat /etc/haproxy/haproxy.cfg
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend minio
    description "Minio for sovcombank.group domain clients"
    bind *:80
    capture response header X-Amz-Request-Id len 16
    default_backend minio

backend minio
    description "Minio"
    balance leastconn
    mode http
    option httpchk
    http-check send meth GET uri /minio/health/live ver HTTP/1.1 hdr host minio.mcs.local
    http-check expect status 200-399
    option log-health-checks

    option http-keep-alive
    no option httpclose
    no option http-server-close
    option forwardfor

    server minio-c1-node1 minio-c1-node1.mcs.local:9000 check inter 2s rise 5 fall 2
    server minio-c1-node2 minio-c1-node2.mcs.local:9000 check inter 2s rise 5 fall 2
    server minio-c1-node3 minio-c1-node3.mcs.local:9000 check inter 2s rise 5 fall 2
```

Config for bucket:
```
root@minio-c1-lb1:~# minio-client ilm tier ls MYMINIO/test
Name    |Type   |Endpoint                      |Bucket         |Prefix  |Region  |Storage-Class
S3TIER  |minio  |https://hb.ru-msk.vkcs.cloud  |tier-test-xxx  |tst/    |        |

root@minio-c1-lb1:~# minio-client ilm rule ls  MYMINIO/test
┌────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                             │
├──────────────────────┬─────────┬────────┬──────┬──────────────┬────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS │ DAYS TO TIER │ TIER   │
├──────────────────────┼─────────┼────────┼──────┼──────────────┼────────┤
│ cl1j06nm0vm0n7sjftrg │ Enabled │ -      │ -    │            1 │ S3TIER │
└──────────────────────┴─────────┴────────┴──────┴──────────────┴────────┘
```
 All files added to bucket after 2 days has been moved to tier.

2. Expanding the pool
Adding new host 
- host minio-c1-node4/10.0.0.5 - s3 node
- host minio-c1-node5/10.0.0.15 - s3 node
- host minio-c1-node6/10.0.0.7 - s3 node
All host have configurations:
* 2 CPU
* 4 GB mem
* System disk 24GB /
* Disk 16GB /mnt/disk1
* Disk 16GB /mnt/disk2
* Disk 16GB /mnt/disk3
* Disk 16GB /mnt/disk4
```
root@minio-c1-node4:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           392M  1.1M  391M   1% /run
/dev/vda1        23G  4.1G   18G  20% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/vde         16G  537M   16G   4% /mnt/disk1
/dev/vdd         16G  537M   16G   4% /mnt/disk3
/dev/vdc         16G  537M   16G   4% /mnt/disk4
/dev/vdb         16G  537M   16G   4% /mnt/disk2
tmpfs           392M  4.0K  392M   1% /run/user/1000
```

New Configuration  minio on all host:
```
root@minio-c1-node4:~# cat /etc/default/minio
MINIO_VOLUMES="http://minio-c1-node{1...3}.mcs.local:9000/mnt/disk{1...4} http://minio-c1-node{4...6}.mcs.local:9000/mnt/disk{1...4}"
MINIO_OPTS="--console-address :9001"
MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=redacted

```

Configuration of haproxy:
```
root@minio-c1-lb1:~# cat /etc/haproxy/haproxy.cfg
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend minio
    description "Minio for sovcombank.group domain clients"
    bind *:80
    capture response header X-Amz-Request-Id len 16
    default_backend minio

backend minio
    description "Minio"
    balance leastconn
    mode http
    option httpchk
    http-check send meth GET uri /minio/health/live ver HTTP/1.1 hdr host minio.mcs.local
    http-check expect status 200-399
    option log-health-checks

    option http-keep-alive
    no option httpclose
    no option http-server-close
    option forwardfor

    server minio-c1-node1 minio-c1-node1.mcs.local:9000 check inter 2s rise 5 fall 2
    server minio-c1-node2 minio-c1-node2.mcs.local:9000 check inter 2s rise 5 fall 2
    server minio-c1-node3 minio-c1-node3.mcs.local:9000 check inter 2s rise 5 fall 2
    server minio-c1-node4 minio-c1-node4.mcs.local:9000 check inter 2s rise 5 fall 2
    server minio-c1-node5 minio-c1-node5.mcs.local:9000 check inter 2s rise 5 fall 2
    server minio-c1-node6 minio-c1-node6.mcs.local:9000 check inter 2s rise 5 fall 2

```

```
root@minio-c1-lb1:~# minio-client admin info MYMINIO/
●  minio-c1-node1.mcs.local:9000
   Uptime: 23 hours
   Version: 2023-11-01T18:37:25Z
   Network: 6/6 OK
   Drives: 4/4 OK
   Pool: 1

●  minio-c1-node2.mcs.local:9000
   Uptime: 23 hours
   Version: 2023-11-01T18:37:25Z
   Network: 6/6 OK
   Drives: 4/4 OK
   Pool: 1

●  minio-c1-node3.mcs.local:9000
   Uptime: 23 hours
   Version: 2023-11-01T18:37:25Z
   Network: 6/6 OK
   Drives: 4/4 OK
   Pool: 1

●  minio-c1-node4.mcs.local:9000
   Uptime: 23 hours
   Version: 2023-11-01T18:37:25Z
   Network: 6/6 OK
   Drives: 4/4 OK
   Pool: 2

●  minio-c1-node5.mcs.local:9000
   Uptime: 23 hours
   Version: 2023-11-01T18:37:25Z
   Network: 6/6 OK
   Drives: 4/4 OK
   Pool: 2

●  minio-c1-node6.mcs.local:9000
   Uptime: 23 hours
   Version: 2023-11-01T18:37:25Z
   Network: 6/6 OK
   Drives: 4/4 OK
   Pool: 2

Pools:
   1st, Erasure sets: 1, Drives per erasure set: 12
   2nd, Erasure sets: 1, Drives per erasure set: 12

13 GiB Used, 1 Bucket, 177,687 Objects
24 drives online, 0 drives offline
```


3. Adding more files to the package and waiting for 2 days

4. Checking the availability of files
Correct behavior:
```
root@minio-c1-lb1:~# minio-client stat MYMINIO/test/1/adduser/examples/adduser.local.conf.examples/skel/dot.bash_profile
Name      : dot.bash_profile
Date      : 2023-11-02 04:26:01 UTC
Size      : 1.3 KiB
ETag      : 7e25a53e7588cdb05870608583b05b19
Type      : file
Metadata  :
  X-Amz-Storage-Class: S3TIER
  Content-Type       : application/octet-stream

root@minio-c1-lb1:~# minio-client cp MYMINIO/test/1/adduser/examples/adduser.local.conf.examples/skel/dot.bash_profile /tmp/ --json
{
 "status": "success",
 "source": "MYMINIO/test/1/adduser/examples/adduser.local.conf.examples/skel/dot.bash_profile",
 "target": "/tmp/dot.bash_profile",
 "size": 1290,
 "totalCount": 1,
 "totalSize": 0
}
{
 "status": "success",
 "total": 1290,
 "transferred": 1290,
 "speed": 13254.685713780187
}

```

incorrect behavior (some part of the files):
```
root@minio-c1-lb1:~# minio-client stat  MYMINIO/test/4/apport/NEWS.gz
Name      : NEWS.gz
Date      : 2023-11-02 04:28:46 UTC
Size      : 45 KiB
ETag      : 203aed2666d9bde475abb067afca921b
Type      : file
Metadata  :
  X-Amz-Storage-Class: S3TIER
  Content-Type       : application/gzip


root@minio-c1-lb1:~# minio-client --json cp  MYMINIO/test/4/apport/NEWS.gz /tmp/
{
 "status": "success",
 "source": "MYMINIO/test/4/apport/NEWS.gz",
 "target": "/tmp/NEWS.gz",
 "size": 46414,
 "totalCount": 1,
 "totalSize": 0
}
{
 "status": "error",
 "error": {
  "message": "Failed to copy `http://localhost/test/4/apport/NEWS.gz`.",
  "cause": {
   "message": "unexpected EOF",
   "error": {}
  },
  "type": "error"
 }
}


root@minio-c1-lb1:~# minio-client stat MYMINIO/test/2/apt/examples/configure-index
Name      : configure-index
Date      : 2023-11-02 04:27:03 UTC
Size      : 26 KiB
ETag      : 726bcad4e2d555214ca1d6d570f7d2f9
Type      : file
Metadata  :
  Content-Type       : application/octet-stream
  X-Amz-Storage-Class: S3TIER

root@minio-c1-lb1:~# minio-client cp --json MYMINIO/test/2/apt/examples/configure-index /tmp/
{
 "status": "success",
 "source": "MYMINIO/test/2/apt/examples/configure-index",
 "target": "/tmp/configure-index",
 "size": 26148,
 "totalCount": 1,
 "totalSize": 0
}
{
 "status": "error",
 "error": {
  "message": "Failed to copy `http://localhost/test/2/apt/examples/configure-index`.",
  "cause": {
   "message": "unexpected EOF",
   "error": {}
  },
  "type": "error"
 }
}


```

With Debug:
```
root@minio-c1-lb1:~# minio-client --debug cp --json MYMINIO/test/2/apt/examples/configure-index /tmp/
minio-client: <DEBUG> GET /test/?location= HTTP/1.1
Host: localhost
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231108T034409Z
Accept-Encoding: gzip

minio-client: <DEBUG> HTTP/1.1 200 OK
Content-Length: 128
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 08 Nov 2023 03:44:09 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 13fd1dddc198a66a2be6c5316f866284c6aa4af17941933c6a5cdc95e4eed4ea
X-Amz-Request-Id: 179588FBB5108A1F
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

minio-client: <DEBUG> Response Time:  7.498243ms

minio-client: <DEBUG> HEAD /test/2/apt/examples/configure-index HTTP/1.1
Host: localhost
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231108T034409Z

minio-client: <DEBUG> HTTP/1.1 200 OK
Content-Length: 26148
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Wed, 08 Nov 2023 03:44:09 GMT
Etag: "726bcad4e2d555214ca1d6d570f7d2f9"
Last-Modified: Thu, 02 Nov 2023 04:27:03 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 6dadb366fe418cabbb9c70e460a548543a3b2eb339c7e623dc02c4bf6689a604
X-Amz-Request-Id: 179588FBB578C5A5
X-Amz-Storage-Class: S3TIER
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

minio-client: <DEBUG> Response Time:  10.32442ms

{
 "status": "success",
 "source": "MYMINIO/test/2/apt/examples/configure-index",
 "target": "/tmp/configure-index",
 "size": 26148,
 "totalCount": 1,
 "totalSize": 0
}
minio-client: <DEBUG> HEAD /test/2/apt/examples/configure-index HTTP/1.1
Host: localhost
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231108T034409Z

minio-client: <DEBUG> HTTP/1.1 200 OK
Content-Length: 26148
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Wed, 08 Nov 2023 03:44:09 GMT
Etag: "726bcad4e2d555214ca1d6d570f7d2f9"
Last-Modified: Thu, 02 Nov 2023 04:27:03 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 6c6c4d6627746654b4b41bea40a73f2b790a9b5595239c1b7913c6d80ca4c243
X-Amz-Request-Id: 179588FBB62126A6
X-Amz-Storage-Class: S3TIER
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

minio-client: <DEBUG> Response Time:  5.996026ms

minio-client: <DEBUG> GET /test/2/apt/examples/configure-index HTTP/1.1
Host: localhost
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;if-match;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
If-Match: "726bcad4e2d555214ca1d6d570f7d2f9"
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231108T034409Z
Accept-Encoding: gzip

minio-client: <DEBUG> HTTP/1.1 200 OK
Content-Length: 26148
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Wed, 08 Nov 2023 03:44:09 GMT
Etag: "726bcad4e2d555214ca1d6d570f7d2f9"
Last-Modified: Thu, 02 Nov 2023 04:27:03 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 1f406b32efdc23c930eb868512ad877efd625b603104620a1db3e1850896dbec
X-Amz-Request-Id: 179588FBB68133F2
X-Amz-Storage-Class: S3TIER
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

minio-client: <DEBUG> Response Time:  81.915154ms

{
 "status": "error",
 "error": {
  "message": "Failed to copy `http://localhost/test/2/apt/examples/configure-index`.",
  "cause": {
   "message": "unexpected EOF",
   "error": {}
  },
  "type": "error",
  "trace": [
   {
    "line": 332,
    "file": "client-fs.go",
    "func": "cmd.(*fsClient).put"
   },
   {
    "line": 337,
    "file": "common-methods.go",
    "func": "cmd.putTargetStream",
    "env": {
     "Tags": [
      "",
      "/tmp/configure-index"
     ]
    }
   },
   {
    "line": 607,
    "file": "common-methods.go",
    "func": "cmd.uploadSourceToTargetURL",
    "env": {
     "Tags": [
      "http://localhost/test/2/apt/examples/configure-index"
     ]
    }
   },
   {
    "line": 610,
    "file": "cp-main.go",
    "func": "cmd.doCopySession",
    "env": {
     "Tags": [
      "http://localhost/test/2/apt/examples/configure-index"
     ]
    }
   }
  ],
  "sysinfo": {
   "host.arch": "amd64",
   "host.cpus": "2",
   "host.lang": "go1.21.3",
   "host.name": "minio-c1-lb1",
   "host.os": "linux",
   "mem.heap.total": "12 MiB",
   "mem.heap.used": "4.3 MiB",
   "mem.total": "18 MiB",
   "mem.used": "4.3 MiB"
  }
 }
}
```


## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include minio version or commit id or PR# that caused this regression, if you have these details. -->

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`):
```
root@minio-c1-node4:~# minio --version
minio version RELEASE.2023-11-01T18-37-25Z (commit-id=4b4a98d5e59354870325ad19703fba03d1b104c2)
Runtime: go1.21.3 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2023 MinIO, Inc.
```
* Server setup and configuration:
  * 2 CPU
  * 4 GB mem
  * System disk 24GB /
  * Disk 16GB /mnt/disk1
  * Disk 16GB /mnt/disk2
  * Disk 16GB /mnt/disk3
  * Disk 16GB /mnt/disk4

* Operating System and version (`uname -a`):
```
root@minio-c1-node4:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@minio-c1-node4:~# uname -a
Linux minio-c1-node4 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```



Discussion / hints:
Looks like a duplicate of #18369 
additional debug  for 
```
root@minio-c1-lb1:~# minio-client --json cp  MYMINIO/test/4/apport/NEWS.gz /tmp/3e3e3e3e3/
{
 "status": "success",
 "source": "MYMINIO/test/4/apport/NEWS.gz",
 "target": "/tmp/3e3e3e3e3/NEWS.gz",
 "size": 46414,
 "totalCount": 1,
 "totalSize": 0
}
{
 "status": "error",
 "error": {
  "message": "Failed to copy `http://localhost/test/4/apport/NEWS.gz`.",
  "cause": {
   "message": "unexpected EOF",
   "error": {}
  },
  "type": "error"
 }
}
```


```
minio-client  admin trace --call s3 -v MYMINIO

minio-c1-node6.mcs.local:9000 [REQUEST s3.HeadObject] [2023-11-08T12:41:27.043] [Client IP: 127.0.0.1]
minio-c1-node6.mcs.local:9000 HEAD /test/4/apport/NEWS.gz
minio-c1-node6.mcs.local:9000 Proto: HTTP/1.1
minio-c1-node6.mcs.local:9000 Host: localhost
minio-c1-node6.mcs.local:9000 Content-Length: 0
minio-c1-node6.mcs.local:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
minio-c1-node6.mcs.local:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio-c1-node6.mcs.local:9000 X-Amz-Date: 20231108T054127Z
minio-c1-node6.mcs.local:9000 X-Forwarded-For: 127.0.0.1
minio-c1-node6.mcs.local:9000 Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3bfaa4a5575db8ebc9c26717067186307c96de727c01f13298ff4b3899610231
minio-c1-node6.mcs.local:9000
minio-c1-node6.mcs.local:9000 [RESPONSE] [2023-11-08T12:41:27.050] [ Duration 7.46ms  ↑ 93 B  ↓ 0 B ]
minio-c1-node6.mcs.local:9000 200 OK
minio-c1-node6.mcs.local:9000 ETag: "203aed2666d9bde475abb067afca921b"
minio-c1-node6.mcs.local:9000 Server: MinIO
minio-c1-node6.mcs.local:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
minio-c1-node6.mcs.local:9000 Vary: Origin,Accept-Encoding
minio-c1-node6.mcs.local:9000 X-Amz-Id-2: 6dadb366fe418cabbb9c70e460a548543a3b2eb339c7e623dc02c4bf6689a604
minio-c1-node6.mcs.local:9000 X-Amz-Request-Id: 17958F625DAAF8E0
minio-c1-node6.mcs.local:9000 X-Content-Type-Options: nosniff
minio-c1-node6.mcs.local:9000 Accept-Ranges: bytes
minio-c1-node6.mcs.local:9000 Content-Length: 46414
minio-c1-node6.mcs.local:9000 Content-Type: application/gzip
minio-c1-node6.mcs.local:9000 Last-Modified: Thu, 02 Nov 2023 04:28:46 GMT
minio-c1-node6.mcs.local:9000 X-Xss-Protection: 1; mode=block
minio-c1-node6.mcs.local:9000 x-amz-storage-class: S3TIER
minio-c1-node6.mcs.local:9000
minio-c1-node6.mcs.local:9000
minio-c1-node5.mcs.local:9000 [REQUEST s3.HeadObject] [2023-11-08T12:41:27.034] [Client IP: 127.0.0.1]
minio-c1-node5.mcs.local:9000 HEAD /test/4/apport/NEWS.gz
minio-c1-node5.mcs.local:9000 Proto: HTTP/1.1
minio-c1-node5.mcs.local:9000 Host: localhost
minio-c1-node5.mcs.local:9000 Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3bfaa4a5575db8ebc9c26717067186307c96de727c01f13298ff4b3899610231
minio-c1-node5.mcs.local:9000 Content-Length: 0
minio-c1-node5.mcs.local:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
minio-c1-node5.mcs.local:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio-c1-node5.mcs.local:9000 X-Amz-Date: 20231108T054127Z
minio-c1-node5.mcs.local:9000 X-Forwarded-For: 127.0.0.1
minio-c1-node5.mcs.local:9000
minio-c1-node5.mcs.local:9000 [RESPONSE] [2023-11-08T12:41:27.041] [ Duration 7.163ms  ↑ 93 B  ↓ 0 B ]
minio-c1-node5.mcs.local:9000 200 OK
minio-c1-node5.mcs.local:9000 ETag: "203aed2666d9bde475abb067afca921b"
minio-c1-node5.mcs.local:9000 Server: MinIO
minio-c1-node5.mcs.local:9000 X-Content-Type-Options: nosniff
minio-c1-node5.mcs.local:9000 x-amz-storage-class: S3TIER
minio-c1-node5.mcs.local:9000 Content-Type: application/gzip
minio-c1-node5.mcs.local:9000 Content-Length: 46414
minio-c1-node5.mcs.local:9000 Last-Modified: Thu, 02 Nov 2023 04:28:46 GMT
minio-c1-node5.mcs.local:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
minio-c1-node5.mcs.local:9000 Vary: Origin,Accept-Encoding
minio-c1-node5.mcs.local:9000 X-Amz-Id-2: 13fd1dddc198a66a2be6c5316f866284c6aa4af17941933c6a5cdc95e4eed4ea
minio-c1-node5.mcs.local:9000 X-Amz-Request-Id: 17958F625D248D4C
minio-c1-node5.mcs.local:9000 X-Xss-Protection: 1; mode=block
minio-c1-node5.mcs.local:9000 Accept-Ranges: bytes
minio-c1-node5.mcs.local:9000
minio-c1-node5.mcs.local:9000
minio-c1-node1.mcs.local:9000 [REQUEST s3.GetObject] [2023-11-08T12:41:27.052] [Client IP: 127.0.0.1]
minio-c1-node1.mcs.local:9000 GET /test/4/apport/NEWS.gz
minio-c1-node1.mcs.local:9000 Proto: HTTP/1.1
minio-c1-node1.mcs.local:9000 Host: localhost
minio-c1-node1.mcs.local:9000 If-Match: "203aed2666d9bde475abb067afca921b"
minio-c1-node1.mcs.local:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
minio-c1-node1.mcs.local:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio-c1-node1.mcs.local:9000 X-Amz-Date: 20231108T054127Z
minio-c1-node1.mcs.local:9000 X-Forwarded-For: 127.0.0.1
minio-c1-node1.mcs.local:9000 Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;if-match;x-amz-content-sha256;x-amz-date, Signature=b2d5bd28022c16ad43ba105b862c504b7ad03e6d7df66fcc32b78a920d4ff3e8
minio-c1-node1.mcs.local:9000 Content-Length: 0
minio-c1-node1.mcs.local:9000 <BLOB>
minio-c1-node1.mcs.local:9000 [RESPONSE] [2023-11-08T12:41:27.134] [ Duration 82.158ms  ↑ 102 B  ↓ 16 KiB ]
minio-c1-node1.mcs.local:9000 200 OK
minio-c1-node1.mcs.local:9000 x-amz-storage-class: S3TIER
minio-c1-node1.mcs.local:9000 Accept-Ranges: bytes
minio-c1-node1.mcs.local:9000 Server: MinIO
minio-c1-node1.mcs.local:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
minio-c1-node1.mcs.local:9000 X-Amz-Id-2: 6c6c4d6627746654b4b41bea40a73f2b790a9b5595239c1b7913c6d80ca4c243
minio-c1-node1.mcs.local:9000 X-Amz-Request-Id: 17958F625E3A9628
minio-c1-node1.mcs.local:9000 X-Content-Type-Options: nosniff
minio-c1-node1.mcs.local:9000 X-Xss-Protection: 1; mode=block
minio-c1-node1.mcs.local:9000 Content-Length: 46414
minio-c1-node1.mcs.local:9000 Content-Type: application/gzip
minio-c1-node1.mcs.local:9000 ETag: "203aed2666d9bde475abb067afca921b"
minio-c1-node1.mcs.local:9000 Last-Modified: Thu, 02 Nov 2023 04:28:46 GMT
minio-c1-node1.mcs.local:9000 Vary: Origin,Accept-Encoding
minio-c1-node1.mcs.local:9000 <BLOB>
minio-c1-node1.mcs.local:9000
minio-c1-node2.mcs.local:9000 [REQUEST s3.GetBucketLocation] [2023-11-08T12:41:27.030] [Client IP: 127.0.0.1]
minio-c1-node2.mcs.local:9000 GET /test/?location=
minio-c1-node2.mcs.local:9000 Proto: HTTP/1.1
minio-c1-node2.mcs.local:9000 Host: localhost
minio-c1-node2.mcs.local:9000 Authorization: AWS4-HMAC-SHA256 Credential=admin/20231108/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f0049d06778d296fd37d0a5898dd60724808962a69250f66172c576450d70eec
minio-c1-node2.mcs.local:9000 Content-Length: 0
minio-c1-node2.mcs.local:9000 User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 minio-client/RELEASE.2023-10-30T18-43-32Z
minio-c1-node2.mcs.local:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio-c1-node2.mcs.local:9000 X-Amz-Date: 20231108T054127Z
minio-c1-node2.mcs.local:9000 X-Forwarded-For: 127.0.0.1
minio-c1-node2.mcs.local:9000
minio-c1-node2.mcs.local:9000 [RESPONSE] [2023-11-08T12:41:27.032] [ Duration 2.577ms  ↑ 93 B  ↓ 128 B ]
minio-c1-node2.mcs.local:9000 200 OK
minio-c1-node2.mcs.local:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
minio-c1-node2.mcs.local:9000 X-Amz-Id-2: fe0b38b6f6141906a07c243960ccdc8ae3de33362b20de3ab918889d190ae729
minio-c1-node2.mcs.local:9000 X-Amz-Request-Id: 17958F625CE1A98E
minio-c1-node2.mcs.local:9000 Content-Type: application/xml
minio-c1-node2.mcs.local:9000 Content-Length: 128
minio-c1-node2.mcs.local:9000 Server: MinIO
minio-c1-node2.mcs.local:9000 Vary: Origin,Accept-Encoding
minio-c1-node2.mcs.local:9000 X-Content-Type-Options: nosniff
minio-c1-node2.mcs.local:9000 X-Xss-Protection: 1; mode=block
minio-c1-node2.mcs.local:9000 Accept-Ranges: bytes
minio-c1-node2.mcs.local:9000 <?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></LocationConstraint>
minio-c1-node2.mcs.local:9000

```
If the new pool is taken out of operation, the behavior becomes correct
Fascinating that seems like a rather silly bug.
> Fascinating that seems like a rather silly bug.

Since the problem is solved by decommissioning the new pool, I suspect that the problem is related to proxying requests to neighboring pool hosts.
will tracking help
```
minio-client  admin trace -a -v MYMINIO >xxx.log
```
?

I'm only worried about specifying Authorization: Bearer in the log file.

No it is not related to that, it is some minor issue. 

Fix patch:
diff --git a/.github/workflows/vulncheck.yml b/.github/workflows/vulncheck.yml
index d263b16f1c384..25899b942ccc9 100644
--- a/.github/workflows/vulncheck.yml
+++ b/.github/workflows/vulncheck.yml
@@ -22,7 +22,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v3
         with:
-          go-version: 1.21.3
+          go-version: 1.21.4
           check-latest: true
       - name: Get official govulncheck
         run: go install golang.org/x/vuln/cmd/govulncheck@latest
diff --git a/cmd/storage-datatypes.go b/cmd/storage-datatypes.go
index 7595ddadbd8c8..beab780450f14 100644
--- a/cmd/storage-datatypes.go
+++ b/cmd/storage-datatypes.go
@@ -296,7 +296,11 @@ func (fi FileInfo) GetDataDir() string {
 // InlineData returns true if object contents are inlined alongside its metadata.
 func (fi FileInfo) InlineData() bool {
 	_, ok := fi.Metadata[ReservedMetadataPrefixLower+"inline-data"]
-	return ok
+	// Earlier MinIO versions didn't reset "x-minio-internal-inline-data"
+	// from fi.Metadata when the object was tiered. So, tiered objects
+	// would return true for InlineData() in these versions even though the
+	// object isn't inlined in xl.meta
+	return ok && !fi.IsRemote()
 }
 
 // SetInlineData marks object (version) as inline.
diff --git a/cmd/xl-storage-format-v2.go b/cmd/xl-storage-format-v2.go
index efaf19c5042dc..2531f233ef9cc 100644
--- a/cmd/xl-storage-format-v2.go
+++ b/cmd/xl-storage-format-v2.go
@@ -536,6 +536,10 @@ func (j xlMetaV2Object) InlineData() bool {
 	return ok
 }
 
+func (j *xlMetaV2Object) ResetInlineData() {
+	delete(j.MetaSys, ReservedMetadataPrefixLower+"inline-data")
+}
+
 const (
 	metaTierStatus    = ReservedMetadataPrefixLower + TransitionStatus
 	metaTierObjName   = ReservedMetadataPrefixLower + TransitionedObjectName
@@ -1440,6 +1444,7 @@ func (x *xlMetaV2) DeleteVersion(fi FileInfo) (string, error) {
 			err = x.setIdx(i, *ver)
 		case fi.TransitionStatus == lifecycle.TransitionComplete:
 			ver.ObjectV2.SetTransition(fi)
+			ver.ObjectV2.ResetInlineData()
 			err = x.setIdx(i, *ver)
 		default:
 			x.versions = append(x.versions[:i], x.versions[i+1:]...)


Test patch: