Repository: minio/minio (language: go)

Issue:
Inconsistent versions between s3.ListObjectVersions s3.GetObjectTagging
For some objects *s3.GetObjectTagging* returns a 404 error with `The specified version does not exist` message although *s3.ListObjectVersions* returns this versions et *s3.GetObject* seems to work.
The *s3.DeleteObject* call on the version succeed, but it's not actually deleted.

## Expected Behavior
Either that the versions to does not exist or that *s3.GetObjectTagging* returns without error. 

## Current Behavior
Inconsistent.

## Steps to Reproduce (for bugs)
I can't reproduce it on new objects, it's probably old object's versions that are borked.

## Context
A few months ago we expended the pool and tried to rebalance the cluster and hit #17421, so we just stopped the rebalance and never run it again.
We also have tiering enabled for old versions so most (not all) versions in this state are on tiered storage.

I suspect that broken versions are still in the original erasure zone while newer versions are in a new one.

## Possible Solution

Starting the rebalance. But would it help ? And is it safe in this state ?

## Your Environment
* Version used (`minio --version`): 2023-09-30T07:02:29Z
* Server setup and configuration: K8s v1.26.5+k0s &  MinIO operator v5.0.9
* Operating System and version (`uname -a`): Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux


Discussion / hints:
Could you post the full HTTP requests, including the payload of the list request?
```
mcli: <DEBUG> GET /wombat-z-wo-de/?delimiter=%2F&encoding-type=url&prefix=urn%3Aoid%3A99544&versions= HTTP/1.1
Host: hot-objects.liiib.re
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 mcli/RELEASE.2023-10-04T06-52-56Z
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231009T133846Z
Accept-Encoding: gzip

mcli: <DEBUG> HTTP/1.1 200 OK
Content-Length: 47824
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Mon, 09 Oct 2023 13:38:46 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: c193288baa267169ce33add640c106a6854350740472e7d5c00c0311830d2c54
X-Amz-Request-Id: 178C740583AAAEFE
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
```

```
mcli: <DEBUG> GET /wombat-z-wo-de/urn%3Aoid%3A99544?tagging=&versionId=c3d44ac9-bd23-4121-a186-407ec1a8c72b HTTP/1.1
Host: hot-objects.liiib.re
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 mcli/RELEASE.2023-10-04T06-52-56Z
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20231009T134038Z
Accept-Encoding: gzip

mcli: <DEBUG> HTTP/1.1 404 Not Found
Content-Length: 369
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Mon, 09 Oct 2023 13:40:39 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: c193288baa267169ce33add640c106a6854350740472e7d5c00c0311830d2c54
X-Amz-Request-Id: 178C741FAC55086B
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchVersion</Code><Message>The specified version does not exist.</Message><Key>urn:oid:99544</Key><BucketName>wombat-z-wo-de</BucketName><Resource>/wombat-z-wo-de/urn:oid:99544</Resource><RequestId>178C741FAC55086B</RequestId><HostId>c193288baa267169ce33add640c106a6854350740472e7d5c00c0311830d2c54</HostId></Error>
```
Could you list the output is `mc stat --debug --json --versions ALIAS/wombat-z-wo-de/urn:oid:99544`?
[stat-output.txt](https://hot-objects.liiib.re/share/stat-output.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=hougo%2F20231009%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231009T140821Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=cc3933462c9a73143bd7cee0df3ee7a507f949e26592dde970ba18d8541ab3b5)
Thanks! Looks like everything *should* work. I will see if I can find anything strange in the code.
```
mcli: <DEBUG> HEAD /wombat-z-wo-de/urn%3Aoid%3A99544?versionId=c3d44ac9-bd23-4121-a186-407ec1a8c72b HTTP/1.1
Host: hot-objects.liiib.re
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 mcli/RELEASE.2023-10-04T06-52-56Z
Authorization: AWS4-HMAC-SHA256 Credential=hougo/20231009/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231009T140643Z

mcli: <DEBUG> HTTP/1.1 200 OK
Content-Length: 221245
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/vnd.openxmlformats-officedocument.presentationml.presentation
Date: Mon, 09 Oct 2023 14:06:43 GMT
Etag: "9c9e5eee056fae7d77f6e284b3723ec0"
Last-Modified: Mon, 19 Jun 2023 16:28:24 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 871a9478ef7473f7e6609b4b705e9c6cffec7f8a629ed157410752289343ea33
X-Amz-Request-Id: 178C758BE0CDC37E
X-Amz-Version-Id: c3d44ac9-bd23-4121-a186-407ec1a8c72b
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mcli: <DEBUG> TLS Certificate found: 
mcli: <DEBUG>  >> Country: US
mcli: <DEBUG>  >> Organization: Let's Encrypt
mcli: <DEBUG>  >> Expires: 2023-12-06 00:44:12 +0000 UTC
mcli: <DEBUG> TLS Certificate found: 
mcli: <DEBUG>  >> Country: US
mcli: <DEBUG>  >> Organization: Internet Security Research Group
mcli: <DEBUG>  >> Expires: 2025-09-15 16:00:00 +0000 UTC
mcli: <DEBUG> TLS Certificate found: 
mcli: <DEBUG>  >> Organization: Digital Signature Trust Co.
mcli: <DEBUG>  >> Expires: 2024-09-30 18:14:03 +0000 UTC
mcli: <DEBUG> Response Time:  477.778025ms

{"status":"success","name":"urn:oid:99544","lastModified":"2023-06-19T18:28:24+02:00","size":221245,"etag":"9c9e5eee056fae7d77f6e284b3723ec0","type":"file","metadata":{"Content-Type":"application/vnd.openxmlformats-officedocument.presentationml.presentation"},"versionID":"c3d44ac9-bd23-4121-a186-407ec1a8c72b"}
```

Can't really see anything that would cause this. In the end it should end up in the same GetObjectInfo call. Any ideas @harshavardhana 

Fix patch:
diff --git a/cmd/erasure-server-pool.go b/cmd/erasure-server-pool.go
index 2305e1bb20b36..e6b9f39035174 100644
--- a/cmd/erasure-server-pool.go
+++ b/cmd/erasure-server-pool.go
@@ -2451,12 +2451,12 @@ func (z *erasureServerPools) GetObjectTags(ctx context.Context, bucket, object s
 		return z.serverPools[0].GetObjectTags(ctx, bucket, object, opts)
 	}
 
-	idx, err := z.getPoolIdxExistingWithOpts(ctx, bucket, object, opts)
+	oi, _, err := z.getLatestObjectInfoWithIdx(ctx, bucket, object, opts)
 	if err != nil {
 		return nil, err
 	}
 
-	return z.serverPools[idx].GetObjectTags(ctx, bucket, object, opts)
+	return tags.ParseObjectTags(oi.UserTags)
 }
 
 // TransitionObject - transition object content to target tier.


Test patch: