Repository: minio/minio (language: go)

Issue:
S3:CopyObject on transitioned objects is not working
## Expected Behavior
If S3:CopyObject on a transitioned object is executed, a new, non-transitioned object version with same content/tags/... is created.
The object copy is accessible.

## Current Behavior
If S3:CopyObject is executed, a non-accessible object copy is created.
Trying to download the copied object from the Web-UI results in an 500 internal server error:

```API: GetObject(bucket=test, object=two/urn:oid:119, versionId=null)
Time: 21:17:32 UTC 10/19/2022
DeploymentID: 09d296a9-ea83-485e-8c65-b485a938e7b6
RequestID: 171F9508E874B820
RemoteHost: 10.244.1.216
Host: 10.244.1.216:9000
UserAgent: MinIO (linux; amd64) minio-go/v7.0.41 MinIO Console/(dev)
Error: sio: authentication failed (sio.Error)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/api-errors.go:2172:cmd.toAPIErrorCode()
       4: cmd/api-errors.go:2197:cmd.toAPIError()
       3: cmd/object-handlers.go:568:cmd.objectAPIHandlers.getObjectHandler()
       2: cmd/object-handlers.go:629:cmd.objectAPIHandlers.GetObjectHandler()
       1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
```
All Metadata from the original object is visible within the Web-UI at the object copy, including the X-Amz-Storage-Class (Tier name) of the original object.
After deleting the original, transitioned object and trying to access the object copy, the error changes from "500 internal server error" to 404 Not Found.

## Steps to Reproduce (for bugs)
1. Create a new Object
2. Add a Tier
3. Add a Life-Cycle Policy to transition current objects after some days to the tier of step 2
4. Wait until your object is transitioned
5. Use S3 Server-Side Copy (S3:CopyObject) to copy the transitioned object inside the bucket
6. Try to access/download the copied object

## Your Environment
* Version used (`minio --version`): minio version RELEASE.2022-10-15T19-57-03Z (commit-id=85fc7cea979b8da1689798e1e290847fd6517c55)
* Server setup and configuration:
  - Minio is deployed in Single-Node Single-Drive setup within a Kubernetes environment (RKE2, v1.24.6)
  - KES is used for SSE-KMS
  - test bucket has bucket encryption enabled by using SSE-KMS
  - test bucket is unversioned
  - Storage tier is a S3 compatible provider (Backblaze B2)
* Operating System and version (`uname -a`): Debian 11, SMP Debian 5.10.140-1 (2022-09-02) x86_64 x86_64 x86_64 GNU/Linux



Discussion / hints:
Thanks for the report we will take a look. 
I followed the steps to reproduce with the only difference that I used another MinIO deployment as warm tier instead of Backblaze's B2. I **couldn't** recreate this issue. Could you try to recreate this issue using another MinIO deployment as your warm-tier?
I used the following docker-compose to create repeatable setup. In this setup, 
- `minio` service refers to the primary MinIO object storage, listening on `hostIP:9000`
- `miniotier` service refers to the other MinIO deployment used as warm tier, listening on `hostIP:9011`

```
version: '3.7'
x-minio-common: &minio-common
  image: minio/minio:${RELEASE}
  networks:
    - minio_net
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3

services:
  minio:
    <<: *minio-common
    command: server /data1
    ports:
      - "9000:9000"
    volumes:
      - data-1:/data1:rw
    environment:
      - MINIO_ROOT_USER=minio
      - MINIO_ROOT_PASSWORD=minio123
      - MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=

  miniotier:
    <<: *minio-common
    command: server /data1
    ports:
      - "9011:9000"
    volumes:
      - datatier-1:/data1:rw
    environment:
      - MINIO_ROOT_USER=minio
      - MINIO_ROOT_PASSWORD=minio123

volumes:
  data-1:
  datatier-1:

networks:
  minio_net:
```
I used the following script to create the test object in preparation for reproducing this issue. You can copy this script to say `test-15906.sh` locally and run it like so,
Usage:
```
./test-15906.sh minio-test <hostIP>
```

```sh
#!/bin/bash

ALIAS=$1
IP=$2

if [ $# -lt 2 ]
then
    echo "missing ALIAS and IP"
    exit 1
fi

./mc alias set $ALIAS-target http://$IP:9011 minio minio123

./mc mb $ALIAS-target/testbucket

# Add a remote tier
echo "adding a remote tier"
./mc admin tier add minio $ALIAS WARMTIER-1 --endpoint http://$IP:9011 --access-key minio --secret-key minio123 --bucket testbucket --prefix testprefix-1

# Make a bucket to set ILM transition on
./mc mb $ALIAS/mybucket

./mc encrypt set sse-s3 $ALIAS/mybucket

# For immediate tiering
./mc ilm add --tier "WARMTIER-1" --transition-days="0" $ALIAS/mybucket

./mc cp /etc/hosts $ALIAS/mybucket/obj-1
```
Thank you for your investigation.

I was able to reproduce the bug with your provided test setup.
To perform a Server-Side Copy, i used rclone copyto. I extended your testscript to reproduce the actual bug directly:

test.sh:
```
#!/bin/bash

ALIAS=$1
IP=$2

if [ $# -lt 2 ]
then
    echo "missing ALIAS and IP"
    exit 1
fi

./mc alias set $ALIAS-tier http://$ip:9011/ minio minio123
./mc alias set $ALIAS http://$ip:9000/ minio minio123
./mc mb $ALIAS-tier/testbucket

# Add a remote tier
echo "adding a remote tier"
./mc admin tier add minio $ALIAS WARMTIER-1 --endpoint http://$ip:9011/ --access-key minio --secret-key minio123 --bucket testbucket --prefix testprefix-1

# Make a bucket to set ILM transition on
./mc mb $ALIAS/mybucket

./mc encrypt set sse-s3 $ALIAS/mybucket

# For immediate tiering
./mc ilm add --tier "WARMTIER-1" --transition-days="0" $ALIAS/mybucket

./mc cp /etc/hosts $ALIAS/mybucket/obj-1

# Execute a Server-Side Copy of the Object
./rclone copyto minio:mybucket/obj-1 minio:mybucket/obj-1-cpy --config=./rclone.conf -vvv

# Head copied object
./mc stat $ALIAS/mybucket/obj-1-cpy
./mc head $ALIAS/mybucket/obj-1-cpy --debug
``` 

rclone.conf:
```
[minio]
type = s3
provider = Minio
access_key_id = minio
secret_access_key = minio123
endpoint = http://SERVER_IP:9000/
server_side_encryption = AES256
```

Please change the SERVER_IP in rclone.conf to the one in your test setup.
To reproduce the second part of the bug, execute the following commands after the test script has finished:

```
# Remove the original object
./mc rm test/mybucket/obj-1

# Access the object copy
./mc head test/mybucket/obj-1-cpy
```
This will result in the follwoing error:

```
mc: <ERROR> Unable to read from `test/mybucket/obj-1-cpy`. The specified key does not exist.`
```

But the object copy is still listed inside the bucket:
```
foo@ubuntu-22:~/workspace/minio$ ./mc ls test/mybucket
[2022-10-24 14:20:03 CEST]   342B STANDARD obj-1-cpy
```


Fix patch:
diff --git a/cmd/object-handlers.go b/cmd/object-handlers.go
index 41f8f508028db..3c47d4076f297 100644
--- a/cmd/object-handlers.go
+++ b/cmd/object-handlers.go
@@ -909,6 +909,11 @@ func getCpObjMetadataFromHeader(ctx context.Context, r *http.Request, userMeta m
 	// to change the original one.
 	defaultMeta := make(map[string]string, len(userMeta))
 	for k, v := range userMeta {
+		// skip tier metadata when copying metadata from source object
+		switch k {
+		case metaTierName, metaTierStatus, metaTierObjName, metaTierVersionID:
+			continue
+		}
 		defaultMeta[k] = v
 	}
 
diff --git a/cmd/xl-storage-format-v2.go b/cmd/xl-storage-format-v2.go
index 14a25ffc3bd47..1e9e6fb921f2d 100644
--- a/cmd/xl-storage-format-v2.go
+++ b/cmd/xl-storage-format-v2.go
@@ -456,9 +456,9 @@ func (j xlMetaV2DeleteMarker) ToFileInfo(volume, path string) (FileInfo, error)
 	fi.ReplicationState = GetInternalReplicationState(j.MetaSys)
 	if j.FreeVersion() {
 		fi.SetTierFreeVersion()
-		fi.TransitionTier = string(j.MetaSys[ReservedMetadataPrefixLower+TransitionTier])
-		fi.TransitionedObjName = string(j.MetaSys[ReservedMetadataPrefixLower+TransitionedObjectName])
-		fi.TransitionVersionID = string(j.MetaSys[ReservedMetadataPrefixLower+TransitionedVersionID])
+		fi.TransitionTier = string(j.MetaSys[metaTierName])
+		fi.TransitionedObjName = string(j.MetaSys[metaTierObjName])
+		fi.TransitionVersionID = string(j.MetaSys[metaTierVersionID])
 	}
 
 	return fi, nil
@@ -487,7 +487,7 @@ func (j *xlMetaV2DeleteMarker) Signature() [4]byte {
 // its contents and false otherwise.
 func (j xlMetaV2Object) UsesDataDir() bool {
 	// Skip if this version is not transitioned, i.e it uses its data directory.
-	if !bytes.Equal(j.MetaSys[ReservedMetadataPrefixLower+TransitionStatus], []byte(lifecycle.TransitionComplete)) {
+	if !bytes.Equal(j.MetaSys[metaTierStatus], []byte(lifecycle.TransitionComplete)) {
 		return true
 	}
 
@@ -503,11 +503,18 @@ func (j xlMetaV2Object) InlineData() bool {
 	return ok
 }
 
+const (
+	metaTierStatus    = ReservedMetadataPrefixLower + TransitionStatus
+	metaTierObjName   = ReservedMetadataPrefixLower + TransitionedObjectName
+	metaTierVersionID = ReservedMetadataPrefixLower + TransitionedVersionID
+	metaTierName      = ReservedMetadataPrefixLower + TransitionTier
+)
+
 func (j *xlMetaV2Object) SetTransition(fi FileInfo) {
-	j.MetaSys[ReservedMetadataPrefixLower+TransitionStatus] = []byte(fi.TransitionStatus)
-	j.MetaSys[ReservedMetadataPrefixLower+TransitionedObjectName] = []byte(fi.TransitionedObjName)
-	j.MetaSys[ReservedMetadataPrefixLower+TransitionedVersionID] = []byte(fi.TransitionVersionID)
-	j.MetaSys[ReservedMetadataPrefixLower+TransitionTier] = []byte(fi.TransitionTier)
+	j.MetaSys[metaTierStatus] = []byte(fi.TransitionStatus)
+	j.MetaSys[metaTierObjName] = []byte(fi.TransitionedObjName)
+	j.MetaSys[metaTierVersionID] = []byte(fi.TransitionVersionID)
+	j.MetaSys[metaTierName] = []byte(fi.TransitionTier)
 }
 
 func (j *xlMetaV2Object) RemoveRestoreHdrs() {
@@ -626,16 +633,16 @@ func (j xlMetaV2Object) ToFileInfo(volume, path string) (FileInfo, error) {
 	}
 	fi.DataDir = uuid.UUID(j.DataDir).String()
 
-	if st, ok := j.MetaSys[ReservedMetadataPrefixLower+TransitionStatus]; ok {
+	if st, ok := j.MetaSys[metaTierStatus]; ok {
 		fi.TransitionStatus = string(st)
 	}
-	if o, ok := j.MetaSys[ReservedMetadataPrefixLower+TransitionedObjectName]; ok {
+	if o, ok := j.MetaSys[metaTierObjName]; ok {
 		fi.TransitionedObjName = string(o)
 	}
-	if rv, ok := j.MetaSys[ReservedMetadataPrefixLower+TransitionedVersionID]; ok {
+	if rv, ok := j.MetaSys[metaTierVersionID]; ok {
 		fi.TransitionVersionID = string(rv)
 	}
-	if sc, ok := j.MetaSys[ReservedMetadataPrefixLower+TransitionTier]; ok {
+	if sc, ok := j.MetaSys[metaTierName]; ok {
 		fi.TransitionTier = string(sc)
 	}
 	if crcs := j.MetaSys[ReservedMetadataPrefixLower+"crc"]; len(crcs) > 0 {
@@ -1528,16 +1535,16 @@ func (x *xlMetaV2) AddVersion(fi FileInfo) error {
 		}
 
 		if fi.TransitionStatus != "" {
-			ventry.ObjectV2.MetaSys[ReservedMetadataPrefixLower+TransitionStatus] = []byte(fi.TransitionStatus)
+			ventry.ObjectV2.MetaSys[metaTierStatus] = []byte(fi.TransitionStatus)
 		}
 		if fi.TransitionedObjName != "" {
-			ventry.ObjectV2.MetaSys[ReservedMetadataPrefixLower+TransitionedObjectName] = []byte(fi.TransitionedObjName)
+			ventry.ObjectV2.MetaSys[metaTierObjName] = []byte(fi.TransitionedObjName)
 		}
 		if fi.TransitionVersionID != "" {
-			ventry.ObjectV2.MetaSys[ReservedMetadataPrefixLower+TransitionedVersionID] = []byte(fi.TransitionVersionID)
+			ventry.ObjectV2.MetaSys[metaTierVersionID] = []byte(fi.TransitionVersionID)
 		}
 		if fi.TransitionTier != "" {
-			ventry.ObjectV2.MetaSys[ReservedMetadataPrefixLower+TransitionTier] = []byte(fi.TransitionTier)
+			ventry.ObjectV2.MetaSys[metaTierName] = []byte(fi.TransitionTier)
 		}
 		if len(fi.Checksum) > 0 {
 			ventry.ObjectV2.MetaSys[ReservedMetadataPrefixLower+"crc"] = fi.Checksum


Test patch: