Repository: minio/minio (language: go)

Issue:
minio does not heal metadata on transitionned object (ilm tiering)
<!--- Provide a general summary of the issue in the Title above -->
When using tiering the hot/front tier hold the metadata for all objects accross all tiers.
The metadata also holds data of inlined objects (size < 128K)

## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

- When I heal a drive on the hot tier cluster, I should not have minio log errors for objects that have already transitionned to the backend tier.
Right now, I have error like this one below for all transitionned objects:
`Error: unable to heal object thanos/index: Storage resources are insufficient for the read operation thanos/index (*fmt.wrapError)`
The object can be listed or stat: 
`sudo mc ls myminio-local/thanos/index 
[2024-10-03 11:17:11 CEST] 512MiB COLD-THANOS index`

- When I heal a drive on the hot tier cluster, minio must be able to re-generate the missing xl.meta files for metadata of transitionned objects. If this is not possible, then minio will progressively loose data at each disk replacement.

## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
minio tries to heal all objects from a tiered setup, even if the object is not stored on the current cluster.
OK, but the healing on the hot tier should work without error even if the object already transitionned to the backend tier.
Otherwise, it means that healing is not supported once tiering is setup.

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`): 2024-09-22T00:33:43Z
* Server setup and configuration:
* Operating System and version (`uname -a`):



Discussion / hints:
The healing is now terminated on the node that I completely rebuild.

All objects that were transitionned have no metadata on this new node.
In the disk the files .meta.xl does not exist for transitionned object. Thus the metadata are corrupted on the new node.

This is a HUGE bug to us, since it basically means Tiering looses data.
Hello,

Guys, I don't get it.
To my view minio tiering is loosing data and should not be considered prod ready.
From what I see, when you use parity=2 once you completely rebuilt 1 node from the hot tier then any incident on the transitionned objects that were part of the first rebuilt will cause a data loss.

It bites me in production but it can be easily reproduced.

I have a setup with 4 VM in EC4:2 (hot tier) and 1 single VM as cold tier. 
I generated data with warp that transitionned to the cold tier.

using the commands below to simulate an outage on a given object I can observe that the healing engine of minio cannot heal metadata of transitionned objects:

```
f='85.zvG4d0a7a)Y1t)GZ.rnd'
mc ls hot/buc/tnxXHrgA/$f
find /data/1/buc -type f | grep $f
mc ls hot/buc/tnxXHrgA/$f; date; sudo systemctl stop minio; date ; sudo rm $(find /data/1/buc -type f | grep $f); sudo systemctl start minio ; date ; mc ls hot/buc/tnxXHrgA/$f
find /data/1/buc -type f | grep $f
mc admin heal -r --scan=deep hot/buc/tnxXHrgA/
find /data/1/buc -type f | grep $f
mc ls hot/buc/tnxXHrgA/$f
mc cp hot/buc/tnxXHrgA/$f /tmp/
mc stat hot/buc/tnxXHrgA/$f
```

Please see the screenshots below where I applied this iteratively on 3 VMs from the hot tier to observe data loss

1/ Every host from the hot tier hold the metadata from the cold tier
![Screenshot from 2024-10-28 15-44-51](https://github.com/user-attachments/assets/cfca1b06-4c26-4483-b59a-85c4eb689625)

2/ Simulate hardware-replacement / incident on VM u20-1 : the data can still be accessed after the outage. Given we are using EC4:2 here and 1 erasure shard was wiped, the quorum is maintained (1 out of 4).
![Screenshot from 2024-10-28 15-49-40](https://github.com/user-attachments/assets/2f8ff820-d559-4fc8-a4da-51a4c4b0900b)

3/ Simulate hardware-replacement / incident on VM u20-3 : the data cannot be accessed anymore
![Screenshot from 2024-10-28 15-57-11](https://github.com/user-attachments/assets/00cbd210-bcbc-4e6b-a732-283f63c29d33)
In addition the heal still sees 50 objects in this directory despite the fact that one of the objects cannot be listed consistently.
Given we are using EC4:2 here and 2 erasure shard were wiped, the quorum is not maintained (2 out of 4) which explained inconsistent lists.

4/ Simulate hardware-replacement / incident on VM u20-4 : the data cannot be accessed anymore
![Screenshot from 2024-10-28 16-09-30](https://github.com/user-attachments/assets/e9e101fb-24f4-4d0f-938a-8bb105c3ef81)
In addition the heal still sees 50 objects in this directory despite the fact that one of the objects cannot be listed consistently.

5/ As can be seen the data is still there on the cold tier : I have 63 objects in this bucket. However in the front tier I see that all objects have transitionned to minio-cold and I can only list 62 objects.
![Screenshot from 2024-10-28 17-21-33](https://github.com/user-attachments/assets/337238c4-901b-46d9-bb3d-63d052f9df4a)



I understand that unit tests in tiering is probably challenging since transition use its own schedule and cannot be force at delays below 1 day, but I am very astonished to find this bug in a production ready solution.
FTR : auto heal of non transitioned objects works fine on this setup.
![minio-normal-healing](https://github.com/user-attachments/assets/75fa53f8-d949-489a-888f-4e43ce815cb4)


please note that these tests were done with bucket versioning disabled and config api `object_max_versions=1`
When @krisis is back (next week AFAIK) he should be able to look at it. Could you collect the directories of an unhealing object from all disks and attach it here as a zip?
I will do the same experiment with the latest version of minio and attach the archive here tomorrow.
I did not find an easy way to trick the clock in order to speed up transition in less than 1 day.
I tried to run minio with `faketime -f "+1d"` but it fails.
Hello,
No surprises but I can confirm that the bug is still there on the latest version of minio.
I redid the same experiment : 
```
mc admin trace -a local | grep -i -e testfile_241028_idx20 -e  heal

rm testfile_*idx*
for i in {11..20}; do dd if=/dev/urandom bs=1K count=$i of=testfile_$(date '+%y%m%d')_idx$i; done
mc cp testfile_*idx* local/buc/adir/
mc ls -r local/buc/

mc admin info local | tail -12
date
f='testfile_241028_idx20'
mc ls local/buc/adir/$f
find /data/1/buc -type f | grep $f
ll /data/1/buc/adir/$f/xl.meta
mc admin heal -r --scan=deep -a --force local/buc
for i in range {1..10}; do printf "$(mc ls -r local/buc/ | wc -l) "; done; echo ""
mc ls local/buc/adir/$f; date; sudo systemctl stop minio; date ; sudo rm -f $(find /data/1/buc -type f | grep $f); ll /data/1/buc/adir/$f/xl.meta; sudo systemctl start minio ; date ; ll /data/1/buc/adir/$f/xl.meta; mc ls local/buc/adir/$f;
for i in range {1..10}; do printf "$(mc ls -r local/buc/ | wc -l) "; done; echo ""
mc admin info local | tail -2
ll /data/1/buc/adir/$f/xl.meta
mc stat local/buc/adir/$f
mc admin heal -r --scan=deep -a --force local/buc
ll /data/1/buc/adir/$f/xl.meta
mc ls local/buc/adir/$f
mc cp local/buc/adir/$f /tmp/
mc stat local/buc/adir/$f
```

and same results : simulated drive replacement causes data loss since xl.meta of transitionned object is never repaired on hot tier.

Screenshots of the tests : 
- Setup
![1-minio-setup-version](https://github.com/user-attachments/assets/d1935c59-b4b3-4aff-a61d-9b0fc4805c0c)
- Setup
![2-minio-tiering-files](https://github.com/user-attachments/assets/b962bf81-a3bf-4506-91b0-618b4501566c)
- simulated incident on vm1 (hot)
![3-vm1-incident-lost-file1](https://github.com/user-attachments/assets/7c45ed15-ea34-47fa-9421-ad1f165df2ea)
- simulated incident on vm2 (hot)
![4-vm2-incident-lost-file1](https://github.com/user-attachments/assets/0d7e9a0c-1c64-4a04-8b7b-ba1bae2e3403)
- simulated incident on vm3 (hot)
![5-vm3-incident-lost-file1](https://github.com/user-attachments/assets/28d12403-2dee-4a14-a6cf-7fb671469ca4)

Archive of the whole minio drives from the 4 vm of the hot tier :  
[minio-tiering-data-loss.zip](https://github.com/user-attachments/files/17571995/minio-tiering-data-loss.zip)

@julienlau Thanks for your files & scripts. Yes. It does seem like there is a healing glitch on tiered objects.

Also, you can remove your sensationalist "LOOSE DATA", since your data isn't lost. It is just inaccessible. I know you just do this and your blog post for attention.

While we appreciate your situation, we have to prioritize our paying customers for non-trivial issues. Thank you for your understanding.
I've edited the title, even though in my view the data is lost once the metadata is lost.

I am sure you guys are doing the best you can, and I thank you for that. However, I am sorry to say that this is not the kind of bug I find acceptable in a production-grade storage solution, even if it is open source. I know this is not the Enterprise Edition I use, but data durability is the core of any storage solution. You may also understand that on my side, the cluster using Minio + tiering is just a bleeding edge pain, with every hardware fault causing a lot of inconsistency issues and manual repairs.

I agree that it may be possible to rebuild the data, but I foresee a lot of pain here (I have no expertise in Minio source code). From my perspective, it would require taking snapshots or stopping writes to the hot+cold clusters (ouch!), performing a full scan of all metadata in the hot+cold cluster (ouch!), and rebuilding the missing metadata file in the front tier using the metadata from the cold tier.
@julienlau 👍🏼 I will see what I can do. 
more facts : the data on the cold tier looks like that

EDIT : re-upload the same image due to 404
![Screenshot from 2024-10-31 12-00-50](https://github.com/user-attachments/assets/de40635e-f27b-43df-b0c6-7100b03f208e)

Yeah, we should be fine with data.

What seems to happen is that when data is tiered the part data obviously isn't there, and this condition isn't ignored all the way through the healing process.
With the data you sent, I can heal `dira` with one shard missing on master. With transitioned data, it requires [N/2+1](https://github.com/minio/minio/blob/b6b7cddc9c419d2e047063c30a92d14e264c5eca/cmd/erasure-metadata.go#L518) (=3) shards, and disallows for the split brain scenario, so this makes it not heal in 2+2 setups.

I can't see your images above. For whatever reason opening the full image gives 404.
I redid the upload the same image.

Nice to here that you manage to heal it the 1 shard missing scenario and thank you for your help. 
I did no try to manually copy the xl.meta of a transitioned object from one node to another, is it what you've done ? 
 
@julienlau I will send a PR that allows for healing 2 shards, but it will need some thorough review, since it is a sensitive system and I don't know all the details of it.

Fix patch:
diff --git a/.github/workflows/replication.yaml b/.github/workflows/replication.yaml
index 753ecf8b92fa2..71d4e14650c9d 100644
--- a/.github/workflows/replication.yaml
+++ b/.github/workflows/replication.yaml
@@ -40,6 +40,7 @@ jobs:
           sudo sysctl net.ipv6.conf.all.disable_ipv6=0
           sudo sysctl net.ipv6.conf.default.disable_ipv6=0
           make test-ilm
+          make test-ilm-transition
 
       - name: Test PBAC
         run: |
diff --git a/Makefile b/Makefile
index a68987747bb34..4d65c39c55992 100644
--- a/Makefile
+++ b/Makefile
@@ -60,6 +60,10 @@ test-ilm: install-race
 	@echo "Running ILM tests"
 	@env bash $(PWD)/docs/bucket/replication/setup_ilm_expiry_replication.sh
 
+test-ilm-transition: install-race
+	@echo "Running ILM tiering tests with healing"
+	@env bash $(PWD)/docs/bucket/lifecycle/setup_ilm_transition.sh
+
 test-pbac: install-race
 	@echo "Running bucket policies tests"
 	@env bash $(PWD)/docs/iam/policies/pbac-tests.sh
diff --git a/cmd/erasure-healing-common.go b/cmd/erasure-healing-common.go
index 1e39c2c261305..7c3c8a4d87b2f 100644
--- a/cmd/erasure-healing-common.go
+++ b/cmd/erasure-healing-common.go
@@ -392,8 +392,8 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
 		if metaErrs[i] != nil {
 			continue
 		}
-		meta := partsMetadata[i]
 
+		meta := partsMetadata[i]
 		if meta.Deleted || meta.IsRemote() {
 			continue
 		}
@@ -442,13 +442,17 @@ func disksWithAllParts(ctx context.Context, onlineDisks []StorageAPI, partsMetad
 	}
 
 	for i, onlineDisk := range onlineDisks {
-		if metaErrs[i] == nil && !hasPartErr(dataErrsByDisk[i]) {
-			// All parts verified, mark it as all data available.
-			availableDisks[i] = onlineDisk
-		} else {
-			// upon errors just make that disk's fileinfo invalid
-			partsMetadata[i] = FileInfo{}
+		if metaErrs[i] == nil {
+			meta := partsMetadata[i]
+			if meta.Deleted || meta.IsRemote() || !hasPartErr(dataErrsByDisk[i]) {
+				// All parts verified, mark it as all data available.
+				availableDisks[i] = onlineDisk
+				continue
+			}
 		}
+
+		// upon errors just make that disk's fileinfo invalid
+		partsMetadata[i] = FileInfo{}
 	}
 
 	return
diff --git a/cmd/erasure-metadata.go b/cmd/erasure-metadata.go
index 885c42c0d4b6b..ce053a07d2f73 100644
--- a/cmd/erasure-metadata.go
+++ b/cmd/erasure-metadata.go
@@ -514,8 +514,10 @@ func listObjectParities(partsMetadata []FileInfo, errs []error) (parities []int)
 		if metadata.Deleted || metadata.Size == 0 {
 			parities[index] = totalShards / 2
 		} else if metadata.TransitionStatus == lifecycle.TransitionComplete {
-			// For tiered objects, read quorum is N/2+1 to ensure simple majority on xl.meta. It is not equal to EcM because the data integrity is entrusted with the warm tier.
-			parities[index] = totalShards - (totalShards/2 + 1)
+			// For tiered objects, read quorum is N/2+1 to ensure simple majority on xl.meta.
+			// It is not equal to EcM because the data integrity is entrusted with the warm tier.
+			// However, we never go below EcM, in case of a EcM=EcN setup.
+			parities[index] = max(totalShards-(totalShards/2+1), metadata.Erasure.ParityBlocks)
 		} else {
 			parities[index] = metadata.Erasure.ParityBlocks
 		}
diff --git a/cmd/erasure-object.go b/cmd/erasure-object.go
index 8ee2ae6ddadac..80c8196804a9e 100644
--- a/cmd/erasure-object.go
+++ b/cmd/erasure-object.go
@@ -2376,7 +2376,11 @@ func (er erasureObjects) TransitionObject(ctx context.Context, bucket, object st
 	}()
 
 	var rv remoteVersionID
-	rv, err = tgtClient.Put(ctx, destObj, pr, fi.Size)
+	rv, err = tgtClient.PutWithMeta(ctx, destObj, pr, fi.Size, map[string]string{
+		"name": object, // preserve the original name of the object on the remote tier object metadata.
+		// this is just for future reverse lookup() purposes (applies only for new objects)
+		// does not apply retro-actively on already transitioned objects.
+	})
 	pr.CloseWithError(err)
 	if err != nil {
 		traceFn(ILMTransition, nil, err)
diff --git a/cmd/warm-backend-azure.go b/cmd/warm-backend-azure.go
index c39f7f4d5d05d..bb087e749cca8 100644
--- a/cmd/warm-backend-azure.go
+++ b/cmd/warm-backend-azure.go
@@ -26,6 +26,7 @@ import (
 	"strings"
 
 	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
 	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
 	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
 	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
@@ -59,10 +60,15 @@ func (az *warmBackendAzure) getDest(object string) string {
 	return destObj
 }
 
-func (az *warmBackendAzure) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
+func (az *warmBackendAzure) PutWithMeta(ctx context.Context, object string, r io.Reader, length int64, meta map[string]string) (remoteVersionID, error) {
+	azMeta := map[string]*string{}
+	for k, v := range meta {
+		azMeta[k] = to.Ptr(v)
+	}
 	resp, err := az.clnt.UploadStream(ctx, az.Bucket, az.getDest(object), io.LimitReader(r, length), &azblob.UploadStreamOptions{
 		Concurrency: 4,
 		AccessTier:  az.tier(), // set tier if specified
+		Metadata:    azMeta,
 	})
 	if err != nil {
 		return "", azureToObjectError(err, az.Bucket, az.getDest(object))
@@ -74,6 +80,10 @@ func (az *warmBackendAzure) Put(ctx context.Context, object string, r io.Reader,
 	return remoteVersionID(vid), nil
 }
 
+func (az *warmBackendAzure) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
+	return az.PutWithMeta(ctx, object, r, length, map[string]string{})
+}
+
 func (az *warmBackendAzure) Get(ctx context.Context, object string, rv remoteVersionID, opts WarmBackendGetOpts) (r io.ReadCloser, err error) {
 	if opts.startOffset < 0 {
 		return nil, InvalidRange{}
diff --git a/cmd/warm-backend-gcs.go b/cmd/warm-backend-gcs.go
index 3246fd1c8f63f..8cacb71c0f1fc 100644
--- a/cmd/warm-backend-gcs.go
+++ b/cmd/warm-backend-gcs.go
@@ -47,16 +47,17 @@ func (gcs *warmBackendGCS) getDest(object string) string {
 	return destObj
 }
 
-// FIXME: add support for remote version ID in GCS remote tier and remove this.
-// Currently it's a no-op.
-
-func (gcs *warmBackendGCS) Put(ctx context.Context, key string, data io.Reader, length int64) (remoteVersionID, error) {
+func (gcs *warmBackendGCS) PutWithMeta(ctx context.Context, key string, data io.Reader, length int64, meta map[string]string) (remoteVersionID, error) {
 	object := gcs.client.Bucket(gcs.Bucket).Object(gcs.getDest(key))
-	// TODO: set storage class
 	w := object.NewWriter(ctx)
 	if gcs.StorageClass != "" {
 		w.ObjectAttrs.StorageClass = gcs.StorageClass
 	}
+	w.ObjectAttrs.Metadata = meta
+	if _, err := xioutil.Copy(w, data); err != nil {
+		return "", gcsToObjectError(err, gcs.Bucket, key)
+	}
+
 	if _, err := xioutil.Copy(w, data); err != nil {
 		return "", gcsToObjectError(err, gcs.Bucket, key)
 	}
@@ -64,6 +65,12 @@ func (gcs *warmBackendGCS) Put(ctx context.Context, key string, data io.Reader,
 	return "", w.Close()
 }
 
+// FIXME: add support for remote version ID in GCS remote tier and remove this.
+// Currently it's a no-op.
+func (gcs *warmBackendGCS) Put(ctx context.Context, key string, data io.Reader, length int64) (remoteVersionID, error) {
+	return gcs.PutWithMeta(ctx, key, data, length, map[string]string{})
+}
+
 func (gcs *warmBackendGCS) Get(ctx context.Context, key string, rv remoteVersionID, opts WarmBackendGetOpts) (r io.ReadCloser, err error) {
 	// GCS storage decompresses a gzipped object by default and returns the data.
 	// Refer to https://cloud.google.com/storage/docs/transcoding#decompressive_transcoding
diff --git a/cmd/warm-backend-minio.go b/cmd/warm-backend-minio.go
index 94c09f22603b7..005f7e341a684 100644
--- a/cmd/warm-backend-minio.go
+++ b/cmd/warm-backend-minio.go
@@ -78,7 +78,7 @@ func optimalPartSize(objectSize int64) (partSize int64, err error) {
 	return partSize, nil
 }
 
-func (m *warmBackendMinIO) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
+func (m *warmBackendMinIO) PutWithMeta(ctx context.Context, object string, r io.Reader, length int64, meta map[string]string) (remoteVersionID, error) {
 	partSize, err := optimalPartSize(length)
 	if err != nil {
 		return remoteVersionID(""), err
@@ -87,10 +87,15 @@ func (m *warmBackendMinIO) Put(ctx context.Context, object string, r io.Reader,
 		StorageClass:         m.StorageClass,
 		PartSize:             uint64(partSize),
 		DisableContentSha256: true,
+		UserMetadata:         meta,
 	})
 	return remoteVersionID(res.VersionID), m.ToObjectError(err, object)
 }
 
+func (m *warmBackendMinIO) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
+	return m.PutWithMeta(ctx, object, r, length, map[string]string{})
+}
+
 func newWarmBackendMinIO(conf madmin.TierMinIO, tier string) (*warmBackendMinIO, error) {
 	// Validation of credentials
 	if conf.AccessKey == "" || conf.SecretKey == "" {
diff --git a/cmd/warm-backend-s3.go b/cmd/warm-backend-s3.go
index de7e7d96d5a30..f46b88fb66689 100644
--- a/cmd/warm-backend-s3.go
+++ b/cmd/warm-backend-s3.go
@@ -56,14 +56,19 @@ func (s3 *warmBackendS3) getDest(object string) string {
 	return destObj
 }
 
-func (s3 *warmBackendS3) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
+func (s3 *warmBackendS3) PutWithMeta(ctx context.Context, object string, r io.Reader, length int64, meta map[string]string) (remoteVersionID, error) {
 	res, err := s3.client.PutObject(ctx, s3.Bucket, s3.getDest(object), r, length, minio.PutObjectOptions{
 		SendContentMd5: true,
 		StorageClass:   s3.StorageClass,
+		UserMetadata:   meta,
 	})
 	return remoteVersionID(res.VersionID), s3.ToObjectError(err, object)
 }
 
+func (s3 *warmBackendS3) Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error) {
+	return s3.PutWithMeta(ctx, object, r, length, map[string]string{})
+}
+
 func (s3 *warmBackendS3) Get(ctx context.Context, object string, rv remoteVersionID, opts WarmBackendGetOpts) (io.ReadCloser, error) {
 	gopts := minio.GetObjectOptions{}
 
diff --git a/cmd/warm-backend.go b/cmd/warm-backend.go
index 2389f3b1f7d36..91a936004d021 100644
--- a/cmd/warm-backend.go
+++ b/cmd/warm-backend.go
@@ -38,6 +38,7 @@ type WarmBackendGetOpts struct {
 // WarmBackend provides interface to be implemented by remote tier backends
 type WarmBackend interface {
 	Put(ctx context.Context, object string, r io.Reader, length int64) (remoteVersionID, error)
+	PutWithMeta(ctx context.Context, object string, r io.Reader, length int64, meta map[string]string) (remoteVersionID, error)
 	Get(ctx context.Context, object string, rv remoteVersionID, opts WarmBackendGetOpts) (io.ReadCloser, error)
 	Remove(ctx context.Context, object string, rv remoteVersionID) error
 	InUse(ctx context.Context) (bool, error)
diff --git a/docs/bucket/lifecycle/setup_ilm_transition.sh b/docs/bucket/lifecycle/setup_ilm_transition.sh
new file mode 100755
index 0000000000000..975ea81f853f9
--- /dev/null
+++ b/docs/bucket/lifecycle/setup_ilm_transition.sh
@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+
+set -x
+
+trap 'catch $LINENO' ERR
+
+# shellcheck disable=SC2120
+catch() {
+	if [ $# -ne 0 ]; then
+		echo "error on line $1"
+		for site in sitea siteb; do
+			echo "$site server logs ========="
+			cat "/tmp/${site}_1.log"
+			echo "==========================="
+			cat "/tmp/${site}_2.log"
+		done
+	fi
+
+	echo "Cleaning up instances of MinIO"
+	pkill minio
+	pkill -9 minio
+	rm -rf /tmp/multisitea
+	rm -rf /tmp/multisiteb
+	if [ $# -ne 0 ]; then
+		exit $#
+	fi
+}
+
+catch
+
+export MINIO_CI_CD=1
+export MINIO_BROWSER=off
+export MINIO_KMS_AUTO_ENCRYPTION=off
+export MINIO_PROMETHEUS_AUTH_TYPE=public
+export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
+unset MINIO_KMS_KES_CERT_FILE
+unset MINIO_KMS_KES_KEY_FILE
+unset MINIO_KMS_KES_ENDPOINT
+unset MINIO_KMS_KES_KEY_NAME
+
+if [ ! -f ./mc ]; then
+	wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
+		chmod +x mc
+fi
+
+minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
+	"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
+minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
+	"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
+
+minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
+	"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
+minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
+	"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
+
+# Wait to make sure all MinIO instances are up
+
+export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001
+export MC_HOST_siteb=http://minioadmin:minioadmin@127.0.0.1:9004
+
+./mc ready sitea
+./mc ready siteb
+
+./mc mb --ignore-existing sitea/bucket
+./mc mb --ignore-existing siteb/bucket
+
+sleep 10s
+
+## Add warm tier
+./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9004 --access-key minioadmin --secret-key minioadmin --bucket bucket
+
+## Add ILM rules
+./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER
+./mc ilm rule list sitea/bucket
+
+./mc cp README.md sitea/bucket/README.md
+
+until $(./mc stat sitea/bucket/README.md --json | jq -r '.metadata."X-Amz-Storage-Class"' | grep -q WARM-TIER); do
+	echo "waiting until the object is tiered to run heal"
+	sleep 1s
+done
+./mc stat sitea/bucket/README.md
+
+success=$(./mc admin heal -r sitea/bucket/README.md --json --force | jq -r 'select((.name == "bucket/README.md") and (.after.color == "green")) | .after.color == "green"')
+if [ "${success}" != "true" ]; then
+	echo "Found bug expected transitioned object to report 'green'"
+	exit 1
+fi
+
+catch


Test patch: