Repository: minio/minio (language: go)

Issue:
Two-Site Active-Active Replication. Deleting an object is not replicated while/if one site was down.
The situation is pretty similar to the #13352

Two clusters of minio with same configuration (cluster: 4 nodes with 1 disk per node)

## Expected Behavior
If the second cluster is down and delete operation is performed on the first cluster, then delete operation must be reproduced on the second cluster after it comes up.

## Current Behavior
Delete operation do not replicates to the second cluster (if it was down while performing delete operation).

## Possible Solution
Don't now.

## Steps to Reproduce (for bugs)
1. Deploy two similar clusters

```console
$ mcli admin info minio1
●  minio11.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  minio12.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  minio13.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  minio14.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

Pools:
   1st, Erasure sets: 1, Drives per erasure set: 4
```

```console
$ mcli admin info minio2
●  minio21.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  minio22.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  minio23.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

●  minio24.labs.lan:9000
   Uptime: 18 hours
   Version: 2023-09-30T07:02:29Z
   Network: 4/4 OK
   Drives: 1/1 OK
   Pool: 1

Pools:
   1st, Erasure sets: 1, Drives per erasure set: 4
```
2. Create bucket in the 1st cluster
```console
$ mcli mb minio1/test-delete
Bucket created successfully `minio1/test-delete`.
```
3. Configure Site Replication using the MinIO Console or Command Line (it's no matter)
4. Copy file to the bucket (1st cluster) and chek file's metadata in both clusters
```console
$ mcli cp --quiet ./test.txt minio1/test-delete/
`/tmp/test.txt` -> `minio1/test-delete/test.txt`
Total: 0 B, Transferred: 2 B, Speed: 65 B/s

$ mcli stat minio1/test-delete/test.txt
Name      : test.txt
Date      : 2023-10-04 12:43:34 EEST
Size      : 2 B
ETag      : b026324c6904b2a9cb4b88d6d61c81d1
VersionID : 1d7351b7-acf3-40e3-bf77-d5144aafd3a8
Type      : file
Metadata  :
  Content-Type: text/plain
Replication Status: COMPLETED

$ mcli stat minio2/test-delete/test.txt
Name      : test.txt
Date      : 2023-10-04 12:43:34 EEST
Size      : 2 B
ETag      : b026324c6904b2a9cb4b88d6d61c81d1
VersionID : 1d7351b7-acf3-40e3-bf77-d5144aafd3a8
Type      : file
Metadata  :
  Content-Type: text/plain
Replication Status: REPLICA
```
5. Add ILM rule for this bucket (1st cluster)
```console
$ mcli ilm rule add --noncurrent-expire-days 1 --expire-delete-marker minio1/test-delete/
Lifecycle configuration rule added with ID `ckej8n6t01nripk0eemg` to minio1/test-delete/.
```
6. Completely shutdown 2nd cluster
7. Delete test file (1st cluster)
```console
$ mcli rm --quiet --force --versions minio1/test-delete/test.txt
Removed `minio1/test-delete/test.txt` (versionId=1d7351b7-acf3-40e3-bf77-d5144aafd3a8).
```
8. Start 2nd cluster
9. Check if the file exists (1st cluster)
```console
$ mcli stat minio1/test-delete/test.txt
<EMPTY OUTPUT>
```
10. Check if the file exists (2nd cluster)
```console
$ mcli stat minio2/test-delete/test.txt
Name      : test.txt
Date      : 2023-10-04 12:43:34 EEST
Size      : 2 B
ETag      : b026324c6904b2a9cb4b88d6d61c81d1
VersionID : 1d7351b7-acf3-40e3-bf77-d5144aafd3a8
Type      : file
Metadata  :
  Content-Type: text/plain
Replication Status: REPLICA
```
Deleting of the file will not be replicated to the second cluster when it will start.
(I have waited more than 2 hours)

**If don't add ILM rule (step 5), then delete operation is replicates as expected.**

## Context
Having two or more clusters with an Active-Active replication meaning that all data operations must be replicated to each other. Even if one of the clusters was unreacheble (network interrupts or other problems) It has to replicate past operations of the "neighbours" clusters.

## Regression
Don't know.

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`):
`minio version RELEASE.2023-09-30T07-02-29Z (commit-id=22d2dbc4e68a20ada8c82edf1cc68026e7b0e3d6)`

* Operating System and version (`uname -a`):
`5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux`



Discussion / hints:
Deletes when occurred when the site is down is taken over by a scanner to heal the object, after when the target comes back online.

It may take 2hrs or even more - it is not instantaneous. It simply depends on if scanner picked up this object for heal on replication. 

ILM rules must be applied to both clusters in replication. ILM rules take precedence over replication if set, so to make sure both sides apply the same rule you must independently apply same ILM rules on both sites.
> It may take 2hrs or even more - it is not instantaneous.

> ILM rules must be applied to both clusters in replication.

Ok, I will repeat the test (also with ILM ruls applied to both clusters) then will wait several days and will provide you a feedback.

Thanks.
### Repeated steps of the test

1. Upload test file
```console
$ mcli ls --versions minio1/test-delete/
<EMPTY OUTPUT>

$ mcli ls --versions minio2/test-delete/
<EMPTY OUTPUT>

$ mcli cp --quiet ./test.txt minio1/test-delete/
`/tmp/test.txt` -> `minio1/test-delete/test.txt`
Total: 0 B, Transferred: 2 B, Speed: 105 B/s

$ mcli stat --versions minio1/test-delete/
Name      : test.txt
Date      : 2023-10-04 17:28:16 EEST
Size      : 2 B
ETag      : b026324c6904b2a9cb4b88d6d61c81d1
VersionID : c6c7cff6-ca20-423b-b12c-079b472de9ca
Type      : file
Metadata  :
  Content-Type: text/plain
Replication Status: COMPLETED

$ mcli stat --versions minio2/test-delete/
Name      : test.txt
Date      : 2023-10-04 17:28:16 EEST
Size      : 2 B
ETag      : b026324c6904b2a9cb4b88d6d61c81d1
VersionID : c6c7cff6-ca20-423b-b12c-079b472de9ca
Type      : file
Metadata  :
  Content-Type: text/plain
Replication Status: REPLICA
```
3. Add ILM rules for non-current versions
```console
$ mcli ilm rule list minio1/test-delete/
mcli: <ERROR> Unable to get lifecycle. The lifecycle configuration does not exist.

$ mcli ilm rule list minio2/test-delete/
mcli: <ERROR> Unable to get lifecycle. The lifecycle configuration does not exist.

$ mcli ilm rule add --noncurrent-expire-days 1 --expire-delete-marker minio1/test-delete/
Lifecycle configuration rule added with ID `ckenggut01nu23lmpfbg` to minio1/test-delete/.

$ mcli ilm rule add --noncurrent-expire-days 1 --expire-delete-marker minio2/test-delete/
Lifecycle configuration rule added with ID `ckengi6t01nu240ft0j0` to minio2/test-delete/.
```
4. Stop second cluster
5. Delete test file (1st cluster)
```console
$ mcli rm --quiet --force --versions minio1/test-delete/test.txt
Removed `minio1/test-delete/test.txt` (versionId=c6c7cff6-ca20-423b-b12c-079b472de9ca).
```
6. Start 2nd cluster
7. Check if the file exists (1st cluster)
```console
$ mcli stat minio1/test-delete/test.txt
<EMPTY OUTPUT>
```
8. Check if the file exists (2nd cluster)
```console
$ mcli stat minio2/test-delete/test.txt
Name      : test.txt
Date      : 2023-10-04 17:28:16 EEST
Size      : 2 B
ETag      : b026324c6904b2a9cb4b88d6d61c81d1
VersionID : c6c7cff6-ca20-423b-b12c-079b472de9ca
Type      : file
Metadata  :
  Content-Type: text/plain
Replication Status: REPLICA
```

Let's wait a few days…
After 24h file is still there…

```console
$ date
Thu Oct  5 17:35:37 EEST 2023

$ mcli stat --versions minio2/test-delete/test.txt
Name      : test.txt
Date      : 2023-10-04 17:28:16 EEST
Size      : 2 B
ETag      : b026324c6904b2a9cb4b88d6d61c81d1
VersionID : c6c7cff6-ca20-423b-b12c-079b472de9ca
Type      : file
Metadata  :
  Content-Type: text/plain
Replication Status: REPLICA
```
It won't be there in 48hrs, 24hrs is not sufficient to expire in ILM.

Fix patch:
diff --git a/cmd/data-scanner.go b/cmd/data-scanner.go
index a72b2cc89aed7..2ed4b72c3bdfa 100644
--- a/cmd/data-scanner.go
+++ b/cmd/data-scanner.go
@@ -929,7 +929,8 @@ func (i *scannerItem) applyLifecycle(ctx context.Context, o ObjectLayer, oi Obje
 
 	versionID := oi.VersionID
 	rCfg, _ := globalBucketObjectLockSys.Get(i.bucket)
-	lcEvt := evalActionFromLifecycle(ctx, *i.lifeCycle, rCfg, oi)
+	replcfg, _ := getReplicationConfig(ctx, i.bucket)
+	lcEvt := evalActionFromLifecycle(ctx, *i.lifeCycle, rCfg, replcfg, oi)
 	if i.debug {
 		if versionID != "" {
 			console.Debugf(applyActionsLogPrefix+" lifecycle: %q (version-id=%s), Initial scan: %v\n", i.objectPath(), versionID, lcEvt.Action)
@@ -1129,7 +1130,7 @@ func (i *scannerItem) applyActions(ctx context.Context, o ObjectLayer, oi Object
 	return size
 }
 
-func evalActionFromLifecycle(ctx context.Context, lc lifecycle.Lifecycle, lr lock.Retention, obj ObjectInfo) lifecycle.Event {
+func evalActionFromLifecycle(ctx context.Context, lc lifecycle.Lifecycle, lr lock.Retention, rcfg *replication.Config, obj ObjectInfo) lifecycle.Event {
 	event := lc.Eval(obj.ToLifecycleOpts())
 	if serverDebugLog {
 		console.Debugf(applyActionsLogPrefix+" lifecycle: Secondary scan: %v\n", event.Action)
@@ -1161,6 +1162,9 @@ func evalActionFromLifecycle(ctx context.Context, lc lifecycle.Lifecycle, lr loc
 			}
 			return lifecycle.Event{Action: lifecycle.NoneAction}
 		}
+		if rcfg != nil && !obj.VersionPurgeStatus.Empty() && rcfg.HasActiveRules(obj.Name, true) {
+			return lifecycle.Event{Action: lifecycle.NoneAction}
+		}
 	}
 
 	return event
diff --git a/cmd/erasure-object.go b/cmd/erasure-object.go
index 2bac709182bbd..6c22a42e297bf 100644
--- a/cmd/erasure-object.go
+++ b/cmd/erasure-object.go
@@ -1628,10 +1628,12 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
 
 	var lc *lifecycle.Lifecycle
 	var rcfg lock.Retention
+	var replcfg *replication.Config
 	if opts.Expiration.Expire {
 		// Check if the current bucket has a configured lifecycle policy
 		lc, _ = globalLifecycleSys.Get(bucket)
 		rcfg, _ = globalBucketObjectLockSys.Get(bucket)
+		replcfg, _ = getReplicationConfig(ctx, bucket)
 	}
 
 	// expiration attempted on a bucket with no lifecycle
@@ -1684,7 +1686,7 @@ func (er erasureObjects) DeleteObject(ctx context.Context, bucket, object string
 
 	if opts.Expiration.Expire {
 		if gerr == nil {
-			evt := evalActionFromLifecycle(ctx, *lc, rcfg, goi)
+			evt := evalActionFromLifecycle(ctx, *lc, rcfg, replcfg, goi)
 			var isErr bool
 			switch evt.Action {
 			case lifecycle.NoneAction:
diff --git a/cmd/erasure-server-pool-decom.go b/cmd/erasure-server-pool-decom.go
index 73a3c40fc8832..384fd9ebbc816 100644
--- a/cmd/erasure-server-pool-decom.go
+++ b/cmd/erasure-server-pool-decom.go
@@ -734,6 +734,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 
 	// Check if bucket is object locked.
 	lr, _ := globalBucketObjectLockSys.Get(bi.Name)
+	rcfg, _ := getReplicationConfig(ctx, bi.Name)
 
 	for setIdx, set := range pool.sets {
 		set := set
@@ -745,7 +746,7 @@ func (z *erasureServerPools) decommissionPool(ctx context.Context, idx int, pool
 			versioned := vc != nil && vc.Versioned(object)
 			objInfo := fi.ToObjectInfo(bucket, object, versioned)
 
-			evt := evalActionFromLifecycle(ctx, *lc, lr, objInfo)
+			evt := evalActionFromLifecycle(ctx, *lc, lr, rcfg, objInfo)
 			switch {
 			case evt.Action.DeleteRestored(): // if restored copy has expired,delete it synchronously
 				applyExpiryOnTransitionedObject(ctx, z, objInfo, evt, lcEventSrc_Decom)
@@ -1048,6 +1049,7 @@ func (z *erasureServerPools) checkAfterDecom(ctx context.Context, idx int) error
 
 			// Check if bucket is object locked.
 			lr, _ := globalBucketObjectLockSys.Get(bi.Name)
+			rcfg, _ := getReplicationConfig(ctx, bi.Name)
 
 			filterLifecycle := func(bucket, object string, fi FileInfo) bool {
 				if lc == nil {
@@ -1056,7 +1058,7 @@ func (z *erasureServerPools) checkAfterDecom(ctx context.Context, idx int) error
 				versioned := vc != nil && vc.Versioned(object)
 				objInfo := fi.ToObjectInfo(bucket, object, versioned)
 
-				evt := evalActionFromLifecycle(ctx, *lc, lr, objInfo)
+				evt := evalActionFromLifecycle(ctx, *lc, lr, rcfg, objInfo)
 				switch {
 				case evt.Action.DeleteRestored(): // if restored copy has expired,delete it synchronously
 					applyExpiryOnTransitionedObject(ctx, z, objInfo, evt, lcEventSrc_Decom)
diff --git a/cmd/erasure-server-pool-rebalance.go b/cmd/erasure-server-pool-rebalance.go
index 58e7c8676f761..783b49976cd89 100644
--- a/cmd/erasure-server-pool-rebalance.go
+++ b/cmd/erasure-server-pool-rebalance.go
@@ -440,6 +440,7 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
 	lc, _ := globalLifecycleSys.Get(bucket)
 	// Check if bucket is object locked.
 	lr, _ := globalBucketObjectLockSys.Get(bucket)
+	rcfg, _ := getReplicationConfig(ctx, bucket)
 
 	pool := z.serverPools[poolIdx]
 	const envRebalanceWorkers = "_MINIO_REBALANCE_WORKERS"
@@ -467,7 +468,7 @@ func (z *erasureServerPools) rebalanceBucket(ctx context.Context, bucket string,
 			versioned := vc != nil && vc.Versioned(object)
 			objInfo := fi.ToObjectInfo(bucket, object, versioned)
 
-			evt := evalActionFromLifecycle(ctx, *lc, lr, objInfo)
+			evt := evalActionFromLifecycle(ctx, *lc, lr, rcfg, objInfo)
 			if evt.Action.Delete() {
 				globalExpiryState.enqueueByDays(objInfo, evt, lcEventSrc_Rebal)
 				return true
diff --git a/cmd/erasure-server-pool.go b/cmd/erasure-server-pool.go
index e70b95c7c29fb..2305e1bb20b36 100644
--- a/cmd/erasure-server-pool.go
+++ b/cmd/erasure-server-pool.go
@@ -1390,7 +1390,7 @@ func (z *erasureServerPools) ListObjects(ctx context.Context, bucket, prefix, ma
 		objInfo, err := z.GetObjectInfo(ctx, bucket, path.Dir(prefix), ObjectOptions{NoLock: true})
 		if err == nil {
 			if opts.Lifecycle != nil {
-				evt := evalActionFromLifecycle(ctx, *opts.Lifecycle, opts.Retention, objInfo)
+				evt := evalActionFromLifecycle(ctx, *opts.Lifecycle, opts.Retention, opts.Replication.Config, objInfo)
 				if evt.Action.Delete() {
 					globalExpiryState.enqueueByDays(objInfo, evt, lcEventSrc_s3ListObjects)
 					if !evt.Action.DeleteRestored() {
@@ -1414,7 +1414,7 @@ func (z *erasureServerPools) ListObjects(ctx context.Context, bucket, prefix, ma
 		objInfo, err := z.GetObjectInfo(ctx, bucket, prefix, ObjectOptions{NoLock: true})
 		if err == nil {
 			if opts.Lifecycle != nil {
-				evt := evalActionFromLifecycle(ctx, *opts.Lifecycle, opts.Retention, objInfo)
+				evt := evalActionFromLifecycle(ctx, *opts.Lifecycle, opts.Retention, opts.Replication.Config, objInfo)
 				if evt.Action.Delete() {
 					globalExpiryState.enqueueByDays(objInfo, evt, lcEventSrc_s3ListObjects)
 					if !evt.Action.DeleteRestored() {
diff --git a/cmd/metacache-server-pool.go b/cmd/metacache-server-pool.go
index f454393b8ec72..80af7b2a6e76e 100644
--- a/cmd/metacache-server-pool.go
+++ b/cmd/metacache-server-pool.go
@@ -370,7 +370,7 @@ func applyBucketActions(ctx context.Context, o listPathOptions, in <-chan metaCa
 
 			objInfo := fi.ToObjectInfo(o.Bucket, obj.name, versioned)
 			if o.Lifecycle != nil {
-				act := evalActionFromLifecycle(ctx, *o.Lifecycle, o.Retention, objInfo).Action
+				act := evalActionFromLifecycle(ctx, *o.Lifecycle, o.Retention, o.Replication.Config, objInfo).Action
 				skip = act.Delete()
 				if act.DeleteRestored() {
 					// do not skip DeleteRestored* actions
@@ -398,7 +398,7 @@ func applyBucketActions(ctx context.Context, o listPathOptions, in <-chan metaCa
 			objInfo := version.ToObjectInfo(o.Bucket, obj.name, versioned)
 
 			if o.Lifecycle != nil {
-				evt := evalActionFromLifecycle(ctx, *o.Lifecycle, o.Retention, objInfo)
+				evt := evalActionFromLifecycle(ctx, *o.Lifecycle, o.Retention, o.Replication.Config, objInfo)
 				if evt.Action.Delete() {
 					globalExpiryState.enqueueByDays(objInfo, evt, lcEventSrc_s3ListObjects)
 					if !evt.Action.DeleteRestored() {
diff --git a/cmd/object-handlers.go b/cmd/object-handlers.go
index 8c0ef24334b25..8d3e9574dc374 100644
--- a/cmd/object-handlers.go
+++ b/cmd/object-handlers.go
@@ -474,7 +474,8 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
 		// Automatically remove the object/version if an expiry lifecycle rule can be applied
 		if lc, err := globalLifecycleSys.Get(bucket); err == nil {
 			rcfg, _ := globalBucketObjectLockSys.Get(bucket)
-			event := evalActionFromLifecycle(ctx, *lc, rcfg, objInfo)
+			replcfg, _ := getReplicationConfig(ctx, bucket)
+			event := evalActionFromLifecycle(ctx, *lc, rcfg, replcfg, objInfo)
 			if event.Action.Delete() {
 				// apply whatever the expiry rule is.
 				applyExpiryRule(event, lcEventSrc_s3GetObject, objInfo)
@@ -732,7 +733,8 @@ func (api objectAPIHandlers) headObjectHandler(ctx context.Context, objectAPI Ob
 		// Automatically remove the object/version if an expiry lifecycle rule can be applied
 		if lc, err := globalLifecycleSys.Get(bucket); err == nil {
 			rcfg, _ := globalBucketObjectLockSys.Get(bucket)
-			event := evalActionFromLifecycle(ctx, *lc, rcfg, objInfo)
+			replcfg, _ := getReplicationConfig(ctx, bucket)
+			event := evalActionFromLifecycle(ctx, *lc, rcfg, replcfg, objInfo)
 			if event.Action.Delete() {
 				// apply whatever the expiry rule is.
 				applyExpiryRule(event, lcEventSrc_s3HeadObject, objInfo)


Test patch: