Repository: FoundationDB/fdb-kubernetes-operator
Pull request #1581 (2023-04-10 23:20:40)

Add taint feature to auto replace tainted node
# Description

Fixes: https://github.com/FoundationDB/fdb-kubernetes-operator/issues/507

When a node is tainted by a key, operator will mark pods on the node as NodeTaintDetected.  If a pods stay in  NodeTaintDetected for long enough, operator will mark the pod as unhealthy and replace it with replacement logic.

SRE can configure the tainted keys operator should react to. Each tainted key has a tainted duration which defines how long the pod should stay in tainted mode before it is marked for replacement.

Change sets include
 * Set logger to output to stdout for replacements_test;
 * Add k8s node concept to unit test framework so that we can test node related features (e.g., taint a node, fail a node);
 * Update node and cluster config to propogate its effect to all reconcilation loop;
 * Update NodeTaintDetected condition time with node's tainted time; so that a pod is removed "closely" after configured duration after its tainted time;
 * Add taint tests in update_status_test.
   This test focuses on the testing the taint logic isolately;
 * Add taint test in replace_failed_process_groups_test.
   This test focuses on testing the interaction between taint function and other reconciliation loops
 * Add end-to-end test to taint a node in k8s cluster and verify the feature replaces pods on tainted node as expected.

## Type of change
- New feature (non-breaking change which adds functionality)

## Testing

Unit tests and end-to-end test on EKS. 

*Do we need to perform additional testing once this is merged, or perform in a larger testing environment?*
No.

## Documentation

TODO: Update user manual and design doc. 

*Did you update relevant documentation within this repository?*

*If this change is adding new functionality, do we need to describe it in our user manual?*

*If this change is adding or removing subreconcilers, have we updated the core technical design doc to reflect that?*

*If this change is adding new safety checks or new potential failure modes, have we documented and how to debug potential issues?*

## Follow-up
Future work:
- [ ] Consider to add a separate replacementTimeDuration to replace tainted node faster. Right now, cluster only replaces a failed process after 2 hours. The reaction time is too long for some taint key (e.g., maintenance or network disconnection).
- [ ]  Consider to reduce the detection latency of a tainted node. Right now, the detection latency is the latency of operator's reconciliation invocation plus operator's reconciliation's execution time. Since reconciliation may not be invoked for a long time (hour), operator may not detect a 10min taint quickly enough.


foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: d8c466ec173b74abc49e0aa5b0a2cf82aa7efa4e
* Duration 4:11:13
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/cf6d8605-6c92-4272-8cc7-fd1c9921df71.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/cf6d8605-6c92-4272-8cc7-fd1c9921df71/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 9f4d7fc4be8f898a8ec9c8bb39eb6f1e01d4eb7e
* Duration 4:11:31
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/44425a47-fcd2-4b27-8203-110d5aaded4c.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/44425a47-fcd2-4b27-8203-110d5aaded4c/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 0f05ff78803c1b5a3a5afcb149e7c01d456ba783
* Duration 4:11:30
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/4ae1dd1d-754f-4047-a0c8-1893868f13a1.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/4ae1dd1d-754f-4047-a0c8-1893868f13a1/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: d9d7e02563d16b404e289ae93f5c48489062d367
* Duration 4:11:20
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/37ad537f-236e-4f4c-80e2-8d92d15d6005.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/37ad537f-236e-4f4c-80e2-8d92d15d6005/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 7a30f2f55c164389a753c0db7ebc63ee5488158c
* Duration 4:11:26
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/a17631db-ee75-45c6-9cfc-9000ef481666.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/a17631db-ee75-45c6-9cfc-9000ef481666/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 9b190f87d510d7a32404eb4d74f18cf00a02bcf0
* Duration 4:11:21
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/76b1739d-aea4-4b5a-bdd5-cb89b3da45a1.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/76b1739d-aea4-4b5a-bdd5-cb89b3da45a1/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: a3affc1ef8b6923d9c9911c664c2171e673c8ae8
* Duration 4:11:10
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/6596c917-6161-45bf-96b5-bb7b05e210a5.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/6596c917-6161-45bf-96b5-bb7b05e210a5/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: b20161ce2fccb1e16f648a8cab833ff7d25d9e05
* Duration 4:11:16
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/ab7305b5-bf56-47b6-b8b9-16b0a68e33a1.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/ab7305b5-bf56-47b6-b8b9-16b0a68e33a1/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 30011c9f03b650e8cc50b542aa6e415cb03d3f79
* Duration 4:11:02
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/d31ff016-31ed-4b6c-89ad-356ddc178f05.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/d31ff016-31ed-4b6c-89ad-356ddc178f05/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 9bee1450858503d2bbdc827243db06550801e7fd
* Duration 0:05:34
* Result: :x: **FAILED**
* Error: `reference not found for primary source and source version 9bee1450858503d2bbdc827243db06550801e7fd`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/681d29a0-a9a4-4436-8c97-40a3a2f4182c.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: ca5f1d7d180fa7354a8a407d90a959e58ef8f003
* Duration 4:11:15
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/b0e30f43-dad6-43cd-a1a0-791f923344d7.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/b0e30f43-dad6-43cd-a1a0-791f923344d7/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: dabc93df0d14d0021dfbfb26443862075e1ddc5d
* Duration 4:11:15
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/1b511324-2b10-4052-a496-c41042016c2f.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/1b511324-2b10-4052-a496-c41042016c2f/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 4b7c8491d2f6adc1bc437569fbe512ad66f42083
* Duration 4:10:59
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/48e3e106-9e92-4bd1-b811-aab530cbf9cf.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/48e3e106-9e92-4bd1-b811-aab530cbf9cf/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: fca2015effd060842234ec0ec150d730309da426
* Duration 0:05:40
* Result: :x: **FAILED**
* Error: `reference not found for primary source and source version fca2015effd060842234ec0ec150d730309da426`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/75018558-70f6-44c2-ad5c-4657fe2c9d2e.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 1edf305613a67d1f20054a0dc5a873b2d9621fa6
* Duration 3:14:40
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/0bf622a6-7727-439e-be1c-af577a640234.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/0bf622a6-7727-439e-be1c-af577a640234/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 1c0bf4b3bfdcf065dde07eb192c6c67d7a9c03af
* Duration 2:29:51
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/58fe0697-6872-4f97-9dcc-ae09dbf2b30b.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/58fe0697-6872-4f97-9dcc-ae09dbf2b30b/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 3991f1a2b623ab3445e8e48af48290e33136988a
* Duration 3:24:05
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/3c21151f-c8dd-46ab-b3bd-ed1487a561d4.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/3c21151f-c8dd-46ab-b3bd-ed1487a561d4/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 75fdfc44c8f27b7cfc23daac07a3518b72e457cd
* Duration 2:40:02
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/36afc35f-494e-4883-bc59-43dd3d70575d.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/36afc35f-494e-4883-bc59-43dd3d70575d/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: b1a0598a24923a43a0960f78398b33cabe3b1e87
* Duration 2:34:24
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/0d6ba609-ce88-449c-afa1-9ec06ea33396.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/0d6ba609-ce88-449c-afa1-9ec06ea33396/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 18ed052f4219572a71083bf4f908f2959972c42d
* Duration 2:45:47
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/ab464277-9b00-4b8b-a21f-84ba60c98799.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/ab464277-9b00-4b8b-a21f-84ba60c98799/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 870cfeab5da97cf56f35ebee035ebaf3284bb304
* Duration 3:16:37
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/104cc2db-9a29-4e54-95ee-bb75f19aee68.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/104cc2db-9a29-4e54-95ee-bb75f19aee68/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: ff0a746d4757c8a25e278186534f956a89409c38
* Duration 1:57:41
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/d4cb4094-128e-4260-8ce9-cce0d7a44774.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/d4cb4094-128e-4260-8ce9-cce0d7a44774/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 7a49837f65a4b2c02957a19e73691ebb5191ef13
* Duration 2:31:46
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/4079eca9-6ed0-4a43-9b0f-594008f5cee8.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/4079eca9-6ed0-4a43-9b0f-594008f5cee8/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: ca61fee9323dccd9f0c75106d02224978e989b7e
* Duration 3:49:36
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/9e1b2626-2cd8-44ec-ac6a-538546dd484a.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/9e1b2626-2cd8-44ec-ac6a-538546dd484a/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 0c51732ae400ef9cca0b0f76dddfea248a3f5295
* Duration 2:32:10
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/74f15dcf-5572-486a-bb5c-b3035e8d0dc1.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/74f15dcf-5572-486a-bb5c-b3035e8d0dc1/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: c79bf51c2e0e6e740cd30a515a0a74a19950d748
* Duration 4:09:55
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/0b89338a-0e64-453f-be36-705bcb66bb3c.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/0b89338a-0e64-453f-be36-705bcb66bb3c/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 090905aa2f5ae79f1be461ff6963666526204393
* Duration 4:09:41
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/bf8157a8-1b07-4121-a3bf-8c5bb399a021.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/bf8157a8-1b07-4121-a3bf-8c5bb399a021/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 5096fcc31e72d0ab5b01bc1c5d2866c1675163dc
* Duration 2:07:40
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/52d72af8-05a9-416b-9ffa-667cfe280ff8.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/52d72af8-05a9-416b-9ffa-667cfe280ff8/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 05f282b84b022ec18d866ba30b1e5816a345e697
* Duration 2:33:56
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/2a415675-5d08-4c95-908a-086d4d17983b.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/2a415675-5d08-4c95-908a-086d4d17983b/workspace.zip) zip file of the working directory (available for 30 days)

xumengpanda: e2e test result by focusing on the 5 tainted tests
```


SSSSSSS
P [PENDING]
Operator when Changing the TLS setting should disable or enable TLS and keep the cluster available [e2e]
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:579
------------------------------
SSSSSSSSSSSSSS
P [PENDING]
Operator when setting the empty config to true should stop all running processes [e2e]
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:1003
------------------------------
S
P [PENDING]
Operator when a process group is set to be blocked for removal should exclude the Pod but not remove the resources [e2e]
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:1101
------------------------------
[AfterSuite] 
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:102
2023/05/04 03:27:54 start cleaning up chaos mesh client with 1 experiment(s)
2023/05/04 03:27:54 Start deleting ptga5tvlhf65pvgt8o9rk3v1t51epwuw
2023/05/04 03:27:54 Chaos ptga5tvlhf65pvgt8o9rk3v1t51epwuw is deleted.
2023/05/04 03:27:54 finished all tests, start deleting namespace mengxu-h3m83pta
[AfterSuite] PASSED [15.045 seconds]
------------------------------
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
[ReportAfterSuite] PASSED [0.002 seconds]
------------------------------

Ran 5 of 30 Specs in 1172.955 seconds
SUCCESS! -- 5 Passed | 0 Failed | 3 Pending | 22 Skipped
PASS | FOCUSED
FAIL	github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator	1172.985s
FAIL
make: *** [Makefile:69: test_operator.run] Error 1

```
foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: f4cb7a62d96bcf4fa8b19ae742456b0b8fe0be05
* Duration 2:31:04
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/e92638ca-4885-434f-8e13-205230fb2302.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/e92638ca-4885-434f-8e13-205230fb2302/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 6cd82ded8a61bf92e67d6ffdad6f888d5a3e193e
* Duration 2:32:32
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/617efc94-da8f-46aa-aa36-523bd8261b83.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/617efc94-da8f-46aa-aa36-523bd8261b83/workspace.zip) zip file of the working directory (available for 30 days)

xumengpanda: ```
2023/05/05 22:03:56 reconciled name=fdb-cluster-57vzpdb4, namespace=mengxu-1drl2po1
• [6.331 seconds]
------------------------------
SSSSSSS
P [PENDING]
Operator when Changing the TLS setting should disable or enable TLS and keep the cluster available [e2e]
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:567
------------------------------
SSSSSSSSSSSSSS
P [PENDING]
Operator when setting the empty config to true should stop all running processes [e2e]
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:991
------------------------------
S
P [PENDING]
Operator when a process group is set to be blocked for removal should exclude the Pod but not remove the resources [e2e]
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:1089
------------------------------
[AfterSuite] 
/root/src/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:102
2023/05/05 22:04:12 start cleaning up chaos mesh client with 1 experiment(s)
2023/05/05 22:04:12 Start deleting skifoch4j44491j15l93xatpgqev4oc9
2023/05/05 22:04:12 Chaos skifoch4j44491j15l93xatpgqev4oc9 is deleted.
2023/05/05 22:04:12 finished all tests, start deleting namespace mengxu-1drl2po1
[AfterSuite] PASSED [15.045 seconds]
------------------------------
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
[ReportAfterSuite] PASSED [0.002 seconds]
------------------------------

Ran 5 of 30 Specs in 1379.051 seconds
SUCCESS! -- 5 Passed | 0 Failed | 3 Pending | 22 Skipped
PASS | FOCUSED
FAIL	github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator	1379.082s
```
foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: b51b797fdddc299d5f2c1ebc92cd6ff6d633aaec
* Duration 2:45:29
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/979ba2ca-96f8-4c91-84b4-c73566d4fed0.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/979ba2ca-96f8-4c91-84b4-c73566d4fed0/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 9f4ddc3fed1b6f4fd1c086d8c701b20d1fbd7781
* Duration 2:45:13
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/610eccbc-7c8d-41e9-b606-fcb40c6d6464.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/610eccbc-7c8d-41e9-b606-fcb40c6d6464/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 5032bc65eb7434bd2dc04de78f98da192ba579bb
* Duration 2:58:01
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/ce9e4fb3-d9e0-48e2-900e-1fed3967211b.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/ce9e4fb3-d9e0-48e2-900e-1fed3967211b/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 0ec06c90b5b662ce5260373dceb764fb97647030
* Duration 2:55:14
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/0870d3fe-f360-4244-a280-1ec1b9713746.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/0870d3fe-f360-4244-a280-1ec1b9713746/workspace.zip) zip file of the working directory (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 1cec691a2471a95942c09ee5287b0308464de830
* Duration 2:25:00
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/7c3d82fc-2757-463c-95c0-8a16ecbd64b1.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/7c3d82fc-2757-463c-95c0-8a16ecbd64b1/workspace.zip) zip file of the working directory (available for 30 days)


Patch:
diff --git a/Makefile b/Makefile
index 41e1f86f..5535ed95 100644
--- a/Makefile
+++ b/Makefile
@@ -152,11 +152,6 @@ manifests: ${MANIFESTS}
 
 ${MANIFESTS}: ${CONTROLLER_GEN} ${GO_SRC}
 	$(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
-	# Per default controller-gen will generate a ClusterRole for our example we want to use a Role and the namespace marker doesn't
-	# work since it requires a namespace and kustomize doesn't support to change the Kind.
-	@mv ./config/rbac/role.yaml ./config/rbac/role.yaml.tmp
-	@sed 's/kind: ClusterRole/kind: Role/g' ./config/rbac/role.yaml.tmp > ./config/rbac/role.yaml
-	@rm ./config/rbac/role.yaml.tmp
 
 # Run go fmt against code
 fmt: bin/fmt_check
diff --git a/api/v1beta2/foundationdbcluster_types.go b/api/v1beta2/foundationdbcluster_types.go
index da0561b3..f046def0 100644
--- a/api/v1beta2/foundationdbcluster_types.go
+++ b/api/v1beta2/foundationdbcluster_types.go
@@ -62,7 +62,8 @@ type FoundationDBClusterList struct {
 	Items           []FoundationDBCluster `json:"items"`
 }
 
-var conditionsThatNeedReplacement = []ProcessGroupConditionType{MissingProcesses, PodFailing, MissingPod, MissingPVC, MissingService, PodPending}
+var conditionsThatNeedReplacement = []ProcessGroupConditionType{MissingProcesses, PodFailing, MissingPod, MissingPVC,
+	MissingService, PodPending, NodeTaintReplacing}
 
 func init() {
 	SchemeBuilder.Register(&FoundationDBCluster{}, &FoundationDBClusterList{})
@@ -361,18 +362,39 @@ func (processGroupStatus *ProcessGroupStatus) MarkForRemoval() {
 }
 
 // NeedsReplacement checks if the ProcessGroupStatus has conditions so that it should be removed
-func (processGroupStatus *ProcessGroupStatus) NeedsReplacement(failureTime int) (bool, int64) {
-	var missingTime *int64
-	for _, condition := range conditionsThatNeedReplacement {
-		conditionTime := processGroupStatus.GetConditionTime(condition)
-		if conditionTime != nil && (missingTime == nil || *missingTime > *conditionTime) {
-			missingTime = conditionTime
+func (processGroupStatus *ProcessGroupStatus) NeedsReplacement(failureTime int, taintReplacementTime int) (bool, int64) {
+	var earliestFailureTime int64 = math.MaxInt64
+	var earliestTaintReplacementTime int64 = math.MaxInt64
+
+	if processGroupStatus.IsMarkedForRemoval() {
+		return false, 0
+	}
+
+	for _, conditionType := range conditionsThatNeedReplacement {
+		conditionTimePtr := processGroupStatus.GetConditionTime(conditionType)
+		if conditionTimePtr == nil {
+			continue
+		}
+
+		conditionTime := *conditionTimePtr
+		if conditionType == NodeTaintReplacing {
+			if earliestTaintReplacementTime > conditionTime {
+				earliestTaintReplacementTime = conditionTime
+			}
+		} else {
+			if earliestFailureTime > conditionTime {
+				earliestFailureTime = conditionTime
+			}
 		}
 	}
 
 	failureWindowStart := time.Now().Add(-1 * time.Duration(failureTime) * time.Second).Unix()
-	if missingTime != nil && *missingTime < failureWindowStart && !processGroupStatus.IsMarkedForRemoval() {
-		return true, *missingTime
+	if earliestFailureTime < failureWindowStart {
+		return true, earliestFailureTime
+	}
+	taintWindowStart := time.Now().Add(-1 * time.Duration(taintReplacementTime) * time.Second).Unix()
+	if earliestTaintReplacementTime < taintWindowStart {
+		return true, earliestTaintReplacementTime
 	}
 
 	return false, 0
@@ -519,6 +541,17 @@ func (processGroupStatus *ProcessGroupStatus) UpdateCondition(conditionType Proc
 	}
 }
 
+// UpdateConditionTime will update the conditionType's condition time to newTime
+// If the conditionType does not exist, the function is no-op
+func (processGroupStatus *ProcessGroupStatus) UpdateConditionTime(conditionType ProcessGroupConditionType, newTime int64) {
+	for i, condition := range processGroupStatus.ProcessGroupConditions {
+		if condition.ProcessGroupConditionType == conditionType {
+			processGroupStatus.ProcessGroupConditions[i].Timestamp = newTime
+			break
+		}
+	}
+}
+
 // addCondition will add the condition to the ProcessGroupStatus.
 // If the old ProcessGroupStatus already contains the condition the condition is reused to contain the same timestamp.
 func (processGroupStatus *ProcessGroupStatus) addCondition(oldProcessGroups []*ProcessGroupStatus, processGroupID ProcessGroupID, conditionType ProcessGroupConditionType) {
@@ -534,24 +567,26 @@ func (processGroupStatus *ProcessGroupStatus) addCondition(oldProcessGroups []*P
 		break
 	}
 
+	// Check if we already got this condition in the current ProcessGroupStatus
+	// This check must execute before checking oldProcessGroupStatus; otherwise, we will add duplicate condition
+	// when both processGroupStatus and oldProcessGroupStatus have the conditionType
+	for _, condition := range processGroupStatus.ProcessGroupConditions {
+		if condition.ProcessGroupConditionType == conditionType {
+			return
+		}
+	}
+
 	// Check if we got a condition for the condition type for the ProcessGroupStatus
 	if oldProcessGroupStatus != nil {
 		for _, condition := range oldProcessGroupStatus.ProcessGroupConditions {
 			if condition.ProcessGroupConditionType == conditionType {
-				// We found a condition with the above condition type
+				// Assumption: processGroupStatus doesn't have the conditionType
 				processGroupStatus.ProcessGroupConditions = append(processGroupStatus.ProcessGroupConditions, condition)
 				return
 			}
 		}
 	}
 
-	// Check if we already got this condition in the current ProcessGroupStatus
-	for _, condition := range processGroupStatus.ProcessGroupConditions {
-		if condition.ProcessGroupConditionType == conditionType {
-			return
-		}
-	}
-
 	// We didn't find any condition so we create a new one
 	processGroupStatus.ProcessGroupConditions = append(processGroupStatus.ProcessGroupConditions, NewProcessGroupCondition(conditionType))
 }
@@ -609,7 +644,7 @@ func FilterByConditions(processGroupStatus []*ProcessGroupStatus, conditionRules
 	return result
 }
 
-// MatchesConditions checks if the provided conditions are matching the current conditions of the process group.
+// MatchesConditions checks if the provided conditionRules matches in the process group's conditions
 //
 // If a condition is mapped to true in the conditionRules map, this condition must be present in the process group.
 // If a condition is mapped to false in the conditionRules map, the condition must be absent in the process group.
@@ -655,6 +690,18 @@ func (processGroupStatus *ProcessGroupStatus) GetConditionTime(conditionType Pro
 	return nil
 }
 
+// GetCondition returns the ProcessGroupStatus's ProcessGroupCondition that matches the conditionType;
+// It returns nil if the ProcessGroupStatus doesn't have a matching condition
+func (processGroupStatus *ProcessGroupStatus) GetCondition(conditionType ProcessGroupConditionType) *ProcessGroupCondition {
+	for _, condition := range processGroupStatus.ProcessGroupConditions {
+		if condition.ProcessGroupConditionType == conditionType {
+			return condition
+		}
+	}
+
+	return nil
+}
+
 // NewProcessGroupCondition creates a new ProcessGroupCondition of the given time with the current timestamp.
 func NewProcessGroupCondition(conditionType ProcessGroupConditionType) *ProcessGroupCondition {
 	return &ProcessGroupCondition{
@@ -701,6 +748,11 @@ const (
 	PodPending ProcessGroupConditionType = "PodPending"
 	// ReadyCondition is currently only used in the metrics.
 	ReadyCondition ProcessGroupConditionType = "Ready"
+	// NodeTaintDetected represents a Pod's node is tainted but not long enough for operator to replace it.
+	// If a node is tainted with a taint that shouldn't trigger replacements, NodeTaintDetected won't be added to the pod
+	NodeTaintDetected ProcessGroupConditionType = "NodeTaintDetected"
+	// NodeTaintReplacing represents a Pod whose node has been tainted and the operator should replace the Pod
+	NodeTaintReplacing ProcessGroupConditionType = "NodeTaintReplacing"
 )
 
 // AllProcessGroupConditionTypes returns all ProcessGroupConditionType
@@ -717,6 +769,8 @@ func AllProcessGroupConditionTypes() []ProcessGroupConditionType {
 		SidecarUnreachable,
 		PodPending,
 		ReadyCondition,
+		NodeTaintDetected,
+		NodeTaintReplacing,
 	}
 }
 
@@ -743,6 +797,10 @@ func GetProcessGroupConditionType(processGroupConditionType string) (ProcessGrou
 		return SidecarUnreachable, nil
 	case "PodPending":
 		return PodPending, nil
+	case "NodeTaintDetected":
+		return NodeTaintDetected, nil
+	case "NodeTaintReplacing":
+		return NodeTaintReplacing, nil
 	}
 
 	return "", fmt.Errorf("unknown process group condition type: %s", processGroupConditionType)
@@ -943,6 +1001,29 @@ type MaintenanceModeOptions struct {
 	MaintenanceModeTimeSeconds *int `json:"maintenanceModeTimeSeconds,omitempty"`
 }
 
+// TaintReplacementOption defines the taint key and taint duration the operator will react to a tainted node
+// Example of TaintReplacementOption
+//   - key: "example.org/maintenance"
+//     durationInSeconds: 7200 # Ensure the taint is present for at least 2 hours before replacing Pods on a node with this taint.
+//   - key: "*" # The wildcard would allow to define a catch all configuration
+//     durationInSeconds: 3600 # Ensure the taint is present for at least 1 hour before replacing Pods on a node with this taint
+//
+// Setting durationInSeconds to the maximum of int64 will practically disable the taint key.
+// When a Node taint key matches both an exact TaintReplacementOption key and a wildcard key, the exact matched key will be used.
+type TaintReplacementOption struct {
+	// Tainted key
+	// +kubebuilder:validation:MaxLength=256
+	// +kubebuilder:validation:Pattern:=^([\-._\/a-z0-9A-Z\*])*$
+	// +kubebuilder:validation:Required
+	Key *string `json:"key,omitempty"`
+
+	// The tainted key must be present for DurationInSeconds before operator replaces pods on the node with this taint;
+	// DurationInSeconds cannot be a negative number.
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:Minimum=0
+	DurationInSeconds *int64 `json:"durationInSeconds,omitempty"`
+}
+
 // AutomaticReplacementOptions controls options for automatically replacing
 // failed processes.
 type AutomaticReplacementOptions struct {
@@ -955,6 +1036,11 @@ type AutomaticReplacementOptions struct {
 	// The default is 7200 seconds, or 2 hours.
 	FailureDetectionTimeSeconds *int `json:"failureDetectionTimeSeconds,omitempty"`
 
+	// TaintReplacementTimeSeconds controls how long a pod stays in NodeTaintReplacing condition
+	// before it is automatically replaced.
+	// The default is 1800 seconds, i.e., 30min
+	TaintReplacementTimeSeconds *int `json:"taintReplacementTimeSeconds,omitempty"`
+
 	// MaxConcurrentReplacements controls how many automatic replacements are allowed to take part.
 	// This will take the list of current replacements and then calculate the difference between
 	// maxConcurrentReplacements and the size of the list. e.g. if currently 3 replacements are
@@ -964,6 +1050,10 @@ type AutomaticReplacementOptions struct {
 	// +kubebuilder:default:=1
 	// +kubebuilder:validation:Minimum=0
 	MaxConcurrentReplacements *int `json:"maxConcurrentReplacements,omitempty"`
+
+	// TaintReplacementOption controls which taint label the operator will react to.
+	// +kubebuilder:validation:MaxItems=32
+	TaintReplacementOptions []TaintReplacementOption `json:"taintReplacementOptions,omitempty"`
 }
 
 // ProcessSettings defines process-level settings.
@@ -2146,6 +2236,11 @@ func (cluster *FoundationDBCluster) GetFailureDetectionTimeSeconds() int {
 	return pointer.IntDeref(cluster.Spec.AutomationOptions.Replacements.FailureDetectionTimeSeconds, 7200)
 }
 
+// GetTaintReplacementTimeSeconds returns cluster.Spec.AutomationOptions.Replacements.TaintReplacementTimeSeconds or if unset the default 1800
+func (cluster *FoundationDBCluster) GetTaintReplacementTimeSeconds() int {
+	return pointer.IntDeref(cluster.Spec.AutomationOptions.Replacements.TaintReplacementTimeSeconds, 1800)
+}
+
 // GetSidecarContainerEnableLivenessProbe returns cluster.Spec.SidecarContainer.EnableLivenessProbe or if unset the default true
 func (cluster *FoundationDBCluster) GetSidecarContainerEnableLivenessProbe() bool {
 	return pointer.BoolDeref(cluster.Spec.SidecarContainer.EnableLivenessProbe, true)
@@ -2401,3 +2496,9 @@ func (cluster *FoundationDBCluster) Validate() error {
 
 	return fmt.Errorf(strings.Join(validations, ", "))
 }
+
+// IsTaintFeatureDisabled return true if operator is configured to not replace Pods tainted Nodes OR
+// if operator's TaintReplacementOptions is not set.
+func (cluster *FoundationDBCluster) IsTaintFeatureDisabled() bool {
+	return len(cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions) == 0
+}
diff --git a/api/v1beta2/zz_generated.deepcopy.go b/api/v1beta2/zz_generated.deepcopy.go
index 33682424..9be5d044 100644
--- a/api/v1beta2/zz_generated.deepcopy.go
+++ b/api/v1beta2/zz_generated.deepcopy.go
@@ -41,11 +41,23 @@ func (in *AutomaticReplacementOptions) DeepCopyInto(out *AutomaticReplacementOpt
 		*out = new(int)
 		**out = **in
 	}
+	if in.TaintReplacementTimeSeconds != nil {
+		in, out := &in.TaintReplacementTimeSeconds, &out.TaintReplacementTimeSeconds
+		*out = new(int)
+		**out = **in
+	}
 	if in.MaxConcurrentReplacements != nil {
 		in, out := &in.MaxConcurrentReplacements, &out.MaxConcurrentReplacements
 		*out = new(int)
 		**out = **in
 	}
+	if in.TaintReplacementOptions != nil {
+		in, out := &in.TaintReplacementOptions, &out.TaintReplacementOptions
+		*out = make([]TaintReplacementOption, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AutomaticReplacementOptions.
@@ -1675,6 +1687,31 @@ func (in *RoutingConfig) DeepCopy() *RoutingConfig {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *TaintReplacementOption) DeepCopyInto(out *TaintReplacementOption) {
+	*out = *in
+	if in.Key != nil {
+		in, out := &in.Key, &out.Key
+		*out = new(string)
+		**out = **in
+	}
+	if in.DurationInSeconds != nil {
+		in, out := &in.DurationInSeconds, &out.DurationInSeconds
+		*out = new(int64)
+		**out = **in
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TaintReplacementOption.
+func (in *TaintReplacementOption) DeepCopy() *TaintReplacementOption {
+	if in == nil {
+		return nil
+	}
+	out := new(TaintReplacementOption)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Version) DeepCopyInto(out *Version) {
 	*out = *in
diff --git a/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml b/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml
index 5f6bdc72..d1659ec2 100644
--- a/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml
+++ b/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml
@@ -9925,6 +9925,22 @@ spec:
                         default: 1
                         minimum: 0
                         type: integer
+                      taintReplacementOptions:
+                        items:
+                          properties:
+                            durationInSeconds:
+                              format: int64
+                              minimum: 0
+                              type: integer
+                            key:
+                              maxLength: 256
+                              pattern: ^([\-._\/a-z0-9A-Z\*])*$
+                              type: string
+                          type: object
+                        maxItems: 32
+                        type: array
+                      taintReplacementTimeSeconds:
+                        type: integer
                     type: object
                   useLocalitiesForExclusion:
                     type: boolean
diff --git a/config/deployment/rbac_role_binding.yaml b/config/deployment/rbac_role_binding.yaml
index eb528e6a..3a76d186 100644
--- a/config/deployment/rbac_role_binding.yaml
+++ b/config/deployment/rbac_role_binding.yaml
@@ -5,8 +5,9 @@ metadata:
   name: manager-rolebinding
 roleRef:
   apiGroup: rbac.authorization.k8s.io
-  kind: Role
+  kind: ClusterRole
   name: fdb-kubernetes-operator-manager-role
 subjects:
 - kind: ServiceAccount
   name: fdb-kubernetes-operator-controller-manager
+
diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
index 4872dfe7..b5ff077b 100644
--- a/config/rbac/role.yaml
+++ b/config/rbac/role.yaml
@@ -1,6 +1,6 @@
 ---
 apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
+kind: ClusterRole
 metadata:
   creationTimestamp: null
   name: manager-role
@@ -22,6 +22,14 @@ rules:
   - patch
   - update
   - watch
+- apiGroups:
+  - ""
+  resources:
+  - nodes
+  verbs:
+  - get
+  - list
+  - watch
 - apiGroups:
   - apps
   resources:
diff --git a/config/samples/deployment.yaml b/config/samples/deployment.yaml
index ef634446..38d8dedd 100644
--- a/config/samples/deployment.yaml
+++ b/config/samples/deployment.yaml
@@ -4,7 +4,7 @@ metadata:
   name: fdb-kubernetes-operator-controller-manager
 ---
 apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
+kind: ClusterRole
 metadata:
   creationTimestamp: null
   name: fdb-kubernetes-operator-manager-role
@@ -26,6 +26,14 @@ rules:
   - patch
   - update
   - watch
+- apiGroups:
+  - ""
+  resources:
+  - nodes
+  verbs:
+  - get
+  - list
+  - watch
 - apiGroups:
   - apps
   resources:
@@ -118,7 +126,7 @@ metadata:
   name: fdb-kubernetes-operator-manager-rolebinding
 roleRef:
   apiGroup: rbac.authorization.k8s.io
-  kind: Role
+  kind: ClusterRole
   name: fdb-kubernetes-operator-manager-role
 subjects:
 - kind: ServiceAccount
diff --git a/controllers/cluster_controller.go b/controllers/cluster_controller.go
index ffd82b91..755962f6 100644
--- a/controllers/cluster_controller.go
+++ b/controllers/cluster_controller.go
@@ -77,6 +77,7 @@ func NewFoundationDBClusterReconciler(podLifecycleManager podmanager.PodLifecycl
 // +kubebuilder:rbac:groups=apps.foundationdb.org,resources=foundationdbclusters/status,verbs=get;update;patch
 // +kubebuilder:rbac:groups="",resources=pods;configmaps;persistentvolumeclaims;events;secrets;services,verbs=get;list;watch;create;update;patch;delete
 // +kubebuilder:rbac:groups="coordination.k8s.io",resources=leases,verbs=get;list;watch;create;update;patch;delete
+// +kubebuilder:rbac:groups="",resources=nodes,verbs=get;list;watch
 
 // Reconcile runs the reconciliation logic.
 func (r *FoundationDBClusterReconciler) Reconcile(ctx context.Context, request ctrl.Request) (ctrl.Result, error) {
@@ -181,7 +182,9 @@ func (r *FoundationDBClusterReconciler) Reconcile(ctx context.Context, request c
 	}
 
 	if cluster.Status.Generations.Reconciled < originalGeneration || delayedRequeue {
-		clusterLog.Info("Cluster was not fully reconciled by reconciliation process", "status", cluster.Status.Generations)
+		clusterLog.Info("Cluster was not fully reconciled by reconciliation process", "status", cluster.Status.Generations,
+			"CurrentGeneration", cluster.Status.Generations.Reconciled,
+			"OriginalGeneration", originalGeneration, "DelayedRequeue", delayedRequeue)
 
 		return ctrl.Result{Requeue: true}, nil
 	}
diff --git a/controllers/replace_failed_process_groups.go b/controllers/replace_failed_process_groups.go
index 7c79530c..bf34a198 100644
--- a/controllers/replace_failed_process_groups.go
+++ b/controllers/replace_failed_process_groups.go
@@ -31,7 +31,7 @@ import (
 // replaced.
 type replaceFailedProcessGroups struct{}
 
-// reconcile runs the reconciler's work.
+// return non-nil requeue if a process has been replaced
 func (c replaceFailedProcessGroups) reconcile(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster) *requeue {
 	logger := log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name, "reconciler", "replaceFailedProcessGroups")
 	// If the EmptyMonitorConf setting is set we expect that all fdb processes in this part of the cluster are missing. In order
diff --git a/controllers/update_status.go b/controllers/update_status.go
index c1193067..da39c8b7 100644
--- a/controllers/update_status.go
+++ b/controllers/update_status.go
@@ -23,10 +23,12 @@ package controllers
 import (
 	"context"
 	"fmt"
+	"math"
 	"sort"
 	"time"
 
 	"github.com/FoundationDB/fdb-kubernetes-operator/internal/locality"
+	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/podmanager"
 	"github.com/go-logr/logr"
@@ -39,6 +41,7 @@ import (
 	"k8s.io/apimachinery/pkg/api/equality"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
 	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/utils/pointer"
 )
 
 // updateStatus provides a reconciliation step for updating the status in the
@@ -112,7 +115,7 @@ func (updateStatus) reconcile(ctx context.Context, r *FoundationDBClusterReconci
 	// Update the running version based on the reported version of the FDB processes
 	version, err := getRunningVersion(versionMap, cluster.Status.RunningVersion)
 	if err != nil {
-		return &requeue{curError: err}
+		return &requeue{curError: fmt.Errorf("update_status skipped due to error in getRunningVersion: %w", err)}
 	}
 	cluster.Status.RunningVersion = version
 
@@ -133,7 +136,7 @@ func (updateStatus) reconcile(ctx context.Context, r *FoundationDBClusterReconci
 		for _, coordinator := range databaseStatus.Client.Coordinators.Coordinators {
 			address, err := fdbv1beta2.ParseProcessAddress(coordinator.Address.String())
 			if err != nil {
-				return &requeue{curError: err}
+				return &requeue{curError: fmt.Errorf("update_status skipped due to error in ParseProcessAddress: %w", err)}
 			}
 
 			if address.Flags["tls"] {
@@ -153,7 +156,7 @@ func (updateStatus) reconcile(ctx context.Context, r *FoundationDBClusterReconci
 
 	configMap, err := internal.GetConfigMap(cluster)
 	if err != nil {
-		return &requeue{curError: err}
+		return &requeue{curError: fmt.Errorf("update_status skipped due to error in GetConfigMap: %w", err)}
 	}
 
 	status.ProcessGroups = make([]*fdbv1beta2.ProcessGroupStatus, 0, len(cluster.Status.ProcessGroups))
@@ -165,12 +168,12 @@ func (updateStatus) reconcile(ctx context.Context, r *FoundationDBClusterReconci
 
 	pods, pvcs, err := refreshProcessGroupStatus(ctx, r, cluster, &status)
 	if err != nil {
-		return &requeue{curError: err}
+		return &requeue{curError: fmt.Errorf("update_status skipped due to error in refreshProcessGroupStatus: %w", err)}
 	}
 
-	status.ProcessGroups, err = validateProcessGroups(ctx, r, cluster, &status, processMap, configMap, pods, pvcs)
+	status.ProcessGroups, err = validateProcessGroups(ctx, r, cluster, &status, processMap, configMap, pods, pvcs, logger)
 	if err != nil {
-		return &requeue{curError: err}
+		return &requeue{curError: fmt.Errorf("update_status skipped due to error in validateProcessGroups: %w", err)}
 	}
 	removeDuplicateConditions(status)
 
@@ -390,7 +393,10 @@ func checkAndSetProcessStatus(r *FoundationDBClusterReconciler, cluster *fdbv1be
 		correct = commandLine == process.CommandLine && versionMatch && !cluster.Spec.Buggify.EmptyMonitorConf
 
 		if !correct {
-			log.Info("IncorrectProcess", "expected", commandLine, "got", process.CommandLine, "expectedVersion", cluster.Spec.Version, "version", process.Version, "processGroupID", processGroupStatus.ProcessGroupID)
+			log.Info("IncorrectProcess", "expected", commandLine, "got", process.CommandLine,
+				"expectedVersion", cluster.Spec.Version,
+				"version", process.Version, "processGroupID", processGroupStatus.ProcessGroupID,
+				"emptyMonitorConf is ", cluster.Spec.Buggify.EmptyMonitorConf)
 		}
 	}
 
@@ -401,8 +407,10 @@ func checkAndSetProcessStatus(r *FoundationDBClusterReconciler, cluster *fdbv1be
 	return nil
 }
 
-func validateProcessGroups(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBClusterStatus, processMap map[fdbv1beta2.ProcessGroupID][]fdbv1beta2.FoundationDBStatusProcessInfo, configMap *corev1.ConfigMap, pods []*corev1.Pod, pvcs *corev1.PersistentVolumeClaimList) ([]*fdbv1beta2.ProcessGroupStatus, error) {
+// Validate and set progressGroup's status
+func validateProcessGroups(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBClusterStatus, processMap map[fdbv1beta2.ProcessGroupID][]fdbv1beta2.FoundationDBStatusProcessInfo, configMap *corev1.ConfigMap, pods []*corev1.Pod, pvcs *corev1.PersistentVolumeClaimList, logger logr.Logger) ([]*fdbv1beta2.ProcessGroupStatus, error) {
 	var err error
+	nodeMap := make(map[string]*corev1.Node)
 	processGroups := status.ProcessGroups
 	processGroupsWithoutExclusion := make(map[fdbv1beta2.ProcessGroupID]fdbv1beta2.None, len(cluster.Spec.ProcessGroupsToRemoveWithoutExclusion))
 
@@ -413,6 +421,11 @@ func validateProcessGroups(ctx context.Context, r *FoundationDBClusterReconciler
 	podMap := internal.CreatePodMap(cluster, pods)
 	pvcMap := internal.CreatePVCMap(cluster, pvcs)
 
+	disableTaintFeature := cluster.IsTaintFeatureDisabled()
+	if disableTaintFeature {
+		logger.Info("Disable taint feature", "Disabled", disableTaintFeature)
+	}
+
 	for _, processGroup := range processGroups {
 		pod, podExists := podMap[processGroup.ProcessGroupID]
 		// If the process group is not being removed and the Pod is not set we need to put it into
@@ -521,7 +534,7 @@ func validateProcessGroups(ctx context.Context, r *FoundationDBClusterReconciler
 			pvc = &pvcValue
 		}
 
-		err = validateProcessGroup(ctx, r, cluster, pod, pvc, configMapHash, processGroup)
+		err = validateProcessGroup(ctx, r, cluster, pod, pvc, configMapHash, processGroup, disableTaintFeature, nodeMap, logger)
 		if err != nil {
 			return processGroups, err
 		}
@@ -532,8 +545,9 @@ func validateProcessGroups(ctx context.Context, r *FoundationDBClusterReconciler
 
 // validateProcessGroup runs specific checks for the status of an process group.
 // returns failing, incorrect, error
-func validateProcessGroup(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster, pod *corev1.Pod, currentPVC *corev1.PersistentVolumeClaim, configMapHash string, processGroupStatus *fdbv1beta2.ProcessGroupStatus) error {
-	logger := log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name, "reconciler", "updateStatus")
+func validateProcessGroup(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster,
+	pod *corev1.Pod, currentPVC *corev1.PersistentVolumeClaim, configMapHash string, processGroupStatus *fdbv1beta2.ProcessGroupStatus,
+	disableTaintFeature bool, nodeMap map[string]*corev1.Node, logger logr.Logger) error {
 	processGroupStatus.UpdateCondition(fdbv1beta2.MissingPod, pod == nil, cluster.Status.ProcessGroups, processGroupStatus.ProcessGroupID)
 	if pod == nil {
 		return nil
@@ -586,6 +600,9 @@ func validateProcessGroup(ctx context.Context, r *FoundationDBClusterReconciler,
 	if !incorrectPVC && desiredPvc != nil {
 		incorrectPVC = !metadataMatches(currentPVC.ObjectMeta, desiredPvc.ObjectMeta)
 	}
+	if incorrectPVC {
+		log.Info("ValidateProcessGroup found incorrectPVC", "CurrentPVC", currentPVC, "DesiredPVC", desiredPvc)
+	}
 
 	processGroupStatus.UpdateCondition(fdbv1beta2.MissingPVC, incorrectPVC, cluster.Status.ProcessGroups, processGroupStatus.ProcessGroupID)
 
@@ -624,6 +641,80 @@ func validateProcessGroup(ctx context.Context, r *FoundationDBClusterReconciler,
 	processGroupStatus.UpdateCondition(fdbv1beta2.PodFailing, failing, cluster.Status.ProcessGroups, processGroupStatus.ProcessGroupID)
 	processGroupStatus.UpdateCondition(fdbv1beta2.PodPending, false, cluster.Status.ProcessGroups, processGroupStatus.ProcessGroupID)
 
+	if !disableTaintFeature {
+		// Update taint status
+		err = updateTaintCondition(ctx, r, cluster, pod, processGroupStatus, nodeMap, logger)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// updateTaintCondition checks pod's node taint label and update pod's taint-related condition accordingly
+func updateTaintCondition(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster,
+	pod *corev1.Pod, processGroupStatus *fdbv1beta2.ProcessGroupStatus, nodeMap map[string]*corev1.Node, logger logr.Logger) error {
+	node, ok := nodeMap[pod.Spec.NodeName]
+	log.V(1).Info("Get pod's node", "Pod", pod.Name, "Pod's node name", pod.Spec.NodeName, "node map size", len(nodeMap))
+	if !ok {
+		node = &corev1.Node{}
+		err := r.Get(ctx, client.ObjectKey{Name: pod.Spec.NodeName}, node)
+		if err != nil {
+			return fmt.Errorf("get pod %s node %s fails with error :%w", pod.Name, pod.Spec.NodeName, err)
+		}
+		nodeMap[pod.Spec.NodeName] = node
+		log.V(1).Info("Set nodeMap cache", "Pod", pod.Name, "Pod's node name", pod.Spec.NodeName, "Node", node)
+	}
+
+	// Check the tainted duration and only mark the process group tainted after the configured tainted duration
+	// TODO: https://github.com/FoundationDB/fdb-kubernetes-operator/issues/1583
+	hasMatchingTaint := false
+	for _, taint := range node.Spec.Taints {
+		hasExactMatch := hasExactMatchedTaintKey(cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions, taint.Key)
+		for _, taintConfiguredKey := range cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions {
+			taintConfiguredKeyString := pointer.StringDeref(taintConfiguredKey.Key, "")
+			if taintConfiguredKeyString == "" || pointer.Int64Deref(taintConfiguredKey.DurationInSeconds, math.MinInt64) < 0 {
+				logger.Info("Cluster's TaintReplacementOption is disabled", "Key",
+					taintConfiguredKey.Key, "DurationInSeconds", taintConfiguredKey.DurationInSeconds,
+					"Pod", pod.Name)
+				continue
+			}
+			if taint.Key == "" {
+				// Node's taint is not properly set, skip the node's taint
+				logger.Info("Taint is not properly set", "Node", node.Name, "Taint", taint)
+				continue
+			}
+			// If a wildcard is specified as key, it will be used as default value, except there is an exact match defined for the taint key.
+			if (hasExactMatch && taint.Key != taintConfiguredKeyString) ||
+				(!hasExactMatch && taintConfiguredKeyString != "*") {
+				continue
+			}
+
+			hasMatchingTaint = true
+			processGroupStatus.UpdateCondition(fdbv1beta2.NodeTaintDetected, true, cluster.Status.ProcessGroups, processGroupStatus.ProcessGroupID)
+			// Use node taint's timestamp as the NodeTaintDetected condition's starting time
+			if !taint.TimeAdded.IsZero() && taint.TimeAdded.Time.Unix() < *processGroupStatus.GetConditionTime(fdbv1beta2.NodeTaintDetected) {
+				processGroupStatus.UpdateConditionTime(fdbv1beta2.NodeTaintDetected, taint.TimeAdded.Unix())
+			}
+			taintDetectedTime := pointer.Int64Deref(processGroupStatus.GetConditionTime(fdbv1beta2.NodeTaintDetected), math.MaxInt64)
+			if time.Now().Unix()-taintDetectedTime > pointer.Int64Deref(taintConfiguredKey.DurationInSeconds, math.MaxInt64) {
+				processGroupStatus.UpdateCondition(fdbv1beta2.NodeTaintReplacing, true, cluster.Status.ProcessGroups, processGroupStatus.ProcessGroupID)
+				logger.Info("Add NodeTaintReplacing condition", "Pod", pod.Name, "Node", node.Name,
+					"TaintKey", taint.Key, "TaintDetectedTime", taintDetectedTime,
+					"TaintDuration", int64(time.Since(time.Unix(taintDetectedTime, 0))),
+					"TaintValue", taint.Value, "TaintEffect", taint.Effect,
+					"ClusterTaintDetectionDuration", time.Duration(pointer.Int64Deref(taintConfiguredKey.DurationInSeconds, math.MaxInt64)))
+			}
+		}
+	}
+	if !hasMatchingTaint {
+		// Remove NodeTaintDetected condition if the pod is no longer on a tainted node;
+		// This is needed especially when a node's status is flapping
+		// We do not remove NodeTaintDetected because the NodeTaintDetected condition is only added when taint is there for configured long time
+		processGroupStatus.UpdateCondition(fdbv1beta2.NodeTaintDetected, false, cluster.Status.ProcessGroups, processGroupStatus.ProcessGroupID)
+	}
+
 	return nil
 }
 
@@ -750,3 +841,12 @@ func getRunningVersion(versionMap map[string]int, fallback string) (string, erro
 
 	return currentCandidate.String(), nil
 }
+
+func hasExactMatchedTaintKey(taintReplacementOptions []fdbv1beta2.TaintReplacementOption, nodeTaintKey string) bool {
+	for _, configuredTaintKey := range taintReplacementOptions {
+		if *configuredTaintKey.Key == nodeTaintKey {
+			return true
+		}
+	}
+	return false
+}
diff --git a/docs/cluster_spec.md b/docs/cluster_spec.md
index ab9f58b6..ac854571 100644
--- a/docs/cluster_spec.md
+++ b/docs/cluster_spec.md
@@ -30,6 +30,7 @@ This Document documents the types introduced by the FoundationDB Operator to be
 * [ProcessSettings](#processsettings)
 * [RequiredAddressSet](#requiredaddressset)
 * [RoutingConfig](#routingconfig)
+* [TaintReplacementOption](#taintreplacementoption)
 * [DataCenter](#datacenter)
 * [DatabaseConfiguration](#databaseconfiguration)
 * [ExcludedServers](#excludedservers)
@@ -47,7 +48,9 @@ AutomaticReplacementOptions controls options for automatically replacing failed
 | ----- | ----------- | ------ | -------- |
 | enabled | Enabled controls whether automatic replacements are enabled. The default is false. | *bool | false |
 | failureDetectionTimeSeconds | FailureDetectionTimeSeconds controls how long a process must be failed or missing before it is automatically replaced. The default is 7200 seconds, or 2 hours. | *int | false |
+| taintReplacementTimeSeconds | TaintReplacementTimeSeconds controls how long a pod stays in NodeTaintReplacing condition before it is automatically replaced. The default is 1800 seconds, i.e., 30min | *int | false |
 | maxConcurrentReplacements | MaxConcurrentReplacements controls how many automatic replacements are allowed to take part. This will take the list of current replacements and then calculate the difference between maxConcurrentReplacements and the size of the list. e.g. if currently 3 replacements are queued (e.g. in the processGroupsToRemove list) and maxConcurrentReplacements is 5 the operator is allowed to replace at most 2 process groups. Setting this to 0 will basically disable the automatic replacements. | *int | false |
+| taintReplacementOptions | TaintReplacementOption controls which taint label the operator will react to. | [][TaintReplacementOption](#taintreplacementoption) | false |
 
 [Back to TOC](#table-of-contents)
 
@@ -458,6 +461,17 @@ RoutingConfig allows configuring routing to our pods, and services that sit in f
 
 [Back to TOC](#table-of-contents)
 
+## TaintReplacementOption
+
+TaintReplacementOption defines the taint key and taint duration the operator will react to a tainted node Example of TaintReplacementOption   - key: \"example.org/maintenance\"     durationInSeconds: 7200 # Ensure the taint is present for at least 2 hours before replacing Pods on a node with this taint.   - key: \"*\" # The wildcard would allow to define a catch all configuration     durationInSeconds: 3600 # Ensure the taint is present for at least 1 hour before replacing Pods on a node with this taint  Setting durationInSeconds to the maximum of int64 will practically disable the taint key. When a Node taint key matches both an exact TaintReplacementOption key and a wildcard key, the exact matched key will be used.
+
+| Field | Description | Scheme | Required |
+| ----- | ----------- | ------ | -------- |
+| key | Tainted key | *string | false |
+| durationInSeconds | The tainted key must be present for DurationInSeconds before operator replaces pods on the node with this taint; DurationInSeconds cannot be a negative number. | *int64 | false |
+
+[Back to TOC](#table-of-contents)
+
 ## FoundationDBCustomParameter
 
 FoundationDBCustomParameter defines a single custom knob
diff --git a/fdbclient/admin_client.go b/fdbclient/admin_client.go
index 4abab7d3..aa79565f 100644
--- a/fdbclient/admin_client.go
+++ b/fdbclient/admin_client.go
@@ -498,6 +498,20 @@ func getRemainingAndExcludedFromStatus(status *fdbv1beta2.FoundationDBStatus, ad
 	notExcludedAddresses := map[string]fdbv1beta2.None{}
 	fullyExcludedAddresses := map[string]fdbv1beta2.None{}
 	visitedAddresses := map[string]fdbv1beta2.None{}
+
+	// If there are more than 1 active generations we can not handout any information about excluded processes based on
+	// the cluster status information as only the latest log processes will have the log process role. If we don't check
+	// for the active generations we have the risk to remove a log process that still has mutations on it that must be
+	// popped.
+	if status.Cluster.RecoveryState.ActiveGenerations > 1 {
+		return exclusionStatus{
+			inProgress:      nil,
+			fullyExcluded:   nil,
+			notExcluded:     addresses,
+			missingInStatus: nil,
+		}
+	}
+
 	for _, addr := range addresses {
 		notExcludedAddresses[addr.MachineAddress()] = fdbv1beta2.None{}
 	}
diff --git a/internal/replacements/replace_failed_process_groups.go b/internal/replacements/replace_failed_process_groups.go
index 855eec9f..3a202cf0 100644
--- a/internal/replacements/replace_failed_process_groups.go
+++ b/internal/replacements/replace_failed_process_groups.go
@@ -82,7 +82,7 @@ ProcessGroupLoop:
 			}
 		}
 
-		needsReplacement, missingTime := processGroupStatus.NeedsReplacement(cluster.GetFailureDetectionTimeSeconds())
+		needsReplacement, missingTime := processGroupStatus.NeedsReplacement(cluster.GetFailureDetectionTimeSeconds(), cluster.GetTaintReplacementTimeSeconds())
 		if !needsReplacement {
 			continue
 		}
diff --git a/kubectl-fdb/cmd/analyze.go b/kubectl-fdb/cmd/analyze.go
index cf79b559..6b84cc87 100644
--- a/kubectl-fdb/cmd/analyze.go
+++ b/kubectl-fdb/cmd/analyze.go
@@ -23,11 +23,12 @@ package cmd
 import (
 	"encoding/json"
 	"fmt"
+	"strings"
+	"time"
+
 	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
 	"k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/rest"
-	"strings"
-	"time"
 
 	"context"
 
@@ -295,7 +296,7 @@ func analyzeCluster(cmd *cobra.Command, kubeClient client.Client, cluster *fdbv1
 			printStatement(cmd, statement, errorMessage)
 		}
 
-		needsReplacement, _ := processGroup.NeedsReplacement(0)
+		needsReplacement, _ := processGroup.NeedsReplacement(0, 0)
 		if needsReplacement && autoFix {
 			failedProcessGroups = append(failedProcessGroups, string(processGroup.ProcessGroupID))
 		}
diff --git a/kubectl-fdb/cmd/remove_process_group.go b/kubectl-fdb/cmd/remove_process_group.go
index 03822030..d7342dbe 100644
--- a/kubectl-fdb/cmd/remove_process_group.go
+++ b/kubectl-fdb/cmd/remove_process_group.go
@@ -153,7 +153,7 @@ func replaceProcessGroups(kubeClient client.Client, clusterName string, ids []st
 				continue
 			}
 
-			needsReplacement, _ := processGroupStatus.NeedsReplacement(0)
+			needsReplacement, _ := processGroupStatus.NeedsReplacement(0, 0)
 			if needsReplacement {
 				processGroupIDs = append(processGroupIDs, processGroupStatus.ProcessGroupID)
 			}
diff --git a/mock-kubernetes-client/client/client.go b/mock-kubernetes-client/client/client.go
index f35824a8..37e69b29 100644
--- a/mock-kubernetes-client/client/client.go
+++ b/mock-kubernetes-client/client/client.go
@@ -28,6 +28,7 @@ import (
 
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/equality"
+	"k8s.io/apimachinery/pkg/api/errors"
 	"k8s.io/apimachinery/pkg/api/meta"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
@@ -75,7 +76,7 @@ func NewMockClientWithHooks(scheme *runtime.Scheme, createHooks []func(ctx conte
 		return nil
 	}
 
-	podCreateHook := func(_ context.Context, client *MockClient, object ctrlClient.Object) error {
+	podCreateHook := func(ctx context.Context, client *MockClient, object ctrlClient.Object) error {
 		pod, isPod := object.(*corev1.Pod)
 		if !isPod {
 			return nil
@@ -89,7 +90,19 @@ func NewMockClientWithHooks(scheme *runtime.Scheme, createHooks []func(ctx conte
 			pod.Status.Phase = corev1.PodRunning
 		}
 
-		return nil
+		nodeName := fmt.Sprintf("%s-node", pod.Name)
+		node := corev1.Node{
+			ObjectMeta: metav1.ObjectMeta{Name: nodeName},
+		}
+		err := client.Get(ctx, ctrlClient.ObjectKey{Name: nodeName}, &node)
+		if pod.Spec.NodeName == "" { // Do not override user-set NodeName so that we can simulate multiple pods on a single node
+			pod.Spec.NodeName = node.Name
+		}
+		if errors.IsNotFound(err) {
+			return client.Create(context.Background(), &node)
+		}
+
+		return err
 	}
 
 	return &MockClient{


Test patch:
diff --git a/api/v1beta2/foundationdbcluster_types_test.go b/api/v1beta2/foundationdbcluster_types_test.go
index 1616880e..41021076 100644
--- a/api/v1beta2/foundationdbcluster_types_test.go
+++ b/api/v1beta2/foundationdbcluster_types_test.go
@@ -4035,7 +4035,7 @@ var _ = Describe("[api] FoundationDBCluster", func() {
 		})
 
 		JustBeforeEach(func() {
-			needsReplacement, timestamp = processGroup.NeedsReplacement(60)
+			needsReplacement, timestamp = processGroup.NeedsReplacement(60, 60)
 		})
 
 		Context("with no conditions", func() {
diff --git a/controllers/cluster_controller_test.go b/controllers/cluster_controller_test.go
index 41ec67ea..1da92acb 100644
--- a/controllers/cluster_controller_test.go
+++ b/controllers/cluster_controller_test.go
@@ -57,6 +57,9 @@ import (
 var firstLogIndex = 1
 var firstStorageIndex = 13
 
+// reloadCluster reloads FoundationDBCluster object info to its local copy.
+// It's necessary in unit test when reconciler changes FoundationDBCluster object but the local's state is not updated.
+// We reloadCluster to bring client state consistent with the latest FoundationDBCluster object
 func reloadCluster(cluster *fdbv1beta2.FoundationDBCluster) (int64, error) {
 	generations, err := reloadClusterGenerations(cluster)
 	if generations.HasPendingRemoval > 0 {
diff --git a/controllers/replace_failed_process_groups_test.go b/controllers/replace_failed_process_groups_test.go
index f7cbc92d..d671e107 100644
--- a/controllers/replace_failed_process_groups_test.go
+++ b/controllers/replace_failed_process_groups_test.go
@@ -22,10 +22,16 @@ package controllers
 
 import (
 	ctx "context"
+	"math/rand"
 	"time"
 
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/fdbadminclient/mock"
+	"github.com/go-logr/logr"
+	"sigs.k8s.io/controller-runtime/pkg/client"
 
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/utils/pointer"
 
 	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
@@ -63,6 +69,568 @@ var _ = Describe("replace_failed_process_groups", func() {
 		result = replaceFailedProcessGroups{}.reconcile(ctx.Background(), clusterReconciler, cluster)
 	})
 
+	Context("replace pod on tainted node", func() {
+		taintKeyStar := "*"
+		taintKeyStarDuration := int64(20)
+		taintKeyMaintenance := "foundationdb.org/maintenance"
+		taintKeyMaintenanceDuration := int64(10)
+		var allPods []*corev1.Pod
+		var allPvcs *corev1.PersistentVolumeClaimList
+		var podOnTaintedNode *corev1.Pod
+		var targetPodProcessGroupID fdbv1beta2.ProcessGroupID
+		var node *corev1.Node
+		var adminClient *mock.AdminClient
+		var configMap *corev1.ConfigMap
+		var processMap map[fdbv1beta2.ProcessGroupID][]fdbv1beta2.FoundationDBStatusProcessInfo
+		var logger logr.Logger
+
+		BeforeEach(func() {
+			logger = log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name, "test", "replace pod on tainted node")
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{
+				{
+					Key:               &taintKeyStar,
+					DurationInSeconds: &taintKeyStarDuration,
+				},
+				{
+					Key:               &taintKeyMaintenance,
+					DurationInSeconds: &taintKeyMaintenanceDuration,
+				},
+			}
+			cluster.Spec.AutomationOptions.Replacements.FailureDetectionTimeSeconds = pointer.Int(5)
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementTimeSeconds = pointer.Int(1)
+			// Update cluster config so that generic reconciliation will work
+			err := k8sClient.Update(ctx.TODO(), cluster)
+			Expect(err).NotTo(HaveOccurred())
+
+			adminClient, err = mock.NewMockAdminClientUncast(cluster, k8sClient)
+
+			Expect(err).NotTo(HaveOccurred())
+			databaseStatus, err := adminClient.GetStatus()
+			Expect(err).NotTo(HaveOccurred())
+			processMap = make(map[fdbv1beta2.ProcessGroupID][]fdbv1beta2.FoundationDBStatusProcessInfo)
+			for _, process := range databaseStatus.Cluster.Processes {
+				processID, ok := process.Locality["process_id"]
+				// if the processID is not set we fall back to the instanceID
+				if !ok {
+					processID = process.Locality["instance_id"]
+				}
+				processMap[fdbv1beta2.ProcessGroupID(processID)] = append(processMap[fdbv1beta2.ProcessGroupID(processID)], process)
+			}
+
+			allPods, err = clusterReconciler.PodLifecycleManager.GetPods(ctx.TODO(), clusterReconciler, cluster, internal.GetPodListOptions(cluster, "", "")...)
+			Expect(err).NotTo(HaveOccurred())
+
+			allPvcs = &corev1.PersistentVolumeClaimList{}
+			err = clusterReconciler.List(ctx.TODO(), allPvcs, internal.GetPodListOptions(cluster, "", "")...)
+			Expect(err).NotTo(HaveOccurred())
+
+			configMap = &corev1.ConfigMap{}
+			err = k8sClient.Get(ctx.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: cluster.Name + "-config"}, configMap)
+			Expect(err).NotTo(HaveOccurred())
+
+			pods, err := clusterReconciler.PodLifecycleManager.GetPods(ctx.TODO(), clusterReconciler, cluster, internal.GetSinglePodListOptions(cluster, "storage-1")...)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(pods)).To(Equal(1))
+
+			podOnTaintedNode = pods[0] // Future: choose a random pod to test
+			node = &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{Name: podOnTaintedNode.Spec.NodeName},
+			}
+			targetPodProcessGroupID = internal.GetProcessGroupIDFromMeta(cluster, podOnTaintedNode.ObjectMeta)
+
+			// Call validateProcessGroups to set processGroupStatus to tainted condition
+			processGroupsStatus, err := validateProcessGroups(ctx.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(processGroupsStatus)).To(BeNumerically(">", 4))
+			processGroup := processGroupsStatus[len(processGroupsStatus)-4]
+			Expect(processGroup.ProcessGroupID).To(Equal(fdbv1beta2.ProcessGroupID("storage-1")))
+			Expect(len(processGroupsStatus[0].ProcessGroupConditions)).To(Equal(0))
+		})
+
+		It("should not replace a pod whose condition is NodeTaintDetected but not NodeTaintReplacing ", func() {
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now()},
+				},
+			}
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+			log.Info("Taint node", "Node name", podOnTaintedNode.Name, "Node taints", node.Spec.Taints)
+
+			processGroupsStatus, err := validateProcessGroups(ctx.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(processGroupsStatus)).To(BeNumerically(">", 4))
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(1))
+			Expect(targetProcessGroupStatus.ProcessGroupConditions[0].ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintDetected))
+
+			Expect(replaceFailedProcessGroups{}.reconcile(ctx.TODO(), clusterReconciler, cluster)).To(BeNil())
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		It("should replace a pod whose condition is NodeTaintReplacing but not NodeTaintDetected", func() {
+			// This test case covers the scenario when a node is tainted for a while and then no longer tainted
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+			log.Info("Taint node", "Node name", podOnTaintedNode.Name, "Node taints", node.Spec.Taints)
+
+			processGroupsStatus, err := validateProcessGroups(ctx.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(processGroupsStatus)).To(BeNumerically(">", 4))
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(2))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected)).NotTo(Equal(nil))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing)).NotTo(Equal(nil))
+
+			node.Spec.Taints = []corev1.Taint{}
+			err = k8sClient.Update(ctx.TODO(), node)
+			Expect(err).NotTo(HaveOccurred())
+			processGroupsStatus, err = validateProcessGroups(ctx.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(processGroupsStatus)).To(BeNumerically(">", 4))
+			targetProcessGroupStatus = fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(1))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing)).NotTo(BeNil())
+
+			Expect(replaceFailedProcessGroups{}.reconcile(ctx.TODO(), clusterReconciler, cluster)).To(BeNil())
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		It("should replace a pod that is both NodeTaintDetected and NodeTaintReplacing ", func() {
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			log.Info("Taint node", "Node name", podOnTaintedNode.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			processGroupsStatus, err := validateProcessGroups(ctx.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(processGroupsStatus)).To(BeNumerically(">", 4))
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(2))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected).ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintDetected))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing).ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintReplacing))
+
+			// cluster won't replace a failed process until GetTaintReplacementTimeSeconds() later
+			Expect(replaceFailedProcessGroups{}.reconcile(ctx.TODO(), clusterReconciler, cluster)).To(BeNil())
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+
+			Eventually(func() *requeue {
+				result = replaceFailedProcessGroups{}.reconcile(ctx.TODO(), clusterReconciler, cluster)
+				return result
+			}).WithTimeout(time.Duration(cluster.GetTaintReplacementTimeSeconds()*3) * time.Second).WithPolling(1 * time.Second).ShouldNot(BeNil())
+
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{targetPodProcessGroupID}))
+		})
+
+		It("should replace a pod that has NodeTaintReplacing condition but no longer has NodeTaintDetected condition", func() {
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			log.Info("Taint node", "Node name", podOnTaintedNode.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			processGroupsStatus, err := validateProcessGroups(ctx.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(processGroupsStatus)).To(BeNumerically(">", 4))
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(2))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected).ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintDetected))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing).ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintReplacing))
+
+			Eventually(func() *requeue {
+				return replaceFailedProcessGroups{}.reconcile(ctx.TODO(), clusterReconciler, cluster)
+			}).WithTimeout(time.Duration(cluster.GetTaintReplacementTimeSeconds()*3) * time.Second).WithPolling(1 * time.Second).ShouldNot(BeNil())
+
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{targetPodProcessGroupID}))
+		})
+
+		It("should not replace a pod that is on a flapping tainted node", func() {
+			// Flapping tainted node
+			tainted := int64(0)
+			for taintTimeOffset := taintKeyMaintenanceDuration * 2; taintTimeOffset >= 0; taintTimeOffset-- {
+				tainted = taintTimeOffset % 2
+				if tainted == 0 {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-1 * time.Second * time.Duration(taintTimeOffset))},
+						},
+					}
+				} else {
+					node.Spec.Taints = []corev1.Taint{}
+				}
+				err = k8sClient.Update(ctx.TODO(), node)
+				Expect(err).NotTo(HaveOccurred())
+				log.Info("Taint node", "Tainted", tainted, "Node name", podOnTaintedNode.Name, "Node taints", node.Spec.Taints, "Now", time.Now())
+			}
+			Expect(tainted).To(Equal(int64(0)))
+
+			processGroupsStatus, err := validateProcessGroups(ctx.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(processGroupsStatus)).To(BeNumerically(">", 4))
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(1))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected).ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintDetected))
+
+			// cluster won't replace the process
+			time.Sleep(time.Second * time.Duration(cluster.GetTaintReplacementTimeSeconds()+1))
+			Expect(replaceFailedProcessGroups{}.reconcile(ctx.TODO(), clusterReconciler, cluster)).To(BeNil())
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		It("should replace a pod that is both NodeTaintDetected and NodeTaintReplacing with cluster reconciliation", func() {
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			log.Info("Taint node", "Node name", node.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			result, err := reconcileClusterWithCustomRequeueLimit(cluster, 1)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(result.Requeue).To(BeTrue())
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+
+			Eventually(func() *corev1.Pod {
+				_, err = reconcileCluster(cluster)
+				Expect(err).NotTo(HaveOccurred())
+				return (getPodByProcessGroupID(cluster, internal.GetProcessClassFromMeta(cluster, podOnTaintedNode.ObjectMeta), internal.GetProcessGroupIDFromMeta(cluster, podOnTaintedNode.ObjectMeta)))
+			}).WithTimeout(time.Duration(cluster.GetTaintReplacementTimeSeconds()*3) * time.Second).WithPolling(1 * time.Second).Should(BeNil())
+
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		It("exact matched taint key should be preferred over wildcard key", func() {
+			err = k8sClient.Get(ctx.TODO(), client.ObjectKey{Namespace: cluster.Namespace, Name: cluster.Name}, cluster)
+			Expect(err).NotTo(HaveOccurred())
+			taintKeyStarDurationShort := int64(5)
+			taintKeyMaintenanceDurationLong := int64(50)
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{
+				{
+					Key:               &taintKeyStar,
+					DurationInSeconds: &taintKeyStarDurationShort,
+				},
+				{
+					Key:               &taintKeyMaintenance,
+					DurationInSeconds: &taintKeyMaintenanceDurationLong,
+				},
+			}
+			Expect(k8sClient.Update(ctx.TODO(), cluster)).NotTo(HaveOccurred())
+
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyStarDurationShort))},
+				},
+			}
+			log.Info("Taint node", "Node name", node.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			_, err = reconcileCluster(cluster)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(k8sClient.Get(ctx.TODO(), client.ObjectKey{Namespace: cluster.Namespace, Name: cluster.Name}, cluster)).NotTo(HaveOccurred())
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(1))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected).ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintDetected))
+
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDurationLong))},
+				},
+			}
+			log.Info("Taint node", "Node name", node.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			Eventually(func() *corev1.Pod {
+				_, err = reconcileCluster(cluster)
+				Expect(err).NotTo(HaveOccurred())
+				return (getPodByProcessGroupID(cluster, internal.GetProcessClassFromMeta(cluster, podOnTaintedNode.ObjectMeta), internal.GetProcessGroupIDFromMeta(cluster, podOnTaintedNode.ObjectMeta)))
+			}).WithTimeout(time.Duration(cluster.GetTaintReplacementTimeSeconds()*3) * time.Second).WithPolling(1 * time.Second).Should(BeNil())
+
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		// This test is flaky because if code is executed slower, the tainted node will be there for too long and its Pod will be removed
+		PIt("should not replace a pod that is on a flapping tainted node", FlakeAttempts(3), func() {
+			// Flapping tainted node
+			tainted := int64(0)
+			for taintTimeOffset := taintKeyMaintenanceDuration * 3; taintTimeOffset >= 0; taintTimeOffset-- {
+				tainted = taintTimeOffset % 2
+				if tainted == 0 {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-1 * time.Second * time.Duration(taintTimeOffset))},
+						},
+					}
+				} else {
+					node.Spec.Taints = []corev1.Taint{}
+				}
+				err = k8sClient.Update(ctx.TODO(), node)
+				Expect(err).NotTo(HaveOccurred())
+				log.Info("Taint node", "Not tainted", tainted, "Node name", podOnTaintedNode.Name, "Node taints", node.Spec.Taints, "Now", time.Now())
+			}
+			Expect(tainted).To(Equal(int64(0))) // ensure last Taint is empty
+			startTime := time.Now()
+
+			result, err := reconcileClusterWithCustomRequeueLimit(cluster, 1)
+			Expect(err).NotTo(HaveOccurred())
+			// pod with any condition is considered as unhealthy, but the pod won't be replaced w/o the NodeTaintReplacing condition
+			Expect(result.Requeue).To(BeTrue())
+
+			Expect(k8sClient.Get(ctx.TODO(), client.ObjectKey{Namespace: cluster.Namespace, Name: cluster.Name}, cluster)).NotTo(HaveOccurred())
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(1))
+			Expect(targetProcessGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected).ProcessGroupConditionType).To(Equal(fdbv1beta2.NodeTaintDetected))
+
+			// Wait long enough to satisfy cluster-wide threshold to replace tainted node
+			time.Sleep(time.Second * time.Duration(cluster.GetTaintReplacementTimeSeconds()))
+
+			// Sanity check test has not take taintKeyMaintenanceDuration seconds to execute
+			Expect(time.Now().Unix() - startTime.Unix()).To(BeNumerically("<", taintKeyMaintenanceDuration))
+			// still should not replace the target process group because it is not marked as failed
+			_, err = reconcileCluster(cluster)
+			Expect(err).NotTo(HaveOccurred())
+			// target pod should not be removed by reconciliation
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+			Expect(getPodByProcessGroupID(cluster, internal.GetProcessClassFromMeta(cluster, podOnTaintedNode.ObjectMeta), internal.GetProcessGroupIDFromMeta(cluster, podOnTaintedNode.ObjectMeta))).NotTo(BeNil())
+		})
+
+		It("should not replace a pod on tainted node when cluster disables taint feature before node is tainted", func() {
+			// Disable taint feature before a node is tainted
+			// TODO: Disable taint feature AFTER a node is tainted
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{}
+			Expect(k8sClient.Update(ctx.TODO(), cluster)).NotTo(HaveOccurred())
+
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			log.Info("Taint node", "Node name", node.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			Eventually(func() int {
+				_, err := reconcileCluster(cluster)
+				Expect(err).NotTo(HaveOccurred())
+				targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+				return len(targetProcessGroupStatus.ProcessGroupConditions)
+			}).WithTimeout(time.Duration(cluster.GetTaintReplacementTimeSeconds()*3) * time.Second).WithPolling(1 * time.Second).Should(Equal(0))
+
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		It("should not replace a pod on tainted node when cluster disables taint feature immediately after node is tainted", func() {
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			log.Info("Taint node", "Node name", node.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{}
+			Expect(k8sClient.Update(ctx.TODO(), cluster)).NotTo(HaveOccurred())
+
+			Eventually(func() []fdbv1beta2.ProcessGroupID {
+				_, err := reconcileCluster(cluster)
+				Expect(err).NotTo(HaveOccurred())
+				// target pod should not be removed by reconciliation;
+				// targetProcessGroupStatus may or may not have its TaintDetected condition updated
+				return (getRemovedProcessGroupIDs(cluster))
+
+			}).WithTimeout(time.Second * time.Duration(cluster.GetTaintReplacementTimeSeconds()*3)).WithPolling(time.Second * time.Duration(1)).Should(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		It("should replace a pod on tainted node when the cluster disables and reenables taint feature", func() {
+			// Disable taint feature before a node is tainted
+			// TODO: Disable taint feature AFTER a node is tainted
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{}
+			Expect(k8sClient.Update(ctx.TODO(), cluster)).NotTo(HaveOccurred())
+
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			log.Info("Taint node", "Node name", node.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			result, err := reconcileCluster(cluster)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(result.Requeue).To(BeFalse())
+			// target pod should not be removed by reconciliation
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+			targetProcessGroupStatus := fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, targetPodProcessGroupID)
+			Expect(len(targetProcessGroupStatus.ProcessGroupConditions)).To(Equal(0))
+
+			// Enable taint feature
+			// Refresh cluster version before we update the cluster again
+			Expect(k8sClient.Get(ctx.TODO(), client.ObjectKey{Namespace: cluster.Namespace, Name: cluster.Name}, cluster)).NotTo(HaveOccurred())
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{
+				{
+					Key:               &taintKeyStar,
+					DurationInSeconds: &taintKeyStarDuration,
+				},
+				{
+					Key:               &taintKeyMaintenance,
+					DurationInSeconds: &taintKeyMaintenanceDuration,
+				},
+			}
+			Expect(k8sClient.Update(ctx.TODO(), cluster)).NotTo(HaveOccurred())
+
+			Eventually(func() *corev1.Pod {
+				_, err = reconcileCluster(cluster)
+				Expect(err).NotTo(HaveOccurred())
+				// target pod should have been removed by reconciliation
+				return getPodByProcessGroupID(cluster, internal.GetProcessClassFromMeta(cluster, podOnTaintedNode.ObjectMeta), internal.GetProcessGroupIDFromMeta(cluster, podOnTaintedNode.ObjectMeta))
+			}).WithTimeout(time.Duration(cluster.GetTaintReplacementTimeSeconds()*3) * time.Second).WithPolling(1 * time.Second).Should(BeNil())
+		})
+
+		It("should replace a pod with NodeTaintReplacing condition when the conditions duration is longer than FailureDetectionTimeSeconds but shorter than TaintReplacementTimeSeconds", func() {
+			// Test FailureDetectionTimeSeconds < TaintReplacementTimeSeconds scenario
+			cluster.Spec.AutomationOptions.Replacements.FailureDetectionTimeSeconds = pointer.Int(1)
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementTimeSeconds = pointer.Int(5)
+			Expect(k8sClient.Update(ctx.TODO(), cluster)).NotTo(HaveOccurred())
+
+			node.Spec.Taints = []corev1.Taint{
+				{
+					Key:       taintKeyMaintenance,
+					Value:     "rack_maintenance",
+					Effect:    corev1.TaintEffectNoExecute,
+					TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+				},
+			}
+			log.Info("Taint node", "Node name", node.Name, "Node taints", node.Spec.Taints, "TaintTime", node.Spec.Taints[0].TimeAdded.Time, "Now", time.Now())
+			Expect(k8sClient.Update(ctx.TODO(), node)).NotTo(HaveOccurred())
+
+			result, err := reconcileClusterWithCustomRequeueLimit(cluster, 1)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(result.Requeue).To(BeTrue())
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+
+			Eventually(func() *corev1.Pod {
+				result, err = reconcileCluster(cluster)
+				Expect(err).NotTo(HaveOccurred())
+				// target pod should have been removed by reconciliation
+				return getPodByProcessGroupID(cluster, internal.GetProcessClassFromMeta(cluster, podOnTaintedNode.ObjectMeta), internal.GetProcessGroupIDFromMeta(cluster, podOnTaintedNode.ObjectMeta))
+			}).WithTimeout(time.Duration(cluster.GetTaintReplacementTimeSeconds()*3) * time.Second).WithPolling(1 * time.Second).Should(BeNil())
+
+			Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+		})
+
+		When("multiple nodes are tainted", func() {
+			var taintedNodes []*corev1.Node
+			var setValidTaint bool
+			BeforeEach(func() {
+				concurrentTaints := 2
+				Expect(len(allPods)).To(BeNumerically(">", concurrentTaints))
+				taintedNodesIndex := map[int]struct{}{}
+				taintedNodes = []*corev1.Node{}
+				var taintKey string
+				var taintTimeAdded *metav1.Time
+				for len(taintedNodesIndex) < concurrentTaints {
+					taintedNodesIndex[rand.Intn(len(allPods))] = struct{}{}
+				}
+				for key := range taintedNodesIndex {
+					curPod := allPods[key]
+					curNode := &corev1.Node{
+						ObjectMeta: metav1.ObjectMeta{Name: curPod.Spec.NodeName},
+					}
+					taintedNodes = append(taintedNodes, curNode)
+				}
+				for i, taintedNode := range taintedNodes {
+					if i%2 == 0 {
+						taintKey = ""
+						taintTimeAdded = &metav1.Time{Time: time.Now()}
+					} else {
+						taintKey = taintKeyMaintenance
+						taintTimeAdded = nil
+					}
+					if setValidTaint {
+						taintKey = taintKeyMaintenance
+						taintTimeAdded = &metav1.Time{Time: time.Now()}
+					}
+					taintedNode.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKey,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: taintTimeAdded,
+						},
+					}
+
+					Expect(k8sClient.Update(ctx.TODO(), taintedNode)).NotTo(HaveOccurred())
+					log.Info("Taint node", "Index", i, "Node name", taintedNode.Name, "Node taints", taintedNode.Spec.Taints)
+				}
+				// Replace all tainted nodes in one reconciliation loop
+				cluster.Spec.AutomationOptions.MaxConcurrentReplacements = &concurrentTaints
+				Expect(k8sClient.Update(ctx.TODO(), cluster)).NotTo(HaveOccurred())
+
+				time.Sleep(time.Second * time.Duration(*cluster.Spec.AutomationOptions.Replacements.TaintReplacementTimeSeconds+1))
+			})
+
+			It("should remove all pods on tainted nodes", func() {
+				retry := len(taintedNodes) * 2 // Hack: Ensure reconciler replaces all tainted nodes
+				for {                          // re-run reconcileCluster up to retry times, assuming each reconciliation replaces at least one pod
+					result, err := reconcileCluster(cluster)
+					Expect(err).NotTo(HaveOccurred())
+					if !result.Requeue || retry <= 0 {
+						break
+					}
+					time.Sleep(time.Microsecond * time.Duration(500)) // Removing this will cause test failure because not all tainted pods are removed
+					retry = retry - 1
+				}
+
+				Expect(getRemovedProcessGroupIDs(cluster)).To(Equal([]fdbv1beta2.ProcessGroupID{}))
+			})
+		})
+
+	})
+
 	Context("with no missing processes", func() {
 		It("should return nil",
 			func() {
@@ -382,8 +950,7 @@ var _ = Describe("replace_failed_process_groups", func() {
 	})
 })
 
-// getRemovedProcessGroupIDs returns a list of ids for the process groups that
-// are marked for removal.
+// getRemovedProcessGroupIDs returns a list of ids for the process groups that are marked for removal.
 func getRemovedProcessGroupIDs(cluster *fdbv1beta2.FoundationDBCluster) []fdbv1beta2.ProcessGroupID {
 	results := make([]fdbv1beta2.ProcessGroupID, 0)
 	for _, processGroupStatus := range cluster.Status.ProcessGroups {
@@ -393,3 +960,18 @@ func getRemovedProcessGroupIDs(cluster *fdbv1beta2.FoundationDBCluster) []fdbv1b
 	}
 	return results
 }
+
+func getPodByProcessGroupID(cluster *fdbv1beta2.FoundationDBCluster, processClass fdbv1beta2.ProcessClass, processGroupID fdbv1beta2.ProcessGroupID) *corev1.Pod {
+	pods, err := clusterReconciler.PodLifecycleManager.GetPods(ctx.TODO(), clusterReconciler, cluster, internal.GetPodListOptions(cluster, processClass, string(processGroupID))...)
+	if err != nil {
+		return nil
+	}
+
+	for _, pod := range pods {
+		if internal.GetProcessGroupIDFromMeta(cluster, pod.ObjectMeta) == processGroupID {
+			return pod
+		}
+	}
+
+	return nil
+}
diff --git a/controllers/suite_test.go b/controllers/suite_test.go
index 7995cc72..3d31598f 100644
--- a/controllers/suite_test.go
+++ b/controllers/suite_test.go
@@ -129,6 +129,10 @@ func reconcileCluster(cluster *fdbv1beta2.FoundationDBCluster) (reconcile.Result
 	return reconcileObject(clusterReconciler, cluster.ObjectMeta, requeueLimit)
 }
 
+func reconcileClusterWithCustomRequeueLimit(cluster *fdbv1beta2.FoundationDBCluster, customRequeueLimit int) (reconcile.Result, error) {
+	return reconcileObject(clusterReconciler, cluster.ObjectMeta, customRequeueLimit)
+}
+
 func reconcileBackup(backup *fdbv1beta2.FoundationDBBackup) (reconcile.Result, error) {
 	return reconcileObject(backupReconciler, backup.ObjectMeta, requeueLimit)
 }
@@ -142,7 +146,7 @@ func reconcileObject(reconciler reconcile.Reconciler, metadata metav1.ObjectMeta
 	result := reconcile.Result{Requeue: true}
 	var err error
 	for result.Requeue && attempts > 0 {
-		log.Info("Running test reconciliation")
+		log.Info("Running test reconciliation", "Attempts", attempts)
 		attempts--
 
 		result, err = reconciler.Reconcile(context.TODO(), reconcile.Request{NamespacedName: types.NamespacedName{Namespace: metadata.Namespace, Name: metadata.Name}})
diff --git a/controllers/update_status_test.go b/controllers/update_status_test.go
index 82f86a15..100b6b8f 100644
--- a/controllers/update_status_test.go
+++ b/controllers/update_status_test.go
@@ -22,9 +22,11 @@ package controllers
 
 import (
 	"context"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"time"
 
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	logf "sigs.k8s.io/controller-runtime/pkg/log"
+
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/fdbadminclient/mock"
 
 	"k8s.io/utils/pointer"
@@ -42,6 +44,249 @@ import (
 
 var _ = Describe("update_status", func() {
 	var storageOneProcessGroupID fdbv1beta2.ProcessGroupID = "storage-1"
+	logger := logf.Log.WithName("testController")
+
+	Context("validate process group on taint node", func() {
+		var cluster *fdbv1beta2.FoundationDBCluster
+		var err error
+		var pod *corev1.Pod                                   // Pod to be tainted
+		var processGroupStatus *fdbv1beta2.ProcessGroupStatus // Target pod's processGroup
+		var node *corev1.Node                                 // Target pod's node
+		var pvc *corev1.PersistentVolumeClaim                 // Target pod's pvc
+		var nodeMap map[string]*corev1.Node
+		taintKeyWildcard := "*"
+		taintKeyWildcardDuration := int64(20)
+		taintKeyMaintenance := "foundationdb/maintenance"
+		taintKeyMaintenanceDuration := int64(5)
+		taintKeyUnhealthy := "foundationdb/unhealthy"
+		taintKeyUnhealthyDuration := int64(11)
+		taintValue := "unhealthy"
+
+		BeforeEach(func() {
+			var pods []*corev1.Pod
+			var allPods []*corev1.Pod
+			nodeMap = make(map[string]*corev1.Node)
+			cluster = internal.CreateDefaultCluster()
+			err = setupClusterForTest(cluster)
+			Expect(err).NotTo(HaveOccurred())
+			// Define cluster's taint policy
+			cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{
+				{
+					Key:               &taintKeyWildcard,
+					DurationInSeconds: &taintKeyWildcardDuration,
+				},
+				{
+					Key:               &taintKeyMaintenance,
+					DurationInSeconds: &taintKeyMaintenanceDuration,
+				},
+			}
+
+			pods, err = clusterReconciler.PodLifecycleManager.GetPods(context.TODO(), clusterReconciler, cluster, internal.GetSinglePodListOptions(cluster, storageOneProcessGroupID)...)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(len(pods)).To(Equal(1))
+
+			pod = pods[0]
+			node = &corev1.Node{
+				ObjectMeta: metav1.ObjectMeta{Name: pod.Spec.NodeName},
+			}
+
+			allPods, err = clusterReconciler.PodLifecycleManager.GetPods(context.TODO(), clusterReconciler, cluster, internal.GetPodListOptions(cluster, "", "")...)
+			Expect(err).NotTo(HaveOccurred())
+
+			allPvcs := &corev1.PersistentVolumeClaimList{}
+			err = clusterReconciler.List(context.TODO(), allPvcs, internal.GetPodListOptions(cluster, "", "")...)
+			Expect(err).NotTo(HaveOccurred())
+
+			processGroupStatus = fdbv1beta2.FindProcessGroupByID(cluster.Status.ProcessGroups, internal.GetProcessGroupIDFromMeta(cluster, pod.ObjectMeta))
+			pvcMap := internal.CreatePVCMap(cluster, allPvcs)
+			pvcValue, pvcExists := pvcMap[processGroupStatus.ProcessGroupID]
+			if pvcExists {
+				pvc = &pvcValue
+			}
+			log.Info("Target processGroupStatus Info", "ProcessGroupID", processGroupStatus.ProcessGroupID,
+				"Conditions size", len(processGroupStatus.ProcessGroupConditions),
+				"Conditions", processGroupStatus.ProcessGroupConditions,
+				"AllPods size", len(allPods))
+		})
+
+		When("process group's node is tainted", func() {
+			JustBeforeEach(func() {
+				log.Info("Taint node", "Node name", pod.Name, "Node taints", node.Spec.Taints)
+				Expect(k8sClient.Update(context.TODO(), node)).NotTo(HaveOccurred())
+
+				err = validateProcessGroup(context.TODO(), clusterReconciler, cluster, pod, pvc, pod.ObjectMeta.Annotations[fdbv1beta2.LastConfigMapKey], processGroupStatus, cluster.IsTaintFeatureDisabled(), nodeMap, logger)
+				Expect(err).NotTo(HaveOccurred())
+			})
+
+			When("the taint was just added", func() {
+				BeforeEach(func() {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now()},
+						},
+					}
+				})
+
+				It("should get NodeTaintDetected condition", func() {
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(1))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected)).NotTo(Equal(nil))
+				})
+			})
+
+			When("the taint was just added long enough to to mark the Pod for replacement", func() {
+				BeforeEach(func() {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+						},
+					}
+				})
+
+				It("should get both NodeTaintDetected and NodeTaintReplacing condition", func() {
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(2))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected)).NotTo(Equal(nil))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing)).NotTo(Equal(nil))
+				})
+			})
+
+			When("disable and enable taint feature", func() {
+				BeforeEach(func() {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+						},
+					}
+					// Disable cluster's taint mechanism
+					cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{}
+				})
+
+				It("should enable cluster taint feature", func() {
+					// Check taint feature is disabled
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(0))
+
+					// Enable taint feature
+					cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{
+						{
+							Key:               &taintKeyWildcard,
+							DurationInSeconds: &taintKeyWildcardDuration,
+						},
+						{
+							Key:               &taintKeyMaintenance,
+							DurationInSeconds: &taintKeyMaintenanceDuration,
+						},
+					}
+
+					err = validateProcessGroup(context.TODO(), clusterReconciler, cluster, pod, pvc, pod.ObjectMeta.Annotations[fdbv1beta2.LastConfigMapKey], processGroupStatus, cluster.IsTaintFeatureDisabled(), nodeMap, logger)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(2))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected)).NotTo(Equal(nil))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing)).NotTo(Equal(nil))
+				})
+			})
+
+			When("cluster disables taint feature but node has wildcard taint", func() {
+				BeforeEach(func() {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyWildcard,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyWildcardDuration+1))},
+						},
+					}
+					// Disable cluster's taint mechanism
+					cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{}
+				})
+
+				It("should disable taint feature", func() {
+					Expect(err).NotTo(HaveOccurred())
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(0))
+				})
+			})
+
+			When("node is tainted with a key different from cluster configured taint key", func() {
+				BeforeEach(func() {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyUnhealthy,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyUnhealthyDuration+1))},
+						},
+					}
+					// Only handle taintKeyMaintenance taint key
+					cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{
+						{
+							Key:               &taintKeyMaintenance,
+							DurationInSeconds: &taintKeyMaintenanceDuration,
+						},
+					}
+				})
+
+				It("pods on the node should not be tainted", func() {
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(0))
+					// Add taintKeyUnhealthy to handle the node's taint key
+					cluster.Spec.AutomationOptions.Replacements.TaintReplacementOptions = []fdbv1beta2.TaintReplacementOption{
+						{
+							Key:               &taintKeyMaintenance,
+							DurationInSeconds: &taintKeyMaintenanceDuration,
+						},
+						{
+							Key:               &taintKeyUnhealthy,
+							DurationInSeconds: &taintKeyUnhealthyDuration,
+						},
+					}
+
+					err = validateProcessGroup(context.TODO(), clusterReconciler, cluster, pod, pvc, pod.ObjectMeta.Annotations[fdbv1beta2.LastConfigMapKey], processGroupStatus, cluster.IsTaintFeatureDisabled(), nodeMap, logger)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(2))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected)).NotTo(BeNil())
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing)).NotTo(BeNil())
+				})
+			})
+
+			When("node taint is removed", func() {
+				// Taint key is removed from tainted node. NodeTaintDetected condition should be removed, but NodeTaintReplacing condition still exist
+				BeforeEach(func() {
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     taintValue,
+							Effect:    corev1.TaintEffectNoExecute,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+						},
+					}
+				})
+
+				It("should keep NodeTaintReplacing condition", func() {
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(2))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected)).NotTo(BeNil())
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing)).NotTo(BeNil())
+
+					// Remove the node taint
+					node.Spec.Taints = []corev1.Taint{}
+					err = k8sClient.Update(context.TODO(), node)
+					Expect(err).NotTo(HaveOccurred())
+					nodeMap = make(map[string]*corev1.Node)
+					log.Info("Remove node taint", "Node name", pod.Name, "Node taints", node.Spec.Taints, "Now", time.Now())
+
+					err = validateProcessGroup(context.TODO(), clusterReconciler, cluster, pod, pvc, pod.ObjectMeta.Annotations[fdbv1beta2.LastConfigMapKey], processGroupStatus, cluster.IsTaintFeatureDisabled(), nodeMap, logger)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(1))
+					Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintReplacing)).NotTo(Equal(nil))
+				})
+			})
+		})
+	})
 
 	Context("validate process group", func() {
 		var cluster *fdbv1beta2.FoundationDBCluster
@@ -52,8 +297,10 @@ var _ = Describe("update_status", func() {
 		var err error
 		var allPods []*corev1.Pod
 		var allPvcs *corev1.PersistentVolumeClaimList
+		var nodeMap map[string]*corev1.Node
 
 		BeforeEach(func() {
+			nodeMap = make(map[string]*corev1.Node)
 			cluster = internal.CreateDefaultCluster()
 			Expect(setupClusterForTest(cluster)).NotTo(HaveOccurred())
 
@@ -99,7 +346,7 @@ var _ = Describe("update_status", func() {
 				processGroupStatus := fdbv1beta2.NewProcessGroupStatus("storage-1337", fdbv1beta2.ProcessClassStorage, []string{"1.1.1.1"})
 				// Reset the status to only tests for the missing Pod
 				processGroupStatus.ProcessGroupConditions = []*fdbv1beta2.ProcessGroupCondition{}
-				Expect(validateProcessGroup(context.TODO(), clusterReconciler, cluster, nil, nil, "", processGroupStatus)).NotTo(HaveOccurred())
+				Expect(validateProcessGroup(context.TODO(), clusterReconciler, cluster, nil, nil, "", processGroupStatus, cluster.IsTaintFeatureDisabled(), nodeMap, logger)).NotTo(HaveOccurred())
 				Expect(len(processGroupStatus.ProcessGroupConditions)).To(Equal(1))
 				Expect(processGroupStatus.ProcessGroupConditions[0].ProcessGroupConditionType).To(Equal(fdbv1beta2.MissingPod))
 			})
@@ -107,7 +354,7 @@ var _ = Describe("update_status", func() {
 
 		When("a process group is fine", func() {
 			It("should not get any condition assigned", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 				Expect(len(processGroupStatus)).To(BeNumerically(">", 4))
 				processGroup := processGroupStatus[len(processGroupStatus)-4]
@@ -122,7 +369,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should get a condition assigned", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				missingProcesses := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.MissingPod, false)
@@ -141,7 +388,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should get a condition assigned", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				incorrectProcesses := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.IncorrectCommandLine, false)
@@ -159,7 +406,7 @@ var _ = Describe("update_status", func() {
 				})
 
 				It("should get a condition assigned", func() {
-					processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+					processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 					Expect(err).NotTo(HaveOccurred())
 
 					incorrectProcesses := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.IncorrectCommandLine, false)
@@ -180,7 +427,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should get a condition assigned", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				missingProcesses := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.MissingProcesses, false)
@@ -201,7 +448,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should get a condition assigned", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				incorrectPods := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.IncorrectPodSpec, false)
@@ -227,7 +474,7 @@ var _ = Describe("update_status", func() {
 			It("should get a condition assigned", func() {
 				// Ensure that we have the same number of Pods and processes
 				Expect(allPods).To(HaveLen(len(processMap)))
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				missingProcesses := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.PodFailing, false)
@@ -248,7 +495,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should get a condition assigned", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				failingPods := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.PodFailing, false)
@@ -269,7 +516,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should get a condition assigned", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				failingPods := fdbv1beta2.FilterByCondition(processGroupStatus, fdbv1beta2.PodFailing, false)
@@ -294,7 +541,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should mark the process group for removal", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				removalCount := 0
@@ -325,7 +572,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should be mark the process group for removal without exclusion", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				removalCount := 0
@@ -365,7 +612,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should mark the process group as unreachable", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				unreachableCount := 0
@@ -389,7 +636,7 @@ var _ = Describe("update_status", func() {
 				})
 
 				It("should remove the condition", func() {
-					processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+					processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 					Expect(err).NotTo(HaveOccurred())
 
 					unreachableCount := 0
@@ -417,7 +664,7 @@ var _ = Describe("update_status", func() {
 			})
 
 			It("should mark the process group as Pod pending", func() {
-				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs)
+				processGroupStatus, err := validateProcessGroups(context.TODO(), clusterReconciler, cluster, &cluster.Status, processMap, configMap, allPods, allPvcs, logger)
 				Expect(err).NotTo(HaveOccurred())
 
 				pendingCount := 0
diff --git a/e2e/fixtures/factory.go b/e2e/fixtures/factory.go
index 143d946a..0f48727e 100644
--- a/e2e/fixtures/factory.go
+++ b/e2e/fixtures/factory.go
@@ -764,3 +764,23 @@ func (factory *Factory) GetSidecarImage() string {
 func (factory *Factory) GetFoundationDBImage() string {
 	return prependRegistry(factory.options.registry, factory.options.fdbImage)
 }
+
+// UpdateNode update node definition
+func (fdbCluster *FdbCluster) UpdateNode(node *corev1.Node) {
+	gomega.Eventually(func() bool {
+		err := fdbCluster.getClient().Update(ctx.Background(), node)
+		return err == nil
+	}).WithTimeout(time.Duration(2) * time.Minute).WithPolling(2 * time.Second).Should(gomega.BeTrue())
+}
+
+// GetNode return Node with the given name
+func (fdbCluster *FdbCluster) GetNode(name string) *corev1.Node {
+	// Retry if for some reasons an error is returned
+	node := &corev1.Node{}
+	gomega.Eventually(func() error {
+		return fdbCluster.getClient().
+			Get(ctx.TODO(), client.ObjectKey{Name: name}, node)
+	}).WithTimeout(2 * time.Minute).WithPolling(1 * time.Second).ShouldNot(gomega.HaveOccurred())
+
+	return node
+}
diff --git a/e2e/fixtures/fdb_cluster.go b/e2e/fixtures/fdb_cluster.go
index 1339318d..4de582c2 100644
--- a/e2e/fixtures/fdb_cluster.go
+++ b/e2e/fixtures/fdb_cluster.go
@@ -364,7 +364,7 @@ func (fdbCluster *FdbCluster) SetDatabaseConfiguration(
 	return fdbCluster.WaitForReconciliation()
 }
 
-// UpdateClusterSpec ensures that the FoundationDBCluster will be updated in Kubernetes. This method as a retry mechanism
+// UpdateClusterSpec ensures that the FoundationDBCluster will be updated in Kubernetes. This method has a retry mechanism
 // implemented and ensures that the provided (local) Spec matches the Spec in Kubernetes.
 func (fdbCluster *FdbCluster) UpdateClusterSpec() {
 	fdbCluster.UpdateClusterSpecWithSpec(fdbCluster.cluster.Spec.DeepCopy())
@@ -394,6 +394,8 @@ func (fdbCluster *FdbCluster) UpdateClusterSpecWithSpec(desiredSpec *fdbv1beta2.
 
 		specUpdated := equality.Semantic.DeepEqual(fetchedCluster.Spec, *desiredSpec)
 		log.Println("UpdateClusterSpec: specUpdated:", specUpdated)
+		log.Printf("FetchedSpec %+v\n", fetchedCluster.Spec)
+		log.Printf("DesiredSpec %+v\n", desiredSpec)
 		if specUpdated {
 			return true
 		}
@@ -405,7 +407,7 @@ func (fdbCluster *FdbCluster) UpdateClusterSpecWithSpec(desiredSpec *fdbv1beta2.
 		}
 		// Retry here and let the method fetch the latest version of the cluster again until the spec is updated.
 		return false
-	}).WithTimeout(1 * time.Minute).WithPolling(1 * time.Second).Should(gomega.BeTrue())
+	}).WithTimeout(10 * time.Minute).WithPolling(1 * time.Second).Should(gomega.BeTrue())
 
 	fdbCluster.cluster = fetchedCluster
 }
@@ -978,6 +980,14 @@ func (fdbCluster *FdbCluster) SetEmptyMonitorConf(enable bool) error {
 	return nil
 }
 
+// SetClusterTaintConfig set fdbCluster's TaintReplacementOptions
+func (fdbCluster *FdbCluster) SetClusterTaintConfig(taintOption []fdbv1beta2.TaintReplacementOption, taintReplacementTimeSeconds *int) {
+	curClusterSpec := fdbCluster.GetCluster().Spec.DeepCopy()
+	curClusterSpec.AutomationOptions.Replacements.TaintReplacementOptions = taintOption
+	curClusterSpec.AutomationOptions.Replacements.TaintReplacementTimeSeconds = taintReplacementTimeSeconds
+	fdbCluster.UpdateClusterSpecWithSpec(curClusterSpec)
+}
+
 // GetProcessCounts returns the process counts of the current FoundationDBCluster.
 func (fdbCluster *FdbCluster) GetProcessCounts() (fdbv1beta2.ProcessCounts, error) {
 	return fdbCluster.cluster.GetProcessCountsWithDefaults()
@@ -1016,21 +1026,33 @@ func (fdbCluster *FdbCluster) GetCustomParameters(
 	return fdbCluster.cluster.Spec.Processes[processClass].CustomParameters
 }
 
-// EnsurePodIsDeleted validates that a Pod is either not existing or is marked as deleted with a non-zero deletion timestamp.
-func (fdbCluster *FdbCluster) EnsurePodIsDeleted(podName string) {
-	gomega.Eventually(func() bool {
-		pod := &corev1.Pod{}
-		err := fdbCluster.getClient().
-			Get(ctx.TODO(), client.ObjectKey{Namespace: fdbCluster.Namespace(), Name: podName}, pod)
+// CheckPodIsDeleted return true if Pod no longer exists at the executed time point
+func (fdbCluster *FdbCluster) CheckPodIsDeleted(podName string) bool {
+	pod := &corev1.Pod{}
+	err := fdbCluster.getClient().
+		Get(ctx.TODO(), client.ObjectKey{Namespace: fdbCluster.Namespace(), Name: podName}, pod)
 
-		log.Println("error: ", err, "pod", pod.ObjectMeta)
-		if err != nil {
-			return kubeErrors.IsNotFound(err)
+	log.Println("error: ", err, "pod", pod.ObjectMeta)
+	if err != nil {
+		if kubeErrors.IsNotFound(err) {
+			return true
 		}
+	}
 
-		// For our case it's enough to validate that the Pod is marked for deletion.
-		return !pod.DeletionTimestamp.IsZero()
-	}).WithTimeout(5 * time.Minute).WithPolling(1 * time.Second).Should(gomega.BeTrue())
+	return !pod.DeletionTimestamp.IsZero()
+}
+
+// EnsurePodIsDeletedWithCustomTimeout validates that a Pod is either not existing or is marked as deleted with a non-zero deletion timestamp.
+// It times out after timeoutMinutes.
+func (fdbCluster *FdbCluster) EnsurePodIsDeletedWithCustomTimeout(podName string, timeoutMinutes int) {
+	gomega.Eventually(func() bool {
+		return fdbCluster.CheckPodIsDeleted(podName)
+	}).WithTimeout(time.Duration(timeoutMinutes) * time.Minute).WithPolling(1 * time.Second).Should(gomega.BeTrue())
+}
+
+// EnsurePodIsDeleted validates that a Pod is either not existing or is marked as deleted with a non-zero deletion timestamp.
+func (fdbCluster *FdbCluster) EnsurePodIsDeleted(podName string) {
+	fdbCluster.EnsurePodIsDeletedWithCustomTimeout(podName, 5)
 }
 
 // SetUseDNSInClusterFile enables DNS in the cluster file. Enable this setting to use DNS instead of IP addresses in
@@ -1125,3 +1147,8 @@ func (fdbCluster *FdbCluster) SetBuggifyBlockRemoval(blockRemovals []fdbv1beta2.
 	fdbCluster.cluster.Spec.Buggify.BlockRemoval = blockRemovals
 	fdbCluster.UpdateClusterSpec()
 }
+
+// GetAutomationOptions return the fdbCluster's AutomationOptions
+func (fdbCluster *FdbCluster) GetAutomationOptions() fdbv1beta2.FoundationDBClusterAutomationOptions {
+	return fdbCluster.cluster.Spec.AutomationOptions
+}
diff --git a/e2e/fixtures/fdb_cluster_specs.go b/e2e/fixtures/fdb_cluster_specs.go
index 250ee5b1..d636fb5d 100644
--- a/e2e/fixtures/fdb_cluster_specs.go
+++ b/e2e/fixtures/fdb_cluster_specs.go
@@ -56,13 +56,15 @@ func (factory *Factory) createFDBClusterSpec(
 				Key: "foundationdb.org/none",
 			},
 			AutomationOptions: fdbv1beta2.FoundationDBClusterAutomationOptions{
-				// We have to wait long enough to ensure the operator is not recreating to many Pods at the same time.
+				// We have to wait long enough to ensure the operator is not recreating too many Pods at the same time.
 				WaitBetweenRemovalsSeconds: pointer.Int(0),
 				Replacements: fdbv1beta2.AutomaticReplacementOptions{
 					Enabled: pointer.Bool(true),
 					// Setting this to 5 minutes is reasonable to prevent the operator recreating Pods when they wait for
 					// new ec2 instances.
 					FailureDetectionTimeSeconds: pointer.Int(300),
+					// Setting TaintReplacementTimeSeconds as half of FailureDetectionTimeSeconds to make taint replacement faster
+					TaintReplacementTimeSeconds: pointer.Int(150),
 					MaxConcurrentReplacements:   pointer.Int(2),
 				},
 				// Allow the operator to remove all Pods that are excluded and marked for deletion to remove at once.
diff --git a/e2e/fixtures/fdb_operator_client.go b/e2e/fixtures/fdb_operator_client.go
index 2ab0f260..21834729 100644
--- a/e2e/fixtures/fdb_operator_client.go
+++ b/e2e/fixtures/fdb_operator_client.go
@@ -165,6 +165,33 @@ subjects:
 - kind: ServiceAccount
   name: fdb-kubernetes-operator-controller-manager
 ---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: {{ .Namespace }}-operator-manager-clusterrole
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - nodes
+  verbs:
+  - get
+  - watch
+  - list
+---
+  apiVersion: rbac.authorization.k8s.io/v1
+  kind: ClusterRoleBinding
+  metadata:
+    name: {{ .Namespace }}-operator-manager-clusterrolebinding
+  roleRef:
+    apiGroup: rbac.authorization.k8s.io
+    kind: ClusterRole
+    name: {{ .Namespace }}-operator-manager-clusterrole
+  subjects:
+  - kind: ServiceAccount
+    name: fdb-kubernetes-operator-controller-manager
+    namespace: {{ .Namespace }}
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
diff --git a/e2e/test_operator/operator_test.go b/e2e/test_operator/operator_test.go
index d4ccb204..fe269004 100644
--- a/e2e/test_operator/operator_test.go
+++ b/e2e/test_operator/operator_test.go
@@ -37,11 +37,13 @@ import (
 
 	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
 	"github.com/FoundationDB/fdb-kubernetes-operator/e2e/fixtures"
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
 	chaosmesh "github.com/chaos-mesh/chaos-mesh/api/v1alpha1"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/utils/pointer"
 )
 
@@ -91,7 +93,7 @@ var _ = BeforeSuite(func() {
 	if factory.ChaosTestsEnabled() {
 		factory.ScheduleInjectPodKill(
 			fixtures.GetOperatorSelector(fdbCluster.Namespace()),
-			"*/1 * * * *",
+			"*/2 * * * *",
 			chaosmesh.OneMode,
 		)
 	}
@@ -124,6 +126,242 @@ var _ = Describe("Operator", Label("e2e"), func() {
 		}
 	})
 
+	PWhen("nodes in the cluster are tainted", func() {
+		taintKeyMaintenance := "maintenance"
+		taintKeyMaintenanceDuration := int64(2)
+		taintKeyStar := "*"
+		taintKeyStarDuration := int64(5)
+		var initialPods *corev1.PodList
+		var taintedNode *corev1.Node
+		var taintedNodes []*corev1.Node
+		historyTaintedNodes := make(map[string]fdbv1beta2.None)
+		numNodesTainted := 2 // TODO: Change to higher number 5
+		ensurePodIsDeletedTimeoutMinutes := 20
+
+		BeforeEach(func() {
+			// Cleanup tainted nodes in case previous test fails which will leave tainted nodes behind
+			for nodeName := range historyTaintedNodes {
+				log.Printf("BeforeEach Reset node as untainted:%s\n", nodeName)
+				node := fdbCluster.GetNode(nodeName)
+				node.Spec.Taints = []corev1.Taint{}
+				fdbCluster.UpdateNode(node)
+			}
+
+			initialPods = fdbCluster.GetStatelessPods()
+
+			// Setup cluster's taint config and wait for long enough
+			fdbCluster.SetClusterTaintConfig([]fdbv1beta2.TaintReplacementOption{
+				{
+					Key:               &taintKeyMaintenance,
+					DurationInSeconds: &taintKeyMaintenanceDuration,
+				},
+			}, pointer.Int(1))
+
+			Expect(len(fdbCluster.GetCluster().Spec.AutomationOptions.Replacements.TaintReplacementOptions)).To(BeNumerically(">=", 1))
+
+			Expect(*fdbCluster.GetAutomationOptions().Replacements.Enabled).To(BeTrue())
+
+		})
+
+		AfterEach(func() {
+			Expect(fdbCluster.ClearProcessGroupsToRemove()).NotTo(HaveOccurred())
+			// Reset taint related options to default value in e2e test
+			fdbCluster.SetClusterTaintConfig([]fdbv1beta2.TaintReplacementOption{}, pointer.Int(150))
+			// untaint the nodes
+			log.Printf("AfterEach Cleanup: Untaint the single node:%s\n", taintedNode.Name)
+			historyTaintedNodes[taintedNode.Name] = fdbv1beta2.None{}
+			taintedNode = fdbCluster.GetNode(taintedNode.Name)
+			taintedNode.Spec.Taints = []corev1.Taint{}
+			fdbCluster.UpdateNode(taintedNode)
+			for i, node := range taintedNodes {
+				historyTaintedNodes[node.Name] = fdbv1beta2.None{}
+				log.Printf("Cleanup: Untaint the %dth node:%s\n", i, node.Name)
+				node = fdbCluster.GetNode(node.Name)
+				node.Spec.Taints = []corev1.Taint{}
+				fdbCluster.UpdateNode(node)
+			}
+			taintedNodes = []*corev1.Node{}
+		})
+
+		When("cluster enables taint feature on focused maintenance taint key", func() {
+			It("should remove the targeted Pod whose Node is tainted", func() {
+				replacedPod := fixtures.RandomPickOnePod(initialPods.Items)
+				// Taint replacePod's node
+				taintedNode = fdbCluster.GetNode(replacedPod.Spec.NodeName)
+				taintedNode.Spec.Taints = []corev1.Taint{
+					{
+						Key:       taintKeyMaintenance,
+						Value:     "rack_maintenance",
+						Effect:    corev1.TaintEffectPreferNoSchedule,
+						TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+					},
+				}
+				log.Printf("Taint node: Pod name:%s Node name:%s Node taints:%+v TaintTime:%+v Now:%+v\n", replacedPod.Name,
+					taintedNode.Name, taintedNode.Spec.Taints, taintedNode.Spec.Taints[0].TimeAdded.Time, time.Now())
+				fdbCluster.UpdateNode(taintedNode)
+
+				err := fdbCluster.WaitForReconciliation()
+				Expect(err).NotTo(HaveOccurred())
+
+				fdbCluster.EnsurePodIsDeletedWithCustomTimeout(replacedPod.Name, ensurePodIsDeletedTimeoutMinutes)
+			})
+
+			It("should remove all Pods whose Nodes are tainted", func() {
+				replacedPods := fixtures.RandomPickPod(initialPods.Items, numNodesTainted)
+				for _, pod := range replacedPods {
+					// Taint replacePod's node
+					node := fdbCluster.GetNode(pod.Spec.NodeName)
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectPreferNoSchedule,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Millisecond * time.Duration(taintKeyMaintenanceDuration*1000+int64(rand.Intn(1000))))},
+						},
+					}
+					log.Printf("Taint node: Pod name:%s Node name:%s Node taints:%+v TaintTime:%+v Now:%+v\n", pod.Name,
+						node.Name, node.Spec.Taints, node.Spec.Taints[0].TimeAdded.Time, time.Now())
+					fdbCluster.UpdateNode(node)
+					taintedNodes = append(taintedNodes, node)
+				}
+
+				err := fdbCluster.WaitForReconciliation()
+				Expect(err).NotTo(HaveOccurred())
+
+				for i, pod := range replacedPods {
+					log.Printf("Ensure %dth pod:%s is deleted\n", i, pod.Name)
+					fdbCluster.EnsurePodIsDeletedWithCustomTimeout(pod.Name, ensurePodIsDeletedTimeoutMinutes)
+				}
+			})
+		})
+
+		When("cluster enables taint feature on star taint key", func() {
+			taintKeyMaintenanceDuration = int64(1)
+			taintKeyStar = "*"
+			taintKeyStarDuration = int64(5)
+
+			BeforeEach(func() {
+				// Custom TaintReplacementOptions to taintKeyStar
+				fdbCluster.SetClusterTaintConfig([]fdbv1beta2.TaintReplacementOption{
+					{
+						Key:               &taintKeyStar,
+						DurationInSeconds: &taintKeyStarDuration,
+					},
+				}, pointer.Int(1))
+
+				automationOptions := fdbCluster.GetAutomationOptions()
+				Expect(len(automationOptions.Replacements.TaintReplacementOptions)).To(BeNumerically(">=", 1))
+				Expect(*automationOptions.Replacements.Enabled).To(BeTrue())
+
+			})
+
+			It("should eventually remove the targeted Pod whose Node is tainted", func() {
+				replacedPod := fixtures.RandomPickOnePod(initialPods.Items)
+				// Taint replacePod's node
+				node := fdbCluster.GetNode(replacedPod.Spec.NodeName)
+				node.Spec.Taints = []corev1.Taint{
+					{
+						Key:       taintKeyMaintenance,
+						Value:     "rack_maintenance",
+						Effect:    corev1.TaintEffectPreferNoSchedule,
+						TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Second * time.Duration(taintKeyMaintenanceDuration+1))},
+					},
+				}
+				log.Printf("Taint node: Pod name:%s Node name:%s Node taints:%+v TaintTime:%+v Now:%+v\n", replacedPod.Name,
+					node.Name, node.Spec.Taints, node.Spec.Taints[0].TimeAdded.Time, time.Now())
+				fdbCluster.UpdateNode(node)
+
+				err := fdbCluster.WaitForReconciliation()
+				Expect(err).NotTo(HaveOccurred())
+
+				fdbCluster.EnsurePodIsDeletedWithCustomTimeout(replacedPod.Name, ensurePodIsDeletedTimeoutMinutes)
+			})
+
+			It("should eventually remove all Pods whose Nodes are tainted", func() {
+				replacedPods := fixtures.RandomPickPod(initialPods.Items, numNodesTainted)
+				for _, pod := range replacedPods {
+					// Taint replacePod's node
+					node := fdbCluster.GetNode(pod.Spec.NodeName)
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectPreferNoSchedule,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Millisecond * time.Duration(taintKeyStarDuration*1000+int64(rand.Intn(1000))))},
+						},
+					}
+					log.Printf("Taint node: Pod name:%s Node name:%s Node taints:%+v TaintTime:%+v Now:%+v\n", pod.Name,
+						node.Name, node.Spec.Taints, node.Spec.Taints[0].TimeAdded.Time, time.Now())
+					fdbCluster.UpdateNode(node)
+				}
+
+				err := fdbCluster.WaitForReconciliation()
+				Expect(err).NotTo(HaveOccurred())
+
+				for i, pod := range replacedPods {
+					log.Printf("Ensure %dth pod:%s is deleted\n", i, pod.Name)
+					fdbCluster.EnsurePodIsDeletedWithCustomTimeout(pod.Name, ensurePodIsDeletedTimeoutMinutes)
+				}
+			})
+		})
+
+		When("cluster disables taint feature with empty taint option", func() {
+			taintKeyMaintenanceDuration = int64(1)
+			taintKeyStar = "*"
+
+			BeforeEach(func() {
+				// Custom TaintReplacementOptions to taintKeyStar
+				fdbCluster.SetClusterTaintConfig([]fdbv1beta2.TaintReplacementOption{}, pointer.Int(1))
+
+				automationOptions := fdbCluster.GetAutomationOptions()
+				Expect(len(automationOptions.Replacements.TaintReplacementOptions)).To(BeNumerically("==", 0))
+				Expect(*automationOptions.Replacements.Enabled).To(BeTrue())
+
+			})
+
+			It("should not remove any Pod whose Nodes are tainted", func() {
+				targetPods := fixtures.RandomPickPod(initialPods.Items, numNodesTainted)
+				var targetPodProcessGroupIDs []fdbv1beta2.ProcessGroupID
+				for _, pod := range targetPods {
+					targetPodProcessGroupIDs = append(targetPodProcessGroupIDs, internal.GetProcessGroupIDFromMeta(fdbCluster.GetCluster(), pod.ObjectMeta))
+					// Taint replacePod's node
+					node := fdbCluster.GetNode(pod.Spec.NodeName)
+					node.Spec.Taints = []corev1.Taint{
+						{
+							Key:       taintKeyMaintenance,
+							Value:     "rack_maintenance",
+							Effect:    corev1.TaintEffectPreferNoSchedule,
+							TimeAdded: &metav1.Time{Time: time.Now().Add(-time.Millisecond * time.Duration(taintKeyStarDuration*1000+int64(rand.Intn(1000))))},
+						},
+					}
+					log.Printf("Taint node: Pod name:%s Node name:%s Node taints:%+v TaintTime:%+v Now:%+v\n", pod.Name,
+						node.Name, node.Spec.Taints, node.Spec.Taints[0].TimeAdded.Time, time.Now())
+					fdbCluster.UpdateNode(node)
+				}
+
+				Consistently(func() bool {
+					for _, groupID := range targetPodProcessGroupIDs {
+						processGroupStatus := fdbv1beta2.FindProcessGroupByID(fdbCluster.GetCluster().Status.ProcessGroups, groupID)
+						Expect(processGroupStatus).NotTo(BeNil())
+						Expect(processGroupStatus.GetCondition(fdbv1beta2.NodeTaintDetected)).To(BeNil())
+					}
+					return true
+				}).WithTimeout(time.Duration(*fdbCluster.GetAutomationOptions().Replacements.TaintReplacementTimeSeconds-1) * time.Second).WithPolling(1 * time.Second).Should(BeTrue())
+
+				// Wait for operator to replace the pod
+				time.Sleep(time.Second * time.Duration(*fdbCluster.GetAutomationOptions().Replacements.TaintReplacementTimeSeconds+1))
+				err := fdbCluster.WaitForReconciliation()
+				Expect(err).NotTo(HaveOccurred())
+
+				for i, pod := range targetPods {
+					log.Printf("Ensure %dth pod:%s is not deleted\n", i, pod.Name)
+					deleted := fdbCluster.CheckPodIsDeleted(pod.Name)
+					Expect(deleted).To(BeFalse())
+				}
+			})
+		})
+	})
+
 	When("replacing a Pod", func() {
 		var replacedPod corev1.Pod
 
diff --git a/e2e/test_operator_ha_upgrades/operator_ha_upgrade_test.go b/e2e/test_operator_ha_upgrades/operator_ha_upgrade_test.go
index 7aefc004..effa4377 100644
--- a/e2e/test_operator_ha_upgrades/operator_ha_upgrade_test.go
+++ b/e2e/test_operator_ha_upgrades/operator_ha_upgrade_test.go
@@ -389,4 +389,49 @@ var _ = Describe("Operator HA Upgrades", Label("e2e"), func() {
 		fixtures.GenerateUpgradeTableEntries(testOptions),
 	)
 
+	DescribeTable(
+		"upgrading a cluster when no remote storage processes are restarted",
+		func(beforeVersion string, targetVersion string) {
+			isAtLeast := factory.OperatorIsAtLeast(
+				"v1.14.0",
+			)
+
+			if !isAtLeast {
+				Skip("operator doesn't support feature for test case")
+			}
+
+			clusterSetup(beforeVersion, false)
+
+			// Select remote storage processes and use the buggify option to skip those
+			// processes during the restart command.
+			storagePods := fdbCluster.GetRemote().GetStoragePods()
+			Expect(storagePods.Items).NotTo(BeEmpty())
+
+			ignoreDuringRestart := make(
+				[]fdbv1beta2.ProcessGroupID,
+				0,
+				len(storagePods.Items),
+			)
+
+			for _, pod := range storagePods.Items {
+				ignoreDuringRestart = append(
+					ignoreDuringRestart,
+					fdbv1beta2.ProcessGroupID(pod.Labels[fdbCluster.GetRemote().GetCachedCluster().GetProcessGroupIDLabel()]),
+				)
+			}
+
+			log.Println(
+				"Selected Pods:",
+				ignoreDuringRestart,
+				"to be skipped during the restart",
+			)
+			fdbCluster.GetRemote().SetIgnoreDuringRestart(ignoreDuringRestart)
+
+			// The cluster should still be able to upgrade.
+			Expect(fdbCluster.UpgradeCluster(targetVersion, true)).NotTo(HaveOccurred())
+		},
+		EntryDescription("Upgrade from %[1]s to %[2]s when no remote storage processes are restarted"),
+		fixtures.GenerateUpgradeTableEntries(testOptions),
+	)
+
 })
diff --git a/e2e/test_operator_upgrades/operator_upgrades_test.go b/e2e/test_operator_upgrades/operator_upgrades_test.go
index c3fb35ce..c636670b 100644
--- a/e2e/test_operator_upgrades/operator_upgrades_test.go
+++ b/e2e/test_operator_upgrades/operator_upgrades_test.go
@@ -1027,4 +1027,18 @@ var _ = Describe("Operator Upgrades", Label("e2e"), func() {
 		EntryDescription("Upgrade from %[1]s to %[2]s"),
 		fixtures.GenerateUpgradeTableEntries(testOptions),
 	)
+
+	DescribeTable(
+		"upgrading a cluster with a pending pod",
+		func(beforeVersion string, targetVersion string) {
+			clusterSetup(beforeVersion, true)
+			pendingPod := fixtures.RandomPickOnePod(fdbCluster.GetPods().Items)
+			// Set the pod in pending state.
+			Expect(fdbCluster.SetPodAsUnschedulable(pendingPod)).NotTo(HaveOccurred())
+			upgradeAndVerify(fdbCluster, targetVersion)
+		},
+		EntryDescription("Upgrade from %[1]s to %[2]s with a pending pod"),
+		fixtures.GenerateUpgradeTableEntries(testOptions),
+	)
+
 })
diff --git a/fdbclient/admin_client_test.go b/fdbclient/admin_client_test.go
index aa403e36..e62af1a5 100644
--- a/fdbclient/admin_client_test.go
+++ b/fdbclient/admin_client_test.go
@@ -118,6 +118,41 @@ var _ = Describe("admin_client_test", func() {
 				nil,
 				[]fdbv1beta2.ProcessAddress{addr5},
 			),
+			Entry("when the process is excluded but the cluster status has multiple generations",
+				&fdbv1beta2.FoundationDBStatus{
+					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+						RecoveryState: fdbv1beta2.RecoveryState{
+							ActiveGenerations: 2,
+						},
+						Processes: map[fdbv1beta2.ProcessGroupID]fdbv1beta2.FoundationDBStatusProcessInfo{
+							"1": {
+								Address:  addr1,
+								Excluded: true,
+							},
+							"2": {
+								Address: addr2,
+							},
+							"3": {
+								Address: addr3,
+							},
+							"4": {
+								Address:  addr4,
+								Excluded: true,
+								Roles: []fdbv1beta2.FoundationDBStatusProcessRoleInfo{
+									{
+										Role: "tester",
+									},
+								},
+							},
+						},
+					},
+				},
+				[]fdbv1beta2.ProcessAddress{addr4},
+				nil,
+				[]fdbv1beta2.ProcessAddress{addr4},
+				nil,
+				nil,
+			),
 		)
 	})
 
diff --git a/internal/replacements/replacements_test.go b/internal/replacements/replacements_test.go
index 01a309da..02aefcf3 100644
--- a/internal/replacements/replacements_test.go
+++ b/internal/replacements/replacements_test.go
@@ -24,6 +24,7 @@ import (
 	"fmt"
 
 	logf "sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/controller-runtime/pkg/log/zap"
 
 	"k8s.io/utils/pointer"
 
@@ -41,6 +42,7 @@ import (
 var _ = Describe("replace_misconfigured_pods", func() {
 	var cluster *fdbv1beta2.FoundationDBCluster
 	var log logr.Logger
+	logf.SetLogger(zap.New(zap.UseDevMode(true), zap.WriteTo(GinkgoWriter)))
 
 	BeforeEach(func() {
 		log = logf.Log.WithName("replacements")
@@ -85,7 +87,6 @@ var _ = Describe("replace_misconfigured_pods", func() {
 			Expect(err).NotTo(HaveOccurred())
 
 			pod.Spec = *spec
-
 			err = internal.NormalizeClusterSpec(cluster, internal.DeprecationOptions{})
 			Expect(err).NotTo(HaveOccurred())
 		})
diff --git a/internal/test_helper.go b/internal/test_helper.go
index 637d47f2..4c9474fe 100644
--- a/internal/test_helper.go
+++ b/internal/test_helper.go
@@ -47,6 +47,7 @@ func CreateDefaultCluster() *fdbv1beta2.FoundationDBCluster {
 				Replacements: fdbv1beta2.AutomaticReplacementOptions{
 					Enabled:                     pointer.Bool(true),
 					FailureDetectionTimeSeconds: pointer.Int(1),
+					TaintReplacementTimeSeconds: pointer.Int(1),
 				},
 				WaitBetweenRemovalsSeconds: pointer.Int(0),
 			},
diff --git a/kubectl-fdb/cmd/cordon_test.go b/kubectl-fdb/cmd/cordon_test.go
index 3fb655a3..2655ca59 100644
--- a/kubectl-fdb/cmd/cordon_test.go
+++ b/kubectl-fdb/cmd/cordon_test.go
@@ -24,6 +24,7 @@ import (
 	"bytes"
 	"context"
 	"fmt"
+
 	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"