Repository: redhat-openshift-ecosystem/opct
Pull request #160: OPCT-332: added a controller to mutate e2e pods failed to schedule
Author: login: mtulio
Labels: name: kind/feature
description: Categorizes issue or PR as related to a new feature.
name: lgtm
description: Indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: target-release/opct-v0.6
Created: 2025-03-06T20:01:26Z | Merged: 2025-03-14T18:15:00Z

**What this PR does / why we need it**:

This change is proposing a controller to watch the e2e pods failed to schedule, mutating those to apply the OPCT  dedicated toleration, targeting to prevent false-positive failures in the conformance results due the dedicated node introduced as required by OPCT.

The controller is started by a command `adm e2e-dedicated controller`, distributed with the same image of `opct`. The deployment will be created in `opct` namespace when the command `run` is executed. The destroy flow will remove all resources in `opct` namespace, not requiring a dedicated logic to remove the controller deployment.

Rehearsal Tests: https://github.com/openshift/release/pull/62503

Done checklist:
- [x] Validate if the tests failed to schedule due dedicated node have been resolved
- [x] Validate CI job using this feature
- [x] Reproduce if bugs linked to the Epic* is related to dedicated node failure, and this solution would resolved it
    - This change resolves all e2e tests choosing randomly workers node. We've observing when the worker node selected to be the dedicated node is the first in the node list, there is more chance to fall into this issue. Although the command `e2e-dedicated taint-node` tries to select the "best node" by preventing scheduling in nodes with monitoring stack, preventing lack of resources while running the workflow. Considering we don't have control of how monitoring stack schedule (and we don't want to tied on this), and partners don't use this logic, the controller will resolve the problem more generically.
    - Here are examples of bugs reporting failures due dedicated node[1][2][3][4]. The Rehearsal jobs[5][6] also is example running a version with the dedicated controller with (shiny) improvement in the false-positives failures.
- [x] Collect the controller logs. https://github.com/redhat-openshift-ecosystem/provider-certification-plugins/pull/78

*Epic [OPCT-332](https://issues.redhat.com/browse/OPCT-332)

**Which issue(s) this PR fixes**:

Fixes/Closes issues: [1][2][3][4]

References:
- [1] https://issues.redhat.com/browse/OPCT-277
- [2] [OPCT-310](https://issues.redhat.com/browse/OPCT-310)
- [3] [OPCT-283](https://issues.redhat.com/browse/OPCT-283)
- [4] [OCPBUGS-31407](https://issues.redhat.com/browse/OCPBUGS-31407) (partially)
- [5] https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-opct-external-aws-ccm/1898524710761664512/artifacts/opct-external-aws-ccm/opct-conformance-test-results/build-log.txt
- [6] https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_release/62503/rehearse-62503-periodic-ci-openshift-release-master-nightly-4.17-opct-external-aws-ccm/1900206675432837120/artifacts/opct-external-aws-ccm/opct-conformance-test-results/artifacts/log-results.txt

**Checklist**
- [x] Subject and description added to both, commit and PR.
- [x] Relevant issues have been referenced.
- [x] This change includes docs. 
- [ ] This change includes unit tests.



Diff:
diff --git a/docs/devel/guide.md b/docs/devel/guide.md
index 63002ab9..e3d1bc34 100644
--- a/docs/devel/guide.md
+++ b/docs/devel/guide.md
@@ -228,3 +228,39 @@ mkdocs serve
 ```
 
 Then you will be able to access the docs locally on the address: http://127.0.0.1:8000/
+
+
+## e2e-dedicated controller
+
+The e2e-dedicated controller is used to watch pods failing
+to schedule and apply tolerations to allow scheduling in
+the OPCT dedicated node, preventing false positives caused
+by the required OPCT environment (dedicated worker node).
+
+### Checking Logs in a Live Cluster
+
+You can check the logs in a live cluster using the following command:
+
+```sh
+oc logs -n opct -l app=opct-dedicated-e2e-controller -f
+```
+
+### Inspecting the Logs in the Archive File
+
+Alternatively you can inspect the controller logs from the archive
+collected in the artifacts collector step:
+
+```sh
+mkdir results-raw results-must-gather
+result_file=opct_202503132124_1899a26c-1564-4eab-b51a-2a39ca6caee4.tar.gz
+mustgather_file=results-raw/plugins/99-openshift-artifacts-collector/results/global/artifacts_must-gather.tar.xz
+
+# Extract the raw results
+tar xfz ${result_file} -C results-raw
+
+# Extracts the must-gather
+tar xfJ ${mustgather_file} -C results-must-gather/
+
+# Inspect the logs
+less results-must-gather/must-gather-opct/inspect-opct/namespaces/opct/pods/opct-dedicated-e2e-controller-*/controller/controller/logs/current.log
+```
diff --git a/pkg/cmd/adm/e2ed/controller.go b/pkg/cmd/adm/e2ed/controller.go
new file mode 100644
index 00000000..15714a89
--- /dev/null
+++ b/pkg/cmd/adm/e2ed/controller.go
@@ -0,0 +1,242 @@
+package e2ed
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	types "github.com/redhat-openshift-ecosystem/opct/pkg"
+
+	log "github.com/sirupsen/logrus"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	"github.com/spf13/cobra"
+	"k8s.io/apimachinery/pkg/fields"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/rest"
+	"k8s.io/client-go/tools/cache"
+	"k8s.io/client-go/util/retry"
+)
+
+// newCmdE2eDedicatedController returns a new cobra.Command for starting the e2e-dedicated controller.
+// The controller watches all pods failing to schedule due to dedicated node configuration, specifically
+// for OPCT, which requires a toleration configuration in the pod spec. Failed pods will be mutated to
+// use the required toleration, preventing false-positive failures.
+func newCmdE2eDedicatedController() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "controller",
+		Short: "Start the e2e-dedicated controller.",
+		Long: `Start the e2e-dedicated controller to watch all pods failing to schedule due the
+		dedicated node configuration, specifically for OPCT, which requires a toleration configuration
+		in the pod spec.
+		Failed pods will be mutated to use the required toleration, preventing false-positive failures.`,
+	}
+
+	cmd.Run = func(cmd *cobra.Command, args []string) {
+		controllerRun()
+	}
+
+	return cmd
+}
+
+type podMutateStatus struct {
+	Successed uint64
+	Failed    uint64
+	Skipped   uint64
+}
+
+var (
+	podCounters      = make(map[string]*podMutateStatus)
+	podCountersMutex sync.Mutex
+)
+
+// getPodKey constructs a unique key for a pod using its namespace and name.
+func getPodKey(namespace, podName string) string {
+	return fmt.Sprintf("%s/%s", namespace, podName)
+}
+
+// ensureCounter checks if a counter exists for the given pod key and initializes it if not.
+func ensureCounter(podKey string) {
+	podCountersMutex.Lock()
+	defer podCountersMutex.Unlock()
+	if _, ok := podCounters[podKey]; !ok {
+		podCounters[podKey] = &podMutateStatus{}
+	}
+}
+
+func showCounter(podKey string) {
+	pk := podCounters[podKey]
+	log.Debugf("Metrics counter for %s: success(%d) skipped(%d) failed(%d)", podKey, pk.Successed, pk.Skipped, pk.Failed)
+}
+
+// incCounterFailure increments the failure counter for the given pod key.
+func incCounterFailure(podKey string) {
+	ensureCounter(podKey)
+	podCountersMutex.Lock()
+	podCounters[podKey].Failed += 1
+	podCountersMutex.Unlock()
+	showCounter(podKey)
+}
+
+// incCounterSkipped increments the skipped counter for the given pod key.
+func incCounterSkipped(podKey string) {
+	ensureCounter(podKey)
+	podCountersMutex.Lock()
+	podCounters[podKey].Skipped += 1
+	podCountersMutex.Unlock()
+	showCounter(podKey)
+}
+
+// incCounterSuccess increments the success counter for the given pod key.
+func incCounterSuccess(podKey string) {
+	ensureCounter(podKey)
+	podCountersMutex.Lock()
+	podCounters[podKey].Successed += 1
+	podCountersMutex.Unlock()
+	showCounter(podKey)
+}
+
+// showCounters prints the current counters for all pods.
+func showCounters(stop chan struct{}) {
+	previousTotal := 0
+	backoff := 10 * time.Second
+	for {
+		select {
+		case <-stop:
+			return
+		default:
+			// Show summary of total pods changed for this controller.
+			failed := 0
+			skipped := 0
+			successed := 0
+			for podKey := range podCounters {
+				failed += int(podCounters[podKey].Failed)
+				skipped += int(podCounters[podKey].Skipped)
+				successed += int(podCounters[podKey].Successed)
+			}
+			// show summary with backoff to prevent many messages.
+			if previousTotal != len(podCounters) {
+				log.Printf("Metrics summary: Total pods changed: %d. successed(%d) skipped(%d) failed(%d)", len(podCounters), successed, skipped, failed)
+				previousTotal = len(podCounters)
+				time.Sleep(1 * time.Second)
+				continue
+			}
+			time.Sleep(backoff)
+			backoff *= 2
+			if backoff > 2*time.Minute {
+				backoff = 2 * time.Minute
+				log.Printf("No change in metrics in the last 2 minutes.")
+			}
+		}
+
+	}
+}
+
+// controllerRun starts the dedicated-e2e controller which watches for pods failing to schedule
+// in the e2e namespaces and mutates them to include the required toleration.
+func controllerRun() {
+	log.Info("Starting the e2e-dedicated controller...")
+
+	config, err := rest.InClusterConfig()
+	if err != nil {
+		log.Fatalf("Failed to get in-cluster config: %v. Ensure the KUBECONFIG environment variable is set or config is in the path:\n", err)
+	}
+
+	clientset, err := kubernetes.NewForConfig(config)
+	if err != nil {
+		log.Fatalf("Failed to create clientset: %v\n", err)
+	}
+
+	log.Info("Creating the informer to watch pods failed to schedule in e2e namespaces...")
+
+	// Create the informer to watch pods every one second, then mutate when updated. The mutation
+	// will add the required toleration.
+	_, controller := cache.NewInformerWithOptions(cache.InformerOptions{
+		ListerWatcher: cache.NewListWatchFromClient(
+			clientset.CoreV1().RESTClient(),
+			"pods",
+			corev1.NamespaceAll,
+			fields.Everything(),
+		),
+		ObjectType: &corev1.Pod{},
+		Handler: cache.ResourceEventHandlerDetailedFuncs{
+			UpdateFunc: func(oldObj, newObj interface{}) {
+				newPod := newObj.(*corev1.Pod)
+				// ensure only e2e and opct (for tests) will be changed
+				if strings.HasPrefix(newPod.Namespace, "e2e-") || strings.HasPrefix(newPod.Namespace, "opct") {
+					for _, condition := range newPod.Status.Conditions {
+						// act only when the pod failed to schedule due the opct environment: one random worker node has taints preventing scheduling the node.
+						if condition.Type == corev1.PodScheduled && condition.Status == corev1.ConditionFalse && condition.Reason == corev1.PodReasonUnschedulable {
+							handleFailedScheduling(clientset, newPod)
+						}
+					}
+				}
+			},
+		},
+		ResyncPeriod: 1 * time.Second,
+	})
+
+	stop := make(chan struct{})
+	defer close(stop)
+	go controller.Run(stop)
+	go showCounters(stop)
+
+	select {}
+}
+
+// handleFailedScheduling is the update informer function handler to mutate the pod object adding the
+// required tolerations, when not exists. Informer function handlers can't return errors,
+// when operation fails, it will be logged in the default log handler.
+func handleFailedScheduling(clientset *kubernetes.Clientset, pod *corev1.Pod) {
+	podKey := getPodKey(pod.Namespace, pod.Name)
+	skipped := false
+	log.Debugf("[%s] starting the handler for failed scheduling pods", podKey)
+
+	retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		pod, err := clientset.CoreV1().Pods(pod.Namespace).Get(context.Background(), pod.Name, metav1.GetOptions{})
+		if err != nil {
+			// Update functions can't return errors
+			log.Errorf("[%s] failed to get the pod by name: %v", podKey, err)
+			return nil
+		}
+
+		// add tolerations only if not yet applied to the node.
+		hasToleration := false
+		for _, toleration := range pod.Spec.Tolerations {
+			if toleration.Key == types.DedicatedNodeRoleLabel {
+				hasToleration = true
+				break
+			}
+		}
+
+		if hasToleration {
+			log.Debugf("[%s] skipping pod already has the required toleration", podKey)
+			incCounterSkipped(podKey)
+			skipped = true
+			return nil
+		}
+		toleration := corev1.Toleration{
+			Key:      types.DedicatedNodeRoleLabel,
+			Operator: corev1.TolerationOpExists,
+			Effect:   corev1.TaintEffectNoSchedule,
+		}
+		pod.Spec.Tolerations = append(pod.Spec.Tolerations, toleration)
+
+		_, updateErr := clientset.CoreV1().Pods(pod.Namespace).Update(context.Background(), pod, metav1.UpdateOptions{})
+		if updateErr != nil {
+			log.Errorf("[%s] failed to update pod: %v", podKey, updateErr)
+		}
+		return updateErr
+	})
+
+	if retryErr != nil {
+		log.Errorf("[%s] failed to update pod: %v", podKey, retryErr)
+		incCounterFailure(podKey)
+	} else if !skipped {
+		log.Infof("[%s] successfully added toleration to pod", podKey)
+		incCounterSuccess(podKey)
+	}
+}
diff --git a/pkg/cmd/adm/e2ed/root.go b/pkg/cmd/adm/e2ed/root.go
index 97fd46a6..56ee4e19 100644
--- a/pkg/cmd/adm/e2ed/root.go
+++ b/pkg/cmd/adm/e2ed/root.go
@@ -11,6 +11,7 @@ func NewCommandE2eDedicated() *cobra.Command {
 		Short: "Administrative commands for e2e-dedicated node configuration",
 	}
 
+	cmd.AddCommand(newCmdE2eDedicatedController())
 	cmd.AddCommand(taintNodeCmd)
 
 	return cmd
diff --git a/pkg/cmd/adm/e2ed/taintNode.go b/pkg/cmd/adm/e2ed/taintNode.go
index 152d4a42..c3606a02 100644
--- a/pkg/cmd/adm/e2ed/taintNode.go
+++ b/pkg/cmd/adm/e2ed/taintNode.go
@@ -137,5 +137,4 @@ func taintNodeRun(cmd *cobra.Command, args []string) {
 	if err := applyTaintToNode(kclient, taintNodeArgs.nodeName); err != nil {
 		log.Fatalf("Failed to apply taint to node: %v", err)
 	}
-	log.Infof("Successfully applied taint to node %s", taintNodeArgs.nodeName)
 }
diff --git a/pkg/destroy/destroy.go b/pkg/destroy/destroy.go
index 48da6301..3dc1dcde 100644
--- a/pkg/destroy/destroy.go
+++ b/pkg/destroy/destroy.go
@@ -44,20 +44,18 @@ func NewCmdDestroy() *cobra.Command {
 				return err
 			}
 
-			err = o.DeleteSonobuoyEnv(sclient)
-			if err != nil {
+			log.Info("removing opct namespace...")
+			if err := o.DeleteSonobuoyEnv(sclient); err != nil {
 				log.Warn(err)
 			}
 
-			log.Info("removing non-openshift NS...")
-			err = o.DeleteTestNamespaces(kclient)
-			if err != nil {
+			log.Info("removing stale e2e namespaces...")
+			if err := o.DeleteTestNamespaces(kclient); err != nil {
 				log.Warn(err)
 			}
 
 			log.Info("restoring privileged environment...")
-			err = o.RestoreSCC(kclient)
-			if err != nil {
+			if err := o.RestoreSCC(kclient); err != nil {
 				log.Warn(err)
 			}
 
diff --git a/pkg/run/run.go b/pkg/run/run.go
index 64ed9fd8..1faa3bf6 100644
--- a/pkg/run/run.go
+++ b/pkg/run/run.go
@@ -20,7 +20,9 @@ import (
 	"github.com/vmware-tanzu/sonobuoy/pkg/config"
 	"github.com/vmware-tanzu/sonobuoy/pkg/plugin/loader"
 	"github.com/vmware-tanzu/sonobuoy/pkg/plugin/manifest"
+	appsv1 "k8s.io/api/apps/v1"
 	v1 "k8s.io/api/core/v1"
+	"k8s.io/utils/ptr"
 
 	"github.com/redhat-openshift-ecosystem/opct/pkg"
 	"github.com/redhat-openshift-ecosystem/opct/pkg/client"
@@ -28,6 +30,7 @@ import (
 	"github.com/redhat-openshift-ecosystem/opct/pkg/wait"
 	rbacv1 "k8s.io/api/rbac/v1"
 	kerrors "k8s.io/apimachinery/pkg/api/errors"
+	kresource "k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/client-go/kubernetes"
@@ -464,6 +467,65 @@ func (r *RunOptions) PreRunSetup(kclient kubernetes.Interface) error {
 	}
 	log.Infof("Created %s ClusterRoleBinding", pkg.PrivilegedClusterRoleBinding)
 
+	// Create the Deployment of dedicated-e2e-controller.
+	// The controller watches for new pods failed to schedule, and apply
+	// tolerations.
+	// TODO(mtulio): change the image registry to dynamic to support disconnected setup.
+	if r.dedicated {
+		deployment := &appsv1.Deployment{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      pkg.DedicatedControllerName,
+				Namespace: pkg.CertificationNamespace,
+				Labels:    pkg.SonobuoyDefaultLabels,
+			},
+			Spec: appsv1.DeploymentSpec{
+				Replicas: ptr.To[int32](1),
+				Selector: &metav1.LabelSelector{
+					MatchLabels: map[string]string{
+						"app": pkg.DedicatedControllerName,
+					},
+				},
+				Template: v1.PodTemplateSpec{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels: map[string]string{
+							"app": pkg.DedicatedControllerName,
+						},
+					},
+					Spec: v1.PodSpec{
+						ServiceAccountName: pkg.SonobuoyServiceAccountName,
+						Containers: []v1.Container{
+							{
+								Name:            "controller",
+								Image:           pkg.ControllerImage,
+								ImagePullPolicy: v1.PullAlways,
+								Command: []string{
+									"opct",
+									"adm",
+									"e2e-dedicated",
+									"controller",
+								},
+								Resources: v1.ResourceRequirements{
+									Limits: v1.ResourceList{
+										v1.ResourceCPU:    kresource.MustParse("128m"),
+										v1.ResourceMemory: kresource.MustParse("256Mi"),
+									},
+									Requests: v1.ResourceList{
+										v1.ResourceMemory: kresource.MustParse("64Mi"),
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		}
+
+		_, err := kclient.AppsV1().Deployments(pkg.CertificationNamespace).Create(context.TODO(), deployment, metav1.CreateOptions{})
+		if err != nil {
+			return errors.Wrap(err, "error creating e2e-dedicated-controller deployment")
+		}
+	}
+
 	return nil
 }
 
diff --git a/pkg/types.go b/pkg/types.go
index e669079b..eb0f8caa 100644
--- a/pkg/types.go
+++ b/pkg/types.go
@@ -19,10 +19,12 @@ const (
 	SonobuoyLabelComponentName     = "component"
 	SonobuoyLabelComponentValue    = "sonobuoy"
 	DefaultToolsRepository         = "quay.io/opct"
+	ControllerImage                = "quay.io/opct/opct:v0.6.0-alpha.1"
 	PluginsImage                   = "plugin-openshift-tests:v0.5.2"
 	CollectorImage                 = "plugin-artifacts-collector:v0.5.2"
 	MustGatherMonitoringImage      = "must-gather-monitoring:v0.5.2"
 	OpenShiftTestsImage            = "image-registry.openshift-image-registry.svc:5000/openshift/tests"
+	DedicatedControllerName        = "opct-dedicated-e2e-controller"
 )
 
 var (


Reviews:
user: login: rvanderp3
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T21:21:51Z
user: login: mtulio
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T21:23:42Z
user: login: mtulio
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T21:25:34Z
user: login: copilot-pull-request-reviewer[bot]
body: ## PR Overview

This PR introduces a dedicated e2e controller to mutate pods that fail to schedule due to dedicated node configuration, preventing false-positive test failures.  
- Renames and refactors the node setup command to "taint-node" under the e2ed package.  
- Adds a new deployment creation for the e2e-dedicated controller in the run package.  
- Introduces the new controller command with informer logic to automatically update pods with the required toleration.

### Reviewed Changes

| File                                      | Description                                                        |
| ----------------------------------------- | ------------------------------------------------------------------ |
| pkg/cmd/adm/e2ed/taintNode.go              | Refactors the node setup command and updates help text.            |
| pkg/run/run.go                            | Adds deployment creation logic for the dedicated e2e controller.     |
| pkg/cmd/adm/e2ed/root.go                   | Registers the new e2e-dedicated commands.                            |
| pkg/cmd/adm/e2ed/controller.go             | Implements controller logic to watch and mutate unscheduled pods.    |
| pkg/types.go                              | Introduces constants for controller image and dedicated controller name.  |
| pkg/cmd/adm/root.go                        | Registers the new e2e-dedicated command in the admin command tree.   |

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.



state: COMMENTED
author_association: NONE
submitted_at: 2025-03-06T21:33:52Z
user: login: mtulio
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T22:19:57Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces a dedicated controller and associated administrative commands to support mutating e2e pods that fail to schedule due to dedicated node configuration issues, and deprecates the older setup-node command.  
- Added a new "taint-node" command to apply dedicated node taints.  
- Introduced the "controller" command and its implementation to automatically mutate pods with the required toleration.  
- Updated the deployment logic to create a dedicated controller deployment and removed the deprecated setup-node command.

### Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

<details>
<summary>Show a summary per file</summary>

| File                                  | Description                                                                              |
| ------------------------------------- | ---------------------------------------------------------------------------------------- |
| pkg/cmd/adm/e2ed/taintNode.go          | Added the taint-node command for setting dedicated node taints.                           |
| pkg/cmd/adm/e2ed/root.go               | Created a new root command for e2e-dedicated subcommands.                                  |
| pkg/run/run.go                         | Integrated the dedicated controller deployment creation into the run pipeline.           |
| pkg/cmd/adm/e2ed/controller.go         | Introduced the dedicated controller for mutating pods with the required toleration.        |
| pkg/destroy/destroy.go                 | Updated logging and error handling when destroying test namespaces and restoring state.    |
| pkg/types.go                           | Added constants for the controller image and dedicated controller name.                  |
| pkg/cmd/adm/root.go                    | Updated the adm root command to include the new e2e-dedicated commands.                    |
| pkg/cmd/adm/setupNode.go               | Removed the deprecated setup-node command.                                               |
</details>



<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**pkg/cmd/adm/e2ed/controller.go:57**
* [nitpick] The parameter names in getPodKey may be confusing since the function is later invoked with the namespace first and pod name second; consider renaming the parameters (e.g., namespace, podName) or adjusting the order for clarity.
```
func getPodKey(podName, namespace string) string {
```
</details>


state: COMMENTED
author_association: NONE
submitted_at: 2025-03-13T03:53:09Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces a new controller for e2e pods that failed to schedule, along with new administrative commands to manage node taints and controller deployment. It also removes the deprecated "setup-node" command and updates the cleanup and configuration routines accordingly.
- Adds a new taint-node command for applying dedicated node taints.
- Creates a dedicated e2e controller deployment when the dedicated flag is enabled.
- Cleans up deprecated commands and updates namespace/resource deletion flows.

### Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

<details>
<summary>Show a summary per file</summary>

| File                                     | Description                                                                          |
| ---------------------------------------- | ------------------------------------------------------------------------------------ |
| pkg/cmd/adm/e2ed/taintNode.go             | Introduces a new command to apply required labels and taints to the chosen node.      |
| pkg/run/run.go                           | Adds deployment logic for the dedicated e2e controller with updated image settings.   |
| pkg/cmd/adm/e2ed/root.go                  | Registers the new e2e-dedicated administrative command.                              |
| pkg/cmd/adm/e2ed/controller.go            | Implements the controller to watch for and mutate pods failing to schedule.          |
| pkg/destroy/destroy.go                    | Enhances logging and cleanup messages during environment destruction.                |
| pkg/types.go                             | Updates constants with new controller name and image details.                        |
| pkg/cmd/adm/root.go                      | Modifies the root administrative command by removing deprecated commands and adding e2ed. |
| pkg/cmd/adm/setupNode.go                 | Removes the deprecated setup-node command.                                            |
</details>





state: COMMENTED
author_association: NONE
submitted_at: 2025-03-13T03:56:36Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces a new e2e-dedicated controller to watch pods that fail scheduling and automatically apply the required toleration, helping prevent false-positive failures in conformance tests. Key changes include:
- Implementation of the dedicated controller in the new cmd package, with corresponding CLI integration.
- Deployment setup for the controller in the run package and documentation updates describing its usage and log inspection.
- Minor logging improvements and cleanup in auxiliary commands and destroy routines.

### Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

<details>
<summary>Show a summary per file</summary>

| File                                 | Description                                                                                  |
| ------------------------------------ | -------------------------------------------------------------------------------------------- |
| docs/devel/guide.md                  | Adds documentation for the e2e-dedicated controller, covering command usage and log inspection.|
| pkg/run/run.go                       | Creates the Deployment for the dedicated controller and adjusts image and registry settings. |
| pkg/cmd/adm/e2ed/controller.go       | Implements the controller logic that watches pods and adds the required toleration.          |
| pkg/cmd/adm/taintNode.go              | Removes duplicate log messages in the taint node command.                                    |
| pkg/cmd/adm/e2ed/root.go              | Integrates the new controller command into the administrative CLI commands.                  |
| pkg/destroy/destroy.go               | Refactors resource cleanup steps and logging in the destroy workflow.                         |
| pkg/types.go                         | Introduces constants for the controller image and dedicated controller name.                 |
</details>



<details>
<summary>Comments suppressed due to low confidence (2)</summary>

**pkg/cmd/adm/e2ed/controller.go:46**
* [nitpick] The field name 'Successed' is non-standard; consider renaming it to 'Succeeded' for clarity.
```
Successed uint64
```
**pkg/cmd/adm/e2ed/controller.go:1**
* The new controller logic currently lacks unit tests; please consider adding tests to verify its behavior.
```
package e2ed
```
</details>


state: COMMENTED
author_association: NONE
submitted_at: 2025-03-14T13:49:56Z

Review comments:
diff_hunk: @@ -464,6 +464,8 @@ func (r *RunOptions) PreRunSetup(kclient kubernetes.Interface) error {
 	}
 	log.Infof("Created %s ClusterRoleBinding", pkg.PrivilegedClusterRoleBinding)
 
+	// Create the e2e-dedicated-controller deployment
path: pkg/run/run.go
user: login: rvanderp3
body: did you intend to have something here?
created_at: 2025-03-06T21:21:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -464,6 +464,8 @@ func (r *RunOptions) PreRunSetup(kclient kubernetes.Interface) error {
 	}
 	log.Infof("Created %s ClusterRoleBinding", pkg.PrivilegedClusterRoleBinding)
 
+	// Create the e2e-dedicated-controller deployment
path: pkg/run/run.go
user: login: mtulio
body: maybe it is old code? I cant find
created_at: 2025-03-06T21:23:42Z
author_association: COLLABORATOR
diff_hunk: @@ -464,6 +464,8 @@ func (r *RunOptions) PreRunSetup(kclient kubernetes.Interface) error {
 	}
 	log.Infof("Created %s ClusterRoleBinding", pkg.PrivilegedClusterRoleBinding)
 
+	// Create the e2e-dedicated-controller deployment
path: pkg/run/run.go
user: login: mtulio
body: iirc I did some cleanup in the last commit
created_at: 2025-03-06T21:25:34Z
author_association: COLLABORATOR
diff_hunk: @@ -12,28 +14,43 @@ import (
 	"k8s.io/client-go/kubernetes"
 )
 
+const computeNodeLabelSelector = "node-role.kubernetes.io/worker="
+const monitoringNamespace string = "openshift-monitoring"
+const monitoringPrometheusLabelSelector string = "prometheus=k8s"
+
 type setupNodeInput struct {
 	nodeName string
 	yes      bool
 }
 
-var setupNodeArgs setupNodeInput
-var setupNodeCmd = &cobra.Command{
-	Use:     "setup-node",
-	Example: "opct adm setup-node",
-	Short:   "Setup the node for the validation process.",
-	Run:     setupNodeRun,
+var taintNodeArgs setupNodeInput
+var taintNodeCmd = &cobra.Command{
+	Use:     "taint-node",
+	Example: "opct adm e2e-dedicated taint-node [options]",
+	Short:   "Setup the 'dedicated' node for the validation environment.",
+	Long: `Setup the 'dedicated' nodes to the test environment used by OPCT.
+The 'dedicated' node is used to host all the service and jobs used during the workflow
+execution. It is used to prevent disruption or evictions in the test environment.
+The command 'taint-node' automatically discovery a node which is not hosting the
+monitoring services, such as Prometheus pods, setting up quickly the test environment
+without forcing the controllers to rebalance.
+
+You can automatically accept the selected node with the option --yes.
+
+Alternativaly you can set the custom node with the option --node <node-name>.
path: pkg/cmd/adm/e2ed/taintNode.go
user: login: Copilot
body: Consider correcting the typo 'Alternativaly' to 'Alternatively'.
```suggestion
Alternatively you can set the custom node with the option --node <node-name>.
```
created_at: 2025-03-06T21:33:51Z
author_association: NONE
diff_hunk: @@ -0,0 +1,203 @@
+package e2ed
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"time"
+
+	types "github.com/redhat-openshift-ecosystem/opct/pkg"
+
+	log "github.com/sirupsen/logrus"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	"github.com/spf13/cobra"
+	"k8s.io/apimachinery/pkg/fields"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/rest"
+	"k8s.io/client-go/tools/cache"
+	"k8s.io/client-go/util/retry"
+)
+
+// NewCmdE2eDedicatedController returns a new cobra.Command for starting the e2e-dedicated controller.
+// The controller watches all pods failing to schedule due to dedicated node configuration, specifically
+// for OPCT, which requires a toleration configuration in the pod spec. Failed pods will be mutated to
+// use the required toleration, preventing false-positive failures.
+func NewCmdE2eDedicatedController() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "controller",
+		Short: "Start the e2e-dedicated controller.",
+		Long: `Start the e2e-dedicated controller to watch all pods failing to schedule due the
+		dedicated node configuration, specifically for OPCT, which requires a toleration configuration
+		in the pod spec.
+		Failed pods will be mutated to use the required toleration, preventing false-positive failures.`,
+	}
+
+	cmd.Run = func(cmd *cobra.Command, args []string) {
+		controllerRun()
+	}
+
+	return cmd
+}
+
+type podMutateStatus struct {
+	Successed uint64
+	Failed    uint64
+	Skipped   uint64
+}
+
+var podCounters map[string]*podMutateStatus
+
path: pkg/cmd/adm/e2ed/controller.go
user: login: Copilot
body: podCounters is accessed from multiple goroutines without synchronization, which may lead to race conditions. Consider using a mutex or a concurrent-safe map to protect its access.
```suggestion
var (
	podCounters = make(map[string]*podMutateStatus)
	podCountersMutex sync.Mutex
)
```
created_at: 2025-03-06T21:33:52Z
author_association: NONE
diff_hunk: @@ -0,0 +1,219 @@
+package e2ed
path: pkg/cmd/adm/e2ed/controller.go
user: login: mtulio
body: TODO: move the controller logic to a dedicated package structure (removing from command package)+
created_at: 2025-03-10T22:19:56Z
author_association: COLLABORATOR
diff_hunk: @@ -228,3 +228,39 @@ mkdocs serve
 ```
 
 Then you will be able to access the docs locally on the address: http://127.0.0.1:8000/
+
+
+## e2e-dedicated controller
+
+The e2e-dedicated controller is used to watch pods failing
+to schedule and apply tolerations to allow scheduling in
+the OPCT dedicated node, preventing false positives caused
+by the required OPCT environment (dedicated worker node).
+
+### Checking Logs in a Live Cluster
+
+You can check the logs in a live cluster using the following command:
+
+```sh
+oc logs -n opct -l app=opct-dedicated-e2e-controller -f
+```
+
+### Inspecting the Logs in a the Archive file
+
path: docs/devel/guide.md
user: login: Copilot
body: [nitpick] The header contains a redundant article; consider revising it to something like '### Inspecting the Logs in the Archive File'.
```suggestion
### Inspecting the Logs in the Archive File
```
created_at: 2025-03-14T13:49:55Z
author_association: NONE
diff_hunk: @@ -464,6 +467,66 @@ func (r *RunOptions) PreRunSetup(kclient kubernetes.Interface) error {
 	}
 	log.Infof("Created %s ClusterRoleBinding", pkg.PrivilegedClusterRoleBinding)
 
+	// Create the Deployment of dedicated-e2e-controller.
+	// The controller watches for new nodes and labels them with the
path: pkg/run/run.go
user: login: Copilot
body: The comment suggests the controller watches nodes, but the controller implementation actually watches pods; please update the comment to accurately reflect its behavior.
```suggestion
	// The controller watches for new pods and labels their nodes with the
```
created_at: 2025-03-14T13:49:55Z
author_association: NONE

PR comments:
user: login: vr4manta
created_at: 2025-03-14T18:09:54Z
body: /lgtm
/approve
author_association: COLLABORATOR
user: login: openshift-ci[bot]
created_at: 2025-03-14T18:10:00Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/redhat-openshift-ecosystem/opct/pull/160#issuecomment-2725421422" title="Approved">vr4manta</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=redhat-openshift-ecosystem%2Fopct).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/redhat-openshift-ecosystem/opct/blob/main/OWNERS)~~ [vr4manta]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE