Repository: argoproj/argo-rollouts

Issue:
Rollout stuck in infinite "Progressing" even after progressDeadlineSeconds exceeds
<!-- If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argo rollouts slack [channel](https://argoproj.github.io/community/join-slack). -->

Checklist:

* [x] I've included steps to reproduce the bug.
* [x] I've included the version of argo rollouts.

**Describe the bug**

In the default scenario, Rollouts work as expected: after a ReplicaSet becomes healthy, it starts the analysis and promotes the new stable version. However, I encountered an issue when deploying a broken revision.

For example, I intentionally misconfigured the `startupProbe` to make it fail, so the ReplicaSet could never become ready. I expected the Rollout to mark this revision as **Degraded** and abort the preview release once `progressDeadlineSeconds` was exceeded. This behavior worked as expected for the first broken revision, but did not for revision 4, which has different image tag and is broken.

**To Reproduce**

My configurations:

`bluegreen-service.yaml`:
```
apiVersion: v1
kind: Service
metadata:
  name: bluegreen-demo-preview
  labels:
    app: bluegreen-demo
spec:
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: bluegreen-demo
---
apiVersion: v1
kind: Service
metadata:
  name: bluegreen-demo
  labels:
    app: bluegreen-demo
spec:
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: bluegreen-demo
```

`bluegreen-rollout.yaml`:
```
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: bluegreen-demo
  labels:
    app: bluegreen-demo
spec:
  replicas: 3
  revisionHistoryLimit: 1
  progressDeadlineSeconds: 60
  progressDeadlineAbort: true
  selector:
    matchLabels:
      app: bluegreen-demo
  template:
    metadata:
      labels:
        app: bluegreen-demo
    spec:
      containers:
      - name: bluegreen-demo
        image: argoproj/rollouts-demo:blue # initially "blue", but changed to "green" to create a new "broken" release
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m
        startupProbe: # initially commented section, but uncommented once I need to break the new release.
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 30
  strategy:
    blueGreen:
      autoPromotionEnabled: false
      activeService: bluegreen-demo
      previewService: bluegreen-demo-preview
```
And a `kustomization.yaml` to deploy it:
```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- bluegreen-rollout.yaml
- bluegreen-service.yaml
```

1. Start with a fresh Rollout object with a healthy revision (Revision 1).
2. Deploy a broken revision (Revision 2) with the misconfigured health check.
Result: Rollout correctly marked it as Degraded after progressDeadlineSeconds and aborted the release.
3. Deploy a healthy revision (Revision 3).
Result: Revision 3 became the new stable version.
4. Deploy another broken revision (Revision 4).
Issue: The Rollout got stuck in an infinite Progressing state with the message: “active service cutover pending.”

**Expected behavior**

The Rollout should mark Revision 4 as **Degraded** and abort the release, similar to how it handled Revision 2.

**Screenshots**

<img width="711" alt="Initial release" src="https://github.com/user-attachments/assets/eac34b1a-7128-43ee-9a13-4d1134b394a2">

---

<img width="782" alt="Broken revision 2  expected behavior" src="https://github.com/user-attachments/assets/aaf6b29c-b275-47f0-9872-9a43a943e1e9">

---

<img width="744" alt="Successful revision 3" src="https://github.com/user-attachments/assets/ee884f47-4c3a-48f7-8647-291fb5bcdd82">

---

<img width="800" alt="Broken revision 4, stuck in infinite progressing state" src="https://github.com/user-attachments/assets/6622c707-00d8-4279-ad35-3d2c057284ed">

---

**Version**

**Argo Rollouts Version**: 2.37.7 (Chart version)
**Argo Rollouts Image**: quay.io/argoproj/argo-rollouts:v1.7.2

**Logs**

Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts
[controller-logs.txt](https://github.com/user-attachments/files/18038448/controller-logs.txt)


Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=bluegreen-demo
[bluegreen-rollout-grep-logs.txt](https://github.com/user-attachments/files/18038443/bluegreen-rollout-grep-logs.txt)

---
<!-- Issue Author: Don't delete this message to encourage other users to support your issue! -->
**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.


Discussion:
@zachaller: Is this repeatable 100% of the time I am not able to reproduce on master and was trying to decide if it's fixed or what I am missing in the steps.@tyoras: Hi @zachaller, I see the fix was reverted. Should this issue be reopened?

Solution patch:
diff --git a/USERS.md b/USERS.md
index 79a014db..85cce2b0 100644
--- a/USERS.md
+++ b/USERS.md
@@ -59,6 +59,7 @@ Organizations below are **officially** using Argo Rollouts. Please send a PR wit
 1. [Tuhu](https://www.tuhu.cn/)
 1. [Twilio SendGrid](https://sendgrid.com)
 1. [Ubie](https://ubie.life/)
+1. [Verkada](https://verkada.com)
 1. [VISITS Technologies](https://visits.world/en)
 1. [WeLab Bank](https://www.welab.bank/)
 1. [Yotpo](https://www.yotpo.com/)
diff --git a/cmd/rollouts-controller/main.go b/cmd/rollouts-controller/main.go
index 762d6ba4..e0a3267d 100644
--- a/cmd/rollouts-controller/main.go
+++ b/cmd/rollouts-controller/main.go
@@ -24,6 +24,7 @@ import (
 
 	"github.com/argoproj/argo-rollouts/metricproviders"
 	"github.com/argoproj/argo-rollouts/rollout"
+	"github.com/argoproj/argo-rollouts/utils/errors"
 	"github.com/argoproj/argo-rollouts/utils/record"
 
 	"github.com/argoproj/argo-rollouts/controller"
@@ -116,12 +117,12 @@ func newCommand() *cobra.Command {
 			defaults.SetTraefikVersion(traefikVersion)
 
 			config, err := clientConfig.ClientConfig()
-			checkError(err)
+			errors.CheckError(err)
 			config.QPS = qps
 			config.Burst = burst
 			namespace := metav1.NamespaceAll
 			configNS, _, err := clientConfig.Namespace()
-			checkError(err)
+			errors.CheckError(err)
 			if namespaced {
 				namespace = configNS
 				log.Infof("Using namespace %s", namespace)
@@ -131,15 +132,15 @@ func newCommand() *cobra.Command {
 			kubeclientmetrics.AddMetricsTransportWrapper(config, k8sRequestProvider.IncKubernetesRequest)
 
 			kubeClient, err := kubernetes.NewForConfig(config)
-			checkError(err)
+			errors.CheckError(err)
 			argoprojClient, err := clientset.NewForConfig(config)
-			checkError(err)
+			errors.CheckError(err)
 			dynamicClient, err := dynamic.NewForConfig(config)
-			checkError(err)
+			errors.CheckError(err)
 			discoveryClient, err := discovery.NewDiscoveryClientForConfig(config)
-			checkError(err)
+			errors.CheckError(err)
 			smiClient, err := smiclientset.NewForConfig(config)
-			checkError(err)
+			errors.CheckError(err)
 			resyncDuration := time.Duration(rolloutResyncPeriod) * time.Second
 			kubeInformerFactory := kubeinformers.NewSharedInformerFactoryWithOptions(
 				kubeClient,
@@ -150,7 +151,7 @@ func newCommand() *cobra.Command {
 				options.LabelSelector = instanceIDSelector.String()
 			}
 			jobKubeClient, _, err := metricproviders.GetAnalysisJobClientset(kubeClient)
-			checkError(err)
+			errors.CheckError(err)
 			jobNs := metricproviders.GetAnalysisJobNamespace()
 			if jobNs == "" {
 				// if not set explicitly use the configured ns
@@ -205,9 +206,9 @@ func newCommand() *cobra.Command {
 			)
 
 			mode, err := ingressutil.DetermineIngressMode(ingressVersion, kubeClient.DiscoveryClient)
-			checkError(err)
+			errors.CheckError(err)
 			ingressWrapper, err := ingressutil.NewIngressWrapper(mode, kubeClient, kubeInformerFactory)
-			checkError(err)
+			errors.CheckError(err)
 
 			if pprofAddress != "" {
 				mux := controller.NewPProfServer()
@@ -217,7 +218,7 @@ func newCommand() *cobra.Command {
 			var cm *controller.Manager
 
 			enabledControllers, err := getEnabledControllers(controllersEnabled)
-			checkError(err)
+			errors.CheckError(err)
 
 			// currently only supports running analysis controller independently
 			if enabledControllers[controllerAnalysis] {
@@ -371,12 +372,6 @@ func createFormatter(logFormat string) log.Formatter {
 	return formatType
 }
 
-func checkError(err error) {
-	if err != nil {
-		log.Fatal(err)
-	}
-}
-
 func getEnabledControllers(controllersEnabled []string) (map[string]bool, error) {
 	enabledControllers := make(map[string]bool)
 	for _, controller := range controllersEnabled {
diff --git a/go.mod b/go.mod
index 57628714..84d9d542 100644
--- a/go.mod
+++ b/go.mod
@@ -8,7 +8,7 @@ require (
 	github.com/argoproj/pkg v0.13.6
 	github.com/aws/aws-sdk-go-v2 v1.36.3
 	github.com/aws/aws-sdk-go-v2/config v1.29.8
-	github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.44.0
+	github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.44.1
 	github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.31.3
 	github.com/aws/smithy-go v1.22.3
 	github.com/blang/semver v3.5.1+incompatible
diff --git a/go.sum b/go.sum
index e1fe0796..6b6c3280 100644
--- a/go.sum
+++ b/go.sum
@@ -110,8 +110,8 @@ github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34 h1:SZwFm17ZUNNg5Np0io
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34/go.mod h1:dFZsC0BLo346mvKQLWmoJxT+Sjp+qcVR1tRVHQGOH9Q=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3 h1:bIqFDwgGXXN1Kpp99pDOdKMTTb5d2KyU5X/BZxjOkRo=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3/go.mod h1:H5O/EsxDWyU+LP/V8i5sm8cxoZgc2fdNR9bxlOFrQTo=
-github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.44.0 h1:0cF07Fs0CT8XSLGGFqp0VNJD+sb447S8UQU7hz95xJo=
-github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.44.0/go.mod h1:HJlcOk+S/wjJuR/8jPa8GhnEKdKqqiQ5wjsE1PjuO1o=
+github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.44.1 h1:ac0UBlcUK+tFcFiAuNbtKqUEtM+iyQgmffEhUACGwD0=
+github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.44.1/go.mod h1:HJlcOk+S/wjJuR/8jPa8GhnEKdKqqiQ5wjsE1PjuO1o=
 github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.31.3 h1:Avh8YS+sgb2OKRht0wdNwY8tqtsCzVrmc8dG8Wfy9LI=
 github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.31.3/go.mod h1:HbtHaw/hnNPaiqcyYnheILVyn81wOZiX9n2gYF5tPmM=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3 h1:eAh2A4b5IzM/lum78bZ590jy36+d/aFLgKF/4Vd1xPE=
diff --git a/rollout/replicaset.go b/rollout/replicaset.go
index e374b982..8412420b 100644
--- a/rollout/replicaset.go
+++ b/rollout/replicaset.go
@@ -107,6 +107,16 @@ func (c *rolloutContext) removeScaleDownDeadlines() error {
 			toRemove = append(toRemove, c.stableRS)
 		}
 	}
+	for _, rs := range c.otherRSs {
+		remainScaleDownDeadlines, err := replicasetutil.GetTimeRemainingBeforeScaleDownDeadline(rs)
+		if err != nil {
+			c.log.Warnf("%v", err)
+			continue
+		}
+		if replicasetutil.HasScaleDownDeadline(rs) && remainScaleDownDeadlines == nil {
+			toRemove = append(toRemove, rs)
+		}
+	}
 	for _, rs := range toRemove {
 		err := c.removeScaleDownDelay(rs)
 		if err != nil {
diff --git a/server/server.go b/server/server.go
index 9a3e65e3..0d01e277 100644
--- a/server/server.go
+++ b/server/server.go
@@ -8,7 +8,6 @@ import (
 	"strings"
 	"time"
 
-	"github.com/argoproj/pkg/errors"
 	"github.com/golang/protobuf/ptypes/empty"
 	"github.com/grpc-ecosystem/grpc-gateway/runtime"
 	log "github.com/sirupsen/logrus"
@@ -39,6 +38,7 @@ import (
 	"github.com/argoproj/argo-rollouts/pkg/kubectl-argo-rollouts/cmd/undo"
 	"github.com/argoproj/argo-rollouts/pkg/kubectl-argo-rollouts/info"
 	"github.com/argoproj/argo-rollouts/pkg/kubectl-argo-rollouts/viewcontroller"
+	"github.com/argoproj/argo-rollouts/utils/errors"
 	"github.com/argoproj/argo-rollouts/utils/json"
 	versionutils "github.com/argoproj/argo-rollouts/utils/version"
 )
diff --git a/utils/errors/errors.go b/utils/errors/errors.go
new file mode 100644
index 00000000..760abcf1
--- /dev/null
+++ b/utils/errors/errors.go
@@ -0,0 +1,10 @@
+package errors
+
+import log "github.com/sirupsen/logrus"
+
+// CheckError is a convenience function to fatally log an exit if the supplied error is non-nil
+func CheckError(err error) {
+	if err != nil {
+		log.Fatal(err)
+	}
+}



Test patch:
diff --git a/rollout/bluegreen_test.go b/rollout/bluegreen_test.go
index f8abaa9f..5a9bd7cf 100644
--- a/rollout/bluegreen_test.go
+++ b/rollout/bluegreen_test.go
@@ -1406,6 +1406,7 @@ func TestBlueGreenReadyToScaleDownOldReplica(t *testing.T) {
 	f.objects = append(f.objects, r2)
 	f.serviceLister = append(f.serviceLister, s)
 
+	f.expectPatchReplicaSetAction(rs2)
 	updatedRSIndex := f.expectUpdateReplicaSetAction(rs2)
 	patchIndex := f.expectPatchRolloutAction(r2)
 	f.run(getKey(r2, t))