Repository: argoproj/argo-rollouts

Issue:
Rollout stuck indefinitely when new pods fail to come up – abort and progressDeadline NOT triggering
<!-- If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argo rollouts slack [channel](https://argoproj.github.io/community/join-slack). -->

Checklist:

* [x] I've included steps to reproduce the bug.
* [x] I've included the version of argo rollouts.

**Describe the bug**

When deploying a Canary Rollout (using workloadRef) in Argo Rollouts v1.8.0, if the new version's pods fail to come up (e.g. due to ImagePullBackOff), the rollout remains stuck in a "Progressing" state indefinitely. Even with `progressDeadlineSeconds: 600` and `progressDeadlineAbort: true` configured, the rollout never marks itself as failed and manual abort commands (`kubectl argo rollouts abort <rollout-name>`) do not take effect unless the failing ReplicaSet is manually deleted. This blocks any subsequent updates, as the rollout remains stuck with the failing new ReplicaSet.

**To Reproduce**

1. Create a Canary Rollout using `workloadRef` to an existing Deployment.
2. Update the rollout to use a new image that is misconfigured or non-existent so that pods fail (e.g., trigger ImagePullBackOff).
3. Confirm that the rollout status remains "Progressing" with a message like:

```
Name:            xapproxy
Namespace:       red
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/4
  SetWeight:     10
  ActualWeight:  0
Images:         dockerhub.io/istio/proxyv2:1.17.1 (canary, stable)
                dockerhub.io/myproxy:25.2.1.3-745.76dfaae (canary)
                dockerhub.io/myproxy:25.2.1.3-746.76dfaae (stable)
```

4. Run the command:
```bash
kubectl argo rollouts abort <rollout-name> -n <namespace>
```

and observe that the rollout does not transition to a failed/aborted state. 

5. As a workaround, executing argo undo/abort:
`kubectl argo rollouts undo <rollout-name> -n <namespace>` or `kubectl argo rollouts abort <rollout-name> -n <namespace>`

followed by manually deleting the failing ReplicaSet (e.g., `kubectl delete rs <failing-replicaset> -n <namespace>`) is required before any new version updates are applied.

**Expected behavior**

- The rollout controller should detect that the new ReplicaSet’s pods are not becoming Ready within the defined progress deadline and automatically abort or mark the rollout as failed.
- The abort command should work regardless of the pod state, allowing the rollout to revert to the stable ReplicaSet and enabling new updates.
- No manual intervention (such as deleting the failing ReplicaSet) should be necessary.

**Version**

Argo Rollouts version: 1.8.0 (stable, released ~2 weeks ago)






Discussion:
@zachaller: Can you confirm if this behavior exists on older versions? Does this also only effect workloadRef, if you don't use workloadRef does it behave as expected?@revandarth: @zachaller  sorry for the late response. 

This behavior does not happen with v1.7.2. If a pod becomes stuck, the earlier version allows for subsequent updates, and the abort command operates correctly. It shows the same behavior regardless of whether workloadRef or without. 

Please let me if you require additional information to reproduce this issue. Others are experiencing similar issue and have reported them on Slack, please see the following thread.

https://cloud-native.slack.com/archives/C01U781DW2E/p1739561010345209@nebojsa-prodana: Impacted by this as well, I suspect that it might be related to this change - https://github.com/argoproj/argo-rollouts/pull/3602/files#diff-bc2f3a4d5d2fd799a169ceccb8bf97ae8b8a45f6a69009525c24e41fd846ef5eR329

as it causes an early exit that does not consider progressdeadline.

@revandarth are you using destinationrule subset based canaries or are you using service based? I believe the above change will only affect subset based canaries@nebojsa-prodana: UPDATE: I just tried the latest master and the issue is not present in there when using Rollouts directly with istio subset based traffic routing.

EDIT: It doesn't always solve the issue, some services with unhealthy pods are still experiencing the problem.


Could we make a new release so that we do not have to rely on an unreleased version?@darthkurak: Have the same issue. Checked on 1.8.2 and helm chart: 2.39.3. It is a very annoying issue. If someone deploys a failing service - it will stuck unless manually remove the broken Replica Set.@zachaller: Can I get an exact configuration and a step by step reproduction.@dtamrazovupgrade: Edit: With the same setup, I've reproduced this issue on v1.8.0 as well, and _cannot_ reproduce it on v1.7.2. So it looks like the regression was introduced in v1.8.0.

@zachaller I've reproduced it with the following setup. This issue did not occur on our prior version v1.7.1, we only ran into it after upgrading to v1.8.2

**Note:** This issue does _not_ occur with a basic canary that does not include `.spec.strategy.trafficRouting` and the `VirtualService` + `DestinationRule` below.

Argo Rollouts version: v1.8.2

Manifests:

```yaml
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: my-srvc
  namespace: my-namespace
spec:
  hosts:
  - my-srvc.my-namespace.svc.cluster.local
  http:
  - name: primary
    route:
    - destination:
        host: my-srvc
        port:
          number: 8080
        subset: stable
      weight: 100
    - destination:
        host: my-srvc
        port:
          number: 8080
        subset: canary
      weight: 0
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  annotations:
    argo-rollouts.argoproj.io/managed-by-rollouts: my-srvc
  name: my-srvc
  namespace: my-namespace
spec:
  host: my-srvc
  subsets:
  - labels:
      app: my-srvc
    name: canary
  - labels:
      app: my-srvc
    name: stable
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-srvc
  namespace: my-namespace
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-srvc
  strategy:
    canary:
      maxSurge: 25%
      maxUnavailable: 25%
      steps:
      - setWeight: 20
      - pause:
          duration: 1m
      - setWeight: 100
      trafficRouting:
        istio:
          destinationRule:
            canarySubsetName: canary
            name: my-srvc
            stableSubsetName: stable
          virtualService:
            name: my-srvc
            routes:
            - primary
  template:
    spec:
      containers:
      - image: nginx:alpine
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - sleep 10
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          limits:
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 128Mi
```

Steps to reproduce: 


1. Apply the manifests included in the output
2. Observe the rollout gets to a healthy state
3. Change the image on the rollout to eg `nginx:does-not-exist`
4. Note that the canary replicaset cannot get healthy pods as they crash with an `ImagePullBackoff`
  - The same can be reproduced with any container you can get running & reliably crash during the rollout process; an invalid image reference is simply the easiest way to get an unhealthy canary replica set
6. Abort the rollout
7. Observe that `.status.aborted: true` is set on the Rollout, yet the canary replica set is not being cleaned up
8. The only remedy is to manually delete the canary replica set to get the Rollout unstuck@pjiang-dev: @dtamrazovupgrade @zachaller I tried reproducing this using the same setup above and did encounter the bug. However i noticed that the rollout did not include a canary/stable service. Once i added a canary and stable service, the abort rollout and timeout worked fine. See the example below

```
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-srvc
  namespace: my-namespace
spec:
  replicas: 3
  progressDeadlineSeconds: 600
  progressDeadlineAbort: true
  selector:
    matchLabels:
      app: my-srvc
  strategy:
    canary:
      maxSurge: 25%
      maxUnavailable: 25%
      canaryService: my-srvc-canary
      stableService: my-srvc-stable
      steps:
      - setWeight: 20
      - pause:
          duration: 1m
      - setWeight: 100
      trafficRouting:
        istio:
          destinationRule:
            canarySubsetName: canary
            name: my-srvc
            stableSubsetName: stable
          virtualService:
            name: my-srvc
            routes:
            - primary
  template:
    metadata:
      labels:
        app: my-srvc
    spec:
      containers:
      - image: nginx:does-not-exist
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - sleep 10
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          limits:
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: my-srvc-stable
  namespace: my-namespace
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: my-srvc
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: my-srvc-canary
  namespace: my-namespace
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: my-srvc
  type: ClusterIP
```

My guess on why the abort doesn't work in this setup and fails silently:
1. Abort requires reconciliation to complete
2. Reconciliation requires traffic routing to succeed
3. Istio is stopping traffic due to unhealthy pods/replicaset and reconcile exists before completion

My proposal is either:
1. Properly Log the abort failure instead of silently failing and document
2. Fix abort so it will work no matter the rollout traffic setup
What do you think @zachaller 

Solution patch:
diff --git a/USERS.md b/USERS.md
index 2ab181bb..2ae9baac 100644
--- a/USERS.md
+++ b/USERS.md
@@ -32,6 +32,7 @@ Organizations below are **officially** using Argo Rollouts. Please send a PR wit
 1. [Hepsiburada](https://hepsiburada.com/)
 1. [Ibotta](https://home.ibotta.com/)
 1. [Intuit](https://www.intuit.com/)
+1. [Microsoft](https://microsoft.com/)
 1. [New Relic](https://newrelic.com/)
 1. [Nitro](https://www.gonitro.com)
 1. [Nozzle](https://nozzle.io)
diff --git a/docs/generated/kubectl-argo-rollouts/kubectl-argo-rollouts.md b/docs/generated/kubectl-argo-rollouts/kubectl-argo-rollouts.md
index 473f555d..13bdeb25 100644
--- a/docs/generated/kubectl-argo-rollouts/kubectl-argo-rollouts.md
+++ b/docs/generated/kubectl-argo-rollouts/kubectl-argo-rollouts.md
@@ -26,7 +26,7 @@ kubectl argo rollouts promote guestbook
 kubectl argo rollouts abort guestbook
 
 # Retry the guestbook rollout
-kubectl argo rollouts retry guestbook
+kubectl argo rollouts retry rollout guestbook
 ```
 
 ## Options
diff --git a/pkg/kubectl-argo-rollouts/cmd/cmd.go b/pkg/kubectl-argo-rollouts/cmd/cmd.go
index 24610550..db1748d2 100644
--- a/pkg/kubectl-argo-rollouts/cmd/cmd.go
+++ b/pkg/kubectl-argo-rollouts/cmd/cmd.go
@@ -41,7 +41,7 @@ const (
   %[1]s abort guestbook
 
   # Retry the guestbook rollout
-  %[1]s retry guestbook`
+  %[1]s retry rollout guestbook`
 )
 
 // NewCmdArgoRollouts returns new instance of rollouts command.
diff --git a/rollout/trafficrouting/istio/istio.go b/rollout/trafficrouting/istio/istio.go
index a0055cf0..1d25e55c 100644
--- a/rollout/trafficrouting/istio/istio.go
+++ b/rollout/trafficrouting/istio/istio.go
@@ -326,8 +326,13 @@ func (r *Reconciler) UpdateHash(canaryHash, stableHash string, additionalDestina
 	if r.rollout.Spec.Strategy.Canary.CanaryService == "" && r.rollout.Spec.Strategy.Canary.StableService == "" {
 
 		for _, rs := range r.replicaSets {
-			if *rs.Spec.Replicas > 0 && !replicasetutil.IsReplicaSetAvailable(rs) {
-				return fmt.Errorf("delaying destination rule switch: ReplicaSet %s not fully available", rs.Name)
+			if *rs.Spec.Replicas > 0 {
+				rsHash := rs.Labels[v1alpha1.DefaultRolloutUniqueLabelKey]
+				// Only check availability for ReplicaSets that will receive traffic
+				if (rsHash == stableHash || rsHash == canaryHash) && rsHash != "" && !replicasetutil.IsReplicaSetAvailable(rs) {
+					r.log.Infof("delaying destination rule switch: ReplicaSet %s not fully available", rs.Name)
+					return nil
+				}
 			}
 		}
 	}



Test patch:
diff --git a/rollout/trafficrouting/istio/istio_test.go b/rollout/trafficrouting/istio/istio_test.go
index db0cef87..731455d4 100644
--- a/rollout/trafficrouting/istio/istio_test.go
+++ b/rollout/trafficrouting/istio/istio_test.go
@@ -3200,3 +3200,143 @@ func TestGetHttpRouteIndexesToPatch(t *testing.T) {
 		assert.Nil(t, indexes)
 	})
 }
+
+// TestUpdateHashAbortScenarios tests both abort and non-abort scenarios for ReplicaSet availability checks
+func TestUpdateHashAbortScenarios(t *testing.T) {
+	createRollout := func(abort bool) *v1alpha1.Rollout {
+		ro := rolloutWithDestinationRule()
+		ro.Spec.Strategy.Canary.CanaryService = ""
+		ro.Spec.Strategy.Canary.StableService = ""
+		ro.Status.Abort = abort
+		return ro
+	}
+
+	createDestinationRuleObj := func() *unstructured.Unstructured {
+		return unstructuredutil.StrToUnstructuredUnsafe(`
+apiVersion: networking.istio.io/v1alpha3
+kind: DestinationRule
+metadata:
+  name: istio-destrule
+  namespace: default
+spec:
+  subsets:
+  - name: stable
+  - name: canary
+`)
+	}
+
+	createUnavailableReplicaSet := func(ro *v1alpha1.Rollout, hash string) *appsv1.ReplicaSet {
+		return &appsv1.ReplicaSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "rs-" + hash,
+				UID:       uuid.NewUUID(),
+				Namespace: metav1.NamespaceDefault,
+				Labels: map[string]string{
+					v1alpha1.DefaultRolloutUniqueLabelKey: hash,
+				},
+			},
+			Spec: appsv1.ReplicaSetSpec{
+				Replicas: func() *int32 { i := int32(1); return &i }(),
+				Template: ro.Spec.Template,
+			},
+			Status: appsv1.ReplicaSetStatus{
+				Replicas:          1,
+				AvailableReplicas: 0,
+			},
+		}
+	}
+
+	createAvailableReplicaSet := func(ro *v1alpha1.Rollout, hash string) *appsv1.ReplicaSet {
+		return &appsv1.ReplicaSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "rs-" + hash,
+				UID:       uuid.NewUUID(),
+				Namespace: metav1.NamespaceDefault,
+				Labels: map[string]string{
+					v1alpha1.DefaultRolloutUniqueLabelKey: hash,
+				},
+			},
+			Spec: appsv1.ReplicaSetSpec{
+				Replicas: func() *int32 { i := int32(1); return &i }(),
+				Template: ro.Spec.Template,
+			},
+			Status: appsv1.ReplicaSetStatus{
+				Replicas:          1,
+				AvailableReplicas: 1,
+			},
+		}
+	}
+
+	setupReconciler := func(ro *v1alpha1.Rollout, client *dynamicfake.FakeDynamicClient, rsList []*appsv1.ReplicaSet) *Reconciler {
+		vsvcLister, druleLister := getIstioListers(client)
+		r := NewReconciler(ro, client, record.NewFakeEventRecorder(), vsvcLister, druleLister, rsList)
+		client.ClearActions()
+		return r
+	}
+
+	t.Run("only checks relevant ReplicaSets for traffic routing", func(t *testing.T) {
+		ro := createRollout(false)
+		obj := createDestinationRuleObj()
+		client := testutil.NewFakeDynamicClient(obj)
+
+		stableRS := createAvailableReplicaSet(ro, "stable123")   // Available - will receive traffic
+		canaryRS := createUnavailableReplicaSet(ro, "canary456") // Unavailable - will receive traffic
+		otherRS := createUnavailableReplicaSet(ro, "other789")   // Unavailable - but won't receive traffic
+
+		rsList := []*appsv1.ReplicaSet{stableRS, canaryRS, otherRS}
+		r := setupReconciler(ro, client, rsList)
+
+		// This should return nil (delaying update) because canary RS (which will receive traffic) is not available
+		err := r.UpdateHash("canary456", "stable123")
+		assert.NoError(t, err)
+
+		// Verify no actions were taken because update was delayed
+		actions := client.Actions()
+		assert.Len(t, actions, 0)
+	})
+
+	t.Run("succeeds when all traffic-receiving ReplicaSets are available", func(t *testing.T) {
+		ro := createRollout(false)
+		obj := createDestinationRuleObj()
+		client := testutil.NewFakeDynamicClient(obj)
+
+		stableRS := createAvailableReplicaSet(ro, "stable123") // Available - will receive traffic
+		canaryRS := createAvailableReplicaSet(ro, "canary456") // Available - will receive traffic
+		otherRS := createUnavailableReplicaSet(ro, "other789") // Unavailable - but won't receive traffic
+
+		rsList := []*appsv1.ReplicaSet{stableRS, canaryRS, otherRS}
+		r := setupReconciler(ro, client, rsList)
+
+		// This should succeed because all ReplicaSets that will receive traffic are available
+		// otherRS being unavailable doesn't matter since it won't receive traffic
+		err := r.UpdateHash("canary456", "stable123")
+		assert.NoError(t, err)
+
+		// Verify DestinationRule was updated
+		actions := client.Actions()
+		assert.Len(t, actions, 1)
+		assert.Equal(t, "update", actions[0].GetVerb())
+	})
+
+	t.Run("works correctly during abort scenarios", func(t *testing.T) {
+		ro := createRollout(true) // Aborted rollout
+		obj := createDestinationRuleObj()
+		client := testutil.NewFakeDynamicClient(obj)
+
+		stableRS := createAvailableReplicaSet(ro, "stable123")
+		canaryRS := createUnavailableReplicaSet(ro, "canary456")
+		otherRS := createUnavailableReplicaSet(ro, "other789")
+
+		rsList := []*appsv1.ReplicaSet{stableRS, canaryRS, otherRS}
+		r := setupReconciler(ro, client, rsList)
+
+		// Since stable is available, this should succeed
+		err := r.UpdateHash("", "stable123") // Empty canary hash = no traffic to canary
+		assert.NoError(t, err)
+
+		// Verify DestinationRule was updated
+		actions := client.Actions()
+		assert.Len(t, actions, 1)
+		assert.Equal(t, "update", actions[0].GetVerb())
+	})
+}