Change:
newStatus := calculateStatus(rc, filteredPods, manageReplicasErr) // Always updates status as pods come up or die. <del> if err := updateReplicationControllerStatus(rm.kubeClient.Core().ReplicationControllers(rc.Namespace), *rc, newStatus); err != nil { </del> <ins> updatedRC, err := updateReplicationControllerStatus(rm.kubeClient.Core().ReplicationControllers(rc.Namespace), *rc, newStatus) if err != nil { </ins> // Multiple things could lead to this update failing.  Returning an error causes a requeue without forcing a hotloop return err } <ins> // Resync the ReplicationController after MinReadySeconds as a last line of defense to guard against clock-skew. if manageReplicasErr == nil && updatedRC.Spec.MinReadySeconds > 0 && updatedRC.Status.ReadyReplicas == *(updatedRC.Spec.Replicas) && updatedRC.Status.AvailableReplicas != *(updatedRC.Spec.Replicas) { rm.enqueueControllerAfter(updatedRC, time.Duration(updatedRC.Spec.MinReadySeconds)*time.Second) } </ins> return manageReplicasErr }

Discussion:
Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.