Change:
The scenario is a single replica with an attached EBS. The create seems to happen right after the delete not waiting for the pod to be terminated. This results in the two kubelets fighting over the EBS which generally results in AWS becoming unhappy meaning the EBS has to be force detached. From the docs I would have thought deployments would wait until the pod is actually killed. Is this a bug or am i misinterpreting the docs?
As a work around, what happens if you use a rolling update with MaxSurge=0 and MaxUnavailable=1?
Just assigned for triage, not to fix the issue. Can clear your assignment once this is triaged.
Deployments with size of 1 should be the same (both in that it's really a Recreate and it has the same problem even though it only matters for Recreate), larger deployments should be rolling.
It's because Deployment scales old RS down to 0 (=0, and RS controller will observe this change later and then kill its pods) and then scales up the new one. So it's possible there's a race here.
Opened that partially fixes this.
Why it's closed? It's not fixed: Recreate still doesn't wait for termination.
what version are you using?
1.6.2-00 from deb package. I'm changing the Deployment to use different configmap. K8s recreates pod, but the new one can't listen on ports that are probably used by the old one (using ). Both are appearing in at the same time (old one is 'Terminating').
This regressed in 1.6 but was fixed again in I don't think it was cherry-picked back to 1.6 though and the cherry-pick is not clean. are you interested in backporting it to the release-1.6 branch?
the cherry-pick isn't clean mostly because of comments. Don't have time to actually build and test the PR though.
I think this is still an issue:
I cannot reproduce it anymore, can you provide a reproducer?
Not easily at the moment -- we ended up changing our pods to handle multiple running simultaneously gracefully.
Hi All, I'm still seeing this on the following version: I am using recreate and can see my current pod go into a terminating state and my new pod come up. It is my understanding that the new pod should not come up until the old pod is terminated completely.
Seeing this on 1.14.3 too - AWS hardware went bad and stuff was in terminating (but still running) and our stateful singleton was brought back up alongside
We see the same problem with 1.14.3 where the old pod is Terminating and the new pod is in ContainerCreating even if we set the Got problem with Volume is already used by pod(s)...
The easiest way I've found to reproduce this is by using bash interactively - it swallows SIGTERM giving a bigger window to observe the behavior. One thing to note, the behavior is different with an explicit delete and one caused by a changing annotation. When changing an annotation, the new pod won't be scheduled until the old one is completely gone, but an explicit delete will create the new pod right away. That said, I don't know how it will behave with other events, say a node going away or being drained. Here is a way to reproduce it. Delete the pod () and watch the new one spin up right away.
recreate deployments wait for the termination of an old version only during a deployment upgrade. The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use . can you please provide a reproducer? Thanks.
Same issue here with version v1.17.6: Deployment with "Recreate" strategy: If I try to delete a pod, the old pod is still Terminating, the new pod has already get initialized and Running:
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see the comment right above you "recreate deployments wait for the termination of an old version only during a deployment upgrade. The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use statefulsets."

Discussion:
import ( "k8s.io/kubernetes/pkg/apis/extensions" <ins> unversionedclient "k8s.io/kubernetes/pkg/client/unversioned" </ins> "k8s.io/kubernetes/pkg/controller" <ins> rsutil "k8s.io/kubernetes/pkg/util/replicaset" "k8s.io/kubernetes/pkg/util/wait" </ins> ) // rolloutRecreate implements the logic for recreating a replica set.

return err } allRSs := append(oldRSs, newRS) <ins> activeOldRSs := controller.FilterActiveReplicaSets(oldRSs) </ins> // scale down old replica sets <del> scaledDown, err := dc.scaleDownOldReplicaSetsForRecreate(controller.FilterActiveReplicaSets(oldRSs), deployment) </del> <ins> scaledDown, err := dc.scaleDownOldReplicaSetsForRecreate(activeOldRSs, deployment) </ins> if err != nil { return err }

return dc.updateDeploymentStatus(allRSs, newRS, deployment) } <ins> // Wait for all old replica set to scale down to zero. if err := dc.waitForInactiveReplicaSets(activeOldRSs); err != nil { return err } </ins> // If we need to create a new RS, create it now // TODO: Create a new RS without re-listing all RSs. if newRS == nil {

return scaled, nil } <ins> // waitForInactiveReplicaSets will wait until all passed replica sets are inactive and have been noticed // by the replica set controller. func (dc *DeploymentController) waitForInactiveReplicaSets(oldRSs []*extensions.ReplicaSet) error { for i := range oldRSs { rs := oldRSs[i] condition := rsutil.ReplicaSetIsInactive(dc.client.Extensions(), rs) if err := wait.ExponentialBackoff(unversionedclient.DefaultRetry, condition); err != nil { return err } } return nil } </ins> // scaleUpNewReplicaSetForRecreate scales up new replica set when deployment strategy is "Recreate" func (dc *DeploymentController) scaleUpNewReplicaSetForRecreate(newRS *extensions.ReplicaSet, deployment *extensions.Deployment) (bool, error) { scaled, _, err := dc.scaleReplicaSetAndRecordEvent(newRS, deployment.Spec.Replicas, deployment)

return selector.Matches(podLabelsSelector) }, nil } <ins> // ReplicaSetIsInactive returns a condition that will be true when a replica set is inactive ie. // it has zero running replicas. func ReplicaSetIsInactive(c unversionedextensions.ExtensionsInterface, replicaSet *extensions.ReplicaSet) wait.ConditionFunc { // If we're given a ReplicaSet where the status lags the spec, it either means that the // ReplicaSet is stale, or that the ReplicaSet manager hasn't noticed the update yet. // Polling status.Replicas is not safe in the latter case. desiredGeneration := replicaSet.Generation return func() (bool, error) { rs, err := c.ReplicaSets(replicaSet.Namespace).Get(replicaSet.Name) if err != nil { return false, err } return rs.Status.ObservedGeneration >= desiredGeneration && rs.Spec.Replicas == 0 && rs.Status.Replicas == 0 && rs.Status.FullyLabeledReplicas == 0, nil } } </ins>