Change:
A developer accidentally made two deployments with labels that overlapped, such that one of the deployment's selectors matched the pods from both deployments. In the repro below the labels are exactly the same, but this also happened one one set of labels is a subset of another. Expected: An error? Actual: Cluster CPU went past 100% and everything else broke. Crazy thrashing with kubernetes shutting down pods & restarting new ones so quickly that we could not even get logs. They never made it past the Pending state. We are running kubernetes 1.2 on GKE. Here's a minimal repro, just apply it (on a cluster you don't need! D: )
See also , . My guess is that this is due to pod relabeling. cc
this shouldn't be an issue in 1.4 anymore
I just ran into this issue and I'm on v1.4.4. My cluster goes into a crazy Pod creation frenzy even though the number of Pod replicas in both the deployments is set to 1 because one of the overlapping Deployment seems to be "redeploying" itself thousands of times. On a single node k8s it quickly reached the limit of 110 pods and makes the cluster unusable until I delete one of the deployments. This is a very nasty bug as it will overrun the whole cluster. I my case the matching labels are only partial. Both deployments have the label "run=nginx" but one of them has a second selected label as well. What happens is that one of the deployments gets the annotation: "" And that Deployment goes into a loop of "redeploying" itself so it tries to create hundreds of pods and overruns the cluster. Looking at the live Deployment yaml it shows "generation : 6616". I've attached the two test Deployments files to reproduce this and I noticed that the ORDER you deploy them in MATTERS! $ kubectl create -f deployment- (and wait until it is fully deployed) $ kubectl create -f deployment- (and watch all hell break loose) I recommend using a disposable cluster even though, if you are quick enough, you can recover by deleting one of the Deployments (it will take a while!) and then its ReplicaSet. Client/Server versions: Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Yes it's a bug. deployment-one has selector & label run=my-nginx deployment-two has selector & label run=my-nginx, other-label=somevalue deployment-two was updated later In the code we look for overlapping deployments by checking if one's selector matches the other's label: In this case, deployment-one selector matches deployment-two (but not the other way around), so only deployment-one knows there's an overlapping deployment. The we choose the deployment that's last updated to ignore (skip syncing): In this case, deployment-one assumes deployment-two will skip syncing itself, so deployment-one continues. But deployment-two doesn't know there's an overlapping deployment. Updating the statement for finding overlapping deployment should solve it.
So you mean that the original/older Deployment always "wins" and the newer deployed one doesn't get any Pods launched or the Deployment "relaunched" even though after deployment the older Deployment is the one in violation (like in my use case)?
Sent the
In this case they overlap with each other (I mean, not only the older one is in violation). The one whose selector is last updated should be the one which stops syncing. Otherwise it'll be surprising when someone adds a new deployment or update a deployment's selector causes my existing deployment to stop working. The bug is because the latter one doesn't know it overlap with the former one. With the fix it can detect other overlapping deployments correctly.
Hello Guys! I think that I have a relevant question related to this issue. I would appreciate any answer. That configuration, which generate this bad behavior. I mean to have a common label between two deploys wouldn't be required to implement a canary release? If the canary release was just a different nginx's version? I am asking this because I have this configuration ,even though I don't have this mess hapening, my deployment never reachs my desired state. If I describe this deployment I got a warning: !!!WARNING!!! This deployment has overlapping label selector with deployment "my-nginx" and won't behave as expected. Please fix it before continue. I am using kubernetes 1.5.1, have I miss something related to concept or implementation of canary release?
can you post the deployment yaml files you are running? Seems like they are overlapping. Are both deployments broken or is it just one of them?
Hello Just one was broken, unfortunately I overwrite the files with another deployment configuration. Probably. I was my mistake, because now everything is working. I got my canary release through the files that follows: I am sharing because I got here ins this thread with this problem.These configurations could be useful to someone with the same problem. Thank you for your attention to my question. Deployment stable apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ise spec: replicas: 3 template: metadata: labels: app: "ise" track: "stable" spec: containers: name: "nginx" image: "nginx:1.8.1" ports: containerPort: 80 lifecycle: preStop: exec: command: ["/usr/sbin/nginx","-s","quit"] livenessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1 Deployment canary apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ise-canary spec: replicas: 1 template: metadata: labels: app: "ise" track: "canary" spec: containers: name: "nginx" image: "nginx:1.9.12" ports: containerPort: 80 lifecycle: preStop: exec: command: ["/usr/sbin/nginx","-s","quit"] livenessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1 Service apiVersion: v1 kind: Service metadata: name: ise spec: ports: port: 80 selector: app: ise type: LoadBalancer
Between two overlapping deployments, one being broken is expected today - you should also see an event about it. This won't be an issue in releases after 1.6. I am assuming deployments in old clusters (1.6+) in the future will contain owner references plus maybe we will have selector generation. Related discussion in and , Adriano S. Fonseca < :

Discussion:
// the newer overlapping ones (only sync the oldest one). New/old is determined by when the // deployment's selector is last updated. func (dc *DeploymentController) handleOverlap(d *extensions.Deployment) error { <del> selector, err := unversioned.LabelSelectorAsSelector(d.Spec.Selector) if err != nil { return fmt.Errorf("deployment %s/%s has invalid label selector: %v", d.Namespace, d.Name, err) } </del> deployments, err := dc.dLister.Deployments(d.Namespace).List(labels.Everything()) if err != nil { return fmt.Errorf("error listing deployments in namespace %s: %v", d.Namespace, err) } overlapping := false for _, other := range deployments { <del> if !selector.Empty() && selector.Matches(labels.Set(other.Spec.Template.Labels)) && d.UID != other.UID { </del> <ins> foundOverlaps, err := util.OverlapsWith(d, other) if err != nil { return err } if foundOverlaps { </ins> deploymentCopy, err := util.DeploymentDeepCopy(other) if err != nil { return err

} return ti.Before(tj) } <ins> // OverlapsWith returns true when two given deployments are different and overlap with each other func OverlapsWith(current, other *extensions.Deployment) (bool, error) { if current.UID == other.UID { return false, nil } currentSelector, err := unversioned.LabelSelectorAsSelector(current.Spec.Selector) if err != nil { return false, fmt.Errorf("deployment %s/%s has invalid label selector: %v", current.Namespace, current.Name, err) } otherSelector, err := unversioned.LabelSelectorAsSelector(other.Spec.Selector) if err != nil { // Broken selectors from other deployments shouldn't block current deployment. Just log the error and continue. glog.V(2).Infof("Skip overlapping check: deployment %s/%s has invalid label selector: %v", other.Namespace, other.Name, err) return false, nil } return (!currentSelector.Empty() && currentSelector.Matches(labels.Set(other.Spec.Template.Labels))) || (!otherSelector.Empty() && otherSelector.Matches(labels.Set(current.Spec.Template.Labels))), nil } </ins>

Expect(err).NotTo(HaveOccurred()) deploymentName = "second-deployment" By(fmt.Sprintf("Creating deployment %q with overlapping selector", deploymentName)) <ins> podLabels["other-label"] = "random-label" </ins> d = newDeployment(deploymentName, replicas, podLabels, nginxImageName, nginxImage, extensions.RollingUpdateDeploymentStrategyType, nil) deployOverlapping, err := c.Extensions().Deployments(ns).Create(d) Expect(err).NotTo(HaveOccurred(), "Failed creating the second deployment")