Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): canary, too many pods, pods dying, label selector

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.2.0

Environment:

Cloud provider or hardware configuration: baremetal

OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core)

Kernel (e.g. uname -a): 4.4.24-1

Install tools:

Others: flannel (0.5.3)

What happened:

I am trying to deploy a canary as described at http://kubernetes.io/docs/user-guide/managing-deployments/#canary-deployments. I have 2 deployments which are identical except for their names and # of replicas. Deployment A has replicas=6 and deployment B has replicas=1, both having the same app label. I have one service which includes both deployments via the label selector. I am able to launch deployment A and everything runs fine. However when I launch deployment B, I notice 3 pods are created for the deployment. Even more problematic is it seems the pods in deployment B are in a cycle of dying and being recreated. The interesting thing is, if I simply change the app label on deployment B and relaunch it, the correct number of replicas are created and the pod stays alive. But obviously that won't work for canarying.

What you expected to happen:

Canary deployment should honor number of replicas and be stable.

How to reproduce it (as minimally and precisely as possible):

Create 2 deployments that share the same app label and 1 service which selects for the app label. Launch first deployment and service. Launch second deployment and observe above.

Anything else do we need to know:

While looking at the kubelet logs to see why the pods might be dying I see things like below related to the pods going up and down.

Error running pod "server-canary-408865492-yf1g4_qa(d2c8e37c-9428-11e6-8c1b-90e2bac3873c)" container "configurator": runContainer: API error (404): {"message":"oci runtime error: container_linux.go:1215: running lstat on namespace path "/proc/28386/ns/ipc" caused "lstat /proc/28386/ns/ipc: no such file or directory""}

Error running pod "408865492-ifa0s_qa(d1727ff2-9428-11e6-8c1b-90e2bac3873c)" container "server": GenerateRunContainerOptions: impossible: cannot find the mounted volumes for pod "server-canary-408865492-ifa0s_qa(d1727ff2-9428-11e6-8c1b-90e2bac3873c)"

kubelet[17062]: , failed to "StartContainer" for "blah" with RunContainerError: "runContainer: API error (409): {"message":"cannot join network of a non running container: 947f639bbcccc5643a4812833bf5a9797bc36f6539c1677629c3ac9aa634f013"}\n"

pod_workers.go:138] Error syncing pod 3d082c94-942c-11e6-8c1b-90e2bac3873c, skipping: [failed to "StartContainer" for "server" with RunContainerError: "runContainer: API error (500): {"message":"oci runtime error: process_linux.go:245: running exec setns process for init caused \"exit status 1\""}\n"

Issue 2:
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce/458/

Failed: [k8s.io] Downward API volume should update labels on modification [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:113
Timed out after 65.000s.
Expected
<string>: content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"
content of file "/etc/labels": key1="value1"
key2="value2"

to contain substring
<string>: key3="value3"

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:112

Previous issues for this test: #28416 #31055 #33627