Issue 1:
PR ##26089 moved some PD tests to flaky to resolve issue #26127 (and duplicate #26076). These tests started failing due to PR #21709.

@rootfs is working on a patch to de-flake the tests.

This issue is to track the fix and should be closed when:

The PD tests have been deflaked (verify by running the "pd.go" e2e tests back to back for a couple of hours to verify the issue is fixed).

The flaky label has been removed from the "pd.go" tests that were marked flaky in #26089

CC @kubernetes/sig-storage

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

kubeadm, kube-dns

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

# kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: AWS

OS (e.g. from /etc/os-release):

# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial

Kernel (e.g. uname -a):

# uname -a
Linux ip-10-1-24-223 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:29:54 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Install tools: kubeadm

Others:

What happened:

kubeadm failed to start kube-dns.

What you expected to happen:

kube-dns starts successfully.

How to reproduce it (as minimally and precisely as possible):

I followed the instructions. http://kubernetes.io/docs/getting-started-guides/kubeadm/

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
# sudo sh -c "cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF"
# sudo apt-get update -y
# sudo apt-get install -y docker.io kubelet kubeadm kubectl kubernetes-cni
# sudo kubeadm init

Anything else do we need to know:

# kubectl get pod --all-namespaces=true
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system dummy-2088944543-qjcd9 1/1 Running 0 32s
kube-system kube-apiserver-ip-10-1-24-144 1/1 Running 0 37s
kube-system kube-discovery-1150918428-d2nsx 1/1 Running 0 31s
kube-system kube-dns-654381707-vf4te 0/3 ContainerCreating 0 19s
kube-system kube-proxy-5vjdd 1/1 Running 0 19s

# kubectl get events -n kube-system
LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
1m 1m 1 dummy-2088944543-qjcd9 Pod Normal Scheduled {default-scheduler } Successfully assigned dummy-2088944543-qjcd9 to ip-10-1-24-144
1m 1m 1 dummy-2088944543-qjcd9 Pod spec.containers{dummy} Normal Pulled {kubelet ip-10-1-24-144} Container image "gcr.io/google_containers/pause-amd64:3.0" already present on machine
1m 1m 1 dummy-2088944543-qjcd9 Pod spec.containers{dummy} Normal Created {kubelet ip-10-1-24-144} Created container with docker id bee2cc05825c; Security:[seccomp=unconfined]
1m 1m 1 dummy-2088944543-qjcd9 Pod spec.containers{dummy} Normal Started {kubelet ip-10-1-24-144} Started container with docker id bee2cc05825c
1m 1m 1 dummy-2088944543 ReplicaSet Normal SuccessfulCreate {replicaset-controller } Created pod: dummy-2088944543-qjcd9
1m 1m 1 dummy Deployment Normal ScalingReplicaSet {deployment-controller } Scaled up replica set dummy-2088944543 to 1
1m 1m 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/etcd-amd64:2.2.5"
1m 1m 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/etcd-amd64:2.2.5"
1m 1m 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 8f0fdf66dbb5; Security:[seccomp=unconfined]
1m 1m 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 8f0fdf66dbb5
1m 1m 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-apiserver-amd64:v1.4.4"
1m 1m 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-apiserver-amd64:v1.4.4"
1m 1m 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 0926c48e85b0; Security:[seccomp=unconfined]
1m 1m 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 0926c48e85b0
1m 1m 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4"
1m 1m 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4"
1m 1m 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 46485fd67302; Security:[seccomp=unconfined]
1m 1m 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 46485fd67302
1m 1m 1 kube-discovery-1150918428-d2nsx Pod Normal Scheduled {default-scheduler } Successfully assigned kube-discovery-1150918428-d2nsx to ip-10-1-24-144
1m 1m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-discovery-amd64:1.0"
1m 1m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-discovery-amd64:1.0"
1m 1m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Created {kubelet ip-10-1-24-144} Created container with docker id bc5677a97797; Security:[seccomp=unconfined]
1m 1m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Started {kubelet ip-10-1-24-144} Started container with docker id bc5677a97797
1m 1m 1 kube-discovery-1150918428 ReplicaSet Normal SuccessfulCreate {replicaset-controller } Created pod: kube-discovery-1150918428-d2nsx
1m 1m 1 kube-discovery Deployment Normal ScalingReplicaSet {deployment-controller } Scaled up replica set kube-discovery-1150918428 to 1
1m 1m 1 kube-dns-654381707-vf4te Pod Normal Scheduled {default-scheduler } Successfully assigned kube-dns-654381707-vf4te to ip-10-1-24-144
1s 1m 68 kube-dns-654381707-vf4te Pod Warning FailedSync {kubelet ip-10-1-24-144} Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-654381707-vf4te_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-654381707-vf4te_kube-system(c37672ee-bb00-11e6-9cb2-06d39f467f73)\" using network plugins \"cni\": cni config unintialized; Skipping pod"

1m 1m 1 kube-dns-654381707 ReplicaSet Normal SuccessfulCreate {replicaset-controller } Created pod: kube-dns-654381707-vf4te
1m 1m 1 kube-dns Deployment Normal ScalingReplicaSet {deployment-controller } Scaled up replica set kube-dns-654381707 to 1
1m 1m 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-proxy-amd64:v1.4.4"
1m 1m 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-proxy-amd64:v1.4.4"
1m 1m 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Created {kubelet ip-10-1-24-144} Created container with docker id c24e0db7ab80; Security:[seccomp=unconfined]
59s 59s 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Started {kubelet ip-10-1-24-144} Started container with docker id c24e0db7ab80
1m 1m 1 kube-proxy DaemonSet Normal SuccessfulCreate {daemon-set } Created pod: kube-proxy-5vjdd
1m 1m 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-scheduler-amd64:v1.4.4"
1m 1m 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-scheduler-amd64:v1.4.4"
1m 1m 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 8f2d941c1e27; Security:[seccomp=unconfined]
1m 1m 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 8f2d941c1e27

1s 1m 68 kube-dns-654381707-vf4te Pod Warning FailedSync {kubelet ip-10-1-24-144} Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-654381707-vf4te_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-654381707-vf4te_kube-system(c37672ee-bb00-11e6-9cb2-06d39f467f73)\" using network plugins \"cni\": cni config unintialized; Skipping pod" 🤔

I tried kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml, but kube-dns didn't start.

# kubectl get pod --all-namespaces=true
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system dummy-2088944543-qjcd9 1/1 Running 0 1h
kube-system etcd-ip-10-1-24-144 1/1 Running 0 1h
kube-system kube-apiserver-ip-10-1-24-144 1/1 Running 0 1h
kube-system kube-controller-manager-ip-10-1-24-144 1/1 Running 0 1h
kube-system kube-discovery-1150918428-d2nsx 1/1 Running 0 1h
kube-system kube-dns-654381707-vf4te 0/3 ContainerCreating 0 1h
kube-system kube-flannel-ds-oxl8w 2/2 Running 0 4m
kube-system kube-proxy-5vjdd 1/1 Running 0 1h
kube-system kube-scheduler-ip-10-1-24-144 1/1 Running 0 1h

# kubectl get events -n kube-system
LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
59m 59m 1 dummy-2088944543-qjcd9 Pod Normal Scheduled {default-scheduler } Successfully assigned dummy-2088944543-qjcd9 to ip-10-1-24-144
59m 59m 1 dummy-2088944543-qjcd9 Pod spec.containers{dummy} Normal Pulled {kubelet ip-10-1-24-144} Container image "gcr.io/google_containers/pause-amd64:3.0" already present on machine
59m 59m 1 dummy-2088944543-qjcd9 Pod spec.containers{dummy} Normal Created {kubelet ip-10-1-24-144} Created container with docker id bee2cc05825c; Security:[seccomp=unconfined]
59m 59m 1 dummy-2088944543-qjcd9 Pod spec.containers{dummy} Normal Started {kubelet ip-10-1-24-144} Started container with docker id bee2cc05825c
59m 59m 1 dummy-2088944543 ReplicaSet Normal SuccessfulCreate {replicaset-controller } Created pod: dummy-2088944543-qjcd9
59m 59m 1 dummy Deployment Normal ScalingReplicaSet {deployment-controller } Scaled up replica set dummy-2088944543 to 1
1h 1h 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/etcd-amd64:2.2.5"
1h 1h 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/etcd-amd64:2.2.5"
1h 1h 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 8f0fdf66dbb5; Security:[seccomp=unconfined]
1h 1h 1 etcd-ip-10-1-24-144 Pod spec.containers{etcd} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 8f0fdf66dbb5
1h 1h 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-apiserver-amd64:v1.4.4"
1h 1h 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-apiserver-amd64:v1.4.4"
1h 1h 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 0926c48e85b0; Security:[seccomp=unconfined]
1h 1h 1 kube-apiserver-ip-10-1-24-144 Pod spec.containers{kube-apiserver} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 0926c48e85b0
1h 1h 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4"
1h 1h 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4"
1h 1h 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 46485fd67302; Security:[seccomp=unconfined]
1h 1h 1 kube-controller-manager-ip-10-1-24-144 Pod spec.containers{kube-controller-manager} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 46485fd67302
59m 59m 1 kube-discovery-1150918428-d2nsx Pod Normal Scheduled {default-scheduler } Successfully assigned kube-discovery-1150918428-d2nsx to ip-10-1-24-144
59m 59m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-discovery-amd64:1.0"
59m 59m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-discovery-amd64:1.0"
59m 59m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Created {kubelet ip-10-1-24-144} Created container with docker id bc5677a97797; Security:[seccomp=unconfined]
59m 59m 1 kube-discovery-1150918428-d2nsx Pod spec.containers{kube-discovery} Normal Started {kubelet ip-10-1-24-144} Started container with docker id bc5677a97797
59m 59m 1 kube-discovery-1150918428 ReplicaSet Normal SuccessfulCreate {replicaset-controller } Created pod: kube-discovery-1150918428-d2nsx
59m 59m 1 kube-discovery Deployment Normal ScalingReplicaSet {deployment-controller } Scaled up replica set kube-discovery-1150918428 to 1
59m 59m 1 kube-dns-654381707-vf4te Pod Normal Scheduled {default-scheduler } Successfully assigned kube-dns-654381707-vf4te to ip-10-1-24-144
1m 59m 3489 kube-dns-654381707-vf4te Pod Warning FailedSync {kubelet ip-10-1-24-144} Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-654381707-vf4te_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-654381707-vf4te_kube-system(c37672ee-bb00-11e6-9cb2-06d39f467f73)\" using network plugins \"cni\": cni config unintialized; Skipping pod"

1s 1m 70 kube-dns-654381707-vf4te Pod Warning FailedSync {kubelet ip-10-1-24-144} Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-654381707-vf4te_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-654381707-vf4te_kube-system(c37672ee-bb00-11e6-9cb2-06d39f467f73)\" using network plugins \"cni\": open /run/flannel/subnet.env: no such file or directory; Skipping pod"

59m 59m 1 kube-dns-654381707 ReplicaSet Normal SuccessfulCreate {replicaset-controller } Created pod: kube-dns-654381707-vf4te
59m 59m 1 kube-dns Deployment Normal ScalingReplicaSet {deployment-controller } Scaled up replica set kube-dns-654381707 to 1
2m 2m 1 kube-flannel-ds-oxl8w Pod spec.containers{kube-flannel} Normal Pulling {kubelet ip-10-1-24-144} pulling image "quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64"
1m 1m 1 kube-flannel-ds-oxl8w Pod spec.containers{kube-flannel} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64"
1m 1m 1 kube-flannel-ds-oxl8w Pod spec.containers{kube-flannel} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 3f87aa05dd39; Security:[seccomp=unconfined]
1m 1m 1 kube-flannel-ds-oxl8w Pod spec.containers{kube-flannel} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 3f87aa05dd39
1m 1m 1 kube-flannel-ds-oxl8w Pod spec.containers{install-cni} Normal Pulled {kubelet ip-10-1-24-144} Container image "quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64" already present on machine
1m 1m 1 kube-flannel-ds-oxl8w Pod spec.containers{install-cni} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 95c33cbcc458; Security:[seccomp=unconfined]
1m 1m 1 kube-flannel-ds-oxl8w Pod spec.containers{install-cni} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 95c33cbcc458
2m 2m 1 kube-flannel-ds DaemonSet Normal SuccessfulCreate {daemon-set } Created pod: kube-flannel-ds-oxl8w
59m 59m 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-proxy-amd64:v1.4.4"
59m 59m 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-proxy-amd64:v1.4.4"
59m 59m 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Created {kubelet ip-10-1-24-144} Created container with docker id c24e0db7ab80; Security:[seccomp=unconfined]
59m 59m 1 kube-proxy-5vjdd Pod spec.containers{kube-proxy} Normal Started {kubelet ip-10-1-24-144} Started container with docker id c24e0db7ab80
59m 59m 1 kube-proxy DaemonSet Normal SuccessfulCreate {daemon-set } Created pod: kube-proxy-5vjdd
1h 1h 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Pulling {kubelet ip-10-1-24-144} pulling image "gcr.io/google_containers/kube-scheduler-amd64:v1.4.4"
1h 1h 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Pulled {kubelet ip-10-1-24-144} Successfully pulled image "gcr.io/google_containers/kube-scheduler-amd64:v1.4.4"
1h 1h 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Created {kubelet ip-10-1-24-144} Created container with docker id 8f2d941c1e27; Security:[seccomp=unconfined]
1h 1h 1 kube-scheduler-ip-10-1-24-144 Pod spec.containers{kube-scheduler} Normal Started {kubelet ip-10-1-24-144} Started container with docker id 8f2d941c1e27