Change:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-08-11T20:29:08Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: VMware vSphere 5.5 OS (e.g. from /etc/os-release): Debian 8 jessie (via referenced by Getting Started Guide) Kernel (e.g. ): Linux kubernetes-minion-1 3.16.0-4-amd64 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x8664 GNU/Linux Install tools: bundled cluster/kube- script Others: What happened: Following the 'Getting Started' guide exactly results in successful deployment of 1 master and 4 minion machines in vSphere, however the first three minions lack the ip routing table entry necessary to reach the pod network on the 4th minion (ie. 10.244.3.0/24). Editing cluster/vsphere/config- and Increasing NUMNODES to a higher value, such as 8, also results in the expected 8 minions however as previously minions 1 through 7 are missing the IP routing table for the pod network on the 8th minion (ie. 10.244.7.0/24). Manually ssh'ing in and adding the missing route on all the appropriate minions restores the correct behavior of the cluster. What you expected to happen: Each minion must have routing table entries for the pod networks of all other minions in the cluster, otherwise Bad Things(tm) will happen. How to reproduce it (as minimally and precisely as possible): Exactly follow the quickstart documentation at Anything else do we need to know: Console logs and routing table inspection of resulting minion nodes are attached below for deployment of both a 4 node and a separate 8 node cluster. root cat cluster/vsphere/config- | grep NUMNODES NUMNODES=4 root KUBERNETESPROVIDER=vsphere cluster/kube- ... Starting cluster using provider: vsphere ... calling verify-prereqs ... calling kube-up Identity : (rsa w/o comment) Starting master VM (this can take a minute)... kubernetes-server-linux- 100% 351MB 175.7MB/s 00:02 kubernetes- 100% 45KB 44.6KB/s 00:00 uploaded /tmp/kubernetes.q1La14/master- to /tmp/master- Using master: kubernetes-master (external IP: 172.16.6.156) Starting node VMs (this can take a minute)... uploaded /tmp/kubernetes.q1La14/node-start-3.sh to /tmp/node-start-3.sh uploaded /tmp/kubernetes.q1La14/node-start-0.sh to /tmp/node-start-0.sh uploaded /tmp/kubernetes.q1La14/node-start-1.sh to /tmp/node-start-1.sh uploaded /tmp/kubernetes.q1La14/node-start-2.sh to /tmp/node-start-2.sh Found kubernetes-minion-1 at 172.16.6.141 Found kubernetes-minion-2 at 172.16.6.169 Found kubernetes-minion-3 at 172.16.6.217 Found kubernetes-minion-4 at 172.16.6.137 Waiting for salt-master to be up on kubernetes-master ... This may take several minutes. Bound to 60 attempts........ [salt-master running] Waiting for all packages to be installed on kubernetes-master ... This may take several minutes. Bound to 60 attempts................... [sudo salt "kubernetes-master" state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed Waiting for salt-minion to be up on kubernetes-minion-1 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-1 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-1 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-2 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-2 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-2 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-3 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-3 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-3 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-4 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-4 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-4 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for master and node initialization. This will continually check to see if the API for kubernetes is reachable. This might loop forever if there was some uncaught error during start up. OK Waiting for kubernetes-minion-1 to become available... OK Waiting for kubernetes-minion-2 to become available... OK Waiting for kubernetes-minion-3 to become available... OK Waiting for kubernetes-minion-4 to become available... OK Setting up routes check if cbr0 bridge is ready on kubernetes-minion-1 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-2 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-3 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-4 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed finding network of cbr0 bridge on node kubernetes-minion-1 finding network of cbr0 bridge on node kubernetes-minion-2 finding network of cbr0 bridge on node kubernetes-minion-3 finding network of cbr0 bridge on node kubernetes-minion-4 setting up routes for kubernetes-minion-1setting up routes for kubernetes-minion-2setting up routes for kubernetes-minion-3setting up routes for kubernetes-minion-4Kubernetes cluster created. cluster "vspherekubernetes" set. user "vspherekubernetes" set. context "vspherekubernetes" set. switched to context "vspherekubernetes". Wrote config for vspherekubernetes to Sanity checking cluster... Kubernetes cluster is running. The master is running at: https://172.16.6.156 The user name and password to use is located in ... calling validate-cluster Waiting for 4 ready nodes. 3 ready nodes, 4 registered. Retrying. Waiting for 4 ready nodes. 3 ready nodes, 4 registered. Retrying. Found 4 node(s). NAME STATUS AGE 172.16.6.137 Ready 36s 172.16.6.141 Ready 2m 172.16.6.169 Ready 2m 172.16.6.217 Ready 1m Validate output: NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-1 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} Cluster validation succeeded Done, listing cluster services: Kubernetes master is running at https://172.16.6.156 KubeDNS is running at https://172.16.6.156/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://172.16.6.156/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard To further debug and diagnose cluster problems, use 'kubectl cluster-info dump' default via 172.16.6.1 dev eth0 244.0.0/24 dev cbr0 proto kernel scope link src 10.244.0.1 244.1.0/24 via 172.16.6.169 dev eth0 244.2.0/24 via 172.16.6.217 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.141 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.137 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.141 dev eth0 244.1.0/24 dev cbr0 proto kernel scope link src 10.244.1.1 244.2.0/24 via 172.16.6.217 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.169 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.137 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.141 dev eth0 244.1.0/24 via 172.16.6.169 dev eth0 244.2.0/24 dev cbr0 proto kernel scope link src 10.244.2.1 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.217 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.137 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.141 dev eth0 244.1.0/24 via 172.16.6.169 dev eth0 244.2.0/24 via 172.16.6.217 dev eth0 244.3.0/24 dev cbr0 proto kernel scope link src 10.244.3.1 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.137 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 NUMNODES=8 root KUBERNETESPROVIDER=vsphere cluster/kube- ... Starting cluster using provider: vsphere ... calling verify-prereqs ... calling kube-up Identity : (rsa w/o comment) Starting master VM (this can take a minute)... kubernetes-server-linux- 100% 351MB 117.1MB/s 00:03 kubernetes- 100% 45KB 44.6KB/s 00:00 uploaded /tmp/kubernetes.3bDOMi/master- to /tmp/master- Using master: kubernetes-master (external IP: 172.16.6.158) Starting node VMs (this can take a minute)... uploaded /tmp/kubernetes.3bDOMi/node-start-4.sh to /tmp/node-start-4.sh uploaded /tmp/kubernetes.3bDOMi/node-start-7.sh to /tmp/node-start-7.sh uploaded /tmp/kubernetes.3bDOMi/node-start-5.sh to /tmp/node-start-5.sh uploaded /tmp/kubernetes.3bDOMi/node-start-0.sh to /tmp/node-start-0.sh uploaded /tmp/kubernetes.3bDOMi/node-start-2.sh to /tmp/node-start-2.sh uploaded /tmp/kubernetes.3bDOMi/node-start-6.sh to /tmp/node-start-6.sh uploaded /tmp/kubernetes.3bDOMi/node-start-3.sh to /tmp/node-start-3.sh uploaded /tmp/kubernetes.3bDOMi/node-start-1.sh to /tmp/node-start-1.sh Found kubernetes-minion-1 at 172.16.6.239 Found kubernetes-minion-2 at 172.16.6.130 Found kubernetes-minion-3 at 172.16.6.124 Found kubernetes-minion-4 at 172.16.6.198 Found kubernetes-minion-5 at 172.16.6.84 Found kubernetes-minion-6 at 172.16.6.160 Found kubernetes-minion-7 at 172.16.6.208 Found kubernetes-minion-8 at 172.16.6.98 Waiting for salt-master to be up on kubernetes-master ... This may take several minutes. Bound to 60 attempts.......... [salt-master running] Waiting for all packages to be installed on kubernetes-master ... This may take several minutes. Bound to 60 attempts....................... [sudo salt "kubernetes-master" state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed Waiting for salt-minion to be up on kubernetes-minion-1 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-1 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-1 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-2 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-2 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-2 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-3 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-3 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-3 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-4 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-4 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-4 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-5 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-5 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-5 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-6 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-6 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-6 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-7 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-7 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-7 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for salt-minion to be up on kubernetes-minion-8 .... This may take several minutes. Bound to 60 attempts [salt-minion running] Waiting for all salt packages to be installed on kubernetes-minion-8 .... This may take several minutes. Bound to 60 attempts [sudo salt kubernetes-minion-8 state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"] passed OK Waiting for master and node initialization. This will continually check to see if the API for kubernetes is reachable. This might loop forever if there was some uncaught error during start up. OK Waiting for kubernetes-minion-1 to become available... OK Waiting for kubernetes-minion-2 to become available... OK Waiting for kubernetes-minion-3 to become available... OK Waiting for kubernetes-minion-4 to become available... OK Waiting for kubernetes-minion-5 to become available... OK Waiting for kubernetes-minion-6 to become available... OK Waiting for kubernetes-minion-7 to become available... OK Waiting for kubernetes-minion-8 to become available... OK Setting up routes check if cbr0 bridge is ready on kubernetes-minion-1 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-2 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-3 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-4 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-5 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-6 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-7 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed check if cbr0 bridge is ready on kubernetes-minion-8 This may take several minutes. Bound to 60 attempts [sudo ifconfig cbr0 | grep -oP "inet addr:KS+"] passed finding network of cbr0 bridge on node kubernetes-minion-1 finding network of cbr0 bridge on node kubernetes-minion-2 finding network of cbr0 bridge on node kubernetes-minion-3 finding network of cbr0 bridge on node kubernetes-minion-4 finding network of cbr0 bridge on node kubernetes-minion-5 finding network of cbr0 bridge on node kubernetes-minion-6 finding network of cbr0 bridge on node kubernetes-minion-7 finding network of cbr0 bridge on node kubernetes-minion-8 setting up routes for kubernetes-minion-1setting up routes for kubernetes-minion-2setting up routes for kubernetes-minion-3setting up routes for kubernetes-minion-4setting up routes for kubernetes-minion-5setting up routes for kubernetes-minion-6setting up routes for kubernetes-minion-7setting up routes for kubernetes-minion-8Kubernetes cluster created. cluster "vspherekubernetes" set. user "vspherekubernetes" set. context "vspherekubernetes" set. switched to context "vspherekubernetes". Wrote config for vsphere_kubernetes to Sanity checking cluster... Kubernetes cluster is running. The master is running at: https://172.16.6.158 The user name and password to use is located in ... calling validate-cluster Waiting for 8 ready nodes. 7 ready nodes, 8 registered. Retrying. Waiting for 8 ready nodes. 7 ready nodes, 8 registered. Retrying. Found 8 node(s). NAME STATUS AGE 172.16.6.124 Ready 4m 172.16.6.130 Ready 4m 172.16.6.160 Ready 2m 172.16.6.198 Ready 3m 172.16.6.208 Ready 1m 172.16.6.239 Ready 5m 172.16.6.84 Ready 2m 172.16.6.98 Ready 42s Validate output: NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-1 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} Cluster validation succeeded Done, listing cluster services: Kubernetes master is running at https://172.16.6.158 KubeDNS is running at https://172.16.6.158/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://172.16.6.158/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Found kubernetes-minion-1 at 172.16.6.239 Found kubernetes-minion-2 at 172.16.6.130 Found kubernetes-minion-3 at 172.16.6.124 Found kubernetes-minion-4 at 172.16.6.198 Found kubernetes-minion-5 at 172.16.6.84 Found kubernetes-minion-6 at 172.16.6.160 Found kubernetes-minion-7 at 172.16.6.208 Found kubernetes-minion-8 at 172.16.6.98 default via 172.16.6.1 dev eth0 244.0.0/24 dev cbr0 proto kernel scope link src 10.244.0.1 244.1.0/24 via 172.16.6.130 dev eth0 244.2.0/24 via 172.16.6.124 dev eth0 244.3.0/24 via 172.16.6.198 dev eth0 244.4.0/24 via 172.16.6.84 dev eth0 244.5.0/24 via 172.16.6.160 dev eth0 244.6.0/24 via 172.16.6.208 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.239 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.98 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.239 dev eth0 244.1.0/24 dev cbr0 proto kernel scope link src 10.244.1.1 244.2.0/24 via 172.16.6.124 dev eth0 244.3.0/24 via 172.16.6.198 dev eth0 244.4.0/24 via 172.16.6.84 dev eth0 244.5.0/24 via 172.16.6.160 dev eth0 244.6.0/24 via 172.16.6.208 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.130 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.98 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.239 dev eth0 244.1.0/24 via 172.16.6.130 dev eth0 244.2.0/24 dev cbr0 proto kernel scope link src 10.244.2.1 244.3.0/24 via 172.16.6.198 dev eth0 244.4.0/24 via 172.16.6.84 dev eth0 244.5.0/24 via 172.16.6.160 dev eth0 244.6.0/24 via 172.16.6.208 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.124 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.98 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.239 dev eth0 244.1.0/24 via 172.16.6.130 dev eth0 244.2.0/24 via 172.16.6.124 dev eth0 244.3.0/24 dev cbr0 proto kernel scope link src 10.244.3.1 244.4.0/24 via 172.16.6.84 dev eth0 244.5.0/24 via 172.16.6.160 dev eth0 244.6.0/24 via 172.16.6.208 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.198 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.98 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.239 dev eth0 244.1.0/24 via 172.16.6.130 dev eth0 244.2.0/24 via 172.16.6.124 dev eth0 244.3.0/24 via 172.16.6.198 dev eth0 244.4.0/24 dev cbr0 proto kernel scope link src 10.244.4.1 244.5.0/24 via 172.16.6.160 dev eth0 244.6.0/24 via 172.16.6.208 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.84 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.98 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.239 dev eth0 244.1.0/24 via 172.16.6.130 dev eth0 244.2.0/24 via 172.16.6.124 dev eth0 244.3.0/24 via 172.16.6.198 dev eth0 244.4.0/24 via 172.16.6.84 dev eth0 244.5.0/24 dev cbr0 proto kernel scope link src 10.244.5.1 244.6.0/24 via 172.16.6.208 dev eth0 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.160 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.98 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.239 dev eth0 244.1.0/24 via 172.16.6.130 dev eth0 244.2.0/24 via 172.16.6.124 dev eth0 244.3.0/24 via 172.16.6.198 dev eth0 244.4.0/24 via 172.16.6.84 dev eth0 244.5.0/24 via 172.16.6.160 dev eth0 244.6.0/24 dev cbr0 proto kernel scope link src 10.244.6.1 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.208 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 via 172.16.6.98 dev eth0 default via 172.16.6.1 dev eth0 244.0.0/24 via 172.16.6.239 dev eth0 244.1.0/24 via 172.16.6.130 dev eth0 244.2.0/24 via 172.16.6.124 dev eth0 244.3.0/24 via 172.16.6.198 dev eth0 244.4.0/24 via 172.16.6.84 dev eth0 244.5.0/24 via 172.16.6.160 dev eth0 244.6.0/24 via 172.16.6.208 dev eth0 244.7.0/24 dev cbr0 proto kernel scope link src 10.244.7.1 172.16.6.0/24 dev eth0 proto kernel scope link src 172.16.6.98 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
Further to this issue; kubernetes-console, where the kube- script is being executed, is a freshly installed Debian 8 jessie machine deployed from a netinst iso. All Debian packages are fully up to date.
It appears that under certain conditions the cbr0 interface is brought up but not immediately configured with the final subnet selection. Due to the order of operations this would always manifest with the highest numbered (ie. last to deploy) minion in the cluster. The existing script would blindly push configuration for the default 172.18.0.0/16 subnet into the cluster, only for the cbr0 interface on that node to eventually become reconfigured with the correct final subnet, thereby breaking networking for containers subsequently deployed to that node. In the above logs the indications of this behaviour include the presence of route '172.18.0.0/16 via 172.16.6.98 dev eth0' on the master with no corresponding interface on node 172.16.6.98 (kubernetes-minion-8), and repetitions of 'Waiting for 8 ready nodes. 7 ready nodes, 8 registered. Retrying.' when calling 'validate-cluster' (ie. minion-8 is not yet fully configured by the time the end of the script is reached). PR attempts to address this issue by testing the subnet returned when cbr0 interface definition is retrieved from each node by the script and retrying every 5 seconds for up to 60 seconds if the network is still 172.18.0.0/16. If the subnet assigned has not changed from default within 60 seconds period the script aborts, consistent with other scripting failures.
Yes this race condition might cause this scenario [ if the bridge network is not setup then there might not be a route setup].

Discussion:
MASTER_CPU=1 NODE_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_NODES}})) <del> NODE_IP_RANGES="10.244.0.0/16" </del> <ins> NODE_IP_RANGES="10.244.0.0/16" # Min Prefix supported is 16 </ins> MASTER_IP_RANGE="${MASTER_IP_RANGE:-10.246.0.0/24}" NODE_MEMORY_MB=2048 NODE_CPU=1

# identify the subnet assigned to the node by the kubernetes controller manager. KUBE_NODE_BRIDGE_NETWORK=() for (( i=0; i<${#NODE_NAMES[@]}; i++)); do <del> printf " finding network of cbr0 bridge on node  ${NODE_NAMES[$i]}n" network=$(kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} 'sudo ip route show | grep -E "dev cbr0" | cut -d     " " -f1') KUBE_NODE_BRIDGE_NETWORK+=("${network}") done </del> <ins> printf " finding network of cbr0 bridge on node  ${NODE_NAMES[$i]}n" network="" top2_octets_final=$(echo $NODE_IP_RANGES | awk -F "." '{ print $1 "." $2 }') # Assume that a 24 bit mask per node attempt=0 max_attempt=60 while true ; do attempt=$(($attempt+1)) network=$(kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} 'sudo ip route show | grep -E "dev cbr0" | cut -d     " " -f1') top2_octets_read=$(echo $network | awk -F "." '{ print $1 "." $2 }') if [[ "$top2_octets_read" == "$top2_octets_final" ]]; then break fi if (( $attempt == $max_attempt )); then echo echo "(Failed) Waiting for cbr0 bridge to come up @ ${NODE_NAMES[$i]}" echo exit 1 fi printf "." sleep 5 done printf "n" KUBE_NODE_BRIDGE_NETWORK+=("${network}") done </ins> # Make the pods visible to each other and to the master. # The master needs have routes to the pods for the UI to work. local j for (( i=0; i<${#NODE_NAMES[@]}; i++)); do <del> printf "setting up routes for ${NODE_NAMES[$i]}" kube-ssh "${KUBE_MASTER_IP}" "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[${i}]} gw ${KUBE_NODE_IP_ADDRESSES[${i}]}" for (( j=0; j<${#NODE_NAMES[@]}; j++)); do if [[ $i != $j ]]; then kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[$j]} gw ${KUBE_NODE_IP_ADDRESSES[$j]}" fi done </del> <ins> printf "setting up routes for ${NODE_NAMES[$i]}n" printf " adding route to ${MASTER_NAME} for network ${KUBE_NODE_BRIDGE_NETWORK[${i}]} via ${KUBE_NODE_IP_ADDRESSES[${i}]}n" kube-ssh "${KUBE_MASTER_IP}" "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[${i}]} gw ${KUBE_NODE_IP_ADDRESSES[${i}]}" for (( j=0; j<${#NODE_NAMES[@]}; j++)); do if [[ $i != $j ]]; then printf " adding route to ${NODE_NAMES[$j]} for network ${KUBE_NODE_BRIDGE_NETWORK[${i}]} via ${KUBE_NODE_IP_ADDRESSES[${i}]}n" kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[$j]} gw ${KUBE_NODE_IP_ADDRESSES[$j]}" fi done printf "n" </ins> done }

printf "Waiting for salt-master to be up on ${KUBE_MASTER} ...n" remote-pgrep ${KUBE_MASTER_IP} "salt-master" <ins> printf "Waiting for all packages to be installed on ${KUBE_MASTER} ...n" kube-check  ${KUBE_MASTER_IP} 'sudo salt "kubernetes-master" state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"' </ins> local i for (( i=0; i<${#NODE_NAMES[@]}; i++)); do printf "Waiting for salt-minion to be up on ${NODE_NAMES[$i]} ....n" remote-pgrep ${KUBE_NODE_IP_ADDRESSES[$i]} "salt-minion" <ins> printf "Waiting for all salt packages to be installed on ${NODE_NAMES[$i]} .... n" kube-check  ${KUBE_MASTER_IP} 'sudo salt '"${NODE_NAMES[$i]}"' state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"' </ins> printf " OKn" done <del> printf "Waiting for init highstate to be done on all nodes (this can take a few minutes) ...n" kube-check  ${KUBE_MASTER_IP} 'sudo salt '''*''' state.show_highstate -t 50' printf "Waiting for all packages to be installed on all nodes (this can take a few minutes) ...n" kube-check  ${KUBE_MASTER_IP} 'sudo salt '''*''' state.highstate -t 50 | grep -E "Failed:[[:space:]]+0"' </del> echo echo "Waiting for master and node initialization."