Code change:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vSphere, "message": "no endpoints available for service "kube-dns"", kube-dns, etc -- similar issues include: however it's not quite the same Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): Using provided vmdk Kernel (e.g. ): and Install tools: GOVC + the script What happened: Our environment utilizes the RFC1918 10.0.0.0/8 block, so I edited to utilize an address space we are not using. This is my current configuration: After the deployment is finished, I am able to login to the dashboard via the URL provided via , however, when I check the DNS server I see: When I list services, I see: When I start a container and try to ping kube-dns, I am unable to: I can ping external IP addresses: Here's the configuration on each minion and the master: What you expected to happen: I expect containers can reach ping eachother. How to reproduce it (as minimally and precisely as possible): I've reproduced this problem using both 172.16.0.0/16 addresses as well as the 192.168.0.0/16 addresses. Anything else do we need to know: I followed the documentation here: and the only change I made was to the network configuration.
I just noticed that doesn't include any routes for the service IPs I defined (e.g. has no routing table entries on any of the minions, only ). I'm not sure how to fix it, but it looks like the DNS server can make requests to the upstream DNS, it just can't get responses back.
cc
Some additional info I probably should have included the first time:
I just noticed that while from inside a container I can ping google: I cannot ping our internal DNS which is at IP . So I think I'm getting closer to where the problem actually is. I also think the 192.168.120.120 was not actually unreachable, just not pingable. For example: Returns right away, while doing the same to IP hangs. I'm not sure why when using our DNS in the of my master/minions doesn't seem to give them the ability to resolve internal assets on our network.
Can you provide your output for iptables also
Hi imkin, Sure thing! Minion 1 - 10.248.30.38 Minion 2 - 10.248.30.40 Minion 3 - 10.248.30.37 Minion 4 - 10.248.30.39 Master - 10.248.30.36 Let me know what else I can provide, I haven't had much time to continue troubleshooting this week, but getting this to work would be a really great template for building our bare metal deployment. Thanks! ZC
Can you try with DNSSERVERIP="192.168.128.120"
Hi - I will edit that in the initial configuration and re-up my cluster this morning to see if that helps things work out of the box so to speak. Thanks! ZC
Hey I have edited my configuration as follows: I have re-run the script and everything came up and validated. I am still experiencing the same behavior, however. I now see this in the logs for the DNS service via the dashboard: I see that the service can't be reached by kubectl either (it doesn't even return kube-dns like it did previously): I also notice that on the minions, none of them seem to have routes for the higher IP ranges:
This looks to be the same as posting a PR shortly.
Awesome!! Thanks - I'm grabbing the from your pull request and will re-open this issue if that did not resolve it (once it's closed by the merge).
In the second case [when your kube-dns had the ip 192.168.128.120] where was your kube-dns?
based on the logs you posted you were definitely impacted by the issue in but I am not sure if that is the only issue on going on your setup. Please let us know how the fix goes for your environment.
There is no route for 192.168.3.0/24

Related discussion:
text: # identify the subnet assigned to the node by the kubernetes controller manager. KUBE_NODE_BRIDGE_NETWORK=() for (( i=0; i<${#NODE_NAMES[@]}; i++)); do <del> printf " finding network of cbr0 bridge on node  ${NODE_NAMES[$i]}n" network=$(kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} 'sudo ip route show | grep -E "dev cbr0" | cut -d     " " -f1') KUBE_NODE_BRIDGE_NETWORK+=("${network}") done </del> <ins> printf " finding network of cbr0 bridge on node  ${NODE_NAMES[$i]}n" network="" top2_octets_final=$(echo $NODE_IP_RANGES | awk -F "." '{ print $1 "." $2 }') # Assume that a 24 bit mask per node attempt=0 max_attempt=60 while true ; do attempt=$(($attempt+1)) network=$(kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} 'sudo ip route show | grep -E "dev cbr0" | cut -d     " " -f1') top2_octets_read=$(echo $network | awk -F "." '{ print $1 "." $2 }') if [[ "$top2_octets_read" == "$top2_octets_final" ]]; then break fi if (( $attempt == $max_attempt )); then echo echo "(Failed) Waiting for cbr0 bridge to come up @ ${NODE_NAMES[$i]}" echo exit 1 fi printf "." sleep 5 done printf "n" KUBE_NODE_BRIDGE_NETWORK+=("${network}") done </ins> # Make the pods visible to each other and to the master. # The master needs have routes to the pods for the UI to work. local j for (( i=0; i<${#NODE_NAMES[@]}; i++)); do <del> printf "setting up routes for ${NODE_NAMES[$i]}" kube-ssh "${KUBE_MASTER_IP}" "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[${i}]} gw ${KUBE_NODE_IP_ADDRESSES[${i}]}" for (( j=0; j<${#NODE_NAMES[@]}; j++)); do if [[ $i != $j ]]; then kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[$j]} gw ${KUBE_NODE_IP_ADDRESSES[$j]}" fi done </del> <ins> printf "setting up routes for ${NODE_NAMES[$i]}n" printf " adding route to ${MASTER_NAME} for network ${KUBE_NODE_BRIDGE_NETWORK[${i}]} via ${KUBE_NODE_IP_ADDRESSES[${i}]}n" kube-ssh "${KUBE_MASTER_IP}" "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[${i}]} gw ${KUBE_NODE_IP_ADDRESSES[${i}]}" for (( j=0; j<${#NODE_NAMES[@]}; j++)); do if [[ $i != $j ]]; then printf " adding route to ${NODE_NAMES[$j]} for network ${KUBE_NODE_BRIDGE_NETWORK[${i}]} via ${KUBE_NODE_IP_ADDRESSES[${i}]}n" kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[$j]} gw ${KUBE_NODE_IP_ADDRESSES[$j]}" fi done printf "n" </ins> done }