Issue 1:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vSphere, "message": "no endpoints available for service "kube-dns"", kube-dns, etc -- similar issues include: #24407 however it's not quite the same

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: vSphere

OS (e.g. from /etc/os-release): Using provided vmdk

Kernel (e.g. uname -a): Linux kubernetes-minion-4 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux and Linux kubernetes-master 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux

Install tools: GOVC + the cluster/vsphere/config-default.sh script

What happened: Our environment utilizes the RFC1918 10.0.0.0/8 block, so I edited cluster/vsphere/config-default.sh to utilize an address space we are not using. This is my current configuration:

NODE_IP_RANGES="192.168.0.0/17"
MASTER_IP_RANGE="${MASTER_IP_RANGE:-192.168.128.0/17}"
SERVICE_CLUSTER_IP_RANGE="192.168.120.0/21" # formerly PORTAL_NET
DNS_SERVER_IP="192.168.120.120"

After the deployment is finished, I am able to login to the dashboard via the URL provided via kubectl cluster-info, however, when I check the DNS server I see:

> curl -sk https://admin:password@10.248.30.36/api/v1/proxy/namespaces/kube-system/services/kube-dns
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"kube-dns\"",
"reason": "ServiceUnavailable",
"code": 503
}

When I list services, I see:

> kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 192.168.120.120 <none> 53/UDP,53/TCP 50m
kubernetes-dashboard 192.168.127.194 <none> 80/TCP 50m

When I start a container and try to ping kube-dns, I am unable to:

> kubectl exec -ti testing-network-2429464084-8jauq /bin/bash
root@testing-network-2429464084-8jauq:/# ping 192.168.120.120
PING 192.168.120.120 (192.168.120.120) 56(84) bytes of data.
^C
--- 192.168.120.120 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4032ms

I can ping external IP addresses:

# ping 216.58.192.142
PING 216.58.192.142 (216.58.192.142) 56(84) bytes of data.
64 bytes from 216.58.192.142: icmp_seq=1 ttl=50 time=52.0 ms
64 bytes from 216.58.192.142: icmp_seq=2 ttl=50 time=51.9 ms
^C
--- 216.58.192.142 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 51.968/51.999/52.031/0.230 ms

Here's the configuration on each minion and the master:

Master - IP: 10.248.30.36
kube@kubernetes-master:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0 proto kernel scope link src 10.248.30.36
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0
192.168.128.0/17 dev cbr0 proto kernel scope link src 192.168.128.1

Minion 1 - IP: 10.248.30.38
kube@kubernetes-minion-1:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0 proto kernel scope link src 10.248.30.38
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 dev cbr0 proto kernel scope link src 192.168.0.1
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0

Minion 2 - IP: 10.248.30.40
kube@kubernetes-minion-2:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0 proto kernel scope link src 10.248.30.40
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 dev cbr0 proto kernel scope link src 192.168.1.1
192.168.2.0/24 via 10.248.30.37 dev eth0

Minion 2 - IP: 10.248.30.37
kube@kubernetes-minion-3:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0 proto kernel scope link src 10.248.30.37
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 dev cbr0 proto kernel scope link src 192.168.2.1

Minion 4 - IP: 10.248.30.39
kube@kubernetes-minion-4:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0 proto kernel scope link src 10.248.30.39
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0
192.168.3.0/24 dev cbr0 proto kernel scope link src 192.168.3.1

What you expected to happen: I expect containers can reach ping eachother.

How to reproduce it (as minimally and precisely as possible): I've reproduced this problem using both 172.16.0.0/16 addresses as well as the 192.168.0.0/16 addresses.

Anything else do we need to know: I followed the documentation here: http://kubernetes.io/docs/getting-started-guides/vsphere/ and the only change I made was to the network configuration.

Issue 2:
The test-cmd script currently does not test resizing replication controllers; this is a central use-case and should have coverage in that script.