Issue 1:
From #26823 on the 1-2 release branch.

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/26823/kubernetes-pull-test-unit-integration/29130/

=== RUN TestEtcdDeleteCollectionNotFound
I0603 16:07:09.501076 31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:09.021087134 -0700 PDT):
[10.438489ms] [10.438489ms] Object encoded
[12.938103ms] [2.499614ms] Version checked
[478.819054ms] [465.880951ms] Object created
[479.50163ms] [682.576Âµs] END
==================
WARNING: DATA RACE
Write by goroutine 136:
testing.(*common).FailNow()
/tmp/workdir/go/src/testing/testing.go:335 +0x41
testing.(*common).Fatalf()
/tmp/workdir/go/src/testing/testing.go:379 +0x94
k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound.func1()
/workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:607 +0x1eb

Previous write by goroutine 127:
testing.(*common).FailNow()
/tmp/workdir/go/src/testing/testing.go:335 +0x41
testing.(*common).Fatalf()
/tmp/workdir/go/src/testing/testing.go:379 +0x94
k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound.func1()
/workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:607 +0x1eb

Goroutine 136 (running) created at:
k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound()
/workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:609 +0x674
testing.tRunner()
/tmp/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 127 (finished) created at:
k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound()
/workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:609 +0x674
testing.tRunner()
/tmp/workdir/go/src/testing/testing.go:456 +0xdc
==================
I0603 16:07:31.563128 31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:30.904025457 -0700 PDT):
[4.074994ms] [4.074994ms] Object encoded
[4.108285ms] [33.291Âµs] Version checked
[388.137603ms] [384.029318ms] Object created
[658.894085ms] [270.756482ms] END
I0603 16:07:33.703849 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:31.608403789 -0700 PDT):
[21.447Âµs] [21.447Âµs] About to list etcd node
[1.910621611s] [1.910600164s] Etcd node listed
[2.095241867s] [184.620256ms] Node list decoded
[2.095271215s] [29.348Âµs] END
I0603 16:07:40.311964 31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:39.339138415 -0700 PDT):
[8.543508ms] [8.543508ms] Object encoded
[8.57157ms] [28.062Âµs] Version checked
[657.103509ms] [648.531939ms] Object created
[972.615398ms] [315.511889ms] END
I0603 16:07:41.685855 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:40.451788092 -0700 PDT):
[28.395Âµs] [28.395Âµs] About to list etcd node
[1.233569593s] [1.233541198s] Etcd node listed
[1.23389272s] [323.127Âµs] Node list decoded
[1.233916803s] [24.083Âµs] END
I0603 16:07:41.989831 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:40.452513447 -0700 PDT):
[17.67Âµs] [17.67Âµs] About to list etcd node
[1.53678103s] [1.53676336s] Etcd node listed
[1.537119905s] [338.875Âµs] Node list decoded
[1.537147305s] [27.4Âµs] END
I0603 16:07:42.136526 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:40.433819252 -0700 PDT):
[21.971Âµs] [21.971Âµs] About to list etcd node
[1.701880738s] [1.701858767s] Etcd node listed
[1.702409739s] [529.001Âµs] Node list decoded
[1.702443019s] [33.28Âµs] END
I0603 16:07:46.832506 31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:46.368422356 -0700 PDT):
[403.833Âµs] [403.833Âµs] Object encoded
[423.064Âµs] [19.231Âµs] Version checked
[463.224621ms] [462.801557ms] Object created
[463.920768ms] [696.147Âµs] END
I0603 16:07:48.421105 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:46.933007109 -0700 PDT):
[20.505Âµs] [20.505Âµs] About to list etcd node
[1.487499823s] [1.487479318s] Etcd node listed
[1.487894779s] [394.956Âµs] Node list decoded
[1.487921821s] [27.042Âµs] END
I0603 16:07:55.195380 31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:54.18723978 -0700 PDT):
[569.053Âµs] [569.053Âµs] Object encoded
[595.08Âµs] [26.027Âµs] Version checked
[1.007338588s] [1.006743508s] Object created
[1.007968516s] [629.928Âµs] END
I0603 16:07:55.848223 31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:55.195738587 -0700 PDT):
[265.59Âµs] [265.59Âµs] Object encoded
[283.195Âµs] [17.605Âµs] Version checked
[469.233449ms] [468.950254ms] Object created
[652.176438ms] [182.942989ms] END
I0603 16:07:56.997531 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.904154956 -0700 PDT):
[29.063Âµs] [29.063Âµs] About to list etcd node
[1.092862614s] [1.092833551s] Etcd node listed
[1.093186047s] [323.433Âµs] Node list decoded
[1.093211067s] [25.02Âµs] END
I0603 16:07:57.032214 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.879245406 -0700 PDT):
[37.82Âµs] [37.82Âµs] About to list etcd node
[1.152358883s] [1.152321063s] Etcd node listed
[1.152757988s] [399.105Âµs] Node list decoded
[1.152786584s] [28.596Âµs] END
I0603 16:07:57.421529 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.8518515 -0700 PDT):
[25.787Âµs] [25.787Âµs] About to list etcd node
[1.568348264s] [1.568322477s] Etcd node listed
[1.569206493s] [858.229Âµs] Node list decoded
[1.569266984s] [60.491Âµs] END
I0603 16:07:58.072211 31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.853393954 -0700 PDT):
[23.36Âµs] [23.36Âµs] About to list etcd node
[2.218247378s] [2.218224018s] Etcd node listed
[2.218613175s] [365.797Âµs] Node list decoded
[2.218650374s] [37.199Âµs] END
I0603 16:08:04.185381 31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:08:03.844808624 -0700 PDT):
[418.697Âµs] [418.697Âµs] Object encoded
[445.452Âµs] [26.755Âµs] Version checked
[339.711037ms] [339.265585ms] Object created
[340.374126ms] [663.089Âµs] END
--- FAIL: TestEtcdDeleteCollectionNotFound (83.95s)
etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
etcd_test.go:614: Unexpected error: <nil>
etcd_test.go:593: Unexpected error: pods "foo" already exists

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

kube-dns, resolv.conf, timeout, skydns

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

1.3.6 / kube-dns v17.1

Environment:

Cloud provider or hardware configuration:

AWS

OS (e.g. from /etc/os-release):

centos7

Kernel (e.g. uname -a):

Install tools:

Others:

What happened:

Host resolv.conf, as used in kube-dns container:

search eu-west-1.compute.internal
nameserver 10.10.224.4
nameserver 10.10.228.4
nameserver 10.10.232.4
options rotate attempts:1 timeout:1
options ndots:5

If I disable 10.10.224.4, every few queries it takes 5s to resolve (resulting in timeouts on other pods). Here's what it looks like when running from the kube-dns pod:

# time nslookup gatech.edu.eu-west-1.compute.internal
nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server: 10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real 0m 5.01s
user 0m 0.00s
sys 0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server: 10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real 0m 5.00s
user 0m 0.00s
sys 0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server: 10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server: 10.50.224.4

Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real 0m 5.01s
user 0m 0.00s
sys 0m 0.00s
/ #
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server: 10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server: 10.50.224.4
Address 1: 10.50.224.4

And from another pod:

/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM
130.207.160.173 RAW

real 0m0.009s
user 0m0.000s
sys 0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM
130.207.160.173 RAW

real 0m2.007s
user 0m0.000s
sys 0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM
130.207.160.173 RAW

real 0m0.006s
user 0m0.000s
sys 0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM
130.207.160.173 RAW

real 0m0.005s
user 0m0.000s
sys 0m0.001s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM
130.207.160.173 RAW

real 0m2.008s
user 0m0.000s
sys 0m0.002s

I can sort of work around this by removing the eu-west-1.compute.internal search domain, which prevents the search NXDOMAIN that causes random hangs.

What you expected to happen:

I expect it to resolve the NXDOMAIN quickly even if one of the nameservers is down. Or at worst, take up to timeout:1. The timeout:1 option seems to be ignored, and it takes 5 seconds to timeout whenever it hits the down server.

Also the rotate option seems ignored, so all the load from kube-dns happens on the primary dns server. (need to verify this, just noticed from quick observation that the primary server gets almost all the queries)

Ideally, in my opinion, kube-dns would use something like musl and resolve against all the nameservers in parallel, making it robust to failure.

How to reproduce it (as minimally and precisely as possible):

Have multiple nameservers in host's resolv.conf

Turn off the first nameserver

Try to resolve domains that would normally return NXDOMAIN

Anything else do we need to know: