Change:
I have observed the DNS pod sometimes getting stuck. Need a health-check that tries to do a lookup or something like that.
also DNS should probably come up with N=2 by default
Why is this attached to 1.0 if it's P3? One of those is wrong.
Health checks I can believe. We're in the process of doing that for everything else -- Nothing else is HA, so I don't buy N=2 for DNS but not for other components.
Other components are somewhat harder to make HA. DNS is trivially HA-able. But not too important, it's available after the fact, too. , Brian Grant wrote:
so you mean a generic solution for health checking, or just you are adding health check for other service. In other words should I start adding health check for DNS?
I'd still like to see a health-check for DNS, but I am not sure what is appropriate, though. Maybe have kube2sky open an HTTP server and answer /healthz iff it can complete a non-existing DNS query? Better ideas? It seems the collected other fixes add up to a more stable DNS server? , Piotr Szczesniak wrote:
I assume this is about adding liveness probes to one or more of the containers in the dns pod: Is dig in the skydns image? We could use an exec probe to perform a DNS query.
duh, exec probe is obviously better :) But it doesn't need to get a name, it just needs to complete and not hang. so something like sh -c "alarm 5 nslookup foobar" maybe? , Brian Grant wrote:
leaving unassigned since I'm not working on it right now
are you currently working on this? If not, I'll grab it. Thanks! --brendan
I think that was looking into this today.
Yeah, I sat on it a little today due to not being able to get it to actually work, but I'm pretty sure it wasn't working because exec liveness probes as a whole are broken (). Sent PR to resolve this once exec probes are functional.

Discussion:
<ins> apiVersion: v1beta3 </ins> kind: ReplicationController <del> apiVersion: v1beta1 id: kube-dns namespace: default labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" desiredState: </del> <ins> metadata: labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" name: kube-dns namespace: default spec: </ins> replicas: {{ pillar['dns_replicas'] }} <del> replicaSelector: </del> <ins> selector: </ins> k8s-app: kube-dns <del> podTemplate: labels: name: kube-dns k8s-app: kube-dns kubernetes.io/cluster-service: "true" desiredState: manifest: version: v1beta2 id: kube-dns dnsPolicy: "Default"  # Don't use cluster DNS. containers: - name: etcd image: gcr.io/google_containers/etcd:2.0.9 command: [ "/usr/local/bin/etcd", "--addr", "127.0.0.1:4001", "--bind-addr", "127.0.0.1:4001", "-initial-cluster-token=skydns-etcd", ] - name: kube2sky image: gcr.io/google_containers/kube2sky:1.4 volumeMounts: - name: dns-token mountPath: /etc/dns_token readOnly: true command: [ # entrypoint = "/kube2sky", "-domain={{ pillar['dns_domain'] }}", "-kubecfg_file=/etc/dns_token/kubeconfig", ] - name: skydns image: gcr.io/google_containers/skydns:2015-03-11-001 command: [ # entrypoint = "/skydns", "-machines=http://localhost:4001", "-addr=0.0.0.0:53", "-domain={{ pillar['dns_domain'] }}.", ] ports: - name: dns containerPort: 53 protocol: UDP volumes: - name: dns-token source: secret: target: kind: Secret namespace: default name: token-system-dns </del> <ins> template: metadata: labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" name: kube-dns spec: containers: - name: etcd image: gcr.io/google_containers/etcd:2.0.9 command: - /usr/local/bin/etcd - --addr - 127.0.0.1:4001 - --bind-addr - 127.0.0.1:4001 - -initial-cluster-token=skydns-etcd - name: kube2sky image: gcr.io/google_containers/kube2sky:1.4 args: # entrypoint = "/kube2sky" - -domain={{ pillar['dns_domain'] }} - -kubecfg_file=/etc/dns_token/kubeconfig volumeMounts: - mountPath: /etc/dns_token name: dns-token readOnly: true - name: skydns image: gcr.io/google_containers/skydns:2015-03-11-001 args: # entrypoint = "/skydns" - -machines=http://localhost:4001 - -addr=0.0.0.0:53 - -domain={{ pillar['dns_domain'] }}. ports: - containerPort: 53 name: dns protocol: UDP livenessProbe: exec: command: - "/bin/sh" - "-c" # The health check succeeds by virtue of not hanging. It'd be nice # to also check local services are known, but if that's broken then # etcd or kube2sky has to be restarted, not skydns. - "nslookup foobar 127.0.0.1 &> /dev/null; echo ok" initialDelaySeconds: 30 timeoutSeconds: 5 dnsPolicy: Default  # Don't use cluster DNS. volumes: - name: dns-token secret: secretName: token-system-dns </ins>