# Debug Services

Debug ServicesAn issue that comes up rather frequently for new installations of Kubernetes is that a Service is not working properly. You've run your Pods through a Deployment (or other workload controller) and created a Service, but you get no response when you try to access it. This document will hopefully help you to figure out what's going wrong.Running commands in a PodFor many steps here you will want to see what a Pod running in the cluster sees. The simplest way to do this is to run an interactive busybox Pod:kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox sh Note:If you don't see a command prompt, try pressing enter.If you already have a running Pod that you prefer to use, you can run a command in it using:kubectl exec <POD-NAME> -c <CONTAINER-NAME> -- <COMMAND> SetupFor the purposes of this walk-through, let's run some Pods. Since you're probably debugging your own Service you can substitute your own details, or you can follow along and get a second data point.kubectl create deployment hostnames --image=registry.k8s.io/serve_hostname deployment.apps/hostnames created kubectl commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands.Let's scale the deployment to 3 replicas.kubectl scale deployment hostnames --replicas=3 deployment.apps/hostnames scaled Note that this is the same as if you had started the Deployment with the following YAML:apiVersion: apps/v1 kind: Deployment metadata: labels: app: hostnames name: hostnames spec: selector: matchLabels: app: hostnames replicas: 3 template: metadata: labels: app: hostnames spec: containers: - name: hostnames image: registry.k8s.io/serve_hostname The label "app" is automatically set by kubectl create deployment to the name of the Deployment.You can confirm your Pods are running:kubectl get pods -l app=hostnames NAME READY STATUS RESTARTS AGE hostnames-632524106-bbpiw 1/1 Running 0 2m hostnames-632524106-ly40y 1/1 Running 0 2m hostnames-632524106-tlaok 1/1 Running 0 2m You can also confirm that your Pods are serving. You can get the list of Pod IP addresses and test them directly.kubectl get pods -l app=hostnames \ -o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}' 10.244.0.5 10.244.0.6 10.244.0.7 The example container used for this walk-through serves its own hostname via HTTP on port 9376, but if you are debugging your own app, you'll want to use whatever port number your Pods are listening on.From within a pod:for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do wget -qO- $ep done This should produce something like:hostnames-632524106-bbpiw hostnames-632524106-ly40y hostnames-632524106-tlaok If you are not getting the responses you expect at this point, your Pods might not be healthy or might not be listening on the port you think they are. You might find kubectl logs to be useful for seeing what is happening, or perhaps you need to kubectl exec directly into your Pods and debug from there.Assuming everything has gone to plan so far, you can start to investigate why your Service doesn't work.Does the Service exist?The astute reader will have noticed that you did not actually create a Service yet - that is intentional. This is a step that sometimes gets forgotten, and is the first thing to check.What would happen if you tried to access a non-existent Service? If you have another Pod that consumes this Service by name you would get something like:wget -O- hostnames Resolving hostnames (hostnames)... failed: Name or service not known. wget: unable to resolve host address 'hostnames' The first thing to check is whether that Service actually exists:kubectl get svc hostnames No resources found. Error from server (NotFound): services "hostnames" not found Let's create the Service. As before, this is for the walk-through - you can use your own Service's details here.kubectl expose deployment hostnames --port=80 --target-port=9376 service/hostnames exposed And read it back:kubectl get svc hostnames NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hostnames ClusterIP 10.0.1.175 <none> 80/TCP 5s Now you know that the Service exists.As before, this is the same as if you had started the Service with YAML:apiVersion: v1 kind: Service metadata: labels: app: hostnames name: hostnames spec: selector: app: hostnames ports: - name: default protocol: TCP port: 80 targetPort: 9376 In order to highlight the full range of configuration, the Service you created here uses a different port number than the Pods. For many real-world Services, these values might be the same.Any Network Policy Ingress rules affecting the target Pods?If you have deployed any Network Policy Ingress rules which may affect incoming traffic to hostnames-* Pods, these need to be reviewed.Please refer to Network Policies for more details.Does the Service work by DNS name?One of the most common ways that clients consume a Service is through a DNS name.From a Pod in the same Namespace:nslookup hostnames Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: hostnames Address 1: 10.0.1.175 hostnames.default.svc.cluster.local If this fails, perhaps your Pod and Service are in different Namespaces, try a namespace-qualified name (again, from within a Pod):nslookup hostnames.default Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: hostnames.default Address 1: 10.0.1.175 hostnames.default.svc.cluster.local If this works, you'll need to adjust your app to use a cross-namespace name, or run your app and Service in the same Namespace. If this still fails, try a fully-qualified name:nslookup hostnames.default.svc.cluster.local Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: hostnames.default.svc.cluster.local Address 1: 10.0.1.175 hostnames.default.svc.cluster.local Note the suffix here: "default.svc.cluster.local". The "default" is the Namespace you're operating in. The "svc" denotes that this is a Service. The "cluster.local" is your cluster domain, which COULD be different in your own cluster.You can also try this from a Node in the cluster:Note:10.0.0.10 is the cluster's DNS Service IP, yours might be different.nslookup hostnames.default.svc.cluster.local 10.0.0.10 Server: 10.0.0.10 Address: 10.0.0.10#53 Name: hostnames.default.svc.cluster.local Address: 10.0.1.175 If you are able to do a fully-qualified name lookup but not a relative one, you need to check that your /etc/resolv.conf file in your Pod is correct. From within a Pod:cat /etc/resolv.conf You should see something like:nameserver 10.0.0.10 search default.svc.cluster.local svc.cluster.local cluster.local example.com options ndots:5 The nameserver line must indicate your cluster's DNS Service. This is passed into kubelet with the --cluster-dns flag.The search line must include an appropriate suffix for you to find the Service name. In this case it is looking for Services in the local Namespace ("default.svc.cluster.local"), Services in all Namespaces ("svc.cluster.local"), and lastly for names in the cluster ("cluster.local"). Depending on your own install you might have additional records after that (up to 6 total). The cluster suffix is passed into kubelet with the --cluster-domain flag. Throughout this document, the cluster suffix is assumed to be "cluster.local". Your own clusters might be configured differently, in which case you should change that in all of the previous commands.The options line must set ndots high enough that your DNS client library considers search paths at all. Kubernetes sets this to 5 by default, which is high enough to cover all of the DNS names it generates.Does any Service work by DNS name?If the above still fails, DNS lookups are not working for your Service. You can take a step back and see what else is not working. The Kubernetes master Service should always work. From within a Pod:nslookup kubernetes.default Server: 10.0.0.10 Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local If this fails, please see the kube-proxy section of this document, or even go back to the top of this document and start over, but instead of debugging your own Service, debug the DNS Service.Does the Service work by IP?Assuming you have confirmed that DNS works, the