Question: How do I resolve HTTP 503 (Service unavailable) errors when I access a Kubernetes Service in an Amazon EKS cluster?

I get HTTP 503 (Service unavailable) errors when I connect to a Kubernetes Service that runs in my Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

Short descriptionHTTP 503 errors are server-side errors. They occur when you connect to a Kubernetes Service pod located in an Amazon EKS cluster that's configured for a load balancer.To troubleshoot HTTP 504 errors, see How do I resolve HTTP 504 errors in Amazon EKS?To troubleshoot HTTP 503 errors, complete the following troubleshooting steps.

ResolutionCheck if the pod label matches the value that's specified in Kubernetes Service selector1.    Run the following command to get the value of the selector:$ kubectl describe service service_name -n your_namespaceNote: Replace service_name with your service name and your_namespace with your service namespace.Example output:Name:                     service-nameNamespace:                pod-nameLabels:                   noneAnnotations:              noneSelector:                 app.kubernetes.io/name=namespaceType:                     NodePortIP Families:              noneIP:                       10.100.17.189IPs:                      10.100.17.189Port:                     unset  80/TCPTargetPort:               80/TCPNodePort:                 unset  31560/TCPEndpoints:                noneSession Affinity:         noneExternal Traffic Policy:  ClusterEvents:                   noneIn the preceding output, the example selector value is app.kubernetes.io/name=namespace.2.    Check if there are pods with the label app.kubernetes.io/name=namespace:$ kubectl get pods -n your_namespace -l "app.kubernetes.io/name=namespace"Example output:No resources found in your_namespace namespace.If no resources are found with the value you searched for, then you get an HTTP 503 error.Verify that the pods defined for the Kubernetes Service are runningUse the label in the Kubernetes Service selector to verify that the pods exist and are in Running state:$ kubectl -n your_namespace get pods -l "app.kubernetes.io/name=your_namespace"Output:NAME                               READY   STATUS             RESTARTS   AGEPOD_NAME                           0/1     ImagePullBackOff   0          3m54sCheck if the pods can pass the readiness probe for your Kubernetes deployment1.    Verify that the application pods can pass the readiness probe. For more information, see Configure liveness, readiness, and startup probes (from the Kubernetes website).2.    Check the readiness probe for the pod:$ kubectl describe pod pod_name -n your_namespace | grep -i readinessNote: replace pod_name with your pad name and your_namespace with your namespace.Example output:Readiness:      tcp-socket :8080 delay=5s timeout=1s period=2s #success=1 #failure=3Warning  Unhealthy  2m13s (x298 over 12m)  kubelet            Readiness probe failed:In the preceding output, you can see Readiness probe failed.Note: This step provides helpful output only if the application is listening on the right path and port. Check the curl output with the curl -Ivk command, and make sure the path defined at the service level is getting a valid response. For example, 200 ms is a good response.Check the capacity for your Classic Load BalancerIf you get an intermittent HTTP 503 error, then your Classic Load Balancer doesn't have enough capacity to handle the request. To resolve this issue, make sure that your Classic Load Balancer has enough capacity and your worker nodes can handle the request rate.Verify that your instances are registeredYou also get an HTTP 503 error if there are no registered instances. To resolve this issue, try the following solutions:Verify that the security groups for the worker node have an inbound rule that allows port access on the node port to worker nodes. Also, verify that no NAT rules are blocking network traffic on the node port ranges.Verify that the custom security group that's specified for the Classic Load Balancer is allowed inbound access on the worker nodes.Make sure that there are worker nodes in every Availability Zone that's specified by the subnets.

Related informationWhy did I receive an HTTP 5xx error when connecting to web servers running on EC2 instances configured to use Classic Load Balancing?HTTP 503: Service unavailableMonitor your Classic Load BalancerMonitor your Application Load BalancersTroubleshoot a Classic Load Balancer: HTTP errors