Change:
What happened: In my cluster sometimes readiness the probes are failing. But the application works fine. What you expected to happen: Successful Readiness Probe. How to reproduce it (as minimally and precisely as possible): We have few clusters with different workloads. Only in cluster with big number of short living pods we have this issue. But not on all nodes. We can't reproduce this error on other our clusters (that have same configuration but different workload). How i found the issue? I deployed a daemonset: Then i started to listen events on all pods of this daemonset. After a couple of time i received next events: Those events where on ~50% of pods of this daemonset. On the nodes where the pods with failed probes was placed, I collected the logs of kubelet. And there was errors like: I was thinking that sometimes the nginx in pod really response slowly. For excluding this theory, I created a short script that curl the application in pod and store response time in a file: I run this script on node where the pod is placed for 30 minutes and i get the following: There was measurements and the longest response time was . In logs of pod there are only message with response code 200 (from my requests and requests of readiness probes): It means that part of probes are successful. Then i stopped the curl script (because the big number of logs). I waited while new error with failed probe appears in kubelet logs. And in logs of that pod with nginx I didn't find the request generated by this probe: If I restart the kubelet - the error don't disappear. Have someone any suggestions about this? Environment: Kubernetes version: 1.12.1 Cloud provider or hardware configuration: hardware OS (e.g: ): ubuntu 16.04 Kernel (e.g. ): 4.15.0-66-generic Install tools: Network plugin and version (if this is a network-related bug): calico:v3.1.3 Seems like the problem appears in many different installations - /sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
Are you sure the application don't hit the resources limits? In my case, the application starting fine, then the container start using more resources until he hit the limit. After that the readiness probes fail
Yes, I am sure. I collected the metrics. Cpu usage on pod was around 0.001 Ram - around 4mb. This is a test application (pure nginx image) that did not do anything (no traffic is sent to it).
/area kubelet
any update on this? we have been facing similar issues since few weeks
This is the first such report I have seen. There's nothing obvious about why this would happen. It's possible kubelet is too busy and starved for CPU and the probe happened to be thing that got thrashed. How many pods are on this machine? How busy is it? It's possible the node itself is thrashing or something and OOM behavior is weird. Does dmesg show any OOMs? It's possible some other failure down deep in kubelet is being translated into this? You could try running kubelet at a higher log level to get more details on what is happening. A lot of bugs have been fixed since 1.12, so we'd need to try to reproduce this and then try again in a more recent version. Is there any way you can help boil down a simpler reproduction?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
For example, if you see the above get pods, of one pod is failing almost every day. A restart would fix this. However, we're unable to find the root cause. But I don't see any abnormal numbers on CPU or memory or thread count. Doing a would give me that has failed. How do we debug in such scenario? Also, this is happening for this particular deployment only. The other pods are running fine. "returns-console--pbjf8 1/1 Running 0 23h returns-console--plqdr 1/1 Running 0 23h returns-console--tlb5k 1/1 Running 0 23h returns-console--tr2kd 1/1 Running 0 23h returns-console--vfj2n 1/1 Running 0 23h"
We too are facing the same issue in our cluster. Kubernetes Version - 1.16.8 Kernel - 4.4.218-1 OS - Centos 7 Installed using Kubespray In our case, timeouts were not related to application but related to specific nodes in cluster. In our 13 node cluster, node 1 to 4 had some kind of issue wherein the pods running on these nodes had random failures due to timeouts. Checked that there weren't any cpu aur memory usage spikes also. P.S We are using NodePort for production use case. Is it possible that the node port setup cannot handle too many socket connections?
I have no idea what might cause spurious probe failues. have you heard any other reports of this? are you saying PROBES failed (always same node) or user traffic failed? If you have any other information about what was going on with those nodes when the failures happen, it would help. Check for OOMs, high CPU usage, conntrack failures?
Conntrack shows hardly 2 or 3 errors. Memory consumption is 60-65% per node. Just found that the timeouts were for almost all request and not just probe. We istio lately to check connection stats and understand if the behaviour was due to application. But the findings were weird. Istio itself is now failing readiness probe quite frequently ! 157 failures in ~3 hours Would like to add that kubernetes is running in our Datacenter. And since iptables version is 1.4.21, --random-fully is not being implemented. But since all machines have same configuration, we ruled out this possibility
I apologize for not having a lot of details to share but I'd add my 2 cents. We recently upgraded from Istio 1.4.4 to 1.5.4 and started seeing the issues described by OP. Lots of liveness / readiness issues there were not happening before. It SEEMS like adding let's say a 20sec initial delay had helped in most cases. At this point we are still seeing it and not sure what the root cause is. running on EKS 1.15 (control plane) / 1.14 managed nodeGroups
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
I'm afraid the only way to know more is to get something like a tcpdump from both inside and outside the pod, which captures one or more failing requests. Possible?
I'm getting same issue. Couple Nginx+PHP Pods running on huge instances in parallel with couple other small applications. These are staging apps+nodes without constant traffic. I constantly receive notification that these Nginx+PHP app has restarted... specifically Nginx container of these Pods. At the same time other apps running in the same namespace, nodes never restart. Liveness on an Nginx container looks like this: UPD: Strange thing is that completely distinct deployments you can see above staging4 staging5 stagingN - above 10 deployments fail at once. My possible problem might be in missing which is default
Having the same problem here. Error cca 3 - 10 times a day: The service operates normally and responds to /status in 5ms, though. Also, and that is a bigger problem, at similar random times some pods refuse to connect to each other. Running on Azure Kubernetes Service V 1.14.8
I'm facing the same issue as well, increasing timeoutSeconds didn't help. Runing on Kubernetes v1.16.7 on AWS deployed via KOPS
I appreciate the extra reports. It sounds like something is really weird. I'll repeat myself from above: I'm afraid the only way to know more is to get something like a tcpdump from both inside and outside the pod, which captures one or more failing requests. Is that possible? Without that I am flying very blind. I don't see this experimentally and I'm not flooded with reports of this, so it's going to be difficult to pin down. If you say you see it repeatedly, please try to capture a pcap?
I'll try to get a dump if I'm able to replicate this issue consistently, since it tends to happen randomly. Just to clarify, what exactly did you mean by tcpdump outside the pod? Did you mean tcpdump of the node where the pod resides? Sorry, I'm relatively new to this :sweat_smile:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
/remove-triage unresolved
We are also seeing this issue - trying to run a Django+WSGI+nginx server that works on a lower version of K8s, when we try this on Linode's managed Kubernetes service - LKE we see this
UPD: So since my last comment, I've updated timeouts to something more realistic. e.g.: Everything was quiet since then. But the day before yesterday, and yesterday: My prometheus-operatoralertmanger and slack have have exploded. And today by magic it is silent again not a single alert. My apps is a usual PHP+Nginx pod. Staging environments deployed on 3 huge boxes. Just a dozen preview deploys which are deployed only once, and then opened by devs only once. No load at all. For example, the app had 10 staging deploys. so app-staging1..10 And all containers of exactly that one app were failing all at once. i repeat virtually not related deployments spread on 3 nodes. In pod, only Nginx containers were restarting. In logs, I was able to catch that actual Nginx was answering 499 to the liveness probe. Nothing else. Still investigating. While ~5 other PHP apps deploys on the same staging nodes never restart.
I am also facing the same issue. Running on Kubernetes v1.16.8 on AWS EKS. Any workarounds?
I am also facing the same issue with AKS 1.16.9. This is occuring consistently. Is there any way to investigate the issue?
Looks like this error message is a symptom of a more general problem of node-to-pod connectivity issues. In our case, we confirmed that by using from the node to the pod's internal IP as suggested in That IP was perfectly routable from the other pods and even from many other nodes, but not from that particular node it ended up on. The root cause turned out to be that the br-int (bridge) interface that connects the pod networks and the NSX overlay logical switch was failing to come up because of missing static routes. Fixing that magically solved the problem. In other cases, the root cause may be different, but the first step would be to confirm whether the node can actually access that pod, e.g. ssh and traceroute or curl into the readiness probe ( when Istio sidecar containers fail to get ready), etc.
experiencing this issue in multiple GKE clusters. just started about 5 days ago. right now i'm having to drain the bad node and remove it from the pool. running
My issue was in a REST API dependency that was checked by readinessProbe. Doubles readinessTimeout helped.
What happened: After updating EKS from v1.14.9-eks- to v1.15.11-eks- some of our ReplicaSets and DaemonSets Liveness/Readiness probes started failing with It seems that random pods are affected. I can successfully curl to them from other pods. All nodes and pods are fine, not CPU/Memory/Storage issues. Two nodes are running 1.15 and the rest 1.14. There doesn't seem to be a correlation between node version and the issue. Nothing obvious in or or Probes are mostly configured like this (truncated): I will increase the timeouts but it would be good to solve this. What changed in 1.15? What you expected to happen: Liveness/Readiness probes not to fail. How to reproduce it (as minimally and precisely as possible): Upgrade EKS 1.14 to 1.15 Environment: Kubernetes version (use ):Cloud provider or hardware configuration: EKS OS (e.g: ): 14: 15 Install tools: AWS Console Network plugin and version (if this is a network-related bug): amazon-k8s-cni:v1.6.3-eksbuild.1
For those of us running a managed is not something we should ever have to deal with. What version and where are you running k8s?
I am using AWS EKS 1.16 and have been getting the same problems recently.
Update: After two days of checking everything (including cluster dump) the issue was with the application/pod itself. The application was failing to respond to Liveness/Readiness probes (in a specific scenario) and it was killed. Initially no obvious reason for this was found, and it correlated with EKS upgrade, so we blamed (wrongly) eks 1.15. initialDelaySeconds didn't help as the pod was fine starting up, timeoutSeconds did help. Things to test: Run constant ping to your affected pods simulating Liveness/Readiness probes and see when they are failing.Enable more logs in your applications Extend Liveness/Readiness probes timeout for easier troubleshooting use tcpdump, it might help.
Dealing with the same issue on a GKE cluster, except with a twist. Not to add to the fire but maybe help other's have the same issue. I have a spring-boot app with actuator - running actuator on a separate port- In this setup, using named container ports, it events: The following works: When removing the use of named containerPorts for readiness/liveness it passed the checks: I haven't been able to find logs/records of what endpoint it's calling, but I suspect it's still pod IP + port, but for some reason it works?
I'm also seeing this 3+ times a day after upgrading my cluser to 1.16.11-gke.5 on GKE a few weeks ago. I ignored it for a while but the past few days I've been running load tests against my services with k6 and I'm seeing sporadic latency in my cluster. I'm beginning to think it's related. After a full day spent trying to narrowing it down I can confirm the latency is any network connection between any pod and its connection to redis, cloudsql, or one another over HTTP. I'm still investigating, but I've shut down all external logging (Datadog) and when under moderate load it shows up every 5-6 seconds with a 200-1500ms delay affecting any and all pods. I can't speak for the loads these other systems are under, but my guess is the health check readiness timeout is a symptom of a more pervasive networking issue. EDIT: I should add, the readiness timeouts occur when run against my Nginx Ingress which was installed via Helm - not against a pod containing a container I built or configured readiness probes for.
Now facing this mainly for the tiller pod (helm), all other pods are generally working fine. I did shutdown all the EKS worker nodes and then recreated, that seems to solve the issue. Looks like a kubernetes 1.16 issue to me.
I can't share a full tcpdump of our internal services, but I have done tcpdumps and have some more to share about this issue. The first is I upgraded my cluster to GKE cluster to 1.17.9-gke.1503, and upgraded my nginx ingress via Helm (ingress-nginx/ingress-nginx) to chart version ingress-nginx-2.15.0. I'm still seeing an occasional timeouts but I still don't believe this is isolated to the ingress. When I have the system under moderate load I can see the my service throughput bottoms out occasionally and it's affecting every connection in the cluster. With throughput is consistent my service response time is averaging ~50ms, when throughput bottoms out it jumps to 700+ms. This shows throughput from Service-A to Service-B: ! And this shows throughput from Service-B to its highly available Redis store: ! At the time of the capture everything had been cached in Redis, but the same issues are seen when querying the managed database. The throughput drops also line up with TCP DUP ACK's to our external logging platform. I'm trying to wade through tcpdumps of kube-dns and kube-proxy but they're a lot to look at. This is our staging cluster, a 4 node n1-standard-1 cluster all running in us-central1-c where I have limited the number of service instances to 1 pod per service. However, the problem is consistent on the more beefy production cluster where all pod's throughput drops at fairly consistent intervals.
We had this same problem that ended up being due to our pod consuming more CPU than allowed by the pod's resources limits cpu. It seemed this prevented networking in general to be blocked to the pod during the high CPU time.
we that issue, but in our case(k8s 1.19.2, Ubuntu 18.04) we disabled CPU limits completely(removed them from deployment descriptors and kubelet is running with ) - that didn't help.
do I understand correctly that for you issue was resolved when instead of named ports you started to use actual ports in probes?
Hi ! I am facing the same issue, I have ever happened the problem on GKE 1.14&1.15 . My cluster have two deployment with nginx container , these deployment have set liveness/readiness probe. Only one deployment(six replica) sometimes all pods get unhealth at the same time. Is possible the issue happened on nginx container with liveness/readiness probe?
Hi! Same issue here: Kubernetes v1.16.8 On-Premise deployment (VMware) via Kubespray CNI: Canal Nodes: Ubuntu 18.04 Using metallb with BGP implementation for LoadBalancer IPs. I'm getting "Liveness probe failed: Get XXX net/http: request canceled (Client.Timeout exceeded while awaiting headers)" randomly every 3h o 4h for some of our k8s applications (not readiness problems). In our case, we check that: Pods are not reaching to resource limit so it's not related to limit issue. Pods have no restarts, so the canceled request is "sporadic" (not consecutive). Pods with this random events are coming from daemonset, deployments and statefulset. Is not disruptive as you can see, but we have random alerts in out monitoring systems. So it would be nice to find the reason. If I find something more, will report. EDIT: Modified timeoutSeconds on Liveness but still getting errors. Keep looking...
Same problem here: EKS v1.16 Django application with Nginx and gunicorn Running the same request via a pod in the same namespace returns a 200 response. EDIT: Even setting longer timeouts behaviour is extremely inconsistent (No Readiness probe failures to several counts)
Ran into a similar issue this week after upgrading EKS 1.17 to 1.18 where my grafana cluster would come partially up. 2 of the 3 pods would return with liveness or readiness probe failed connection refused. I tried many different solutions with no luck. Eventually, the only thing that seem to fixed the issue was performing a rolling restart of the worker nodes as suggested by Now all the pods are coming up successfully even after I manually terminate them.
We too are facing the same issue in our cluster.
Please make sure your all working nodes can access the Internet. I also encountered a similar problem. The reason is: k8s-master and k8s-work2 can go online k8s-work1 no internet
Hello is now the problem solved?
Same issue, sometimes I get this K8S event:
We're also seeing the same issue: Random failures of the Liveness and Readiness probes (that we are very very sure are not rooted in the application) with using Kubernetes 1.15 Managed EKS at AWS:
Adding to this: ( and me both work on the same issue within our K8s / EKS cluster, so the version etc. poste apply to this) I think we have a proof that there is a networking issue between the Kubelet and the container. We have had an http based health check in place; and a http-based readyness endpoint. We have had problems with our cluster, as is restarted contains seemingly random, giving failing health checks as reason. Of course, we started to dig into why our container fails (as, we all know, "SQL is not broken"). After a lot of deep and thought digging into metrics and logs, we came to the conclusion that our container (running a rail based web app) is just fine. We concluded that our network setup is somehow flawed (being strange, given that we use EKS for a reason…), so we switched the health check to a simple command to see if this mitigates the problem. We left the readiness endpoint alone, so this is still using HTTP. Now, we see readiness probe failures, but no health check failures any more. In addition, we can see that our app is happily serving http request, while we see times where the readiness endpoint is not called. ! Our view on this: see that the container is reachable and responsive; the same time, the readiness check endpoint is NOT being served by the app. This means that the , for some reason, is not able to send readiness probe http request to the container for some seconds, and then starts to get them trough later. At the same time, request coming in from an external load balancer are still served. As fare as we understand, the process runs on the node, so this looks like the networking for "node-local" request is unreliable. This looks like a serious issue to us, as the health checks killed our app without good reason, repeatedly. I understand that this problem looks a lot like people are using K8S for unstable apps and blame the platform instead of doing their homework — but, as far as I can tell, what we see proves that this is not the case for us and that there is a problem. It would be great to get some feedback on this issue -- let me know if there is a misguided thought. This is complex stuff, after all. :)
We have reviewed our findings on this, and here is my update: ready endpoint is called by the , the gaps were due to some unrelated issue with our logging back end. have reviewed all calls and the response time for all readiness checks, and the were well within the expected time range We have had no outages any more since switching the health checks to an based health check, still hinting a problem with http related checks. Also, we have switched from to port, but still see the readiness endpoint failing, so this does not influence the failure of http probes for us. We will now switch to exec based readiness probes and post if this helps. Sorry for the noise, but I hope this helps to clarify the situation for people having this problem.
Same issue
We have replaced the with an doing basically the same thing, which so far works without any problems.
I'm observing the same behaviour. EKS 1.17 w/ instances I have an app deployed with cpu.request = 0.25 cores, 1 core Max CPU usage today of any of the pods in this deployment is about 0.2 core (so, less than even the request) Seeing occasional liveness probe failed and readiness probe failed events for each of the pods in this deployment. To test this, I've port-forwarded to one of the pods, and have a simple shell script running on my own machine that curls the port-forwarded endpoint used by liveness probe once per second and logs if non-200 response returned. Even while observing liveness and readiness probe failure events, this curl loop has returned 200s only.
I was facing the same problem in prod but then I tried hitting pod id with the actual port number which resulted in a 200 but the response time was some where close to 7 seconds . But the usual response time is somewhere near 120 ms. So if once in a while the health point records a lag there is a unhealthy pod request and to the surprise this kept on happening every 10 s and the pod got killed . I eventually tracked the kills to a pod lying on a node and deleted it . Still this is not the solution as I keep seeing this often in my clusters. Please help
We're seeing similar behaviour as described too on a 1.17.5 cluster - all pods on certain nodes failing their http based healthchecks with timeout errors and getting restarted when they don't need to be. Just a heads up to those looking to move to an exec based healthcheck as a workaround, you may run into which substantially increases system load. Funnily enough, this is the main reason we moved to the http based healthchecks (only to run into this issue).
Wow. Can't believe this. We've been facing an issue with sudden pod restarts because of the liveness probes failing when we were sure the servers were behaving properly. We now implemented the instead of and so far everything looks good.
Same here, I've noticed the same issue in my GKE cluster. Don't know when it started, but it happens for many containers which I haven't updated in more than a year and now they have a lot of probes failing whereas before they never did, so it's certainly not the container.
We too are facing the same issue in our cluster. Kubernetes Version - 1.16.8 kernel version: 3.10.0-957.5.1.el7.x86_64 OS - Centos 7
Also I suggest to rename the topic of this issue from "Sometimes" to "VERY frequently" which a) reflects reality better and b) might move this issue up the priority list because I find it kind of annoying that Kubernetes probes have been so flaky and unreliable for such a long time now without anyone making any real progress here.
Seeing a similar behavior in an AKS cluster (Azure). Kubernetes OS and Kernel Since this is a managed Kubernetes, we don't have much visibility into the managed part, but I can say we are trying to utilise the nodes well, so we pack them up with a lot of pods (up to about 50), that have many containers, so I might have ~200 containers running in a node. Replacing with a simple running resolved it completely!
same issue on AKS . can we have some feedback from kubernetes maintainers?
same issue on: Kops v1.18.3 +AWS +EC2 (r5 family) K8S v1.18.9 (Ubuntu 20.04.1 LTS base image) replacing httpGet with a simple exec running curl resolved it completely in my case too!
same issue on EKS 1.16 I noticed there is a potential correlation with the node network I/O. I was doing some heavy query in Thanos, and it caused a network spike, and at the same time some of the pods running on the same node as thanos pods got restarted due to liveness probe timeout. It happened three times so far which made me think it's not just a coincidence.
We are seeing this on EKS 1.16 as well. It does seem to be related to node newtork I/O since s were working fine before we enabled log forwarding daemonsets (we generate a huge amount of logs).
cc
For liveness probes, I replaced with and it has worked fine since. To me this makes more sense anyways. A liveness probe should fail only when a restart is the only thing that will fix the issue. A busy server (too busy to respond to a httpGet in time) is not such an issue.
In my case curl workaround worked for a short time. Main issue was aggressive throttling - . So in case of restarts problem it's good to check value of For time being I had to disable limits globally ( in kubelet), after kernel patching I will go backto limits. Currently and works perfectly, the only downside is lack of limits ;P
same issue on 1.16
Experienced this bug as well in 100's of clusters. I am running v1.18, and not in the cloud. Did the httpGet to tcpSocket liveness probe work around. Looks like that fixes the main pod, but other pods are having issue with network connections.
k8s-1.14.8 (Managed by Aliyun cloud provider) CentOS 7.7.1908 linux kernel 3.10.0-1062.18.1.el7.x86_64 coredns 1.3.1 shows some pods fail to response readinessProbe time to time: I identified the cause was the pod spent 5s to do dns lookup, further investigation showed the node where the failing pod ran on lost incoming udp packet time to time, so the pod resent dns query 5s later but the readinessProbe timeout is exactly 5s. After reboot the k8s node where the failing pod ran on, and make sure coredns pods are not scheduled to these suspicious nodes, the issue disappeared. I guess the physical machine where the k8s node is on has traffic congestion sometimes, or linux kernel has some bug that drops incoming udp packet.
Somebody pointed me to this link: (It's in Chinese, but most important information is in English) It's recommended to insert this podConfig to your pod template: The most important option is .
Does anyone know if this issue affects v1.19+ ? We're facing similar problems with v1.18.x on multiple clusters.
Yes, facing the same problem with a 1.20.2 cluster.
Hello , How did you managed to get these issue fixed ? Could it be possible to paste the entire block changed ? I am using calico so need to check for the changes.
- we just moved to use the probe to replace the . Here is how it looks for one of our containers: It eliminated the error for us.
seems IBM reached the same conclusion: They offer a job to do the update of Pods for on OCP, might work on K8s as well. see above readme for details
I've seen this behaviour with on prem clusters on 1.19 and now with 1.20 on large bare metal nodes. In our findings so far it seems if the pod fs is slow the http probes will fail at intervals. The more pvc's attached the the pod the more likely something like will show a value over a few secs. If there is no pvc's attached we dont see any http probe failures.
Had you got a chance to test the limits after kernel patching?
We recently upgraded multiple k8s clusters to v1.20 and are now having this issue occur at a much higher rate than previously (on v1.18). ! ^You can see exactly when we started the upgrades this week. Those are k8s events that match .
/triage accepted
Environment: Amazon EKS, K8s version , AMI release version , AMI type , instance type We're having the same issue since Monday this week, tried everything, but one interesting observation is: the frequency of the occurrence is a lot higher when the nodes have significant memory usage... This could be our app leaking memory, but in that case I would expect K8s to let the app use up to 99% of and then "Evict" the pod and start a new one. This would've been a totally fine behavior, but instead we're getting: Our nodes all have 16 Gb of RAM, but Datadog monitoring shows that the memory usage exceeds the total available memory somehow: ! If the host OS is swapping memory - this could explain why the application processes the requests extremely slowly, slower than liveness probe timeout value (I increased it from 1 to 3 seconds to compensate). But why would the nodes be swapping memory? I thought K8s does not allocate pods if there isn't enough physical memory available?
Our production issues are now fully resolved. Thank you everyone for sharing your solutions!!! We're running on latest EKS, K8s version 1.20. probes with basic exec + curl (see below). This was suggested by a few people above. My guess is when the nodes are under high load, kubelet process couldn't spin up fast enough to check the health status of a pod. This makes "think" that a few pods are unhealthy, so k8s kills them and spins up new ones to replace. This makes the problem worse, because while a few pods are unhealthy the rest have to bear the load. On top of that the pod startup is CPU intensive in our case, it takes about 45 seconds (I know - slow). probe runs in an existing running pod/container, so it's very efficient and lightweight. , tweak timings of to continuously monitor pods health, and use instead of . This was my misunderstanding of probes: I used to think that is only used for pod startup and once the pod is started and healthy - takes over and monitors its "liveness" status. It turns out, I was completely wrong: It turns out, is what I needed to wait for the pod to start up - duh. is used to continuously monitor the pod, and make sure it is "ready" to receive HTTP requests (in our case). is only needed when the application is incapable of dying on its own (Erland, Elixir?). Our app is Rails, so if it dies - the container process is killed and the non-zero status code. In this case K8s simply starts a new pod automatically, no need to check "liveness". The is needed to redirect the page HTML, so that we don't pollute our logs with it. It's great to see the system purring peacefully, even under high load: <img width="533" alt="CleanShot 2021-07-08 at 09 33 38 src="https://user-"
thanks for sharing slution. according to the above conclusion, have you tried to modify kubelet's reserved resources? kubelet thanks in advance
I have tried this setting. And I also set the Still hit the same issue.
Thank you so much for help! My problem did solved with simple exec curl, fantastic!! Thanks for all and especial for
Thank you!, timeoutSeconds: 2 & exec both work, but again it all depend on your node busy status
I didn't try to change the standard resource reservations and limits for kubelet services. Another K8s upgrade is available - 1.21, but this time instead of just upgrading our production, I'm going to set up another new cluster and gradually move workloads to it. Just to avoid this type of issues.
did you have any luck after updating to ?
I have the same problem with a v1.21.6 cluster. Both probes are set to the same port and path: From kubelet logs, both probes sent at about the same time, but liveness probe succeeded, readiness probe failed.
Hi did your problem solved with simple exec curl When the problem has occurred, was there a high load on the node where the pod is located? (the same as above)
I can't reproduce that failure with the following manifest, check previous comment
I think I found the problem of my case. I run tcpdump and captured the failed probe. ! is the pod ip. is the ip of an interface created by nodelocal dns cache. The host didn't reply to the SYN+ACK of the 3-way handsake. I checked iptable rules of the host and found: There rule are by I enabled iptable trace and found the SYN+ACK packet was dropped by calico firewall rule. So whenever the source port is 8080, the probe will fail. (We changed local port range to 1024 ) And I can confirm with below command: I think this maybe a bug of nodelocal dns cache. There should be rules to allow incoming traffic to 8080 like
wow, impressive, so the nodelocal dns cache are interferring with the probes traffic and calico is dropping it since the conntrack entry is invalid can you confirm that kubelet is using 169.254.25.10 as origin ip for the probe connections? cc:
That's what is supposed to do, but these rules are only for traffic that are sent/received by 169.254.25.10. As Antionio asked in the previous comment, can you confirm that kubelet uses "169.254.25.10" to send out healthcheck requests?
Yes I can confirm. Below is the packets captured from the pod
What does the routing table on the host look like? I would expect the default route to use eth0 ip and the nodelocaldns interface to NOT show up in the routing table at all. That's one reason we add it as a dummy device -
The CNI plugin we use is , and the routing table on the host looks like:
I checked all the nodes of our cluster, and found that when accessing pod, some use ip as the source, some use . We manually deleted interface and recreated it before, so the index of was bigger than and . is service ip of After reboot, the index of became smaller. is the host ip of . This may be related to how linux select the source ip.
this is explained here http://linux- you confirmed your system has that src parameter interestingly, your host doesn't have an IP on that subnet, 172.16.0.128 is not a valid address, is the network address I'm assuming that not having an IP in the same subnet, then the kernel choose another algorithm that ends picking the 169.254.25.10 as source. If your host has an address on the pod subnet, i.e. 172.16.0.129 in this case, I think that it will use that IP to reach the pods I honestly can't say how to fix this from kubernetes perspective, and how kubelet can participate on the source address selection .... It seems straight forward to solve from the CNI perspective and to fall into that realm ...
Actually the route rule by canal has no src param, as shows I now just reboot the nodes to workaroud this.
Is this the reason "exec curl" can workaround httpGet probe issue? Will this still work after 1.20?
I had the same issue, but in my case, I had 2 containers inside of the same POD, and both of them there a readiness probe setup, and I am looking in the probe for the wrong container
In our case (AKS, K8s v1.21.2, ASP.NET Core 6) the issue is reproducible with a test app that is stressing the CPU for 1.5 seconds every 300-600 seconds at random. These short spikes are not visible on the azure portal under Insights. I've not tried running it without CPU limits yet (the limit is 50m right now). Increasing the probing timeout to 3 seconds fixes the issue. Querying the readiness probe with curl from a pod in the same namespace shows that sometimes it takes more that 2 seconds to respond, but this is not visible in the app's logs, all response times are short. Regular ICMP pings are always quick, no more than 20ms.
I've had another dig into what might be going on here, because this keeps making less and less sense. Given that load can reproduce the issue (thanks Aleksey), I'm wondering if it's down to the go net/http client's implementation of timeouts and concurrency. Every probe gets associated with a worker object in the kubelet, and the . Every probe period, when that worker gets a chance to run, it opens a client connection and makes a request. The request is done using the deadline semantics, so the current time when the request is being constructed is marked and then the timeout is to that. Once that deadline is passed, even if the all the data has been returned and is waiting in the socket buffer, then the call fails with a timeout. So, I'm wondering, is there any potential thread-thrashing happening here? How many goroutines are getting queued up during these load events, and are they getting stuck inside their deadline waiting for CPU time, then getting timed out? There's also this page which suggests that there might be a problem with the dial timeout in net/http which might be another direction to look of starting the connection. Is your test app tunable? Could you try probes with timeouts of [1,2,3,4,5] seconds and stress the CPU for [1.5,2.5,3.5,4.5,5.5] seconds to see if it's directly attributable to the CPU stress?
, I don't have time to look into it further at the moment unfortunately. We'll increase the probing timeouts for now. I tried with the CPU limit removed and the issue is gone (it's been running for 3 hours now). I think we might be looking at different issues because this is not just a kubelet issue in my case (curl is also seeing the delays). Sharing the code just in case (the logic is in ): The curl test:
Curl from another container could be getting delayed by the pod being in an unready state. If you turn off the readiness probe and leave the liveness probe, I suspect much of that would go away, with curl timeouts happening when the livenessProbes start to fail. To be clear, you're taking the CPU limit off your spiky app? In which case it probably is just that your container is using too much CPU, and not responding to the probe within the timeout. would prevent execution of processes in the container once it's used 50m of CPU in any timeslice.
With the CPU limit removed I did get one probe failure eventually after running it for 20 hours.
Observed same issue with (k8s v1.22.1,dataceneter,Rhel 8.3) and timeoutSeconds: 10 resolved our issue.
What kind of timeout can you use if that workload is just running in a container?
Hi yesterday I recreated my self-hosted kubernetes clusters (they're deployed on EC2 intances) with the latest versions of kubernetes, 1.23.4, using kubeadm and containerd. After deploying our app on both clusters and experiencing load spikes on the "prod" cluster, my app pods kept restarting with the and on the second container within the pod I saw the previous post and it worked for me, have no clue why, thank you
Has anyone ever run into issues where the probes fail occasionally with EOF (what does an EOF imply, no response?)? I have been able to confirm that the service is responsive and the endpoint works but would fail from time to time with a connection refused. Thanks
We have encountered the same issue past many days, initially we thought this problem can be at the api level, Just wanted to confirm once if this error has some issue with probes
I have the same issue of kubelet timing out on the liveness and readiness probe on a pod only during the load testing. The resources on the pod do not go beyond the limits. The strange thing is the same application pod works well in the same cluster but on the different nodes. That isolates the problem to a particular node. I did a curl of readiness and liveness probes within the pod and also from the other pods within the same node pods and outside this node pods as the probes are in failed state. The URL on the liveness probes and readiness probes work well giving 204 return status. It is only the kubelet that has problem in reaching to the pods during the test. Kubeproxy looks good on the node. Restarting the kubelet and docker did not help. Restarting the node too did not help. Increasing the timeouts on the probes did not help. After the testing, the pods return to normal healthy state. Kubernetes bare metal v1.14.3. Is there any way to troubleshoot this problem from the aspect of kubelet?
On v1.21.2 AKS - same issue. Resolved using fix increasing timeoutSeconds to 10. Default-backend has timeoutSeconds set to 5 by default. Will attempt to decrease from 10 and look for anything useful. Odd issue since it's only impacting single cluster -- others with same config intermittently show the same error but do not go into CrashLoop.
for me it worked after increasing the livenessprobe.timeoutSeconds to 15 seconds and the liveness probe errors stopped. seems like my app is taking a lot oftime to respond with https and slow network and low CPU resources. Thanks for the suggestion.
On the node where your Pod is running (can be a worker node) Source is the .
We are experiencing the same issue on . If workaround mentioned above works, it comes with the downside that not all application containers would have curl installed. Some of them might be distroless. And some might be upstream containers that we don't own and we can't easily install another library like curl. So it would ultimately still be nice to find out why the kubelet runs into these issue.
Did you ever find the reasoning or solution for this? Running into a similar issue
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Experiencing this problem in production level application. I am running a application on Kubernetes and liveness probe keeps killing the application time to time, usually on the high load. There was no problem with or without high load. Initial settings liveness probe, readiness probe both using app's endpoint which just returns . with What I've done check settings increase probe timeout from 2 -10 change probe method: to thing double the replicas(10 -20) And it is still happening. Now I'm about to remove liveness probe after finding out . I guess readiness probe is enough for app.
It seems really problematic if users are abandoning liveness probes because they are not reliable. We should not ignore this. Do we need to find a way to give the probers more CPU scheduling time? Higher realtime prio? Maybe a distinct process from kubelet?
I guess first step is to prove which side of the link is at fault. If we can prove that it was the pod that timed out, at least we can say "we do X, Y, Z to give probes the highest prio on the kubelet side, you need to do A, B, C to make sure you respond to to them" and "look here to see the proof that we sent the request - you didn't respond". If switching to exec works but http doesn't, I am baffled. Calling exec should eat MORE cpu time than an HTTP probe, I think.
but not kubelet cpu time :) , I think the kubelet is the bottleneck, since is the one that has to spawn the goroutines for the probes, with exec it delegates the task to the container runtime via CRI (ref )
It has to make a connection through CRI, so it should be at least as much, no? Antonio Ojea
To be honest I just came to the conclusion by the comments and analysis exposed on this issuehttpProbe fails with timeout and curl exec in container doesn't , but you are right, is not that golang can not handle 100 requests per second ... If someone in the thread with a reproducer can get a kubelet pprof of cpu that will help to clarify things
deployment pprof shows a considerable amount of time dialing/http probing ! I have to compare against too Checking my machine default values: Resources: Timers: 3 probes per container every second create a 3 TCP connections per second (it has to wait 60 second on TIME-WAIT state holding the socket) that consume an ephemeral port (we have per tuple) and a conntrack entry. Once the TIME-WAIT timer expires it frees the conntrack entry and the socket. In the example above, we create a connections at a rate of 3 per pod with 100 pods = 300 connections per second, that will be freed after 60 seconds. So, we'll have a constant number of 59x300 = sockets in TIME-WAIT and conntrack entries just for having the probes that doesn't seem to be worrisome if we have conntrack entries, but it leaves only - ephemeral ports for things like DNS and any other hostNetwork stuff
/assign
Why is kubelet opening new connections for every probe? Shouldn't it re-use the connection when it can?
I's been that way since v1.2 that is the conntrack wait period, it will disappear once the socket closes. The TIMEWAIT period for the socket is a constant of the kernel, however, users can override it per socket using the SOLINGER options. I see 2 possible directions: connections that means we have to refactor all the probes (for http I think that golang will handle it) SOLINGER to 0 or a much lower value (this is probing so the risk of loosing non-ack data is not important) and decrease the TIMEWAIT value
Option 1 is the "right" answer, but may be harder. The probe code isn't my favorite... Antonio Ojea
let me see how far I go with 1
I did 1 and 2 XD
I have to leave 1 out, the results are very confusing and not deterministic. The way http reuses connections c differs between http1 and http2, and golang stdlib behave also different for those protocols. I think that option 2 for probes is the best option, less disruptive and the results I obtained locally are very promising.
Will you revisit HTTP reuse? Antonio Ojea
That's a can of worms, http1 Vs http2 thing is complex , let's this soak and revisit one we are able to get some data about the impact of this change
FWIW - I found this article useful in explaining TIME-WAIT and potential options to use in Linux -
I am still facing the same issue.
Same here, still facing this issue.
please indicate with which version are you facing the issue and how many probes are configured in the node, and if they are http/TCP probes or exec probes
hi, i'm facing this issue right now, too, in on of my namespaces after Digital Ocean did an automatic Update to 1.24.16-do.0. it pulls the image successfully, but then crashes and throws me Liveness probe failed: Get "http://": context deadline exceeded (Client.Timeout exceeded while awaiting headers) when curling the IP inside the pod i'm getting a 137 error. i may have about 5 or more http/TCP probes in the node.
No, that would fail to detect a pod doing graceful shutdown, wouldn't it? Kubelet needs to know whether the pod is accepting new connections, not just whether it's continuing to process old connections.
do you have example of how to implement option 2 (since option 1 doesn't seem to be doable at this point)? We are experiencing these a lot and would like a proper way to address all our http probes. Can this happen at the probe spec level or does this change need to happen at the host level? If so what changes are you recommending we add to the bootstrap in order to resolve this. Having it documented would greatly be helpful here.
it is already implemented in
ok so still happening for me on 1.26 host, have to confirm on 1.27. Do you know which release this made it into? Seems like there is an open PR to fix the http reuse which would fix option 1.
Looks like it was part of 1.27. I’ll upgrade that cluster this week.
did you get a change to upgrade and did you see any improvement?
Hello, I am facing this intemitient issue since some time. Usually it can heal itself after a while, or killing the project contour pods which causing restarting them fixing the problem, until next time. Some of the cluster output logs: "reason": "Unhealthy", "message": "Liveness probe failed: Get " net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)", "source": { "component": "kubelet", "host": "" }, "reason": "Unhealthy", "message": "Readiness probe failed: Get " context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "source": { "component": "kubelet", "host": "" }, "reason": "Unhealthy", "message": "Liveness probe failed: command "/bin/calico-node -felix-live -bird-live" timed out", "source": { "component": "kubelet", "host": "" }, "firstTimestamp": "2024-01-05T18:21:46Z", "lastTimestamp": "2024-04-25T13:46:25Z", "involvedObject": { "kind": "Pod", "namespace": "kube-system", "name": "coredns--lw7mj", "uid": "xyz", "apiVersion": "v1", "resourceVersion": "" }, "reason": "NodeNotReady", "message": "Node is not ready", "source": { "component": "node-controller" }, "firstTimestamp": "2024-04-24T16:33:27Z" "reason": "Unhealthy", "message": "Liveness probe failed: Get " context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "source": { "component": "kubelet", "host": "" }, "firstTimestamp": "2024-04-24T16:52:25Z", "lastTimestamp": "2024-04-25T13:51:05Z", "count": 3, 2024-03-27 04:48:27.774 [INFO][1] 350: Main client watcher loop 2024-03-27 04:52:59.896 [ERROR][1] 290: Error getting cluster information config ClusterInformation="default" error=Get "": context deadline exceeded 2024-03-27 04:52:59.896 [ERROR][1] 269: Failed to verify datastore error=Get "": context deadline exceeded 2024-03-27 04:53:01.900 [ERROR][1] 304: Received bad status code from apiserver error=an error on the server ("[+]ping okn[+]log okn[-]etcd failed: reason withheldn[+]poststarthook/start-kube-apiserver-admission-initializer okn[+]poststarthook/generic-apiserver-start-informers okn[+]poststarthook/priority-and-fairness-config-consumer okn[+]poststarthook/priority-and-fairness-filter okn[+]poststarthook/start-apiextensions-informers okn[+]poststarthook/start-apiextensions-controllers okn[+]poststarthook/crd-informer-synced okn[+]poststarthook/bootstrap-controller okn[+]poststarthook/rbac/bootstrap-roles okn[+]poststarthook/scheduling/bootstrap-system-priority-classes okn[+]poststarthook/priority-and-fairness-config-producer okn[+]poststarthook/start-cluster-authentication-info-controller okn[+]poststarthook/aggregator-reload-proxy-client-cert okn[+]poststarthook/start-kube-aggregator-informers okn[+]poststarthook/apiservice-registration-controller okn[+]poststarthook/apiservice-status-available-controller okn[+]poststarthook/kube-apiserver-autoregistration okn[+]autoregister-completion okn[+]poststarthook/apiservice-openapi-controller oknhealthz check failed") has prevented the request from succeeding status=500 2024-03-27 05:44:29.841 [INFO][1] 378: Terminating main client watcher loop
I am also facing the same issue Readiness probe failed: Get “http://*:8085/actuator/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
makes sense, but this is not the only issue at play here. I love how everyone completely ignored in and in , the latter didn't even got a single upvote. I am running 1.27.7, and still having this issue, so it is not [only] . It does look to me like mentioned above is exactly what is happening (albeit in addition to other issue[s]), and I did notice random DNS resolution failures in my logs too. I am going to try what they suggested: Can't immediately tell if it helps of not, as there isn't a reliable way to reproduce the issue. If I get it all correct, the following config: Gets me to at least 4s on potential DNS resolution request, plus 1s extra for an actual probe - my probe will need at least 5s timeout. There got to be an underlying issue the DNS resolution request fails in the first place though. In my GKE clusters, Google deploys a coredns service as a local cache on every node, so it should be a local name resolution. Why would it fail?
This issue become a magnet for multiple unrelated issues, for the case of http probes like this I may be missing something , but how can dns be related if there is no dns resolution on the kubelet and if those value apply to the pods, and is kubelet who does the http query?
you are right. I got excited a little too soon, because DNS errors clicked and I assumed that containers are addressed by name. I later validated in code that is wasn't the case, and also, you are right that it would logically make zero sense since probe is done by kubelet and pod dns options apply to the container. And surely, my issues continue even with these dns options. My leading theory right now is the AWS Load Balancer Controller. It sets up every node of the cluster into the target group, even these that are not running target pods (in instance mode, since I am using cilium, not VPC CNI, so IP mode is not available to me). It also do not set up any healthchecks by default, so my users would typically set it to their NodePort service. What happens as a result is, deployments with a low number of replicas receiving health checks requests from all other nodes in the cluster. Consider a rather large cluster i.e. hundreds of 16xlarge EC2 instances... some nodes that happen to run low replica deployments really struggle. I just watched one node conntrack table grow 50k entries in less than a second. I think I'm gonna have to address this issue first by adding node selectors and fixing health checks on all the ingresses before I can figure out if there are any other remaining issues in this noise.
no worries, we understand the frustration of users with these enigmatic problems, that is why is important for users to report the behaviors and the details of the investigations so we can try to figure out the problems and solve them, if we focus on the symptoms and the conclusions every individual arrives , then little progress can be done. Probes are being revisited these days so these feedback is extremely valuable cc: Can you expand on why the users decide to use a NodePort Service as a target?
sorry, I think I got you even more confused. I was talking about AWS LB health checks, not kubelet probes. Let me try to sum it up better on the off chance someone else got the same thing going on in their environment: is Cilium with disabled eBPF Host-Routing but kube-proxy being still replaced (results in all cilium traffic still being a subject for iptables/conntrack). Load Balancer Controller in Instance mode backed by NodePort services, because IP mode requires VPC CNI or other VPC IPAM capable CNI. using above class and configuring annotation with other healthcheck related annotations, effectively targeting the NodePort service. large multi-tenant cluster with hundreds of nodes, with many of deployments being small i.e. under 5 replicas. of these small services tends to have their own ingress each. to anti affinity and topology spread constraints, larger deployments typically force lots of these small deployments onto same nodes, resulting often one particular node of the cluster running often a single replica of the deployment NodePort service is backed by. This results in each of the hundreds of AWS LBs sending health check request to every node of the cluster (every node on the target group) which is routed by cilium to a small number of nodes running target pods. That exhausts conntrack table which in turn leads to other issues, including issues with kubelet probes, which is what raised the alerts... So I do not think this is related to this KEP, this is completely separate topic. I got to make all my users to set to their respective node pools to limit what is to AWS LB target groups, and also make them fix the health check using the Cilium echo endpoint (LB should only be concerned with the node health, not health of the pods). Also I should probably make my users consolidate Ingress Groups so the number of LBs goes down as well. I expect conntrack table to get back under control with this, and I wonder if my other issues i.e. kubelet unexplained porobe failures stops after that.
Just a hypothesis: If the pod gets timeout on DNS query, kubelet will get timeout on wating for HTTP response headers.

Discussion:
<ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package prober import ( "context" "fmt" "net" "net/http" "sync" "sync/atomic" "testing" "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/client-go/kubernetes/fake" "k8s.io/client-go/tools/record" kubepod "k8s.io/kubernetes/pkg/kubelet/pod" "k8s.io/kubernetes/pkg/kubelet/prober/results" "k8s.io/kubernetes/pkg/kubelet/status" statustest "k8s.io/kubernetes/pkg/kubelet/status/testing" kubeletutil "k8s.io/kubernetes/pkg/kubelet/util" utilpointer "k8s.io/utils/pointer" ) // TCP sockets goes through a TIME-WAIT state (default 60 sec) before being freed, // causing conntrack entries and ephemeral ports to be hold for 60 seconds // despite the probe may have finished in less than 1 second. // If the rate of probes is higher than the rate the OS recycles the ports used, // it can consume a considerable number of ephemeral ports or conntrack entries. // These tests verify that after certain period the probes keep working, if the probes // don't close the sockets faster, they will start to fail. // The test creates a TCP or HTTP server to fake a pod. It creates 1 pod with 600 fake // containers each and runs one probe for each of these containers (all the probes comes // from the same process, same as in the Kubelet, and targets the same IP:port to verify // that the ephemeral port is not exhausted. // The default port range on a normal Linux system has 28321 free ephemeral ports per // tuple srcIP,srcPort:dstIP:dstPort:Proto: /proc/sys/net/ipv4/ip_local_port_range 32768 60999 // 1 pods x 600 containers/pod x 1 probes/container x 1 req/sec = 600 req/sec // 600 req/sec x 59 sec = 35400 // The test should run out of ephemeral ports in less than one minute and start failing connections // Ref: https://github.com/kubernetes/kubernetes/issues/89898#issuecomment-1383207322 func TestTCPPortExhaustion(t *testing.T) { const ( numTestPods   = 1 numContainers = 600 ) if testing.Short() { t.Skip("skipping TCP port exhaustion in short mode") } tests := []struct { name string http bool // it can be tcp or http }{ {"TCP", false}, {"HTTP", true}, } for _, tt := range tests { t.Run(fmt.Sprintf(tt.name), func(t *testing.T) { podManager := kubepod.NewBasicPodManager(nil) podStartupLatencyTracker := kubeletutil.NewPodStartupLatencyTracker() m := NewManager( status.NewManager(&fake.Clientset{}, podManager, &statustest.FakePodDeletionSafetyProvider{}, podStartupLatencyTracker), results.NewManager(), results.NewManager(), results.NewManager(), nil, // runner &record.FakeRecorder{}, ).(*manager) defer cleanup(t, m) now := time.Now() fakePods := make([]*fakePod, numTestPods) for i := 0; i < numTestPods; i++ { fake, err := newFakePod(tt.http) if err != nil { t.Fatalf("unexpected error creating fake pod: %v", err) } defer fake.stop() handler := fake.probeHandler() fakePods[i] = fake pod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID:       types.UID(fmt.Sprintf("pod%d", i)), Name:      fmt.Sprintf("pod%d", i), Namespace: "test", }, Spec: v1.PodSpec{}, Status: v1.PodStatus{ Phase:  v1.PodPhase(v1.PodReady), PodIPs: []v1.PodIP{{IP: "127.0.0.1"}}, }, } for j := 0; j < numContainers; j++ { // use only liveness probes for simplicity, initial state is success for them pod.Spec.Containers = append(pod.Spec.Containers, v1.Container{ Name:          fmt.Sprintf("container%d", j), LivenessProbe: newProbe(handler), }) pod.Status.ContainerStatuses = append(pod.Status.ContainerStatuses, v1.ContainerStatus{ Name:        fmt.Sprintf("container%d", j), ContainerID: fmt.Sprintf("pod%d://container%d", i, j), State: v1.ContainerState{ Running: &v1.ContainerStateRunning{ StartedAt: metav1.Now(), }, }, Started: utilpointer.Bool(true), }) } podManager.AddPod(&pod) m.statusManager.SetPodStatus(&pod, pod.Status) m.AddPod(&pod) } t.Logf("Adding %d pods with %d containers each in %v", numTestPods, numContainers, time.Since(now)) ctx, cancel := context.WithTimeout(context.Background(), 59*time.Second) defer cancel() var wg sync.WaitGroup wg.Add(1) go func() { defer wg.Done() for { var result results.Update var probeType string select { case result = <-m.startupManager.Updates(): probeType = "startup" case result = <-m.livenessManager.Updates(): probeType = "liveness" case result = <-m.readinessManager.Updates(): probeType = "readiness" case <-ctx.Done(): return } switch result.Result.String() { // The test will fail if any of the probes fails case "Failure": t.Errorf("Failure %s on contantinerID: %v Pod %v", probeType, result.ContainerID, result.PodUID) case "UNKNOWN": // startup probes t.Logf("UNKNOWN state for %v", result) default: } } }() wg.Wait() // log the number of connections received in each pod for debugging test failures. for _, pod := range fakePods { n := pod.connections() t.Logf("Number of connections %d", n) } }) } } func newProbe(handler v1.ProbeHandler) *v1.Probe { return &v1.Probe{ ProbeHandler:     handler, TimeoutSeconds:   1, PeriodSeconds:    1, SuccessThreshold: 1, FailureThreshold: 3, } } // newFakePod runs a server (TCP or HTTP) in a random port func newFakePod(httpServer bool) (*fakePod, error) { ln, err := net.Listen("tcp", "127.0.0.1:0") if err != nil { return nil, fmt.Errorf("failed to bind: %v", err) } f := &fakePod{ln: ln, http: httpServer} // spawn an http server or a TCP server that counts the number of connections received if httpServer { var mu sync.Mutex visitors := map[string]struct{}{} go http.Serve(ln, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() if _, ok := visitors[r.RemoteAddr]; !ok { atomic.AddInt64(&f.numConnection, 1) visitors[r.RemoteAddr] = struct{}{} } })) } else { go func() { for { conn, err := ln.Accept() if err != nil { // exit when the listener is closed return } atomic.AddInt64(&f.numConnection, 1) // handle request but not block go func(c net.Conn) { defer c.Close() // read but swallow the errors since the probe doesn't send data buffer := make([]byte, 1024) c.Read(buffer) // respond conn.Write([]byte("Hi back!n")) }(conn) } }() } return f, nil } type fakePod struct { ln            net.Listener numConnection int64 http          bool } func (f *fakePod) probeHandler() v1.ProbeHandler { port := f.ln.Addr().(*net.TCPAddr).Port var handler v1.ProbeHandler if f.http { handler = v1.ProbeHandler{ HTTPGet: &v1.HTTPGetAction{ Host: "127.0.0.1", Port: intstr.FromInt(port), }, } } else { handler = v1.ProbeHandler{ TCPSocket: &v1.TCPSocketAction{ Host: "127.0.0.1", Port: intstr.FromInt(port), }, } } return handler } func (f *fakePod) stop() { f.ln.Close() } func (f *fakePod) connections() int { return int(atomic.LoadInt64(&f.numConnection)) } </ins>

<ins> //go:build !windows // +build !windows /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package probe import ( "net" "syscall" ) // ProbeDialer returns a dialer optimized for probes to avoid lingering sockets on TIME-WAIT state. // The dialer reduces the TIME-WAIT period to 1 seconds instead of the OS default of 60 seconds. // Using 1 second instead of 0 because SO_LINGER socket option to 0 causes pending data to be // discarded and the connection to be aborted with an RST rather than for the pending data to be // transmitted and the connection closed cleanly with a FIN. // Ref: https://issues.k8s.io/89898 func ProbeDialer() *net.Dialer { dialer := &net.Dialer{ Control: func(network, address string, c syscall.RawConn) error { return c.Control(func(fd uintptr) { syscall.SetsockoptLinger(int(fd), syscall.SOL_SOCKET, syscall.SO_LINGER, &syscall.Linger{Onoff: 1, Linger: 1}) }) }, } return dialer } </ins>

<ins> //go:build windows // +build windows /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package probe import ( "net" "syscall" ) // ProbeDialer returns a dialer optimized for probes to avoid lingering sockets on TIME-WAIT state. // The dialer reduces the TIME-WAIT period to 1 seconds instead of the OS default of 60 seconds. // Using 1 second instead of 0 because SO_LINGER socket option to 0 causes pending data to be // discarded and the connection to be aborted with an RST rather than for the pending data to be // transmitted and the connection closed cleanly with a FIN. // Ref: https://issues.k8s.io/89898 func ProbeDialer() *net.Dialer { dialer := &net.Dialer{ Control: func(network, address string, c syscall.RawConn) error { return c.Control(func(fd uintptr) { syscall.SetsockoptLinger(syscall.Handle(fd), syscall.SOL_SOCKET, syscall.SO_LINGER, &syscall.Linger{Onoff: 1, Linger: 1}) }) }, } return dialer } </ins>

DisableKeepAlives:  true, Proxy:              http.ProxyURL(nil), DisableCompression: true, // removes Accept-Encoding header <ins> // DialContext creates unencrypted TCP connections // and is also used by the transport for HTTPS connection DialContext: probe.ProbeDialer().DialContext, </ins> }) return httpProber{transport, followNonLocalRedirects} }

type tcpProber struct{} <del> // Probe returns a ProbeRunner capable of running an TCP check. </del> <ins> // Probe checks that a TCP connection to the address can be opened. </ins> func (pr tcpProber) Probe(host string, port int, timeout time.Duration) (probe.Result, string, error) { return DoTCPProbe(net.JoinHostPort(host, strconv.Itoa(port)), timeout) }

// If the socket fails to open, it returns Failure. // This is exported because some other packages may want to do direct TCP probes. func DoTCPProbe(addr string, timeout time.Duration) (probe.Result, string, error) { <del> conn, err := net.DialTimeout("tcp", addr, timeout) </del> <ins> d := probe.ProbeDialer() d.Timeout = timeout conn, err := d.Dial("tcp", addr) </ins> if err != nil { // Convert errors to failures to handle timeouts. return probe.Failure, err.Error(), nil