I'm Alex Movergan, DevOps team lead at Altenar. I focus on automation in general and on improving troubleshooting skills within my team. In this article, I'll share a captivating tale that revolves around Kubernetes, validation webhooks, kubespray, and Calico. Join me on this DevOps journey as we explore real-world scenarios and practical solutions, unraveling the intricacies of troubleshooting in a Kubernetes environment. Continue reading to delve into the world of Kubernetes troubleshooting. Every time a new engineer joins our team, they undergo an onboarding process before diving into production. As part of this process, they build a small instance of our Sportsbook. Our Sportsbook operates on Kubernetes, which is deployed on several VMWare clusters. This particular story takes place during the onboarding of one of our new engineers, highlighting essential technological aspects that will provide a better understanding of our environment.   Our infrastructure is based on the following technologies: Kubernetes cluster is deployed by kubespray on top of VMWare. Calico in IPVS mode is used as CNI. MetaLLB used for internal load balancers Flux for Kubernetes infrastructure settings such as network policy, namespaces, and PSS.  The task is to build a Kubernetes cluster using our internal terraform module that invokes VMware provider and Kubespray under the hood. The cluster consists of six workers and one master. It all began when our new engineer approached me and my senior colleagues for assistance after days of troubleshooting. Initially, the issue seemed simple: the MetaLLB AddressPool deployment was failing, accompanied by the following error:   "What is happening here?" you may ask. Well, during installation, MetaLLB creates a ValidatingAdmissionWebhook. This webhook is called by the kube-api server to validate an object before creation. However, it is currently encountering a timeout, leading to the failure.   Let's start with the basics. While calling the same URL from other pods works fine, the kube-api continues to flood logs with the error. This leads us to the conclusion that the MetaLLB controller is present and functioning correctly. To isolate the issue, we eliminated other components. We scaled down the flux pods to 0 and removed all network policies from the cluster. However, the result remains the same: the webhook service is accessible from other pods but kube-api. Therefore, we can conclude that it is not a network policy problem. To replicate the issue and proceed further, we face a challenge. The kube-api pod does not have a shell, so we cannot directly access it and make the same URL call. To overcome this, we added a sidecar container. Since kube-api is not a regular pod but a static pod on the master node, we SSHed into the VM and modified the /etc/kubernetes/manifests/kube-apiserver.yaml file. Specifically, we addd netshoot as a sidecar container: An important aspect to note is the presence of the hostNetwork: true attribute. This attribute indicates that the kube-api pod is running on a node and has access to the node's network. To summarize the situation, we encountered the same outcome: the curl command failed with a timeout: At this point, we no longer needed to rely solely on kube-api logs; we can now reproduce the issue ourselves, which simplifies the troubleshooting process. Moving forward, let's employ heavy artillery a powerful tool for network troubleshooting: tcpdump. Personally, I find tcpdump indispensable in network troubleshooting scenarios (and it always comes to network troubleshooting, doesn't it?). Open tcpdump in one terminal tab and run the curl command in another:   If you paid close attention, you may have noticed that we used curl to connect to10.202.145.97:443, while tcpdump was running on 10.202.146.97.9443. This distinction arises from the way Calico operates, utilizing iptables and ipset to convert service IP:port into pod IP:port. In this case, 10.202.146.97.9443 represents the address and port of the metallb-controller pod.   In conclusion, we are sending traffic but not receiving any response.   The next step involves identifying the network segment where the traffic is being lost. To achieve this, we add Netshoot to the metallb-controller pod, allowing us to verify if the traffic reaches the pod.   Let's repeat the experiment, this time running tcpdump on both ends of the traffic pipeline: On kube-api pod: On metallb-controller pod: This is becoming quite intriguing. It appears that the traffic successfully reaches the final destination, with the metallb-controller providing replies each time. However, the issue lies in delivering packets back to the kube-api pod. After spending approximately two hours on trial and error, we decided to take a lunch break. Despite our break, we couldn't stop discussing the issue and managed to come up with a few other tests we can attempt to resolve it. Let's try to create a simple pod with netshoot and 'hostNetwork' option set to 'true'. A few "ctr+c" "ctr+v" and we have another static pod running on the same master node.   We proceeded with the same steps, and as expected, curl failed with a timeout once again. To simplify the testing further, we attempted to ping the metallb-controller from our netshoot container. On netshoot: On metallb-controller: The results were quite similar to what we observed earlier with curl. If you have reached this point, it is evident that the issue lies on the network side. Without delving into the specific documentation we explored regarding Calico, IPVS, and Kubernetes networking, I must say it has been quite an adventure. The next idea we had was to ping the kube-api server pod from the metallb-controller: Wait, what? Excuse me? What on earth is happening here? We attempted the same command multiple times and even manually entered the correct IP address, but the outcome remained unchanged. What could possibly be the issue? Not to mention, we also tried using telnet and curl, and they all failed in the same manner.  Another tool that I find quite handy is strace. So, let's strace telnet and connect it to the kube-api pod IP: Upon examination, we noticed that the connect() function returned an error of "-1 EINVAL (Invalid argument)". Intrigued, we decided to dive into the Linux kernel functions documentation. How exciting!  Funny enough, the first result from Google did not contain the description of this failure result, Luckily, the second result proved to be much more informative: In conclusion, there seems to be a network configuration issue with the IP addresses. The question remains, though: Where exactly is the problem originating from?   At this point, we knew we are on the right track. We started from Calico IPPool configuration: Then terraform code we used for deployment: Next pod cidr on nodes:  Ah, we have identified the discrepancy! It appears that Calico has a "blockSize: 26" setting, whereas the "k8s_network_node_prefix" is set to 25.   Therefore, the network configuration is incorrect. I must mention that this is a test cluster created solely for the purpose of exploring cluster building with our toolchain. Hence, all the networks in the cluster are "/24". However, even though it is a test cluster, it was deployed with six workers and one master. Simple network calculations reveal that a "/24" network cannot be divided into seven "/26" subnets, let alone seven "/25" subnets:   Now, the question arises: Why did this occur? We passed a specific value for a parameter. To uncover the root cause, let's delve into the code of kubespray:  This part: And this part too: What we realized is that even though we used  in our terraform module code, this variable was never used by kubespray because calico_pool_blocksize comes first and it has a default value: At this juncture, we concluded our investigations and successfully identified the root cause. With this newfound clarity, our action plan is crystal clear: Obtain a larger network range.   Rectify the Terraform module for Kubernetes deployment. Perform a complete redeployment of the cluster from scratch.  Even though the root cause is trivial as always, as a result of our troubleshooting experience, we have decided to document our steps in this article, in the hope that it may prove valuable to others facing similar challenges. Thank you very much for your attention. I remain available to address any questions or comments you may have.  DevOps Automation Manager