Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As we update Kube-proxy to version 1.21, we observe a stream of following error entries within the Kube-proxy logs: Skipping invalid IP: ". As we do not see anything wrong with the EKS cluster itself, this message should not be reported as an error. an EKS cluster v1.21 and launched a managed nodegroup within this cluster. use the managed add-on or manage Kube-proxy yourself. Either way, the EKS provided kube-proxy image (kube-proxy:v1.21.2-eksbuild.2) is based on the upstream image and this error can be traced to the upstream code as I will be outlining in the next section. logs for the kube-proxy pod: kubectl logs -n kube-system <kube-proxy-pod-nameIn my replication, I could analyze this: These errors were recurring in the kube-proxy logs every 15 minutes which I concluded after watching the log stream for a 1+ hour window. To be able to make sense of the logs better, I increased verbosity of kubeproxy to 9: kubectl patch ds kube-proxy -n kube-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command/1", "value":"--v=9"}]' In the logs, I could notice the following snippet recurring periodically (every 15 minutes): This hinted that a 'resync' was behind causing this errors being pushed every 15 minutes, however, before diving deeper into the root cause there, I searched for the origin of 'Skipping invalid IP: ' line and tried to reverse engineer back to the root cause for these errors. For this, I went through the '' in kube-proxy upstream github (release-1.21 branch). I started with the 'MapIPsByIPFamily' function which is where this error is handled and thrown [1]. This error is returned to it by a child function getIPFamilyFromIP [2]. The purpose of this function is to identify if the IP string passed is IPv6 or IPv4. So, I backtracked a bit from the 'MapIPsByIPFamily' function and observed that it was referenced on line 186 by 'newBaseServiceInfo' in the file of kube-proxy [3]. Here I noticed that 'service.Spec.ExternalIPs' is passed as string to the 'MapIPsByIPFamily' function. Now, not all services have an ExternalIP defined in their spec. This is when an empty string '' is passed to the latter and hence the snippet : [klog.Errorf("Skipping invalid IP: %s", ip)] translates to 'Skipping invalid IP: ' in the actual error message. However, in the following error handling block: Rather than simply logging an error, there could be a check for this case of an empty string (as this is pretty common for clusters to have services without ExternalIP defined). If it is not possible to handle this error in this snippet for some reason, the verbosity level could be highered to atleast 4 so that the default kube-proxy image that runs at level 2 generally is not bombarded with a stream of recurring errors that are not actually errors. Can you please help fix this issue? [1] [2] [3] Kubernetes version (use ): 1.21 Cloud provider or hardware configuration: EKS with EC2 worker nodes OS (e.g: ): Amazon Linux 2 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
/triage accepted nice report thanks /cc
Thanks. is your service defined with ?
I performed a 'kubectl get svc <svc_name-o yaml' and I could confirm that my services do not have an explicit .Spec.ExternalIPs = {""}. The output for one of them was as follows:
/assign
There was a similar problem with blank spaces too on the field , in this case the IPs were invalid because a blank space was appended to the IP If is not an ExternalIPs, then it has to be in the Status.LoadBalancer.Ingress.IP I think that if the function errors, we should return the error instead of logging inside, so the caller can handle it and raise a meaningful error, pointing the user to the specific service and field , what do you think?
iirc we have to ignore and log the error. Because there some cloud providers that sets this as " ". And since there is no validation in place for that, we can not afford to add validation now (break the api). What i am looking into is limit the amount of times we log this error on proxy or remove it at all.
yeah, validation train has left the station some time ago :smile: I was trying to mean that the error should point to the service causing the issue, so users can act on it ...
Thanks for taking a look guys, really appreciate your help with this I could see that the fix has been planned as part of v1.23 milestone. Can you please clarify if this fix can/would be backported to previous Kubernetes versions and if there is a timeline for the same. This will help plan if we will continue to see such errors for some time now. Thanks again, good day !

Discussion:
"net" "net/http" "strconv" <ins> "strings" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types"

if ipFamily, err := getIPFamilyFromIP(ip); err == nil { ipFamilyMap[ipFamily] = append(ipFamilyMap[ipFamily], ip) } else { <del> klog.ErrorS(nil, "Skipping invalid IP", "ip", ip) </del> <ins> // this function is called in multiple places. All of which // have sanitized data. Except the case of ExternalIPs which is // not validated by api-server. Specifically empty strings // validation. Which yields into a lot of bad error logs. // check for empty string if len(strings.TrimSpace(ip)) != 0 { klog.ErrorS(nil, "Skipping invalid IP", "ip", ip) } </ins> } } return ipFamilyMap