Repository: canonical/microk8s

Issue:
NodePort services with externalTrafficPolicy set to Local don't work on worker nodes
<!--
   Thank you for submitting an issue. Please fill in the template below
   information about the bug you encountered.
-->

#### Summary
NodePort services with externalTrafficPolicy set to Local don't work on worker nodes, they only work on master nodes.  Please reference bug for another example:  https://github.com/canonical/microk8s/issues/2971
Setting the externalTrafficPolicy to Cluster fixes the iptables issues, however this will not allow for actual sourceip to be accessible.

#### What Should Happen Instead?
All nodes, including worker nodes should respond on the nodeport as configured and send traffic to the local pod.

#### Reproduction Steps


install daemonset with nodeport service enabled, with externalTrafficPolicy set to Local.

#### Introspection Report
[inspection-report-20231221_150119.tar.gz](https://github.com/canonical/microk8s/files/13743270/inspection-report-20231221_150119.tar.gz)

You can see various log entries related to the iptables configuration:
 for example: 
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "istio-ingress-int/istio-internal-ingress-gw:status-port has no local endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 31081 -j DROP

#### Can you suggest a fix?
<!-- (This section is optional). How do you propose that the issue be fixed? -->

#### Are you interested in contributing with a fix?
<!-- yes/no, or @mention maintainers. Community contributions are welcome. -->

<!-- Thank you for making MicroK8s better -->



Discussion:
Hi @dfry 

The iptables rule you have listed seems to come from Istio? I'm not then sure if this comes from Kubernetes itself or Istio. In any case, this does not look like a MicroK8s issue to me. 

Do you have a reproducer on a MicroK8s cluster without Istio?
Hi @neoaggelos 

The nodeport service in this case belongs to an istio ingress gateway daemonset.  If you look at the other issue that I mentioned, you will see an example not related to istio: https://github.com/canonical/microk8s/issues/2971

Cheers
OK, I see, thanks for this, I was able to reproduce the issue. I started a 2-node cluster (u1 is control plane, u2 is worker):

```
root@u1:~# k get node -o wide
NAME   STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
u2     Ready    <none>   12m   v1.29.0   10.0.0.79     <none>        Ubuntu 20.04.6 LTS   6.2.0-39-generic   containerd://1.6.15
u1     Ready    <none>   12m   v1.29.0   10.0.0.93     <none>        Ubuntu 20.04.6 LTS   6.2.0-39-generic   containerd://1.6.15
```

In the kube-proxy logs of `u2`, I see:

```
Dec 28 11:03:43 u2 microk8s.daemon-kubelite[4870]: E1228 11:03:43.676655    4870 server.go:1039] "Failed to retrieve node info" err="nodes \"10.0.0.79\" not found"
Dec 28 11:03:43 u2 microk8s.daemon-kubelite[4870]: I1228 11:03:43.676673    4870 server.go:1020] "Can't determine this node's IP, assuming loopback; if this is incorrect, please set the --bind-address flag"
Dec 28 11:03:43 u2 microk8s.daemon-kubelite[4870]: I1228 11:03:43.680519    4870 server.go:652] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
Dec 28 11:03:43 u2 microk8s.daemon-kubelite[4870]: I1228 11:03:43.680530    4870 server_others.go:168] "Using iptables Proxier"
Dec 28 11:03:43 u2 microk8s.daemon-kubelite[4870]: I1228 11:03:43.682141    4870 server_others.go:512] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR for family" ipFamily="IPv6"
Dec 28 11:03:43 u2 microk8s.daemon-kubelite[4870]: I1228 11:03:43.682149    4870 server_others.go:529] "Defaulting to no-op detect-local"
```

For some reason, it uses `10.0.0.79` as the node name, instead of `u2`. This is because we set `--hostname-override=$ip` in the kube-proxy args:

```
root@u2:~# grep hostname-override /var/snap/microk8s/current/args/kube-proxy
--hostname-override=10.0.0.79
```

Therefore, kube-proxy is looking for local endpoints, but cannot find any since it's looking for a different node (that does not exist).

The solution is to remove the `--hostname-override` line from the kube-proxy args, then restart microk8s with

```
sudo snap restart microk8s.daemon-kubelite
```

Then, kube-proxy uses the right nodename (`u2`). The logs now say:

```
Dec 28 11:03:03 u2 microk8s.daemon-kubelite[2939]: I1228 11:03:03.479105    2939 server.go:1050] "Successfully retrieved node IP(s)" IPs=["10.0.0.79"]
```

And I can curl the NodePort service from the worker node as well.

The bug seems to be in https://github.com/canonical/microk8s/blob/master/scripts/wrappers/join.py#L327-L328 and https://github.com/canonical/microk8s/blob/master/scripts/wrappers/join.py#L345-L346. We always set the --hostname-override variable, but we should only do it when the flag is already set, similar to what we do for kubelet's node-ip in https://github.com/canonical/microk8s/blob/master/scripts/wrappers/join.py#L467-L468 

Thanks again for raising this @dfry 
thanks @neoaggelos 
Currently, I am building my clusters thru an ansible playbook and my workaround is to only use master nodes.   What would be the approximate release cycle for this type of bug/fix as you have suggested above?  This is just an estimate for my internal planning purposes.

Cheers
If you are using ansible to automate this, a temporary workaround can be to add an extra task that removes the --hostname-override line from the worker node kube-proxy configs (in /var/snap/microk8s/current/args/kube-proxy) and restarts microk8s.

A proper bugfix can be expected to land on early January (pace is currently slow due to holiday break) on supported `1.2x/edge` tracks, and will find its way to stable with the next Kubernetes patchlevel releases.
thanks @neoaggelos 

Resolving patch:
diff --git a/scripts/wrappers/join.py b/scripts/wrappers/join.py
--- a/scripts/wrappers/join.py
+++ b/scripts/wrappers/join.py
@@ -312,7 +312,7 @@ def create_kubeconfig(token, ca, master_ip, api_port, filename, user):
         try_set_file_permissions(config)
 
 
-def update_kubeproxy(token, ca, master_ip, api_port, hostname_override):
+def update_kubeproxy(token, ca, master_ip, api_port):
     """
     Configure the kube-proxy
 
@@ -320,16 +320,14 @@ def update_kubeproxy(token, ca, master_ip, api_port, hostname_override):
     :param ca: the ca
     :param master_ip: the master node IP
     :param api_port: the API server port
-    :param hostname_override: the hostname override in case the hostname is not resolvable
     """
     create_kubeconfig(token, ca, master_ip, api_port, "proxy.config", "kubeproxy")
     set_arg("--master", None, "kube-proxy")
-    if hostname_override:
-        set_arg("--hostname-override", hostname_override, "kube-proxy")
+    set_arg("--hostname-override", None, "kube-proxy")
     service("restart", "proxy")
 
 
-def update_cert_auth_kubeproxy(token, master_ip, master_port, hostname_override):
+def update_cert_auth_kubeproxy(token, master_ip, master_port):
     """
     Configure the kube-proxy
 
@@ -337,13 +335,11 @@ def update_cert_auth_kubeproxy(token, master_ip, master_port, hostname_override)
     :param ca: the ca
     :param master_ip: the master node IP
     :param master_port: the master node port where the cluster agent listens
-    :param hostname_override: the hostname override in case the hostname is not resolvable
     """
     proxy_token = "{}-proxy".format(token)
     get_client_cert(master_ip, master_port, "proxy", proxy_token, "/CN=system:kube-proxy", False)
     set_arg("--master", None, "kube-proxy")
-    if hostname_override:
-        set_arg("--hostname-override", hostname_override, "kube-proxy")
+    set_arg("--hostname-override", None, "kube-proxy")
 
 
 def update_kubeproxy_cidr(cidr):
@@ -769,7 +765,7 @@ def join_dqlite_worker_node(info, master_ip, master_port, token):
     store_base_kubelet_args(info["kubelet_args"])
     update_kubelet_node_ip(info["kubelet_args"], hostname_override)
     update_kubelet_hostname_override(info["kubelet_args"])
-    update_cert_auth_kubeproxy(token, master_ip, master_port, hostname_override)
+    update_cert_auth_kubeproxy(token, master_ip, master_port)
     update_cert_auth_kubelet(token, master_ip, master_port)
     subprocess.check_call(
         [f"{snap()}/actions/common/utils.sh", "create_worker_kubeconfigs"],
@@ -891,12 +887,10 @@ def join_etcd(connection_parts, verify=True):
     update_flannel(info["etcd"], master_ip, master_port, token)
 
     if api_authn_mode == "Token":
-        update_kubeproxy(
-            info["kubeproxy"], info["ca"], master_ip, info["apiport"], hostname_override
-        )
+        update_kubeproxy(info["kubeproxy"], info["ca"], master_ip, info["apiport"])
         update_kubelet(info["kubelet"], info["ca"], master_ip, info["apiport"])
     elif api_authn_mode == "Cert":
-        update_cert_auth_kubeproxy(info["kubeproxy"], master_ip, master_port, hostname_override)
+        update_cert_auth_kubeproxy(info["kubeproxy"], master_ip, master_port)
         update_cert_auth_kubelet(info["kubelet"], master_ip, master_port)
         subprocess.check_call(
             [


Test patch: