Repository: canonical/microk8s
Pull request: Clean up Calico virtual interfaces on uninstall (#2903)
#### Summary

This PR cleans up Calico interfaces that are left after uninstalling microk8s as described in #2903.

The first part is deleting the vxlan interface that is used for nodes - the change here is actually fixing a typo, the deletion was already intended and present in the code, however the interface name was not set properly.

The second part is ensuring that only interfaces part of the cluster are being deleted, so Calico interfaces that are tied to pods. For that all the interfaces are computed the same way Calico computes them (referred to as WorkloadEndpoints) and dumped on service stop, so that they can be picked up by the remove hook.

   Closes #2903 

#### Changes

Changes are done in the remove hook for the CNI, there is a helper script that is invoked when stopping `kubelite` (to dump WEPs on service stop) and a `stop-command` added to the `kubelite` daemon in *snapcraft.yaml* to invoke the helper script. I've also added the same functionality when invoking `microk8s stop`, although it was part of experimentation and I can leave it out in the final version of the PR based on feedback received.

#### Testing

```bash
merkata@mastodon:~$ ip a| grep cali
22: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    inet 10.1.139.192/32 scope global vxlan.calico
23: caliafcdeb2d69b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
24: cali99c376db89a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
25: cali9fe2e03e266@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
merkata@mastodon:~$ sudo snap remove --purge microk8s
[sudo] password for merkata: 
microk8s removed
merkata@mastodon:~$ ip a| grep cali
merkata@mastodon:~$ 
```

#### Possible Regressions

I haven't checked for possible regressions, though I don't think that in this case there should be any.

#### Checklist

* [X ] Read the [contributions](https://github.com/canonical/microk8s/blob/master/CONTRIBUTING.md) page.
* [X ] Submitted the [CLA form](https://ubuntu.com/legal/contributors/agreement), if you are a first time contributor.
* [ ] The introduced changes are covered by unit and/or integration tests.

#### Notes

If needed, I can try and add a test in CI for this case.

Problem statement:
Calico virtual network interfaces not cleaned up on uninstall
Uninstalling the microk8s snap does not clean up any network interfaces created by Calico. Occured after uninstalling `v1.23.3` using `snap`.

```
~
$ ip a
...
66: cali2b0ba0cec00@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-b3f14500-8dda-de2f-233b-230670bd6549
69: cali8653ab46e7e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c8103f93-8e92-a74f-8349-4a6a8d8739c6
70: cali013b9543920@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-d92cae3d-14e9-52fe-714f-8eeb6e616a93
71: cali454300f8321@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-440c924a-53f2-db2a-6119-878235dcefc2
73: cali47f89a3fd26@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-dde0ef58-9a14-13f0-750b-c1ad956eb986
74: calidece2355e7d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-35e00d0b-5b63-368b-bdcb-c944812a5c3b
75: calie54eabdbf18@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-d82db011-72c8-8568-d2e9-477f9126e7c7
76: calia914aed7549@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-f86cddaf-c570-2eb1-59a1-24f908cc8ff4
191: mpqemubr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:f5:38:89 brd ff:ff:ff:ff:ff:ff
    inet 10.16.236.1/24 brd 10.16.236.255 scope global mpqemubr0
       valid_lft forever preferred_lft forever
192: tap-6dcbc7c2e1d: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master mpqemubr0 state DOWN group default qlen 1000
    link/ether da:53:d4:89:dc:12 brd ff:ff:ff:ff:ff:ff
195: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default 
    link/ether 66:ca:5c:9c:07:32 brd ff:ff:ff:ff:ff:ff
    inet 10.1.14.0/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
197: tap-de1c947ae73: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master mpqemubr0 state DOWN group default qlen 1000
    link/ether 72:bb:a5:26:05:88 brd ff:ff:ff:ff:ff:ff
198: calid192816f978@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c1b71b54-1575-3847-3e3a-b20294af24d5
199: cali9e1d6b0e75c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-149861b0-a057-236f-266b-bfd9a3bc25cb
200: cali1b305feaa92@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c22c1576-7c05-7e73-dd20-cb64261a7f31
203: calif5c17f87f23@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-509a5393-fb11-49d1-246e-a58dadc1f3ca
204: tap-cf7589c2edb: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master mpqemubr0 state DOWN group default qlen 1000
    link/ether 0e:c3:30:c4:a1:a2 brd ff:ff:ff:ff:ff:ff
210: calibe999c73374@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-718acdac-30d3-f02c-7469-8eec9515d7b0
211: cali12d4a061371@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-7605a0e0-301f-b3b9-7cd9-6efbe52dc07d
212: tap-227b550eb5b: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master mpqemubr0 state DOWN group default qlen 1000
    link/ether ba:2e:72:3b:78:54 brd ff:ff:ff:ff:ff:ff
```


```
~ 
$ sudo snap remove microk8s
snap "microk8s" is not installed
```

Hints from issue discussion:
Thank you for reporting this @simskij 

We will probably need to take care these interfaces in the remove hook [1]. Special care should be taken when MicroK8s runs on lxc so we do not destroy interfaces used by other containers.

[1] https://github.com/ubuntu/microk8s/blob/master/snap/hooks/remove
Hey, i would like to work on this issue, so before i start working on it, would you mind sparing your time explaining what the issue is about? as i'm new to this field. 
Hi @gurmatsinghsour, thank you for volunteering.

When pods are created certain (cali and tap) interfaces are created. The issue here is that we do not clean these interfaces when we remove the snap.

I think you should start my building the snap [1] and then work on removing the extra interfaces in the script called when the snap is removed [2].

[1] https://github.com/canonical/microk8s/blob/master/docs/build.md#building-the-snap-from-source
[2] https://github.com/canonical/microk8s/blob/master/docs/build.md#building-the-snap-from-source

Commit message: fix(calico): Remove unsupported flag

Patch to review:
diff --git a/microk8s-resources/default-hooks/remove.d/10-cni-link b/microk8s-resources/default-hooks/remove.d/10-cni-link
index 3c80d81d15..3e4be2df28 100755
--- a/microk8s-resources/default-hooks/remove.d/10-cni-link
+++ b/microk8s-resources/default-hooks/remove.d/10-cni-link
@@ -4,11 +4,15 @@
 
 if ! is_strict || (is_strict && snapctl is-connected network-control)
 then
-  for link in cni0 calico.vxlan
+  for link in cni0
   do
     if "${SNAP}/sbin/ip" link show "${link}"
     then
       "${SNAP}/sbin/ip" link delete "${link}" || true
     fi
   done
+  for calink in $("${SNAP}/sbin/ip" -j link show |\
+                  "${SNAP}/usr/bin/jq" -r '.[].ifname | select(test("cali[a-f0-9]"))')
+  do "${SNAP}/sbin/ip" link delete "${calink}"
+  done
 fi
diff --git a/tests/test-cluster.py b/tests/test-cluster.py
index ce8b68a921..0c0571e0a8 100644
--- a/tests/test-cluster.py
+++ b/tests/test-cluster.py
@@ -278,6 +278,25 @@ def test_calico_in_nodes(self):
                 assert False
             print("Calico found in node {}".format(vm.vm_name))
 
+    def test_calico_interfaces_removed_on_snap_remove(self):
+        """
+        Test that calico interfaces are not present on the node
+        when the microk8s snap is removed.
+        """
+        vm = VM(backend)
+        vm.setup(channel_to_test)
+        print("Waiting for machine {}".format(vm.vm_name))
+        vm.run("/snap/bin/microk8s.status --wait-ready --timeout 240")
+        timeout = time.time() + 240
+        while time.time() <= timeout:
+            pods = vm.run("/snap/bin/microk8s.kubectl get po -n kube-system -o wide")
+            for line in pods.decode().splitlines():
+                if "calico" in line and "Running" in line:
+                    break
+            time.sleep(5)
+        vm.run("snap remove --purge microk8s")
+        assert vm.run("/usr/bin/ip a | grep cali") == ""
+
     def test_nodes_in_ha(self):
         """
         Test all nodes are seeing the database while removing nodes


Code review comments:
diff_hunk: @@ -11,8 +11,8 @@ then
       "${SNAP}/sbin/ip" link delete "${link}" || true
     fi
   done
-  for calink in $("${SNAP}/sbin/ip" -j --pretty link show |\
-                 awk '/cali/ {print $2}' | tr -d '",')
+  for calink in $("${SNAP}/sbin/ip" -j link show |\
+                  "${SNAP}/usr/bin/jq" -r '.[].ifname | select(test("cali[a-f0-9]"))')
path: microk8s-resources/default-hooks/remove.d/10-cni-link
text: @user1:
One final fix for the regex.
```suggestion
                  "${SNAP}/usr/bin/jq" -r '.[].ifname | select(test("^cali[a-f0-9]*$"))')
```

@author:
Nice! I've changed it, note I'm using the regex you suggested above, so that `vxlan.calico` is also picked up:

```shell
merkata@user2:~/dev/charm-k8s-postgresql$ ip -j link show| /snap/microk8s/current/usr/bin/jq -r '.[].ifname | select(test("^cali[a-f0-9]*$"))'
cali95ad96c92c3
merkata@user2:~/dev/charm-k8s-postgresql$ ip -j link show| /snap/microk8s/current/usr/bin/jq -r '.[].ifname | select(test("^vxlan.calico|cali[a-f0-9]*$"))'
vxlan.calico
cali95ad96c92c3
```