# Vendor-Agnostic Reconfiguration of Kubernetes Clusters in Cloud Federations
## 5 Reconfiguration Tactics
### CNI Network Plugin  
Using the CNI networking plugin, we can bring more sophisticated networking capabilities to our cluster, such as network policies to customize connectivity of pods. However, switching to CNI plugins in clusters of a hosted product type depends on the vendors' proprietary interfaces. At the time of research, Calico was the leading CNI plugin to support network policies and was also employed by all three studied vendors. Therefore,  
Figure 3: Redhat-COP PodPresets admission webhook server. this section takes GKE and the Calico network plugin as the example to introduce our vendor-agnostic reconfiguration tactics, of which Figure 4 illustrates the general steps. Firstly, similar to the tactics for reconfiguring the Kubelet, a privileged Daemonset reconfigures Kubelet to run in the CNI mode by overriding the Kubelet configuration file. Kubelet discovers and executes CNI Plugins from the /opt/cni/bin directory by default. However, in GKE's compute instances, we do not have access to this folder, so we point cni-bin-dir field in the Kubelet configuration file to the location of our Calico installation. Directly installing the default Calico network plugin in a GKE cluster could make the control plane-to-pod communication unavailable. It is because our privileged Daemonset and the Calico network plugin cannot be deployed on the control plane. During installation, the plugin will set up a tunnel device on each node to enable an overlay pod network supporting the pod communication across nodes. Such tunnel devices encapsulate a pod IP packet with another IP packet at the level of the cluster VPC network. However, when we directly install the plugin in a GKE cluster, the tunnel devices do not exist on the vendor-managed control plane. In addition, since the Calico network plugin uses the Calico IP Address Management (IPAM) engine to allocate IP addresses to pods dynamically from its managed IP pool, the cluster VPC network will not have routing information for these addresses. For example, Figure 5 illustrates a situation where the API server can not communicate with a webhook server, when the default Calico plugins are installed. The webhook server pod IP address 10.0.0.2 assigned by Calico is not in the pod IP range 10.124.0.1/24 of its node. Moreover, as Calico cannot be installed on the GKE control plane, the packet from the API server is not encapsulated with node-level IP packets. Therefore, the cluster VPC network cannot route the packet with the destination IP address 10.0.0.2 properly. However, due to tunnels' encapsulation, pod-to-pod communication across worker nodes is still available. In summary, as the packets from the control plane are not encapsulated, and the network does not have routing information for Calico-managed IP pools, the control plane-to-pod communication is lost. A possible consequence of such disconnection is that the API server cannot send requests to deployed admission webhook servers (ex. the PodPresets webhook server, as discussed in Section 5.1) and thus lost the extensibility of admission logic. To solve this disconnection problem, before deploying the Calico network plugin, we have to modify its manifest, configuring it to run in nonoverlay mode and use the host-local IPAM. The Calico network plugin running in the nonoverlay mode will not encapsulate IP packets from containers and directly send them to the underlying network stack and then to the cloud network infrastructure. The host-local IPAM assigns IP addresses to pods from the IP range configured in the cluster VPC network. The control plane assigns such IP range to nodes when they join the cluster. Therefore, as the network of GKE is VPC-native by default [48], it will still be aware of pod IP addresses and able to forward traffic from the control plane to pods, as illustrated in Figure 6.  
Figure 4: Steps for reconfiguring the Calico CNI plugin in GKE. Although the reconfiguration tactic is developed for GKE and the Calico network plugin, it is easily adapted to other vendors and network plugins.