# Vendor-Agnostic Reconfiguration of Kubernetes Clusters in Cloud Federations
###### Abstract
The latter is because K8s vendors may hide configuration manifests that are defined as part of the open-source K8s distribution on GitHub. Such hiding either involves locking configuration parameters and plug-in components in a particular setting or encapsulating the configuration manifests behind a proprietary customization API. Such proprietary API only offers reconfiguration of some configuration settings and increases the complexity of vendor lock-in avoidance. In this paper, we present a unified and vendor-agnostic cluster reconfiguration approach, so that cluster administrators and application managers can avoid vendor lock-in limitations when reconfiguring clusters and can also change a broader set of configuration settings than offered in the proprietary customization interfaces. In particular, we make the following contributions:  
* We propose a feature-oriented approach where K8s clusters can be reconfigured by means of declarative configuration of desired optional features. An optional feature corresponds here to a specific functionality of Kubernetes that is not enabled by default, yet it is clearly described in the open-source documentation of Kubernetes with precise instructions how to enable it. Based on a previous case study of feature incompatibilities between three leading K8s vendors--Azure Kubernetes Service (AKS), Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE)--we have identified more than 30 optional features that were stable or highly demanded but that were locked by at least two vendors in different enabled or disabled states, leading to feature incompatibilities that violate at least one of the aforementioned consistency requirements;
* We account all feature incompatibilities to three configuration manifests of the open-source K8s distribution that are partially or completely hidden by proprietary customization interfaces of the three vendors;
* We describe in detail what are the most prevailing vendor-agnostic reconfiguration tactics in industry for changing a broad set of configuration settings. We point out that these tactics are all based on imperative configuration management that suggests one-off installation without further monitoring. This is not in line with the Kubernetes philosophy of declarative configuration management where a separate control loop continuously monitors for differences between desired and actual system configuration states;
* We extend KubeFed, a popular tool for federation of K8s clusters, with an API and autonomic controller for declarative feature compatibility management. As such, cluster administrators and application managers can submit feature configuration manifests to this API to specify what desired features all member clusters in a federation should have. The controller detects missing features in the member clusters. If a cluster does not support one or more desired features, the controller will apply the aforementioned imperative reconfigurations tactics to install them. It will further monitor the member cluster and generate events to report successful or pending installation of the desired features;
* We make an empirical evaluation of the controller and the three vendor-agnostic reconfiguration tactics with respect to (i) the impact of the reconfigured K8s features on the performance of applications, (ii) the disruption of running applications during the reconfiguration process and (iii) the total reconfiguration time. The remainder of the paper is organized as follows. In Section 2, we introduce the technical background, mainly focused on Kubernetes and cluster federations. Subsequently,in Section 3, we investigate related work for managing vendor lock-in avoidance in both cloud computing and container-based clusters. Then, in Section 4, we make an analysis of the configuration manifests that cause the overall feature incompatibility problem. The vendor-agnostic reconfiguration tactics for these problematic configuration manifests are elaborated in Section 5.