Project: Antrea (Runtime / Cloud Native Network)
File: upgrade.md

The Antrea Multi-cluster feature is introduced from v1.5.0. There is no data-plane related changes from release v1.5.0, so Antrea deployment and Antrea Multi-cluster deployment are indenpendent. However, we suggest to keep Antrea and Antrea Multi-cluster in the same version considering there will be data-plane change involved in the future. Please refer to to learn the requirement of Antrea upgrade. This doc focuses on Multi-cluster deployment only. The goal is to support 'graceful' upgrade. Multi-cluster upgrade will not have disruption to data-plane of member clusters, but there can be downtime of processing new configurations when individual components restart: During Leader Controller restart, a new member cluster, ClusterSet or ResourceExport will not be processed. This is because the Controller also runs the validation webhooks for MemberClusterAnnounce, ClusterSet and ResourceExport. During Member Controller restart, a new ClusterSet will not be processed, this is because the Controller runs the validation webhooks for ClusterSet. Our goal is to support version skew for different Antrea Multi-cluster components, but the Multi-cluster feature is still in Alpha version, and the API is not stable yet. Our recommendation is always to upgrade Antrea Multi-cluster to the same version for a ClusterSet. Antrea Leader Controller: must be upgraded first Antrea Member Controller: must the same version as the Antrea Leader Controller. Antctl: must not be newer than the Antrea Leader/Member Controller. Please notice Antctl for Multi-cluster is added since v1.6.0. In one ClusterSet, We recommend all member and leader clusters deployed with the same version. During Leader controller upgrade, resource export/import between member clusters is not supported. Before all member clusters are upgraded to the same version as Leader controller, the feature introduced in old version should still work cross clusters, but no guarantee for the feature in new version. It should have no impact during upgrade to those imported resources like Service, Endpoints or AntreaClusterNetworkPolicy. Prior to Antrea v1.13, the `ClusterClaim` CRD is used to define both the local Cluster ID and the ClusterSet ID. Since Antrea v1.13, the `ClusterClaim` CRD is removed, and the `ClusterSet` CRD solely defines a ClusterSet. The name of a `ClusterSet` CR must match the ClusterSet ID, and a new `clusterID` field specifies the local Cluster ID. After upgrading Antrea Multi-cluster Controller from a version older than v1.13, the new version Multi-cluster Controller can still recognize and work with the old version `ClusterClaim` and `ClusterSet`

However, we still suggest updating the `ClusterSet` CR to the new version after upgrading Multi-cluster Controller. You just need to update the existing `ClusterSet` CR and add the right `clusterID` to the spec. An example `ClusterSet` CR is like the following: ```yaml apiVersion: multicluster.crd.antrea.io/v1alpha2 kind: ClusterSet metadata: name: test-clusterset # This value must match the ClusterSet ID. namespace: kube-system spec: clusterID: test-cluster-north # The new added field since v1.13. leaders: clusterID: test-cluster-north secret: "member-north-token" server: "https://172.18.0.1:6443" namespace: antrea-multicluster ``` You may also delete the `ClusterClaim` CRD after the upgrade, and then all existing `ClusterClaim` CRs will be removed automatically after the CRD is deleted. ```bash kubectl delete crds clusterclaims.multicluster.crd.antrea.io ``` The Antrea Multi-cluster APIs are built using K8s CustomResourceDefinitions and we follow the same versioning scheme as the K8s APIs and the same . Other than the most recent API versions in each track, older API versions must be supported after their announced deprecation for a duration of no less than: GA: 12 months Beta: 9 months Alpha: N/A (can be removed immediately) K8s has a on the removal ofAPI object versions that have been persisted to storage. We adopt the following rules for the CustomResources which are persisted by the K8s apiserver. Alpha API versions may be removed at any time. The must be used for CRDs to indicate that a particular version of the resource has been deprecated. Beta and GA API versions must be supported after deprecation for the respective durations stipulated above before they can be removed. For deprecated Beta and GA API versions, a must be provided along with each Antrea release, until the API version is removed altogether. Please refer to to learn the details. Following is the Antrea Multi-cluster feature list. For the details of each feature, please refer to . | Feature | Supported in | | -- | | | Service Export/Import | v1.5.0 | | ClusterNetworkPolicy Replication | v1.6.0 | When you are trying to directly apply a newer Antrea Multi-cluster YAML manifest, as provided with , you will probably meet an issue like below if you are upgrading Multi-cluster components from v1.5.0 to a newer one: ```log label issue:The Deployment "antrea-mc-controller" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"antrea", "component":"antrea-mc-controller"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable ``` The issue is caused by the label change introduced by . The reason is mutation of label selectors on Deployments is not allowed in `apps/v1beta2` and forward. You need to delete the Deployment "antrea-mc-controller" first, then run `kubectl apply -f` with the manifest of the newer version.