Project: Gloo (Orchestration & Management / API Gateway)
File: canary.md

title: Canary upgrade weight: 30 description: Use a canary upgrade model to upgrade Gloo Edge, such as in production environments. Use a canary upgrade model to upgrade Gloo Edge, such as in production environments. In the canary model, you run two deployments of `gloo` in your cluster: the existing deployment that runs your current version, and a canary deployment that runs the version you want to upgrade to. After you create the canary deployment, you verify that it handles traffic as you expect before upgrading to the new version. This approach helps you reduce potential downtime for production upgrades. Before you begin, follow the guide to complete these preparatory steps: Review important changes made to Gloo Edge in version {{< readfile file="static/content/versiongeosslatest_minor.md" markdown="true">}}, including CRD, Helm, CLI, and feature changes. Upgrade your current version to the latest patch. Upgrade any dependencies to the required supported versions. Consider other steps to prepare for upgrading. Review frequently-asked questions about the upgrade process. Follow the steps in to update `glooctl` to the version you want to upgrade to. Now you're ready to upgrade. The steps vary depending on your Gloo Edge installation. Set the version to upgrade Gloo Edge to in an environment variable, such as the latest patch version for open source (`{{< readfile file="static/content/versiongeosslatest.md" markdown="true">}}`) or enterprise (`{{< readfile file="static/content/versiongeelatest.md" markdown="true">}}`). ```sh export CANARY_VERSION=<version> ``` Update and pull the Gloo Edge Helm chart for the canary version. {{< tabs >}} {{< tab name="Open Source" >}} ```sh helm repo add gloo https://storage.googleapis.com/solo-public-helm helm repo update helm pull gloo/gloo --version $CANARY_VERSION --untar ``` {{< /tab >}} {{< tab name="Enterprise">}} ```sh helm repo add glooe https://storage.googleapis.com/gloo-ee-helm helm repo update helm pull glooe/gloo-ee --version $CANARY_VERSION --untar ``` {{< /tab >}} {{< /tabs >}} Apply the CRDs for the canary version to your cluster. The Gloo Edge CRDs are designed to be backward compatible, so the new CRDs should not impact the performance of your existing installation. For more information, see the list of for {{< readfile file="static/content/versiongeosslatest_minor.md" markdown="true">}}. {{< tabs >}} {{< tab name="Open Source" >}} ```sh kubectl apply -f gloo/crds ``` {{< /tab >}} {{< tab name="Enterprise">}} ```sh kubectl apply -f gloo-ee/charts/gloo/crds ``` {{< /tab >}} {{< /tabs >}} Install the canary version of Gloo Edge in your cluster in a new namespace, `gloo-system-$CANARY_VERSION`. {{< tabs >}} {{< tab name="Open Source" >}} ```sh glooctl install gateway \ --create-namespace \ -n gloo-system-$CANARY_VERSION \ --version $CANARY_VERSION ``` {{< /tab >}} {{< tab name="Enterprise">}} Note that you must set your license key by using the `--license-key $LICENSEKEY` flag or including the `licensekey: $LICENSE_KEY` setting in your values file. If you do not have a license key, . ```sh glooctl install gateway enterprise \ --create-namespace \ -n gloo-system-$CANARY_VERSION \ --version $CANARY_VERSION \ --license-key $LICENSE_KEY ``` {{< /tab >}} {{< /tabs >}} Verify that your existing and canary versions of Gloo Edge are running. ```shell kubectl get all -n gloo-system kubectl get all -n gloo-system-$CANARY_VERSION ``` Check the to modify any custom resources for any changes or new capabilities in {{< readfile file="static/content/versiongeosslatest_minor.md"

applicable, add steps to walk users though updating crs for any breaking changes--> Test your routes and monitor the metrics of the canary version. ```shell glooctl check ``` the previous version of Gloo Edge so that your cluster uses the new version going forward. ```shell glooctl uninstall -n gloo-system ``` In the canary upgrade model for Gloo Edge Federation, you start with an existing version of Gloo Edge Federation in your management cluster that manages one or more remote Gloo Edge instances. To test a new version, you install the canary version of Gloo Edge Federation in a new namespace on the management cluster. Then, you install one or more Gloo Edge instances that run the canary version in remote test clusters, register the test clusters, and check your setup. Finally, you deregister the remote test clusters from the previous version of Gloo Edge Federation, and uninstall the previous Gloo Edge Federation version from the management cluster. Set your Gloo Edge license key in an environment variable. If you do not have a license key, . ```sh export LICENSE_KEY=<version> ``` Set the version to upgrade Gloo Edge to in an environment variable, such as the latest patch version (`{{< readfile file="static/content/versiongeelatest.md" markdown="true">}}`). ```sh export CANARY_VERSION=<version> ``` Update and pull the Gloo Edge Federation Helm chart for the canary version. ```shell helm repo add gloo-fed https://storage.googleapis.com/gloo-fed-helm helm repo update helm pull gloo-fed/gloo-fed --version $CANARY_VERSION --untar ``` Apply the CRDs for the canary version to your management cluster. The Gloo Edge CRDs are designed to be backward compatible, so the new CRDs should not impact the performance of your existing installation. However, if after evaluating the canary installation you decide to continue to use the existing installation, you can easily remove any added CRDs by referring to the listed changes to CRD names and running `kubectl delete crd <CRD>`. Then, to re-apply previous versions of CRDs, you can run `helm pull gloo/gloo --version <previous_version> --untar` and `kubectl apply -f gloo/crds`. ``` kubectl apply -f gloo-fed/crds ``` Install the canary version of Gloo Edge Federation in a new namespace, `gloo-system-$CANARY_VERSION`, in your management cluster. ``` helm install gloo-fed gloo-fed/gloo-fed \ --create-namespace \ -n gloo-system-$CANARY_VERSION \ --version $CANARY_VERSION \ --set-string licensekey=$LICENSEKEY ``` Verify that your existing and canary versions of Gloo Edge Federation are running. ```shell kubectl get all -n gloo-system kubectl get all -n gloo-system-$CANARY_VERSION ``` on one of your remote clusters. Make sure that the version of Gloo Edge Enterprise matches the canary version of Gloo Edge Federation that you installed. Check the for any changes or new capabilities in {{< readfile file="static/content/versiongeosslatest_minor.md" markdown="true">}}. In your management cluster, with these changes to test your canary version of Gloo Edge Federation. Because Gloo Edge Federation scans all federated resources on the management cluster, you might want to reuse the existing federated resources in the `gloo-system` namespace. <!--If applicable, add steps to walk users though updating crs for any breaking changes--> {{% expand "Expand for an example federated virtual service configuration" %}} This example configuration creates a federated virtual service with different features across versions 1.12 and

POSSIBLE UPDATE for something that is new in 1.16 instead--> Creates a shared `gloo-fed` namespace to show that both the existing and canary Gloo Edge Federation instances can watch for changes to a resource in the same namespace. Places the federated virtual service resource in both the `remote1` existing version cluster (1.12) and the `remote2` canary version cluster (1.13). Configures settings such as `numRetries: 10` that are shared across versions. Adds the `retryBackOff` that is available in version 1.13 but not in 1.12. As such, the 1.12 federated resource ignores this new configuration. ```yaml kubectl create ns gloo-fed kubectl apply -f - <<EOF apiVersion: fed.gateway.solo.io/v1 kind: FederatedVirtualService metadata: name: my-fed-vs namespace: gloo-fed spec: placement: clusters: remote1 remote2 namespaces: gloo-system template: metadata: name: my-vs spec: virtualHost: domains: '*' routes: matchers: prefix: / routeAction: single: upstream: name: default-petstore-8080 namespace: gloo-system options: retries: retryOn: 'connect-failure' numRetries: 10 perTryTimeout: '5s' retryBackOff: baseInterval: 1s maxInterval: 3s EOF ``` {{% /expand %}} Check that the updated, federated custom resources from the management cluster are propagated to the remote test cluster. In the management cluster, verify that the federated resources are propagated to the remote clusters by reviewing the federated resource configuration. This example gets the configuration for the federated virtual service from the previous step. ```sh kubectl get federatedvirtualservice -n gloo-fed -o yaml ``` In this example output, the `PLACED` state in the status section confirms that the resource is federated: {{< highlight yaml "hl_lines=10 16" >}} ... status: namespacedPlacementStatuses: gloo-system-$CANARY_VERSION: clusters: remote2: namespaces: gloo-system: state: PLACED gloo-system: clusters: remote1: namespaces: gloo-system: state: PLACED {{< /highlight >}} In the remote clusters, confirm that the federated resources are created. The following commands are based on the previous step's example.<!--If step 8 was updated for 1.16, update this step as well for the 1.16 feature--> In the existing version's remote cluster, verify that the resource is federated with only the existing version's configuration. In the example, the existing 1.12 version has an updated `numRetries: 10`, but no `retryBackOff` section, which is available only in version 1.13 or later. {{< highlight yaml "hl_lines=5" >}} kubectl get virtualservices -n gloo-system --context remote1 -o yaml ... options: retries: numRetries: 10 perTryTimeout: 5s retryOn: connect-failure {{< /highlight >}} In the canary version's remote cluster, verify that the resource is federated with only the canary version's configuration. In the example, the canary 1.13 version has an updated `numRetries: 10`, as well as a `retryBackOff` section. {{< highlight yaml "hl_lines=6 9-11" >}} kubectl get virtualservices -n gloo-system --context remote2 -o yaml ... options: retries: numRetries: 10 perTryTimeout: 5s retryOn: connect-failure retryBackOff: baseInterval: 1s maxInterval: 3s {{< /highlight >}} Repeat the previous steps for each federated resource that you want to check. Test your routes and monitor the metrics of the canary version in your remote test cluster. ```shell glooctl check ``` Shift traffic to the canary version of Gloo Edge Federation by uninstalling the previous version. your other remote clusters that still use the previous version of Gloo Edge

the previous version of Gloo Edge Federation so that your management cluster uses only the new version. ```shell glooctl uninstall -n gloo-system ``` Optionally delete the previous version namespace from each remote cluster. The deregister command does not clean up the namespace and custom resources in the previous version. ```shell kubectl delete ns gloo-system ``` As you test your environment with the canary version of Gloo Edge, you might need to roll back to the previous version. You can follow a canary model for the rollback. Set the previous version that you want to roll back to as an environment variable. ```shell export ROLLBACK_VERSION=<version> ``` If you already removed the previous version of Gloo Edge from your cluster, re-install Gloo Edge at the rollback version. Update and pull the Gloo Edge Helm chart for the rollback version. The Helm charts vary depending on the Gloo Edge installation option. {{< tabs >}} {{< tab name="Open Source" >}} ```sh helm repo update helm pull gloo/gloo --version $ROLLBACK_VERSION --untar ``` {{< /tab >}} {{< tab name="Enterprise">}} ```sh helm repo update helm pull glooe/gloo-ee --version $ROLLBACK_VERSION --untar ``` {{< /tab >}} {{< tab name="Federation">}} ```sh helm repo update helm pull gloo-fed/gloo-fed --version $ROLLBACK_VERSION --untar ``` {{< /tab >}} {{< /tabs >}} Install the rollback version of Gloo Edge in a new namespace in your cluster. {{< tabs >}} {{< tab name="Open Source" >}} ```sh glooctl install gateway \ --create-namespace \ -n gloo-system-$ROLLBACK_VERSION \ --version $ROLLBACK_VERSION ``` {{< /tab >}} {{< tab name="Enterprise">}} ```sh glooctl install gateway enterprise \ --create-namespace \ -n gloo-system-$ROLLBACK_VERSION \ --version $ROLLBACK_VERSION \ --license-key $LICENSE_KEY ``` {{< /tab >}} {{< tab name="Federation">}} ```sh helm install gloo-fed gloo-fed/gloo-fed \ --create-namespace \ -n gloo-system-$ROLLBACK_VERSION \ --version $ROLLBACK_VERSION \ --set-string licensekey=$LICENSEKEY ``` {{< /tab >}} {{< /tabs >}} Revert any changes to custom resources that you made during the upgrade. For differences between versions, check the and the . Gloo Edge Federation: Shift traffic from the canary version to the rollback version of Gloo Edge Federation. your test remote clusters that use the canary version of Gloo Edge Federation. using the rollback version on each remote cluster. Optionally delete the canary version namespace. The deregister command does not clean up the namespace and custom resources in the previous version. ```shell kubectl delete ns gloo-system-$CANARY_VERSION ``` the canary version of Gloo Edge from your cluster. ```shell glooctl uninstall -n gloo-system-$CANARY_VERSION ``` Remove any new CRDs that were added in the canary version that are not present in the rollback version. Check the list of in {{< readfile file="static/content/versiongeosslatest_minor.md" markdown="true">}} for the names of any newly added CRDs. Delete any added CRDs. ``` kubectl delete crd <CRD> ``` Apply the Gloo CRDs from the rollback version Helm chart to your cluster. In Gloo Edge Federation, this cluster is the local management cluster. {{< tabs >}} {{< tab name="Open Source" >}} ```sh kubectl apply -f gloo/crds ``` {{< /tab >}} {{< tab name="Enterprise">}} ```sh kubectl apply -f gloo-ee/charts/gloo/crds ``` {{< /tab >}} {{< tab name="Federation">}} ```sh kubectl apply -f gloo-fed/crds ``` {{< /tab >}} {{< /tabs >}} Test your routes and monitor the metrics of the rollback version. ```shell glooctl check ```