# Deploying MongoDB Across Multiple Kubernetes Clusters With MongoDBMulti

This article is part of a three-parts series on deploying MongoDB across multiple Kubernetes clusters using the operators.

-   Deploying the MongoDB Enterprise Kubernetes Operator on Google Cloud

-   Mastering MongoDB Ops Manager

-   Deploying MongoDB Across Multiple Kubernetes Clusters With MongoDBMulti

With the latest version of the MongoDB Enterprise Kubernetes Operator, you can deploy MongoDB resources across multiple Kubernetes clusters! By running your MongoDB replica set across different clusters, you can ensure that your deployment remains available even in the event of a failure or outage in one of them. The MongoDB Enterprise Kubernetes Operator's Custom Resource Definition (CRD), MongoDBMulti, makes it easy to run MongoDB replica sets across different Kubernetes environments and provides a declarative approach to deploying MongoDB, allowing you to specify the desired state of your deployment and letting the operator handle the details of achieving that state.

> ⚠️ Support for multi-Kubernetes-cluster deployments of MongoDB is a preview feature and not yet ready for Production use. The content of this article is meant to provide you with a way to experiment with this upcoming feature, but should not be used in production as breaking changes may still occur. Support for this feature during preview is direct with the engineering team and on a best-efforts basis, so please let us know if trying this out at kubernetes-product@mongodb.com. Also feel free to get in touch with any questions, or if this is something that may be of interest once fully released.

## Overview of MongoDBMulti CRD

Developed by MongoDB, MongoDBMulti Custom Resource allows for the customization of resilience levels based on the needs of the enterprise application.

-   Single region (Multi A-Z) consists of one or more Kubernetes clusters where each cluster has nodes deployed in different availability zones in the same region. This type of deployment protects MongoDB instances backing your enterprise applications against zone and Kubernetes cluster failures.

-   Multi Region consists of one or more Kubernetes clusters where you deploy each cluster in a different region, and within each region, deploy cluster nodes in different availability zones. This gives your database resilience against the loss of a Kubernetes cluster, a zone, or an entire cloud region.

By leveraging the native capabilities of Kubernetes, the MongoDB Enterprise Kubernetes Operator performs the following tasks to deploy and operate a multi-cluster MongoDB replica set:

-   Creates the necessary resources, such as Configmaps, secrets, service objects, and StatefulSet objects, in each member cluster. These resources are in line with the number of replica set members in the MongoDB cluster, ensuring that the cluster is properly configured and able to function.

-   Identifies the clusters where the MongoDB replica set should be deployed using the corresponding MongoDBMulti Custom Resource spec. It then deploys the replica set on the identified clusters.

-   Watches for the creation of the MongoDBMulti Custom Resource spec in the central cluster.

-   Uses a mounted kubeconfig file to communicate with member clusters. This allows the operator to access the necessary information and resources on the member clusters in order to properly manage and configure the MongoDB cluster.

-   Watches for events related to the CentralCluster and MemberCluster in order to confirm that the multi-Kubernetes-cluster deployment is in the desired state.

You should start by constructing a central cluster. This central cluster will host the Kubernetes Operator, MongoDBMulti Custom Resource spec, and act as the control plane for the multi-cluster deployment. If you deploy Ops Manager with the Kubernetes Operator, the central cluster may also host Ops Manager.

You will also need a service mesh. I will be using Istio, but any service mesh that provides a fully qualified domain name resolution between pods across clusters should work.

Communication between replica set members happens via the service mesh, which means that your MongoDB replica set doesn't need the central cluster to function. Keep in mind that if the central cluster goes down, you won't be able to use the Kubernetes Operator to modify your deployment until you regain access to this cluster.  

## Using the MongoDBMulti CRD

Alright, let's get started using the operator and build something! For this tutorial, we will need the following tools: 

-   gcloud 

-   gke-cloud-auth-plugin

-   Go  v1.17 or later

-   Helm

-   kubectl

-   kubectx

-   Git.

We need to set up a master Kubernetes cluster to host the MongoDB Enterprise Multi-Cluster Kubernetes Operator and the Ops Manager. You will need to create a GKE Kubernetes cluster by following the instructions in Part 1 of this series. Then, we should install the MongoDB Multi-Cluster Kubernetes Operator  in the `mongodb` namespace, along with the necessary CRDs. This will allow us to utilize the operator to effectively manage and operate our MongoDB multi cluster replica set. For instructions on how to do this, please refer to the relevant section of Part 1. Additionally, we will need to install the Ops Manager, as outlined in Part 2 of this series.

### Creating the clusters

After master cluster creation and configuration, we need three additional GKE clusters, distributed across three different regions: `us-west2`, `us-central1`, and `us-east1`. Those clusters will host MongoDB replica set members. 

```bash
CLUSTER_NAMES=(mdb-cluster-1 mdb-cluster-2 mdb-cluster-3)
ZONES=(us-west2-a us-central1-a us-east1-b)

for ((i=0; i<${#CLUSTER_NAMES@]:0:1}; i++)); do
  gcloud container clusters create "${CLUSTER_NAMES[$i]}" \
    --zone "${ZONES[$i]}" \
    --machine-type n2-standard-2 --cluster-version="${K8S_VERSION}" \
    --disk-type=pd-standard --num-nodes 1
done
```

The clusters have been created, and we need to obtain the credentials for them.

```bash
for ((i=0; i<${#CLUSTER_NAMES[@]:0:1}; i++)); do
  gcloud container clusters get-credentials "${CLUSTER_NAMES[$i]}" \
    --zone "${ZONES[$i]}"
done
```

After successfully creating the Kubernetes master and MongoDB replica set clusters, installing the Ops Manager and all required software on it, we can check them using `[kubectx`.

```bash
kubectx
```

You should see all your Kubernetes clusters listed here. Make sure that you only have the clusters you just created and remove any other unnecessary clusters using `kubectx -d ` for the next script to work.

```bash
gke_lustrous-spirit-371620_us-central1-a_mdb-cluster-2
gke_lustrous-spirit-371620_us-east1-b_mdb-cluster-3
gke_lustrous-spirit-371620_us-south1-a_master-operator
gke_lustrous-spirit-371620_us-west2-a_mdb-cluster-1
```

We need to create the required variables: `MASTER` for a master Kubernetes cluster, and `MDB_1`, `MDB_2`, and `MDB_3` for clusters which will host MongoDB replica set members. Important note: These variables should contain the full Kubernetes cluster names.

```bash
KUBECTX_OUTPUT=($(kubectx))
CLUSTER_NUMBER=0
for context in  "${KUBECTX_OUTPUT@]}"; do
 if [[ $context == *"master"* ]]; then
    MASTER="$context"
 else
    CLUSTER_NUMBER=$((CLUSTER_NUMBER+1))
    eval  "MDB_$CLUSTER_NUMBER=$context"
 fi
done
```

Your clusters are now configured and ready to host the MongoDB Kubernetes Operator.

### Installing Istio

Install [Istio (I'm using v 1.16.1) in a multi-primary mode on different networks, using the install_istio_separate_network script. To learn more about it, see the Multicluster Istio documentation. I have prepared a code that downloads and updates `install_istio_separate_network.sh` script variables to currently required ones, such as full K8s cluster names and the version of Istio.

```bash
REPO_URL="https://github.com/mongodb/mongodb-enterprise-kubernetes.git"
SUBDIR_PATH="mongodb-enterprise-kubernetes/tools/multicluster"
SCRIPT_NAME="install_istio_separate_network.sh"
ISTIO_VERSION="1.16.1"
git clone "$REPO_URL"
for ((i = 1; i <= ${#CLUSTER_NAMES@]}; i++)); do
  eval mdb="\$MDB_${i}"
  eval k8s="CTX_CLUSTER${i}"
  sed -i'' -e "s/export ${k8s}=.*/export CTX_CLUSTER${i}=${mdb}/"  "$SUBDIR_PATH/$SCRIPT_NAME"
done
sed -i'' -e "s/export VERSION=.*/export VERSION=${ISTIO_VERSION}/"  "$SUBDIR_PATH/$SCRIPT_NAME"
```

Install Istio in a multi-primary mode on different Kubernetes clusters via the following command.

```bash
yes | "$SUBDIR_PATH/$SCRIPT_NAME"
```

Execute the[ multi-cluster kubeconfig creator tool. By default, the Kubernetes Operator is scoped to the `mongodb` namespace, although it can be installed in a different namespace as well. Navigate to the directory where you cloned the Kubernetes Operator repository in an earlier step, and run the tool. Got to Multi-Cluster CLI documentation to lean more about `multi cluster cli`.

```bash
CLUSTERS=$MDB_1,$MDB_2,$MDB_3
cd  "$SUBDIR_PATH"
go run main.go setup \
  -central-cluster="${MASTER}" \
  -member-clusters="${CLUSTERS}" \
  -member-cluster-namespace="mongodb" \
  -central-cluster-namespace="mongodb"
```
### Verifying cluster configurations

Let's check the configurations we have made so far. I will switch the context to cluster #2.

```bash
kubectx $MDB_2
```

You should see something like this in your terminal.

```bash
Switched to context "gke_lustrous-spirit-371620_us-central1-a_mdb-cluster-2"
```

We can see `istio-system` and `mongodb` namespaces created by the scripts

```bash
kubectl get ns

NAME              STATUS   AGE
default           Active   62m
istio-system      Active   7m45s
kube-node-lease   Active   62m
kube-public       Active   62m
kube-system       Active   62m
mongodb           Active   41s
```

and the MongoDB Kubernetes operator service account is ready.

```bash
kubectl -n mongodb get sa

default                                     1         55s
mongodb-enterprise-operator-multi-cluster   1         52s
```

Next, execute the following command on the clusters, specifying the context for each of the member clusters in the deployment. The command adds the label `istio-injection=enabled`' to the`'mongodb` namespace on each member cluster. This label activates Istio's injection webhook, which allows a sidecar to be added to any pods created in this namespace.

```bash
CLUSTER_ARRAY=($MDB_1 $MDB_2 $MDB_3)
for CLUSTER in "${CLUSTER_ARRAY@]}"; do     
  kubectl label --context=$CLUSTER namespace mongodb istio-injection=enabled
done
```

### Installing the MongoDB multi cluster Kubernetes operator

Now the MongoDB Multi Cluster Kubernetes operator must be installed on the master-operator cluster and be aware of the all Kubernetes clusters which are part of the Multi Cluster. This step will add the multi cluster Kubernetes operator to each of our clusters. 

First, switch context to the master cluster.

```bash
kubectx $MASTER
```

The `mongodb-operator-multi-cluster` operator needs to be made aware of the newly created Kubernetes clusters by updating the operator config through Helm. This procedure was tested with `mongodb-operator-multi-cluster` version `1.16.3`.

```bash
helm upgrade --install mongodb-enterprise-operator-multi-cluster mongodb/enterprise-operator \
  --namespace mongodb \
  --set namespace=mongodb \
  --version="${HELM_CHART_VERSION}" \
  --set operator.name=mongodb-enterprise-operator-multi-cluster \
  --set "multiCluster.clusters={${CLUSTERS}}" \
  --set operator.createOperatorServiceAccount=false \
  --set multiCluster.performFailover=false
```

Check if the MongoDB Enterprise Operator multi cluster pod on the master cluster is running.

```bash
kubectl -n mongodb get pods
```

```bash
NAME                                                   READY STATUS    RESTARTS   AGE
mongodb-enterprise-operator-multi-cluster-688d48dfc6    1/1  Running 0  8s
```

It's now time to link all those clusters together using the MongoDB Multi CRD. The Kubernetes API has already been extended with a MongoDB-specific object - `mongodbmulti`.

```bash
kubectl -n mongodb get crd | grep multi
```

```bash
mongodbmulti.mongodb.com                         
```

You should also review after the installation logs and ensure that there are no issues or errors.

```bash
POD=$(kubectl -n mongodb get po|grep operator|awk '{ print $1 }')
kubectl -n mongodb logs -f po/$POD
```

We are almost ready to create a multi cluster MongoDB Kubernetes replica set! We need to configure the required service accounts for each member cluster.

```bash
for CLUSTER in "${CLUSTER_ARRAY[@]}"; do
  helm template --show-only templates/database-roles.yaml mongodb/enterprise-operator --namespace "mongodb" | kubectl apply -f - --context=${CLUSTER} --namespace mongodb; 
done
```

Also, let's generate Ops Manager API keys and add our IP addresses to the Ops Manager access list. Get the Ops Manager (created as described in [Part 2) URL.  Make sure you switch the context to master. 

```bash
kubectx $MASTER
URL=http://$(kubectl -n "${NAMESPACE}" get svc ops-manager-svc-ext -o jsonpath='{.status.loadBalancer.ingress0].ip}:{.spec.ports[0].port}')
echo $URL
```
Log in to Ops Manager, and generate public and private API keys. When you create API keys, don't forget to add your current IP address to API Access List.

To do so, log in to the Ops Manager and go to `ops-manager-db` organization.

![Ops Manager provides a organizations and projects hierarchy to help you manage your Ops Manager deployments. In the organizations and projects hierarchy, an organization can contain many projects

Click `Access Manager` on the left-hand side, and choose Organization Access then choose `Create API KEY`  in the top right corner.

The key must have a name (I use `mongodb-blog`) and permissions must be set to `Organization Owner` .

When you click Next, you will see your `Public Key`and `Private Key`. Copy those values and save them --- you will not be able to see the private key again. Also, make sure you added your current IP address to the API access list.

Get the public and private keys generated by the API key creator and paste them into the Kubernetes secret.

```bash
kubectl apply -f - <
  privateKey: 
EOF
```

You also need an  `Organization ID`. You can see the organization ID by clicking on the gear icon in the top left corner.

Copy the `Organization ID` and paste to the Kubernetes config map below.

```bash
kubectl apply -f - <
EOF
```

The Ops Manager instance has been configured, and you have everything needed to add the MongoDBMultiCRD to your cluster.

### Using the MongoDBMultiCRD

Finally, we can create a MongoDB replica set that is distributed across three Kubernetes clusters in different regions. I have updated the Kubernetes manifest with the full names of the Kubernetes clusters. Let's apply it now!

```bash
MDB_VERSION=6.0.2-ent
kubectl apply -f - <