How To Deploy 3 Node MongoDB Shard Cluster On AWS
By: Rakesh Bhatt
Tags: Database, Mean Stack, Clustering, Mongodb, DevOps

Photo by Safar Safarov on Unsplash

In this tutorial, we will deploy a MongoDB sharded cluster in AWS. You can try this tutorial to deploy sharded cluster into any cloud with a little change.

Currently, as per MongoDB guidelines you need 3 nodes as config server replica set, at least 3 shard nodes where each node is 3 member replica set and one or more mongos server.

According to the official documentation, we need to have below infrastructure:

3 AWS EC2 instances for config servers (3 Node replica set) 9 AWS EC2 Instances for the deployment of 3 shards where each shard has a replica set of 3 nodes 1 AWS EC2 instance for mongos

But for this tutorial, we will deploy the shard cluster with below infrastructure:

1 AWS EC2 instance for config servers (1 Node replica set) 2 AWS EC2 Instances for the deployment of 2 shards where each shard have a replica set of 1 node 1 AWS EC2 instance for mongos

Let’s do it then —

Config Server

In the sharded cluster, every operation/query has certain metadata stored in it, which decides the path of data storage along with MongoDB read and write operations.

Config servers is a key database that holds metadata information to operate sharded cluster

Deployment: — Config sever needs to be deployed in 3 node replica set and making it a replica set is mandatory. We will deploy a single node replica set for this tutorial.

Mongos

Mongos is a router which works as the main host, You can use MongoDB client libraries to operate on a sharded cluster with any number of Mongos hosts.

Deployment: — Multiple mongos routers can be connected with the config server. As they don’t have any database of their own. They communicate with config servers to operate with sharded cluster.

Having multiple mongos server help in the case where one mongos server is down client will use other mogos server to query cluster. All mongodb client libraries support multiple mongos hosts.

Shards

Shards are mongod instances that contain database and collection data. Distribution of data is decided by config servers based on shard keys configured by the mongos server.

Deployment: — By MongoDB guidelines, we need to have a 3 node replica set per shard. For simplicity we will be creating two shards here with 1 node replica set.

You can choose any cloud you wish (AWS/Oracle/AZure etc). For this article we will use AWS.

Create 3 Ubuntu instances and name them

1 Server instance for config server/mongos (config_instance) 2 Servers for shards (shard1_instance/shard2_instance)

MongoDB Installation:

You can follow MongoDB guidelines — https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/

MongoDB Version:

You have to be a little cautious here in case you are trying to restore any kind of data into sharded cluster. mongorestore doesn’t work above 4.2.x because of transaction support

cannot be part of a backup strategy for 4.2+ sharded clusters that have sharded transactions in progress, as backups created with ** Notes on MongoDB site - mongodump and mongorestore be part of a backup strategy for 4.2+ sharded clusters that have sharded transactions in progress, as backups created with mongodump do not maintain the atomicity guarantees of transactions across shards.

Installation procedure

Ssh into config_instance and install MongoDB from the given link. MongoDB will place a config file at location /etc/mongod.conf Open /etc/mongod.conf file and change data directory or log directory by changing properties dbPath and systemLog path. if you are changing default directories, do not forget to give ownership to mongodb user created by during mongodb installations using below command

sudo chown mongodb:mongodb /volume_paths -R

3. After installation run mongod service using command

sudo service mongod restart

4. If you want to go ahead with key file security lets create a key file too

openssl rand - base64 756 > /home/ < userename > /

chmod 400 shardauthkey

5. Now open /etc/mongod.conf in the text editor of your choice vim/nano etc. and copy below setting to the bottom of /etc/mongod.conf file

sharding:

clusterRole: configsvr

replication:

replSetName: config

net:

bindIp: 0.0.0.0

6. We have declared server cluster role and we have set up a replica set name as well in mongodb configuration.

7. Let’s restart mongod service. After restart lets enter to the mongod shell

8. Enter mongod shell using below command. Here port is 27017 which is a default port configured in mongod.conf file

mongod --port=27017

9.Once the mongod shell is open enter below command

{

_id: "config",

configsvr: true,

members: [{

_id: 0,

host: "<interanl/elastic_ip_of_config_server>:27017"

}, ]

}

You will see a replica set has been initiated. We have initiated a single node replica set for now. You can add more instances in the replica set if you need to. You will see shell turned into primary after a while.

10. Now set up a default user for authentication which is very important. Enter command after opening mongo shell

use admin

db.createUser({

user: "config",

pwd: "config@123",

roles: [{

role: "root",

db: "admin"

}]

})

11. Exit from the terminal now enter the mongo shell again with below command:

mongo --port=2707 -u config -p config@123 --authenticationDatabase=admin

12. Open config file again /etc/mongod.conf and append below lines at the bottom of the file

security:

keyfile: /home/<username>/shardauthkey

Restart mongod service. We have set up our config server.

Shard Servers:

Shard-1:

Ssh into shard1_instance and shard2_instance we have created at the beginning of this tutorial. You can setup both shards with same procedure mentioned below Install MongoDB as defined at the beginning of this article Copy key file from config server put it in user home directory of both shard instances Give key file suitable permissions. We need to use this key file in our sharded cluster so give it requiried permissions so that it can be used

sudo chown 400 /home/<username>/shardauthkey

5. Open again /etc/mongod.config file in any text editor and place below lines at the bottom of file.

sharding:

clusterRole: shardsvr

replication:

replSetName: shard1

net:

bindIp: 0.0.0.0

For shard2 just change the name of the replica set to shard2 for ease

6. Now enter into mongod shell with below command

mongod --port=27017

7. Initiate a replica set. Its a single node replica set. Add more server to this shard if you wish

rs.initiate({

_id: "shardsvr",

members: [{

_id: 0,

host: "elastic_ip/public_ip/private_ip>:27018"

}, ]

})

8. After few seconds you will see mongod shell turning to primary

9. Run below command and create auth user here

use admin

db.createUser({

user: "shard",

pwd: "shard@123",

roles: [{

role: "root",

db: "admin"

}]

})

10. Open config file again /etc/mongod.conf and append below lines at the bottom of the file

security:

keyfile: /home/ < username > /shardauthkey

11. We have added keyfile as well which we have copied from config server where we initially created it

12. Now restart mongod and enter the shell with below command

mongo --port=2707 -u shard -p shard@123 --authenticationDatabase=admin

Done

Now we are done with our initial setup. But the major question is How we will enable each node to communicate with each other.

Network conditioning

Config server: config_server need to have direct access to <shar_1_ip> on port 27017 and <shard_2_ip> on port 27017. Shard servers need to communicate to config server so they need access to <config_server_ip> on port 27017

AWS

I will explain about AWS. Similar way you can set up in any cloud

<shard_1_ip> : (Can be elastic IP/internal IP) Use internal_ip for better security

<shard_2_ip>: (Can be elastic IP/internal IP) Use internal_ip for better security

<config_server_ip>: (Can be elastic IP/internal IP) Use internal_ip for better security

In security group of shard server on AWS, give inbound permission to config_server_ip onport 27017. In security group of config server on AWS, give inbound permission to shard_1_ip and shard_2_ip on port 27017

On some servers, you need to edit firewall rules. Go ahead and edit in the same way

Our cluster is ready to start communicating with each other

Mongos:

Ssh to config_instance Create mongos.conf file in /etc/

systemLog:

destination: file

logAppend: true

path: <log file directory>

net:

port: 27020

bindIp: 0.0.0.0

processManagement:

timeZoneInfo: /usr/share/zoneinfo

sharding:

configDB: config/<inter/public/elasticip/>:27017

security:

keyFile: /home/<username>/shardauthkey

3. Now run mongos

mongos —config /etc/mongos.conf

4. To run mongos without interruptions we need to daemonize mongos like mongod. In the latest ubuntu version systemctl service can be added for mongos. Or if you want to use you can use supervisor as well.

5. Now login to mongos using the below command. We have changed the port in mongo command as config server is running on port 27017 and mongos is running on port 27020

mongo --port=27020 -u shard -p shard@123 --authenticationDatabase=admin

6. Once you have entered into mongo shell you can add shards to cluster now.

sh.addShard( "shard_1<public/private/elasticip>:27017")

sh.addShard( "shard_2<public/private/elasticip>:27017")

7. Add a new user to provide access to mongodb clients

use admin db.createUser({ user: "sharduser",

pwd: "sharduser@123",

roles: [ { role: "root", db: "admin" } ] })

8. Done we have setup the cluster

9. We can initiate a few sharded collection in mongos if we have already had them otherwise we can add them later

sh.enableSharding("<database_name>")

sh.shardCollection("<database_name>.<collection_name>", {

< shard key field >: "hashed"

})

Shard Keys

Be cautious of shard keys as they provide information about the way data across the entire cluster will be distributed. Shard keys also put some restriction on collections for data update and querying them

For more details look into the below article:

https://docs.mongodb.com/manual/core/sharding-shard-key/

Using mongos

We have setup mongos in the config server. Go to its security group, give inbound rule on config_server public IP, and port 27020 to all.

Now you can use shard cluster with below command

mongo —host <public_ip_of_config_server> —port 27020 -u sharduser -p sharduser@123 --authenticationDatabase=admin

Use it in any mongo client library. You can scale a cluster to 3 nodes replica set per server and similarly multiple mongos too.

I hope this will clarify your doubts about MongoDB shard cluster setup.

Thanks for reading