Develop and Deploy a Scalable RESTful API using Node.js & Mongo. | by Saleh Alkhalifa | Towards Data Science

Introduction to APIsGetting Software InstalledCreating a Remote Mongo Database (AWS)Creating the Node ServerDeploying your API (Heroku)
Introduction to APIs
Getting Software Installed
Creating a Remote Mongo Database (AWS)
Creating the Node Server
Deploying your API (Heroku)
An API, or Application Programming Interface, can be simply defined as an interaction between various software components. When a user clicks a button to see a list of their Facebook friends, likes on Instagram, or emails within an inbox, data is generally being exchanged in one way or another through a web service API.
There are a number of different types of web service APIs such as:
SOAP — Simple Object Access ProtocolXML-RPC — XML Format for Data TransferJSON-RPC — JSON Format for Data TransferREST — Representational State Transfer
SOAP — Simple Object Access Protocol
XML-RPC — XML Format for Data Transfer
JSON-RPC — JSON Format for Data Transfer
REST — Representational State Transfer
For the purposes of this article, we will be focusing specifically on the utilization of REST as an architectural principle.
APIs function as methods allowing for the transfer of data from the client-side, to an API server which interacts with a database of some sort. Data is generally retrieved based on a set of conditions, and then returned back to the client.
One of the most commonly used examples for an API generally involve the stock market, companies and daily prices. While keeping the diagram above in mind, a user (client) may request the price history data for a particular company. The client-side would make an API call specifying a specific company, and the API server would retrieve that information from the database, and return it to the user. Note that the user never interacts with the database, only the API server.
An API call is an action in which an endpoint of a server is specified and request, and the server responds with the requested content. Take for example the process of logging into a social media website. A user would fill out their login information, and then click “login”. Upon clicking this button, a number of API calls are made to authenticate the user and then retrieve the user’s data. This in of itself is an example of an API call. Let us take a look at a simple example of a URL in which stock price data is being requested for a specific company.
In the above example, you see the URL request broken down by section. We begin with an HTTPs protocol, followed by the name of a particular domain of interest. Following the best practices of a RESTful API, we then specify the path of the api as well as the version. We then specify the exact end point of interest. Notice that the endpoint follows a particular “funnel down” approach in which we narrow down the scope of the data as we go from the left to right. We first start with all of the companies, and then specifically selection Apple as the company of interest, and then specifically request the prices from that one company. This is a classic RESTful architectural pattern.
When interacting with an API server, there are four main methods one can take: GET, POST, PUT, and DELETE. For a more detailed explanation, please visit my last article in which in discuss this in more detail in a Python setting.
In the following steps, we will be developing a Node.js API using Mongo as our database. Let us go ahead and get started.
You can download Node.js directly from their website. Select the LTS option which is recommended for most users, and install that locally on your machine. You can confirm the installation by running “node” on your command line.
(base) C:\Users\alkhalifas> node
...and you should see a welcome message from nodejs. If you do not, please re-run the installation of node.
On the other hand the MongoDB Community Edition can be downloaded using this link. Select the version appropriate for your OS, and click download and install. You can confirm the installation of mongo by running either “mongo” to access the client, or “mongod” to start the database server.
(base) C:\Users\alkhalifas> mongo(base) C:\Users\alkhalifas> mongod
There are a number of remote database options ranging from scalable relational database to object databases depending on the specific usecase. For the purposes of this tutorial, we will be deploying our API to an object database known as Mongo. A Mongo database can be used via the CLI we installed in the previous section, or via a GUI. To get started, you will need to navigate the to https://www.mongodb.com/.
Create a new account by selecting the “Start Free” option in the main page. once you have signed up for a new account, navigate to the clusters section of the website and select “Create a New Cluster”.
For the Cloud Provider & Region, select AWS and change the region closest to you. When selecting a region, be sure to select one with with “Free Tier” option.
Keep all other settings such as Cluster Tier & Additional Settings as their defaults. Feel free to change the cluster name. Finally, click “Create Cluster”. You will need to wait a few minutes while the cluster gets provisioned. Follow the tutorial that will auto-start at the end of the provisioning. This will walk you through creating the clusters credentials. Be sure to note the username, password, and collection. We will need those in a few minutes.
Back on the main cluster page, find your new cluster and click the “Connect” button:
A new menu will appear with three options. Select the option called “Connect to MongoDB Compass”.
Clicking this option will bring up another menu consisting of two steps. Step 1 is to download Compass which is a GUI interface allowing you to interact with the database. The alternative to this is using the CLI we installed in the previous section, however, it is highly recommended you use the GUI. Step 2 is to copy the connection string. Be sure to substitute <password> for your actual password you created in the tutorial. Save this connection string, we will need this in a few moments.
With Atlas installed, you can now open the GUI, connected to your database using the connection string, and create new data for your database.
We can get stared by creating a new directory and then initializing npm (which was installed with Node). You will be asked to fill out a few items for the project such as the name, author, etc. These are completely optional, but recommended as best practice. Keep most items as their default values.
(base) C:\Users\alkhalifas> mkdir node-api-books(base) C:\Users\alkhalifas> mkdir cd node-api-books(base) C:\Users\alkhalifas\node-api-books> npm init
At this point, the project is created and ready for development. I would recommend you open the directory with your favorite IDE such as IntelliJ, WebStorm, or VisualCode — all will function the same.
We will be creating a few files within the project, each of which server a very specific function. Theoretically, we could group the contents of all of these files into server.js, however, decoupling the contents based on function is considered best practice when developing a node API, which is what we will complete here today. We beign with server.js which is the main file any incoming request will interact with. This then prompts the controller which determines the specific function to carry out based on the endpoint. The service then communications with the DAO (Data Access Object), which carries out a specific function in conjunction with the model and schema of our database — both of which will be governed using the mongoose library.
To get stared, let us go ahead and install two libraries we will need: (1) Express to handle the backend web application framework of the API, and mongoose to help with the database interaction with Mongo.
npm install express --savenpm install mongoose --save
The server.js file acts as the main ‘gatekeeper’ for all API calls. There are three main functions we will set up within this file:
Mange access for incoming callsGuide traffic to the controllerConnecting to the Mongo database
Mange access for incoming calls
Guide traffic to the controller
Connecting to the Mongo database
Managing access is commonly done through what is known as a CORS policy. This can be configured using the CORS library which allows you to specify the domains that you do and do not wish to grant access to. For the purposes of this tutorial, we will grant access to all domains by listing the “*” in the code below.
app.use(function (req, res, next) {    res.header('Access-Control-Allow-Origin', '*');    res.header('Access-Control-Allow-Headers', 'Content-Type, X-Requested-With, Origin');    res.header('Access-Control-Allow-Methods', 'GET, POST, PUT, PATCH, DELETE, OPTIONS');    next();
Once access is determined and granted, the file then passes the API request to controller which will determine the type of request by determining the endpoint. In order to access the controller, we must ‘require’ it.
require("./controllers/book-controller")(app);
Finally, we will connect to the remote Mongo database using the Mongoose library. Paste the connection from the previous section that was copied from the MongoDB website.
mongoose.connect('<connection string>',                 {useNewUrlParser: true,                     useUnifiedTopology: true});
The controller is responsible for outlining the endpoints of your API that users can navigate to. For example, one particular endpoint we can create is www.website.com/api/v1/books which should return a list of all books.
app.get('/api/v1/books/', (req, res) =>    bookService.findAllBooks()        .then(allBooks => res.json(allBooks)));
Perhaps we could even narrow down the scope and return a single book by specifying the book’s ID. For that, we could use an endpoint of www.website.com/api/v1/books/123, in which 123 is the book’s ID or primary key. We could also set up an endpoint to query by author.
app.get('/api/v1/books/:bid', (req, res) =>    bookService.findBooksById(req.params['bid'])        .then(book => res.json(book)));app.get('/api/v1/author/:aname', (req, res) =>    bookService.findBookByAuthor(req.params['aname'])        .then(books => res.json(books)));
In addition to retrieving values, we can also specify other CRUD operations here such as deleting, posting, and updating our database entries.
app.delete('/api/v1/books/:bid', (req, res) =>    bookService.deleteBookById(req.params['bid'])        .then(status => res.send(status)));app.post('/api/v1/books/', (req, res) =>    bookService.addNewBook(req.body)        .then(newBook => res.json(newBook)));app.put('/api/v1/books/:bid', (req, res) =>    bookService.updateBook(req.params['bid'], req.body)        .then(status => res.send(status)));
The book-service file is responsible for connecting the book-controller to the book-dao (Database Access Object). Within this file we simply create a connection by creating a new constant and connecting it to the dao function. In the case of finding all books:
const findAllBooks = () => bookDao.findAllBooks()
The book-dao file is responsible for connecting the data access object to the database object’s model. It is here that we can now use Mongoose operations such as find() and findById(id). For example, we can find all book values using:
const findAllBooks = () => bookModel.find()
The book-model file is responsible for handling the model and its association to the predefined schema in the file book-schema. One of the benefits of defining a schema and connecting it to the model is ensuring that all entries will be identify in their format. Within this file we define the attributes of our Book object in the sense that a book can have a title and author which are of the type String. A publication date which is of the type Date. An edition in the form of an integer. And finally a type which must be of one of three available options of type String.
const bookSchema = mongoose.Schema({    title: String,    author: String,    pubDate: Date,    edition: Number,    type: {        type: String,        enum: ['HARD_COVER', 'SOFT_COVER', 'E_BOOK']},}, {collection: 'books'});
Running the Node Server:
Depending on your IDE of choice, there may be some built in capabilities to run your server directly from your menu bar. If not, you can always run this via the command line. Navigate to the folder containing the server.js file.
(base) C:\Users\alkhalifas\node-api-books> node server.js
In your web browser, navigate to http://localhost:3000/api/v1/books which should now display a list of books in JSON format.
With that, the server is now ready for deployment on Heroku. Go ahead and add and commit all your code thus far to GitHub.
There are many servers where one can deploy their API, one of which is Heroku. If you have not done so already, navigate to www.heroku.com and create a free new account.
Within the apps menu, click on “New” and then “Create new app”.
Assign the application a name and region, and click “create”. You will then be guided to the application’s menu.
Within the application, navigate to the “Deploy” section.
Next, select your deployment method of choice. You are free to deploy your API via the Heroku CLI, or via the GitHub integration.
Once the application has been deployed, click on the “Open App” option at the top of the menu and you will be directed to the API where you can now navigate to the API endpoints we prepared.
Within this article we covered the basic concepts of RESTful APIs including their characteristics, structure, and architecture. We then provisioned and deployed a mongo database using MongoDB and AWS. Next, we prepared a best-practice Node server with full CRUD operations to manage our database of books. Finally, we deployed the server on Heroku allowing us to access our API from anywhere. The code for this project can be cloned from by GitHub account.