Taking Advantage of MongoDB’s ‘Change Streams’ for real-time IoT applications using Node.js
By: Tomas Ambrogi
Tags: Nodejs, IoT, Servers, Clustering, Mongodb

Introduction

A few months back I was involved in a project which had a microcontroller with a couple of sensors connected to a PC feeding it new data constantly. The PC would store and process this data, generating event triggers which were sent through sockets to a web page. I needed a way to store each new event while at the same time keep the web page updated in real time.

Why MongoDB?

MongoDB is a NoSQL (non-relational) database, which means it doesn’t require a predefined table schema. This can be useful especially when working with big loads of unstructured data that may or may not change during your project’s lifetime. Given that a simple change in the project’s specifications may require adding new sensors, or new data stored from these sensors, a non-relational database fitted my needs perfectly.

The Node Server

Although there are other frameworks and programming languages for server development, Node has proven to be easy to implement and deploy. I took advantage of Express and developed the web page using Vue because… let's face it, Vue is simply awesome.

So, let's dig into the code:

server.js:

var MongoClient = require('mongodb').MongoClient; const pipeline = [{$project: { documentKey: false }}]; MongoClient.connect("mongodb://localhost:27017?replicaSet=mongo-repl", { useNewUrlParser: true }, function(err, database) { if (err) throw err; var db = database.db("demo-db"); console.log("[mongodb] Mongo connection established"); // Show database docs for debugging console.log("[mongodb] Mongo db dump: "); db.collection("data").find({},{fields: {"_id" : false }}).toArray(function(err, result) { if (err) throw err; console.log(result); }); const changeStream = db.collection("data").watch(pipeline); changeStream.on("change", function(change) { dataObj = {"sensor" : change.fullDocument.sensor} console.log(dataObj); }); });

In order to run this code, there are a couple of things that need to be done first.

You need to install MongoDB server, click on the link for a detailed explanation on how to install it on Ubuntu.

After installation, you will need to create a specific folder within your project folder.

So imagine your project is on ~/project, you need to run:

cd ~/project

mkdir data

And then (and this is the key part):

mongod --port 27017 --dbpath ./data --smallfiles --replSet mongo-repl

And in another terminal run:

mongo --port 27017 --eval 'rs.initiate({"_id":"mongo-repl","members":[{"_id":0,"host":"localhost:27017"}]})'

Ok, so what was all that? Basically, after creating a folder for all our database information, we need to run the mongod service (what we’ve just installed) and specify the port and path for your replica set. Yes, a replica set. This was what caused me so many troubles when attempting this for the first time. In order to be able to watch changes in our database in real time (defined as change streams in mongodb’s documentation) you need to run not one instance of our db, but a cluster. And this is basically what you are doing with the code above.

The first line creates the replica set, specifying your recently created data folder, so that the cluster stores its information there. You also tell mongod the port in which our cluster will be running, and a few other options, with the parameter replSet being the name of our cluster.

After that, you need to enter mongo shell and initialize our recently created cluster, which is what the second command does.

A few notes:

The replica set needs to be initialized only once, there’s no need to run that command every time.

If you’re running it as a script, consider adding ‘&’ at the end of the mongod command to avoid having to use multiple terminals.

Once you’ve done all this we are ready to run our server.js, so:

cd ~/project

npm init

npm install mongodb --save

node server.js

(I’m assuming you’re familiar with Node and npm)

Now you could run another script in a different terminal to insert a new item in the recently created database and see how server.js prints every new insert in the cluster. In the new terminal run:

mongo

This will open Mongo’s console so now we can interact with our database:

use demo-db

db.data.insert({sensor: 'Sensor A'})

And just like that, in our server.js terminal, we should see our DB insert appear as a console log.

Conclusions

IoT is not only changing the way we think about hardware but also how we design our software. Requirements usually include being able to process data in real time and storing it for machine learning and big data analysis. All these requirements have an impact on our server application, so choosing the right tools is essential.

MongoDB has proven to be a very flexible tool for our IoT applications regarding data storage. If all worked out ok you should be able to integrate this simple example into your project.