Node.js Monitoring with Prometheus+Grafana
By: Jack Yeh
Tags: Prometheus, Hello World, Nodejs, Monitoring, Grafana

Less articles. More git clones.

Well, here is the finished sample code repo if you’d like to see it in action:

https://github.com/teamzerolabs/express-monitoring

yarn start to have the server and metrics exporter up and running.

to have the server and metrics exporter up and running. Continue if you want to know how metrics are wired up into prometheus and grafana

and Need help setting things up and give recommendations? Contact us at info@teamzerolabs.com

Hello world

In this article, we will be monitoring this short express app.

If you can setup metrics for this app, you will be ready for any other express monitoring.

Start by cloning the repo, and run yarn dev . Once the server is up and running, try hitting the api in the browser/postman/curl a couple of times. Next, we will learn about prometheus and exporters.

Prometheus — all purpose metric scraper

A quick word on prometheus:

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud

In some monitoring/alerting solutions, the application is responsible for sending events/metrics to be ingested. In Prometheus, we configure it to visit exporters to collect the metric for aggregation.

Here is what prometheus sees during the visit:

Metric pages in a nutshell, names followed by optional labels, followed by a number

Exporter? Never heard of her!

An exporter is any program that will produce the metrics plaintext page (pictured above) when asked. There are all kinds of external exporters that can keep track of:

machine stats: CPU/Memory/Disk remaining size/Network traffic

database stats: Check number of running queries/database sizes

When we add instrumentation code into our Node.JS app, so that it can respond with a metrics page, we have effectively made it an exporter.

Our express server is running on port 4000, if you run yarn start , you will see that the metrics server is running on port 9991. You can see what the current metrics look like at localhost:9991/metrics

Every exporter will expose a minimum of one metric: up . To capture request call counts and response time, we will install the package express-prom-bundle .

Note: there are several techniques and other packages used inside setup.js to aggregate routes together and handle loose trailing slash route matching. We will cover how they are used in future articles!

Here is what the final main.js looks like

Add 4 lines, and we are ready to capture request counts and response time!

Following express-prom-bundle , we insert the middleware before defining routes.

, we insert the middleware before defining routes. After all routes are declared, captureAllRoutes will fetch all url patterns to be matched.

will fetch all url patterns to be matched. Finally, setupMetricsService will run another express app to serve /metrics traffic to outside world.

will run another express app to serve traffic to outside world. Try hitting a couple of routes in the browser, then revisit localhost:9991/metrics , you should see something similar to the following:

Almost there!

Metrics Exporter is ready, what’s next?

We have an exporter running and serving metrics, but no one is listening in yet. If you have docker running on your machine, you can use dockprom to quickly spin Prometheus+Grafana up.

git clone https://github.com/stefanprodan/dockprom

cd dockprom



ADMIN_USER=admin ADMIN_PASSWORD=admin docker-compose up -d

Once docker-compose is up and running, you can visit the prometheus dashboard by going to localhost:9090 , the default http user and password are both admin .

And, you can find the Grafana dashboard at localhost:3000

Connecting exporter to Prometheus

Fresh clone of dockprom includes the following:

Under dockprom/prometheus/prometheus.yml , we find an array of jobs prometheus will scrape. Add the following into the list:

- job_name: 'nodejs'

scrape_interval: 10s

honor_labels: true

static_configs:

- targets: ['host.docker.internal:9991']

Run docker-compose down && docker-compose up -d

Prometheus will now:

Scrape host.docker.internal:9991 (On Mac, host.docker.internal points to the local machine) every 10 seconds.

Refresh the prometheus landing page, and you will see under Status/Targets that our Node.JS app is now under watch!

Next, try to type in http_request_duration_seconds_count in the Graph Query Editor, and this will show up.

Our express app metrics are hooked into prometheus!

One more thing

Prometheus query editor and graph displays are nice, but how do I create and organize an entire dashboard? Here’s where Grafana comes in.

Launch localhost:3000 in the browser, and click on create a new dashboard:

If this is your first dashboard, you must call it Dashboard helloworld.

Click on Add panel, and then select Add Query .

Under Metrics, we can put in http_request_duration_seconds_count , click outside the box, and it will automatically show the graph!

Say hello to your very first panel.

Click Back on the top left, and don’t forget to click on Save to keep this panel around next time you visit Grafana.

Ready to monitor everything

You now have an application that exports metrics to prometheus, and Grafana setup to look at metrics from prometheus! Remember, as long as a program can serve a metrics page to Prometheus, it can be counted as an exporter.

Further Reading