2020 Node.JS Quick Upgrade Guide. Bring your repository into the future…
By: Jack Yeh
Tags: Typescript, Error Handling, Nodejs, Monitoring, Programming

Bring your repository into the future in 5 easy steps.

Photo by Ian Simmonds on Unsplash

Background

Raise your hand if you have a web service written in Node running in production from 2017 and earlier untouched.

…

If, like me, you have more micro-services fingers on your hands, you know that some of them are long overdue for a proper upgrade. Old repository often features the following symptoms:

Nested Callback/Promises everywhere

Ambiguous function input/output types

Mixed formatting and spacing, no clear structure of functionality and naming convention

Lackluster error logging

Zero reporting and monitoring support

Fear not, we know exactly the remedy for each of these symptoms, read on to see how to tackle each sections and bring your repository into the future today!

1. Nested Callbacks/Promises everywhere

Back in 2017, many popular coding libraries did not have support for Promises. We wrote code with dubious execution paths which can lead to hard to track down bugs.

Example: A route that saves a record into database and log an event to analytics.

server.post('/products/', function (req, res, next) {

const record = req.body;

model.save(record, function (err, dbResult) {

if (err) {

next(err);

} logEvent('new_product_created', dbResult.id,

function (err, result) {

if (err) {

logger.error('Failed analytics' + err.message);

}

res.json({ success: true, id: dbResult.id });

});

});

});

Review the version of your library usage, and upgrade them to use Promises and Async/Await:

server.post('/products/', async (req, res, next) => {

const record = req.body;

try {

const dbResult = await model.save(record);

await logEvent('new_product_created', dbResult.id)

.catch(err =>

logger.error('Failed analytics' + err.message));

res.json({ success: true, id: dbResult.id });

} catch (err) {

next(err);

}

});

When updating, keep these rules in mind, they are equally important:

If a function uses callbacks inside, it can be converted to emit Promise objects.

If a function returns a Promise, it should have the “ async ” modifier.

” modifier. Prefer flat level of try catch/Promises over nested level of callbacks.

You also do not have to apply all of these style rules in one attempt. It’s enough to covert them as you go and pursue positive incremental changes with plenty of tests!

2. Ambiguous function input/output types

Functions that try to do too much will end up becoming hard to debug. Picture the following:

function validateObject(input) {

if (Array.isArray(input)) {

return input.map((record) => validateObject(record);

}



if (input.propA === "correct") {

return true;

} else if (input.propC === 42) {

return "meaning of life";

} else {

return ["passed"];

}

} validateObject(["nice", { propA: "correct", propC: 42}]); ==> ?

Whoever calls this function will have to check if the return value is an array, string, boolean. Using functions like this everywhere means your code will be littered with type checks and become bug ridden.

Luckily, you can tackle this issue easily: Start with adding typescript to your project.

Typescript conversion can be done in a piece meal fashion, not requiring the user to add types to every file at the beginning.

I recommend WebStorm, which comes with an amazing built-in linter.

Add types to your function parameters and return values, and watch the editor work its hardest to let you know when any function is returning inconsistent types.

Furthermore, Typescript will remind you about function return types for Async functions!

If you know a function is returning Promise, label it with async and let the editor do the rest of the work for return type correction.

3. Mixed formatting and spacing

Anyone who has ever worked in a team knows that members can have different preferences for empty spaces. To ensure the formatting stays consistent, we usually hook up a linter in the pre-commit hooks.

This step formats the code according to a pre-defined rule before code check-in. But, what if we want to nudge team members to code in a similar style while they are coding, so when we pull our own code changes, it is less jarring?

Answer: Use a code editor that auto-formats on edit, like WebStorm. Turn on the Prettier Integration, and check in lint configuration to the repository.

To add Prettier to the project:

yarn add --dev prettier

Then, add this file .prettierrc at the base level:

{

"tabWidth": 2,

"semi": false,

"singleQuote": true,

"trailingComma": "es5",

"printWidth": 80,

"arrowParens": "always"

}

Setup these file watchers:

And you will never have to think about space formatting rules or argue with a coworker on spaces ever again! Prettier even works for JSON files, keeping everything tidy automatically.

4. Lackluster error logging

After running services in Production for a couple of years, I am sure you have had user complaints. Usually, developers receive tickets on broken behaviors, they (and QA team) do the following:

Try to reproduce the behavior in local environment, and cannot reproduce.

Try to reproduce the behavior in staging environment, and cannot reproduce it.

Try to reproduce the behavior with a dummy user in production environment, and cannot reproduce it.

Try to find error logs around the bug by tailing the logs, this is not feasible for busy systems.

Try to download the logs and grep it or use splunk to find the logs.

Some developers even reach this step:

Refuse the existence of bugs. Ask the user to try again, and the error went away. Concludes that it was the user’s fault.

When a bug occurs in production, ideally the team receives the following:

The exception thrown in the code.

Breadcrumb of events around the error.

Customer’s request context (which url were they hitting, user agent strings)

If your traffic volume is small, you could follow the logs in production and try to reproduce the error. This is a time consuming chore when you have more traffics. Instead of allocating all of our times in search, imagine a service that can catalog all of your runtime errors and display them in a dashboard:

This is what Sentry does.

You establish a connection to Sentry’s server on program startup. Every time there is an exception generated, it is forwarded to Sentry, and they will help record information around the error to give more context.

The setup code is dead-simple:

import { init } from '@sentry/node' function startSentry() {

init({

environment: config.app.env,

dsn: config.sentry.dsn,

enabled: config.app.env === 'production',

})

logger.info(`Sentry configured: ${config.sentry.dsn}`)

} // For known errors:

import { captureException } from '@sentry/node' try {

knownError();

} catch (err) {

captureException(err);

} // For unknown errors:

// You don't have to do anything, Sentry will track it.

If you have customers complaints that is hard to track down, check out Sentry’s offering!

5. Zero reporting and monitoring support

If the service was built in a tight deadline, it probably has its fair share of tech debts. For new comers, tech debts are corner cutting solutions that we put in to meet deadlines.

We often leave // TODO FIXME Shouldn't do this throughout the code when the project is rushed to completion. Rushing a project has its advantages, sometimes you just do not have enough funding/runway to do it proper, or you are afraid that your competitors are about to ship their offering soon.

After tech debts are introduced, they will stick around until the end of the product’s shelve life unless we intervene. But, like all businesses, time is money. We definitely do not want to aimlessly pursue a refactoring or rewrite without weighing the cost.

How can we justify removing technical debt? With Metrics and Numbers.

If we can compile a report of the service itself being unstable or slow, we have better chances of convincing management to let us do clean up work.

For example, a particular route is returning error code 500 30% of the time, which leads to customer support tickets being generated.

Or, one of the image processing function takes the following durations to complete: 5 seconds/60 minutes/Errors out.

Or the most classic example: Our service runs out of memory every week and needs to restart on those thresholds.

If you work on any coding projects at all, you will create and run into technical debts. These debts can lead to bad customer experiences, which directly translates into support tickets, and ends up taking time away from the team that might otherwise have been able to work on new projects.

Fixing issues without keeping tabs on metrics is similar to ignoring the history:

Those who ignore history are doomed to repeat it

So, when you receive customer support tickets, your first reaction should be: How can I measure this, to have reference points in the future?

And not: Let me fix it today, release a patch in production and promptly forget about it.

You can measure the frequency and duration of events with Prometheus:

yarn add prom-client

Then, measuring duration is as simple as the following:

A histogram comes with the following buckets by default, [0.1, 5, 15, 50, 100, 500].

When an event takes less than 499 seconds to complete, it will be placed in the last bucket.

When it takes less than 75 seconds to complete, it will be placed in [ < 100, < 500] buckets.

When it takes less than 0.1 second to complete, it will be placed in all of the buckets.

const client = require('prom-client');

const histogram = new client.Histogram({

name: 'myapp_image_process_duration',

help: 'How long image processing function takes to complete',

}); async function imageProcessingFunc(input) {

const end = histogram.startTimer();

...

... Busy async works

...

end();

}

Over time, our program will produce a series of buckets and how many events are in each bucket.

We can then measure the average user’s experience with Prometheus’s query language.

When this duration is unbearably long, we will know it is the right time to devote energy in a proper fix!

Conclusion

To ensure legacy Node code base live long and stays relevant, I encourage you and your team to pursue the following practices:

Flattening code flows, avoid nested callbacks and promises.

Use TypeScript. Enforce consistent input/output types.

Adopt Prettier and WebStorm.

Ship runtime errors to Sentry or equivalent. Avoid looking through logs and breadcrumbs manually.

Instrument code with Prometheus, find out where the weakest link is. Metrics will help with decision making and buy-ins.

Remember, sustainability is more about principle than speed!

Need advice on porting codebase? You can reach out to us at info@teamzerolabs.com