Using node modules with njs

Planning your ingress-nginx migration? Start here:
kubernetes.nginx.org.

english
русский
news
about
download
security
documentation
faq
books
community
enterprise
community forum (new)
x.com
blog
njs
ingress controller
gateway fabric

Using node modules with njs

Environment
Protobufjs
DNS-packet

Often, a developer wants to use 3rd-party code,
usually available as a library of some kind.
In the JavaScript world, the concept of a module is relatively new,
so there was no standard until recently.
Many platforms (browsers) still don't support modules, which makes code
reuse harder.
This article describes ways to reuse
Node.js code in njs.

Examples in this article use features that appeared in
njs
0.3.8

There is a number of issues
that may arise when 3rd-party code is added to njs:

Multiple files that reference each other and their dependencies

Platform-specific APIs

Modern standard language constructions

The good news is that such problems are not something new or specific to njs.
JavaScript developers face them daily
when trying to support multiple disparate platforms
with very different properties.
There are instruments designed to resolve the above-mentioned issues.

Multiple files that reference each other, and their dependencies

This can be solved by merging all the interdependent code into a single file.
Tools like
browserify or
webpack
accept an entire project and produce a single file containing
your code and all the dependencies.

Platform-specific APIs

You can use multiple libraries that implement such APIs
in a platform-agnostic manner (at the expense of performance, though).
Particular features can also be implemented using the
polyfill approach.

Modern standard language constructions

Such code can be transpiled:
this means performing a number of transformations
that rewrite newer language features in accordance with an older standard.
For example, babel project
can be used to this purpose.

In this guide, we will use two relatively large npm-hosted libraries:

protobufjs —
a library for creating and parsing protobuf messages used by the
gRPC protocol

dns-packet —
a library for processing DNS protocol packets

Environment

This document mostly employs a generic approach
and avoids specific best practice advices concerning Node.js
and JavaScript.
Make sure to consult the corresponding package's manual
before following the steps suggested here.

First (assuming Node.js is installed and operational), let's create an
empty project and install some dependencies;
the commands below assume we are in the working directory:

$ mkdir my_project && cd my_project
$ npx license choose_your_license_here > LICENSE
$ npx gitignore node

$ cat > package.json <<EOF
{
"name": "foobar",
"version": "0.0.1",
"description": "",
"main": "index.js",
"keywords": [],
"author": "somename <some.email@example.com> (https://example.com)",
"license": "some_license_here",
"private": true,
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
}
}
EOF
$ npm init -y
$ npm install browserify

Protobufjs

The library provides a parser
for the .proto interface definitions
and a code generator for message parsing and generation.

In this example, we will use the
helloworld.proto
file
from the gRPC examples.
Our goal is to create two messages:
HelloRequest and
HelloResponse.
We will use the
static
mode of protobufjs instead of dynamically generating classes, because
njs doesn't support adding new functions dynamically
due to security considerations.

Next, the library is installed and
the JavaScript code implementing message marshalling
is generated from the protocol definition:

$ npm install protobufjs
$ npx pbjs -t static-module helloworld.proto > static.js

Thus, the static.js file becomes our new dependency,
storing all the code we need to implement message processing.
The set_buffer() function contains code that uses the
library to create a buffer with the serialized
HelloRequest message.
The code resides in the code.js file:

var pb = require('./static.js');

// Example usage of protobuf library: prepare a buffer to send
function set_buffer(pb)
{
// set fields of gRPC payload
var payload = { name: "TestString" };

// create an object
var message = pb.helloworld.HelloRequest.create(payload);

// serialize object to buffer
var buffer = pb.helloworld.HelloRequest.encode(message).finish();

var n = buffer.length;

var frame = new Uint8Array(5 + buffer.length);

frame[0] = 0; // 'compressed' flag
frame[1] = (n & 0xFF000000) >>> 24; // length: uint32 in network byte order
frame[2] = (n & 0x00FF0000) >>> 16;
frame[3] = (n & 0x0000FF00) >>> 8;
frame[4] = (n & 0x000000FF) >>> 0;

frame.set(buffer, 5);

return frame;
}

var frame = set_buffer(pb);

To ensure it works, we execute the code using node:

$ node ./code.js
Uint8Array [
0, 0, 0, 0, 12, 10,
10, 84, 101, 115, 116, 83,
116, 114, 105, 110, 103
]

You can see that this got us a properly encoded gRPC frame.
Now let's run it with njs:

$ njs ./code.js
Thrown:
Error: Cannot find module "./static.js"
at require (native)
at main (native)

Modules are not supported, so we've received an exception.
To overcome this issue, let's use browserify
or other similar tool.

An attempt to process our existing code.js file will result
in a bunch of JS code that is supposed to run in a browser,
i.e. immediately upon loading.
This isn't something we actually want.
Instead, we want to have an exported function that
can be referenced from the nginx configuration.
This requires some wrapper code.

In this guide, we use
njs cli in all examples for the sake of simplicity.
In real life, you will be using nginx njs module to run your code.

The load.js file contains the library-loading code that
stores its handle in the global namespace:

global.hello = require('./static.js');

This code will be replaced with merged content.
Our code will be using the "global.hello" handle to access
the library.

Next, we process it with browserify
to get all dependencies into a single file:

$ npx browserify load.js -o bundle.js -d

The result is a huge file that contains all our dependencies:

(function(){function......
...
...
},{"protobufjs/minimal":9}]},{},[1])
//# sourceMappingURL..............

To get final "njs_bundle.js" file we concatenate
"bundle.js" and the following code:

// Example usage of protobuf library: prepare a buffer to send
function set_buffer(pb)
{
// set fields of gRPC payload
var payload = { name: "TestString" };

// create an object
var message = pb.helloworld.HelloRequest.create(payload);

// serialize object to buffer
var buffer = pb.helloworld.HelloRequest.encode(message).finish();

var n = buffer.length;

var frame = new Uint8Array(5 + buffer.length);

frame[0] = 0; // 'compressed' flag
frame[1] = (n & 0xFF000000) >>> 24; // length: uint32 in network byte order
frame[2] = (n & 0x00FF0000) >>> 16;
frame[3] = (n & 0x0000FF00) >>> 8;
frame[4] = (n & 0x000000FF) >>> 0;

frame.set(buffer, 5);

return frame;
}

// functions to be called from outside
function setbuf()
{
return set_buffer(global.hello);
}

// call the code
var frame = setbuf();
console.log(frame);

Let's run the file using node to make sure things still work:

$ node ./njs_bundle.js
Uint8Array [
0, 0, 0, 0, 12, 10,
10, 84, 101, 115, 116, 83,
116, 114, 105, 110, 103
]

Now let's proceed further with njs:

$ njs ./njs_bundle.js
Uint8Array [0,0,0,0,12,10,10,84,101,115,116,83,116,114,105,110,103]

The last thing will be to use njs-specific API to convert
array into byte string, so it could be usable by nginx module.
We can add the following snippet before the line
return frame; }:

if (global.njs) {
return String.bytesFrom(frame)
}

Finally, we got it working:

$ njs ./njs_bundle.js |hexdump -C
00000000 00 00 00 00 0c 0a 0a 54 65 73 74 53 74 72 69 6e |.......TestStrin|
00000010 67 0a |g.|
00000012

This is the intended result.
Response parsing can be implemented similarly:

function parse_msg(pb, msg)
{
// convert byte string into integer array
var bytes = msg.split('').map(v=>v.charCodeAt(0));

if (bytes.length < 5) {
throw 'message too short';
}

// first 5 bytes is gRPC frame (compression + length)
var head = bytes.splice(0, 5);

// ensure we have proper message length
var len = (head[1] << 24)
+ (head[2] << 16)
+ (head[3] << 8)
+ head[4];

if (len != bytes.length) {
throw 'header length mismatch';
}

// invoke protobufjs to decode message
var response = pb.helloworld.HelloReply.decode(bytes);

console.log('Reply is:' + response.message);
}

DNS-packet

This example uses a library for generation and parsing of DNS packets.
This a case worth considering because the library and its dependencies
use modern language constructions not yet supported by njs.
In turn, this requires from us an extra step: transpiling the source code.

Additional node packages are needed:

$ npm install @babel/core @babel/cli @babel/preset-env babel-loader
$ npm install webpack webpack-cli
$ npm install buffer
$ npm install dns-packet

The configuration file, webpack.config.js:

const path = require('path');

module.exports = {
entry: './load.js',
mode: 'production',
output: {
filename: 'wp_out.js',
path: path.resolve(__dirname, 'dist'),
},
optimization: {
minimize: false
},
node: {
global: true,
},
module : {
rules: [{
test: /\.m?js$$/,
exclude: /(bower_components)/,
use: {
loader: 'babel-loader',
options: {
presets: ['@babel/preset-env']
}
}
}]
}
};

Note we are using "production" mode.
In this mode webpack does not use "eval" construction
not supported by njs.
The referenced load.js file is our entry point:

global.dns = require('dns-packet')
global.Buffer = require('buffer/').Buffer

We start the same way, by producing a single file for the libraries:

$ npx browserify load.js -o bundle.js -d

Next, we process the file with webpack, which itself invokes babel:

$ npx webpack --config webpack.config.js

This command produces the dist/wp_out.js file, which is a
transpiled version of bundle.js.
We need to concatenate it with code.js
that stores our code:

function set_buffer(dnsPacket)
{
// create DNS packet bytes
var buf = dnsPacket.encode({
type: 'query',
id: 1,
flags: dnsPacket.RECURSION_DESIRED,
questions: [{
type: 'A',
name: 'google.com'
}]
})

return buf;
}

Note that in this example generated code is not wrapped into function and we
do not need to call it explicitly.
The result is in the "dist" directory:

$ cat dist/wp_out.js code.js > njs_dns_bundle.js

Let's call our code at the end of a file:

var b = set_buffer(global.dns);
console.log(b);

And execute it using node:

$ node ./njs_dns_bundle_final.js
Buffer [Uint8Array] [
0, 1, 1, 0, 0, 1, 0, 0,
0, 0, 0, 0, 6, 103, 111, 111,
103, 108, 101, 3, 99, 111, 109, 0,
0, 1, 0, 1
]

Make sure this works as expected, and then run it with njs:

$ njs ./njs_dns_bundle_final.js
Uint8Array [0,1,1,0,0,1,0,0,0,0,0,0,6,103,111,111,103,108,101,3,99,111,109,0,0,1,0,1]

The response can be parsed as follows:

function parse_response(buf)
{
var bytes = buf.split('').map(v=>v.charCodeAt(0));

var b = global.Buffer.from(bytes);

var packet = dnsPacket.decode(b);

var resolved_name = packet.answers[0].name;

// expected name is 'google.com', according to our request above
}