Streaming SQL in Node.js
By: Forbes Lindesay
Tags: Postgres, Sqlite, Nodejs, MySQL, Sql

Sometimes you need a way to look at every row in a database table, and perform some operation on it.

SQL databases are, on the whole, great at handling massive amounts of data. I don’t think it’s an exaggeration to say that most of the companies I’ve seen who use Hadoop, would be better off just adding an index or some RAM to their Postgres/MySQL server. You can do amazing filtering and aggregation within SQL. Sometimes these powerful operations don’t support the thing you need, sometimes you’ll want to process every single row in node.js

Async Iterables

Imagine the following scenario:

We have a database containing a large number of IDs of Tweets from Twitter. At least 100s of thousands.

We want request the number of likes each tweet has from Twitter

We’re processing this as a background job, and don’t want to use too much memory.

We’re going to want to stream the results of our query to get the rows out of SQL. This example works equally well, regardless of whether you’re using @databases/pg for Postgres, @databases/mysql for MySQL or @databases/sqlite for SQLite:

// replace this with your db of choice

import connect, {sql} from '@databases/pg'; const db = connect(); export default async function updateTweets() {

const tweets = db.queryStream(

sql`SELECT id, likes FROM tweets;`

);

for await (const tweet of tweets) {

const likes = await getLikes(tweet.id);

if (likes !== tweet.likes) {

await db.query(

sql`

UPDATE tweets

SET likes = ${likes}

WHERE id = ${tweet.id};

`

);

}

}

}

This will require a very recent version of node, or a transpiler like babel, in order to support the async iterator proposal. It will stream the tweets from our database, allowing a buffer of a few tweets before it begins to tell the database to slow down with the sending of data. As soon as we have received a tweet, and finished processing the tweet before, we can begin processing it.

Node.js Streams

Imagine you need to export the entirety of a very large database table to a CSV file for processing in some other system, e.g. an analytics tool, or for importing into an accounting package. You could wait until you’ve fetched all the records, but it’s much more efficient to start sending data to the client as soon as it’s available. This especially true if they’re downloading it over a (relatively) slow internet connection.

This example works equally well, regardless of whether you’re using @databases/pg for Postgres, @databases/mysql for MySQL. It doesn’t currently support SQLite.

// replace this with your db of choice

import connect, {sql} from '@databases/pg';

import {Map} from 'barrage';

import stringify from 'csv-stringify'; const db = connect(); export default function getTweetsCSV() {

const tweets = db.queryNodeStream(

sql`SELECT id, likes FROM tweets;`

);

const map = new Map(tweet => [tweet.id, tweet.likes]);

const stringifier = stringify();

stringifier.write(['Tweet ID', 'Likes']);

tweets.pipe(map).pipe(stringifier);

tweets.on('error', e => stringifier.emit('error', e);

map.on('error', e => stringifier.emit('error', e);

return stringifier;

}

Here we request all tweets as a node.js Object Stream. We then pipe that stream into a Map stream, which takes each row, represented as an object, and returns the row in the form of an array. Finally, we pipe this mapped stream into the csv-stringifier stream, which takes an array for each line, and outputs a CSV file. We could further pipe that stream into createGzip if the client supports gzip.

Conclusion

Both streaming options add a lot of flexibility.