JavaScript for Data Engineers
By: Kovid Rathee
Tags: Data Science, Programming, Software Development, Data Engineering, JavaScript

DATA ENGINEERING

JavaScript for Data Engineers

A brief introduction to JavaScript-based open-source SQL, orchestration and ETL tools for data engineers

The latest StackOverflow Developer survey deemed JavaScript as the most popular technology closely followed by SQL as the third most popular technology. The former was considered to be a client-side scripting/front end language until a number of years ago when JavaScript based servers got widespread attention. Since then, JavaScript projects have been initiated for almost all major areas of work in the umbrella of software development. Data Engineering is one such field where JavaScript is being used more than ever.

There are already a lot of visualisation libraries written in JavaScript such as D3.js, C3.js, Charts.js and so on. Much has been written about them but not about hardcore data engineering tools related to handling databases, data cleansing, ETL, data pipeline orchestration and so on. Let’s take a look at some of the most popular and useful active JavaScript projects that data engineers could learn and use in their current work.

It is a query builder for PostgreSQL, MySQL, MariaDB, MSSQL, Oracle and Amazon Redshift. With over 300 contributors and about 150 releases till date, it is by far the most popular query builder available today. Fun fact — the author of Slonik wrote this about why using Knex.js is bad for dynamic query building, got a lot of flak — he’s made some general good points in the following piece but NOT enough to convince me that Knex.js is bad!

A JavaScript based database for the browser that works for mobile apps, browsers and node.js applications. It’s really good at handling CSVs & Excel files. This project has 5.2K stars on GitHub and has 51K downloads/month from npm. It is being maintained pretty well with the last code push done a week ago.

There are two projects worth looking at from the Koop project — the first one is, not surprisingly, called Koop — it’s an ETL utility for geospatial data. The project is well maintained and is sponsored by ESRI which is the company to talk about when we talk about location intelligence.

Calling it a complete ETL tool would be a mistake. Firstly, it is just meant for geospatial data as the transform it supports relate to geospatial data. For example, transforming geospatial data on the fly into GeoJSON and Vector Tiles.

A component based programming environment following the principles of Flow-Based Programming (the logic of a program is defined in a graph). Many people will relate this to Airflow or similar orchestrators. While there are similarities in the sense that NoFlo can be programmed to be used somewhat like an orchestrator, it is a bit vast in scope than the popular orchestrators. You can use noflo-nodejs to execute NoFlo code on Node.js.

This is TaskRabbit’s contribution to the open source data engineering community. Empujar is an ETL tool which can be used to do a lot of moving around of data, including a creating and storing backups. Currently, Empujar has support for MySQL, Amazon Redshift, Elasticsearch and S3. Custom connectors can be created easily to include other databases or data sources. Although this tool has been around for a while, it is worth mentioning that the latest PR was not merged as the build was failing.

Honorary Mentions

GruntJS — This is a simple task runner that helps you automate your grunt work like making sure the code is formatted, linted, minified and so on.

— This is a simple task runner that helps you automate your grunt work like making sure the code is formatted, linted, minified and so on. BookshelfJS — An ORM built on Knex.js with transaction support, eager relational loading and support for 1:1, 1:n and n:n relations.

— An ORM built on Knex.js with transaction support, eager relational loading and support for 1:1, 1:n and n:n relations. ObjectionJS — An ORM for Node.js based on Knex.js which fully supports MySQL, MariaDB and PostgreSQL.

— An ORM for Node.js based on Knex.js which fully supports MySQL, MariaDB and PostgreSQL. Slonik — A Node.js based SQL client for PostgreSQL that promotes writing raw SQL and discourages ad-hoc dynamic generation of SQL.

There are many other projects on GitHub but I haven’t picked many of them because most of them are not up to date and haven’t seen a code checkin in a long time.

To conclude, we can say that there are a lot of JavaScript based, well maintained, open-source repositories to help with the day-to-day data engineerie stuff — generating SQL, interacting with databases, moving data around from one place to another, integrating data and visualising it too. With JavaScript being one of the promising languages of the future, it is worth investing in learning JavaScript as it will be more widely used further down the line.

JavaScript In Plain English

Did you know that we have three publications and a YouTube channel? Find links to everything at plainenglish.io!