5 Ways to Query your Relational DB using JavaScript | by Michael Bogan | Towards Data Science

If you’re developing web applications, you’re almost certainly going to be constantly interacting with a database. And when it comes time to select the way you’ll interact, the choices can be overwhelming.
In this article, we’re going to look in detail at 5 different ways to interact with your database using JavaScript, and we’ll talk about the pros and cons of each. We’ll start with the lowest-level choice — SQL Commands — then move through to higher-level abstractions.
Choosing the right database library for your JavaScript application can have a big impact on the maintainability, scalability, and performance of your code, so it’s worth spending some time to figure out your options.
We’re going to use a trivial Express application hosted on Heroku as our example. All the code for this article is in this GitHub repository. Feel free to clone it and follow along.
To run the sample application, you’ll need the following software on your machine:
A unix-like terminal environment (Mac OSX and Linux are fine. If you’re using Windows, you’ll need the Windows Subsystem for Linux).
git (and a github account).
npm (version 6 or later).
The Heroku command-line tool.
If you don’t have a Heroku account already, you’ll need to sign up for a free account. If you don’t want to sign up for Heroku, you can also run the application locally against a local Postgres instance. If you’re comfortable with that, it should be pretty easy to see what changes you need to make instead of deploying to Heroku.
Once you’ve installed all the above, run heroku login in a terminal, and you're ready to get started.
To start, we’ll set up the following:
A trivial Express application that just serves a “Hello, World” web page.
A Postgres database.
Two tables, representing “users” and “comments” (a user has many comments).
Some sample data (in this case, generated via mockaroo.com).
I’ve created a sample application which will set all this up for you (provided you’ve run heroku login as mentioned above). To set it up, please execute the following commands from the command line:
git clone https://github.com/digitalronin/query-database-javascript.git
cd query-database-javascript make setup
This will take a few minutes to complete. While you are waiting, you can view the makefile to see the relevant commands, which carry out the following:
Create a new Heroku application.
Add a Postgres database instance.
Deploy the application to Heroku.
Run a command on Heroku to set up the database tables and import the CSV sample data.
Open the URL of your Heroku application in a new browser window.
At the end of this process, you should see “Hello, World” on a web page.
OK — we’re all set up! We’ve created a database with two tables and some sample data. But we’re not doing anything with it yet. The next step is to enable our web application to retrieve data from the database.
Whenever you interact with a relational database, you do so by sending SQL commands to the network socket on which the database is listening. This is true for all the libraries we’re going to look at in this article — at the lowest level, they all send SQL commands to the database and retrieve whatever output comes back.
So, the first way we’re going to look at interacting with our database is to do just that — send SQL commands. To do this, we’re going to install the pg JavaScript library, which lets us send SQL to a Postgres database and retrieve the results.
To install the pg library, execute the following command:
npm install pg
This will fetch and install the library, and it will add it to your package.json and package-lock.json files. Let’s commit those changes:
git add package.json package-lock.json git
commit -m "Install the pg library"
To talk to our database, we need some details:
The hostname of the machine Postgres is running on.
The network port Postgres is listening on.
The name of the database our data is in.
A username and password with permission to access that data.
Most database libraries will let us establish a connection either by supplying an object to the library which has keys and values for all of those details, or by combining them all into a single “database URL”, which is what we’re going to do.
When you add a database to a Heroku application, you automatically get an environment variable called DATABASE_URL, containing all the details you need to connect to the database. You can see the value of your DATABASE_URL by running:
heroku config
This will output all the environment variables your application can use. There should only be one for now, so you should see something like this in the output:
DATABASE_URL: postgres://clqcouauvejtvw:1b079cad50f3ff9b48948f15a7fa52123bc6795b875348d66886407a266c0f5b@ec2-52-73-247-67.compute-1.amazonaws.com:5432/dfb3aad8c026in
In the case of our example, that breaks down like this:
SQL
{"hostname": "ec2-52-73-247-67.compute-1.amazonaws.com","port": 5432,"database": "dfb3aad8c026in","username": "clqcouauvejtvw","password": "1b079cad50f3ff9b48948f15a7fa52123bc6795b875348d66886407a266c0f5b"}
Your DATABASE_URL value will be different, but the structure will be the same.
Now that we have the pg library installed, and we know how to connect to our database, let’s execute our first example of interacting with a database. We’ll simply fetch the list of users and display them on our web page. At the top of our index.js file, we’ll require our pg library, and create a database connection object.
JavaScript
const { Pool } = require('pg');const conn = new Pool({ connectionString: process.env.DATABASE_URL });
In the express() block, we'll alter the get line to call a method that displays a list of users from the database:
.get('/', (req, res) => listUsers(req, res))
Finally, we’ll implement the listUsers function:
JavaScript
async function listUsers(req, res) {try {const db = await conn.connect()const result = await db.query('SELECT * FROM users');const results = { users: (result) ? result.rows : null};res.render('pages/index', results );db.release();} catch (err) {console.error(err);res.send("Error " + err);}}
This code waits until a connection is established to our database, then sends an SQL query using the query function and retrieves the result.
Now, this step could fail for lots of different reasons, so in the code we test to ensure we’ve got some data and, if we do, we assign result.rows to the key users of our results object. Next, we pass results to the render function, then release our database connection.
In views/pages/index.ejs we have access to the results object, so we can display our user data like this:
HTML
<h1>Users</h1><ul><% users.map((user) => { %><li><%= user.id %> - <%= user.first_name %> <%= user.last_name %></li><% }); %></ul>
You can see the code with these changes here. first_name and last_name are the names of two columns from the users table of our database.
Let’s deploy these changes so we can see the data in our Heroku application:
git add index.js views/pages/index.ejs
git commit -m "Display a list of users"
git push heroku master
This will take a minute or two to deploy. When that command has finished executing, reload your browser and you should see a list of users on the web page.
The above example is for Postgres, but the code for other common relational databases will be similar. For example, if you are using MySQL:
Instead of npm install pg use npm install mysql2 (use mysql2, not mysql - mysql2 is faster and supports async/await)
In index.js you would require mysql like this:
const mysql = require('mysql2/promise');
The listUsers function would look like this:
JavaScript
async function listUsers(req, res) {try {const conn = await mysql.createConnection(process.env.DATABASE_URL);const [rows, fields] = await conn.execute('SELECT * FROM users');const results = { 'users': rows };res.render('pages/index', results );await conn.end();} catch (err) {console.error(err);res.send("Error " + err);}}
views/pages/index.ejs remains the same.
You can see the sample project with these changes here.
Now let’s examine a few libraries which build on top of this foundation, adding layers of abstraction that let you read and manipulate database data in a more “JavaScript-like” way.
So far, we’ve seen how to send raw SQL to the database; statements like:
SELECT * FROM users
If we wanted to get comments by a specific user, say the user whose id is 1, we might use something like this:
SELECT * FROM comments WHERE user_id = 1
There’s nothing wrong with interacting with your database in this way, but it can feel a bit cumbersome, and it requires you to keep “shifting gears” mentally. You think about your JavaScript code in one way, but when you need to think about the data in your database, you have to start thinking in SQL.
The purpose of the rest of the database libraries we’re going to consider is to let you treat the data in your database more like the JavaScript objects and code in your application. “Under the hood” it’s all SQL, but you won’t need to care much about that unless you want to.
The first library we’ll talk about is Knex. The documentation page describes Knex as a “query builder”, and its purpose is to provide a layer of abstraction on top of raw SQL.
Knex requires pg (or MySQL if you’re using a MySQL database). We already have pg installed, so we just add knex like this:
npm install knex
git add package.json package-lock.json
git commit -m "Install the knex library"
The NPM page for knex describes it as a “query builder.” Knex abstracts SQL to a certain extent, but not very far. We still need to understand the underlying SQL, but we can write it in a more JavaScript-like syntax, rather than having to slice and dice SQL strings. More importantly, we can use composition to chain knex terms in a way that is much more comfortable for JavaScript programmers.
So, when we used pg, we had this statement:
const result = await db.query('SELECT * FROM users');
When we use knex, we can write this:
const result = await db.select().from('users');
That might not look like much of a difference, but because of the way we can compose knex function calls, we can also do things like this:
const result = await db.select().from('users').limit(5).offset(8);
Here, we’re getting 5 user records, starting at position 8 in the total set of all possible user records that match our query. You can see the full set of options available in the knex documentation.
Let’s change our Express app to use knex to display some records from our database. First, in index.js replace these two lines:
JavaScript
const { Pool } = require('pg');const conn = new Pool({ connectionString: process.env.DATABASE_URL });
...with this:
JavaScript
const db = require('knex')({client: 'pg',connection: process.env.DATABASE_URL});
Then, change the implementation of listUsers to this:
JavaScript
async function listUsers(req, res) {try {const result = await db.select().from('users').limit(5).offset(5);const results = { 'users': (result) ? result : null};res.render('pages/index', results );} catch (err) {console.error(err);res.send("Error " + err);}}
Our views/pages/index.ejs file can stay exactly the same as before.
Commit, push, and deploy:
git add index.js
git commit -m "Use knex to display user data"
git push heroku master
When you refresh your browser, you should see user records 6 to 10 on the page.
You can see the code with these changes here.
Knex gives us a way of interacting with our database, which is much more like JavaScript, but we are still required to think in a database-centric way when we need to manipulate data.
The next three libraries we’re going to talk about are all built on top of knex (which is built on top of pg or MySQL), and are examples of “object relational mapping” or ORM libraries. As the name implies, the purpose of an ORM library is to translate between data in a relational database and JavaScript objects in your application. What that means is, instead of thinking about records in the users table when you write your JavaScript code, you can think about user objects.
The first library we’ll look at is objection, which is built on top of knex:
npm install objection
git add package.json package-lock.json
git commit -m "Install the objection library"
To highlight some of the utility of ORM libraries, we’re going to alter our application to display users and their comments. Objection is built on top of knex, so in our index.js file, we have to leave the knex block in place, and add a bit more code (I’m putting everything in the index.js file, to keep things simple. In a real application you would break the code into separate files):
const { Model } = require('objection');
Model.knex(db);
This gives us a Model class from which we can inherit to define two classes User and Comment. We’ll define Comment first:
JavaScript
class Comment extends Model {static get tableName() {return 'comments';}}
Our class needs to extend Model, and must implement a tableName function to tell Objection which database table contains the underlying records.
The User class is similar, but we're going to add some behavior to our class; a fullName function, which we can use in our view template. We're also going to tell Objection that Users have Comments (i.e. a user owns zero or more comments). In ORM-speak, this is usually described as a "has many relationship" - i.e. a user has many comments. Here's what the code for this looks like:
JavaScript
class User extends Model {static get tableName() {return 'users';}fullName() {return `${this.first_name} ${this.last_name}`;}static get relationMappings() {return {comments: {relation: Model.HasManyRelation,modelClass: Comment,join: {from: 'users.id',to: 'comments.user_id'}}};}}
We define a relationMappings object inside our User class, with a single key of comments and a value telling Objection that this is a HasManyRelation on the Comment class, where the value of the id column of the users table matches the value of the user_id column of the comments table.
Now that we’ve defined our classes, let’s use them in our code. Here’s the new implementation of listUsers:
JavaScript
async function listUsers(req, res) {try {const users = await User.query().limit(5);for (i in users) {const user = users[i];user.comments = await User.relatedQuery('comments').for(user.id);}const results = { 'users': users };res.render('pages/index', results );} catch (err) {console.error(err);res.send("Error " + err);}}
Here, we fetch 5 users, then for each of those users, we fetch their comments and assign those to the comments property of our user object. In views/pages/index.ejs we can display our users and their comments like this:
HTML
<h1>Users</h1><ul><% users.map((user) => { %><li><%= user.id %> - <%= user.fullName() %></li><ul><% user.comments.map((comment) => { %><li><%= comment.body %></li><% }); %></ul><% }); %></ul>
You can see the code with these changes here. As usual, commit, and push to deploy:
git add index.js views/pages/index.ejs
git commit -m "Show users and comments using Objection"
git push heroku master
Now, when you reload the page, you should see users and comments.
This code highlights a common problem that people run into when using ORM libraries, known as the “N+1 selects” problem.
This is the block of code we used to fetch users and their comments:
JavaScript
const users = await User.query().limit(5);for (i in users) {const user = users[i];user.comments = await User.relatedQuery('comments').for(user.id);}
This works, but is very inefficient. First, we fetch 5 users, then for each of those 5 users, we fetch their comments by making another call to the database. So, we’ve made 1 call for the users, then another 5 calls to get the comments. That’s 5 calls plus the first 1, i.e. 5+1 or N+1 where N == 5. Hence the “N+1 selects” problem.
Unless your database queries are very complex, the time it takes to make a round-trip call to the database is a lot longer than the time it takes the database to calculate and transmit the results of your queries. So, to keep our applications fast, we need to minimize the number of calls to the database, as much as we can. The code above is the exact opposite of this.
For this trivial example, you won’t notice any difference, but for real-world applications, the performance hit can be very serious, and cause lots of problems.
Fortunately, every ORM library has features that make it easy to avoid this problem (provided you know it’s there). Here’s how Objection does it; in index.js, replace the code block above with this:
const users = await User.query().limit(5).withGraphFetched('comments');
This one line does the same as the code block above, but in a much more database-efficient way. Objection will use the relationship information we provided to figure out how to fetch the user data and the comments data in a single query, and unpack and stitch together the results into the same object structure we built before using our for loop.
You can see the code with these changes here.
The next ORM library we’ll look at is Bookshelf.
A lot of the differences between ORM libraries depend on what use-case the library is optimized for. In the case of Bookshelf, it’s clearly designed to make it as easy as possible to render paginated lists of data, which is a very common use case in web applications.
Let’s replace Objection with Bookshelf in our application:
npm uninstall objection
npm install bookshelf
git add package.jsonpackage-lock.json
git commit -m "Replace Objection with Bookshelf"
In index.js, replace these lines:
JavaScript
const { Model } = require('objection');Model.knex(db);
...with this:
JavaScript
const bookshelf = require('bookshelf')(db);
Replace our class definitions with these:
JavaScript
const Comment = bookshelf.model('Comment', {tableName: 'comments'});const User = bookshelf.model('User', {tableName: 'users',comments() {// by default, bookshelf infers that the foreign key is 'user_id'return this.hasMany('Comment');}});
Our listUsers function now looks like this:
JavaScript
async function listUsers(req, res) {try {const models = await new User().fetchPage({pageSize: 5,page: 1,withRelated: ['comments']});users = [];models.map(m => {const user = m.attributes;const comments = m.related('comments');user.comments = comments.map(c => c.attributes);users.push(user);});const results = { 'users': users };res.render('pages/index', results );} catch (err) {console.error(err);res.send("Error " + err);}}
As you can see, the definition of the classes is a bit more concise, but Bookshelf needs a more verbose definition of how to unpack our data to build the users/comments structure. Notice also how the concept of pages of data is built directly into the library’s API.
The code in views/pages/index.ejs is almost identical (I’ve removed the fullName function from the User class):
HTML
<h1>Users</h1><ul><% users.map((user) => { %><li><%= user.id %> - <%= user.first_name %> <%= user.last_name %></li><ul><% user.comments.map((comment) => { %><li><%= comment.body %></li><% }); %></ul><% }); %></ul>
You can see the code with these changes here. And of course, once again commit and deploy.
git add index.js views/pages/index.ejs
git commit -m "Show users and comments using Bookshelf"
git push heroku master
The last library we’re going to look at is Sequelize.
Sequelize is quite opinionated in the way it expects your data to be structured. If you follow its conventions, you can write less code and allow Sequelize to do a lot of the work for you. In particular, Sequelize has a lot of features to help create tables for you, and by default, it will create them following its own structure and naming conventions.
The database we’ve been using isn’t structured in exactly the way Sequelize expects, so we need to add a bit of extra configuration to allow Sequelize to work with it.
To remove bookshelf and install sequelize, run these commands:
npm uninstall bookshelf
npm install sequelize
git add package.json package-lock.json
git commit -m "Replace Bookshelf with Sequelize"
In index.js, replace these lines:
JavaScript
const db = require('knex')({client: 'pg',connection: process.env.DATABASE_URL});const bookshelf = require('bookshelf')(db)
...with these:
JavaScript
const { Sequelize, DataTypes } = require('sequelize');const sequelize = new Sequelize(process.env.DATABASE_URL);
Then, replace the class definitions for User and Comment with this code:
JavaScript
const User = sequelize.define('User', {first_name: { type: DataTypes.STRING },last_name: { type: DataTypes.STRING },email: { type: DataTypes.STRING }},{tableName: 'users',timestamps: false});const Comment = sequelize.define('Comment', {body: { type: DataTypes.STRING }}, {tableName: 'comments',timestamps: false});User.hasMany(Comment, { foreignKey: 'user_id' });
Note that we passed two objects to sequelize.define. The first object defined our object's properties, and the second contained some metadata.
In this case, we told Sequelize that the database table that underpins the User class is called ‘users’ (by default, Sequelize would infer that the table was called ‘Users’), and the timestamps: false tells Sequelize that our table does not have timestamp columns, called createdAt and updatedAt.
Sequelize makes it very easy to write code that will create tables for you, and when it does so, it adds these timestamp columns and sets their values accordingly when you write to the database. The sequelize documentation is excellent, and has more about this.
The foreignKey: 'user_id' that we pass to hasMany is another of the places where we have to tell Sequelize we're not following its conventions. It expects (and would create for us) a column called UserId to link comments to users.
Inside our listUsers function, we can replace all of this code:
JavaScript
const models = await new User().fetchPage({pageSize: 5,page: 1,withRelated: ['comments']});users = [];models.map(m => {const user = m.attributes;const comments = m.related('comments');user.comments = comments.map(c => c.attributes);users.push(user);});
...with this single line:
JavaScript
const users = await User.findAll({ include: Comment });
We also have to make one tiny change in views/pages/index.ejs. Replace this line:
<% user.comments.map((comment) => { %>
...with this (the difference is user.Comments instead of user.comments):
<% user.Comments.map((comment) => { %>
You can see the code with these changes here.
git add index.js views/pages/index.ejs
git commit -m "Show users and comments using Sequelize"
git push heroku master
So there you have it — 5 ways you can query a relational database from your JavaScript application. We started with raw SQL via the pg/mysql library, then looked at the knex query builder, before moving on to three ORM libraries; objection, bookshelf, and sequelize.
So, which is the right choice for your application?
As always, it depends. There’s nothing you can do with an ORM library that you can’t do using a query builder or even raw SQL. Since everything works using SQL “under the hood”. That’s not surprising. Also, even if you decide to use an ORM, most libraries still give you a way to send raw SQL to your database. So what level of abstraction you use depends on the problem you’re trying to solve, and what kind of code you want to focus your attention on.
If you’re making heavy use of the features of your database, maybe with complex views or stored procedures, you might find it easier to use knex or raw SQL. But, for most web applications it’s quite likely that an ORM library will make your life easier by abstracting away the table structure and allowing you to think about your application data as JavaScript objects.
If you’ve decided on ORM, the choice of which ORM library to use isn’t always clear-cut. The landscape of JavaScript libraries is very dynamic. New libraries are created quite often, and older ones fall out of favor. Here are a few things to think about when making your choice:
Scan through the documentation of the library, and see if it’s clear and comprehensive. Then, decide if the way the API is put together makes sense to you. Different libraries use different approaches, and you might find one of them a better fit than others for your requirements and preferences. This is especially true if you are writing code to work with an existing database, or creating your database as you develop your application.
Have a look at the community around the library. Is it something a lot of people are actively using? If so, there will probably be plenty of help and advice available if you need it. Some libraries also have extensive plugin ecosystems around them, and it may be that particular plugins make your life much easier.
A related issue is the age of the library. If it’s been around for a while, it’s more likely that common problems have been found and fixed. If it’s a relatively new library, you might have to figure more things out for yourself (which might be a good thing if you’re someone who loves to play with new, shiny toys and solve puzzles).
Performance is more likely to depend on how you use the library than on the library itself. But, if you absolutely, positively must squeeze the last few microseconds of latency out of your application, then working closer to the database using SQL or knex will be a little faster. Be warned that this is usually quite a marginal benefit, and the cost in the maintainability of your code is very likely to be higher than the gain in benchmark performance.