Level Up Your Asynchronous JavaScript Skills by Implementing a Bluebird-Style Promise.map
By: Yong Jun
Tags: Nodejs, JavaScript, Promises, Web Scraping, Programming

Level Up Your Asynchronous JavaScript Skills by Implementing a Bluebird-Style Promise.map

Better asynchronicity

Photo by Valentin Antonucci on Unsplash.

I do a lot of web scraping with Node.js in my work. Usually, you do not want to fire all your API calls in one go, as doing so is likely to overwhelm other people’s servers, trigger their DDoS protection, or worse, take them offline.

Data scraping on the web is usually done in two steps:

You visit an index page where you find a “listing” of all the sub-resources you can call to fetch some details. Take the example of a property portal. The website might organize the property information by location or every property might have its own page. One by one, you will visit the items on the list from step 1 to fetch out the details.

It is certainly a bad idea to wrap all the step 2 API calls in a giant Promise.all . The correct approach is to use staggering and/or a rate limit. JavaScript’s Promise lends itself well to staggering because it is easy to create waterfall behavior from Promise . You simply .then all your invocations so they occur one after another. For example:

To rate limit, you can introduce a timed delay:

There is a subtle difference between the two delay implementations. The first method waits 1000ms before invoking, while the second method invokes first and waits 1000ms to deliver the result. This difference is made obvious in the diagram below:

Photo by the author.

While the waterfall pattern is useful when the list of items is small, it can be really slow on larger lists, as you are effectively running on a “single thread.” Can we do better?

The answer is yes. And it is using Promise.map , one of the Promise API extensions in bluebird . The magic here is the concurrency option.

What concurrency does is it lets you set the maximum number of “threads” at any one time. Promise.all sets no limit on the number of “threads.” If my list consists of 1,000 items, 1,000 API calls will be fired all at the same time. On the other hand, if I use the waterfall pattern shown above, I end up working on a single “thread.” Promise.map provides a middle ground.

If concurrency is such a useful option, why is it not implemented in Promise.all in the first place?

Promise.all :

Accepts an array as its only argument.

The array contains pending promises and Promise.all will resolve when every promise in the array has resolved.

Promise.map :

Accepts an array and a mapper function.

The mapper function will be invoked on every item in the array, returning a pending promise each time. When all the returned promises resolve, Promise.map resolves.

Since you do not pass in a function with Promise.all , all the pending promises must have been generated somewhere else (by invoking the API call function). So by the time Promise.all receives the collection of pending promises, all the invocations will have already started. Promise.map , on the other hand, is free to schedule invocations any way it wants since it operates on an array of inputs and the mapper function to transform input to output. With that said, it is not impossible to implement Promise.map behavior using Promise.all .

Preferring to use native Promise as much as I can, I find installing a whole bluebird module for a single function uneconomical. So I figured, “Why not write my own Promise.map ?”

The trick to implementing Promise.map with native Promise.all is to call Promise.all on not a mapped version of the input array but a smaller array containing threads that are basically copies of the waterfall pattern shown earlier. The size of this array is governed by the concurrency option. Here’s a visual representation:

And a basic implementation:

Lines 8-12 set up the concurrency threads.

threads. The inner function wrappedMapper (lines 16-23) handles populating the concurrency threads.

Every time wrappedMapper is invoked, it does the following:

Dequeues the next item in the input array. Invokes the mapper function on the input. Waits for the promise returned by the invocation to resolve. Places the resolved item in the output array (at the corresponding index). Calls itself again if there are items left in the input array.

One can see the Promise.map function is a more general implementation of the waterfall pattern. At one extreme, if concurrency is set to 1 , it behaves exactly the same as the simple waterfall pattern. If concurrency is not set, it defaults to the other extreme, which is Infinity . Then it behaves similarly to Promise.all , invoking mapper upfront on the entire input array.

The basic implementation above has two outstanding issues:

The inner function wrappedMapper assumes mapper always returns a Promise or thenable. An undefined property error will be thrown if mapper returns a non-thenable value. According to MDN web docs, Promise.all accepts an iterable — not just array. In the case of Promise.map , passing an iterable means it is possible to provide an input list that is not fixed upfront. This is especially useful for handling retries when API calls fail.

Here is a more complete implementation:

How can you ensure that a thenable is returned from mapper ? One possible way is to check for the existence of the .then method and handle thenable vs. non-thenable separately. A more robust method is to wrap the returned value in Promise.resolve . It ensures that the .then method is always present and avoids the need for type checking and separate handling.

Promise.resolve is applied on line 22.

is applied on line 22. Line 18 replaces array indexing with an iterator object.

We can use the same logic to implement Promise.filter :

That wraps up today’s article on asynchronous programming in JS. Thank you for reading!