votinginfoproject/Metis — pull request #222 by tank157 (2014-08-07 20:04:51+00:00)
Title: Speed up locality calc at cost of higher mongo load

The eachSeries function won’t start the next
element in the series until the prior finishes. WI
locality calcs are taking 15-20 seconds per calc,
and has almost 2000 localities. This is the main
time sink for the WI feed (though it still takes
about an hour to even get to this point).

By using the eachLimit instead, with a limit of 20,
it will put up to 20 calcs into the event processing
loop. These are then able to talk to mongo somewhat
concurrently, increasing the load on mongo but
dramatically reducing the time it takes to run
through the entire locality set. We might need to
try other values for the limit, on my beefy
macbook pro it made the mongo process peak up to
400%.

With this fix in place, the WI feed went from 9+
hours locally to 2 hours.

Assigning to @cap10morgan