goddamnyouryan/coven-api — pull request #3 by antstorm (2014-12-19 21:06:38+00:00)
Title: Using Typhoeus::Hydra to speed up HackerNews sync by executing parallel requests

HackerNews allows you to fetch a single post at a time, which is painfully slow, so I decided to speed it up.

So instead of executing requests one by one, I've added `Typhoeus::Hydra` (since `Typhoeus` is already used in the project) to start multiple requests in parallel.

Here are some performance tests:

1: bundle exec rake posts:sync 2.55s user 0.48s system 2% cpu 1:43.32 total
2: bundle exec rake posts:sync 2.50s user 0.47s system 5% cpu 52.591 total
4: bundle exec rake posts:sync 2.47s user 0.47s system 7% cpu 38.781 total
8: bundle exec rake posts:sync 2.42s user 0.45s system 13% cpu 21.417 total
16: bundle exec rake posts:sync 2.30s user 0.43s system 20% cpu 13.444 total

These tests can not be taken too seriously, since my internet connection is not that great or stable, but a positive trend is very obvious.

I've kept the `CONCURRENCY` constant to 8, because it seems to be the most optimal CPU/time wise.