bundler/bundler — pull request #2863 by burke (2014-02-07 03:14:27+00:00)
Title: Increase API_REQUEST_LIMIT to 750.

The current behaviour of switching to the modern index when asked for >100 gems is quite wasteful.

1. `specs.4.8.gz` is fetched from S3 at 1.6MB (this part is okay).
2. Each of the 75,000 gems metadata in the gzip is [immediately instantiated as an object](https://github.com/bundler/bundler/blob/master/lib/bundler/fetcher.rb#L187-L191).
3. An [O(n^2) traversal](https://github.com/bundler/bundler/blob/master/lib/bundler/index.rb#L104-L114) is done of these 75,000 specs.

In browsing history, it looks to me like the current limit of 100 was chosen somewhat arbitrarily, and based on a very old index size (ie. O(n^2) on 10000 is a lot cheaper than O(n^2) on 75000). Steps 2 and 3 each take ~7 seconds, running on a 2.3GHz i7. Increasing this limit brings my total resolve time down to ~6 seconds.

We (Shopify) likely have one of the higher gem counts going, at ~275 in total including all indirect dependencies. I propose changing this value to at least 750. I haven't benchmarked this high, but even assuming completely linear scaling, we're still looking at 18 seconds with this change vs. 30 seconds without at 750 gems.

Related reading: #2448 and #2556.

It's entirely possible I'm misunderstanding what's going on here, or the motivation behind this `API_REQUEST_LIMIT`, but this *definitely* speeds up bundler runs substantially on our project. Discussion welcome.