tokuhirom/OrePAN2 — pull request #20 by oalders (2014-08-14 20:45:20+00:00)
Title: Look up provides via MetaCPAN

Here is an attempt to use MetaCPAN to do lookups for provides. In my case this took us from 21 minutes to regenerate the index to just under 2 minutes.

A couple of notes.

1) I haven't made using MetaCPAN a configurable option. This is just because I'm not clear on how you would like that done. If you can give me some guidance, I can set that up. Having this always enabled will no doubt annoy people.

2) What I found when using this is that a fair number of versions in my 02packages went from from numbers to undef. This is because OrePAN2 seems to assign version numbers to inner packages which do not have their own versions. So, in this case using MetaCPAN will give you results which are more correct. If someone toggles back and forth between using MetaCPAN and not using it, they'll notice the version numbers swapping in 02packages.

3) I've made do_metacpan_lookup() public so that if you're using the libraries rather than the CLI you can warm up the cache before calling add_index(). Doing the API lookups is actually slower than unpacking the tarballs in some cases. Also, just searching for everything at once is way faster as you save a huge amount of API lookups, so if you can do a bulk lookup, that's better for everyone.

I'm not married to the interface I've created here, so let me know which parts you think are terrible and we'll go from there. We are using this at MaxMind and we depend quite heavily on it, so it was worth it for us to spend a bit of time on these changes.