davidcelis/recommendable — pull request #90 by ajjahn (2014-04-23 04:33:51+00:00)
Title: Scaling/Performance with a large number of raters

With a small number of raters, refreshing similarities and recommendations is quite fast and works great. Unfortunately my system needs to support more than half a million raters. My refresh time for a single rater increases significantly with the total number of raters. Even at 10,000 raters, refreshing can take between 1-3 minutes.

I've tried tweaking (and lowering) the `config.nearest_neighbors`, `config.furthest_neighbors`, and `config.recomendations_to_store` settings without seeing much improvement.

Obviously, 1-3 minutes will only get much longer as the number of raters gets closer to half a million. So here are a few questions:

First, I'm wondering if this situation is unique to me? Is this a typical situation?

Is there a way to limit the similar raters calculation to use a subset of the total raters in the system?

Are there any other places you can suggest I could look to decrease refreshing time?

I noticed a few places in the recommendable code where Redis operations are called inside a loop; is there any reason not to `multi` or `pipeline` in those situations?

Any direction you can offer is appreciated!