eigenhombre/mongocrawl — pull request #1 by mrocklin (2012-05-29 13:48:52+00:00)
Title: Priority

A number of major changes

1) I've removed concurrency for now and made all functions pure (except for affecting the database). We should change this back at some point but I think that we should still pass in the refs/atoms/whatever

2) We cache all git requests into a database

3) We now maintain a priority-queue of users and repos to query next. This is instead of walking out uniformly (note that this walk out uniformly behavior could be recreated by a clever selection of priority function)

4) We now write out to file

5) I've removed the previous two solutions

I'm happy to work off of this branch for a while. There is no rush to get it merged. It's mainly here to serve for conversation.