iParadigms/walker — pull request #57 by dankinder (2014-11-03 22:50:34+00:00)
Title: Fetch manager fast shutdown

- Some refactoring here to break the fetching loop into smaller parts. Certainly something we'll keep working on modularizing
- Made the fetchers able to unclaim hosts immediately after the current fetch instead of waiting until we finish the host off. At this time it'll mean deleting links from the crawl segment that we haven't crawled yet. This does not seem like a big deal because those links will get dispatched again