Refactor.
Refactor for a more functional style.
Better API for crawl cache.
Helper methods for static crawl cache.
Using crawl cache to speed up result of crawling.