Crawl a site to generate various useful documents.