I solved the concurrency problem of the urls to crawl.  The urls now hash to a bin and are then controlled by a single crawler.  So mit.edu will always map to a single bucket and all mit sites will be crawled by the same crawler.