Perf improvements from profiling; reduce impact of blocked main thread; use set intersection instead of some for already-crawled? check, reduce unnecessary lookups by smarter boolean logic in crawl?