handle crawl requests

quickly, accurately, and minimalistically