refactor crawler to handle html parsing on slow pages
trying to manage crawling of slow php sites