Crawler main logic bug fixes.