A couple of bug fixes in the crawler logic,