bug fixes in crawler