slight refactoring to clean up the crawler