Crawler refactored for better flexibility