refactored all crawlers into one crawler function