Split site crawlers out of base file

Allows for easier swap-in of fake site crawlers, maybe as response to
certain debug input.