Test the shit out of the crawler using online tests. Fuck mocking these bastards.