Moved crawler into its own file, added tests (needs more for the edge cases)