Crawler refactor

Complete refactoring of the crawler