Brought crawlers in as app, some refactoring