crawler refactoring