refactoring crawler configuration