refactored crawler