Refactored crawler code.