refactored spider crawler