Switched to threadpool for concurrent crawling operations