Refactoring crawl method, using ConcurrentDictionary