Add multithreading to crawler.