added ability for user to crawl websites using threads.