Added initial support for multithreaded crawling