Made the spider multithreaded in getting the urls.\n Added support for decentralized spider activity for the crawler
Spider running alone.Multithreaded url get working