added GenericCrawler and some other changes