WebCrawler uses redis now for storing and checking web urls