add redis to do dedupe and crawl stage management