This repository contains a crawler which gets the redirect urls from a set of urls