GitHub repository: mozilla/srg-pre-crawl-utils
Primary language: Python
Description: Python scripts for sampling a slice of a crawl list (such as Alexa Top 1M), crawling through each site and fetching the internal links for each site.