GitHub repository: nchamsi/arachnid
Description: Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites