GitHub repository: short-d/crawler
Primary language: Go
Description: Explore the web in parallel on thousands of machines