GitHub repository: schollz/linkcrawler
Primary language: Go
Description: Cross-platform persistent and distributed web crawler :link: