GitHub repository: schollz/crawdad
Primary language: Go
Description: Cross-platform persistent and distributed web crawler :crab: