github.com/puerkitobio/gocrawl (go): Package gocrawl is a polite, slim and concurrent web crawler written in Go.
Keywords: crawler|robots-txt