github.com/jimsmart/grobotstxt (go): Package grobotstxt is a Go port of Google's robots.txt parser and matcher C++ library.  See: https://github.com/google/robotstxt
Keywords: go|robots-exclusion-protocol|robots-txt