github.com/temoto/robotstxt.go (go): Package robotstxt implements the robots.txt Exclusion Protocol as specified in http://www.robotstxt.org/wc/robots.html with various extensions.
Keywords: go|go-library|golang|golang-library|production-ready|robots-txt|status-active|web