github.com/samclarke/robotstxt (go): Package robotstxt parses robots.txt files  Aims to follow the Google robots.txt specification, see: https://developers.google.com/search/reference/robots_txt for more information.
Keywords: golang|robots-exclusion-standard|robots-parser|robots-txt|robotstxt