github.com/hvlck/robots (go): robots.txt parser
Keywords: noctis|robots-txt