Toimik.RobotsProtocol (nuget): Parsers for Robots Exclusion Standard (aka robots.txt), Robots Meta Tag, and X-Robots-Tag.        Visit Project Site for documentation.
Keywords: robots.txt|robots-txt|robotstxt|robots-exclusion-standard|robots-exclusion-protocol|robots-meta-tag|x-robots-tag|robots|meta-tags|robots-parser|robots-tag