HalloJoe.RobotsTxt (nuget): A tool for accessing and querying the contents of a robots.txt file.
Keywords: robots.txt|robotstxt