RobotsTxt (nuget): A robots.txt parser for .NET  Supports ; - Allow directives. - Crawl-delay directives. - Sitemap declarations. - * and $ wildcards.  See https://bitbucket.org/cagdas/robotstxt for usage examples.
Keywords: Robots.Txt|Parser|Parsing|Library