RobotsSharpParser (nuget): This client library enables working with Robots.txt.       Key Features:         - Parse robots.txt into Typed object.         - Lookup Allowed/Disallowed/Crawldelay based on User-Agent.         - Traverse sitemap in robots.txt for urls.        For More info see: https://github.com/nicholasbergesen/robotsSharp/master/README.md
Keywords: robots|parse|web|crawling|robots.txt|scraping|spider|sitemap