Nick.RobotsParser (nuget): https://github.com/nicholasbergesen/RobotsParser/blob/master/README.md
Keywords: robots|parse|web|crawling|robots.txt|scraping|spider|sitemap