InfinityCrawler (nuget): A simple but powerful web crawler library
Keywords: crawler|robot|spider|robots-txt|web-crawler|web-crawling