SkyScraper (nuget): Web scraper / crawler / spider. Supports robots protocol and user agent.
Keywords: spider|crawler|scraper|bot