TurnerSoftware.RobotsExclusionTools (nuget): Robots.txt parsing and querying utility
Keywords: utilities|robots|norobots-rfc|parse|parser|robots-txt|user-agent