GitHub repository: nicholasbergesen/RobotsParser
Primary language: C#
Description: Parse robots.txt and traverse sitemaps.