GitHub repository: CrawlCtrl/CrawlCtrl
Primary language: C#
Description: Deserialize robots.txt files