GitHub repository: scrapy/protego
Primary language: DIGITAL Command Language
Description: A pure-Python robots.txt parser with support for modern conventions.