GitHub repository: webignition/robots-txt-parser
Primary language: PHP
Description: A parser for robots.txt files