GitHub repository: DrakeES/RobotsTxt
Primary language: PHP
Created: 2015-09-21T12:14:09Z
Description: A simple class for parsing robots.txt files and telling whether certain paths are allowed for certain user agents