jmajors/robotstxt (packagist): A small package for parsing websites' robots.txt files
Keywords: php|robots.txt|crawler|disallowed-paths|robotstxt