vipnytt/robotstxtparser (packagist): Robots.txt parsing library, with full support for every directive and specification.
Keywords: parser|google|W3C|spider|robots.txt|robots|web-crawler|yandex|robot|The Web Robots Pages|sitemaps.org|Robot exclusion standard|Sean Conner|Martijn Koster|Robots Exclusion Protocol|REP|robots-parser|robots-txt