diggin/diggin-robotrules (packagist): parser/handler for Robots Exclusion Protocol (robots.txt and more)
Keywords: spider|crawler|robots.txt|robots|robotstxt