GitHub repository: nickmoline/robots-checker
Primary language: PHP
Description: Library to check a url for all possible robots exclusion protocols (robots.txt, http X-Robots-Tag header, meta robots tag, or http status codes)