GitHub repository: KornaPhp/php-robots-txt
Primary language: PHP
Description: Determine if a page may be crawled from robots.txt, robots meta tags and robot headers