spatie/robots-txt (packagist): Determine if a page may be crawled from robots.txt and robots meta tags
Keywords: spatie|robots-txt|crawler|php