tomverran/robots-txt-checker (packagist): Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
Keywords: crawler|robots.txt