GitHub repository: openEuler-BaseService/www-robotrules
Description: A module that parses /robots.txt files