GitHub repository: mdornseif/verBOTen
Primary language: Python
Description: large-scale analysys of robots.txt