GitHub repository: xmsec/RobotsDisallowed
Description: A harvest of the Disallowed directories from the robots.txt files of the world's top websites.