GitHub repository: J300J/RobotScanner
Primary language: Python
Description: WIP you can find pages with interesting information by reading the robot.txt file of a site. # This script reaches out to a site (or list of sites) and scans the pages that the owners don't want you to look at