GitHub repository: robotshell/robotScraper
Primary language: Python
Description: RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.