GitHub repository: r3k4t/pyrobotstxt
Primary language: Python
Created: 2021-03-22T05:57:03Z
Description: A simple python program which find out any website robot.txt file.