GitHub repository: righettod/robots-disallowed-dict-builder
Primary language: Python
Created: 2018-05-27T09:31:37Z
Description: Script generating a dictionary containing the most common DISALLOW clauses from robots.txt file found on CISCO Top 1 million sites