GitHub repository: KlemenVovk/robots-txt-analyzer
Primary language: Java
Created: 2019-01-13T20:43:09Z
Description: A Java application that analyzes the robots.txt file if available and outputs scraping/crawling options.