GitHub repository: BrandwatchLtd/robots
Primary language: Java
Created: 2015-03-27T09:04:23Z
Description: Support for Robots Exclusion Protocol, including parsing and matching against robot.txt directives.