GitHub repository: pandzel/RobotsTxt-HTTP
Primary language: Java
Description: Apache HttpClient wrapper with ability to process and apply robots.txt file directives.