com.github.itechbear:robotstxt (maven): A java clone of Google's robotst.txt parser: https://github.com/google/robotstxt
Keywords: crawler|google-robotst-parser|java|robotstxt