Robotx is a simple but powerful parser for robots.txt files.