Perl stand alone robots.txt parser. With real world experience crawling the web. Googlebot like interpretation of robots.txt.