An alternative to python's robotparser originally by Philip Semanchuk see: http://nikitathespider.com/python/rerp/