DO NOT REVERT: Changing to new library for robots.txt to allow me to not have to manage the nrobots myself. Lib not acting as expected https://code.google.com/p/robotstxt/issues/detail?id=6