Revert "DO NOT REVERT: Changing to new library for robots.txt to allow me to not have to manage the nrobots myself. Lib not acting as expected https://code.google.com/p/robotstxt/issues/detail?id=6"

This reverts commit e4019052b5d27d013df2b397f75b501cae384de6.