- More cleanup work done on RobotsTxtParser class.