Attempting to get the robots.txt parsing to occur.  Right now it is non-functioning.