add ENGAGE_ROBOTS handling for robots.txt
Merge pull request #138 from ryanpitts/development

add ENGAGE_ROBOTS handling for robots.txt