add ENGAGE_ROBOTS handling for robots.txt