modify Robots to take care of disallowed paths