update list of disallows for robot.txt