changed function to modify robots.txt