update config to disallow on robots.txt