Fix the issues with the robots.txt warning for the status code changed.