updates robots.txt and meta robots data