added noindex nofollow and robots.txt