Robots.txt are added to handle search engine indexing