Added robots.txt file, only allowing it to crawl the index.html page as the rest are not complete.