Added robots.txt restrictions