Added disallow dirs to robots.txt"