added disallow directives to robots.txt