added blocked urls to robots.txt