updatetd robots.txt to exclude directories