add changed directories to robots.txt and .htaccess