update generic robots.txt