File list updates and robots.txt