robots.txt, allpost, and other minimal changes