Clean up old robots.txt entries and fix them going forward.

#6537