Clean up old robots.txt entries and fix them going forward.

#6537 
couldn't you run yourself out of memory with this?