Clean up old robots.txt entries and fix them going forward.

#6537 
possible, I tested on scholar where the most groups exist and it ran without issues.  It only runs through the private groups without custom domains.