Correctly delete entries from robots.txt
Fix old entries
Handle Memory
Merge pull request #6538 from openscholar/6537

Clean up old robots.txt entries and fix them going forward.