Remove legacy urls from robots.txt