ensured robots.txt redirects are cached as well