Cache robots.txt responses on client-side

Moving VCL logic to the application

Backporting #3240