Cache robots.txt responses on client-side

Moving VCL logic to the application

@prein / @michalroszka