Cache robots.txt responses on client-side

Moving VCL logic to the application
Merge pull request #3240 from Wikia/robots-txt-caching

Cache robots.txt responses on client-side