Cache robots.txt responses on client-side

Moving VCL logic to the application
Merge pull request #3248 from Wikia/release-217-robots-txt-caching

Cache robots.txt responses on client-side