Simplify robots.txt file

This simplifies the robots file. Before, each instance disallowed the cached urls. This didn't work perfectly behind the ELB's.

Using this approach, the ELB has no influence anymore.