Activate robots.txt for Deltacloud servers - DTACLOUD-440

https://issues.apache.org/jira/browse/DTACLOUD-440