Correcting handling of robots.txt: it does not return 503 if not present, so that continues to serve requests until it is excluded by the load balancer.

Correcting a bug when index.html is not present and we are referencing a directory.