Add option to dynamically serve a /robots.txt

I added a --robots or -r option to dynamically respond to /robots.txt. This allows you to expose you http-server instance without the risk of having your folders indexed by a search engine crawler.