DS-1936 followup formatting fixes

Based on the comments in PR#498 , I've done some additional cleanup of the formatting of our robots.txt files.

* Cleanup formatting of robots.txt so there are no blank lines in a group record
* Minor comments changes

This work is based on the robots.txt documentation from Google at: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

These changes are very minor, so I'll merge this shortly and close the ticket:
https://jira.duraspace.org/browse/DS-1936