DSpace/DSpace — pull request #498 by tdonohue (2014-03-04 22:38:54+00:00)
Title: DS-1936 - Update robots.txt to include a Sitemap entry

This PR does a few things:

1. Updates JSPUI & XMLUI robots.txt to include a "Sitemap" entry
2. Ensures that "Sitemap" entry is autofilled out with [dspace.url]/htmlmap (when 'ant' is run)
3. "Syncs" up the comments in the JSPUI and XMLUI robots.txt. They have gotten vastly out of sync. For this purpose, I admit to having "borrowed" from the great robots.txt at https://scholarworks.iupui.edu/robots.txt

I've tested the process and it seems to work fine. Comments welcome.