Add Base Robots.TXT to root

(DNN-3909) Consider Basic Robots.txt in root

Not a "bug fix" specifically or something broken, however, how about
considering adding a basic robots.txt to the root? This is an old item
that has been discussed, and if generic, could be a good addition of
basics. This one is basically the setup from the dnnsoftware.com site.

I know that the sitemap.aspx and sitemap provider are preferred, however
in every project, we add a basic starter robots.txt file, especially if
it is a single domain/portal site where we can reference the
sitemap.aspx