update robotstxt variable to hide other vsites files dirs #2797

#### Changes
* every vsite now has directives-- either disallow all (private), or
disallow other sites' files dirs (public).
* rename php vars to match variable names
* refactor to reduce identical logic checks
* comments
account for shared domains and ksort directives array #2797
refactor & update comments #2797
rename $sid to $gid #2797
Merge pull request #2928 from openscholar/2797-domain-robots

update robotstxt variable to hide other vsites files dirs #2797