update robotstxt variable to hide other vsites files dirs #2797

#### Changes
* every vsite now has directives-- either disallow all (private), or
disallow other sites' files dirs (public).
* rename php vars to match variable names
* refactor to reduce identical logic checks
* comments