Base of Robots.txt changes

Need to allow for child directory instead of always using root level.