robots.txt change allow all / version

removed crawl delay, robot version, and request rate, added an allow all
for all agents before disallow.
Merge remote-tracking branch 'origin/appdev'