Right way to allow robots complete access according to http://www.robotstxt.org/robotstxt.html