updating robots.txt to include Allow block per best practices.
Merge pull request #4 from cturra/master

internal bug 1026574 to support google search appliance.