updating robots.txt to include Allow block per best practices.