Updated robots.txt to disallow non-main pages.