You have the option to Allow or Disallow crawlers to explore the pages of your website. Ensure you have indicated specifically the rules for crawling your website.