You can block all bots from indexing your site by using the below commands: User-agent:* Disallow: / These lines will prevent any crawler from indexing any part of your site. Whether you prevent certain pages or the entire site is up to you.