If you have sensitive information to which access must be controlled, you need expert advice. Limit what search engines index Since a very important way to find things that shouldn't be found is by use of a search engine, you can control access by the search engines. Also note that restricting access or unlinking a file will not remove it from a search engine. Even re-indexing your site may not get this file removed. They can remain for many months.
Web Robots are programs that explore the web automatically.
They are also sometimes called spiders or crawlers. The search engine indexers are web robots. One way to exclude log files, dynamic pages, and anything else you don't want indexed by specifying the directories or files in a robots.txt file in your root area. Whenever an indexing robot visits your site, it first looks for a file named robots.txt. You can exclude all pages, specific directories, or individual directories.