You should also submit your site to the searchengines of your country, this means if your site meets the submission criteria of these engines Step 5 . In some countrylists it becomes obvious that searchengines use services of other searchengines. But also Ask Jeeves, Excite, Google, Looksmart, Lycos and Fast are important sources. Make sure you get listed in these toplevel databases. A robot is a program that automatically follows the internet's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
Robots have similar functionality as normal web browsers.
Robots do this without any interaction. Web robots are sometimes referred to as Web Wanderers, Web Crawlers, or Spiders. You can include multiple disallow lines in the robots.txt file. The page title We recommend that you first read the Hints and Tips for searching searchengines , before you continue. We think you would like to see your website ranked in the top 10 of the results in case somebody looks for certain keywords.