You can tell exactly what a robot has recorded or failed to record. You can also spot robots that may be making a large number of requests, which can affect your page impression statistics or even burden your server. Those from the major search engines can sometimes be identified from their host names. These often incorporate part of the search engine's name or the company's name. A better way of spotting spiders is to look for their agent names, or what some people call browser names.
Spiders have their own names, just like browsers.
For example, Netscape identifies itself by saying Mozilla . Alta Vista's spider says Scooter , while HotBot's spider is named Slurp . A list of host and agent names for the major search engines is on the SpiderSpotting Chart , and even more in-depth information is available to site subscribers . However, it's useful to know how to spot any robot, because names can change, or new robots can appear. So, let's take a look at how to spot robots when you don't know what to look for.