Although the term spiders has been used interchangeably with robots by many people, Nwana says that spiders are not agents. This is confusing as it seems to be contradictory. At its inception it was possible for individuals to index the contents of the Internet, and to provide subject guides to where the available information on a topic could be found. The use of Internet robots is one way to remedy this problem. As I mentioned above, an Internet robot is a type of Intelligent agent.
Internet robot is a term used to describe a software program that traverses the Internet.
A robot goes to a site and downloads its contents. Once it has done this, it follows the links found in this site to others, and repeats the process. According to Martijn Koster, there are a number of ways a robot will begin its search. Another way is from submissions of URLs to indexing services. Finally a robot may use postings from mailing lists and news services.