This article is a description and primer for World Wide Web robots and search engines, more so it provides sample code and installation instructions. First and foremost, to effectively implement and understand WWW robots one must know something about the World Wide Web. A Robot is a program that traverses the World Wide Web, gathering some sort of information from each site it visits. This journey is accomplished by visiting a web page and then recursively visiting all or some of its linked pages.
Spiders are synonymous with Robots, as are Wanderers.
These names however, have some misleading implications. For instance many people think that a spider or wanderer leaves the home site to work its magic, when in reality it never leaves. I prefer the term Robot and will continue using it throughout this document. A search engine is not a robot. However some search engines rely heavily on robots.