It collects the resources or documents to be indexed. A gatherer starts from a single document, usually the Web server's home directory, then selects the next document to index by following a hypertext link. In fact, it is impossible to collect every document on the Web, because that there are just too many. Furthermore, there is no guarantee that documents on the Web will be maintained for any specific period of time. Robot gatherers are used for some search systems.
Gathers locate Web severs and collect resources.
Typically Gatherers fetch documents by traversing hypertext links of a document. Also, they should have a policy for limiting the traverse. A breadth first strategy will first traverse all the hypertext links to the original document and gather the documents, then examine the gathered documents for further links to follow. The breadth first gatherer does a wide and shallow traverse. 15.2.2 Depth first The depth-first strategy will traverse one link and gather the document to index, then follow one of the links to the gathered document, and go to the depth of a path first, then to another path.