Welcome to MOMspider's Web. Given that a means for automating the traversal process is desired, we need to define the requirements and limitations of such a solution. The primary requirement is that it improve the existing maintenance process by reducing the detrimental effects of human inattentiveness, duplication of effort, and distributed document ownership.
Manual traversal is both time-consuming and boring.
In addition, their only method for testing a link is to actually request and transfer the document contents. Fortunately, these are the characteristics for which automation is most effective. Provided that document metainformation is available in the response headers, such a program can also check for special conditions that would interest the infostructure owner, such as a recent Last-modified date or an approaching Expires date. Furthermore, the program can restrict its focus to the web's structure and not be distracted by the contents of each document. With manual traversal, duplication of effort occurs because different infostructure owners don't see the results of others' traversals.