One needs, in short, something like Alta Vista. Unsurprisingly, he mismanaged the company in the PC era. But even as Digital finished downsizing, Alta Vista was being developed in its research facilities in Palo Alto. A different, non-Propaedian approach to the Web, Alta Vista epitomizes what one might call the theological way of knowing. Studying sacred texts like the Bible, theologians have long felt that every word was important, because they all represented a choice by the Supreme Deity involved.
Concordances are of little use to average people.
But they are uniquely suited to computers' propensities for speed and stupidity. Computers, however, are fast enough to work in this stupid way. Many other indexers use spiders, including the popular Lycos search engine. But Alta Vista has taken its spider technology one step further. Known as Scooter, it consists of a thousand individually operating chunks of software that scuttle throughout the Web like so many robots, looking at individual pages and noting the other pages to which they refer.