They all work differently and, while they all share some data each has found some pages that the others have missed. Each refreshes their database periodically and some hold references for only short periods requiring active monitoring to keep a listing current. Some of the search activity to create these extensive catalogs is performed using automation. Some search information is collected by human intervention. Add to these facts that many web sites change often and many good sites fall to the wayside lost in the domain name shuffle or the inability to accommodate the size of the curious public trying to get access and you have a deep pot of boiling data.
Many links go dead before they are ever discovered.
There is a high degree of volatility to the data pool. Studying the mass of data, its structure and organization, and the tools we mine it with has become a full time occupation. Those who hold the data and those that seek that data wander in a fog trying to find one another. In many cases, specialty search engines are the answer. These are usually categorized as metasearch tools.