Lack of effectiveness appears to have three causes. First, collection selection based on word histograms is not appropriate for heterogeneous collections. Second, relevant documents are scattered over many collections and searching a few collections misses many relevant documents. Third, most existing collection selection metrics lack sound theoretical justifications and hence may not be well tuned to the problem. We propose a new approach to distributed retrieval based on document clustering and language modeling.
Document clustering is used to organize collections around topics.
For documents on the Web, the databases in different special purpose search engines are natural clusters. The graph only includes citing articles where the year of publication is known.