No matter how big the hard disk, it will fill to capacity. By the end of this century large scale digital libraries will be accessed over gigabit networks. A term-level index, which records with each term the exact locations where it appears, is about the same size as the text itself since the space required to store a pointer is not very different from that needed to store a word. To do this we will survey fundamental techniques for Huffman coding, arithmetic coding, Lempel-Ziv coding, Dynamic Markov Chains, and Prediction by Partial Match. We will discuss the nasty intersection between problems of random access, processing time, memory usage, and answering queries with and without the overhead of decompression.
Digital libraries force users into an interactive dialogue and force successive query refinements.
The index is capable of identifying all documents that contain combinations of specified terms or are in some other way judged to be relevant to the set of query terms. Full-text retrieval makes information immediately accessible to the end user without requiring manual organizing or keyword selection. The index terms are drawn from the text itself. In a full-text system, such as mg, there are no fields. Information is retrieved either by specifying a Boolean combination of terms and seeking all documents that include that combination, or by specifying a list of terms and applying a heuristic to gauge the relevance of each document to the list.