Height in a digital search tree and the longest phrase of the Lempel-Ziv scheme

We study the height of a digital search tree (DST in short) built from n random strings generated by an unbiased memoryless source (i.e., all symbols are equally likely). We shall argue that the height of such a tree is equivalent to the length of the longest phrase in the Lempel-Ziv parsing scheme that partitions a random sequence into n phrases. We shall identify four natural regions of the height distribution and characterize them asymptotically for large n. In particular, for the region where most of the probability mass is concentrated, the asymptotic distribution of the height exhibits an exponential of a Gaussian distribution (with an oscillating term) around the most probable value kl = [log 2 n + 2V/~-22 n log2 ( 2V/2~2 n ) + ~ I 1⁄2] +1. More precisely, we shall prove that the asymptotic distr ibution of a digital search tree is either concentrated on the one point kx or the two points kl 1 and kl, which actually proves (slightly modified) Kesten's conjecture quoted in [1]. We derive these results by a combination of analytic methods such as generating functions, Laplace transform, the saddle point method and ideas of applied mathematics such as linearization, asymptotic matching and the WKB method. 1 I n t r o d u c t i o n The heart of some universal da ta compression schemes is the parsing algorithm due to Lempel and Ziv [29]. It partitions a sequence into phrases (b locks)of variable sizes such that a new block is the shortest substring not seen in the past as a phrase. For example, the string 110010100010001000 is parsed into (1)(10)(0)(101)(00)(01)(000)(100). This parsing algori thm plays a crucial role in numerous applications such as efficient transmission of data discriminating between ~ p o r t e d by DOE Grant DE-FG02-96ER25168 and NSF Grants NCR-9415491 and CCR-9804760. Dept. Mathematics, Statistics & Computer Science, University of Illinois at Chicago, Chicago, Illinois 60607-7045. SDepartment of Computer Science, Purdue University, W. Lafayette, IN 47907. information sources, test of randomness, estimating the statistical model of individual sequences, and so forth. The parameters of interest in these applications are: the number of phrases, the number of phrases of a given size, the size of a phrase, the length of a sequence built from a given number of phrases, the length of the longest phrase, etc. Some of them have already been analyzed (e.g., number of phrases [1, 10], the size of a typical phrase [12, 20]). Here we study the distribution of the longest phrase. The Lempel-Ziv parsing scheme can be efficiently implemented by a special digital tree called the digital search tree (in short DST). This finds a myriad of applications in computer science and telecommunications such as dynamic hashing, partial match retrieval of multidimensional data, searching and sorting, pattern matching, conflict resolution algorithms for broadcast communications, da ta compression, coding, security, genes searching, DNA sequencing, genome maps, and so forth. The digital search tree is constructed as follows. We consider n, possibly infinite, strings of symbols from a finite alphabet E (however, for the simplicity we work only with the binary alphabet ~ = {0, 1}). The empty string is stored in the root, while the first string occupies the right or the left child of the root depending whether its first symbol is 'T ' or "0". The remaining strings are stored in available nodes (that are directly at tached to nodes already existing in the tree). The search for an available node follows the prefix structure of a string. The rule is simple: if the next symbol in a string is "1" we move to the right, otherwise we move to the left. The resulting tree has n + 1 internal nodes (cf. Figure 1). Details can be found in [14] and [21]. The parsing scheme according to Lempel-Ziv with a fixed number, n, of phrases (cf. [8, 10, 20]) can be accomplished on the associated digital search tree as follows: We consider an infinite sequence of binary symbols and parti t ion it until we create the first n phrases. Assuming the first phrase is an empty one, we store it in the root of a digital search tree, and all other phrases are stored in internal nodes. When a new phrase is created, the search starts at the root