Dynamic entropy-compressed sequences and full-text indexes

We give new solutions to the Searchable Partial Sums with Indels problem. Given a sequence of <i>n</i> <i>k</i>-bit numbers, we present a structure taking <i>kn</i> + <i>o</i>(<i>kn</i>) bits of space, able of performing operations <i>sum</i>, <i>search</i>, <i>insert</i>, and <i>delete</i>, all in <i>O</i>(log <i>n</i>) worst-case time, for any <i>k</i> = <i>O</i>(log <i>n</i>). This extends previous results by Hon et al. [2003c] achieving the same space and <i>O</i>(log <i>n</i>/log log <i>n</i>) time complexities for the queries, yet offering complexities for <i>insert</i> and <i>delete</i> that are amortized and worse than ours, and supported only for <i>k</i> = <i>O</i>(1). Our result matches an existing lower bound for large values of <i>k</i>. We also give new solutions to the Dynamic Sequence problem. Given a sequence of <i>n</i> symbols in the range [1,σ] with binary zero-order entropy <i>H</i><sub>0</sub>, we present a dynamic data structure that requires <i>nH</i><sub>0</sub> + <i>o</i>(<i>n</i> log σ) bits of space, which is able of performing <i>rank</i> and <i>select</i>, as well as inserting and deleting symbols at arbitrary positions, in <i>O</i>(log <i>n</i> log σ) time. Our result is the <i>first</i> entropy-bound dynamic data structure for <i>rank</i> and <i>select</i> over general sequences. In the case σ = 2, where both previous problems coincide, we improve the dynamic solution of Hon et al. [2003c] in that we compress the sequence. The only previous result with entropy-bound space for dynamic binary sequences is by Blandford and Blelloch [2004], which has the same complexities as our structure, but does not achieve constant 1 multiplying the entropy term in the space complexity. Finally, we present a new dynamic compressed full-text self-index, for a collection of texts over an alphabet of size σ, of overall length <i>n</i> and <i>h</i>th order empirical entropy <i>H<sub>h</sub></i>. The index requires <i>nH<sub>h</sub></i> + <i>o</i>(<i>n</i> log σ) bits of space, for any <i>h</i> ≤ α log<sub>sigma</sub> <i>n</i> and constant 0 < α < 1. It can count the number of occurrences of a pattern of length <i>m</i> in time <i>O</i>(<i>m</i> log <i>n</i> log σ). Each such occurrence can be reported in <i>O</i>(log<sup>2</sup><i>n</i>log log <i>n</i>) time, and displaying a context of length ℓ from a text takes time <i>O</i>(log <i>n</i>(ℓ log σ + log <i>n</i> log log <i>n</i>)). Insertion/deletion of a text to/from the collection takes <i>O</i>(log <i>n</i> log σ) time per symbol. This significantly improves the space of a previous result by Chan et al. [2004] in exchange for a slight time complexity penalty. We achieve at the same time the <i>first</i> dynamic index requiring essentially <i>nH<sub>h</sub></i> bits of space, and the <i>first</i> construction of a compressed full-text self-index within that working space. Previous results achieve at best <i>O</i>(<i>nH<sub>h</sub></i> space with constants larger than 1 [Ferragina and Manzini 2000; Arroyuelo and Navarro 2005] and higher time complexities. An important result we prove in this paper is that the wavelet tree of the Burrows-Wheeler transform of a text, if compressed with a technique that achieves zero-order compression locally (e.g., Raman et al. [2002]), automatically achieves <i>h</i>th order entropy space for any <i>h</i>. This unforeseen relation is essential for the results of the previous paragraph, but it also derives into significant simplifications on many existing static compressed full-text self-indexes that build on wavelet trees.