Title: Tight Bounds and Phase Transitions for Incremental and Dynamic Retrieval
Domain: Computer Science

Question: Determining the optimal redundancy $R$ for retrieval data structures in the incremental and dynamic settings when the universe size is polynomial, i.e., $|\mathcal{U}| = \poly(n)$.

Background: Retrieval data structures are designed to answer key-value queries without explicitly storing the keys, which helps save space. The problem of optimizing these data structures can be formulated in four settings: static, value-dynamic, incremental, and dynamic. Each setting offers different levels of dynamism to the user. The static setting supports only queries, the value-dynamic setting supports queries and updates, the incremental setting supports queries, updates, and insertions, and the dynamic setting supports all operations including deletions. | | Previous research has established optimal bounds for the static and value-dynamic settings. For the static setting, minimal perfect hashing can achieve a redundancy of $R = \Theta(n)$, and further improvements have shown that the optimal redundancy can be $o(n)$ with $O(1)$ query time. For the value-dynamic setting, a recent result established a tight bound of $R = \Theta(n)$. | | The incremental and dynamic settings have also been studied, with the state-of-the-art upper bound achieving a redundancy of $R = \Theta(n \log \log n)$. However, there has been an open question about whether the optimal redundancy could shrink as the value size $v$ increases, especially in the incremental setting.

Definitions: - **Retrieval Data Structure**: | - A data structure that answers key-value queries without storing keys explicitly. | - Given a set of \( n \) keys \( K \subseteq \mathcal{U} \) and a \( v \)-bit value \( f(k) \) for each key \( k \in K \), it supports queries of the form: | \[ | \texttt{Query}(k) = | \begin{cases} | f(k) & \text{if } k \in K \\ | \text{anything} & \text{otherwise} | \end{cases} | \] | | - **Four Settings**: | - **Static**: Supports only queries. | - **Value-dynamic**: Supports queries and updates. | - **Incremental**: Supports queries, updates, and insertions. | - **Dynamic**: Supports all operations including deletions. | | - **Redundancy**: | - If a solution uses space \( nv + R \), it is said to have redundancy \( R \). | - The goal is to determine the optimal redundancy \( R \) for each setting. | | - **Minimal Perfect Hashing**: | - A technique to achieve redundancy \( R = \Theta(n) \) in the static setting. | - Further improvements have shown redundancy can be \( o(n) \) with \( O(1) \) query time. | | - **Phase Transition**: | - A phenomenon where the optimal space redundancy changes significantly as the value size \( v \) approaches \( \log n \). | - In the incremental setting, redundancy decreases from \( \Theta(n \log \log n) \) to \( \Theta(n) \). | | - **Perfect Universe Reducing Hash Function**: | - A hash function \( h: [U_1] \rightarrow [U_2] \) for a hash table \( H \) on universe \( U_1 \) such that for every pair of keys \( k_1, k_2 \) in \( H \), \( h(k_1) \neq h(k_2) \).

Methodology: **Conduct a Technical Overview of the Upper Bound Results**: | - Observe that for an incremental retrieval data structure with a target space usage of \( S \) bits (\( S \geq n \cdot v \)), up to \( \ell \leq n \) key-value pairs inserted so far, it is possible to store an additional \( \leq \frac{S - \ell \cdot v}{\ell} \) bits of information per inserted key. | - Utilize these extra bits to reduce the collision rate for the earlier inserted items. As more key-value pairs are inserted, demonstrate how to implement a universe reduction technique to decrease the amount of extra information stored per key, in order to avoid exceeding the maximum space of \( S \) bits. | - Ensure that this universe reduction step, which occurs after the items have been inserted, does not introduce any new collisions. By performing this universe-reduction step \( O(\log^*(n)) \) times, gracefully adjust the collision rate over time to achieve the upper bound of space usage. | | **Achieve Efficient Time Complexity**: | - Tackle the challenge of efficiently finding a valid universe-reducing hash function over large universes by bucketing the keys and representing the universe-reducing hash function properly, employing lookup table techniques to bypass the efficiency barrier. | - Specifically, partition the fingerprint space \([n \log^2 n]\) into \( n/B \) buckets (\( B := (\log \log n)^{10} \)), and use two \( B \)-wise independent hash functions \( h_{\text{buk}}: U \rightarrow [n/B] \) and \( h_1: U \rightarrow [B \log^2 n] \) to map each key to a pair \((b, f)\), where \( b \) indicates the bucket the key belongs to, and \( f \) is the fingerprint of length \((\log B + 2 \log \log n)\) bits within that bucket. This approach reduces the fingerprint size to \((\log B + 2 \log \log n)\) bits. | - By doing so, reduce the effective size of the universes being dealt with to sub-logarithmic, thereby enabling the use of explicit lookup tables for finding valid universe reductions. | | **Address the Issue of Small Bucket Sizes**: | - When a newly inserted key is mapped to a bucket already containing \( 4B \) keys, insert the key into the collision set instead. Since each bucket is limited to at most \( 4B \) keys (fingerprints), describe the set of fingerprints in each bucket using only \( O(B \cdot t_j) \) bits, which can be encoded within \( O(1) \) words and manipulated efficiently using lookup tables.

Answer: | Answer (direct) | | For a polynomial universe |\mathcal U|=\mathrm{poly}(n), the optimal redundancy R:=S-nv for retrieval data structures is: | • Incremental setting (insert-only): | R_{\text{inc}}=\Theta\!\Big(n\;+\;n\cdot\max\{0,\;\log(\tfrac{\log n}{v})\}\Big). | Equivalently: | • If v\ge c\log n (for a constant c>0), then R_{\text{inc}}=\Theta(n). | • If v=\log n/\log\log n, then R_{\text{inc}}=\Theta(n\log\log\log n). | • If v=\log^{0.99}n, then R_{\text{inc}}=\Theta(n\log\log n). | These bounds are tight: there is an incremental structure with | S\le nv+O(n)+O\!\left(n\log\!\left(\tfrac{\log n}{v}\right)\right) | and a matching lower bound | S\ge nv+\Omega(n)+\Omega\!\left(n\log\!\left(\tfrac{\log n}{v}\right)\right) | \quad (\text{for }|\mathcal U|\ge n^3), | giving the phase transition around v\asymp \log n. Timewise, the construction achieves O(1) queries/updates and O(1) amortized expected insertions. | • Dynamic setting (insertions & deletions): | R_{\text{dyn}}=\Theta(n\log\log n). | Specifically, any dynamic retrieval DS needs | S\ge nv+\Omega(n\log\log n)\quad (|\mathcal U|\ge n^3), | and this is achievable up to constants via succinct dynamic hashing (e.g., using the provided dynamic table with parameter t=2, yielding | S\le nv+O(n\log\log n) and O(1) expected update/query time). | | So, the answer your draft conveyed is right on the incremental side (tight upper+lower) and gives the correct dynamic lower bound; to be fully complete, make explicit that the dynamic upper bound S\le nv+O(n\log\log n) is attainable, hence R_{\text{dyn}}=\Theta(n\log\log n).

Checklist: 1. Setup
   - States |U| = poly(n).
   - Defines redundancy R := S - nv.

2. Incremental formula
   - Gives R_inc = Θ(n + n * max{0, log(log n / v)}).
   - Mentions phase transition at v ≍ log n.

3. Incremental cases
   - v ≥ c log n ⇒ R_inc = Θ(n).
   - v = log n / log log n ⇒ R_inc = Θ(n log log log n).
   - v = log^0.99 n ⇒ R_inc = Θ(n log log n).

4. Incremental bounds
   - Upper bound: S ≤ nv + O(n) + O(n log(log n / v)).
   - Lower bound: S ≥ nv + Ω(n) + Ω(n log(log n / v)), for |U| ≥ n^3.

5. Incremental performance
   - Achieves O(1) query and update.
   - Achieves O(1) amortized insertion.

6. Dynamic formula
   - Gives R_dyn = Θ(n log log n).
   - Requires |U| ≥ n^3.

7. Dynamic bounds
   - Lower bound: S ≥ nv + Ω(n log log n).
   - Upper bound: S ≤ nv + O(n log log n).

8. Dynamic performance
   - Achieves O(1) expected query.
   - Achieves O(1) expected update.

9. Completeness
   - Incremental: bounds are tight.
   - Dynamic: both bounds matched, confirming Θ(n log log n).