Fast and Simple Compact Hashing via Bucketing

Abstract Compact hash tables store a set S of n key-value pairs, where the keys are from the universe $$U = \{0,\ldots ,u-1\}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>U</mml:mi> <mml:mo>=</mml:mo> <mml:mo>{</mml:mo> <mml:mn>0</mml:mn> <mml:mo>,</mml:mo> <mml:mo>…</mml:mo> <mml:mo>,</mml:mo> <mml:mi>u</mml:mi> <mml:mo>-</mml:mo> <mml:mn>1</mml:mn> <mml:mo>}</mml:mo> </mml:mrow> </mml:math> , and the values are $$v$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>v</mml:mi> </mml:math> -bit integers, in close to $${{\mathcal {B}}(u, n)} + nv$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mrow> <mml:mi>B</mml:mi> <mml:mo>(</mml:mo> <mml:mi>u</mml:mi> <mml:mo>,</mml:mo> <mml:mi>n</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> <mml:mo>+</mml:mo> <mml:mi>n</mml:mi> <mml:mi>v</mml:mi> </mml:mrow> </mml:math> bits of space, where $${{\mathcal {B}}(u, n)} = \log _2 {{u} \atopwithdelims (){n}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mrow> <mml:mi>B</mml:mi> <mml:mo>(</mml:mo> <mml:mi>u</mml:mi> <mml:mo>,</mml:mo> <mml:mi>n</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> <mml:mo>=</mml:mo> <mml:msub> <mml:mo>log</mml:mo> <mml:mn>2</mml:mn> </mml:msub> <mml:mfenced> <mml:mfrac> <mml:mi>u</mml:mi> <mml:mi>n</mml:mi> </mml:mfrac> </mml:mfenced> </mml:mrow> </mml:math> is the information-theoretic lower bound for representing the set of keys in S , and support operations insert, delete and lookup on S . Compact hash tables have received significant attention in recent years, and approaches dating back to Cleary [IEEE T. Comput, 1984], as well as more recent ones have been implemented and used in a number of applications. However, the wins on space usage of these approaches are outweighed by their slowness relative to conventional hash tables. In this paper, we demonstrate that compact hash tables based upon a simple idea of bucketing practically outperform existing compact hash table implementations in terms of memory usage and construction time, and existing fast hash table implementations in terms of memory usage (and sometimes also in terms of construction time), while having competitive query times. A related notion is that of a compact hash ID map , which stores a set $${\hat{S}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>S</mml:mi> <mml:mo>^</mml:mo> </mml:mover> </mml:math> of n keys from U , and implicitly associates each key in $${\hat{S}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>S</mml:mi> <mml:mo>^</mml:mo> </mml:mover> </mml:math> with a unique value (its ID), chosen by the data structure itself, which is an integer of magnitude O ( n ), and supports inserts and lookups on $${\hat{S}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mover> <mml:mi>S</mml:mi> <mml:mo>^</mml:mo> </mml:mover> </mml:math> , while using space close to $${{\mathcal {B}}(u,n)}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>B</mml:mi> <mml:mo>(</mml:mo> <mml:mi>u</mml:mi> <mml:mo>,</mml:mo> <mml:mi>n</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> bits. One of our approaches is suitable for use as a compact hash ID map.