Fast Gumbel-Max Sketch and its Applications

The well-known Gumbel-Max Trick for sampling elements from a categorical distribution (or more generally a non-negative vector) and its variants have been widely used in areas such as machine learning and information retrieval. To sample a random element <inline-formula><tex-math notation="LaTeX">$i$</tex-math><alternatives><mml:math><mml:mi>i</mml:mi></mml:math><inline-graphic xlink:href="zhang-ieq1-3237857.gif"/></alternatives></inline-formula> in proportion to its positive weight <inline-formula><tex-math notation="LaTeX">$v_{i}$</tex-math><alternatives><mml:math><mml:msub><mml:mi>v</mml:mi><mml:mi>i</mml:mi></mml:msub></mml:math><inline-graphic xlink:href="zhang-ieq2-3237857.gif"/></alternatives></inline-formula>, the Gumbel-Max Trick first computes a Gumbel random variable <inline-formula><tex-math notation="LaTeX">$g_{i}$</tex-math><alternatives><mml:math><mml:msub><mml:mi>g</mml:mi><mml:mi>i</mml:mi></mml:msub></mml:math><inline-graphic xlink:href="zhang-ieq3-3237857.gif"/></alternatives></inline-formula> for each positive weight element <inline-formula><tex-math notation="LaTeX">$i$</tex-math><alternatives><mml:math><mml:mi>i</mml:mi></mml:math><inline-graphic xlink:href="zhang-ieq4-3237857.gif"/></alternatives></inline-formula>, and then samples the element <inline-formula><tex-math notation="LaTeX">$i$</tex-math><alternatives><mml:math><mml:mi>i</mml:mi></mml:math><inline-graphic xlink:href="zhang-ieq5-3237857.gif"/></alternatives></inline-formula> with the largest value of <inline-formula><tex-math notation="LaTeX">$g_{i}+\ln v_{i}$</tex-math><alternatives><mml:math><mml:mrow><mml:msub><mml:mi>g</mml:mi><mml:mi>i</mml:mi></mml:msub><mml:mo>+</mml:mo><mml:mo form="prefix">ln</mml:mo><mml:msub><mml:mi>v</mml:mi><mml:mi>i</mml:mi></mml:msub></mml:mrow></mml:math><inline-graphic xlink:href="zhang-ieq6-3237857.gif"/></alternatives></inline-formula>. Recently, applications including similarity estimation and weighted cardinality estimation require to generate <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives><mml:math><mml:mi>k</mml:mi></mml:math><inline-graphic xlink:href="zhang-ieq7-3237857.gif"/></alternatives></inline-formula> independent Gumbel-Max variables from high dimensional vectors. However, it is computationally expensive for a large <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives><mml:math><mml:mi>k</mml:mi></mml:math><inline-graphic xlink:href="zhang-ieq8-3237857.gif"/></alternatives></inline-formula> (e.g., hundreds or even thousands) when using the traditional Gumbel-Max Trick. To solve this problem, we propose a novel algorithm, <italic>FastGM</italic>, which reduces the time complexity from <inline-formula><tex-math notation="LaTeX">$O(kn^+)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>O</mml:mi><mml:mo>(</mml:mo><mml:mi>k</mml:mi><mml:msup><mml:mi>n</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="zhang-ieq9-3237857.gif"/></alternatives></inline-formula> to <inline-formula><tex-math notation="LaTeX">$O(k \ln k + n^+)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>O</mml:mi><mml:mo>(</mml:mo><mml:mi>k</mml:mi><mml:mo form="prefix">ln</mml:mo><mml:mi>k</mml:mi><mml:mo>+</mml:mo><mml:msup><mml:mi>n</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="zhang-ieq10-3237857.gif"/></alternatives></inline-formula>, where <inline-formula><tex-math notation="LaTeX">$n^+$</tex-math><alternatives><mml:math><mml:msup><mml:mi>n</mml:mi><mml:mo>+</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="zhang-ieq11-3237857.gif"/></alternatives></inline-formula> is the number of positive elements in the vector of interest. FastGM stops the procedure of Gumbel random variables computing for many elements, especially for those with small weights. We perform experiments on a variety of real-world datasets and the experimental results demonstrate that FastGM is orders of magnitude faster than state-of-the-art methods without sacrificing accuracy or incurring additional expenses.