A Scalable Shannon Entropy Estimator

Abstract Quantified information flow (QIF) has emerged as a rigorous approach to quantitatively measure confidentiality; the information-theoretic underpinning of QIF allows the end-users to link the computed quantities with the computational effort required on the part of the adversary to gain access to desired confidential information. In this work, we focus on the estimation of Shannon entropy for a given program $$\varPi $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>Π</mml:mi> </mml:math> . As a first step, we focus on the case wherein a Boolean formula $$\varphi (X,Y)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>φ</mml:mi> <mml:mo>(</mml:mo> <mml:mi>X</mml:mi> <mml:mo>,</mml:mo> <mml:mi>Y</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> captures the relationship between inputs X and output Y of $$\varPi $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>Π</mml:mi> </mml:math> . Such formulas $$\varphi (X,Y)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>φ</mml:mi> <mml:mo>(</mml:mo> <mml:mi>X</mml:mi> <mml:mo>,</mml:mo> <mml:mi>Y</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> have the property that for every valuation to X , there exists exactly one valuation to Y such that $$\varphi $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>φ</mml:mi> </mml:math> is satisfied. The existing techniques require $$\mathcal {O}(2^m)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>O</mml:mi> <mml:mo>(</mml:mo> <mml:msup> <mml:mn>2</mml:mn> <mml:mi>m</mml:mi> </mml:msup> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> model counting queries, where $$m = |Y|$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>m</mml:mi> <mml:mo>=</mml:mo> <mml:mo>|</mml:mo> <mml:mi>Y</mml:mi> <mml:mo>|</mml:mo> </mml:mrow> </mml:math> . We propose the first efficient algorithmic technique, called $$\mathsf {EntropyEstimation}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>EntropyEstimation</mml:mi> </mml:math> to estimate the Shannon entropy of $$\varphi $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>φ</mml:mi> </mml:math> with PAC-style guarantees, i.e., the computed estimate is guaranteed to lie within a $$(1\pm \varepsilon )$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mo>(</mml:mo> <mml:mn>1</mml:mn> <mml:mo>±</mml:mo> <mml:mi>ε</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> -factor of the ground truth with confidence at least $$1-\delta $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1</mml:mn> <mml:mo>-</mml:mo> <mml:mi>δ</mml:mi> </mml:mrow> </mml:math> . Furthermore, $$\mathsf {EntropyEstimation}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>EntropyEstimation</mml:mi> </mml:math> makes only $$\mathcal {O}(\frac{min(m,n)}{\varepsilon ^2})$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>O</mml:mi> <mml:mo>(</mml:mo> <mml:mfrac> <mml:mrow> <mml:mi>m</mml:mi> <mml:mi>i</mml:mi> <mml:mi>n</mml:mi> <mml:mo>(</mml:mo> <mml:mi>m</mml:mi> <mml:mo>,</mml:mo> <mml:mi>n</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> <mml:msup> <mml:mi>ε</mml:mi> <mml:mn>2</mml:mn> </mml:msup> </mml:mfrac> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> counting and sampling queries, where $$m = |Y|$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>m</mml:mi> <mml:mo>=</mml:mo> <mml:mo>|</mml:mo> <mml:mi>Y</mml:mi> <mml:mo>|</mml:mo> </mml:mrow> </mml:math> , and $$n = |X|$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>n</mml:mi> <mml:mo>=</mml:mo> <mml:mo>|</mml:mo> <mml:mi>X</mml:mi> <mml:mo>|</mml:mo> </mml:mrow> </mml:math> , thereby achieving a significant reduction in the number of model counting queries. We demonstrate the practical efficiency of our algorithmic framework via a detailed experimental evaluation. Our evaluation demonstrates that the proposed framework scales to the formulas beyond the reach of the previously known approaches.