SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underset {\alpha_ {i} \in \mathbb {R} ^ {| C |}} {\operatorname {a r g m i n}} \left\{\sum_ {n = 1} ^ {T} \left(\mathbb {I} (i = d _ {n}) \left(\sum_ {S, \chi_ {S} \in C} \alpha_ {i, S} \chi_ {S} (x _ {n}) - f (x _ {n})\right)\right) ^ {2} + \lambda \| \alpha_ {i} \| _ {1} \right\}. \tag {8}
$$
This theorem illuminates the inherent trade-offs in designing interpretable models. Specifically, to reduce  $\mathbb{I}_{\mathrm{total}}$ , one would typically need to increase  $\mathbb{D}_{\mathrm{total}}$  unless  $f$  itself is close to an interpretable model  $\bar{g}$ . Theorem 4 can also be seen as a quantitative version of the Impossible Trinity Theorem 1 for removal-based explanations.
Now we established the lower bound of  $\mathbb{I}_{\mathrm{total}} + \mathbb{D}_{\mathrm{total}}$  by  $\mathbb{I}_2(f,g^\dagger)$  (the same as  $\mathbb{D}_2(f,g^\dagger)$ ), showing the trade-off between efficiency and consistency, and also bridging the locally consistent interpretations  $g_{i}$  and globally consistent interpretation  $g^{\dagger}$ .
4 Trade-off Between Efficiency and Consistency
Harmonica recovers functions across the entire function space. However, in our setting, if  $\mathcal{N}_x$  is small, it is sufficient to recover a small neighborhood of the function. This inspires us to apply Harmonica to a local space instead of the entire space. By doing so, we obtain more concentrated samples in the local neighborhood. Consequently, minimizing the interpretation error in this neighborhood becomes more manageable. We refer to Harmonica with samples in the local neighborhood as Harmonica-local.
If we relax the consistency requirement, meaning that users are willing to accept minor inconsistencies in interpretations when a slight modification is made to the input, we can achieve smaller interpretation error with Harmonica-anchor, as shown in Algorithm 2. Intuitively, Harmonic-anchor applies multiple Harmonica-local algorithms to different subspaces of the input. Specifically, for a given neighborhood region  $\mathcal{N}x$ , we now randomly select  $k_{x}$  bases  $b_{x,i}$ ,  $(i = 1,2,3,\dots,k_x)$  as interpretation anchors instead of only one basis as Harmonica does. We calculate  $k_{x}$  interpretation models  $g_{x,i}$  on each anchor and denote the coefficient of  $g_{x,i}$  as  $\alpha_{x,i}$ . For simplicity, we omit the subscript  $x$  in the following.
The number of bases  $k$  is highly related to consistency, so we could define the log value  $\log k$  to evaluate the inconsistency of the interpretation models. The minimum inconsistency value is 0, as in Harmonica  $(k = 1)$ , while for attribution methods, the inconsistency should be almost  $n$ , which is the number of input variables. The maximum inconsistency value depends on the neighborhood region  $\mathcal{N}_x$ , since the number of interpretation models should not exceed the number of points in  $\mathcal{N}_x$ . As a special case, the Harmonica algorithm achieves 0 inconsistency, making it the most consistent algorithm.
However, if one only considers the number of different interpretation algorithms, users may still encounter a high level of inconsistency empirically when the interpretations significantly differ from one another. Thus, another critical constraint to add is the spectrum distance constraint among different interpretations of different anchors. Based on Theorem 4, we introduce Harmonica-anchor-constrained in Algorithm 3 (in Appendix). In this algorithm,  $\lambda_{2}$  serves as a penalty coefficient for the difference between  $g_{i}$ , and we solve the problem using iterative gradient descent. Intuitively, a larger
$\lambda_{2}$  restricts the expressive power of the interpretation models  $g_{i}$ , leading to a larger interpretation error.
5.1 Analysis on Polynomial Functions
To investigate the performance of different interpretation methods, we manually examine the output of various algorithms including LIME [47], SHAP [38], Shapley Interaction Index [45], Shapley Taylor [61, 26], Faith-SHAP [62], Harmonica, and Low-degree (Appendix D) for lower-order polynomial functions.
We observe that all algorithms can accurately learn the coefficients of the first-order polynomial. For the second-order polynomial function, only Shapley Taylor, Faith-SHAP, Harmonica, and Low-degree can learn all the coefficients accurately. For the third-order polynomial function, only Faith-SHAP, Harmonica, and Low-degree succeed. Due to space constraints, we defer the details to Appendix F.
In the rest of this section, we conduct experiments to evaluate the interpretation error  $\mathbb{I}_{p,\mathcal{N}_x}(f,g)$  and truthful gap  $\mathbb{T}_{V_C}(f,g)$  of Harmonica and other baseline algorithms on language and vision tasks quantitatively. In our experiments, we choose 2nd order and 3rd order Harmonica algorithms, which correspond to setting  $C$  to include all terms with order at most 2 and 3.
The baseline algorithms chosen for comparison include LIME [47], Integrated Gradients [60], SHAP [38], Integrated Hessians [30], Shapley Taylor interaction index, and Faith-SHAP, where the first three are first-order algorithms, and the last three are second-order algorithms.
The two language tasks we select are the SST-2 [55] dataset for sentiment analysis and the IMDb [40] dataset for movie review classification. The vision task is the ImageNet [33] for image classification. To demonstrate the capability of our interpretation framework applied to vision tasks, we have generated two examples in Figure 2, compared with LIME, Integrated Gradients (IG), and SHAP. Note that all of these methods are applied to the same ground-truth image segmentation provided by the MS-COCO dataset [35]. For ablations on using the SLIC superpixels [2], please refer to Appendix H.
Figure 2: Illustrative examples for applying our interpretation method on MS-COCO dataset.
For the SST-2 dataset, we attempt to interpret a convolutional neural network (see details in Appendix G) trained with the Adam [32] optimizer for 10 epochs. The IMDb dataset contains long paragraphs, and each paragraph has multiple sentences. By default, we use periods, colons, and exclamations to separate sentences. For the ImageNet [33] dataset, we aim to provide class-specific interpretation, meaning that only the class with the maximum predicted probability is considered for each sample. We use the official pre-trained ResNet-101 [28] model from PyTorch.
Figure 3: Visualization of  $L^2$  interpretation error  $\mathbb{I}_{p,\mathcal{N}_x}(f,g)$  of several state-of-the-art interpretation methods evaluated on IMDb and ImageNet datasets.
5.3 Results on Interpretation Error
For a given input sentence  $x$  with length  $l$ , we define the induced neighborhood  $\mathcal{N}_x$  by introducing a masking operation on this sentence. The radius  $0 \leq r \leq l$  is defined as the maximum number of masked words.
Figure 3 displays the  $L^2$  interpretation error evaluated under different neighborhoods with a radius ranging from 1 to  $\infty$  for all the considered datasets. Here,  $\infty$  represents the maximum sentence length, which may vary for different data points. We also inspect  $L^1$  and  $L^0$  norms. Here,  $L^2$  and  $L^1$  are defined according to Eqn. (2) with  $p = 2$  and  $p = 1$ , respectively. And  $L^0$  denotes  $\int_{\mathcal{X}} \mathbb{1}|f(x) - g(x)| \geq 0.1d\mu(x)$ . We can see that Harmonica consistently outperforms all the other baselines on all radii.
Note that for IMDb in Figure 3, we make a slight modification such that the masking operation is performed on sentences in one input paragraph instead (we also change the definition of radii accordingly). For ImageNet in Figure 3, the interpretation error is evaluated on 1000 random images, and the masking operation is performed on 16 superpixels in one input image instead (we also change the definition of radii accordingly). We can see that when the neighborhood's radius is greater than 1, Harmonica outperforms all the other baselines. Specifically, when evaluated on ImageNet dataset, Harmonica-anchor  $(k = 9)$  achieves 31.8 times lower interpretation error  $\mathbb{I}_{2,\mathcal{N}_1}$  compare to Integrated Gradients [60]. Limited by space, the detailed numerical results and interpretation error under other norms are presented in Appendix I. In contrast, Harmonica-local achieves more accurate local consistency but high error outside the neighborhood. The relevant results are deferred in Appendix J due to limited space.
As mentioned in Section 4, Harmonica-anchor further reduces the interpretation error by learning each subspace separately. Figure 3 shows that the  $L_{2}$  interpretation error of Harmonica-anchor achieves lower error than Harmonica on various datasets, and the error further declines as we increase the number of anchors. Full results of other norms and the results of Harmonica-anchor-constrained are shown in Appendix L.
Table 1: Comparison of  $C^3$  truthful gap  $\mathbb{T}_C(f,g)$  results evaluated on IMDb and ImageNet datasets (lower truthful gap means better truthfulness of the interpretation).
Method	Harm. 2nd	Harm. 3rd	LIME	SHAP	IG	IH	Shapley Taylor	Faith-Shap
IMDb	0.540	0.174	5.343	1.390	1.438	1.948	7.005	15.528
ImageNet	0.660	0.246	0.738	2.023	1.848	175.368	0.474	0.430
5.4 Results on Truthful Gap
For convenience, we define the set of bases  $C^d$  up to degree  $d$  as  $C^d = \{\chi_S | S \subseteq [n], |S| \leq d\}$ . We evaluate the truthful gap on the set of bases  $C^3$ ,  $C^2$ , and  $C^1$ . For implementation details on
Calculating the truthful gap, please refer to Appendix K. Table 1 shows the  $C^3$  truthful gap evaluated on different datasets. We can see that Harmonica outperforms all the other baseline algorithms. Due to space limitations,  $C^2$  and  $C^1$  results are provided in Appendix K.
5.5 More Experimental Results
For more experimental results on different image segmentation methods, different neural network architectures, and different choices of baselines, please refer to Appendix H. For more results on the Low-degree algorithm, please refer to Appendix M.
Interpretability is a critical topic in machine learning, and we refer the reader to [20, 37] for insightful general discussions. Below we discuss different types of interpretable models.
Removal-based explanations. Covert et al. [18] presents a unified framework for removal-based model explanation methods, connecting 26 existing techniques such as LIME [47], SHAP [38], Meaningful Perturbations [21], and permutation tests [10]. LIME [47] is a classical method that samples data points following a predefined sampling distribution and computes a function that empirically satisfies local fidelity.
The Shapley value [50, 72, 24] originates from cooperative game theory and is used to allocate the total value generated by a group of players among individual players. It is the unique kind of method that satisfies a few important properties, including efficiency, symmetry, dummy, additivity, etc. Shapley values have been extensively applied to machine learning model explanations [38, 39, 57, 59, 67, 78, 23, 77] and feature importance [17]. Recent research has focused on developing efficient approximation methods for Shapley values [38, 15, 5, 16, 31, 26, 68]. Many works have generalized Shapley values to higher-order feature interactions [45, 24, 61, 41, 63, 1, 62].
Gradient-based explanations. Gradient-based explanations are popular for deep learning models, such as CNNs. These methods include SmoothGrad, Integrated Gradient, GradCAM, DeepLift, LRP, etc. [53, 54, 52, 60, 74, 49, 6, 51, 13, 42, 52, 48]. Although these methods have seen extensive use in various areas, gradient-based methods are often time-consuming and can be insensitive to random model parameterization [3].
Model-specific interpretable models. Interpretable or transparent (white-box) models are inherently ante-hoc and model-specific. One primary goal of utilizing interpretable models is to achieve inherent model interpretability. Prominent approaches include Decision Trees ([66, 7, 75]), Decision Rules ([69, 58]), Decision Sets ([34, 70]), and Linear Models ([64, 65]). Moreover, another research direction attempts to use an interpretable model surrogate to approximate the original black-box models. [14] employs a two-layer additive risk model for interpreting credit risk assessments. [8] suggests an approach called model extraction that greedily learns a decision tree to approximate  $f$ . However, these methods primarily rely on heuristics and lack theoretical guarantees.
In this paper, we tackled the problem of generating consistent interpretations and introduced the impossible trinity theorem of interpretability, under a formal framework for understanding the interplay between interpretability, consistency, and efficiency. Since a consistent interpretation cannot be efficient, we relaxed efficiency to truthfulness, meaning the interpretation matches the target function in a specific subspace. This led to the problem of learning Boolean functions and the proposal of new algorithms based on the Fourier spectrum and a localized version of Harmonica. Our methods showed lower interpretation errors and improved consistency compared to the existing approaches.
While our work offers theoretical insights, many open questions and challenges remain in building more interpretable, consistent, and efficient models. We hope our work serves as a foundation for future research in the area of explainable AI.
Thanks to Jiaye Teng for the useful discussions. This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0150000).
[1] Kjersti Aas, Martin Jullum, and Anders Løland. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artificial Intelligence, 298: 103502, 2021.
[2] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süssstrunk. Slic superpixels. Technical report, EPFL, 2010.
[3] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
[4] Zeyuan Allen-Zhu and Yang Yuan. Improved svrg for non-strongly-convex or sum-of-non-convex objectives. In International Conference on Machine Learning, pp. 1080–1089. PMLR, 2016.
[5] Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In International Conference on Machine Learning, pp. 272-281. PMLR, 2019.
[6] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. *PloS one*, 10(7):e0130140, 2015.
[7] Randall Balestriero. Neural decision trees. ArXiv, abs/1702.07360, 2017.
[8] Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpretability via model extraction. ArXiv, abs/1706.09773, 2017.
[9] William Beckner. Inequalities in fourier analysis. Annals of Mathematics, 102(1):159-182, 1975.
[10] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
[11] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
[13] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 782-791, 2021.
[14] Chaofan Chen, Kangcheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang, and Tong Wang. An interpretable model with globally consistent explanations for credit risk. ArXiv, abs/1811.12615, 2018.
[15] Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. ArXiv, abs/1808.02610, 2019.
[16] Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics, pp. 3457-3465. PMLR, 2021.
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[74] Shawn Xu, Subhashini Venugopalan, and Mukund Sundararajan. Attribution in scale and space. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9677-9686. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.00970.
[75] Yongxin Yang, Irene Garcia Morillo, and Timothy M. Hospedales. Deep neural decision trees. ArXiv, abs/1806.06988, 2018.
[76] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
[77] Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. On explainability of graph neural networks via subgraph explorations. In International Conference on Machine Learning, pp. 12241-12252. PMLR, 2021.
[78] Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10877-10886, 2021.
[79] Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. arXiv preprint arXiv:2308.04371, 2023.
Algorithm 3: Harmonica-anchor-constrained
Input: anchor number  $k$ , model  $f$ , a distance metric function  $d(\cdot, \cdot): (\chi_S, \chi_S) \to \mathbb{R}$  which calculates distance between two anchors (e.g., Hamming distance or  $L_p$  norm), sampling number  $T$ , loss balance coefficient  $\lambda_1, \lambda_2$ , update epochs  $E$ , update rate  $\eta$
Output: interpretation models  $g_{i}$  with coefficients  $\alpha_{i}, i = 1,2,3,\ldots,k$ .
1: Fix random bases  $b_{i}, i = 1,2,3,\dots,k$  as interpretation anchors.
2: Randomly initialize  $\alpha_{i}\in \mathbb{R}^{|C|},i = 1,2,3,\dots,k.$
3: Randomly sample  $T$  bases  $b_{1}, b_{2}, \ldots, b_{T}$  and accordingly, we calculate  $f(x_{1}), f(x_{2}), \ldots, f(x_{T})$ .
4: Assign each basis to the anchor with minimal distance and index with  $d_{i} = \min \{\arg \min_{j}d(b_{i},b_{j}^{x})\} ,i = 1,2,3,\ldots T$  and  $j = 1,2,3,\dots,k$ .
5: for  $e = 1,2,3,\dots ,E$  do
$$
\begin{array}{l} 6: \quad L _ {f} = \frac {1}{T} \sum_ {i = 1} ^ {T} \left(\sum_ {S, \chi_ {S} \in C} \alpha_ {d _ {i}, S} \chi_ {S} (x _ {i}) - f (x _ {i})\right) ^ {2} \\ 7: \quad L _ {r} = \frac {1}{k} \sum_ {i = 1} ^ {k} \| \alpha_ {i} \| _ {1}. \\ 8: \quad L _ {c} = \frac {1}{k} \sum_ {i = 1} ^ {k} \| \alpha_ {i} - \frac {1}{k} \sum_ {j = 1} ^ {k} \alpha_ {j} \| _ {2}. \\ 9: \quad \text {U p d a t e} \alpha_ {i} \text {w i t h} \alpha_ {i} \leftarrow \alpha_ {i} - \eta \frac {\partial \left(L _ {f} + \lambda_ {1} L _ {r} + \lambda_ {2} L _ {c}\right)}{\partial \alpha_ {i}}. \\ \end{array}
$$
A Fourier Analysis of Boolean Function
Fourier analysis of Boolean function is a fascinating field, and we refer the reader to [43] for a more comprehensive introduction. We define the inner product as follows.
Definition A.1 (Inner product). Given two functions  $f, g \in \{-1, 1\}^n \to \mathbb{R}$ , their inner product is:
$$
\langle f, g \rangle \triangleq \underset {x \sim \{- 1, 1 \} ^ {n}} {\mathbb {E}} [ f (x) g (x) ] = 2 ^ {- n} \sum_ {x \in \{- 1, 1 \} ^ {n}} f (x) g (x). \tag {9}
$$
In addition, we define the induced norm  $\| f\| _2\triangleq \sqrt{\langle f,f\rangle}$ , and more generally, for  $p > 0$
$$
\| f \| _ {p} \triangleq \mathbb {E} [ | f (x) | ^ {p} ] ^ {1 / p}. \tag {10}
$$
The inner product defines one kind of similarity between two functions and is invariant under different basis. Specifically, we have the following Theorem.
Definition A.2 (Plancherel's Theorem). Given two functions  $f, g \in \{-1, 1\}^n \to \mathbb{R}$ ,
$$
\langle f, g \rangle = \sum_ {S \subseteq [ n ]} \hat {f} _ {S} \hat {g} _ {S}.
$$
When setting  $f = g$ , we get the Parseval's identity:  $\mathbb{E}[f^2] = \sum_S \hat{f}_S^2$ .
A distribution over a discrete domain  $S$  is often represented as a non-negative function  $f: S \to \mathbb{R}^+$  which is normalized in  $L_1$ , i.e.,  $\sum_{x \in S} f(x) = 1$ .
Definition A.3 (Fourier  $p$ -norm). For any Boolean function  $f \in \{-1, 1\}^n \to \mathbb{R}$ , we define the Fourier (or spectral)  $p$ -norm of  $f$  as
$$
\left\| f \right\| _ {p} \triangleq \left(\sum_ {S \subseteq [ n ]} | \hat {f} _ {S} | ^ {p}\right) ^ {1 / p}. \tag {11}
$$
Definition A.4 (Fourier  $p$ -distance). For any Boolean function  $f, g \in \{-1, 1\}^n \to \mathbb{R}$ , we define the Fourier (or spectral)  $p$ -distance between  $f$  and  $g$  as
$$
\left. \left| \right| f - g \right| _ {p} \triangleq \left(\sum_ {S \subseteq [ n ]} \left| \hat {f} _ {S} - \hat {g} _ {S} \right| ^ {p}\right) ^ {1 / p}. \tag {12}
$$
Pseudo Code for Harmonica-anchor-constrained
Using Fourier  $p$ -norm, we could rephrase Parseval's identity as:  $\| f \|_2 = \left\| f \right\|_2$ .
Proof. Pick  $f \in \mathcal{F} \setminus \mathcal{G}$ . If  $\mathcal{A}$  is consistent with respect to  $f$ , let  $g = \mathcal{A}(f, x) \in \mathcal{G}$  for any  $x \in \mathcal{X}$ . If for every  $x \in \mathcal{X}$ ,  $g(x) = f(x)$ , we know  $g = f \notin \mathcal{G}$ , this is a contradiction. Therefore, there exists  $x \in \mathcal{X}$  such that  $g(x) \neq f(x)$ .
Proof of Lemma 3.4.
Proof. Denote the complement space as  $V_{\bar{C}}$ . We may expand  $f, g, v$  on both bases and get:
$$
\begin{array}{l} \| f - g \| ^ {2} - \inf  _ {v \in V} \| f - v \| _ {2} = \sum_ {S \in C} \langle f - g, \chi_ {S} \rangle^ {2} + \\ \sum_ {S \in \bar {C}} \langle f, \chi_ {S} \rangle^ {2} - \inf  _ {v \in V _ {C}} \left(\sum_ {S \in C} \langle f - v, \chi_ {S} \rangle^ {2} + \sum_ {S \in \bar {C}} \langle f, \chi_ {S} \rangle^ {2}\right) \\ = \sum_ {S \in C} \left\langle f - g, \chi_ {S} \right\rangle^ {2} - \inf  _ {v \in V _ {C}} \left(\sum_ {S \in C} \left\langle f - v, \chi_ {S} \right\rangle^ {2}\right) \\ = \sum_ {S \in C} \left\langle f - g, \chi_ {S} \right\rangle^ {2}, \\ \end{array}
$$
where the last equality holds because we can set the Fourier coefficients  $\hat{v}_S = \hat{f}_S$  for every  $S\in C$  which further gives  $\langle f - v,\chi_S\rangle = 0$
Proof. Taking  $\mu$  as the uniform distribution, then  $\mathbb{I}_0(f,g) = \frac{\sum_{x\in\{-1,1\}^n}\mathbf{1}_{f(x)}\neq g(x)}{2^n} = \frac{|\mathrm{supp}(f - g)|}{2^n}$ . Note  $\mathbb{D}_0(f,g) = \sum_{S\subseteq [n]}\mathbf{1}_{\hat{f} (S)\neq \hat{g} (S)} = |\mathrm{supp}(\hat{f} -\hat{g})|$ . The conclusion follows from Proposition C.1.
Proof of Proposition 3.6.
Proof. From the definition of  $\mathbb{I}_2(f,g)$  and  $\mathbb{D}_2(f,g)$ . By Parseval's identity,  $\mathbb{I}_2^2 (f,g) = 2^{-n}\sum_{x\in \{-1,1\} ^n}(f(x) - g(x))^2 = \sum_{S\subset [n]}(\hat{f} (S) - \hat{g} (S))^2 = \mathbb{D}_2^2 (f,g)$ .
Proof. From Proposition 3.6 (Parseval's identity), we have (notice that our functions  $g_{i}$  are defined on  $\mathcal{X} = \{-1,1\}^n$  globally and then restricted to  $\mathcal{X}_i \subseteq \mathcal{X}$ ).