SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: a system for large-scale machine learning. In Osdi, volume 16, pages 265–283. Savannah, GA, USA, 2016.
[2] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Susstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274-2282, 2012.
[3] Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. In International Conference on Machine Learning, pages 110-119. PMLR, 2021.
[4] Chunrong Ai and Edward C Norton. Interaction terms in logit and probit models. Economics letters, 80(1):123-129, 2003.
[5] David Alvarez-Melis and Tommi S. Jaakkola. On the robustness of interpretability methods. CoRR, abs/1806.08049, 2018.
[6] Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, pages 1-42, 2022.
[7] Randall Balestriero and Richard Baraniuk. Mad max: Affine spline insights into deep learning, 2018. URL https://arxiv.org/abs/1805.06576.
[8] Naman Bansal, Chirag Agarwal, and Anh Nguyen. SAM: The Sensitivity of Attribution Methods to Hyperparameters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8673-8683, 2020.
[9] Mahsa Bazzaz and Seth Cooper. Active learning for classifying 2d grid-based level completeness. arXiv preprint arXiv:2309.04367, 2023.
[10] Andrew Beam and Ben Kompa. Second opinion needed: Communicating uncertainty in medical artificial intelligence. NPJ Digital Medicine, 4, 2021.
[11] Sandeep Bodduluri, Arie Nakhmani, Joseph M. Reinhardt, Carla G. Wilson, Merry-Lynn McDonald, Ramaraju Rudraraju, Byron C. Jaeger, Nirav R. Bhakta, Peter J. Castaldi, Frank C. Sciarba, Chengcui Zhang, Purushotham V. Bangalore, and Surya P. Bhatt. Deep neural network analyses of spirometry for structural phenotyping of chronic obstructive pulmonary disease. JCI Insight, 5(13), July 2020. doi: 10.1172/jci.insight.132781.
[12] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[13] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
[14] Boris Chen, Amir Ziai, Rebecca Tucker, and Yuchen Xie. Match cutting: Finding cuts with smooth visual transitions. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2114-2124, 2022. URL https://api-semanticscholar.org/CorpusID:252846493.
[82] Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The Shapley Taylor Interaction Index. In Proceedings of the 37th International Conference on Machine Learning, pages 9259-9268. PMLR, November 2020.
[83] Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2): 26-31, 2012.
[84] Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In International Conference on Learning Representations, 2018.
[85] Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detection. In International Conference on Learning Representations, 2019.
[86] Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attribution for feature interactions. Advances in neural information processing systems, 33:6147-6159, 2020.
[87] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010.
[88] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[89] Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. Advances in Neural Information Processing Systems, 33:13623-13634, 2020.
[90] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL http:// arxiv.org/abs/1708.07747.
[91] Taedong Yun, Justin Cosentino, Babak Behsaz, Zachary R McCaw, Davin Hill, Robert Luben, Dongbing Lai, John Bates, Howard Yang, Tae-Hwi Schwantes-An, Anthony P Khawaja, Andrew Carroll, Brian D Hobbs, Michael H Cho, Cory Y McLean, and Farhad Hormozdiari. Unsupervised representation learning improves genomic discovery for lung function and respiratory disease prediction. medRxiv, 2023. doi: 10.1101/2023.04.28.23289285. URL https://www.medrxiv.org/content/early/2023/04/29/2023.04.28.23289285.
[92] Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. In The AAAI Conference on Artificial Intelligence (AAAI), 2021.
[93] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[94] Jingyi Zhu. Hessian estimation via stein's identity in black-box problems. arXiv preprint arXiv:2104.01317, 2021.
Machine learning models have become increasingly pervasive in society, from medicine [10, 67, 15] and law [38] to entertainment[14, 9, 80]. Therefore, it is important that users of the technology understand the factors underlying model predictions. To this end we propose SmoothHess for quantifying the feature interactions affecting model output. Potential applications of our method are widespread; SmoothHess may be used to find interactions influencing a model to predict whether a customer will default on a credit loan or if a patient has melanoma. The deeper understanding of the model gleaned from SmoothHess may be used to improve decision making. For instance, a doctor may notice that the SmoothHess feature interactions between the pixels in an image of a lesion don't "make sense", indicating that the model should not be trusted and that a more granular human assessment is required. However, such applications are highly sensitive and the cost of inaccurate predictions, or, in this case, misinterpreted attributions, can be high. An inaccurate interpretation can instill unwarranted trust, or mistrust, in a user. In the most extreme cases this may lead to sub-optimal decision making such as the confident denial of a credit loan to a trustworthy customer or misdiagnosis of a benign lesion as malignant.
In addition, it is important to note the challenges related to ensuring robust, stable, and trustworthy explanations. In particular, recent works have uncovered issues related to the sensitivity of the explainer to small changes in the input [5, 41], adversarial attacks [27, 20, 37], or hyperparameter tuning [8]. Methods have been proposed that attempt to quantify explanation uncertainty [75, 69, 35], however further challenges remain. Thus, as with all methods for explaining machine learning model predictions, we recommend that SmoothHess is used in tandem with the careful consideration of domain experts, who are best equipped to interpret interactions in the context of their field.
We make use of a Lemma from Lin et al. [49] in our proof below, which we relate here:
Lemma 3. (Lin et al. [49]) Denote  $x_0 \in \mathbb{R}^d$ , locally-Lipschitz continuous function  $h(z): \mathbb{R}^d \to \mathbb{R}$ , covariance matrix  $\Sigma \in \mathbb{R}^{d \times d}$ , random vector  $z \in \mathbb{R}^d$  distributed from  $z \sim \mathcal{N}(x_0, \Sigma)$ . If  $\mathbb{E}_z[|h(z)|] < \infty$ , then
$$
\mathbb {E} _ {z} \left[ \Sigma^ {- 1} \left(\left(z - x _ {0}\right) \left(z - x _ {0}\right) ^ {T} - \Sigma\right) \Sigma^ {- 1} h (z) \right] = \mathbb {E} _ {z} \left[ \Sigma^ {- 1} \left(z - x _ {0}\right) \left[ \nabla_ {z} h (z) \right] ^ {T} \right]. \tag {13}
$$
Proof. We define random vector  $z = x_0 + \delta \in \mathbb{R}^d$  distributed from  $z \sim \mathcal{N}(x_0, \Sigma)$ , and for which we denote the density function as  $p_\Sigma$ . We begin by showing  $\mathbb{E}_z[|g(z)|] = \mathbb{E}_\delta[|g(x_0 + \delta)|] < \infty$ . We are given that  $g$  is  $L$ -Lipschitz, i.e.  $\forall a, b \in \mathbb{R}^d$
$$
\left| g (a) - g (b) \right| \leq L \| a - b \| _ {2} \tag {14}
$$
for some  $L > 0$
Now, for any fixed  $\delta \in \mathbb{R}^d$  we have
$$
| g (x _ {0} + \delta) | = | g (x _ {0}) + (g (x _ {0} + \delta) - g (x _ {0})) | \triangleq \text {i n e q .} \leq | g (x _ {0}) | + | g (x _ {0} + \delta) - g (x _ {0}) | \stackrel {{E q. 1 4}} {\leq} \tag {15a}
$$
$$
\left| g \left(x _ {0}\right) \right| + L \| x _ {0} + \delta - x _ {0} \| _ {2} = \left| g \left(x _ {0}\right) \right| + L \| \delta \| _ {2}. \tag {15b}
$$
Thus, we may bound the expectation  $\mathbb{E}_{\delta}[|g(x_0 + \delta)|]$ :
$$
\mathbb {E} _ {\delta} [ | g (x _ {0} + \delta) | ] \leq \mathbb {E} _ {\delta} [ | g (x _ {0}) | + L \| \delta \| _ {2} ] = g (x _ {0}) + L \mathbb {E} _ {\delta} [ \| \delta \| _ {2} ] <   \infty . \tag {16}
$$
Here  $g(x_0) < \infty$  as it is a constant, and it may be seen that  $L\mathbb{E}_{\delta}[\| \delta \| _2] < \infty$  using a simple change of variables. Defining  $\beta = \Sigma^{-\frac{1}{2}}\delta \sim \mathcal{N}(0,I)$  one may write  $\mathbb{E}_{\delta}[\| \delta \| _2] = \mathbb{E}_{\beta}[\| \Sigma^{\frac{1}{2}}\beta \| _2]$ . As a straightforward consequence of Cauchy-Schwarz, for any fixed  $\beta$ , one may write
$$
\left\| \Sigma^ {\frac {1}{2}} \beta \right\| _ {2} \leq \left\| \Sigma^ {\frac {1}{2}} \right\| _ {F} \| \beta \| _ {2} \tag {17}
$$
where  $\| \cdot \| _F$  denotes the Frobenius norm. Noting that  $\| \beta \| _2\sim \mathcal{X}_d$  we use Eq. (17) to see
$$
\mathbb {E} _ {\beta} \left[ \| \Sigma^ {\frac {1}{2}} \beta \| _ {2} \right] \leq \mathbb {E} _ {\beta} \left[ \| \Sigma^ {\frac {1}{2}} \| _ {F} \| \beta \| _ {2} \right] = \| \Sigma^ {\frac {1}{2}} \| _ {F} \mathbb {E} _ {\beta} \left[ \| \beta \| _ {2} \right] = \| \Sigma^ {\frac {1}{2}} \| _ {F} \sqrt {2} \frac {\Gamma ((d + 1) / 2)}{\Gamma (d / 2)} <   \infty \tag {18}
$$
where  $\Gamma (\cdot)$  denotes the Gamma function.
Next, we move the Hessian operator inside the integral.
$$
\nabla_ {x} ^ {2} \left(g * q _ {\Sigma}\right) \left(x _ {0}\right) = \nabla_ {x} ^ {2} \int_ {z \in \mathbb {R} ^ {d}} g (z) q _ {\Sigma} \left(z - x _ {0}\right) d z = \int_ {z \in \mathbb {R} ^ {d}} g (z) \nabla_ {x} ^ {2} q _ {\Sigma} \left(z - x _ {0}\right) d z = \tag {19a}
$$
$$
\int_ {z \in \mathbb {R} ^ {d}} g (z) \nabla_ {x} \nabla_ {x} ^ {T} q _ {\Sigma} (z - x _ {0}) d z = \int_ {z \in \mathbb {R} ^ {d}} g (z) \left(\nabla_ {x} q _ {\Sigma} (z - x _ {0}) (z - x _ {0}) ^ {T} \Sigma^ {- 1}\right) d z = \tag {19b}
$$
$$
\int_ {z \in \mathbb {R} ^ {d}} g (z) \left(\left(\nabla_ {x} (z - x _ {0}) ^ {T} \Sigma^ {- 1}\right) q _ {\Sigma} (z - x _ {0}) + \left(\nabla_ {x} q _ {\Sigma} (z - x _ {0})\right) (z - x _ {0}) ^ {T} \Sigma^ {- 1}\right) d z = \tag {19c}
$$
$$
\int_ {z \in \mathbb {R} ^ {d}} g (z) \left(- I \Sigma^ {- 1} q _ {\Sigma} \left(z - x _ {0}\right) + \left(\nabla_ {x} q _ {\Sigma} \left(z - x _ {0}\right)\right) \left(z - x _ {0}\right) ^ {T} \Sigma^ {- 1}\right) d z = \tag {19d}
$$
$$
\int_ {z \in \mathbb {R} ^ {d}} g (z) \left(- I \Sigma^ {- 1} q _ {\Sigma} \left(z - x _ {0}\right) + q _ {\Sigma} \left(z - x _ {0}\right) \Sigma^ {- 1} \left(z - x _ {0}\right) \left(z - x _ {0}\right) ^ {T} \Sigma^ {- 1}\right) d z = \tag {19e}
$$
$$
\int_ {z \in \mathbb {R} ^ {d}} g (z) q _ {\Sigma} (z - x _ {0}) \left(- I \Sigma^ {- 1} + \Sigma^ {- 1} \left(z - x _ {0}\right) \left(z - x _ {0}\right) ^ {T} \Sigma^ {- 1}\right) d z = \tag {19f}
$$
$$
\int_ {z \in \mathbb {R} ^ {d}} g (z) p _ {\Sigma} (z) \left(- I \Sigma^ {- 1} + \Sigma^ {- 1} \left(z - x _ {0}\right) \left(z - x _ {0}\right) ^ {T} \Sigma^ {- 1}\right) d z = \quad \left(p _ {\Sigma} (z) = q _ {\Sigma} \left(z - x _ {0}\right)\right) (1 9 \mathrm {g})
$$
$$
\mathbb {E} _ {z} [ g (z) (- I \Sigma^ {- 1} + \Sigma^ {- 1} (z - x _ {0}) (z - x _ {0}) ^ {T} \Sigma^ {- 1}) ] = \tag {19h}
$$
$$
\mathbb {E} _ {z} [ g (z) \Sigma^ {- 1} (- I + (z - x _ {0}) (z - x _ {0}) ^ {T} \Sigma^ {- 1}) ] = \tag {19i}
$$
$$
\mathbb {E} _ {z} \left[ g (z) \Sigma^ {- 1} \left(- \Sigma + \left(z - x _ {0}\right) \left(z - x _ {0}\right) ^ {T}\right) \Sigma^ {- 1} \right] = \tag {19j}
$$
$$
\mathbb {E} _ {z} \left[ \Sigma^ {- 1} \left(\left(z - x _ {0}\right) \left(z - x _ {0}\right) ^ {T} - \Sigma\right) \Sigma^ {- 1} g (z) \right] \tag {19k}
$$
Lemma 3 may be applied as  $g$  is Lipschitz, and thus locally-Lipschitz, and  $\mathbb{E}_z[|g(z)|] < \infty$ , yielding
$$
\mathbb {E} _ {z} \left[ \Sigma^ {- 1} \left(\left(z - x _ {0}\right) \left(z - x _ {0}\right) ^ {T} - \Sigma\right) \Sigma^ {- 1} g (z) \right] = \mathbb {E} _ {z} \left[ \Sigma^ {- 1} \left(z - x _ {0}\right) \left[ \nabla_ {z} g (z) \right] ^ {T} \right] \tag {20}
$$
Using a change of variables from  $z$  to  $x_0 + \delta$  we write
$$
\mathbb {E} _ {z} \left[ \Sigma^ {- 1} (z - x _ {0}) \left[ \nabla_ {z} g (z) \right] ^ {T} \right] = \mathbb {E} _ {\delta} \left[ \Sigma^ {- 1} \delta \left[ \nabla_ {x} g \left(x _ {0} + \delta\right) \right] ^ {T} \right] \tag {21}
$$
which, when combined with Eq. (19) and Eq. (20), completes the proof
$$
\nabla_ {x} ^ {2} \left(g * q _ {\Sigma}\right) \left(x _ {0}\right) = \mathbb {E} _ {\delta} \left[ \Sigma^ {- 1} \delta \left[ \nabla_ {x} g \left(x _ {0} + \delta\right) \right] ^ {T} \right]. \tag {22}
$$
We begin by establishing a result expressing the eigenvalues of the symmetrization of a rank 1 matrix in closed form:
Lemma 4. Given  $x, y \in \mathbb{R}^d$  denote  $A = xy^T + yx^T \in \mathbb{R}^{d \times d}$ . The following facts hold:
1. Matrix  $A$  will have the following eigenvalues
- It has  $d - 2$  eigenvalues equal to 0
- The other two eigenvalues are denoted by  $\lambda^{+}(A)$  and  $\lambda^{-}(A)$  will have the following form:
$$
\lambda^ {\pm} (A) = x ^ {T} y \pm \| x \| _ {2} \| y \| _ {2} \tag {23}
$$
2.  $\lambda^{+}(A)$  and  $\lambda^{-}(A)$  are non-negative and non-positive respectively.
3. Given  $x$  and  $y$  are sampled from sub-gaussian distributions, then  $\lambda^{+}(A)$  and  $\lambda^{-}(A)$  are sub-exponential random variables.
Proof. We divide the proof into three sections
1. We have  $\mathrm{rank}(A) \leq 2$ ,  $A \in S_d$ . Therefore  $\exists Q = [e_1|e_2|\ldots|e_d] \in \mathbb{R}^{d \times d}$  s.t.  $Q^T Q = QQ^T = I$ , the column vectors  $e_i \in \mathbb{R}^d$  are orthonormal and
$$
A = Q \Lambda Q ^ {T} \tag {24a}
$$
$$
Q ^ {T} A Q = \Lambda \tag {24b}
$$
where  $\Lambda = \mathrm{diag}([\lambda_1,\lambda_2,0,\dots ,0])\in \mathbb{R}^{d\times d}$ . As eigenvalues are invariant to change of basis,  $A$  has eigenvalues  $\lambda_{1}$  and  $\lambda_{2}$  and the other  $d - 2$  eigenvalues are equal to 0.
It can be seen that  $\operatorname{span}(\{x, y\}) = \operatorname{span}(\{e_1, e_2\})$ . As  $\operatorname{span}(\{e_1, e_2\}) = C(A)$  this can be shown by proving  $\operatorname{span}(\{x, y\}) = C(A)$ . We know  $\exists z_x \in \mathbb{R}^d : z_x \perp x$  and  $\exists z_y \in \mathbb{R}^d : z_y \perp y$ . Thus we have
$$
A z _ {x} = \left(x y ^ {T} + y x ^ {T}\right) z _ {x} = x y ^ {T} z _ {x} + y x ^ {T} z _ {x} = x \left(y ^ {T} z _ {x}\right) \tag {25a}
$$
$$
A z _ {y} = \left(x y ^ {T} + y x ^ {T}\right) z _ {y} = x y ^ {T} z _ {y} + y x ^ {T} z _ {y} = y \left(x ^ {T} z _ {y}\right). \tag {25b}
$$
From the above, we see that  $x, y \in C(A)$ . We know  $\operatorname{rank}(A) \leq 2$ . If  $\operatorname{rank}(A) = 2$  we have  $x \neq y$  and thus  $\operatorname{span}(\{x, y\}) = C(A)$ . If  $\operatorname{rank}(A) = 1$  we have  $x \neq 0, y \neq 0$  and still  $\operatorname{span}(\{x, y\}) = C(A)$ . If  $\operatorname{rank}(A) = 0$  we have  $x = y = 0$  and clearly  $\operatorname{span}(\{x, y\}) = \{0\} = C(A)$ .
As  $\{e_i\}_{i = 1}^d$  are orthonormal, we have that  $x^{T}e_{j} = y^{T}e_{j} = 0$ ,  $\forall j > 2$ . We define
$$
\bar {x} = Q ^ {T} x = \left(e _ {1} ^ {T} x, e _ {2} ^ {T} x, e _ {3} ^ {T} x, \dots , e _ {d} ^ {T} x\right) = \left(e _ {1} ^ {T} x, e _ {2} ^ {T} x, 0, \dots , 0\right) \in \mathbb {R} ^ {d}, \tag {26a}
$$
$$
\bar {y} = Q ^ {T} y = \left(e _ {1} ^ {T} y, e _ {2} ^ {T} y, e _ {3} ^ {T} y, \dots , e _ {d} ^ {T} y\right) = \left(e _ {1} ^ {T} y, e _ {2} ^ {T} y, 0, \dots , 0\right) \in \mathbb {R} ^ {d} \tag {26b}
$$
We define  $Q_{2} = [e_{1}|e_{2}]\in \mathbb{R}^{d\times 2}$  and
$$
\tilde {x} = Q _ {2} ^ {T} x = \left(e _ {1} ^ {T} x, e _ {2} ^ {T} x\right) \in \mathbb {R} ^ {2} \tag {27a}
$$
$$
\tilde {y} = Q _ {2} ^ {T} y = \left(e _ {1} ^ {T} y, e _ {2} ^ {T} y\right) \in \mathbb {R} ^ {2} \tag {27b}
$$
We first show that  $\| \bar{x}\| _2$  and  $\| \bar{y}\| _2$  are equal to  $\| x\| _2$  and  $\| y\| _2$  respectively:
$$
\left\| \bar {x} \right\| _ {2} = \left\| Q ^ {T} x \right\| _ {2} = \sqrt {\left(Q ^ {T} x\right) ^ {T} \left(Q ^ {T} x\right)} = \sqrt {x ^ {T} Q Q ^ {T} x} = \sqrt {x ^ {T} I x} = \| x \| _ {2} \tag {28a}
$$
$$
\left\| \bar {y} \right\| _ {2} = \left\| Q ^ {T} y \right\| _ {2} = \sqrt {\left(Q ^ {T} y\right) ^ {T} \left(Q ^ {T} y\right)} = \sqrt {y ^ {T} Q Q ^ {T} y} = \sqrt {y ^ {T} I y} = \| y \| _ {2} \tag {28b}
$$
Next, we use Eq. (28) to show that  $\| \tilde{x}\| _2$  and  $\| \tilde{y}\| _2$  are equal to  $\| x\| _2$  and  $\| y\| _2$  respectively:
$$
\| \tilde {x} \| _ {2} = \sqrt {\left(e _ {1} ^ {T} x\right) ^ {2} + \left(e _ {2} ^ {T} x\right) ^ {2}} = \| \bar {x} \| _ {2} = \| x \| _ {2} \tag {29a}
$$
$$
\| \tilde {y} \| _ {2} = \sqrt {\left(e _ {1} ^ {T} y\right) ^ {2} + \left(e _ {2} ^ {T} y\right) ^ {2}} = \| \bar {y} \| _ {2} = \| y \| _ {2} \tag {29b}
$$
We show an equality between inner products  $\tilde{x}^T\tilde{y} = x^T y$
$$
\tilde {x} ^ {T} \tilde {y} = \left(e _ {1} ^ {T} x\right) \left(e _ {1} ^ {T} y\right) + \left(e _ {2} ^ {T} x\right) \left(e _ {2} ^ {T} y\right) = \bar {x} ^ {T} \bar {y} = \left(Q ^ {T} x\right) ^ {T} Q ^ {T} y = x ^ {T} Q Q ^ {T} y = \tag {30a}
$$
$$
x ^ {T} I y = x ^ {T} y \tag {30b}
$$
Now, one may write,
$$
Q _ {2} ^ {T} A Q _ {2} = \operatorname {d i a g} \left(\left[ \lambda_ {1}, \lambda_ {2} \right]\right) \tag {31a}
$$
$$
Q _ {2} ^ {T} \left(x y ^ {T} + y x ^ {T}\right) Q _ {2} = \operatorname {d i a g} \left(\left[ \lambda_ {1}, \lambda_ {2} \right]\right) \tag {31b}
$$
$$
Q _ {2} ^ {T} x y ^ {T} Q _ {2} + Q _ {2} ^ {T} y x ^ {T} Q _ {2} = \operatorname {d i a g} ([ \lambda_ {1}, \lambda_ {2} ]) \tag {31c}
$$
$$
\tilde {x} \tilde {y} ^ {T} + \tilde {y} \tilde {x} ^ {T} = \operatorname {d i a g} ([ \lambda_ {1}, \lambda_ {2} ]). \tag {31d}
$$
The following facts hold as a result of Eq. (31d):
-  $\lambda_{1} = 2\tilde{x}_{1}\tilde{y}_{1}$
-  $\lambda_{2} = 2\tilde{x}_{2}\tilde{y}_{2}$
-  $\tilde{x}_1\tilde{y}_2 + \tilde{x}_2\tilde{y}_1 = 0$
Now, we show
$$
\lambda_ {1} = \tilde {x} ^ {T} \tilde {y} + \| \tilde {x} \| _ {2} \| \tilde {y} \| _ {2}, \quad \lambda_ {2} = \tilde {x} ^ {T} \tilde {y} - \| \tilde {x} \| _ {2} \| \tilde {y} \| _ {2} \tag {32}
$$
Which can be derived as such:
$$
\begin{array}{l} \tilde {x} ^ {T} \tilde {y} \pm \| \tilde {x} \| _ {2} \| \tilde {y} \| _ {2} = \tilde {x} _ {1} \tilde {y} _ {1} + \tilde {x} _ {2} \tilde {y} _ {2} \pm \sqrt {\left(\tilde {x} _ {1} ^ {2} + \tilde {x} _ {2} ^ {2}\right) \left(\tilde {y} _ {1} ^ {2} + \tilde {y} _ {2} ^ {2}\right)} \tag {33} \\ = \tilde {x} _ {1} \tilde {y} _ {1} + \tilde {x} _ {2} \tilde {y} _ {2} \pm \sqrt {\left(\tilde {x} _ {1} \tilde {y} _ {1} - \tilde {x} _ {2} \tilde {y} _ {2}\right) ^ {2}} = \tilde {x} _ {1} \tilde {y} _ {1} + \tilde {x} _ {2} \tilde {y} _ {2} \pm | \tilde {x} _ {1} \tilde {y} _ {1} - \tilde {x} _ {2} \tilde {y} _ {2} | = \lambda_ {1} \text {o r} \lambda_ {2} \\ \end{array}
$$
where the second equality comes from the following:
$$
\begin{array}{l} \left(\tilde {x} _ {1} \tilde {y} _ {1} - \tilde {x} _ {2} \tilde {y} _ {2}\right) ^ {2} = \left(\tilde {x} _ {1} \tilde {y} _ {1}\right) ^ {2} - 2 \tilde {x} _ {1} \tilde {x} _ {2} \tilde {y} _ {1} \tilde {y} _ {2} + \left(\tilde {x} _ {2} \tilde {y} _ {2}\right) ^ {2} \\ = \left(\tilde {x} _ {1} \tilde {y} _ {1}\right) ^ {2} - 2 \tilde {x} _ {1} \tilde {x} _ {2} \tilde {y} _ {1} \tilde {y} _ {2} + \left(\tilde {x} _ {2} \tilde {y} _ {2}\right) ^ {2} + \left(\tilde {x} _ {1} \tilde {y} _ {2} + \tilde {x} _ {2} \tilde {y} _ {1}\right) ^ {2} \tag {34} \\ = \left(\tilde {x _ {1}} ^ {2} + \tilde {x _ {2}} ^ {2}\right) \left(\tilde {y _ {1}} ^ {2} + \tilde {y _ {2}} ^ {2}\right). \\ \end{array}
$$
Proving that Eq. (32) holds. Finally, we combine Eq. (29) and Eq. (30) with Eq. (32) to express the eigenvalues of  $A$  as:
$$
\lambda_ {1} = x ^ {T} y + \| x \| _ {2} \| y \| _ {2}, \quad \lambda_ {2} = x ^ {T} y - \| x \| _ {2} \| y \| _ {2} \tag {35}
$$
which we use to denote  $\lambda^{+}(A) = \lambda_{1},\lambda^{-}(A) = \lambda_{2}$
2. As  $|x^T y| \leq \| x \|_2 \| y \|_2$ , it follows that  $\lambda^{+}(A) = x^T y + \| x \|_2 \| y \|_2 \geq 0$  and  $\lambda^{-}(A) = x^T y - \| x \|_2 \| y \|_2 \leq 0$ .
3. We denote  $D = \{1, \dots, d\}$  It can be seen that  $x_{i}y_{i}$  is sub-exponential  $\forall i \in D$ , as a sub-gaussian times a sub-gaussian is sub-exponential. Thus, it follows that
$$
x ^ {T} y = \sum_ {i = 1} ^ {d} x _ {i} y _ {i} \text {i s a s u b - e x p o n e n t i a l r a n d o m v a r i a b l e}, \tag {36}
$$
as the sum of sub-exponential random variables is sub-exponential. Further we see that  $x_{i}^{2}$  and  $y_{i}^{2}$  are sub-exponential as the square of a sub-gaussian is sub-exponential. As the sum of sub-exponentials is sub-exponential we have that  $\sum_{i=1}^{d} x_{i}^{2}, \sum_{i=1}^{d} y_{i}^{2}$  are both sub-exponential random variables. As the square root of a sub-exponential is sub-gaussian we have that
$$
\| x \| _ {2} = \sqrt {\sum_ {i = 1} ^ {d} x _ {i} ^ {2}} \text {i s a s u b - g a u s s i a n r a n d o m v a r i a b l e} \tag {37a}
$$
$$
\| y \| _ {2} = \sqrt {\sum_ {i = 1} ^ {d} y _ {i} ^ {2}} \text {i s a s u b - g a u s s i a n r a n d o m v a r i a b l e} \tag {37b}
$$
As a sub-gaussian times a sub-gaussian is sub-exponential, from Eq. 37 we have that
$$
\| x \| _ {2} \| y \| _ {2} \text {i s a s u b - e x p o n t i a l r a n d o m v a r i a b l e}. \tag {38}
$$
Now we see from Eq. and 36 Eq. 38 that both  $\lambda^{+}(A)$  and  $\lambda^{-}(A)$  are the sum of subexponential random variables and thus are sub-exponential.
We now use the result of Lemma 4 to prove the sample complexity bounds for SmoothHess in Theorem 1:
where  $H = \nabla_{x}^{2}[(f*q_{\Sigma})(x_{0})]$ ,  $C^{+}, C^{-}c^{+}, c^{-} > 0$  are constants depending on the function  $f$  and covariance  $\Sigma$  and  $q_{\Sigma}: \mathbb{R}^d \to \mathbb{R}$  is the density function of  $\mathcal{N}(0, \Sigma)$ .
Proof. As  $x_0, f$  and  $\Sigma$  are fixed, we refer to  $\hat{H}_n(f, x_0, \Sigma)$  as  $\hat{H}_n$  for brevity. We denote  $D = \{1, \ldots, d\}$ . We begin by explicitly expressing our estimator  $\hat{H}_n$  in terms of  $\delta_i$  and  $\nabla_x f(x_0 + \delta_i)$ . From Eq. (8) we have
$$
H _ {n} ^ {\circ} = \frac {1}{n} \sum_ {i = 1} ^ {n} \Sigma^ {- 1} \delta_ {i} \left[ \nabla_ {x} f \left(x _ {0} + \delta_ {i}\right) \right] ^ {T} \tag {39a}
$$
$$
\hat {H} _ {n} = \frac {1}{2} H _ {n} ^ {\circ} + \frac {1}{2} H _ {n} ^ {\circ T} = \tag {39b}
$$