SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Tackling urban mobility with technology. https://europe.googleblog.com/2015/11/tackling-urban-mobility-with-technology.html, 2015. Accessed: 2022-11-06.
[2] Learning with privacy at scale. https://docs-assets developer.apple.com/ml-research/papers/learning-with-privacy-at-scale.pdf, 2017. Accessed: 2022-11-06.
[3] Disclosure avoidance for the 2020 census: An introduction. https://www2.census.gov/library/publications/decennial/2020/2020-census-disclosure-avoidance-handbook.pdf, 2021. Accessed: 2022-11-06.
[4] Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research, 18(1):6446-6531, 2017.
[5] Emmanuel Abbe, Afonso S Bandeira, and Georgina Hall. Exact recovery in the stochastic block model. IEEE Transactions on information theory, 62(1):471-487, 2015.
[6] Jamil Arbas, Hassan Ashtiani, and Christopher Liaw. Polynomial time and private learning of unbounded gaussian mixture models. arXiv preprint arXiv:2303.04288, 2023.
[7] Sanjeev Arora and Satyen Kale. A combinatorial, primal-dual approach to semidefinite programs. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 227-236, 2007.
[8] Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning gaussians and beyond. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 1075-1076. PMLR, 02-05 Jul 2022.
[9] Ainesh Bakshi, Ilias Diakonikolas, Samuel B. Hopkins, Daniel Kane, Sushrut Karmalkar, and Pravesh K. Kothari. Outlier-robust clustering of gaussians and other non-spherical mixtures. In Sandy Irani, editor, 61st IEEE Annual Symposium on Foundations of Computer Science, FOCS 2020, Durham, NC, USA, November 16-19, 2020, pages 149-159. IEEE, 2020.
[10] Ainesh Bakshi, Ilias Diakonikolas, He Jia, Daniel M. Kane, Pravesh K. Kothari, and Santosh S. Vempala. Robustly learning mixtures of  $k$  arbitrary gaussians. In Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 - 24, 2022, pages 1234-1247. ACM, 2022.
[11] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473. IEEE, 2014.
[12] Christian Borgs, Jennifer Chayes, and Adam Smith. Private graphon estimation for sparse graphs. Advances in Neural Information Processing Systems, 28, 2015.
[13] Christian Borgs, Jennifer Chayes, Adam Smith, and Ilias Zadik. Revealing network structure, confidentially: Improved rates for node-private graphon estimation. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 533-543. IEEE, 2018.
[14] Moses Charikar, Jacob Steinhardt, and Gregory Valiant. Learning from untrusted data. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pages 47-60, 2017.
We use **boldface** characters for random variables. We hide multiplicative factors logarithmic in  $n$  using the notation  $\tilde{O}(\cdot), \tilde{\Omega}(\cdot)$ . Similarly, we hide absolute constant multiplicative factors using the standard notation  $O(\cdot), \Omega(\cdot), \Theta(\cdot)$ . Often times we use the letter  $C$  do denote universal constants independent of the parameters at play. We write  $o(1), \omega(1)$  for functions tending to zero (resp. infinity) as  $n$  grows. We say that an event happens with high probability if this probability is at least  $1 - o(1)$ . Throughout the paper, when we say "an algorithm runs in time  $O(q)$ " we mean that the number of basic arithmetic operations involved is  $O(q)$ . That is, we ignore bit complexity issues.
Vectors, matrices, tensors We use  $\mathrm{Id}_n$  to denote the  $n$ -by- $n$  dimensional identity matrix,  $J_n \in \mathbb{R}^{n \times n}$  the all-ones matrix and  $\mathbf{0}_n$ ,  $\mathbf{1}_n \in \mathbb{R}^n$  to denote respectively the zero and the all-ones vectors. When the context is clear we drop the subscript. For matrices  $A, B \in \mathbb{R}^{n \times n}$  we write  $A \succeq B$  if  $A - B$  is positive semidefinite. For a matrix  $M$ , we denote its eigenvalues by  $\lambda_1(M), \ldots, \lambda_n(M)$ , we simply write  $\lambda_i$  when the context is clear. We denote by  $\|M\|$  the spectral norm of  $M$ . We denote by  $\mathbb{R}^{d^{\otimes t}}$  the set of real-valued order- $t$  tensors. for a  $d \times d$  matrix  $M$ , we denote by  $M^{\otimes t}$  the  $t$ -fold Kronecker product  $\underbrace{M \otimes M \otimes \cdots \otimes M}_{t\text{ times}}$ . We define the flattening, or vectorization, of  $M$  to
be the  $d^{t}$ -dimensional vector, whose entries are the entries of  $M$  appearing in lexicographic order. With a slight abuse of notation we refer to this flattening with  $M$ , ambiguities will be clarified from context. We denote by  $N(0,\sigma^2)^{d^{\otimes t}}$  the distribution over Gaussian tensors with  $d^{t}$  entries with standard deviation  $\sigma$ . Given  $u,v\in \{\pm 1\} ^n$ , we use  $\mathrm{Ham}(u,v)\coloneqq \sum_{i = 1}^{n}\mathbb{1}_{[u_i\neq v_i]}$  to denote their Hamming distance. Given a vector  $u\in \mathbb{R}^n$ , we let  $\mathrm{sign}(u)\in \{\pm 1\} ^n$  denote its sign vector. A vector  $u\in \{\pm 1\} ^n$  is said to be balanced if  $\sum_{i = 1}^{n}u_{i} = 0$ .
Graphs We consider graphs on  $n$  vertices and let  $\mathcal{G}_n$  be the set of all graphs on  $n$  vertices. For a graph  $G$  on  $n$  vertices we denote by  $A(G) \in \mathbb{R}^{n \times n}$  its adjacency matrix. When the context is clear we simply write  $A$ . Let  $V(G)$  (resp.  $E(G)$ ) denote the vertex (resp. edge) set of graph  $G$ . Given two graphs  $G, H$  on the same vertex set  $V$ , let  $G \setminus H := (V, E(G) \setminus H(G))$ . Given a graph  $H, H' \subseteq H$  means  $H'$  is a subgraph of  $H$  such that  $V(H') = V(H)$  and  $E(H) \subseteq E(H)$ . The Hamming distance between two graphs  $G, H$  is defined to be the size of the symmetric difference between their edge sets, i.e.  $\operatorname{Ham}(G, H) := |E(G) \triangle E(H)|$ .
A.1 Differential privacy
In this section we introduce standard notions of differential privacy [24].
Definition A.1 (Differential privacy). An algorithm  $\mathcal{M}:\mathcal{V}\to \mathcal{O}$  is said to be  $(\varepsilon ,\delta)$ -differentially private for  $\varepsilon ,\delta >0$  if and only if, for every  $S\subseteq \mathcal{O}$  and every neighboring datasets  $Y,Y^{\prime}\in \mathcal{V}$  we have
$$
\mathbb {P} \left[ \mathcal {M} (Y) \in S \right] \leqslant e ^ {\varepsilon} \cdot \mathbb {P} \left[ \mathcal {M} \left(Y ^ {\prime}\right) \in S \right] + \delta .
$$
To avoid confusion, for each problem we will exactly state the relevant notion of neighboring datasets. Differential privacy is closed under post-processing and composition.
Lemma A.2 (Post-processing). If  $\mathcal{M}:\mathcal{Y}\to \mathcal{O}$  is an  $(\varepsilon ,\delta)$ -differentially private algorithm and  $\mathcal{M}':\mathcal{Y}\rightarrow \mathcal{Z}$  is any randomized function. Then the algorithm  $\mathcal{M}'$  ( $\mathcal{M}(Y)$ ) is  $(\varepsilon ,\delta)$ -differentially private.
In order to talk about composition it is convenient to also consider DP algorithms whose privacy guarantee holds only against subsets of inputs.
Definition A.3 (Differential Privacy Under Condition). An algorithm  $\mathcal{M}:\mathcal{V}\to \mathcal{O}$  is said to be  $(\varepsilon ,\delta)$ -differentially private under condition  $\Psi$  (or  $(\varepsilon ,\delta)$ -DP under condition  $\Psi$ ) for  $\varepsilon ,\delta >0$  if and only if, for every  $S\subseteq \mathcal{O}$  and every neighboring datasets  $Y,Y^{\prime}\in \mathcal{V}$  both satisfying  $\Psi$  we have
It is not hard to see that the following composition theorem holds for privacy under condition.
Lemma A.4 (Composition for Algorithm with Halting, [35]). Let  $\mathcal{M}_1: \mathcal{Y} \to \mathcal{O}_1 \cup \{\bot\}$ ,  $\mathcal{M}_2: \mathcal{O}_1 \times \mathcal{Y} \to \mathcal{O}_2 \cup \{\bot\}$ , ...,  $\mathcal{M}_t: \mathcal{O}_{t-1} \times \mathcal{Y} \to \mathcal{O}_t \cup \{\bot\}$  be algorithms. Furthermore, let  $\mathcal{M}$  denote the algorithm that proceeds as follows (with  $\mathcal{O}_0$  being empty): For  $i = 1, \ldots, t$  compute  $o_i = \mathcal{M}_i(o_{i-1}, Y)$  and, if  $o_i = \bot$ , halt and output  $\bot$ . Finally, if the algorithm has not halted, then output  $o_t$ . Suppose that:
- For any  $1 \leqslant i \leqslant t$ , we say that  $Y$  satisfies the condition  $\Psi_i$  if running the algorithm on  $Y$  does not result in halting after applying  $\mathcal{M}_1, \ldots, \mathcal{M}_i$ .
-  $\mathcal{M}_1$  is  $(\varepsilon_1, \delta_1)$ -DP.
-  $\mathcal{M}_i$  is  $(\varepsilon_i, \delta_i)$ -DP (with respect to neighboring datasets in the second argument) under condition  $\Psi_{i-1}$  for all  $i = \{2, \ldots, t\}$ .
Then  $\mathcal{M}$  is  $(\sum_{i}\varepsilon_{i},\sum_{i}\delta_{i})$  -DP.
A.1.1 Basic differential privacy mechanisms
The Gaussian and the Laplace mechanism are among the most widely used mechanisms in differential privacy. They work by adding a noise drawn from the Gaussian (respectively Laplace) distribution to the output of the function one wants to privatize. The magnitude of the noise depends on the sensitivity of the function.
Definition A.5 (Sensitivity of function). Let  $f: \mathcal{V} \to \mathbb{R}^d$  be a function, its  $\ell_1$ -sensitivity and  $\ell_2$ -sensitivity are respectively
$$
\Delta_{f,1}:= \max_{\substack{Y,Y^{\prime}\in \mathcal{Y}\\ Y,Y^{\prime}\text{are adjacent}}}\| f(Y) - f(Y^{\prime})\|_{1}\qquad \qquad \Delta_{f,2}:= \max_{\substack{Y,Y^{\prime}\in \mathcal{Y}\\ Y,Y^{\prime}\text{are adjacent}}}\| f(Y) - f(Y^{\prime})\|_{2}  .
$$
For function with bounded  $\ell_1$ -sensitivity the Laplace mechanism is often the tool of choice to achieve privacy.
Definition A.6 (Laplace distribution). The Laplace distribution with mean  $\mu$  and parameter  $b > 0$ , denoted by  $Lap(\mu, b)$ , has PDF  $\frac{1}{2b} e^{-|x - \mu| / b}$ . Let  $Lap(b)$  denote  $Lap(0, b)$ .
A standard tail bound concerning the Laplace distribution will be useful throughout the paper.
Fact A.7 (Laplace tail bound). Let  $x \sim \operatorname{Lap}(\mu, b)$ . Then,
$$
\mathbb {P} \left[ | \boldsymbol {x} - \mu | > t \right] \leqslant e ^ {- t / b}.
$$
The Laplace distribution is useful for the following mechanism
Lemma A.8 (Laplace mechanism). Let  $f: \mathcal{Y} \to \mathbb{R}^d$  be any function with  $\ell_1$ -sensitivity at most  $\Delta_{f,1}$ . Then the algorithm that adds  $\text{Lap}\left(\frac{\Delta_{f,1}}{\varepsilon}\right)^{\otimes d}$  to  $f$  is  $(\varepsilon, 0)$ -DP.
It is also useful to consider the "truncated" version of the Laplace distribution where the noise distribution is shifted and truncated to be non-positive.
Definition A.9 (Truncated Laplace distribution). The (negatively) truncated Laplace distribution  $w$  with mean  $\mu$  and parameter  $b$  on  $\mathbb{R}$ , denoted by  $tLap(\mu, b)$ , is defined as  $Lap(\mu, b)$  conditioned on the value being non-positive.
Lemma A.10 (Truncated Laplace mechanism). Let  $f: \mathcal{V} \to \mathbb{R}$  be any function with  $\ell_1$ -sensitivity at most  $\Delta_{f,1}$ . Then the algorithm that adds  $tLap\left(-\Delta_{f,1}\left(1 + \frac{\log(1 / \delta)}{\varepsilon}\right), \Delta_{f,1} / \varepsilon\right)$  to  $f$  is  $(\varepsilon, \delta)$ -DP.
The following tail bound is useful when reasoning about truncated Laplace random variables.
Lemma A.11 (Tail bound truncated Laplace). Suppose  $\mu < 0$  and  $b > 0$ . Let  $x \sim tLap(\mu, b)$ . Then, for  $y < \mu$  we have that
$$
\mathbb {P} [ \boldsymbol {x} <   y ] \leqslant \frac {e ^ {(y - \mu / b)}}{2 - e ^ {\mu / b}}.
$$
In constraint, when the function has bounded  $\ell_2$ -sensitivity, the Gaussian mechanism provides privacy.
Lemma A.12 (Gaussian mechanism). Let  $f: \mathcal{V} \to \mathbb{R}^d$  be any function with  $\ell_2$ -sensitivity at most  $\Delta_{f,2}$ . Let  $0 < \varepsilon, \delta \leqslant 1$ . Then the algorithm that adds  $N\left(0, \frac{\Delta_{f,2}^2 \cdot 2 \log(2 / \delta)}{\varepsilon^2} \cdot \mathrm{Id}\right)$  to  $f$  is  $(\varepsilon, \delta)$ -DP.
A.1.2 Private histograms
Here we present a classical private mechanism to learn a high dimensional histogram.
Lemma A.13 (High-dimensional private histogram learner, see [33]). Let  $q, b, \varepsilon > 0$  and  $0 < \delta < 1/n$ . Let  $\{I_i\}_{i=-\infty}^{\infty}$  be a partition of  $\mathbb{R}$  into intervals of length  $b$ , where  $I_i := \{x \in \mathbb{R} \mid q + (i-1) \cdot b \leqslant x < q + i \cdot b\}$ . Consider the partition of  $\mathbb{R}^d$  into sets  $\{B_{i_1,\dots,i_d}\}_{i_1,\dots,i_d=1}^{\infty}$  where
$$
B _ {i _ {1}, \dots , i _ {d}} := \left\{x \in \mathbb {R} ^ {d} \mid \forall j \in [ d ], x _ {j} \in I _ {i _ {j}} \right\}
$$
Let  $Y = \{y_{1},\ldots ,y_{n}\} \subseteq \mathbb{R}^{d}$  be a dataset of  $n$  points. For each  $B_{i_1,\dots ,i_d}$ , let  $p_{i_1,\dots ,i_d} = \frac{1}{n} |\{j\in [n]\mid y_j\in B_{i_1,\dots ,i_d}\}|$ . For  $n\geqslant \frac{8}{\varepsilon\alpha}\cdot \log \frac{2}{\delta\beta}$ , there exists an efficient  $(\varepsilon ,\delta)$ -differentially private algorithm that returns  $\hat{p}_{1,\dots ,1},\dots ,\hat{p}_{i_1,\dots ,i_d},\dots$  satisfying
$$
\mathbb {P} \left[ \max  _ {i _ {1}, \dots , i _ {d} \in \mathbb {N}} \left| p _ {i _ {1}, \dots , i _ {d}} - \hat {\boldsymbol {p}} _ {i _ {1}, \dots , i _ {d}} \right| \geqslant \alpha \right] \leqslant \beta .
$$
Proof. We consider the following algorithm, applied to each  $i_1,\ldots ,i_d\in \mathbb{N}$  on input  $Y$  ..
1. If  $p_{i_1, \ldots, i_d} = 0$  set  $\hat{p}_{i_1, \ldots, i_d} = 0$ , otherwise let  $\hat{p}_{i_1, \ldots, i_d} = p_{i_1, \ldots, i_d} + \tau$  where  $\tau \sim \mathrm{Lap}\left(0, \frac{2}{n\varepsilon}\right)$ .
2. If  $\hat{\pmb{p}}_{i_1,\dots,i_d} \leqslant \frac{3\log(2 / \delta)}{\varepsilon n}$  set  $\hat{\pmb{p}}_{i_1,\dots,i_d} = 0$
First we argue utility. By construction we get  $\hat{p}_{i_1,\dots,i_d} = 0$  whenever  $p_{i_1,\dots,i_d} = 0$ , thus we may focus on non-zero  $p_{i_1,\dots,i_d}$ . There are at most  $n$  non-zero  $p_{i_1,\dots,i_d}$ . By choice of  $n, \delta$  and by Fact A.7 the maximum over  $n$  independent trials  $\tau \sim \mathrm{Lap}\left(0, \frac{2}{n\varepsilon}\right)$  is bounded by  $\alpha$  in absolute value with probability at least  $\beta$ .
It remains to argue privacy. Let  $Y = \{y_{1},\ldots ,y_{n}\}$ ,  $Y^{\prime} = \{y_1^{\prime},\dots ,y_n^{\prime}\}$  be adjacent datasets. For  $i_{1},\ldots ,i_{d}\in \mathbb{N}$ , let
$$
\begin{array}{l} p _ {i _ {1}, \dots , i _ {d}} = | \{j \in [ n ] \mid y _ {j} \in B _ {i _ {1}, \dots , i _ {d}} \} | \\ p _ {i _ {1}, \dots , i _ {d}} ^ {\prime} = \left| \left\{j \in [ n ] \mid y _ {j} ^ {\prime} \in B _ {i _ {1}, \dots , i _ {d}} \right\} \right|. \\ \end{array}
$$
Since  $Y, Y'$  are adjacent there exists only two sets of indices  $\mathcal{I} := \{i_1, \ldots, i_d\}$  and  $\mathcal{J} := \{j_1, \ldots, j_d\}$  such that  $p_{\mathcal{I}} \neq p_{\mathcal{I}}'$  and  $p_{\mathcal{J}} \neq p_{\mathcal{J}}'$ . Assume without loss of generality  $p_{\mathcal{I}} > p_{\mathcal{I}}'$ . Then it must be  $p_{\mathcal{I}} = p_{\mathcal{I}}' + 1/n$  and  $p_{\mathcal{J}} = p_{\mathcal{J}}' - 1/n$ . Thus by the standard tail bound on the Laplace distribution in Fact A.7 and by Lemma A.8, we immediately get that the algorithm is  $(\varepsilon, \delta)$ -differentially private.
A.2 Sum-of-squares and pseudo-distributions
We introduce here the sum-of-squares notion necessary for our private algorithm learning mixtures of Gaussians. We remark that these notions are not needed for Appendix C.
Let  $w = (w_{1}, w_{2}, \ldots, w_{n})$  be a tuple of  $n$  indeterminates and let  $\mathbb{R}[w]$  be the set of polynomials with real coefficients and indeterminates  $w, \ldots, w_{n}$ . We say that a polynomial  $p \in \mathbb{R}[w]$  is a sum-of-squares (sos) if there are polynomials  $q_{1}, \ldots, q_{r}$  such that  $p = q_{1}^{2} + \dots + q_{r}^{2}$ .
A.2.1 Pseudo-distributions
Pseudo-distributions are generalizations of probability distributions. We can represent a discrete (i.e., finitely supported) probability distribution over  $\mathbb{R}^n$  by its probability mass function  $D\colon \mathbb{R}^n\to \mathbb{R}$  such that  $D\geqslant 0$  and  $\sum_{w\in \mathrm{supp}(D)}D(w) = 1$ . Similarly, we can describe a pseudo-distribution by its mass function. Here, we relax the constraint  $D\geqslant 0$  and only require that  $D$  passes certain low-degree non-negativity tests.
Concretely, a level- $\ell$  pseudo-distribution is a finitely-supported function  $D: \mathbb{R}^n \to \mathbb{R}$  such that  $\sum_w D(w) = 1$  and  $\sum_w D(w)f(w)^2 \geqslant 0$  for every polynomial  $f$  of degree at most  $\ell/2$ . (Here, the summations are over the support of  $D$ .) A straightforward polynomial-interpolation argument shows
that every level- $\infty$ -pseudo distribution satisfies  $D \geqslant 0$  and is thus an actual probability distribution. We define the pseudo-expectation of a function  $f$  on  $\mathbb{R}^d$  with respect to a pseudo-distribution  $D$ , denoted  $\tilde{\mathbb{E}}_{D(w)} f(w)$ , as
$$
\tilde {\mathbb {E}} _ {D (w)} f (w) = \sum_ {w} D (w) f (w). \tag {A.1}
$$
The degree- $\ell$  moment tensor of a pseudo-distribution  $D$  is the tensor  $\mathbb{E}_{D(w)}(1, w_1, w_2, \ldots, w_n)^{\otimes \ell}$ . In particular, the moment tensor has an entry corresponding to the pseudo-expectation of all monomials of degree at most  $\ell$  in  $w$ . The set of all degree- $\ell$  moment tensors of probability distribution is a convex set. Similarly, the set of all degree- $\ell$  moment tensors of degree  $d$  pseudo-distributions is also convex. Key to the algorithmic utility of pseudo-distributions is the fact that while there can be no efficient separation oracle for the convex set of all degree- $\ell$  moment tensors of an actual probability distribution, there's a separation oracle running in time  $n^{O(\ell)}$  for the convex set of the degree- $\ell$  moment tensors of all level- $\ell$  pseudodistributions.
Fact A.14 ([56, 53, 51, 38]). For any  $n,\ell \in \mathbb{N}$ , the following set has a  $n^{O(\ell)}$ -time weak separation oracle (in the sense of [27]):
$$
\left\{\tilde {\mathbb {E}} _ {D (w)} \left(1, w _ {1}, w _ {2}, \dots , w _ {n}\right) ^ {\otimes d} \mid \text {d e g r e e - d p s e u d o - d i s t r i b u t i o n D o v e r} \mathbb {R} ^ {n} \right\}. \tag {A.2}
$$
This fact, together with the equivalence of weak separation and optimization [27] allows us to efficiently optimize over pseudo-distributions (approximately)—this algorithm is referred to as the sum-of-squares algorithm.
The level- $\ell$  sum-of-squares algorithm optimizes over the space of all level- $\ell$  pseudo-distributions that satisfy a given set of polynomial constraints—we formally define this next.
Definition A.15 (Constrained pseudo-distributions). Let  $D$  be a level- $\ell$  pseudo-distribution over  $\mathbb{R}^n$ . Let  $\mathcal{A} = \{f_1 \geqslant 0, f_2 \geqslant 0, \ldots, f_m \geqslant 0\}$  be a system of  $m$  polynomial inequality constraints. We say that  $D$  satisfies the system of constraints  $\mathcal{A}$  at degree  $r$ , denoted  $D \models_r \mathcal{A}$ , if for every  $S \subseteq [m]$  and every sum-of-squares polynomial  $h$  with  $\deg h + \sum_{i \in S} \max \{\deg f_i, r\} \leqslant \ell$ ,
$$
\tilde {\mathbb {E}} _ {D} h \cdot \prod_ {i \in S} f _ {i} \geqslant 0.
$$
We write  $D \models \mathcal{A}$  (without specifying the degree) if  $D \models_0 \mathcal{A}$  holds. Furthermore, we say that  $D \models_r \mathcal{A}$  holds approximately if the above inequalities are satisfied up to an error of  $2^{-n^\ell} \cdot \| h \| \cdot \prod_{i \in S} \| f_i \|$ , where  $\| \cdot \|$  denotes the Euclidean norm $^{22}$  of the coefficients of a polynomial in the monomial basis.
We remark that if  $D$  is an actual (discrete) probability distribution, then we have  $D \models \mathcal{A}$  if and only if  $D$  is supported on solutions to the constraints  $\mathcal{A}$ .
We say that a system  $\mathcal{A}$  of polynomial constraints is explicitly bounded if it contains a constraint of the form  $\{\| w\| ^2\leqslant M\}$ . The following fact is a consequence of Fact A.14 and [27],
Fact A.16 (Efficient Optimization over Pseudo-distributions). There exists an  $(n + m)^{O(\ell)}$ -time algorithm that, given any explicitly bounded and satisfiable system<sup>23</sup>  $\mathcal{A}$  of  $m$  polynomial constraints in  $n$  variables, outputs a level- $\ell$  pseudo-distribution that satisfies  $\mathcal{A}$  approximately.
A.2.2 Sum-of-squares proof
Let  $f_1, f_2, \ldots, f_r$  and  $g$  be multivariate polynomials in  $w$ . A sum-of-squares proof that the constraints  $\{f_1 \geqslant 0, \ldots, f_m \geqslant 0\}$  imply the constraint  $\{g \geqslant 0\}$  consists of sum-of-squares polynomials  $(p_S)_{S \subseteq [m]}$  such that
$$
g = \sum_ {S \subseteq [ m ]} p _ {S} \cdot \Pi_ {i \in S} f _ {i}. \tag {A.3}
$$
22The choice of norm is not important here because the factor  $2^{-n^{\ell}}$  swamps the effects of choosing another norm.
23Here, we assume that the bit complexity of the constraints in  $\mathcal{A}$  is  $(n + m)^{O(1)}$
We say that this proof has degree  $\ell$  if for every set  $S\subseteq [m]$ , the polynomial  $p_S\Pi_{i\in S}f_i$  has degree at most  $\ell$ . If there is a degree  $\ell$  SoS proof that  $\{f_i\geqslant 0\mid i\leqslant r\}$  implies  $\{g\geqslant 0\}$ , we write:
$$
\left\{f _ {i} \geqslant 0 \mid i \leqslant r \right\} \Big | _ {\ell} \left\{g \geqslant 0 \right\}. \tag {A.4}
$$
Sum-of-squares proofs satisfy the following inference rules. For all polynomials  $f, g \colon \mathbb{R}^n \to \mathbb{R}$  and for all functions  $F \colon \mathbb{R}^n \to \mathbb{R}^m$ ,  $G \colon \mathbb{R}^n \to \mathbb{R}^k$ ,  $H \colon \mathbb{R}^p \to \mathbb{R}^n$  such that each of the coordinates of the outputs are polynomials of the inputs, we have:
$$
\frac {\mathcal {A} \left| _ {\ell} \right. \{f \geqslant 0 , g \geqslant 0 \}}{\mathcal {A} \left| _ {\ell} \right. \{f + g \geqslant 0 \}}, \frac {\mathcal {A} \left| _ {\ell} \right. \{f \geqslant 0 \} , \mathcal {A} \left| _ {\ell^ {\prime}} \right. \{g \geqslant 0 \}}{\mathcal {A} \left| _ {\ell + \ell^ {\prime}} \right. \{f \cdot g \geqslant 0 \}} \quad (a d d i t i o n
$$
$$
\frac {\mathcal {A} \left| _ {\ell} \right. \mathcal {B} , \mathcal {B} \left| _ {\ell^ {\prime}} \right. C}{\mathcal {A} \left| _ {\ell , \ell^ {\prime}} \right. C} \quad (\text {t r a n s i t i v i t y})
$$
$$
\frac {\{F \geqslant 0 \} \mid_ {\ell} \left\{G \geqslant 0 \right\}}{\left\{F (H) \geqslant 0 \right\} \mid_ {\ell \cdot \deg (H)} \left\{G (H) \geqslant 0 \right\}}. \quad (\text {s u b s t i t u t i o n})
$$
Low-degree sum-of-squares proofs are sound and complete if we take low-level pseudo-distributions as models.
Concretely, sum-of-squares proofs allow us to deduce properties of pseudo-distributions that satisfy some constraints.
Fact A.17 (Soundness). If  $D \stackrel{[r]}{\underset{r'}{\rightleftharpoons}} \mathcal{A}$  for a level- $\ell$  pseudo-distribution  $D$  and there exists a sum-of-squares proof  $\mathcal{A} \mid_{r'} \mathcal{B}$ , then  $D \stackrel{[r \cdot r' + r']}{\underset{r \cdot r' + r'}{\rightleftharpoons}} \mathcal{B}$ .
If the pseudo-distribution  $D$  satisfies  $\mathcal{A}$  only approximately, soundness continues to hold if we require an upper bound on the bit-complexity of the sum-of-squares  $\mathcal{A}\big|_{r'}B$  (number of bits required to write down the proof).
In our applications, the bit complexity of all sum of squares proofs will be  $n^{O(\ell)}$  (assuming that all numbers in the input have bit complexity  $n^{O(1)}$ ). This bound suffices in order to argue about pseudo-distributions that satisfy polynomial constraints approximately.