SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vern I. Paulsen and Mrinal Raghupathi. An Introduction to the Theory of Reproducing Kernel Hilbert Spaces. Cambridge Studies in Advanced Mathematics. Cambridge University Press, Cambridge, 2016. doi: 10.1017/CBO9781316219232.
Alessandro Rudi and Carlo Ciliberto. PSD Representations for Effective Probability Models. In Advances in Neural Information Processing Systems, volume 34, pages 19411-19422. Curran Associates, Inc., 2021.
Alessandro Rudi, Ulysse Marteau-Ferey, and Francis Bach. Finding Global Minima via Kernel Approximations. arXiv:2012.11978, December 2020.
Walter Rudin. The Basic Theorems of Fourier Analysis. In Fourier Analysis on Groups, chapter 1, pages 1-34. John Wiley & Sons, Ltd, 1990. doi: 10.1002/9781118165621.ch1.
Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008.
Pascal Van Hentenryck. Machine Learning for Optimal Power Flows. INFORMS Tutorials in Operations Research, October 18.
Peter JM Van Laarhoven, Emile HL Aarts, Peter JM van Laarhoven, and Emile HL Aarts. Simulated annealing. Springer, 1987.
Hayato Waki, Sunyoung Kim, Masakazu Kojima, and Masakazu Muramatsu. Sums of Squares and Semidefinite Program Relaxations for Polynomial Optimization Problems with Structured Sparsity. SIAM Journal on Optimization, 17(1):218-242, January 2006. doi: 10.1137/050623802.
Irene Waldspurger, Alexandre d'Aspremont, and Stéphane Mallat. Phase Recovery, MaxCut and Complex Semidefinite Programming, July 2013.
Jie Wang and Victor Magron. Exploiting Sparsity in Complex Polynomial Optimization. Journal of Optimization Theory and Applications, 192(1):335-359, January 2022.
Jie Wang, Victor Magron, and Jean-Bernard Lasserre. Chordal-TSSOS: A Moment-SOS Hierarchy That Exploits Term Sparsity with Chordal Extension. SIAM Journal on Optimization, 31(1): 114-141, January 2021a. doi: 10.1137/20M1323564.
Jie Wang, Victor Magron, and Jean-Bernard Lasserre. TSSOS: A Moment-SOS Hierarchy That Exploits Term Sparsity. SIAM Journal on Optimization, 31(1):30-58, January 2021b. doi: 10.1137/19M1307871.
G.N.Watson.A Treatise on the Theory of Bessel Functions.Cambridge University Press,1922.
Blake Woodworth, Francis Bach, and Alessandro Rudi. Non-Convex Optimization with Certificates and Fast Rates Through Kernel Sums of Squares. In Proceedings of Thirty Fifth Conference on Learning Theory, pages 4620-4642. PMLR, June 2022.
A Extensions
We explore additional extensions of GloptiNets that further enhance its appeal. We first describe a block diagonal structure for the model for faster evaluation, a theoretical splitting scheme for optimization, and finally a warm-start scheme.
A.1 Block diagonal structure for efficient computation
Without any further assumption, we see that a model from definition 1 can be evaluated in  $O(dm^2)$  time; its Fourier coefficient given by lemma 1 in  $O(dm^2r)$ ; the bound on the RKHS norm is computed in  $O(dm^2 + mr^2)$  time thanks to lemma 2; all that enables to compute a certificate, as stated in theorem 2, in  $O(Ndm^2r + mr^2)$  time, where  $N$  is the number of frequencies sampled. If the function  $f$  to be minimized has big  $\mathcal{H}_{\mathrm{s}}$  norm, we might need a large model size  $m$  to have  $f - f_{\star} \approx g$ . Hence, we introduce specific structure on  $G$  which makes it block-diagonal and better conditioned.
Proposition 2 (Block-diagonal PSD model). Let  $g$  be a PSD model as in definition 1, with  $m = bs$  anchors. Split them into  $b$  groups, denoting them  $\mathbf{z}_{ij}$ ,  $i \in [b]$  and  $j \in [s]$ . Compute the Cholesky factorization of each kernel matrix  $T_{i}^{\top}T_{i} = K_{\mathbf{z}_{i}} \in \mathbb{R}^{s \times s}$ . Then, define  $G$  as a block-diagonal matrix, with  $b$  blocks defined as  $G_{i} = \tilde{R}_{i}\tilde{R}_{i}^{\top}$ ,  $\tilde{R}_{i} = T^{-1}R_{i}$ , and  $R_{i} \in \mathbb{R}^{r \times s}$ . Equivalently,
$$
G = \left( \begin{array}{c c c} \tilde {R} _ {1} \tilde {R} _ {1} ^ {\top} & & \\ & \ddots & \\ & & \tilde {R} _ {b} \tilde {R} _ {b} ^ {\top} \end{array} \right), \text {s . t .} g (x) = \sum_ {i = 1} ^ {b} \left\| \tilde {R} _ {i} ^ {\top} K _ {\mathbf {z} _ {\mathrm {i}}} (x) \right\| ^ {2}, K _ {\mathbf {z} _ {\mathrm {i}}} (x) = K (\mathbf {z} _ {i j}, x) _ {1 \leq j \leq s}. \tag {21}
$$
Then  $g$  can be evaluated in  $O(rbs^3 d)$  time,  $\widehat{g}_{\omega}$  in  $O(bs^{2}(dr + s))$  time, and  $\| g\|_{S(\mathcal{H}_s)}^2$  in  $O(b^{2}(rs^{2} + r^{2}s) + bs^{3})$  time. The model has  $(r + d)bs$  real parameters.
Proof. Having  $G$  defined as such, it is psd, of rank at most  $rb \leq sb = m$ . Written  $g(x) = \sum_{i=1}^{b} \| \tilde{R}_i^\top K_{\mathbf{z}_i}(x) \|^2$ , we can compute the Fourier coefficient by applying lemma 1 to each of the  $b$  component. Adding the cost of computing  $G_i = \tilde{R}_i \tilde{R}_i^\top$  results in complexity of  $O(bs^2(dr + s))$ . Finally, note that  $\| g \|_{\mathcal{S}(\mathcal{H}_s)}^2 = \| A \|_{\mathcal{S}(\mathcal{H}_s)}^2$  where
$$
A = ((\varphi (\mathbf {z} _ {1 j})) _ {j \in [ s ]}, \ldots , (\varphi (\mathbf {z} _ {b j})) _ {j \in [ s ]}) (\mathrm {D i a g} G _ {i}) _ {i \in [ b ]} ((\varphi (\mathbf {z} _ {1 j})) _ {j \in [ s ]}, \ldots , (\varphi (\mathbf {z} _ {b j})) _ {j \in [ s ]}) ^ {*}.
$$
Then, defining  $Q$  the matrix of  $b \times b$  blocks of size  $s \times s$  s.t. for  $j, k \in [b]$ ,  $Q_{jk} = K(\mathbf{z}_j, \mathbf{z}_k) \in \mathbb{R}^{s \times s}$ , we have
$$
\left\| A \right\| _ {\mathcal {S} \left(\mathcal {H} _ {s}\right)} ^ {2} = \operatorname {T r} Q (\operatorname {D i a g} G _ {i}) _ {i \in [ b ]} Q (\operatorname {D i a g} G _ {i}) _ {i \in [ b ]} = \sum_ {j, k = 1} ^ {b} \operatorname {T r} G _ {j} Q _ {j k} G _ {k} Q _ {k j}, \tag {22}
$$
and each term in the sum can be written  $\mathrm{Tr}(\tilde{R}_j^\top Q_{jk}\tilde{R}_k)(\tilde{R}_k^\top Q_{kj}\tilde{R}_j^\top) = \| \tilde{R}_j^\top Q_{jk}\tilde{R}_k\|_{HS}^2$ , which is computed in  $O(rs^{2} + r^{2}s)$  time, plus  $O(bs^{3})$  to compute the Cholesky factor.
Denoting  $\varphi_{\mathbf{z}_i} = (\varphi (\mathbf{z}_{ij}))_{1\leq j\leq s}$ , note that
$$
\varphi_ {\mathbf {z} _ {i}} G _ {i} \varphi_ {\mathbf {z} _ {i}} ^ {*} = \varphi_ {\mathbf {z} _ {i}} T _ {i} ^ {- 1} R _ {i} R _ {i} ^ {\top} \left(\varphi_ {\mathbf {z} _ {i}} T _ {i} ^ {- 1}\right) ^ {*} = E _ {i} R _ {i} R _ {i} ^ {\top} E _ {i} ^ {*}, \tag {23}
$$
with  $E_{i} = \varphi_{\mathbf{z}_{i}}T_{i}^{-1}$  an orthonormal basis of  $\mathrm{Span}(\varphi_{\mathbf{z}_{ij}})_{1\leq j\leq s}$  as  $E_{i}^{*}E_{i} = \mathbf{I}_{s}$ . Thus, each model's coefficient is defined on an orthonormal basis, which makes the optimization easier. Of course, this comes at an added  $s^3$  complexity, which could be alleviated by using e.g. an incomplete Cholesky factorization instead.
Remark 4 (Relation to Term Sparsity in POP). The successful application of polynomial hierarchies to problems with thousands of variables relies on making the moment matrix  $M$  having a block structure Wang et al. [2021b,a]. If the monomial basis has size  $m$ , the constraint  $M \succeq 0$  is replaced with  $M = (\mathrm{Diag}M_{i})_{i\in [b]}$  and  $M_{i} \succeq 0$ . This enables to solve b SDP of size at most  $s$  instead of one of size  $m$ . Our model in proposition 2 follows a similar route for having a lower computational budget.
A.2 Global optimization with splitting scheme
While GloptiNets can provide certificates for functions, it falls behind local solvers in terms of competitiveness. The challenge lies in the fact that finding a certificate is considerably more difficult than finding a local minimum, as it necessitates the uniform approximation of the entire function. However, we present a novel algorithmic framework that has the potential to enhance the competitiveness of GloptiNets with local solvers while simultaneously delivering certificates. Our approach involves partitioning the search domain into multiple regions and computing lower bounds for each partition. By discarding portions of the domain where we can certify that the function exceeds a certain threshold, the algorithm progressively simplifies the optimization problem and removes areas from consideration. Moreover, such an approach is naturally well suited to parallel computation.
The algorithm relies on a divide-and-conquer mechanism. First, we split the hypercube  $(-1,1)^d$  in  $N$  regions, where  $N$  is the number of core available. We compute an upper bound with a local solver. For each region, we run GloptiNets in parallel, computing a certificate at regular interval. As soon as the certificate is bigger than the upper bound, we stop the process: we know that the global minimum is not in the associated region. We can then reallocate the freed computing power by splitting the biggest current region, which yields an easier problem. We stop as soon as the region considered are small enough. This is summarized in alg. 2, where  $\mathbb{P}$  indicates the loop run in parallel.
Note that minimizing  $f$  on a hypercube of center  $\mu$  and size  $\sigma$  amounts to minimizing  $x \mapsto f((x - \mu) / \sigma)$  on  $[-1, 1]^d$ , which is another Chebychev polynomial whose coefficients can be evaluated efficiently thanks to the order-2 relation every orthonormal polynomial satisfy. For Chebychev polynomials, that is  $H_{\omega + 1}(x) = 2xH_{\omega}(x) - H_{\omega - 1}(x)$ .
Algorithm 2: Splitting scheme with GloptiNets
Data: A Chebychev polynomial  $f$  with a unique global optimum, a probability  $\delta$ , a number of cores  $N$  and a volume  $\rho < 1 / N$ .  
Result: A certificate on  $f$ :  $f_{\star} \geq C_{\delta}(f)$  with proba.  $1 - \delta_{\star}$ .  
/* Initialization: upper bound and partition  
 $\Pi = \text{partition}([-1,1]^d, N)$ ,  $\delta_{\star} = 0$ ;  
 $\mathbb{P}$ $\text{ub} = \min_{\pi \in \Pi} \{\text{local solver}_{x \in \pi} f(x)\}$ ;  
/* Iterate over the partition  
 $\mathbb{P}$  for  $\pi \in \Pi$ , While length  $(\Pi) > 1$  do  
while  $C_{\delta}(f_{\pi}) < \text{ub do}$ $\lfloor$  Continue optimization;  
Split biggest part:  $\pi_0 = \arg \max_{\pi \in \Pi} \text{Vol}(\pi)$ ; ( $\pi_1, \pi_2$ ) = partition  $(\pi_0, 2)$ ;  
If  $\text{Vol}(\pi_{1,2}) < \rho$ : end this process;  
Update upper bound:  $\text{ub} = \min \left\{\text{ub}, \text{local solver}_{x \in \pi_{1,2}} f(x)\right\}$ ;  
Update search space and  $\delta_{\star}$ :  $\Pi = \Pi \setminus \{\pi, \pi_0\} \cup \{\pi_1, \pi_2\}$ ,  $\delta_{\star} = 1 - (1 - \delta_{\star})(1 - \delta)$ ;  
/* A single region in  $\Pi$  remains  
Returns  $\Pi = \{\pi\}$ ,  $C_{\delta}(f_{\pi})$ ,  $\delta_{\star}$ ;
A.3 Warm restarts
Our model distinguishes itself by leveraging the analytical properties of the objective function, rather than relying solely on algebraic characteristics. This approach offers a notable advantage, as closely related functions can naturally benefit from a warm restart. For example, if we already have a certificate for a function  $f$  using a PSD model  $g$ , and we seek to compute a certificate for a similar function  $\tilde{f} \approx f$ , we can readily employ GloptiNets by initializing the PSD model with  $g$ . Indeed, if  $f - f_{\star} \approx g$ , we can expect  $\tilde{f} - \tilde{f}_{\star} \approx g$ , so we can expect the optimization to be faster.
In contrast, P-SoS methods, which rely on SDP programs, cannot directly adapt to new problems without significant effort. For instance, if a new component is introduced, an entirely new SDP must be solved. Our model's ability to accommodate related yet distinct problems could prove highly valuable in domains with a frequent need to certify different but closely related problems. In the industry, the Optimal Power Flow (OPF) problem requires periodic solves every 5 minutes
Van Hentenryck [18]. With GloptiNets, once the initial challenging solve is performed, subsequent solves become easier assuming minimal changes in supply and demand conditions.
A.4 Optimizing the certificate directly
As explained in section 3.2 where GloptiNets is introduced, we optimize a proxy of the  $L_{\infty}$  norm rather than the certificate of theorems 2 and 3. This proxy is the log-sum-exp on a random batch of  $N$  points. The reason for this is that evaluating an extended k-SoS model  $g(x)$  on  $x \in \mathbb{T}^d$  requires  $O(ds)$  time, while evaluating  $\widehat{g}_{\omega}$  on  $\omega \in \mathbb{Z}^d$  requires  $O(ds^2)$  time. Yet, optimizing the certificate directly could probably help obtaining higher-precision certificate. Lemma 4 in appendix D sketches a method to reduce the computational cost of the Fourier computation from  $O(s^2)$  to  $O(s)$ .
B Kernel defined on the Chebychev basis
In this section we describe the approach we take to model functions written in the Chebychev basis. For  $h$  such a polynomial, a naive approach would simply model  $f = h \circ \cos(2\pi \cdot)$  as a trigonometric polynomial. However, note that the decomposition of  $f$  only has cosine terms. Thus, approximating  $f - f_{\star}$  efficiently requires a PSD model which has only cosine terms in its Fourier decomposition. This is achieved by using a kernel written in the Chebychev basis, as introduce in proposition 1, for which we now provide a proof.
Proof of proposition 1. Let  $x, y \in [-1, 1]$  and  $u, v \in [0, 1/2]$  s.t.  $x, y = \cos(2\pi u), \cos(2\pi v)$ , by bijectivity of the cosine function on  $[0, \pi]$ . From the definition of  $K$  in eq. (19) and the definition of  $q$  in eq. (6), we have that
$$
\begin{array}{l} K (x, y) = \frac {1}{2} \sum_ {\omega \in \mathbb {Z}} \widehat {q} _ {\omega} \left(e ^ {2 \pi \mathrm {i} \omega (u + v)} + e ^ {2 \pi \mathrm {i} \omega (u - v)}\right) \\ = \sum_ {\omega \in \mathbb {Z}} \widehat {q} _ {\omega} e ^ {2 \pi \mathrm {i} \omega u} \cos (2 \pi \omega v) \\ = \widehat {q} _ {0} + 2 \sum_ {\omega \in \mathbb {N}} \widehat {q} _ {\omega} \cos (2 \pi \omega u) \cos (2 \pi \omega v) \\ = \widehat {q} _ {0} + 2 \sum_ {\omega \in \mathbb {N}} \widehat {q} _ {\omega} H _ {\omega} (u) H _ {\omega} (v). \\ \end{array}
$$
Since  $q$  has positive Fourier transform, this makes the feature map of  $K$  explicit with  $K(x,y) = \varphi(u) \cdot \varphi(v)$ ,  $\varphi(u)_\omega = \sqrt{(1 + 1_{\omega \neq 0}) \widehat{q}_\omega} H_\omega(u)$ , for  $\omega \in \mathbb{N}$ . Hence the kernel is a reproducing kernel.
We now use this kernel with the Bessel function  $x \mapsto e^{s(\cos(2\pi x) - 1)}$ , i.e. we define the kernel  $K$  on  $[-1, 1]$  to satisfy
$$
\forall u, v \in (0, 1 / 2), K (\cos (2 \pi u), \cos (2 \pi v)) = \frac {1}{2} \left(e ^ {s (\cos (2 \pi (u + v))} + e ^ {s (\cos (2 \pi (u - v))}\right). \tag {24}
$$
As it was the case for the torus, this kernel enables an easy characterization of a RKHS in which an associated PSD model  $g$  lives.
Lemma 3 (Chebychev coefficient of the Bessel kernel). Let  $g$  be a PSD model as in definition 1, with the kernel  $K$  of eq. (24). Then, the Chebychev coefficient  $\omega \in \mathbb{N}^d$  of  $g$  can be computed in  $O(rdm^2)$  time with
$$
\begin{array}{l} g _ {\omega} = \sum_ {i, j = 1} ^ {m} R _ {i} ^ {\top} R _ {j} \prod_ {\ell = 1} ^ {d} (1 + \mathbf {1} _ {\omega \neq 0}) \frac {e ^ {- 2 \mathbf {s} _ {\ell}}}{2} \left[ I _ {\omega_ {\ell}} \left(2 \mathbf {s} _ {\ell} \sigma_ {- \ell i j}\right) H _ {\omega_ {\ell}} \left(\sigma_ {+ \ell i j}\right) \right. \tag {25} \\ \left. \left. + I _ {\omega_ {\ell}} \left(2 \mathbf {s} _ {\ell} \sigma_ {+ \ell i j}\right) H _ {\omega_ {\ell}} \left(\sigma_ {- \ell i j}\right) \right] \right. \\ \end{array}
$$
$$
\sigma_ {\pm \ell i j} = \cos (2 \pi m _ {\pm \ell i j}), m _ {\pm \ell i j} = (\mathbf {u} _ {\ell i j} \pm \mathbf {u} _ {\ell i j}) / 2, a n d \cos 2 \pi \mathbf {u} _ {\ell i j} = \mathbf {z} _ {\ell i j}.
$$
Expanding  $g$  and definition of Chebychev coefficient. From the definition of  $g$  in eq. (5), we have
$$
g (\mathbf {x}) = \sum_ {i, j = 1} ^ {m} R _ {i} ^ {\top} R _ {j} \prod_ {\ell = 1} ^ {d} K _ {\mathbf {s} _ {\ell}} \left(\mathbf {x} _ {\ell}, \mathbf {z} _ {\ell i}\right) K _ {\mathbf {s} _ {\ell}} \left(\mathbf {x} _ {\ell}, \mathbf {z} _ {\ell j}\right). \tag {26}
$$
We consider  $x, y, z \in (-1, 1)$  and  $s > 0$ . We denote  $u, v, w \in (0, 1/2)$  s.t.
$$
x, y, z = \cos 2 \pi u, \cos 2 \pi v, \cos 2 \pi w
$$
with the bijectivity of  $x \mapsto \cos(2\pi x)$  on  $(0, 1/2)$ . We now compute the Chebychev coefficient of  $x \mapsto K_s(x,y)K_s(x,z)$ . Denoted  $p_{\omega}$ , this is
$$
\forall \omega \in \mathbb {N}, p _ {\omega} = \frac {1 + \mathbf {1} _ {\omega \neq 0}}{\pi} \int_ {- 1} ^ {1} K _ {s} (x, y) K _ {s} (x, z) T _ {\omega} (x) \frac {\mathrm {d} x}{\sqrt {1 - x ^ {2}}},
$$
$$
\forall \omega \in \mathbb {N}, p _ {\omega} = (1 + \mathbf {1} _ {\omega \neq 0}) \int_ {0} ^ {1} K _ {s} (\cos 2 \pi u, \cos 2 \pi v) K _ {s} (\cos 2 \pi u, \cos 2 \pi w) \cos (2 \pi \omega u) \mathrm {d} u. \tag {27}
$$
Chebychev coefficient of kernel product. With the definition of the kernel in proposition 1, eq. (19), we have
$$
\begin{array}{l} K _ {s} (x, y) K _ {s} (x, z) = \frac {1}{4} (p (u + v) + p (u - v)) \times (p (u + w) + p (u - w)) \\ = \frac {e ^ {- 2 s}}{4} \left(e ^ {s \cos 2 \pi (u + v)} + e ^ {s \cos 2 \pi (u - v)}\right) \times \left(e ^ {s \cos 2 \pi (u + w)} + e ^ {s \cos 2 \pi (u - w)}\right) \\ \end{array}
$$
Now use the sum-to-product formula with the cosines to obtain
$$
\begin{array}{l} K _ {s} (x, y) K _ {s} (x, z) = \frac {e ^ {- 2 s}}{4} \left(e ^ {2 s \cos 2 \pi \left(\frac {v - w}{2}\right) \cos 2 \pi \left(u + \frac {v + w}{2}\right)} + e ^ {2 s \cos 2 \pi \left(\frac {v - w}{2}\right) \cos 2 \pi \left(u - \frac {v + w}{2}\right)}\right) \\ \left. + e ^ {2 s \cos 2 \pi \left(\frac {v + w}{2}\right) \cos 2 \pi \left(u + \frac {v - w}{2}\right)} + e ^ {2 s \cos 2 \pi \left(\frac {v + w}{2}\right) \cos 2 \pi \left(u - \frac {v - w}{2}\right)}\right), \tag {28} \\ \end{array}
$$
We simplify this expression by introducing
$$
m _ {\pm} = \frac {1}{2} (v \pm w) \text {a n d} \sigma_ {\pm} = \cos 2 \pi m _ {\pm}. \tag {29}
$$
Then, eq. (28) becomes
$$
\begin{array}{l} K _ {s} (x, y) K _ {s} (x, z) = \frac {e ^ {- 2 s}}{4} \left(e ^ {2 s \sigma_ {-} \cos 2 \pi (u + m _ {+})} + e ^ {2 s \sigma_ {-} \cos 2 \pi (u - m _ {+})}\right) \tag {30} \\ \left. + e ^ {2 s \sigma_ {+} \cos 2 \pi (u + m _ {-})} + e ^ {2 s \sigma_ {+} \cos 2 \pi (u - m _ {-})}\right). \\ \end{array}
$$
We recognize the definition of the kernel (which is not a surprise as we chose the kernel to be stable by product). However, we need variables in  $(0,1 / 2)$  to retrieve the proper definition of the kernel. Instead, we use lemma 5 on eq. (30) combined with eq. (27), to obtain
$$
\begin{array}{l} p _ {\omega} = (1 + \mathbf {1} _ {\omega \neq 0}) \frac {e ^ {- 2 s}}{4} \left(\cos (2 \pi \omega m _ {+}) I _ {\omega} (2 s \sigma_ {-}) + \cos (2 \pi \omega m _ {+}) I _ {\omega} (2 s \sigma_ {-}) \right. \\ \left. \dot {+} \cos (2 \pi \omega m _ {-}) I _ {\omega} (2 s \sigma_ {+}) + \cos (2 \pi \omega m _ {-}) I _ {\omega} (2 s \sigma_ {+})\right), \\ \end{array}
$$
$$
p _ {\omega} = \left(1 + \mathbf {1} _ {\omega \neq 0}\right) \frac {e ^ {- 2 s}}{2} \left(\cos \left(2 \pi \omega m _ {+}\right) I _ {\omega} (2 s \sigma_ {-}) + \cos \left(2 \pi \omega m _ {-}\right) I _ {\omega} (2 s \sigma_ {+})\right). \tag {31}
$$
Equation (31) contains the Chebychev coefficient of the product of two kernel functions as defined in eq. (27). Plugging this result into the definition of  $g$  in eq. (26), and noting that  $\cos(2\pi \omega m_{\pm}) = H_{\omega}(\cos 2\pi m_{\pm}) = H_{\omega}(\sigma_{\pm})$ , we obtain the result.
Figure 3: Certificate vs. RKHS norm of  $f$ , for a given model  $g$  with a fixed number of parameters.  $f$  has 1146 coefficients and  $g$  has 22528 parameters. Best certificate is kept among a set of optimization hyperparameters. As the norm of  $f$  decreases, fitting  $f - f_{\star}$  with  $g$  is easier and the certificate becomes tighter.
Thanks to lemma 3, we see that a model  $g$  defined as in definition 1 with the Bessel kernel  $K_{\mathbf{s}}$  of eq. (24) as its Chebychev coefficients decaying in  $O(I_{\omega}(2s))$ . Hence, it belongs to  $\mathcal{H}_{2\mathrm{s}}$ , the RKHS associated to  $K_{2\mathrm{s}}$ .
C Additional details on the experiments
Tuning the hyperparameters. The time reported in section 4 does not take into account the experiments needed to find a good set of hyperparameters. The parameters tuned were the type of optimizer, the decay of learning rate, and the regularization on the Frobenius norm of  $G$ .
Regularization. Regularization is performed by approximating the  $HS$  norm with a proxy which is faster to compute. We use  $\| R_j^\top R_k\|_{HS}^2$  instead of  $\| \tilde{R}_j^\top Q_{jk}\tilde{R}_k\|_{HS}^2$  in eq. (22).
Hardware. GloptiNets was used with NVIDIA V100 GPUs for the interpolation part, and Intel Xeon CPU E5-2698 v4 @ 2.20GHz for computing the certificate. TSSOS was run on a Apple M1 chip with Mosek solver.
Configuration of TSSOS. We use the lowest possible relaxation order  $d$  (i.e.  $\lceil \deg f / 2\rceil$ ), along with Chordal sparsity. We use the first relaxation step of the hierarchy. In these settings, TSSOS is not guaranteed to converge to  $f_{\star}$  but will execute the fastest.
Certificate vs. number of parameter for a given function. In fig. 2, the target function is a random polynomial of norm 1 or 2, or a kernel mixture with 10 coefficients of norm 1 or 2. The models forming the blue line are defined as in proposition 2, with rank, block size and number of blocks equal to  $(1, bs, 1)$  respectively, with  $bs$  the block size we vary. The number of frequencies sampled to compute the certificate is  $1.6 \cdot 10^{7}$ , and accounts for the fact that the bound on the variance becomes larger than the MOM estimator for large models.
Certificate vs. problem difficulty for a given model. We have 3 related parameters: the quality of the optimization (given by the certificate), the expressivity of the model (given by its number of parameters), and the difficulty of the optimization (given by the norm of the function). In fig. 3, we fix the latter and plot the relation between the first two. Here, we fix the model with parameters (8, 16, 128), and we optimize a polynomial in  $3d$  of degree 12, with RKHS norm ranging from 1 to 20. The certificates obtained are given in fig. 3. The resulting plot exhibits a clear polynomial relation between the certificate and the norm of the function, with a slope of  $-0.88$ . This suggests that the certificate behaves as  $O(\|f\|_{\mathcal{H}_{2s}}^{1/2})$ .
Comparison with TSSOS on the Fourier basis. In table 1, the polynomials  $f$  all have a RKHS norm of 1. The small model is defined as in proposition 2, with rank, block size and number of blocks equal to 4, 32, 8 respectively. For the big models, those values are 8, 128, 16. The certificate is the
Table 2: GloptiNets and TSSOS on random Chebychev polynomials. The same conclusion as in table 1 applies. While TSSOS is very efficient on small problems, its memory requirements grow exponentially with the problem size. GloptiNets has less accuracy, but a computational burden which does not increase with the problem size.
d	p	n	TSSOS	GN-small	GN-big
Certif.	t	Certif.	t	Certif.	t
4	3	255	3.4·10-7	6	1.1·10-2	2·102	4.1·10-3	1·103
4	624	2.1·10-9	153	2.5·10-2	2·102	3.6·10-3	1·103
5	1295	Out of memory!	-	1.8·10-2	2·102	4.2·10-3	2·103
maximum of the Chebychev bound of theorem 2 and the MoM bound of theorem 3. The number of frequencies sampled is  $3.2 \cdot 10^{7}$ .