SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gaspard Beugnot
gaspard.beugnot@inria.fr
Inria, École normale supérieure, CNRS, PSL Research University, 75005 Paris, France
Julien Mairal
julien.mairal@inria.fr
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France
alessandro.rudi@inria.fr
We present a novel approach to non-convex optimization with certificates, which handles smooth functions on the hypercube or on the torus. Unlike traditional methods that rely on algebraic properties, our algorithm exploits the regularity of the target function intrinsic in the decay of its Fourier spectrum. By defining a tractable family of models, we allow at the same time to obtain precise certificates and to leverage the advanced and powerful computational techniques developed to optimize neural networks. In this way the scalability of our approach is naturally enhanced by parallel computing with GPUs. Our approach, when applied to the case of polynomials of moderate dimensions but with thousands of coefficients, outperforms the state-of-the-art optimization methods with certificates, as the ones based on Lasserre's hierarchy, addressing problems intractable for the competitors.
Non-convex optimization is a difficult and crucial task. In this paper, we aim at optimizing globally a non-convex function defined on the hypercube, by providing a certificate of optimality on the resulting solution. Let  $h$  be a smooth function on  $[-1, 1]^d$ . Here we provide an algorithm that given  $\widehat{x}$ , an estimate of the minimizer  $x_{\star}$  of  $h$
$$
x_{\star} = \operatorname *{arg  min}_{x\in [-1,1]^{d}}h(x),
$$
produces an  $\epsilon$ , that constitutes an explicit certificate for the quality of  $\widehat{x}$ , of the form
$$
\left| h \left(x _ {\star}\right) - h (\widehat {x}) \right| \leq \epsilon_ {\delta},
$$
with probability  $1 - \delta$ . The literature abounds of algorithms to optimize non-convex functions. Typically they are either  $(a)$  heuristics, very smart, but with no guarantees of global convergence Moscato et al. [1989], Horst and Pardalos [2013]  $(b)$  variation of algorithms used in convex optimization, which can guarantee convergence only to local minima Boyd and Vandenberghe [2004]  $(c)$  algorithms with only asymptotic guarantees of convergence to a global minimum, but no explicit certificates Van Laarhoven et al. [1987]. In general, the methods recalled above are quite fast to produce some
solution, but don't provide guarantees on its quality, with the result that the produced point can be arbitrarily far from the optimum, so they are used typically where non-reliable results can be accepted.
On the contrary, there are contexts where an explicit quantification of the incurred error is crucial for the task at hand (finance, engineering, scientific validation, safety-critical scenarios Lasserre [2009]). In these cases, more expensive methods that provide certificates are used, such as polynomial sum-of-squares (poly-SoS) Lasserre [2001, 2009]. These kinds of techniques are quite powerful since they provide certificates in the form above, often with machine-precision error. However,  $(a)$  they have reduced applicability since  $h$  must be a multivariate polynomial (possibly sparse, low-degree) and must be known in its analytical form  $(b)$  the resulting algorithm is a semi-definite programming optimization on matrices whose size grows very fast with the number of variables and the degree of the polynomial, becoming intractable already in moderate dimensions and degrees.
Our approach builds and extends the more recent line of works on kernel sum-of-squares, and in particular the work of Woodworth et al. [2022] based on the Fourier analysis. It mitigates the limitations of poly-SoS methods in both aspects: (a) we can deal with any function  $h$  (not necessarily a polynomial) for which the Fourier transform is known and (b) the resulting algorithm leverages the smoothness properties of the objective function as Woodworth et al. [2022] rather than relying on its algebraic structure leading to way more compact representations than poly-SoS. Contrary to Woodworth et al. [2022], we fully leverage the power of the certificate allowing for a drastic reduction of the computational cost of the method. Indeed, we cast the minimization in terms of a way smaller problem, similar to the optimization of a small neural network that, albeit again non-convex, produces efficiently a good candidate on which we then compute the certificate.
Notably, our focus lies on a posteriori guarantees: we define a family of models that allow for efficient computation of certificates. Once the model structure is established, we have ample flexibility in training the model, offering various possibilities to achieve good certificates in practical scenarios, while still using well-established and effective techniques in the field of deep neural networks (DNN) Goodfellow et al. [2016] to reduce the computational burden of the approach.
- We propose a new approach to global optimization with certificates which drastically extends the applicability domain allowed by the state of the art, since it can be applied to any function for which we can compute the Fourier transform (not just polynomials).
- The proposed approach is naturally tailored for GPU computations and provides a refined control of time and memory requirements of the proposed algorithm, contrary to poly-SoS methods (whose complexity scales dramatically and in a rigid way with dimension and degree of the polynomial).
- From a technical viewpoint, we improve the results in Woodworth et al. [2022], by developing a fast stochastic approach to recover the certificate in high probability (theorem 3), and we generalize the formulation of the problem to allow the use of powerful techniques from DNN, still providing a certificate on the result (section 3, in particular alg. 1)
- In practical applications, we are able to provide certificates for functions in moderate dimensions, which surpasses the capabilities of current state-of-the-art techniques. Specifically, as shown in the experiments we can handle polynomials with thousands of coefficients. This achievement marks an important milestone towards utilizing these models to provide certificates for more challenging real-life problems.
1.1 Previous work
Polynomial SoS. In the field of certificate-based polynomial optimization, Lasserre's hierarchy plays a pivotal role Lasserre [2001, 2009]. This hierarchy employs a sequence of SDP relaxations with increasing size proportional to  $O(r^{d})$  (where  $d$  is the dimension of the space and  $r$  is a parameter that upper bounds the degree of the polynomial) and that ultimately converges to the optimal solution when  $r \to \infty$ . While Lasserre's hierarchy is primarily associated with polynomial optimization, its applicability extends beyond this domain. It offers a specific formulation for the more general moment problem, enabling a wide range of applications; see Henrion et al. [2020] for an introduction. For polynomial optimization problems such as in eq. (1), a significant amount of research has been dedicated to leveraging problem structure to improve the scalability of the hierarchy. This research
has predominantly focused on exploiting very specific sparsity patterns among the variables of the polynomial, enabling the handling in these restricted scenarios of instances ranging from a few variables to even thousands of variables Waki et al. [2006], Wang et al. [2021b,a]. There have been theoretical results regarding optimization on the hypercube Bach and Rudi [2023], Laurent and Slot [2022], but there are no algorithms handling them natively. Furthermore, alternative approaches exist that exploit different types of structure, such as the constant trace property Mai et al. [2022].
Kernel SoS. Kernel Sum of Squares (K-SoS) is an emerging research field that originated from the introduction of a novel parametrization for positive functions in Marteau-Ferey et al. [2020]. This approach has found application in various domains, including Optimal Control Berthier et al. [2022], Optimal Transport Muzellec et al. [2021] and modeling probability distribution Rudi and Ciliberto [2021]. In the context of function optimization, two types of theoretical results have been explored: a priori guarantees Rudi et al. [2020] and a posteriori guarantees Woodworth et al. [2022]. A priori guarantees offer insights into the convergence rate towards a global optimum of the function, giving a rate on the number of parameters and the complexity necessary to optimize a function up to a given error. For example, Rudi et al. [2020] proposes a general technique to achieve the global optimum, with error  $\epsilon$  of a function that is  $s$ -times differentiable, by requiring a number of parameters essentially in the order of  $O(\epsilon^{-s/d})$ , allowing to avoid the curse of dimensionality in the rate, when the function is very regular, i.e.,  $s \geq d$ , while typical black-box optimization algorithms have a complexity that scales as  $\epsilon^{-d}$ . A-posteriori guarantees focus on providing a certificate for the minimum found by the algorithm. In particular, Woodworth et al. [2022], provides both a-priori guarantee and a-posteriori certificates; however, the model considered makes it computationally infeasible to provide certificates in dimension larger than 2.
To conclude, approaches based on kernel-SoS allow to extend the applicability of global optimization with certificates methods to a wider family of functions and on exploiting finer regularity properties beyond just the number of variables and the degrees of a polynomial. By comparison, we focus on making the optimization amenable to high-performance GPU computation while retaining an a posteriori certificate of optimality.
2 Computing certificates with extended k-SoS
Without loss of generality (see next remark), with the goal of simplifying the analysis and using powerful tools from harmonic analysis, we cast the problem in terms of minimization of a periodic function  $f$  over the torus,  $[0,1]^d$  (we will denote it also as  $\mathbb{T}^d$ ). In particular, we are interested in minimizing periodic functions for which we know (or we can easily compute) the coefficients of its Fourier representation, i.e.
$$
f _ {\star} = \min  _ {z \in \mathbb {T} ^ {d}} f (z), \quad f (z) = \sum_ {\omega \in \mathbb {Z} ^ {d}} \widehat {f} _ {\omega} e ^ {2 \pi \mathrm {i} \omega \cdot z}, \quad \forall z \in \mathbb {T} ^ {d}, \tag {1}
$$
where  $\mathbb{Z}$  is the set of integers. This setting is already interesting on its own, as it encompasses a large class of smooth functions. It includes notably trigonometric polynomials, i.e. functions which have only a finite number of non-zero Fourier coefficients  $\widehat{f}_{\omega}$ . Optimization of trigonometric polynomials arises in multiple research areas, such as the optimal power flow Van Hentenryck [18] or quantum mechanics Hilling and Sudbery [2010]. Note that this problem is already NP-hard, as it encompasses for instance the Max-Cut problem Waldspurger et al. [2013]. Even so, we will consider the more general case where we can evaluate function values of  $f$ , along with its Fourier coefficient  $\widehat{f}_{\omega}$ , and we have access to its norm in a certain Hilbert space. This norm can be computed numerically for trigonometric polynomials, and more generally reflects the regularity (degree of differentiability) of the function, and thus the difficulty of the problem.
Remark 1 (No loss of generality in working on the torus). Given a (non-periodic) function  $h:[-1,1]^d\to \mathbb{R}$  we can obtain a periodic function whose minimum is exactly  $h_*$  and from which we can recover  $x_{\star}$ . Indeed, following the classical Chebychev construction, define  $\cos (2\pi z)$  as the componentwise application of  $\cos$  to the elements of  $2\pi z$ , i.e.  $\cos (2\pi z)\coloneqq (\cos (2\pi z_1),\ldots ,\cos (2\pi z_d))$  and define  $f$  as  $f(z)\coloneqq h(\cos (2\pi z))$  for  $z\in [0,1]^d$ . It is immediate to see that (a)  $f$  is periodic, and, (b) since  $\cos (2\pi z)$  is invertible on  $[0,1]^d$  and its image is exactly  $[-1,1]^d$ , we have  $h_* = h(x_\star) = f(z_\star)$  where
$$
x _ {\star} = \cos (2 \pi z _ {\star}), \quad a n d \quad z _ {\star} = \min  _ {z \in \mathbb {T} ^ {d}} f (z).
$$
2.1 Certificates for global optimization and k-SoS
A general "recipe" for obtaining a certificates was developed in Woodworth et al. [2022] where, in particular, it was derived the following bound [Woodworth et al., 2022, see Thm. 2]
$$
f _ {\star} \geq \sup  _ {c \in \mathbb {R}, g \in \mathcal {G} _ {+}} c - \| f - c - g \| _ {F}, \tag {2}
$$
where  $\| u\| _F$  is the  $\ell_1$  norm of the Fourier coefficients of a periodic function  $u$ , i.e.
$$
\left\| u \right\| _ {F} := \sum_ {\omega \in \mathbb {Z} ^ {d}} \left| \widehat {u} _ {\omega} \right|, \tag {3}
$$
and the sup is taken over  $\mathcal{G}_{+}$  that is a class of non-negative functions. The paper Woodworth et al. [2022] then chooses  $\mathcal{G}_{+}$  to be the set of positive semidefinite models, leading to a possibly expensive convex SDP problem. Our approach instead starts from the following two observations: (a) the lower bound in eq. (2) holds for any set  $\mathcal{G}_{+}$  of non-negative functions, not necessarily convex, moreover (b) any candidate solution  $(g,c)$  of the supremum in eq. (2) would constitute a lower bound for  $f_{\star}$ , so there is no need to solve eq. (2) exactly. This yields the following theorem
Theorem 1. Given a point  $\widehat{x} \in \mathbb{T}^d$  and a non-negative and periodic function  $g_0: \mathbb{T}^d \to \mathbb{R}_+$ , we have
$$
\left| f (\widehat {x}) - f \left(x _ {\star}\right) \right| \leq \| f - f (\widehat {x}) - g _ {0} \| _ {F} \tag {4}
$$
Proof. Since  $x_{\star}$  is the minimizer of  $f$ , then  $f(x_{\star}) \leq f(\widehat{x})$ . Moreover, since  $c_0 = f(\widehat{x})$  and  $g_0$  are feasible solutions for the r.h.s. of eq. (2), we have
$$
f(\widehat{x})\geq f(x_{\star})\geq \sup_{c\in \mathbb{R}, g\in \mathcal{G}_{+}}c - \| f - c - g\|_{F}\geq c_{0} - \| f - c_{0} - g_{0}\|_{F},
$$
from which we derive that  $0 \leq f(\widehat{x}) - f(x_{\star}) \leq \|f - f(\widehat{x}) - g_0\|_F$ .
In particular, since any good candidate  $g_0$  is enough to produce a certificate, we consider the following class of non-negative functions that can be seen as a two-layer neural network.
Definition 1 (extended K-SoS model on the torus). Let  $K:\mathbb{T}^d\times \mathbb{T}^d\to \mathbb{R}$  be a periodic function in the first variable and let  $m,r\in \mathbb{N}$ . Given a set of anchors  $\mathbf{Z} = (\mathbf{z}_1,\dots ,\mathbf{z}_{\mathbf{m}})\subset \mathbb{T}^d$  and a matrix  $R\in \mathbb{R}^{r\times m}$ , we define the  $K$ -SoS model  $g$  with
$$
\forall \mathbf {x} \in \mathbb {T} ^ {d}, \quad g (\mathbf {x}) = \| R K _ {\mathbf {Z}} (\mathbf {x}) \| _ {2} ^ {2}, \quad a n d \quad K _ {\mathbf {Z}} (\mathbf {x}) = \left(K (\mathbf {x}, \mathbf {z} _ {1}), \dots , K (\mathbf {x}, \mathbf {z} _ {m})\right) \in \mathbb {R} ^ {m}. \tag {5}
$$
The functions represented by the model above are non-negative and periodic. The model is an extension of the k-SoS model presented in Marteau-Ferey et al. [2020], where the points  $(\mathbf{z}_1,\dots ,\mathbf{z}_{\mathbf{m}})$  cannot be optimized. Moreover it has the following benefits at the expense of the convexity in the parameters:
1. The extended k-SoS models benefit of the good approximation properties of k-SoS models described in Marteau-Ferey et al. [2020] and especially Rudi and Ciliberto [2021], since they are a super-set of the k-SoS, that have optimal approximation properties for non-negative functions.
2. The extended model can have a reduced number of parameters, by choosing a matrix  $R$  with  $r = 1$  or  $r \ll m$ . This will drastically improve the cost of the optimization, while not impacting the approximation properties of the model, since a good approximation is still possible with already  $r$  proportional to  $d$  [Rudi et al., 2020, see Thm. 3].
3. The extended model does not require any positive semidefinite constraint on the matrix (contrary to the base model) that is typically a well-known bottleneck to scale up the optimization in the number of parameters Marteau-Ferey et al. [2020]. In the extended model we trade the positive semidefinite constraint with non-convexity. However this allows us to use all the advanced and effective techniques we know for unconstrained (or box-constrained) non-convex optimization for (two-layers) neural networks Goodfellow et al. [2016].
We discuss an efficient representation of these problems in section 3.3.
To conclude the picture on the k-SoS models, a critical aspect of the model is the choice of  $K$ , since it must guarantee good approximation properties and at the same time we need to compute easily its Fourier coefficients since we need to evaluate  $\| f - c - g \|_F$ . To this aim, a good candidate for  $K$  are the reproducing kernels defined on the torus Steinwart and Christmann [2008]. We use shift-invariant kernels, enabling a convenient analysis of the associated RKHS through their Fourier Transform.
Definition 2 (Reproducing kernel on the torus). Let  $q$  be a real function on  $\mathbb{T}^d$ , with positive Fourier Transform and  $q(0) = 1$ . Let  $K$  be the kernel defined with
$$
\forall x, y \in \mathbb {T} ^ {d}, K (x, y) = q (x - y) = \sum_ {\omega \in \mathbb {Z} ^ {d}} \widehat {q} _ {\omega} e ^ {2 \pi \mathrm {i} \omega \cdot (x - y)}. \tag {6}
$$
Then,  $K$  is a r.k bounded by 1. We denote  $\mathcal{H}$  its Reproducing kernel Hilbert Space (RKHS) and by  $\| \cdot \|_{\mathcal{H}}$  the associated RKHS norm
$$
\left\| f \right\| _ {\mathcal {H}} ^ {2} = \sum_ {\omega \in \mathbb {Z} ^ {d}} \left| \widehat {f} _ {\omega} \right| ^ {2} / \widehat {q} _ {\omega}.
$$
Define  $\lambda(x) = q(x)^2$ . We assume that we can compute (and sample from, see next section)  $\widehat{\lambda}_{\omega}$ , i.e., the Fourier transform of  $\lambda$ , corresponding to  $(\widehat{q} \star \widehat{q})_{\omega}$ , for all  $\omega \in \mathbb{Z}^d$ .
By choosing such a  $K$ , the models inherit the good approximation properties derived in Marteau-Ferey et al. [2020], Rudi and Ciliberto [2021]. We conclude by recalling that shift-invariant r.k kernel have a positive Fourier transform due to Bochner's theorem Rudin [1990]. The fact that  $K$  is bounded by 1 can be seen with  $|K(x,x)| = |q(0)| = \sum_{\omega}\widehat{q}_{\omega} = 1$ . Finally, note that the Fourier coefficients of an extended k-SoS model can be computed exactly, as in shown e.g. later in lemma 1.
2.2 Providing certificates with the  $F$ -norm
As discussed in the previous section our approach for providing a certificate on  $f$  relies on first obtaining  $\widehat{x}$  using a fast algorithm without guarantees and solving approximately eq. (2) to obtain the certificate (see theorem 1). With this aim, now we need an efficient way to compute the norm  $\| \cdot \|_F$ . We use here a stochastic approach. Introducing a probability  $\widehat{\lambda}_{\omega}$  (that later will be chosen as a rescaled version of  $\widehat{\lambda}_{\omega}$  in definition 2) on  $\mathbb{Z}^d$  we rewrite the  $F$ -norm
$$
\left\| u \right\| _ {F} = \sum_ {\omega \in \mathbb {Z} ^ {d}} \widehat {\lambda} _ {\omega} \cdot \frac {| \widehat {u} _ {\omega} |}{\widehat {\lambda} _ {\omega}} = \mathbb {E} _ {\omega \sim \widehat {\lambda} _ {\omega}} \left[ \frac {| \widehat {u} _ {\omega} |}{\widehat {\lambda} _ {\omega}} \right] \tag {7}
$$
which yields an objective that is amenable to stochastic optimization. From there, Woodworth et al. [2022] computes a certificate by truncating the sum to a hypercube  $\{\omega ;\| \omega \|_{\infty}\leq N\}$  of size
$N^d$  and bounding the remaining terms with a smoothness assumption on  $u = f - c - g$ , which enables to control the decay of  $\widehat{u}_{\omega}$ . We want to avoid this cost exponential in the dimension so we proceed differently.
Figure 1:  $f - f_{\star}$  is a trigonometric polynomial approximated by an extended k-SoS model  $g$ . The  $L_{\infty}$  norm of the difference (blue) is upper-bounded by the  $F$ -norm (red), which is itself upper bounded by the MoM inequality in theorem 3, with probability  $98\%$ , here showed with respect to the number  $N$  of sampled frequencies. Shaded area shows min/max values across 10 estimations.
Probabilistic estimates with the  $\mathcal{H}$  norm. Given that the  $F$ -norm can be written as an expectation in eq. (7), we approximate it with an empirical mean  $\widehat{S}$  given with  $N$  i.i.d samples  $(\omega_{i})_{1\leq i\leq n} \sim \widehat{\lambda}_{\omega}$ . Now, note that the variance of  $\widehat{S}$  can be upper bounded by a Hilbert norm, as
$$
\widehat {S} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\left| \widehat {u} _ {\omega_ {i}} \right|}{\lambda_ {\omega_ {i}}} \text {, s o t h a t} \operatorname {V a r} \widehat {S} \leq \frac {1}{N} \mathbb {E} \left(\frac {\left| \widehat {u} _ {\omega} \right|}{\widehat {\lambda} _ {\omega}}\right) ^ {2} = \frac {1}{N} \sum_ {\omega \in \mathbb {Z} ^ {d}} \frac {\left| \widehat {u} _ {\omega} \right| ^ {2}}{\widehat {\lambda} _ {\omega}} = \frac {1}{N} \| u \| _ {\mathcal {H} _ {\lambda}} ^ {2}, \tag {8}
$$
with  $\mathcal{H}_{\lambda}$  the RKHS from definition 2 with kernel  $K(x,x^{\prime}) = \sum_{\omega \in \mathbb{Z}^{d}}\widehat{\lambda}_{\omega}e^{2\pi \mathrm{i}\omega \cdot (x - x^{\prime})}$ . This allows to quantify the deviation of  $\widehat{S}$  from  $\mathbb{E}[\widehat{S}] = \| u\| _F$ , with e.g. Chebychev's inequality, as shown in next theorem.
Theorem 2 (Certificate with Chebychev Inequality). Let  $(\widehat{\lambda}_{\omega})_{\omega}$  be a probability distribution on  $\mathbb{Z}^d$ ,  $\delta \in (0,1)$  and  $g$  a positive function. Let  $N > 0$  and  $\widehat{S}$  be the empirical mean of  $|\widehat{f}_{\omega} - c - \widehat{g}_{\omega}| / \widehat{\lambda}_{\omega}$  obtained with  $N$  samples  $\omega_{i} \sim \widehat{\lambda}_{\omega}$ . Then, a certificate with probability  $1 - \delta$  is given with
$$
f _ {\star} \geq c - \widehat {S} - \frac {\| f - c - g \| _ {\mathcal {H} _ {\lambda}}}{\sqrt {N \delta}}, \quad \widehat {S} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\left| \widehat {f} _ {\omega_ {i}} - c \mathbf {1} _ {\omega_ {i} = 0} - \widehat {g} _ {\omega_ {i}} \right|}{\widehat {\lambda} _ {\omega_ {i}}}. \tag {9}
$$
Proof. From its definition in eq. (7), we see that an unbiased estimator of the  $F$ -norm is given by  $\widehat{S}$ . Then, Chebychev's inequality states that  $|\widehat{S} - \|u\|_F|^2 \leq \mathsf{Var} \widehat{S} / \delta$  with probability at least  $1 - \delta$ . Using the computation of the variance in eq. (8), it follows that  $\|u\|_F \leq \widehat{S} + \|f - c - g\|_{\mathcal{H}_\lambda} / \sqrt{N\delta}$  with probability at least  $1 - \delta$ . Plugging this expression into eq. (2), we obtain the result.
Note that the norm in  $\mathcal{H}_{\lambda}$  can be developed with (assuming for conciseness that  $(-c)$  is comprised in the 0-frequency of  $f$ )
$$
\left\| u \right\| _ {\mathcal {H} _ {\lambda}} ^ {2} = \sum_ {\omega \in \mathbb {Z} ^ {d}} \frac {\widehat {f} _ {\omega} ^ {*} (\widehat {f} _ {\omega} - 2 \widehat {g} _ {\omega})}{\widehat {\lambda} _ {\omega}} + \| g \| _ {\mathcal {H} _ {\lambda}} ^ {2} \leq \left(\| f \| _ {\mathcal {H} _ {\lambda}} + \| g \| _ {\mathcal {H} _ {\lambda}}\right) ^ {2}. \tag {10}
$$
Thus, theorem 2 provides a certificate of  $f_{\star}$  as long as we can (i) evaluate the Fourier transform  $\widehat{g}_{\omega}$  of  $g$  and (ii) compute its Hilbert norm in some r.k  $\mathcal{H}_{\lambda}$  induced by  $\widehat{\lambda}_{\omega}$ . In next section, we detail the choice we make to achieve this efficiently, with kernels amenable to GPU computation, scaling to thousands of coefficients.