Title: GloptiNets: Scalable Non-Convex Optimization with Certificates
Venue: NeurIPS 2023
Accepted: yes
Keywords: non-convex optimization

polynomial optimization

kernel sum-of-squares
Abstract: We present a novel approach to non-convex optimization with certificates, which handles smooth functions on the hypercube or on the torus. Unlike traditional methods that rely on algebraic properties, our algorithm exploits the regularity of the target function intrinsic in the decay of its Fourier spectrum. By defining a tractable family of models, we allow {\em at the same time} to obtain precise certificates and to leverage the advanced and powerful computational techniques developed to optimize neural networks. In this way the scalability of our approach is naturally enhanced by parallel computing with GPUs. Our approach, when applied to the case of polynomials of moderate dimensions but with thousands of coefficients, outperforms the state-of-the-art optimization methods with certificates, as the ones based on Lasserre's hierarchy, addressing problems intractable for the competitors.

GloptiNets: Scalable Non-Convex Optimization with Certificates

Abstract

We present a novel approach to non-convex optimization with certificates, which handles smooth functions on the hypercube or on the torus. Unlike traditional methods that rely on algebraic properties, our algorithm exploits the regularity of the target function intrinsic in the decay of its Fourier spectrum. By defining a tractable family of models, we allow at the same time to obtain precise certificates and to leverage the advanced and powerful computational techniques developed to optimize neural networks. In this way the scalability of our approach is naturally enhanced by parallel computing with GPUs. Our approach, when applied to the case of polynomials of moderate dimensions but with thousands of coefficients, outperforms the state-of-the-art optimization methods with certificates, as the ones based on Lasserre's hierarchy, addressing problems intractable for the competitors.

1 Introduction

Non-convex optimization is a difficult and crucial task. In this paper, we aim at optimizing globally a non-convex function defined on the hypercube, by providing a certificate of optimality on the resulting solution. Let h be a smooth function on [ -1 , 1] d . Here we provide an algorithm that given ̂ x , an estimate of the minimizer x glyph[star] of h

produces an glyph[epsilon1] , that constitutes an explicit certificate for the quality of ̂ x , of the form

with probability 1 -δ . The literature abounds of algorithms to optimize non-convex functions. Typically they are either (a) heuristics, very smart, but with no guarantees of global convergence Moscato et al. [1989], Horst and Pardalos [2013] (b) variation of algorithms used in convex optimization, which can guarantee convergence only to local minima Boyd and Vandenberghe [2004] (c) algorithms with only asymptotic guarantees of convergence to a global minimum, but no explicit certificates Van Laarhoven et al. [1987]. In general, the methods recalled above are quite fast to produce some

solution, but don't provide guarantees on its quality, with the result that the produced point can be arbitrarily far from the optimum, so they are used typically where non-reliable results can be accepted.

On the contrary, there are contexts where an explicit quantification of the incurred error is crucial for the task at hand (finance, engineering, scientific validation, safety-critical scenarios Lasserre [2009]). In these cases, more expensive methods that provide certificates are used, such as polynomial sum-of-squares (poly-SoS) Lasserre [2001, 2009]. These kinds of techniques are quite powerful since they provide certificates in the form above, often with machine-precision error. However, (a) they have reduced applicability since h must be a multivariate polynomial (possibly sparse, low-degree) and must be known in its analytical form (b) the resulting algorithm is a semi-definite programming optimization on matrices whose size grows very fast with the number of variables and the degree of the polynomial, becoming intractable already in moderate dimensions and degrees.

Our approach builds and extends the more recent line of works on kernel sum-of-squares , and in particular the work of Woodworth et al. [2022] based on the Fourier analysis. It mitigates the limitations of poly-SoS methods in both aspects: (a) we can deal with any function h (not necessarily a polynomial) for which the Fourier transform is known and (b) the resulting algorithm leverages the smoothness properties of the objective function as Woodworth et al. [2022] rather than relying on its algebraic structure leading to way more compact representations than poly-SoS. Contrary to Woodworth et al. [2022], we fully leverage the power of the certificate allowing for a drastic reduction of the computational cost of the method. Indeed, we cast the minimization in terms of a way smaller problem, similar to the optimization of a small neural network that, albeit again non-convex, produces efficiently a good candidate on which we then compute the certificate.

Notably, our focus lies on a posteriori guarantees: we define a family of models that allow for efficient computation of certificates. Once the model structure is established, we have ample flexibility in training the model, offering various possibilities to achieve good certificates in practical scenarios, while still using well-established and effective techniques in the field of deep neural networks (DNN) Goodfellow et al. [2016] to reduce the computational burden of the approach.

Our contributions can be summarized as follows:

Wepropose a new approach to global optimization with certificates which drastically extends the applicability domain allowed by the state of the art, since it can be applied to any function for which we can compute the Fourier transform (not just polynomials).

The proposed approach is naturally tailored for GPU computations and provides a refined control of time and memory requirements of the proposed algorithm, contrary to poly-SoS methods (whose complexity scales dramatically and in a rigid way with dimension and degree of the polynomial).

From a technical viewpoint, we improve the results in Woodworth et al. [2022], by developing a fast stochastic approach to recover the certificate in high probability (theorem 3), and we generalize the formulation of the problem to allow the use of powerful techniques from DNN, still providing a certificate on the result (section 3, in particular alg. 1)

In practical applications, we are able to provide certificates for functions in moderate dimensions, which surpasses the capabilities of current state-of-the-art techniques. Specifically, as shown in the experiments we can handle polynomials with thousands of coefficients. This achievement marks an important milestone towards utilizing these models to provide certificates for more challenging real-life problems.

1.1 Previous work

Polynomial SoS. In the field of certificate-based polynomial optimization, Lasserre's hierarchy plays a pivotal role Lasserre [2001, 2009]. This hierarchy employs a sequence of SDP relaxations with increasing size proportional to O ( r d ) (where d is the dimension of the space and r is a parameter that upper bounds the degree of the polynomial) and that ultimately converges to the optimal solution when r → ∞ . While Lasserre's hierarchy is primarily associated with polynomial optimization, its applicability extends beyond this domain. It offers a specific formulation for the more general moment problem, enabling a wide range of applications; see Henrion et al. [2020] for an introduction. For polynomial optimization problems such as in eq. (1), a significant amount of research has been dedicated to leveraging problem structure to improve the scalability of the hierarchy. This research

has predominantly focused on exploiting very specific sparsity patterns among the variables of the polynomial, enabling the handling in these restricted scenarios of instances ranging from a few variables to even thousands of variables Waki et al. [2006], Wang et al. [2021b,a]. There have been theoretical results regarding optimization on the hypercube Bach and Rudi [2023], Laurent and Slot [2022], but there are no algorithms handling them natively. Furthermore, alternative approaches exist that exploit different types of structure, such as the constant trace property Mai et al. [2022].

Kernel SoS. Kernel Sum of Squares (K-SoS) is an emerging research field that originated from the introduction of a novel parametrization for positive functions in Marteau-Ferey et al. [2020]. This approach has found application in various domains, including Optimal Control Berthier et al. [2022], Optimal Transport Muzellec et al. [2021] and modeling probability distribution Rudi and Ciliberto [2021]. In the context of function optimization, two types of theoretical results have been explored: a priori guarantees Rudi et al. [2020] and a posteriori guarantees Woodworth et al. [2022]. A priori guarantees offer insights into the convergence rate towards a global optimum of the function, giving a rate on the number of parameters and the complexity necessary to optimize a function up to a given error. For example, Rudi et al. [2020] proposes a general technique to achieve the global optimum, with error glyph[epsilon1] of a function that is s -times differentiable, by requiring a number of parameters essentially in the order of O ( glyph[epsilon1] -s/d ) , allowing to avoid the curse of dimensionality in the rate, when the function is very regular, i.e., s ≥ d , while typical black-box optimization algorithms have a complexity that scales as glyph[epsilon1] -d . A-posteriori guarantees focus on providing a certificate for the minimum found by the algorithm. In particular, Woodworth et al. [2022], provides both a-priori guarantee and a-posteriori certificates; however, the model considered makes it computationally infeasible to provide certificates in dimension larger than 2 .

To conclude, approaches based on kernel-SoS allow to extend the applicability of global optimization with certificates methods to a wider family of functions and on exploiting finer regularity properties beyond just the number of variables and the degrees of a polynomial. By comparison, we focus on making the optimization amenable to high-performance GPU computation while retaining an a posteriori certificate of optimality.

2 Computing certificates with extended k-SoS

Without loss of generality (see next remark), with the goal of simplifying the analysis and using powerful tools from harmonic analysis, we cast the problem in terms of minimization of a periodic function f over the torus, [0 , 1] d (we will denote it also as T d ). In particular, we are interested in minimizing periodic functions for which we know (or we can easily compute) the coefficients of its Fourier representation, i.e.

where Z is the set of integers. This setting is already interesting on its own, as it encompasses a large class of smooth functions. It includes notably trigonometric polynomials, i.e. functions which have only a finite number of non-zero Fourier coefficients ̂ f ω . Optimization of trigonometric polynomials arises in multiple research areas, such as the optimal power flow Van Hentenryck [18] or quantum mechanics Hilling and Sudbery [2010]. Note that this problem is already NP-hard, as it encompasses for instance the Max-Cut problem Waldspurger et al. [2013]. Even so, we will consider the more general case where we can evaluate function values of f , along with its Fourier coefficient ̂ f ω , and we have access to its norm in a certain Hilbert space. This norm can be computed numerically for trigonometric polynomials, and more generally reflects the regularity (degree of differentiability) of the function, and thus the difficulty of the problem.

Remark 1 (No loss of generality in working on the torus) . Given a (non-periodic) function h : [ -1 , 1] d → R we can obtain a periodic function whose minimum is exactly h ∗ and from which we can recover x glyph[star] . Indeed, following the classical Chebychev construction, define cos(2 πz ) as the componentwise application of cos to the elements of 2 πz , i.e. cos(2 πz ) := (cos(2 πz 1 ) , . . . , cos(2 πz d )) and define f as f ( z ) := h (cos(2 πz )) for z ∈ [0 , 1] d . It is immediate to see that (a) f is periodic, and, (b) since cos(2 πz ) is invertible on [0 , 1] d and its image is exactly [ -1 , 1] d , we have h ∗ = h ( x glyph[star] ) = f ( z glyph[star] ) where

We discuss an efficient representation of these problems in section 3.3.

2.1 Certificates for global optimization and k-SoS

A general 'recipe' for obtaining a certificates was developed in Woodworth et al. [2022] where, in particular, it was derived the following bound [Woodworth et al., 2022, see Thm. 2]

where ‖ u ‖ F is the glyph[lscript] 1 norm of the Fourier coefficients of a periodic function u , i.e.

and the sup is taken over G + that is a class of non-negative functions. The paper Woodworth et al. [2022] then chooses G + to be the set of positive semidefinite models , leading to a possibly expensive convex SDP problem. Our approach instead starts from the following two observations: (a) the lower bound in eq. (2) holds for any set G + of non-negative functions, not necessarily convex , moreover (b) any candidate solution ( g, c ) of the supremum in eq. (2) would constitute a lower bound for f glyph[star] , so there is no need to solve eq. (2) exactly. This yields the following theorem

Theorem 1. Given a point ̂ x ∈ T d and a non-negative and periodic function g 0 : T d → R + , we have

Proof. Since x glyph[star] is the minimizer of f , then f ( x glyph[star] ) ≤ f ( ̂ x ) . Moreover, since c 0 = f ( ̂ x ) and g 0 are feasible solutions for the r.h.s. of eq. (2), we have

from which we derive that 0 ≤ f ( ̂ x ) -f ( x glyph[star] ) ≤ ‖ f -f ( ̂ x ) -g 0 ‖ F .

In particular, since any good candidate g 0 is enough to produce a certificate, we consider the following class of non-negative functions that can be seen as a two-layer neural network .

Definition 1 (extended K-SoS model on the torus) . Let K : T d × T d → R be a periodic function in the first variable and let m,r ∈ N . Given a set of anchors Z = ( z 1 , . . . , z m ) ⊂ T d and a matrix R ∈ R r × m , we define the K-SoS model g with

The functions represented by the model above are non-negative and periodic. The model is an extension of the k-SoS model presented in Marteau-Ferey et al. [2020], where the points ( z 1 , . . . , z m ) cannot be optimized. Moreover it has the following benefits at the expense of the convexity in the parameters:

The extended k-SoS models benefit of the good approximation properties of k-SoS models described in Marteau-Ferey et al. [2020] and especially Rudi and Ciliberto [2021], since they are a super-set of the k-SoS, that have optimal approximation properties for non-negative functions.

The extended model can have a reduced number of parameters , by choosing a matrix R with r = 1 or r glyph[lessmuch] m . This will drastically improve the cost of the optimization, while not impacting the approximation properties of the model, since a good approximation is still possible with already r proportional to d [Rudi et al., 2020, see Thm. 3].

The extended model does not require any positive semidefinite constraint on the matrix (contrary to the base model) that is typically a well-known bottleneck to scale up the optimization in the number of parameters Marteau-Ferey et al. [2020]. In the extended model we trade the positive semidefinite constraint with non-convexity. However this allows us to use all the advanced and effective techniques we know for unconstrained (or boxconstrained) non-convex optimization for (two-layers) neural networks Goodfellow et al. [2016].

To conclude the picture on the k-SoS models, a critical aspect of the model is the choice of K , since it must guarantee good approximation properties and at the same time we need to compute easily its Fourier coefficients since we need to evaluate ‖ f -c -g ‖ F . To this aim, a good candidate for K are the reproducing kernels defined on the torus Steinwart and Christmann [2008]. We use shift-invariant kernels, enabling a convenient analysis of the associated RKHS through their Fourier Transform.

Definition 2 (Reproducing kernel on the torus) . Let q be a real function on T d , with positive Fourier Transform and q (0) = 1 . Let K be the kernel defined with

Then, K is a r.k bounded by 1 . We denote H its Reproducing kernel Hilbert Space (RKHS) and by ‖·‖ H the associated RKHS norm

Define λ ( x ) = q ( x ) 2 . We assume that we can compute (and sample from, see next section) ̂ λ ω , i.e., the Fourier transform of λ , corresponding to ( ̂ q glyph[star] ̂ q ) ω , for all ω ∈ Z d .

By choosing such a K , the models inherit the good approximation properties derived in MarteauFerey et al. [2020], Rudi and Ciliberto [2021]. We conclude by recalling that shift-invariant r.k kernel have a positive Fourier transform due to Bochner's theorem Rudin [1990]. The fact that K is bounded by 1 can be seen with | K ( x, x ) | = | q (0) | = ∑ ω ̂ q ω = 1 . Finally, note that the Fourier coefficients of an extended k-SoS model can be computed exactly, as in shown e.g. later in lemma 1.

2.2 Providing certificates with the F -norm

As discussed in the previous section our approach for providing a certificate on f relies on first obtaining ̂ x using a fast algorithm without guarantees and solving approximately eq. (2) to obtain the certificate (see theorem 1). With this aim, now we need an efficient way to compute the norm ‖ · ‖ F . We use here a stochastic approach. Introducing a probability ̂ λ ω (that later will be chosen as a rescaled version of ̂ λ ω in definition 2) on Z d we rewrite the F -norm

which yields an objective that is amenable to stochastic optimization. From there, Woodworth et al. [2022] computes a certificate by truncating the sum to a hypercube { ω ; ‖ ω ‖ ∞ ≤ N } of size

10

3

10

4

10

5

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

N

L

∞

norm

F

-norm

Our bound (Thm. 3)

N d and bounding the remaining terms with a smoothness assumption on u = f -c -g , which enables to control the decay of ̂ u ω . We want to avoid this cost exponential in the dimension so we proceed differently.

Probabilistic estimates with the H norm. Given that the F -norm can be written as an expectation in eq. (7), we approximate it with an empirical mean ̂ S given with N i.i.d samples ( ω i ) 1 ≤ i ≤ n ∼ ̂ λ ω . Now, note that the variance of ̂ S can be upper bounded by a Hilbert norm, as

with H λ the RKHS from definition 2 with kernel K ( x, x ′ ) = ∑ ω ∈ Z d ̂ λ ω e 2 π i ω · ( x -x ′ ) . This allows to quantify the deviation of ̂ S from E [ ̂ S ] = ‖ u ‖ F , with e.g. Chebychev's inequality, as shown in next theorem.

Theorem 2 (Certificate with Chebychev Inequality) . Let ( ̂ λ ω ) ω be a probability distribution on Z d , δ ∈ (0 , 1) and g a positive function. Let N > 0 and ̂ S be the empirical mean of | ̂ f ω -c -̂ g ω | / ̂ λ ω obtained with N samples ω i ∼ ̂ λ ω . Then, a certificate with probability 1 -δ is given with

Proof. From its definition in eq. (7), we see that an unbiased estimator of the F -norm is given by ̂ S . Then, Chebychev's inequality states that | ̂ S -‖ u ‖ F | 2 ≤ Var ̂ S/δ with probability at least 1 -δ . Using the computation of the variance in eq. (8), it follows that ‖ u ‖ F ≤ ̂ S + ‖ f -c -g ‖ H λ / √ Nδ with probability at least 1 -δ . Plugging this expression into eq. (2), we obtain the result.

Note that the norm in H λ can be developed with (assuming for conciseness that ( -c ) is comprised in the 0 -frequency of f )

Thus, theorem 2 provides a certificate of f glyph[star] as long as we can (i) evaluate the Fourier transform ̂ g ω of g and (ii) compute its Hilbert norm in some r.k H λ induced by ̂ λ ω . In next section, we detail the choice we make to achieve this efficiently, with kernels amenable to GPU computation, scaling to thousands of coefficients.

Remark 2 (Using a RKHS norm instead of the F -norm) . Note that since ( ̂ λ ω ) ω sums to 1 , the associated kernel is bounded by 1 . Hence ‖ u ‖ L ∞ ≤ ‖ u ‖ H λ , and the latter could be used instead of the F -norm in eq. (2) . There are two reasons for taking our approach instead. Firstly, the F -norm is always tighter that a RKHS norm (see e.g. [Woodworth et al., 2022, Lem. 4]); secondly, we cannot compute ‖ u ‖ H λ efficiently and have to rely instead on another upper bound. However, taking the number of samples N = O ( ‖ u ‖ 2 H λ ) alleviates this issue.

Exponential concentration bounds with MoM. The scaling in 1 / √ δ in theorem 2 can be prohibitive if one requires a high probability on the result ( δ glyph[lessmuch] 1 ). Hopefully, alternative estimator exist for those cases. The Median-of-Mean estimator is an example, illustrated in theorem 3.

Theorem 3 (Certificate with MoM estimator) . Let ( ̂ λ ω ) ω be a probability distribution on Z d , and δ ∈ (0 , 1) . Draw N > 0 frequencies ω i ∼ ̂ λ ω . Define the MoM estimator with the following: for K ∈ N s.t. δ = e -K/ 8 and N = Kb , b ≥ 1 , write B 1 , . . . , B K a partition of [ N ] ; then

A certificate on f glyph[star] with probability 1 -δ follows, with

Proof. Using e.g. Theorem 4.1 from Devroye et al. [2016] we get that the deviation of the MoM estimator from the expectation is bounded by

Using the upper bound on the variance with the H λ norm given in eq. (8) and plugging the resulting expression into eq. (2), we obtain the result.

To conclude this section, bounding the L ∞ norm from above with the F -norm in eq. (3) enables to obtain a certificate on f , as shown in theorem 1. The F -norm requires an infinite number of computation in the general case, but can be bounded efficiently with a probabilistic estimate, given by theorem 2 or theorem 3. This is summed up in fig. 1. Note that the difference ‖·‖ F -‖·‖ L ∞ is a source of conservatism in the certificate which we do not quantify - yet, the F -norm is optimal for a class of norms, see [Woodworth et al., 2022, Lemma 3].

3 Algorithm and implementation

3.1 Bessel kernel

We now detail the specific choice of kernel we make in order to compute the certificate of theorem 2 or theorem 3 efficiently. Our first observation is to use a kernel stable by product, so that we can easily characterize a Hilbert space the model g belongs to. This restricts the choice to the exponential family. That's why we define, for a parameter s > 0 ,

with I | ω | ( · ) the modified Bessel function of the first kind [Watson, 1922, p.181]. Then, define K s ( x, y ) = q s ( x -y ) as in definition 2, and take a tensor product to extend the definition of K to multiple dimension, i.e. K s ( x , y ) = ∏ d glyph[lscript] =1 K s glyph[lscript] ( x glyph[lscript] , y glyph[lscript] ) for any x , y ∈ T d . We refer to this kernel as the Bessel kernel , and the associated RKHS as H s . It is stable by product as K s ( x, y ) = K s / 2 ( x, y ) 2 . This is key to compute the Fourier transform of the model g , and in contrast to previous approaches which used the exponential kernel with ̂ q ω ∝ e -s | ω | Woodworth et al. [2022], Bach and Rudi [2023].

In the following, g is a K-SoS model defined as in definition 1, with the Bessel kernel of parameter s ∈ R d + defined in eq. (14).

Lemma 1 (Fourier coefficient of the Bessel kernel) . For ω ∈ Z d , the Fourier coefficient of g in ω can be computed in O ( drm 2 ) time with

Proof. From its definition in eq. (5), we rewrite g as

Now, from the definition of the Bessel kernel in eq. (14), we have that for any ( x, y, z ) ∈ T , K ( x, y ) K ( x, z ) = e -2 s e 2 s cos(2 π ( y -z ) / 2) cos 2 π ( x -( y + z ) / 2) . By definition of the modified Bessel function, the Fourier coefficient of this expression are given by I | ω | (2 s cos(2 π ( y -z ) / 2)) . Using this into eq. (16), we get the result.

The second necessary ingredient for using the certificate of theorem 2 is computing a RKHS norm for g . It relies on the inclusion of H 2 s into the bigger space of symmetric operator S ( H s ) .

Lemma 2 (Bound on the RKHS norm of g ) . g belongs to H 2 s , and ‖ g ‖ H 2 s is bounded by the Hilbert-Schmidt norm of S ( H s ) , which can be computed in O ( dm 2 + mr 2 ) time, with

Proof. Assume that d = 1 ; the reasoning can be extended to multiple dimensions with the tensor product. From the computation of the Fourier coefficient in lemma 1 and the fact that I | ω | (2 s cos(2 π · )) ≤ I | ω | (2 s ) , we have that ̂ g ω = O ( I | ω | (2 s )) hence g ∈ H 2 s . Finally, since the kernel is stable by product, H 2 s = H s glyph[circledot] H s , so we can use e.g. [Paulsen and Raghupathi, 2016, Thm. 5.16], with H 1 = H 2 = H s and S ( H s ) = H s ⊗ H s , with the operator A = ( ϕ ( z 1 ) , . . . , ϕ ( z m )) R glyph[latticetop] R ( ϕ ( z 1 ) , . . . , ϕ ( z m )) ∗ ∈ S ( H s ) .

With lemma 2, we have that the model g belongs to H 2 s , so we will naturally use ̂ λ ω = ∏ d glyph[lscript] =1 e -2 s glyph[lscript] I ω (2 s glyph[lscript] ) in theorem 2; said differently, the space H λ introduced in eq. (8) is simply H 2 s defined in eq. (14).

3.2 The algorithm: GloptiNets

We can now describe how GloptiNets yields a certificate on f . The key observation is that no matter how is obtained our model g ( R, z ) from definition 1, we will always be able to compute a certificate with theorems 2 and 3. Thus, even though optimizing eq. (9) w.r.t ( c, R, z ) is highly non-convex, we can use any optimization routine and check empirically its efficiency by looking at the certificate. Finally, thanks to its low-rank structure it is cheaper to evaluate g than evaluating its Fourier coefficient. This is formally shown in proposition 2 in appendix A, where a block-diagonal structure for the model is also introduced. That's why we first optimize sup c,g c -‖ f -c -g ‖ glyph[star] , where ‖·‖ ∗ is a proxy for the L ∞ norm, e.g. the log-sum-exp on a random batch of N points 1 :

This optimization can be carried out by any deep learning libraries with automatic differentiation and any flavour of gradient ascent. Only afterwards do we compute the certificate with theorems 2 and 3. This procedure is summed up in alg. 1.

Algorithm 1: GloptiNets

3.3 Specific implementation for the Chebychev basis

As already observed in Bach and Rudi [2023], a result on trigonometric polynomial on T d directly extends to a real polynomials on [ -1 , 1] d . The reason for that is that minimizing h on [ -1 , 1] d amounts to minimizing the trigonometric polynomial f = h ((cos 2 πx 1 , . . . , cos 2 πx d )) on T d . Note however that f is an even function in all dimension, as for any x ∈ T d , f ( x ) = f ( x 1 , . . . , -x i , . . . , x d ) . Thus, approximating f -f glyph[star] with a K-SoS model of definition 1 is suboptimal, in the sense that we could approximate f only on [0 , 1 / 2] d , which is 2 -d smaller. Put differently, the Fourier coefficient of f are real by design: it would be convenient to enforce this structure in the model g . This is achieved with proposition 1.

Proposition 1 (Kernel defined on the Chebychev basis) . Let q be a real, even function on the torus, bounded by 1 , as in eq. (6) . Let K be the kernel defined on [ -1 , 1] by

Then K is a symmetric, p.d., hence reproducing kernel, bounded by 1 , with explicit feature map given by

The proof is available in appendix B. It simply relies on expanding the definition of K in eq. (19). The resulting expression in eq. (20) exhibits only cosine terms (in the decomposition of x ↦→ K (cos 2 πx, y ) ). This enables to directly extend the PSD models from definition 1 with such kernels. Finally, when used with the Bessel kernel of eq. (14), we recover an easy computation of the Chebychev coefficient, as shown in lemma 3, in O ( drm 2 ) time. This enables to approximate any function expressed on the Chebychev basis. Note that polynomials expressed in other basis can be certified too, by first operating a change of basis.

4 Experiments

The code to reproduce these experiments is available at github.com/gaspardbb/GloptiNets.jl

Settings. Given a function h , we compute a candidate ̂ x with gradient descent and multiple initializations. The goal is then to certify that ̂ x is indeed a global minimizer of h . This is a common setup in the Polynomial-SoS literature Wang and Magron [2022]. To illustrate the influence of the number of parameters, the positive model g defined in definition 1 for GloptiNets designates either a small model GN-small with 1792 parameters, or a bigger model GN-big with 22528 parameters. The latter should have higher expressivity and better interpolate positive functions, leading to tighter certificates. All results for GloptiNets are obtained with confidence 1 -δ = 1 -e -4 ≥ 98% . All other details regarding the experiments are reported in appendix C.

10

1

10

2

10

3

10

-

2

10

-

1

# params

Certificate

poly,

‖

h

‖

H

= 1

poly,

‖

h

‖

H

= 2

kernel,

‖

h

‖

H

= 1

kernel,

‖

h

‖

H

= 2

Polynomials. We first consider the case where h is a random trigonometric polynomial. Note that this is a restrictive analysis, as GloptiNets can handle any smooth functions ( i.e. with infinite non-zero Fourier coefficients). Polynomials have various dimension d , degree p , number of coefficients n , but a constant RKHS norm H 2 1 d . We compare the performances of GloptiNets to TSSOS, in its complex polynomial variant Wang and Magron [2022]. The latter is used with parameters such that it executes the fastest, but without guarantees of convergence to the global minimum f glyph[star] . Table 1 shows the certificates h ( x glyph[star] ) -h ( ̂ x ) and the execution times (lower is better, t in seconds) for TSSOS, GN-small and GN-big. Figure 2 provides certificate on a random polynomial, function of the number of parameters in g .

Kernel mixtures. While polynomials provide ground for comparison with existing work, GloptiNets is not confined to this function class. This is evidenced by experiments on kernel mixtures, where our approach stands as the only viable alternative we are aware of. The function we certify are of the form h ( x ) = ∑ n i =1 α i K ( x i , x ) , where K is the Bessel kernel of eq. (14). Kernel mixtures are ubiquitous in machine learning and arise e.g. when performing kernel ridge regression. Certificates obtained on mixtures are compared with those obtained on polynomials in fig. 2, function of the model size g .

Results. There are two key hindsight about the performances of GloptiNets. Firstly, its certificate does not depend on the structure of the function to optimize. Thus, although GloptiNets does not match the performances of TSSOS on small polynomials, it can tackle polynomials which cannot be handled by competitors, with arbitrarily as many coefficients ( n = ∞ ). For instance, TSSOS cannot handle problems with n ∈ { 489 , 833 } in table 1. More importantly, GloptiNets can certify a richer class of functions than polynomials, among which kernel mixtures. The performances of GloptiNets mostly depends on the complexity of the function to certify, as measured with its RKHS norm.

Secondly, note that a bigger model yields tighter certificate . This is detailed in fig. 2, where the same function f is optimized with various models. The dependency of the certificate on the norm of f is shown in fig. 3 in appendix C, along with experiments with Chebychev polynomials.

5 Limitations

One limitation of GloptiNets is the trade-off resulting from its high flexibility for obtaining a certificate as in alg. 1. While this flexibility offers numerous advantages, it also introduces the need for an extensive hyperparameter search. Although we have identified a set of hyperparameters that align with deep learning practices - utilizing a Momentum optimizer with cosine decay and a large initial learning rate - the optimal settings may vary depending on the specific problem at hand.

In the same vein, the certificates given by GloptiNets are of moderate accuracy. While adding more parameters into the k-SoS model certainly helps (as shown in fig. 2), alternative optimization scheme to interpolate h -h ( ̂ x ) with g might provide easier improvement. For instance, we found that using approximate second-order scheme in alg. 1 is key to obtaining good certificates.

In the specific settings of polynomial optimization, we highlight that our model is not competitive on problems which exhibits some algebraic structure, as for instance term sparsity or the constant trace property. Typically, problems with coefficients of low degrees (less or equal than 2 ), which encompass notably the OPF problem, are really well handled by the family of solvers TSSOS belongs to. Finally, GloptiNets does not handle constraints yet.

6 Conclusion

The GloptiNets algorithm presented in this work lays the foundation for a new family of solvers which provide certificates to non-convex problems. While our approach does not aim to replace the well-established Lasserre's hierarchy for sparse polynomials, it offers a fresh perspective on tackling a new set of problems at scale. Through demonstrations on synthetic examples, we have showcased the potential of our approach. Further research directions include extensive parameter tuning to obtain tighter certificates, with the possibility of leveraging second-order optimization schemes, along with warm-restart schemes for application which requires solving multiple similar problems sequentially.

Acknowledgments. AR acknowleges support of the French government under management of Agence Nationale de la Recherche as part of the 'Investissements d'avenir' program, reference ANR19-P3IA-0001 (PRAIRIE 3IA Institute). AR acknowledges support of the European Research Council (grant REAL 947908). JM was supported by the ERC grant number 101087696 (APHE-LAIA project) and by ANR 3IA MIAI@Grenoble Alpe (ANR-19-P3IA-0003).

References

Francis Bach and Alessandro Rudi. Exponential convergence of sum-of-squares hierarchies for trigonometric polynomials. SIAM Journal on Optimization , 33(3):2137-2159, 2023.

Eloïse Berthier, Justin Carpentier, Alessandro Rudi, and Francis Bach. Infinite-Dimensional Sumsof-Squares for Optimal Control. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 577-582, December 2022. doi: 10.1109/CDC51059.2022.9992396.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Luc Devroye, Matthieu Lerasle, Gabor Lugosi, and Roberto I. Oliveira. Sub-Gaussian Mean Estimators. The Annals of Statistics , 44(6):2695-2725, 2016.

Dinh D˜ ung, Vladimir N. Temlyakov, and Tino Ullrich. Hyperbolic Cross Approximation. arXiv:2211.04889 , April 2017.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT press, 2016.

Didier Henrion, Milan Korda, and Jean-Bernard Lasserre. The Moment-SOS Hierarchy , volume 4 of Optimization and Its Applications . World Scientific Publishing Europe Ltd., December 2020. doi: 10.1142/q0252.

Joseph J. Hilling and Anthony Sudbery. The geometric measure of multipartite entanglement and the singular values of a hypermatrix. Journal of Mathematical Physics , 51(7):072102, July 2010.

Reiner Horst and Panos M Pardalos. Handbook of global optimization , volume 2. Springer Science &Business Media, 2013.

Cédric Josz and Daniel K. Molzahn. Lasserre Hierarchy for Large Scale Polynomial Optimization in Real and Complex Variables. SIAM Journal on Optimization , 28(2):1017-1048, January 2018.

Jean B. Lasserre. Global Optimization with Polynomials and the Problem of Moments. SIAM Journal on Optimization , 11(3):796-817, January 2001. doi: 10.1137/S1052623400366802.

Jean Bernard Lasserre. Moments, Positive Polynomials and Their Applications , volume 1 of Series on Optimization and Its Applications . October 2009. doi: 10.1142/p665.

Monique Laurent and Lucas Slot. An effective version of schmüdgen's positivstellensatz for the hypercube. Optimization Letters , September 2022. doi: 10.1007/s11590-022-01922-5.

Ngoc Hoang Anh Mai, J. B. Lasserre, Victor Magron, and Jie Wang. Exploiting Constant Trace Property in Large-scale Polynomial Optimization. ACM Transactions on Mathematical Software , 48(4):40:1-40:39, December 2022.

Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Non-parametric Models for Non-negative Functions. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 12816-12826. Curran Associates, Inc., 2020.

Pablo Moscato et al. On evolution, search, optimization, genetic algorithms and martial arts: Towards memetic algorithms. Caltech concurrent computation program, C3P Report , 826(1989):37, 1989.

Boris Muzellec, Adrien Vacher, Francis Bach, François-Xavier Vialard, and Alessandro Rudi. Nearoptimal estimation of smooth transport maps with kernel sums-of-squares. arXiv:2112.01907 , December 2021.

Vern I. Paulsen and Mrinal Raghupathi. An Introduction to the Theory of Reproducing Kernel Hilbert Spaces . Cambridge Studies in Advanced Mathematics. Cambridge University Press, Cambridge, 2016. doi: 10.1017/CBO9781316219232.

Alessandro Rudi and Carlo Ciliberto. PSD Representations for Effective Probability Models. In Advances in Neural Information Processing Systems , volume 34, pages 19411-19422. Curran Associates, Inc., 2021.

Alessandro Rudi, Ulysse Marteau-Ferey, and Francis Bach. Finding Global Minima via Kernel Approximations. arXiv:2012.11978 , December 2020.

Walter Rudin. The Basic Theorems of Fourier Analysis. In Fourier Analysis on Groups , chapter 1, pages 1-34. John Wiley & Sons, Ltd, 1990. doi: 10.1002/9781118165621.ch1.

Ingo Steinwart and Andreas Christmann. Support vector machines . Springer Science & Business Media, 2008.

Pascal Van Hentenryck. Machine Learning for Optimal Power Flows. INFORMS Tutorials in Operations Research , October 18.

Peter JM Van Laarhoven, Emile HL Aarts, Peter JM van Laarhoven, and Emile HL Aarts. Simulated annealing . Springer, 1987.

Hayato Waki, Sunyoung Kim, Masakazu Kojima, and Masakazu Muramatsu. Sums of Squares and Semidefinite Program Relaxations for Polynomial Optimization Problems with Structured Sparsity. SIAM Journal on Optimization , 17(1):218-242, January 2006. doi: 10.1137/050623802.

Irène Waldspurger, Alexandre d'Aspremont, and Stéphane Mallat. Phase Recovery, MaxCut and Complex Semidefinite Programming, July 2013.

Jie Wang and Victor Magron. Exploiting Sparsity in Complex Polynomial Optimization. Journal of Optimization Theory and Applications , 192(1):335-359, January 2022.

Jie Wang, Victor Magron, and Jean-Bernard Lasserre. Chordal-TSSOS: A Moment-SOS Hierarchy That Exploits Term Sparsity with Chordal Extension. SIAM Journal on Optimization , 31(1): 114-141, January 2021a. doi: 10.1137/20M1323564.

Jie Wang, Victor Magron, and Jean-Bernard Lasserre. TSSOS: A Moment-SOS Hierarchy That Exploits Term Sparsity. SIAM Journal on Optimization , 31(1):30-58, January 2021b. doi: 10.1137/19M1307871.

N. Watson. A Treatise on the Theory of Bessel Functions . Cambridge University Press, 1922.

Blake Woodworth, Francis Bach, and Alessandro Rudi. Non-Convex Optimization with Certificates and Fast Rates Through Kernel Sums of Squares. In Proceedings of Thirty Fifth Conference on Learning Theory , pages 4620-4642. PMLR, June 2022.

A Extensions

We explore additional extensions of GloptiNets that further enhance its appeal. We first describe a block diagonal structure for the model for faster evaluation, a theoretical splitting scheme for optimization, and finally a warm-start scheme.

A.1 Block diagonal structure for efficient computation

Without any further assumption, we see that a model from definition 1 can be evaluated in O ( drm ) time; its Fourier coefficient given by lemma 1 in O ( dm 2 r ) ; the bound on the RKHS norm is computed in O ( dm 2 + mr 2 ) time thanks to lemma 2; all that enables to compute a certificate, as stated in theorem 2, in O ( Ndm 2 r + mr 2 ) time, where N is the number of frequencies sampled. If the function f to be minimized has big H s norm, we might need a large model size m to have f -f glyph[star] ≈ g . Hence, we introduce specific structure on G which makes it block-diagonal and better conditioned .

Proposition 2 (Block-diagonal PSD model) . Let g be a PSD model as in definition 1, with m = bs anchors. Split them into b groups, denoting them z ij , i ∈ [ b ] and j ∈ [ s ] . Compute the Cholesky factorization of each kernel matrix T glyph[latticetop] i T i = K z i ∈ R s × s . Then, define G as a block-diagonal matrix, with b blocks defined as G i = ˜ R i ˜ R glyph[latticetop] i , ˜ R i = T -1 R i , and R i ∈ R r × s . Equivalently,

Then g can be evaluated in O ( rbs 3 d ) time, ̂ g ω in O ( bs 2 ( dr + s )) time, and ‖ g ‖ 2 S ( H s ) in O ( b 2 ( rs 2 + r 2 s ) + bs 3 ) time. The model has ( r + d ) bs real parameters.

Proof. Having G defined as such, it is psd, of rank at most rb ≤ sb = m . Written g ( x ) = ∑ b i =1 ‖ ˜ R glyph[latticetop] i K z i ( x ) ‖ 2 , we can compute the Fourier coefficient by applying lemma 1 to each of the b component. Adding the cost of computing G i = ˜ R i ˜ R glyph[latticetop] i results in complexity of O ( bs 2 ( dr + s )) . Finally, note that ‖ g ‖ 2 S ( H s ) = ‖ A ‖ 2 S ( H s ) where

Then, defining Q the matrix of b × b blocks of size s × s s.t. for j, k ∈ [ b ] , Q jk = K ( z j , z k ) ∈ R s × s , we have

and each term in the sum can be written Tr( ˜ R glyph[latticetop] j Q jk ˜ R k )( ˜ R glyph[latticetop] k Q kj ˜ R glyph[latticetop] j ) = ‖ ˜ R glyph[latticetop] j Q jk ˜ R k ‖ 2 HS , which is computed in O ( rs 2 + r 2 s ) time, plus O ( bs 3 ) to compute the Cholesky factor.

Denoting ϕ z i = ( ϕ ( z ij )) 1 ≤ j ≤ s , note that

with E i = ϕ z i T -1 i an orthonormal basis of Span( ϕ z ij ) 1 ≤ j ≤ s as E ∗ i E i = I s . Thus, each model's coefficient is defined on an orthonormal basis, which makes the optimization easier. Of course, this comes at an added s 3 complexity, which could be alleviated by using e.g. an incomplete Cholesky factorization instead.

Remark 4 (Relation to Term Sparsity in POP) . The successful application of polynomial hierarchies to problems with thousands of variables rely on making the moment matrix M having a block structure Wang et al. [2021b,a]. If the monomial basis has size m , the constraint M glyph[followsequal] 0 is replaced with M = (Diag M i ) i ∈ [ b ] and M i glyph[followsequal] 0 . This enables to solve b SDP of size at most s instead of one of size m . Our model in proposition 2 follows a similar route for having a lower computational budget.

A.2 Global optimization with splitting scheme

While GloptiNets can provide certificates for functions, it falls behind local solvers in terms of competitiveness. The challenge lies in the fact that finding a certificate is considerably more difficult than finding a local minimum, as it necessitates the uniform approximation of the entire function. However, we present a novel algorithmic framework that has the potential to enhance the competitiveness of GloptiNets with local solvers while simultaneously delivering certificates. Our approach involves partitioning the search domain into multiple regions and computing lower bounds for each partition. By discarding portions of the domain where we can certify that the function exceeds a certain threshold, the algorithm progressively simplifies the optimization problem and removes areas from consideration. Moreover, such an approach is naturally well suited to parallel computation.

The algorithm relies on a divide-and-conquer mechanism. First, we split the hypercube ( -1 , 1) d in N regions, where N is the number of core available. We compute an upper bound with a local solver. For each region, we run GloptiNets in parallel , computing a certificate at regular interval. As soon as the certificate is bigger than the upper bound, we stop the process: we know that the global minimum is not in the associated region. We can then reallocate the freed computing power by splitting the biggest current region, which yields an easier problem. We stop as soon as the region considered are small enough. This is summarized in alg. 2, where P indicates the loop run in parallel.

Note that minimizing f on a hypercube of center µ and size σ amounts to minimizing x ↦→ f (( x -µ ) /σ ) on [ -1 , 1] d , which is another Chebychev polynomial whose coefficients can be evaluated efficiently thanks to the order-2 relation every orthonormal polynomial satisfy. For Chebychev polynomials, that is H ω +1 ( x ) = 2 xH ω ( x ) -H ω -1 ( x ) .

Algorithm 2: Splitting scheme with GloptiNets

A.3 Warm restarts

Our model distinguishes itself by leveraging the analytical properties of the objective function, rather than relying solely on algebraic characteristics. This approach offers a notable advantage, as closely related functions can naturally benefit from a warm restart. For example, if we already have a certificate for a function f using a PSD model g , and we seek to compute a certificate for a similar function ˜ f ≈ f , we can readily employ GloptiNets by initializing the PSD model with g . Indeed, if f -f glyph[star] ≈ g , we can expect ˜ f -˜ f glyph[star] ≈ g , so we can expect the optimization to be faster.

In contrast, P-SoS methods, which rely on SDP programs, cannot directly adapt to new problems without significant effort. For instance, if a new component is introduced, an entirely new SDP must be solved. Our model's ability to accommodate related yet distinct problems could prove highly valuable in domains with a frequent need to certify different but closely related problems. In the industry, the Optimal Power Flow (OPF) problem requires periodic solves every 5 minutes

Van Hentenryck [18]. With GloptiNets, once the initial challenging solve is performed, subsequent solves become easier assuming minimal changes in supply and demand conditions.

A.4 Optimizing the certificate directly

As explained in section 3.2 where GloptiNets is introduced, we optimize a proxy of the L ∞ norm rather than the certificate of theorems 2 and 3. This proxy is the log-sum-exp on a random batch of N points. The reason for this is that evaluating an extended k-SoS model g ( x ) on x ∈ T d requires O ( drs ) time, while evaluating ̂ g ω on ω ∈ Z d requires O ( drs 2 ) time. Yet, optimizing the certificate directly could probably help obtaining higher-precision certificate. Lemma 4 in appendix D sketches a method to reduce the computational cost of the Fourier computation from O ( s 2 ) to O ( s ) .

B Kernel defined on the Chebychev basis

In this section we describe the approach we take to model functions written in the Chebychev basis. For h such a polynomial, a naive approach would simply model f = h ◦ cos(2 π · ) as a trigonometric polynomial. However, note that the decomposition of f only has cosine terms. Thus, approximating f -f glyph[star] efficiently requires a PSD model which has only cosine terms in its Fourier decomposition. This is achieved by using a kernel written in the Chebychev basis, as introduce in proposition 1, for which we now provide a proof.

Proof of proposition 1. Let x, y ∈ [ -1 , 1] and u, v ∈ [0 , 1 / 2 ] s.t. x, y = cos(2 πu ) , cos(2 πv ) , by bijectivity of the cosine function on [0 , π ] . From the definition of K in eq. (19) and the definition of q in eq. (6), we have that

glyph[negationslash]

Since q has positive Fourier transform, this makes the feature map of K explicit with K ( x, y ) = ϕ ( u ) · ϕ ( v ) , ϕ ( u ) ω = √ (1 + 1 ω =0 ) ̂ q ω H ω ( u ) , for ω ∈ N . Hence the kernel is a reproducing kernel.

We now use this kernel with the Bessel function x ↦→ e s (cos(2 πx ) -1) , i.e. we define the kernel K on [ -1 , 1] to satisfy

As it was the case for the torus, this kernel enables an easy characterization of a RKHS in which an associated PSD model g lives.

Lemma 3 (Chebychev coefficient of the Bessel kernel) . Let g be a PSD model as in definition 1, with the kernel K of eq. (24) . Then, the Chebychev coefficient ω ∈ N d of g can be computed in O ( rdm 2 ) time with glyph[negationslash]

where

Proof.

Expanding g and definition of Chebychev coefficient. From the definition of g in eq. (5), we have

We consider x, y, z ∈ ( -1 , 1) and s > 0 . We denote u, v, w ∈ (0 , 1 / 2) s.t.

with the bijectivity of x ↦→ cos(2 πx ) on (0 , 1 / 2) . We now compute the Chebychev coefficient of x ↦→ K s ( x, y ) K s ( x, z ) . Denoted p ω , this is glyph[negationslash]

or equivalently

glyph[negationslash]

Chebychev coefficient of kernel product. With the definition of the kernel in proposition 1, eq. (19), we have

Now use the sum-to-product formula with the cosines to obtain

We simplify this expression by introducing

Then, eq. (28) becomes

We recognize the definition of the kernel (which is not a surprise as we chose the kernel to be stable by product). However, we need variables in (0 , 1 / 2) to retrieve the proper definition of the kernel. Instead, we use lemma 5 on eq. (30) combined with eq. (27), to obtain glyph[negationslash]

which gives

glyph[negationslash]

Equation (31) contains the Chebychev coefficient of the product of two kernel function as defined in eq. (27). Plugging this result into the definition of g in eq. (26), and noting that cos(2 πωm ± ) = H ω (cos 2 πm ± ) = H ω ( σ ± ) , we obtain the result.

1

2

3

4

5

6

7

8

10 9

20

10

-

1

.

5

10

-

1

.

2

‖

f

‖

2

Certificate

Thanks to lemma 3, we see that a model g defined as in definition 1 with the Bessel kernel K s of eq. (24) as its Chebychev coefficients decaying in O ( I ω (2 s )) . Hence, it belongs to H 2 s , the RKHS associated to K 2 s .

C Additional details on the experiments

Tuning the hyperparameters. The time reported in section 4 does not take into account the experiments needed to find a good set of hyperparameters. The parameters tuned were the type of optimizer, the decay of learning rate, and the regularization on the Frobenius norm of G .

Regularization. Regularization is performed by approximating the HS norm with a proxy which is faster to compute. We use ‖ R glyph[latticetop] j R k ‖ 2 HS instead of ‖ ˜ R glyph[latticetop] j Q jk ˜ R k ‖ 2 HS in eq. (22).

Hardware. GloptiNets was used with NVIDIA V100 GPUs for the interpolation part, and Intel Xeon CPU E5-2698 v4 @ 2.20GHz for computing the certificate. TSSOS was run on a Apple M1 chip with Mosek solver.

Configuration of TSSOS. We use the lowest possible relaxation order d ( i.e. glyph[ceilingleft] deg f/ 2 glyph[ceilingright] ), along with Chordal sparsity. We use the first relaxation step of the hierarchy. In these settings, TSSOS is not guaranteed to converge to f glyph[star] but will executes the fastest.

Certificate vs. number of parameter for a given function. In fig. 2, the target function is a random polynomial of norm 1 or 2 , or a kernel mixture with 10 coefficients of norm 1 or 2 . The models forming the blue line are defined as in proposition 2, with rank, block size and number of blocks equal to (1 , bs, 1) respectively, with bs the block size we vary. The number of frequencies sampled to compute the certificate is 1 . 6 · 10 7 , and accounts for the fact that the bound on the variance becomes larger than the MOM estimator for large models.

Certificate vs. problem difficulty for a given model. We have 3 related parameters: the quality of the optimization (given by the certificate), the expressivity of the model (given by its number of parameters), and the difficulty of the optimization (given by the norm of the function). In fig. 3, we fix the latter and plot the relation between the first two. Here, we fix the model with parameters (8 , 16 , 128) , and we optimize a polynomial in 3 d of degree 12 , with RKHS norm ranging from 1 to 20 . The certificates obtained are given in fig. 3. The resulting plot exhibits a clear polynomial relation between the certificate and the norm of the function, with a slope of -0 . 88 . This suggest that the certificate behaves as O ( ‖ f ‖ 1 / 2 H 2 s ) .

Comparison with TSSOS on the Fourier basis. In table 1, the polynomials f all have a RKHS norm of 1 . The small model is defined as in proposition 2, with rank, block size and number of blocks equal to 4 , 32 , 8 respectively. For the big models, those values are 8 , 128 , 16 . The certificate is the

maximum of the Chebychev bound of theorem 2 and the MoM bound of theorem 3. The number of frequencies sampled is 3 . 2 · 10 7 .

Comparison with TSSOS on the Chebychev basis. We compare GloptiNets with TSSOS on random Chebychev polynomials in table 2, similarly to the comparison with trigonometric polynomials in table 1. Minimizing polynomials defined on the canonical basis is easier: contrary to trigonometric polynomials, there is no need to account for the imaginary part of the variable. If d is the dimension, complex polynomials are encoded in a variable of dimension 2 d in TSSOS, following the definition of Hermitian Sum-of-Squares introduced in Josz and Molzahn [2018]. Hence, the random polynomials we consider are characterized by the dimension d and their number of coefficients n ; instead of bounding the degree, we use all the basis elements H ω ( x ) = ∏ d glyph[lscript] =1 H ω glyph[lscript] ( x glyph[lscript] ) for which ‖ ω ‖ ∞ ≤ p . The maximum degree is then dp . The RKHS norm of f is fixed to 1 . As with the comparison on Trigonometric polynomial table 1, we see that GloptiNets provides similar certificates no matter the number of coefficients in f . Even though it lags behind TSSOS for small polynomials, it handles large polynomials which are intractable to TSSOS. The 'small' and 'big' models have the same structure as for the trigonometric polynomials experiments.

Sampling from the Bessel distribution. The function ω ↦→ e -s I ω ( s ) decays rapidly. In fact, with s = 2 , which is the value used to generate the random polynomials, it falls under machine precision as soon as ω > 14 . Thus, we approximate the distribution with a discrete one with weights I ω ( s ) for ω s.t. the result is above the machine precision. We then extend it to multiple dimension with a tensor product. Finally, we use a hash table to store the already sampled frequency, to make the evaluation of million of frequencies much faster. For instance in dimension 5 , sampling 10 6 frequencies from the Bessel distribution of parameter s = 2 on N 5 yields only ≈ 10 4 unique frequencies. This allows for tighter certificates, as it makes the r.h.s of eq. (9), in 1 / N , much smaller. Note that the time to generate this hash table is not reported in tables 1 and 2, and of the order of a few seconds.

Optimizing a kernel mixture. As it is the case with polynomials, when optimizing a function of the form h ( x ) = ∑ n i =1 α i K ( x i , x ) the certificate provided by GloptiNets only depends on the function norm ‖ h ‖ 2 H and not on e.g. the number of coefficients n . This is illustrated in fig. 4.

D Fourier coefficients in linear time

Lemma 4 (Fourier coefficient of the Bessel kernel in linear time) . Let g be an extended k-SoS model as in definition 1. Then, its Fourier transform can be evaluated in linear time in m with

where

and q · , · ( s ) is defined with lemma 6.

Lemma 4 provides a formula for computing ̂ g ω which is linear in m , but which still requires numerical approximation to compute the sum on n ∈ Z d . For instance, restraining the sum to the hyperbolic

10

1

10

2

10

3

10

-

2

10

-

1

10

0

# params

Certificate

‖

h

‖

H

= 1

,

n

= 10

‖

h

‖

H

= 1

,

n

= 100

‖

h

‖

H

= 2

,

n

= 10

‖

h

‖

H

= 2

,

n

= 100

cross D˜ ung et al. [2017]

would result in a complexity of O ( n (log d ) n mr ) and should produce reasonably accurate estimate of ̂ g ω for low n .

Furthermore, since q is real-even w.r.t n , the inner-product in eq. (36) can be simplified by computing only half of the terms.

Proof. From lemma 1, we have that

Introducing

eq. (33) simplifies to

Using lemma 6, for any x, y ∈ T ,

( q glyph[lscript],n depends on ω glyph[lscript] ) so that, f glyph[lscript] defined in eq. (34) now writes

where, for any glyph[lscript] ∈ { 1 , . . . , d } and z ∈ T , we defined

We then define the embedding φ ± : T → ( Z d → C ) be the tensor product of the φ glyph[lscript], ± . Then, eq. (36), enables to write ̂ g ω in eq. (35) as

E Other computation

Lemma 5. Let f be the function defined on ( -1 , 1) with

Then, its Chebychev coefficient are given with

glyph[negationslash]

Proof. The ω ∈ N ∗ . The component ω of a function f on the Chebychev basis is given with

which we conveniently rewrite, with the classical change of variable x = cos 2 πu ,

which is valid for any interval I 1 ⊂ R of length 1 .

Now, for s > 0 , consider the function f defined on ( -1 , 1) with x ↦→ e s cos(arccos( x ) -2 πv ) , or equivalently

Putting eq. (41) into eq. (40), we obtain

The last term is odd, hence integrate to 0 on an interval centered around 0 . Hence,

We recognize the definition of the modified Bessel function of the first kind, defined in eq. (14). Plugging this into eq. (42), we obtain

If ω = 0 , we add a factor 1 / 2 into the definition in eq. (40), which yields

Lemma 6 (Fourier decomposition of Bessel composed with cosine) . Let s > 0 , ω ∈ N and z ∈ T . Then,

and q ω, -n = q ω,n by evenness of the coefficients.

Proof. From the definition of the modified Bessel function of the first kind [Watson, 1922, p.77, Eq. 2], we have

so that

Using the change of variable n = 2( p -k ) + ω into eq. (46), we see that n has the same parity as ω and

Equation (47) can be rewritten

for which eq. (45) is a concise rewriting.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on non-convex global optimization on the hypercube. \nThe approach is built on top of the framework [7] and relies on non-negativity certificates that are not only restricted to non-negative polynomials since it is applicable to any function with computable Fourier coefficients. \nThe methodology is flexible in the sense that one can increase the certificate of the obtained certificate by relying on the so-called k-SOS method. The approach is naturally compatible with GPU computation thanks to a modular Bessel kernel, which is stable by product. The resulting optimization method, called GloptiNets, can be handled with automatic differentiation. Afterwards the algorithm is benchmarked against the TSSOS library designed for sparse polynomial optimization on six problems of dimension 3 or 4. The performance are similar or less good when the number of coefficients is small, and certificates are obtained with GloptiNets on examples for which the SDP solver used by TSSOS runs out of memory.\n\nSTRENGTHS:\nThe methodology allows a certain flexibility as it is not restricted to objective polynomial functions and does not require any assumptions on the structure (e.g., sparsity, symmetry) apart from the hypercube constraint. \nOne can also increase the accuracy of the certificate by considering bigger nets.\n\nWEAKNESSES:\nSome technical explanation are not easy to follow (see the questions below). \n\nThe title of the paper is slightly misleading, I suggest to emphasize that the optimization is performed on the hypercube.\n\nThe paper only provides numerical comparisons for low dimensions (the number of variables is either 3 or 4) and the corresponding benchmarks are cooked by hand (called synthetic in the conclusion), thus do not correspond to any concrete optimization problem.  Even if it is promising, I believe that this methodology in its current state is not yet convincing for optimizers interested in either academic or industrial applications. \n\nIn addition, the accuracy of the obtained certificate is low compared to interior-point methods. What would be the amount of computational time required to obtain a similar accuracy with GN? On Figure 2, the number of parameters required to go from 0.1 to 0.02 increases rather quickly. This can also be concluded from the numbers in Table 1. For problem 3, it is approximately 3 times slower to double the accuracy of the resulting certificate (going from 1.3e-2 to 6.9e-3). So one could believe that it would take a rather considerable amount of time to reach an accuracy of 1e-8.\n\nI believe that the statement \"TSSOS is not guaranteed to converge to f* but executes faster, and thus is on an equal footing with GloptiNets\" is quite misleading for two reasons: \n\n1) The efficiency of TSSOS increases when the optimization problem involves sparser polynomials (with low n), which is not the case here for problems 3 and 6.\n\n2) The efficiency of TSSOS relies on the one of the tool used to solve the SDP. An alternative SDP solver such as COSMO would be certainly much more efficient but would also yield certificates with lower accuracy.\n\nSimilarly the sentence \"poly-SoS methods (whose complexity scales dramatically and in a rigid way with dimension and degree of the polynomial)\" is misleading. For a given certificate accuracy, there is no established estimate showing that the complexity of poly-SOS methods is higher. \nSo I strongly suggest to the authors to either reformulate this statement or to justify it with rigorously established statements. \nThe rigidity statement is also not really accurate as any structure-exploiting variant of the Moment-SOS hierarchy comes with a specific computational cost and convergence rates. \n\nThere is a potentially high source of conservativeness for the resulting lower bound because of the difference between the L-infinity and Fourier norms.\n\nQUESTIONS:\nl123: each z_i is a vector of T^d so shouldn't we have Z \\subset T^d?\n\nl124: which norm is used?\n\nAfter l154: the matrix R suddenly (re-)introduced in Equation (7) is rather confusing. Where does this matrix come from? I guess that it is related to R from definition (1).\n\nFigure 1 is not mentioned in the text and I could not understand it. The F norm color is displayed in orange, not in red. \n\n(8) The expectation notation differs from (7) as there is no sub-index anymore.\n\nIn Figure 2, I cannot see any red.\n\nCould one obtain estimates of the difference between the L-infinity and Fourier norms? Is there a specific class of examples where this difference is significant?\n\nLIMITATIONS:\nThere is a clear section dedicated to limitations with an explicit list:\n- an extensive hyper-parameter search is required\n- the structure is not exploited\n- the setting is restricted to the hypercube\n- the benchmarks are somehow artificial\n\nTo be more convincing, the framework should be adapted to take into account the structure of real-world problems, e.g., the AC-OPF problem mentioned by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents GloptiNets, a method to bound the suboptimality of a given candidate solution to the optimization of a smooth (nonconvex) function on the hypercube.\n\nNonconvex optimization with optimality certificates is difficult. Oftentimes it is relatively easy (fast) to compute a candidate solution, but it is much more difficult (computationally expensive) to bound the suboptimality of the solution. A popular way to provide optimality certificates is via polynomial sums of squares (Lasserre's hierarchy), but it (a) can only handle functions that are polynomials and (b) scales poorly to the dimension of the problem due to computational bottlenecks in semidefinite programming.\n\nGloptiNets tries to tackle the drawbacks of poly-SOS. The key observation is that, given a candidate solution, any positive function provides a lower bound, and hence a bound on the suboptimality. With this key observation, and considering the recent success of machine learning, the overall idea of GloptiNets is to seek a positive function parametrized by \"neural networks\" (more precisely the anchor points $z$ and the matrix $R$ in Definition 1.)\n\nThe rest of the paper mostly focuses on choosing the Kernel function and leveraging sampling (and concentration inequalities) to evaluate the loss function more efficiently (for which the reviewer did not check carefully).\n\nThe experiments of this paper compare GloptiNets with TSSOS, a state-of-the-art polynomial SOS toolbox, on computing certificates for random positive trigonometric polynomials. The experiments show that although TSSOS can obtain high-accuracy certificates, GloptiNets is much more scalable.\n\nSTRENGTHS:\n- The paper tackles the important problem of nonconvex optimization with certificates. \n- The proposed GloptiNets mitigates the drawbacks of existing poly-SOS frameworks, i.e., it can handle non-polynomial problems and it does not rely on semidefinite programming. As a result, it is more scalable.\n\nWEAKNESSES:\nThe biggest issue with this paper is that (i) its experiments are rather limited (I believe there are only 6 problems tested), and (ii) the limited experiments are not even related to applications where optimality certificates are desired. Therefore, it is questionable how useful this technique will be in practice (and it kind of defeats the purpose stated in the introduction). \n\nI would encourage more experiments where the optimality certificates are indeed valuable (for example recently in [robotics](https://arxiv.org/abs/2109.03349)).\n\nQUESTIONS:\nI notice that in the results table, the best suboptimality GloptiNets can produce is in the order of $10^{-3}$. I wonder if this can be improved by, for example\n- increase the number of samples $x_1,\\dots, x_N$,\n- increase the size of the neural network (Fig. 2 partially answers this, but the accuracy there is still far from, say $10^{-9}$ as can be achieved by TSSOS)\n- let the neural networks train for longer time\n- increase the number of samples $\\omega_1,\\dots,\\omega_n$\n\nIf it is possible to achieve the same certificate accuracy as TSSOS, then it is useful to report the number of sample points and training time required to attain the accuracy; if it is not possible, then an explanation of this should be provided (because the positive function class of GloptiNets should subsume the positive function class of TSSOS, and therefore the NN should be able to achieve the same accuracy as TSSOS if enough parameters are allowed?).\n\nLIMITATIONS:\nLimitations are clearly stated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for certain non-convex optimization problems on a hypercube or torus (i.e. with periodic boundary conditions), which also provides a certificate (a measure of how close the fitted function is to the target). \nAs I understand it, the certificate computation utilizes the Fourier basis. \nThey do two sets of experiments using their model GloptiNet: one on random trigonometric and one on Chebyschev polynomials, with increasing number of coefficients. They compare their method's performance against an existing polynomial solver, TSSOS. Their model's certificates are orders of magnitude less tight (~$10^{-2}$) than TSSOS ($10^{-12}$). But their model has the benefit that it takes almost constant time, independent of the complexity of the polynomial, to get within the certificate's accuracy. They also show that the certificate gets tighter as they increase the number of model parameters, though the drop seems slower than linear.\n\nSTRENGTHS:\nThe paper gives a detailed exposition of the derivation of the certificate and discusses existing work extensively. I am no expert in this field, but I could more or less follow the process. \nThe contribution of the work, in addition to providing a certificate, seems to lie in its efficiency. They claim it should work well in higher dimensions, too, and the run time seems to be not affected much by the complexity of the target function. \nAlso, being able to use fast stochastic optimizations of DNN while still providing a certificate for the fit seems to be another benefit of the method. \nSince I am not familiar with this branch of optimization, I cannot assess the novelty that much.\n\nWEAKNESSES:\nA big emphasis is on the derivation of the certificate. While I appreciate the details, it makes it difficult to follow the flow of the paper. Instead a little more details on how the extended k-SoS model is implemented and how the optimization is done would be useful. I see discussions of the details in the appendix, but it is brushed over in the main text. For example, is the R matrix optimized, or fixed? \nAlso, the high value of the certificate and lack of hyperparameter tuning, except model size, weakens the paper (see questions). \n\nAdditionally, the target function in the two experiment, trigonometric and Chebyshev, are both related to the Fourier series, which is used in computing the certificate. I wonder how well the model performs on other polynomials with other bases (see questions).\n\nQUESTIONS:\n1. currently the certificate seems very loose at $O(10^{-2})$. Is this a good number? \n2. How much does hyperparameter tuning (other than model size) affect the certificate? Is there hope for much better values, or does it need dramatic changes?   \n3. Experiments: How would the model perform on polynomials from a dramatically different basis from the Fourier basis (e.g. $x^p$)? I understand this doesn't work on the Torus (no periodicity), but does the hypercube have a similar limitation (meaning you *have to* use the Chebyshev basis due to boundary conditions)?\n4. Baseline methods seem very limited. Is TSSOS the only viable alternative? \n5. In Fig 2, no red curve is presented, but mentioned in the caption.\n\nLIMITATIONS:\nThey discuss the limitations. I would also add discussions about the class of polynomials, if it is currently limited to Chebyshev and trigonometric polynomials. As for societal impact, I don't see any issues, as this is a purely mathematical optimization paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops an approach to compute certificates for non-convex optimization on functions that are optimized over the hypercube or torus. It does so by considering the generic certificate recipe from [7] and then relaxing the positive semi-definite constraint to a class of functions that is easier to optimize that they denote K-SoS. This class of functions depends on the choice of a kernel K, of which they explore different alternatives. The authors consider optimization aspects of this general framework and provide a concrete algorithm that they name GloptiNets. They also provide experiments comparing the tightness of the certificate across the number of parameters and related approaches.\n\nSTRENGTHS:\n1. The paper is clear and beautifully written. It is a pleasure to read.\n\n2. Results are sound to the best of my knowledge. The theoretical derivations are clear and give the proper references to understand which steps have been taken.\n\n3. The authors consider all relevant aspects of the implement of this framework: from expressiveness properties of the function approximation, to probabilistic estimates of the certificate, to practical implementation details.\n\nWEAKNESSES:\nNo major weaknesses identified. Below are some minor suggestions that I hope the authors consider, but should not be taken as criticism towards the work.\n\n\n## Minor suggestions\n\n1. Add x and y labels to Figure 1.\n2. Some aspects of this paper, such as the lower bound formula (2) or the anchor points Z are introduced assuming the reader knows about the related works. I would encourage the authors to aim for a larger audience. In particular, a few words about the intuition can go a long way towards making the work more accessible.\n\n## Typos \n\nL244: eventhough -> even though\nL245 optimisation -> optimization (since American spelling is used elsewhere)\nL93: \"we focus on the computational performances of our model\" it's unclear what the authors mean here by computational performances, is it run-time? tightness of the certificate? the trade-off between these two? Please make precise\n\nQUESTIONS:\n1. What is Z_d in eq. (1)? I don't think it has been defined before.\n\n2. It was not clear to me whether the anchor points Z are given by the problem or are part of the choice of the kernel family. Clarification would be appreciated.\n\nLIMITATIONS:\nI believe the limitations have been correctly addressed by the author in the Limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

comments: SUMMARY:
They present a novel approach to non-convex optimization with certificates, which handles smooth functions on the hypercube or on the torus. Unlike traditional methods that rely on algebraic properties, our algorithm exploits the regularity of the target function intrinsic in the decay of its Fourier spectrum.

STRENGTHS:
The paper is clearly written and the theory is interesting.

WEAKNESSES:
The experimental results are not abundant.

QUESTIONS:
No.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-21