A Generalized Weighted Optimization Method for Computational Learning and Inversion

1 INTRODUCTION . Given N data pairs { xj , yj } Nj=1 , where xj ∈ R , yj ∈ C , j = 1 , . . . , N , we are interested in learning a random Fourier feature ( RFF ) model ( Rahimi & Recht , 2008 ; Liao et al. , 2020 ; Xie et al. , 2020 ) fθ ( x ) = P−1∑ k=0 θke ikx , x ∈ [ 0 , 2π ] , ( 1 ) where P ∈ N is a given positive integer and we used the short-hand notation θ : = ( θ0 , · · · , θP−1 ) T with the superscript T denoting the transpose operation . This exact model as well as its generalization to more complicated setups have been extensively studied ; see for instance Liao & Couillet ( 2018 ) ; Shahrampour & Kolouri ( 2019 ) ; d ’ Ascoli et al . ( 2020 ) ; Li et al . ( 2020 ) ; Özcelikkale ( 2020 ) ; Liu et al . ( 2020 ; 2021 ) and references therein . While this model may seem to be overly simplified from a practical perspective for many real-world applications , it serves as a prototype for theoretical understandings of different phenomena in machine learning models ( Sriperumbudur & Szabo , 2015 ; Belkin et al. , 2020 ; Li et al. , 2021a ) . A common way to computationally solve this learning problem is to reformulate it as an optimization problem where we find θ by minimizing the model and data mismatch for a given dataset . In this paper , we assume that the training data are collected on a uniform grid of x over the domain [ 0 , 2π ] . That is , { xj = 2πjN } N−1 j=0 . Let ωN = exp ( 2πi N ) where i is the imaginary unit . We introduce Ψ ∈ CN×P to be the feature matrix with elements ( Ψ ) jk = ( ωN ) jk , 0 ≤ j ≤ N − 1 , 0 ≤ k ≤ P − 1 . Based on the form of fθ ( x ) in ( 1 ) , we can then write the 2-norm based data mismatch into the form∑N−1 j=0 |fθ ( xj ) − yj |2 = ‖Ψθ − y‖22 where the column data vector y = ( y0 , · · · , yN−1 ) T. The learning problem is therefore recast as a least-squares optimization problem of the form θ̂ = arg min θ ‖Ψθ − y‖22 , ( 2 ) assuming that a minimizer does exist , especially when we restrict θ to an appropriate space . In a general feature regression problem , the Fourier feature { eikx } P−1k=0 is then replaced with a different feature model { ϕk ( x ) } P−1k=0 , while the least-squares form ( 2 ) remains unchanged except that the entries of the matrix Ψ is now Ψjk = ϕk ( xj ) . We emphasize that this type of generalization will be discussed in Section 5 . Moreover , we remark that this least-squares optimization formulation is a classical computational inversion tool in solving the general linear inverse problems of the form Ψθ = y ; see for instance Engl et al . ( 1996 ) ; Tarantola ( 2005 ) and references therein . Previous work on weighted optimization for feature and kernel learning . Xie et al . ( 2020 ) studied the fitting problem for this model under the assumption that the coefficient vector θ is sampled from a distribution with the property that γ is a positive constant , Eθ [ θ ] = 0 , Eθ [ θθ∗ ] = cγΛ−2γ [ P ] , ( 3 ) where the superscript ∗ denotes the Hermitian transpose and the diagonal matrix Λ [ P ] has diagonal elements ( Λ [ P ] ) kk = tk = 1 + k , k ≥ 0 . That is , Λ [ P ] = diag { t0 , t1 , t2 , . . . , tk , . . . , tP−1 } , tk : = 1 + k . ( 4 ) The subscript [ P ] indicates that Λ [ P ] is a diagonal submatrix of Λ that contains its element indexed in the set [ P ] : = { 0 , 1 , · · · , P − 1 } . The normalization constant cγ = 1/ ( ∑P−1 k=0 ( 1 + k ) −2γ ) is only selected so that Eθ [ ‖θ‖2 ] = 1 . It does not play a significant role in the rest of the paper . The main assumption in ( 3 ) says that statistically , the signal to be recovered has algebraically decaying Fourier coefficients . This is simply saying that the target function we are learning is relatively smooth , which is certainly the case for many functions as physical models in practical applications . It was shown in Xie et al . ( 2020 ) that , to learn a model with p ≤ P features , it is advantageous to use the following weighted least-squares formulation θ̂p = Λ −β [ p ] ŵ , with ŵ = arg minθ ‖Ψ [ N×p ] Λ −β [ p ] w − y‖ 2 2 , ( 5 ) when the learning problem is overparameterized , i.e. , p > N . Here , Ψ [ N×p ] ∈ CN×p is the matrix containing the first p columns of Ψ , and β > 0 is some pre-selected exponent that can be different from the γ in ( 3 ) . To be more precise , we define the the generalization error of the learning problem Eβ ( P , p , N ) : = Eθ [ ‖fθ ( x ) − fθ̂p ( x ) ‖ 2 L2 ( [ 0,2π ] ) ] = Eθ [ ‖θ̂p − θ‖22 ] , ( 6 ) where the equality comes from the Parseval ’ s identity , and θ̂p is understood as the vector ( θTp , 0 , · · · , 0 ) T so that θ and θ̂p are of the same length P . The subscript θ in Eθ indicates that the expectation is taken with respect to the distribution of the random variable θ . It was shown in Xie et al . ( 2020 ) that the lowest generalization error achieved from the weighted least-squares approach ( 5 ) in the overparameterized regime ( p > N ) is strictly less than the lowest possible generalization error in the underparameterized regime ( p ≤ N ) . This , together with the analysis and numerical evidence in previous studies such as those in Belkin et al . ( 2019 ; 2020 ) , leads to the understanding that smoother approximations ( i.e. , solutions that are dominated by lower Fourier modes ) give better generalization in learning with the RFF model ( 1 ) . Main contributions of this work . In this work , we analyze a generalized version of ( 5 ) for general feature regression from noisy data . Following the same notations as before , we introduce the following weighted least-squares formulation for feature regression : θ̂ δ p = Λ −β [ p ] ŵ , with ŵ = arg min w ‖Λ−α [ N ] ( Ψ [ N×p ] Λ −β [ p ] w − y δ ) ‖22 , ( 7 ) where the superscript δ on y and θ̂p denotes the fact that the training data contain random noise of level δ ( which will be specified later ) . The exponent α is pre-selected and can be different from β . While sharing similar roles with the weight matrix Λ−β [ P ] , the weight matrix Λ −α [ N ] provides us the additional ability to deal with noise in the training data . Moreover , as we will see later , the weight matrix Λ−α [ N ] does not have to be either diagonal or in the same form as the matrix Λ −β [ p ] ; the current form is to simplify the calculations for the RFF model . It can be chosen based on the a priori information we have on the operator Ψ as well as the noise distribution of the training data . The highlight and also one of the main contributions of our work is that we introduce a new weight matrix Λ−α [ N ] that emphasizes the data mismatch in terms of its various modes , in addition to Λ −β [ p ] , the weight matrix imposed on the unknown feature coefficient vector θ . This type of generalization has appeared in different forms in many computational approaches for solving inverse and learning problems where the standard 2-norm ( or ` 2 in the infinite-dimensional setting ) is replaced with a weighted norm that is either weaker or stronger than the unweighted 2-norm . In this paper , we characterize the impact of the new weighted optimization framework ( 7 ) on the generalization capability of various feature regression and kernel regression models . The new contributions of this work are threefold . First , we discuss in detail the generalized weighted leastsquares framework ( 7 ) in Section 2 and summarize the main results for training with noise-free data in Section 3 for the RFF model in both the overparameterized and the underparameterized regimes . This is the setup considered in Xie et al . ( 2020 ) , but our analysis is based on the proposed weighted model ( 7 ) instead of ( 5 ) as in their work . Second , we provide the generalization error in both two regimes for the case of training with noisy data ; see Section 4 . This setup was not considered in Xie et al . ( 2020 ) , but we demonstrate here that it is a significant advantage of the weighted optimization when data contains noise since the weighting could effectively minimize the influence of the noise and thus improve the stability of feature regression . Third , we extend the same type of results to more general models in feature regression and kernel regression that are beyond the RFF model , given that the operator Ψ satisfies certain properties . In the general setup presented in Section 5 , we derive error bounds in the asymptotic limit when P , N , and p all become very large . Our analysis provides some guidelines on selecting weighting schemes through either the parameter domain weighting or the data domain weighting , or both , to emphasize the features of the unknowns to be learned based on a priori knowledge . 2 GENERALIZED WEIGHTED LEAST-SQUARES FORMULATION . There are four essential elements in the least-squares formulation of the learning problem : ( i ) the parameter to be learned ( θ ) , ( ii ) the dataset used in the training process ( y ) , ( iii ) the feature matrix ( Ψ ) , and ( iv ) the metric chosen to measure the data mismatch between Ψθ and y . Element ( i ) of the problem is determined not only by the data but also by a priori information we have . The information encoded in ( 3 ) reveals that the size ( i.e. , the variance ) of the Fourier modes in the RFF model decays as fast as ( 1 + k ) −2γ . Therefore , the low-frequency modes in ( 1 ) dominate high-frequency modes , which implies that in the learning process , we should search for the solution vectors that have more low-frequency components than the high-frequency components . The motivation behind introducing the weight matrix Λ−β [ p ] in ( 5 ) is exactly to force the optimization algorithm to focus on admissible solutions that are consistent with the a priori knowledge given in ( 3 ) , which is to seek θ whose components |θk|2 statistically decay like ( 1 + k ) −2β . When the problem is formally determined ( i.e. , p = N ) , the operator Ψ is invertible , and the training data are noise-free , similar to the weight matrix Λ−β [ p ] , the weight matrix Λ −α [ N ] does not change the solution of the learning problem . However , as we will see later , these two weight matrices do impact the solutions in various ways under the practical setups that we are interested in , for instance , when the problem is over-parameterized or when the training data contain random noise . The weight matrix Λ−α [ N ] is introduced to handle elements ( ii ) - ( iv ) of the learning problem . First , since Λ−α [ N ] is directly applied to the data y δ , it allows us to suppress ( when α > 0 ) or promote ( when α < 0 ) high-frequency components in the data during the training process . In particular , when transformed back to the physical space , the weight matrix Λ−α [ N ] with α > 0 corresponds to a smoothing convolutional operator whose kernel has Fourier coefficients decaying at the rate k−α . This operator suppresses high-frequency information in the data . Second , Λ−α [ N ] is also directly applied to Ψθ . This allows us to precondition the learning problem by making Λ−α [ N ] Ψ a betterconditioned operator ( in an appropriate sense ) than Ψ , for some applications where the feature matrix Ψ has certain undesired properties . Finally , since Λ−α [ N ] is applied to the residual Ψθ − y , we can regard the new weighted optimization formulation ( 7 ) as the generalization of the classic leastsquares formulation with a new loss function ( a weighted norm ) measuring the data mismatch . Weighting optimization schemes such as ( 7 ) have been studied , implicitly or explicitly , in different settings ( Needell et al. , 2014 ; Byrd & Lipton , 2019 ; Engquist et al. , 2020 ; Li , 2021 ; Yang et al. , 2021 ) . For instance , if we take β = 0 , then we have a case where we rescale the classical leastsquares loss function with the weight Λ−α [ N ] . If we take α = 1 , then this least-squares functional is equivalent to the loss function based on the H−1 norm , instead of the usual L2 norm , of the mismatch between the target function fθ ( x ) and the learned model fθ̂ ( x ) . Based on the asymptotic equivalence between the quadratic Wasserstein metric and the H−1 semi-norm ( on an appropriate functional space ) , this training problem is asymptotically equivalent to the same training problem based on a quadratic Wasserstein loss function ; see for instance Engquist et al . ( 2020 ) for more detailed illustration on the connection . In the classical statistical inversion setting , Λ2α plays the role of the covariance matrix of the additive Gaussian random noise in the data ( Kaipio & Somersalo , 2005 ) . When the noise is sampled from mean-zero Gaussian distribution with covariance matrix Λ2α , a standard maximum likelihood estimator ( MLE ) is often constructed as the minimizer of ( Ψθ − y ) ∗Λ−2α [ N ] ( Ψθ − y ) = ‖Λ −α [ N ] ( Ψθ − y ) ‖ 2 2 . The exact solution to ( 7 ) , with X+ denoting the Moore–Penrose inverse of operator X , is given by θ̂ δ p = Λ −β [ p ] ( Λ−α [ N ] Ψ [ N×p ] Λ −β [ p ] ) + Λ−α [ N ] y δ . ( 8 ) In the rest of this paper , we analyze this training result and highlight the impact of the weight matrices Λ−α [ N ] and Λ −β [ N ] in different regimes of the learning problem . We reproduce the classical bias-variance trade-off analysis in the weighted optimization framework . For that purpose , we utilize the linearity of the problem to decompose θ̂ δ p as θ̂ δ p = Λ −β [ p ] ( Λ −α [ N ] Ψ [ N×p ] Λ −β [ p ] ) +Λ−α [ N ] y + Λ −β [ p ] ( Λ −α [ N ] Ψ [ N×p ] Λ −β [ p ] ) +Λ−α [ N ] ( y δ − y ) , ( 9 ) where the first part is simply θ̂p , the result of learning with noise-free data , while the second part is the contribution from the additive noise . We define the generalization error in this case as Eδα , β ( P , p , N ) = Eθ , δ [ ‖fθ ( x ) − fθ̂δp ( x ) ‖ 2 L2 ( [ 0,2π ] ) ] = Eθ , δ [ ‖θ̂ δ p − θ̂p + θ̂p − θ‖22 ] , ( 10 ) where the expectation is taken over the joint distribution of θ and the random noise δ . By the standard triangle inequality , this generalization error is bounded by sum of the generalization error from training with noise-free data and the error caused by the noise . We will use this simple observation to bound the generalization errors when no exact formulas can be derived . We also look at the variance of the generalization error with respect to the random noise , which is Varδ ( Eθ [ ‖θ̂ δ − θ‖22 ] ) : = Eδ [ ( Eθ [ ‖θ̂ δ − θ‖22 ] − Eθ , δ [ ‖θ̂ δ − θ‖22 ] ) 2 ] . ( 11 ) In the rest of the work , we consider two parameter regimes of learning : ( i ) In the overparameterized regime , we have the following setup of the parameters : N < p ≤ P , and , P = µN , p = νN for some µ , ν ∈ N s.t . µ ≥ ν 1 . ( 12 ) ( ii ) In the underparameterized regime , we have the following scaling relations : p ≤ N ≤ P , and , P = µN for some µ ∈ N. ( 13 ) The formally-determined case of p = N ≤ P is included in both the overparameterized and the underparameterized regimes . We make the following assumptions throughout the work : ( A-I ) The random noise δ in the training data is additive in the sense that yδ = y + δ . ( A-II ) The random vectors δ and θ are independent . ( A-III ) The random noise δ ∼ N ( 0 , σI [ P ] ) for some constant σ > 0 . While assumptions ( A-I ) and ( A-II ) are essential , assumption ( A-III ) is only needed to simplify the calculations . Most of the results we obtain in this paper can be reproduced straightforwardly for the random noise δ with any well-defined covariance matrix .

Summary: This paper studies the weighted least squares, random features model under noise one-dimensional data setting in under-/over-parameterized regime. The derived error bounds demonstrate the impact of noise on the generalization error. Besides, the extension to kernel regression shows that, the selected weighted matrix is helpful to generalization when the RKHS is small (i.e., singular values of \Psi decay fast).