A Generalized Weighted Optimization Method for Computational Learning and Inversion

1 INTRODUCTION . Given N data pairs { xj , yj } Nj=1 , where xj ∈ R , yj ∈ C , j = 1 , . . . , N , we are interested in learning a random Fourier feature ( RFF ) model ( Rahimi & Recht , 2008 ; Liao et al. , 2020 ; Xie et al. , 2020 ) fθ ( x ) = P−1∑ k=0 θke ikx , x ∈ [ 0 , 2π ] , ( 1 ) where P ∈ N is a given positive integer and we used the short-hand notation θ : = ( θ0 , · · · , θP−1 ) T with the superscript T denoting the transpose operation . This exact model as well as its generalization to more complicated setups have been extensively studied ; see for instance Liao & Couillet ( 2018 ) ; Shahrampour & Kolouri ( 2019 ) ; d ’ Ascoli et al . ( 2020 ) ; Li et al . ( 2020 ) ; Özcelikkale ( 2020 ) ; Liu et al . ( 2020 ; 2021 ) and references therein . While this model may seem to be overly simplified from a practical perspective for many real-world applications , it serves as a prototype for theoretical understandings of different phenomena in machine learning models ( Sriperumbudur & Szabo , 2015 ; Belkin et al. , 2020 ; Li et al. , 2021a ) . A common way to computationally solve this learning problem is to reformulate it as an optimization problem where we find θ by minimizing the model and data mismatch for a given dataset . In this paper , we assume that the training data are collected on a uniform grid of x over the domain [ 0 , 2π ] . That is , { xj = 2πjN } N−1 j=0 . Let ωN = exp ( 2πi N ) where i is the imaginary unit . We introduce Ψ ∈ CN×P to be the feature matrix with elements ( Ψ ) jk = ( ωN ) jk , 0 ≤ j ≤ N − 1 , 0 ≤ k ≤ P − 1 . Based on the form of fθ ( x ) in ( 1 ) , we can then write the 2-norm based data mismatch into the form∑N−1 j=0 |fθ ( xj ) − yj |2 = ‖Ψθ − y‖22 where the column data vector y = ( y0 , · · · , yN−1 ) T. The learning problem is therefore recast as a least-squares optimization problem of the form θ̂ = arg min θ ‖Ψθ − y‖22 , ( 2 ) assuming that a minimizer does exist , especially when we restrict θ to an appropriate space . In a general feature regression problem , the Fourier feature { eikx } P−1k=0 is then replaced with a different feature model { ϕk ( x ) } P−1k=0 , while the least-squares form ( 2 ) remains unchanged except that the entries of the matrix Ψ is now Ψjk = ϕk ( xj ) . We emphasize that this type of generalization will be discussed in Section 5 . Moreover , we remark that this least-squares optimization formulation is a classical computational inversion tool in solving the general linear inverse problems of the form Ψθ = y ; see for instance Engl et al . ( 1996 ) ; Tarantola ( 2005 ) and references therein . Previous work on weighted optimization for feature and kernel learning . Xie et al . ( 2020 ) studied the fitting problem for this model under the assumption that the coefficient vector θ is sampled from a distribution with the property that γ is a positive constant , Eθ [ θ ] = 0 , Eθ [ θθ∗ ] = cγΛ−2γ [ P ] , ( 3 ) where the superscript ∗ denotes the Hermitian transpose and the diagonal matrix Λ [ P ] has diagonal elements ( Λ [ P ] ) kk = tk = 1 + k , k ≥ 0 . That is , Λ [ P ] = diag { t0 , t1 , t2 , . . . , tk , . . . , tP−1 } , tk : = 1 + k . ( 4 ) The subscript [ P ] indicates that Λ [ P ] is a diagonal submatrix of Λ that contains its element indexed in the set [ P ] : = { 0 , 1 , · · · , P − 1 } . The normalization constant cγ = 1/ ( ∑P−1 k=0 ( 1 + k ) −2γ ) is only selected so that Eθ [ ‖θ‖2 ] = 1 . It does not play a significant role in the rest of the paper . The main assumption in ( 3 ) says that statistically , the signal to be recovered has algebraically decaying Fourier coefficients . This is simply saying that the target function we are learning is relatively smooth , which is certainly the case for many functions as physical models in practical applications . It was shown in Xie et al . ( 2020 ) that , to learn a model with p ≤ P features , it is advantageous to use the following weighted least-squares formulation θ̂p = Λ −β [ p ] ŵ , with ŵ = arg minθ ‖Ψ [ N×p ] Λ −β [ p ] w − y‖ 2 2 , ( 5 ) when the learning problem is overparameterized , i.e. , p > N . Here , Ψ [ N×p ] ∈ CN×p is the matrix containing the first p columns of Ψ , and β > 0 is some pre-selected exponent that can be different from the γ in ( 3 ) . To be more precise , we define the the generalization error of the learning problem Eβ ( P , p , N ) : = Eθ [ ‖fθ ( x ) − fθ̂p ( x ) ‖ 2 L2 ( [ 0,2π ] ) ] = Eθ [ ‖θ̂p − θ‖22 ] , ( 6 ) where the equality comes from the Parseval ’ s identity , and θ̂p is understood as the vector ( θTp , 0 , · · · , 0 ) T so that θ and θ̂p are of the same length P . The subscript θ in Eθ indicates that the expectation is taken with respect to the distribution of the random variable θ . It was shown in Xie et al . ( 2020 ) that the lowest generalization error achieved from the weighted least-squares approach ( 5 ) in the overparameterized regime ( p > N ) is strictly less than the lowest possible generalization error in the underparameterized regime ( p ≤ N ) . This , together with the analysis and numerical evidence in previous studies such as those in Belkin et al . ( 2019 ; 2020 ) , leads to the understanding that smoother approximations ( i.e. , solutions that are dominated by lower Fourier modes ) give better generalization in learning with the RFF model ( 1 ) . Main contributions of this work . In this work , we analyze a generalized version of ( 5 ) for general feature regression from noisy data . Following the same notations as before , we introduce the following weighted least-squares formulation for feature regression : θ̂ δ p = Λ −β [ p ] ŵ , with ŵ = arg min w ‖Λ−α [ N ] ( Ψ [ N×p ] Λ −β [ p ] w − y δ ) ‖22 , ( 7 ) where the superscript δ on y and θ̂p denotes the fact that the training data contain random noise of level δ ( which will be specified later ) . The exponent α is pre-selected and can be different from β . While sharing similar roles with the weight matrix Λ−β [ P ] , the weight matrix Λ −α [ N ] provides us the additional ability to deal with noise in the training data . Moreover , as we will see later , the weight matrix Λ−α [ N ] does not have to be either diagonal or in the same form as the matrix Λ −β [ p ] ; the current form is to simplify the calculations for the RFF model . It can be chosen based on the a priori information we have on the operator Ψ as well as the noise distribution of the training data . The highlight and also one of the main contributions of our work is that we introduce a new weight matrix Λ−α [ N ] that emphasizes the data mismatch in terms of its various modes , in addition to Λ −β [ p ] , the weight matrix imposed on the unknown feature coefficient vector θ . This type of generalization has appeared in different forms in many computational approaches for solving inverse and learning problems where the standard 2-norm ( or ` 2 in the infinite-dimensional setting ) is replaced with a weighted norm that is either weaker or stronger than the unweighted 2-norm . In this paper , we characterize the impact of the new weighted optimization framework ( 7 ) on the generalization capability of various feature regression and kernel regression models . The new contributions of this work are threefold . First , we discuss in detail the generalized weighted leastsquares framework ( 7 ) in Section 2 and summarize the main results for training with noise-free data in Section 3 for the RFF model in both the overparameterized and the underparameterized regimes . This is the setup considered in Xie et al . ( 2020 ) , but our analysis is based on the proposed weighted model ( 7 ) instead of ( 5 ) as in their work . Second , we provide the generalization error in both two regimes for the case of training with noisy data ; see Section 4 . This setup was not considered in Xie et al . ( 2020 ) , but we demonstrate here that it is a significant advantage of the weighted optimization when data contains noise since the weighting could effectively minimize the influence of the noise and thus improve the stability of feature regression . Third , we extend the same type of results to more general models in feature regression and kernel regression that are beyond the RFF model , given that the operator Ψ satisfies certain properties . In the general setup presented in Section 5 , we derive error bounds in the asymptotic limit when P , N , and p all become very large . Our analysis provides some guidelines on selecting weighting schemes through either the parameter domain weighting or the data domain weighting , or both , to emphasize the features of the unknowns to be learned based on a priori knowledge . 2 GENERALIZED WEIGHTED LEAST-SQUARES FORMULATION . There are four essential elements in the least-squares formulation of the learning problem : ( i ) the parameter to be learned ( θ ) , ( ii ) the dataset used in the training process ( y ) , ( iii ) the feature matrix ( Ψ ) , and ( iv ) the metric chosen to measure the data mismatch between Ψθ and y . Element ( i ) of the problem is determined not only by the data but also by a priori information we have . The information encoded in ( 3 ) reveals that the size ( i.e. , the variance ) of the Fourier modes in the RFF model decays as fast as ( 1 + k ) −2γ . Therefore , the low-frequency modes in ( 1 ) dominate high-frequency modes , which implies that in the learning process , we should search for the solution vectors that have more low-frequency components than the high-frequency components . The motivation behind introducing the weight matrix Λ−β [ p ] in ( 5 ) is exactly to force the optimization algorithm to focus on admissible solutions that are consistent with the a priori knowledge given in ( 3 ) , which is to seek θ whose components |θk|2 statistically decay like ( 1 + k ) −2β . When the problem is formally determined ( i.e. , p = N ) , the operator Ψ is invertible , and the training data are noise-free , similar to the weight matrix Λ−β [ p ] , the weight matrix Λ −α [ N ] does not change the solution of the learning problem . However , as we will see later , these two weight matrices do impact the solutions in various ways under the practical setups that we are interested in , for instance , when the problem is over-parameterized or when the training data contain random noise . The weight matrix Λ−α [ N ] is introduced to handle elements ( ii ) - ( iv ) of the learning problem . First , since Λ−α [ N ] is directly applied to the data y δ , it allows us to suppress ( when α > 0 ) or promote ( when α < 0 ) high-frequency components in the data during the training process . In particular , when transformed back to the physical space , the weight matrix Λ−α [ N ] with α > 0 corresponds to a smoothing convolutional operator whose kernel has Fourier coefficients decaying at the rate k−α . This operator suppresses high-frequency information in the data . Second , Λ−α [ N ] is also directly applied to Ψθ . This allows us to precondition the learning problem by making Λ−α [ N ] Ψ a betterconditioned operator ( in an appropriate sense ) than Ψ , for some applications where the feature matrix Ψ has certain undesired properties . Finally , since Λ−α [ N ] is applied to the residual Ψθ − y , we can regard the new weighted optimization formulation ( 7 ) as the generalization of the classic leastsquares formulation with a new loss function ( a weighted norm ) measuring the data mismatch . Weighting optimization schemes such as ( 7 ) have been studied , implicitly or explicitly , in different settings ( Needell et al. , 2014 ; Byrd & Lipton , 2019 ; Engquist et al. , 2020 ; Li , 2021 ; Yang et al. , 2021 ) . For instance , if we take β = 0 , then we have a case where we rescale the classical leastsquares loss function with the weight Λ−α [ N ] . If we take α = 1 , then this least-squares functional is equivalent to the loss function based on the H−1 norm , instead of the usual L2 norm , of the mismatch between the target function fθ ( x ) and the learned model fθ̂ ( x ) . Based on the asymptotic equivalence between the quadratic Wasserstein metric and the H−1 semi-norm ( on an appropriate functional space ) , this training problem is asymptotically equivalent to the same training problem based on a quadratic Wasserstein loss function ; see for instance Engquist et al . ( 2020 ) for more detailed illustration on the connection . In the classical statistical inversion setting , Λ2α plays the role of the covariance matrix of the additive Gaussian random noise in the data ( Kaipio & Somersalo , 2005 ) . When the noise is sampled from mean-zero Gaussian distribution with covariance matrix Λ2α , a standard maximum likelihood estimator ( MLE ) is often constructed as the minimizer of ( Ψθ − y ) ∗Λ−2α [ N ] ( Ψθ − y ) = ‖Λ −α [ N ] ( Ψθ − y ) ‖ 2 2 . The exact solution to ( 7 ) , with X+ denoting the Moore–Penrose inverse of operator X , is given by θ̂ δ p = Λ −β [ p ] ( Λ−α [ N ] Ψ [ N×p ] Λ −β [ p ] ) + Λ−α [ N ] y δ . ( 8 ) In the rest of this paper , we analyze this training result and highlight the impact of the weight matrices Λ−α [ N ] and Λ −β [ N ] in different regimes of the learning problem . We reproduce the classical bias-variance trade-off analysis in the weighted optimization framework . For that purpose , we utilize the linearity of the problem to decompose θ̂ δ p as θ̂ δ p = Λ −β [ p ] ( Λ −α [ N ] Ψ [ N×p ] Λ −β [ p ] ) +Λ−α [ N ] y + Λ −β [ p ] ( Λ −α [ N ] Ψ [ N×p ] Λ −β [ p ] ) +Λ−α [ N ] ( y δ − y ) , ( 9 ) where the first part is simply θ̂p , the result of learning with noise-free data , while the second part is the contribution from the additive noise . We define the generalization error in this case as Eδα , β ( P , p , N ) = Eθ , δ [ ‖fθ ( x ) − fθ̂δp ( x ) ‖ 2 L2 ( [ 0,2π ] ) ] = Eθ , δ [ ‖θ̂ δ p − θ̂p + θ̂p − θ‖22 ] , ( 10 ) where the expectation is taken over the joint distribution of θ and the random noise δ . By the standard triangle inequality , this generalization error is bounded by sum of the generalization error from training with noise-free data and the error caused by the noise . We will use this simple observation to bound the generalization errors when no exact formulas can be derived . We also look at the variance of the generalization error with respect to the random noise , which is Varδ ( Eθ [ ‖θ̂ δ − θ‖22 ] ) : = Eδ [ ( Eθ [ ‖θ̂ δ − θ‖22 ] − Eθ , δ [ ‖θ̂ δ − θ‖22 ] ) 2 ] . ( 11 ) In the rest of the work , we consider two parameter regimes of learning : ( i ) In the overparameterized regime , we have the following setup of the parameters : N < p ≤ P , and , P = µN , p = νN for some µ , ν ∈ N s.t . µ ≥ ν 1 . ( 12 ) ( ii ) In the underparameterized regime , we have the following scaling relations : p ≤ N ≤ P , and , P = µN for some µ ∈ N. ( 13 ) The formally-determined case of p = N ≤ P is included in both the overparameterized and the underparameterized regimes . We make the following assumptions throughout the work : ( A-I ) The random noise δ in the training data is additive in the sense that yδ = y + δ . ( A-II ) The random vectors δ and θ are independent . ( A-III ) The random noise δ ∼ N ( 0 , σI [ P ] ) for some constant σ > 0 . While assumptions ( A-I ) and ( A-II ) are essential , assumption ( A-III ) is only needed to simplify the calculations . Most of the results we obtain in this paper can be reproduced straightforwardly for the random noise δ with any well-defined covariance matrix .

Summary:  The paper follows and extends the work by Belkin, Hsu, and Xu (2020) and  Liang and Rakhlin (2018) which studies the bias-variance trade-off of the regression/interpolation problem in the under/over-parametrization regions.  In particular, this paper follows the random Fourier model setting in Xie et al. (2020) and analyzed a generalized weighted least-square optimization method that allows the weighting in both the parametrization and data space. The authors derived the generalization error of such weighted least-square framework for the over parametrized and under parameterized regimes and compare them in these two cases.  The general conclusion is that emphasizing low-frequency features provide better generalization ability.  The paper also studies both noise-free and noise cases.