Understanding the role of importance weighting for deep learning

1 INTRODUCTION . Importance weighting is a standard tool for estimating a quantity under a target distribution while only the samples from some source distribution is accessible . It has been drawing extensive attention in the communities of statistics and machine learning . Causal inference for deep learning investigates heavily on the propensity score weighting method that applies the off-policy optimization with counterfactual estimator ( Gilotte et al. , 2018 ; Jiang & Li , 2016 ) , modelling with observational feedback ( Schnabel et al. , 2016 ; Xu et al. , 2020 ) and learning from controlled intervention ( Swaminathan & Joachims , 2015 ) . The importance weighting methods are also applied to characterize distribution shifts for deep learning models ( Fang et al. , 2020 ) , with modern applications in such as the domain adaptation ( Azizzadenesheli et al. , 2019 ; Lipton et al. , 2018 ) and learning from noisy labels ( Song et al. , 2020 ) . Other usages include curriculum learning ( Bengio et al. , 2009 ) and knowledge distillation ( Hinton et al. , 2015 ) , where the weights characterize the model confidence on each sample . To reduce the discrepancy between the source and target distribution for model training , a standard routine is to minimize a weighted risk ( Rubinstein & Kroese , 2016 ) . Many techniques have been developed to this end , and the common strategy is re-weighting the classes proportionally to the inverse of their frequencies ( Huang et al. , 2016 ; 2019 ; Wang et al. , 2017 ) . For example , Cui et al . ∗The work was done when the author was with Walmart Labs . ( 2019 ) proposes re-weighting by the inverse of effective number of samples . The focal loss ( Lin et al. , 2017 ) down-weights the well-classified examples , and the work by Li et al . ( 2019 ) suggests an improved technique which down-weights examples based on the magnitude of the gradients . Despite the empirical successes of various re-weighting methods , it is ultimately not clear how importance weighting lays influence from the theoretical standpoint . The recent study of Byrd & Lipton ( 2019 ) observes from experiments that there is little impact of importance weights on the converged deep neural network , if the data can be separated by the model using gradient descent . They connect this phenomenon to the implicit bias of gradient descent ( Soudry et al. , 2018 ) - a novel topic that studies why over-parameterized models trained on separable data is biased toward solutions that generalize well . Implicit bias of gradient descent has been observed and studied for linear model ( Soudry et al. , 2018 ; Ji & Telgarsky , 2018b ) , linear neural network ( Ji & Telgarsky , 2018a ; Gunasekar et al. , 2018 ) , two-layer neural network with homogeneous activation ( Chizat & Bach , 2020 ) and smooth neural networks ( Nacson et al. , 2019 ; Lyu & Li , 2019 ) . To summarize , those work reveals that the direction of the parameters ( for linear predictor ) and the normalized margin ( for nonlinear predictor ) , regardless of the initialization , respectively converge to those of a max-margin solution . The pivotal role of margin for deep learning models has been explored actively after the long journey of understanding the generalization of over-parameterized neural networks ( Bartlett et al. , 2017 ; Golowich et al. , 2018 ; Neyshabur et al. , 2018 ) . For instance , Wei et al . ( 2019 ) studies the margin of the neural networks for separable data under weak regularization . They show that the normalized margin also converges to the max-margin solution , and provide a generalization bound for a neural network that hinges on its margin . Although there are rich understandings for the implicit bias of gradient descent and the margin-based generalization , very few efforts are dedicated to studying how they adjust to the weighted empiricalrisk minimization ( ERM ) setting . The established results do not directly transfer since importance weighting can change both the optimization geometry and how the generalization is measured . In this paper , we fill in the gap by showing the impact of importance weighting on the implicit bias of gradient descent as well as the generalization performance . By studying the optimization dynamics of linear models , we first reveal the effect of importance weighting on the convergence speed under linearly separable data . When the data is not linearly separable , we characterize the unique role of importance weighting on defining the intercept term upon the implicit bias . We then investigate the non-linear neural network under a weak regularization as Wei et al . ( 2019 ) . We provide a novel generalization bound that reflects how importance weighting leads to the interplay between the empirical risk and a compounding term that consists of the model complexity as well as the deviation between the source target distribution . Based on our theoretical results , we discuss several exploratory developments on importance weighting that are worthy of further investigations . • A good set of weights for learning can be inversely proportional to the hard-to-classify extent . For example , a sample that is close to ( far from ) the oracle decision boundary should have a large ( small ) weight . • If the importance weights are jointly trained according to a weighting model , the impact of the weighting model eventually diminishes after showing strong correlation with the hard-to-classify extent such as margin . • The usefulness of explicit regularization on weighted ERM can be studied , via their impact on the margin , on balancing the empirical loss and the distribution divergence . In summary , our contribution are three folds . • We characterize the impact of importance weighting on the implicit bias of gradient descent . • We find a generalization bound that hinges on the importance weights . For finite-step training , the role of importance weighting on the generalization bound is reflected in how the margin is affected , and how it balances the source and target distribution . • We propose several exploratory topics for importance weighting that worth further investigating from both the application and theoretical perspective . The rest of the paper is organized as follows . In Section 2 , we introduce the background , preliminary results and the experimental setup . In Section 3 and 4 , we demonstrate the influence of the importance weighting for linear and non-linear models in terms of the implicit bias of gradient descent and the generalization performance . We then discuss the extended investigations in Section 5 . 2 PRELIMINARIES . We use bold-font letters for vectors and matrices , uppercase letters for random variables and distributions , and ‖ · ‖ to denote ` 2 norm when no confusion arises . We denote the training data by D = { wi , xi , yi } ni=1 where xi ∈ X denotes the features , yi is binary or categorical , and the importance weight is bounded such that : wi ∈ [ 1/M , M ] for someM > 1 . We mention that the importance weights are often defined with respect to the source distribution Ps from which the training data is drawn , and the target distribution Pt . We do not make this assumption here because importance weighting is often applied for more general purposes . Therefore , wi can be defined arbitrarily . We use f ( θ , x ) to denote the predictor and define F = { f ( θ , · ) | θ ∈ Θ ⊂ Rd } . For the sake of notation , we focus on the binary setting : yi ∈ { −1 , +1 } with f ( θ , x ) ∈ R. However , it will become clear later that our results can be easily extended to the multi-class setting . Consider the weighted empirical risk minimization ( ERM ) task with the risk given byL ( θ ; w ) = 1/n ∑n i=1 wi ` ( yif ( θ , xi ) ) for some non-negative loss function ` ( · ) . The weight-agnostic counterpart is denoted by : L ( θ ) = 1/n ∑n i=1 ` ( yif ( θ , xi ) ) . We focus particularly on the exponential loss ` ( u ) = exp ( −u ) and log loss ` ( u ) = log ( 1 + exp ( −u ) ) . For the multi-class problem where yi ∈ [ k ] , we extend our setup using the softmax function where the logits are now given by { fj ( θ , x ) } kj=1 . For optimization , we consider using gradient descent to minimize the total loss : θ ( t+1 ) ( w ) = θ ( t ) ( w ) − ηt∇L ( θ ; w ) ∣∣ θ=θ ( t ) ( w ) , where the learning rate ηt can be constant or step-dependent . From parameter norm divergence to support vectors . Suppose D is separated by f ( θ ( t ) , x ) after some point during training . The key factor that contributes to the implicit bias for both linear and non-linear predictor under a weak regularization 1 is that the norm of the parameters diverges after separation , i.e . limt→∞ ‖θ ( t ) ‖2 = ∞ , as a consequence of using gradient descent . Now we examine ‖θ ( t ) ( w ) ‖2 . The heuristic is that if ` ( · ) is exponential-like , multiplying by wi only changes its tail property up to a constant while the asymptotic behavior is not affected . In particular , the necessary conditions for norm divergence under gradient descent can be summarized by : • C1 . The loss function ` ( · ) has a exponential tail behavior ( that we formalize in Appendix A.1 ) such that limu→∞ ` ( −u ) = limu→∞∇ ` ( −u ) = 0 ; • C2 . The predictor f ( θ , x ) is α-homogeneous such that f ( c · θ , x ) = cαf ( θ , x ) , ∀c > 0 . In addition , we need certain regularities from f ( θ , x ) to ensure the existence of critical points and the convergence of gradient descent : • C3 . for any x ∈ X , f ( · , x ) is β-smooth and l-Lipschitz on Rd . C1 can be satisfied by the exponential loss , log loss and cross entropy loss under the multi-class setting . For standard deep learning models such as multilayer perceptron ( MLP ) , C2 implies that the activation functions are homogeneous such as ReLU and LeakyReLU , and bias terms are disallowed . C3 is a common technical assumptions whose practical implications are discussed in Appendix A.1 . Among the three necessary conditions , importance weighting only affects C1 up to a constant , so its impact on the norm divergence diminishes in the asymptotic regime . The formal statement is provided as below . Claim 1 . There exists a constant learning rate for gradient descent , such that for any w ∈ [ 1/M , M ] n , with a weak regularization , limt→∞ ∥∥θ ( t ) ( w ) ∥∥ =∞ under C1-C3 . Compared with the previous work , we extend the norm divergence result not only to weighted ERM but a more general setting where a weak regularization is considered . We defer the proof to Appendix A.1 . A direct consequence of parameter norm divergence is that both the risk and the gradient are dominated by the terms with the smallest margin , i.e . arg mini yif ( θ , xi ) , which are also referred to as the `` support vectors '' . To make sense of this point , notice that both the risk and the gradient have the form of : ∑ i Ci exp ( − yif ( θ , xi ) ) , where Ci are low-order terms . Since f ( θ , xi ) = ‖θ‖α2 f ( θ/‖θ‖2 , xi ) due to the homogeneous assumption in C2 , it holds that : 1The regularized loss is given by Lλ ( θ ; w ) = L ( θ ; w ) + λ‖θ‖r for a fixed r > 0 . The weak regularization refers to the case where λ→ 0. limt→∞ exp ( − yif ( θ ( t ) ( w ) , xi ) ) → 0 . Therefore , the decision boundaries may share certain characteristics with the support vector machine ( SVM ) since they rely on the same support vectors . As a matter of fact , the current understandings on the implicit bias of gradient descent are mostly established on the connection with hard-margin SVM : min θ∈Rd ‖θ‖2 s.t . yif ( θ , xi ) ≥ 1 ∀i = 1 , 2 , . . . , n , ( 1 ) whose optimization path coincides with the max-margin problem : max‖θ‖2≤1 mini=1 , ... , n yif ( θ , xi ) , as shown by Nacson et al . ( 2019 ) . Define γ ( θ ) : = mini yif ( θ , xi ) . We use θ∗ to denote the optimal solution and γ∗ = γ ( θ∗ ) : = mini yif ( θ∗ , xi ) to denote the corresponding margin . Implicit bias of gradient descent . We start by considering the weight-agnostic setting . When D is linear separable , it is reasonable to conjecture that the separating hyperplane under a linear f ( θ , · ) overlaps with the solution of hard-margin SVM . Soudry et al . ( 2018 ) and Ji & Telgarsky ( 2018b ) first show that ‖θ ( t ) ‖ converges in direction to θ∗ , i.e . limt→∞ θ ( t ) /‖θ ( t ) ‖2 = θ∗ . For nonlinear predictors , however , the parameter direction is less meaningful . Instead , it has been pointed out that neural networks often achieve perfect separation of the training data ( Zhang et al. , 2016 ) . Therefore , we are more interested in the margin whose pivoting role for the generalization of neural networks is studied extensively ( Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ; Golowich et al. , 2018 ) . Specifically , it has been show in Nacson et al . ( 2019 ) and Lyu & Li ( 2019 ) that the normalized margin , defined by γ̃ ( θ ( t ) ) : = γ ( θ ( t ) /‖θ ( t ) ‖2 ) , converges to the maximum margin γ∗ without regularization . It becomes clear at this point that to understand the role of importance weighting for deep learning , we must characterize the impact of weights on the implicit bias since they reveal the optimization geometry and generalization performance . Formally , we address the following critical questions . • Q1 . Does importance weighting modify the convergence results ( convergence in direction for linear predictor and in normalized margin for nonlinear predictor ) ? • If the convergence results remain unchanged , then : – Q2 . in what way is importance weighting affecting the optimization process ; – Q3 . how does importance weighting influence the generalization from the source distribution to the target distribution ?

Summary: This paper studies the inductive bias of gradient descent (GD) on smooth non-linear models when optimizing a weighted ERM. The authors provide several novel results for the linear and non-linear model cases. For linear models and linearly separable data, they show that GD converges to the hard-margin SVM solution and the convergence rate upper bound is lower for weighted ERMs that have higher weight on low margin points. They further characterize the inductive bias for non-linearly separable data, on a unique non-linearly separable subspace defined by Ji and Telgarsky (2018). For nonlinear models they consider a weak regularization setup. They show that asymptotically GD converges to a max margin predictor, which is similar to the non-weighted ERM case. They prove a generalization bound for weighted ERM and together with experiments provide insights on the generalization performance of GD in this case.