Multi-Objective Online Learning

1 INTRODUCTION . Traditional optimization methods for machine learning are usually designed to optimize a single objective . However , in many real-world machine learning applications , we are often required to optimize multiple correlated objectives simultaneously . For example , in autonomous driving ( Huang et al. , 2019 ; Lu et al. , 2019b ) , the self-driving vehicle needs to learn to solve multiple tasks such as self-localization and object identification at the same time . In online advertising ( Ma et al. , 2018a ; b ) , the advertiser aims to choose the exposure of items to different users so that both the Click-Through Rate ( CTR ) and the Post-Click Conversion Rate ( CVR ) are maximized simultaneously . In many multi-objective scenarios , the objectives may conflict with each other ( Kendall et al. , 2018 ) . Hence , there may not exist any single solution that optimizes all the objectives simultaneously . For example , in the online advertising scenario , merely optimizing either CVR or CTR will incur the degradation of performance of the other ( Ma et al. , 2018a ; b ) . Multi-objective optimization ( MOO ) ( Marler and Arora , 2004 ; Deb , 2014 ) is concerned with optimizing multiple conflicting objectives simultaneously . Many different approaches for MOO have been proposed , which include evolutionary methods ( Murata et al. , 1995 ; Zitzler and Thiele , 1999 ) , scalarization methods ( Fliege and Svaiter , 2000 ) and gradient-based iterative methods ( Désidéri , 2012 ) . Recently , due to the great success of training multi-task deep neural networks , the first-order gradient-based iterative methods , i.e. , Multiple Gradient Descent Algorithm ( MGDA ) and its variants , have regained a significant amount of research interest ( Sener and Koltun , 2018 ; Lin et al. , 2019 ; Yu et al. , 2020 ) . These methods compute a composite gradient based on the gradient information of all the individual objectives and then apply the composite gradient to model update . The determination of the composite gradient is based on a min-norm solver ( Désidéri , 2012 ) which yields the common descent direction of all the objectives . However , compared to the increasingly wide application prospect , the first-order gradient-based iterative algorithms are relatively understudied , especially for the online learning setting . Multiobjective online learning is very important due to reasons in two main aspects . First , due to the data explosion in many real-world scenarios such as various web applications , making in-time prediction requires to perform online learning . Second , the theoretical investigation will lay a solid foundation for the design of new optimizers of multi-task deep neural networks , such as multi-objective Adam . In this paper , we conduct a systematic study of multi-objective online learning . To begin with , we first formulate the framework of Multi-Objective Online Convex Optimization ( MO-OCO ) . The biggest challenge in the design of MO-OCO is the derivation of an appropriate regret definition in the multi-objective setting . Since the multiple objectives form a vector space , we need a discrepancy metric to scalarize the loss vector . Specifically , we adopt the Pareto suboptimality gap ( PSG ) , which is a distance-based discrepancy metric extensively used in multi-objective bandits ( Turgay et al. , 2018 ; Lu et al. , 2019a ) . Then analogously to the single-objective online setting , we define the multi-objective static regret and the multi-objective dynamic regret . However , since PSG is a metric motivated purely from the geometric view , it is intrinsically difficult to directly optimize using firstorder gradient-based iterative methods . To remedy this problem , for the multi-objective dynamic regret , via a highly non-trivial transformation , we derive its equivalent regret in the unconstrained max-min form , which is much easier to optimize . Unfortunately , we further show that such an equivalence does not hold for the multi-objective static regret since PSG always yields non-negative measurements . Hence , in this paper , we mainly focus on studying the multi-objective dynamic regret and leave the complete treatment of its static counterpart as an open problem . Based on the proposed MO-OCO framework , we develop the Online Mirror Multiple Descent ( OMMD ) algorithm . The key module of OMMD is the gradient composition scheme , which utilizes the information of all the individual gradients to compute a composite gradient that tends to descend all the losses simultaneously . By directly applying the min-norm solver ( Désidéri , 2012 ) in the offline setting to determine the composition weights , we give the first variant of OMMD termed as OMMD-I . However , the min-norm solver only uses the instantaneous gradients and ignores the historical information , which can be very unstable in the online setting where the losses and the gradients at different rounds can vary wildly . To make the learning process more stable , we introduce a carefully designed L1-regularizer to the vanilla min-norm solver , which results in the second variant of OMMD , namely OMMD-II . We then give the theoretical analysis of the proposed OMMD algorithm . Specifically , we derive a non-trivial dynamic regret bound of OMMD-II , which includes that of OMMD-I as a special case . The dynamic regret bound is in the same order O ( V 1/3T T 2/3 ) , where T is the time horizon and VT is the temporal variability at T , as that of its single-objective counterpart ( Zhang et al. , 2018 ) . We further show that the regret bound of OMMD-II is lower than that of OMMD-I . To evaluate the effectiveness of our proposed algorithm , we conduct extensive experiments using both simulation datasets and real-world datasets . Specifically , we first design simulation experiments to verify the capability of OMMD-II to track the dynamic online data streams . Then we successfully realize adaptive regularization for online learning using our MO-OCO formalism , which demonstrates the effectiveness of OMMD-II in the convex setting . We further conduct online multitask learning experiments with deep neural networks , the results of which show that both OMMD-I and OMMD-II are effective in the non-convex setting . Moreover , in both simulation and multi-task deep experiments , OMMD-II yields better performance than OMMD-I , which verifies the theoretical superiority of the regularized min-norm method over the vanilla min-norm method . In summary , in this paper , we give the first systematic study of multi-objective online learning , which includes novel framework , algorithm design and theoretical analysis . We believe that our paper paves the way for future research on multiple-objective optimization and multi-task learning . 2 PRELIMINARIES . In this section , we briefly review the necessary background knowledge of online convex optimization and multi-objective optimization . 2.1 ONLINE CONVEX OPTIMIZATION . Online Convex Optimization ( OCO ) ( Zinkevich , 2003 ; Hazan , 2019 ) is the most commonly adopted framework for designing online learning algorithms . It can be viewed as a structured repeated game between a learner and an adversary . At each round t ∈ { 1 , . . . , T } , the learner is required to generate a decision xt from a convex compact set X ⊂ Rn . Then the adversary replies the learner with a convex function ft : X → R and the learner suffers the loss ft ( xt ) . The goal of the learner is to minimize the regret with respect to the best fixed decision in hindsight , i.e. , RS ( T ) = T∑ t=1 ft ( xt ) − min x∗∈X T∑ t=1 ft ( x ∗ ) . Note that the above regret is the static regret ( Hall and Willett , 2013 ) , which compares the learner ’ s cumulative loss with that of a fixed decision . There is another version of regret , namely the dynamic regret ( Hall and Willett , 2013 ; Zhang et al. , 2018 ) , which compares the learner ’ s cumulative loss with that of a sequence of changing decisions , i.e. , RD ( T ) = T∑ t=1 ft ( xt ) − T∑ t=1 min x∗t∈X ft ( x ∗ t ) . Any meaningful regret is required to be sublinear in T , i.e. , limT→∞ RS/D ( T ) /T = 0 , which implies that when T is large enough , the learner can perform as well as the best fixed decision in hindsight ( for static regret ) or the changing optimal decisions at each round ( for dynamic regret ) . Online Mirror Descent ( OMD ) ( Hazan , 2019 ) is a classic first-order online learning algorithm . At each round t ∈ { 1 , . . . , T } , OMD yields its decision using the following formula xt+1 = argmin x∈X η⟨∇ft ( xt ) , x⟩+BR ( x , xt ) , where η is the step size , R : X → R is the regularization function , and BR ( x , x′ ) = R ( x ) − R ( x′ ) − ⟨∇R ( x′ ) , x − x′⟩ is the Bregman divergence induced from R. As a generic algorithm , by instantiating different regularization functions , OMD can induce two important algorithms , i.e. , Online Gradient Descent ( Zinkevich , 2003 ) and Online Exponentiated Gradient ( Hazan , 2019 ) . Several papers work on the dynamic regret of online mirror descent ( Jadbabaie et al. , 2015 ; Shahrampour and Jadbabaie , 2017 ) . 2.2 MULTI-OBJECTIVE OPTIMIZATION . Multiple-objective optimization ( MOO ) is concerned with solving the problems of optimizing multiple objective functions simultaneously ( Zitzler and Thiele , 1999 ; Sener and Koltun , 2018 ) . In general , since different objectives may conflict with each other , there is no single solution that can optimize all the objectives at the same time . Instead , MOO seeks to find solutions that achieve Pareto optimality . In the following , we exposit Pareto optimality and related definitions more formally using a vector-valued loss H = ( h1 , . . . , hm ) ⊤ as objectives , where m ≥ 2 and hi : K → R , i ∈ { 1 , . . . , m } , K ⊂ R , is the i-th loss function . Definition 1 ( Pareto optimality ) . ( a ) For any two solutions x , x′ ∈ K , we say that x dominates x′ , denoted as x ≺ x′ or x′ ≻ x , if hi ( x ) ≤ hi ( x′ ) for all i , and there exists one i such that hi ( x ) < hi ( x′ ) ; otherwise , we say that x does not dominate x′ , denoted as x ⊀ x′ or x′ ⊁ x . ( b ) A solution x∗ ∈ K is called Pareto optimal if it is not dominated by any other solution in K. There may exist multiple Pareto optimal solutions . For example , it is easy to show that the optimizer of any single objective , say , x∗1 ∈ argminx∈K h1 ( x ) , is Pareto optimal . Different Pareto optimal solutions reflect different trade-offs among the objectives ( Sener and Koltun , 2018 ; Lin et al. , 2019 ) . Definition 2 ( Pareto front ) . ( a ) All Pareto optimal solutions form the Pareto set , denoted as PK ( H ) . ( b ) The image of PK ( H ) constitutes the Pareto front , denoted as P ( H ) = { H ( x ) | x ∈ PK ( H ) } . Now that we ’ ve established the notion of optimality in MOO , we proceed to introduce the metrics that measure the discrepancy of an arbitrary solution x ∈ K from being optimal . Recall that , in the single-objective setting with merely one loss function h : Q → R , where Q ⊂ R , for any z ∈ Q , the loss gap h ( z ) − minz′′∈Q h ( z′′ ) is directly the discrepancy measure . However , in MOO with more than one loss , for any x ∈ K , the loss gap H ( x ) − H ( x′′ ) , where x′′ ∈ PK ( H ) , is a vector . Intuitionally , the desired discrepancy metric shall scalarize the vector-valued loss gap and yield the value 0 for any Pareto optimal solution . In general , there are two commonly used discrepancy metrics in MOO , namely Pareto suboptimality gap ( PSG ) ( Turgay et al. , 2018 ) and Hypervolume ( HV ) ( Bradstreet , 2011 ) . As HV is a volume-based metric , it is very difficult to optimize or analyze via iterative algorithms ( Zhang and Golovin , 2020 ) . Hence in this paper , we adopt PSG , which has been extensively used in multi-objective bandits ( Turgay et al. , 2018 ; Lu et al. , 2019a ) . Definition 3 ( Pareto suboptimality gap ) . For any x ∈ K , the Pareto suboptimality gap to a given comparator set K∗ ⊂ K , denoted as ∆ ( x ; K∗ , H ) , is defined as the minimal scalar ϵ ≥ 0 that needs to be subtracted from all entries of H ( x ) , such that H ( x ) − ϵ1 is not dominated by any point in K∗ , where 1 denotes the all-one vector in Rm , i.e. , ∆ ( x ; K∗ , H ) = inf ϵ≥0 ϵ s.t . ∀x′′ ∈ K∗ , ∃i ∈ { 1 , . . . , m } , hi ( x ) − ϵ < hi ( x′′ ) . Clearly , PSG is a distance-based discrepancy metric that motivated from a purely geometric viewpoint . In practice , the comparator set K∗ is often set to be the Pareto set PK ( H ) ( Turgay et al. , 2018 ) . Then for any x ∈ K , its PSG is always non-negative and equals to zero if and only if x ∈ PK ( H ) . Multiple Gradient Descent Algorithm ( MGDA ) is an offline first-order algorithm for MOO ( Fliege and Svaiter , 2000 ; Désidéri , 2012 ) . At each iteration l ∈ { 1 , . . . , L } , where L is the maximum number of iterations , it first computes the gradient ∇hi ( xl ) for each objective i ∈ { 1 , . . . , m } , and then derive the composite gradient gl = ∑m i=1 λ i l∇hi ( xl ) as the convex combination of these multiple gradients ; it applies the composite gradient to execute the gradient descent step to update the decision , i.e. , xl+1 = xl − ηgl , where η is the step size . The core module of MGDA is the determination of the weights λl = ( λ1l , . . . , λ m l ) for the gradient composition , which is given as λl = argmin λl∈∆m ∥ m∑ i=1 λil∇hi ( xl ) ∥22 , where ∆m = { λ ∈ Rm | λi ≥ 0 , i ∈ { 1 , . . . , m } , and ∑m i=1 λ i = 1 } denotes the probabilistic simplex in Rm . This is a min-norm solver which finds the weights in the simplex that yields the minimum L2 norm of the composite gradient . Thus MGDA is also called the min-norm method . Existing works ( Désidéri , 2012 ; Sener and Koltun , 2018 ) have shown that MGDA is guaranteed to decrease all the objectives simultaneously until it reaches a Pareto optimal decision ( under the convex setting where all hi are convex functions ) .

Summary: In the standard online learning model, the aim of the learning algorithm is to minimize its cumulative regret, defined by the difference in cumulative losses between the algorithm and the best decision taken with the benefit of insight in some reference set. In the generalized model of “multi-objective” online learning, the feedback is no longer a single scalar, but a vector of losses, each defined for a distinct objective. The performance of the learning algorithm is often defined in terms of “Pareto regret”, for which each instantaneous regret is captured by the notion of Pareto Suboptimality Gap (PSG). Although multi-objective online learning has been studied in the (possibly contextual and linear) bandit setting, many questions remain open. Notably, this paper focuses on the full-information setting where a vector-valued loss function is supplied at the end of each round. Based on the Online Mirror Descent (OMD) paradigm, two versions of multi-objective OMD are provided and analyzed for the dynamic version of Pareto regret (the static version being left as an open issue). Comparative experiments for both versions are performed on several benchmarks.