Multi-Objective Online Learning

1 INTRODUCTION . Traditional optimization methods for machine learning are usually designed to optimize a single objective . However , in many real-world machine learning applications , we are often required to optimize multiple correlated objectives simultaneously . For example , in autonomous driving ( Huang et al. , 2019 ; Lu et al. , 2019b ) , the self-driving vehicle needs to learn to solve multiple tasks such as self-localization and object identification at the same time . In online advertising ( Ma et al. , 2018a ; b ) , the advertiser aims to choose the exposure of items to different users so that both the Click-Through Rate ( CTR ) and the Post-Click Conversion Rate ( CVR ) are maximized simultaneously . In many multi-objective scenarios , the objectives may conflict with each other ( Kendall et al. , 2018 ) . Hence , there may not exist any single solution that optimizes all the objectives simultaneously . For example , in the online advertising scenario , merely optimizing either CVR or CTR will incur the degradation of performance of the other ( Ma et al. , 2018a ; b ) . Multi-objective optimization ( MOO ) ( Marler and Arora , 2004 ; Deb , 2014 ) is concerned with optimizing multiple conflicting objectives simultaneously . Many different approaches for MOO have been proposed , which include evolutionary methods ( Murata et al. , 1995 ; Zitzler and Thiele , 1999 ) , scalarization methods ( Fliege and Svaiter , 2000 ) and gradient-based iterative methods ( Désidéri , 2012 ) . Recently , due to the great success of training multi-task deep neural networks , the first-order gradient-based iterative methods , i.e. , Multiple Gradient Descent Algorithm ( MGDA ) and its variants , have regained a significant amount of research interest ( Sener and Koltun , 2018 ; Lin et al. , 2019 ; Yu et al. , 2020 ) . These methods compute a composite gradient based on the gradient information of all the individual objectives and then apply the composite gradient to model update . The determination of the composite gradient is based on a min-norm solver ( Désidéri , 2012 ) which yields the common descent direction of all the objectives . However , compared to the increasingly wide application prospect , the first-order gradient-based iterative algorithms are relatively understudied , especially for the online learning setting . Multiobjective online learning is very important due to reasons in two main aspects . First , due to the data explosion in many real-world scenarios such as various web applications , making in-time prediction requires to perform online learning . Second , the theoretical investigation will lay a solid foundation for the design of new optimizers of multi-task deep neural networks , such as multi-objective Adam . In this paper , we conduct a systematic study of multi-objective online learning . To begin with , we first formulate the framework of Multi-Objective Online Convex Optimization ( MO-OCO ) . The biggest challenge in the design of MO-OCO is the derivation of an appropriate regret definition in the multi-objective setting . Since the multiple objectives form a vector space , we need a discrepancy metric to scalarize the loss vector . Specifically , we adopt the Pareto suboptimality gap ( PSG ) , which is a distance-based discrepancy metric extensively used in multi-objective bandits ( Turgay et al. , 2018 ; Lu et al. , 2019a ) . Then analogously to the single-objective online setting , we define the multi-objective static regret and the multi-objective dynamic regret . However , since PSG is a metric motivated purely from the geometric view , it is intrinsically difficult to directly optimize using firstorder gradient-based iterative methods . To remedy this problem , for the multi-objective dynamic regret , via a highly non-trivial transformation , we derive its equivalent regret in the unconstrained max-min form , which is much easier to optimize . Unfortunately , we further show that such an equivalence does not hold for the multi-objective static regret since PSG always yields non-negative measurements . Hence , in this paper , we mainly focus on studying the multi-objective dynamic regret and leave the complete treatment of its static counterpart as an open problem . Based on the proposed MO-OCO framework , we develop the Online Mirror Multiple Descent ( OMMD ) algorithm . The key module of OMMD is the gradient composition scheme , which utilizes the information of all the individual gradients to compute a composite gradient that tends to descend all the losses simultaneously . By directly applying the min-norm solver ( Désidéri , 2012 ) in the offline setting to determine the composition weights , we give the first variant of OMMD termed as OMMD-I . However , the min-norm solver only uses the instantaneous gradients and ignores the historical information , which can be very unstable in the online setting where the losses and the gradients at different rounds can vary wildly . To make the learning process more stable , we introduce a carefully designed L1-regularizer to the vanilla min-norm solver , which results in the second variant of OMMD , namely OMMD-II . We then give the theoretical analysis of the proposed OMMD algorithm . Specifically , we derive a non-trivial dynamic regret bound of OMMD-II , which includes that of OMMD-I as a special case . The dynamic regret bound is in the same order O ( V 1/3T T 2/3 ) , where T is the time horizon and VT is the temporal variability at T , as that of its single-objective counterpart ( Zhang et al. , 2018 ) . We further show that the regret bound of OMMD-II is lower than that of OMMD-I . To evaluate the effectiveness of our proposed algorithm , we conduct extensive experiments using both simulation datasets and real-world datasets . Specifically , we first design simulation experiments to verify the capability of OMMD-II to track the dynamic online data streams . Then we successfully realize adaptive regularization for online learning using our MO-OCO formalism , which demonstrates the effectiveness of OMMD-II in the convex setting . We further conduct online multitask learning experiments with deep neural networks , the results of which show that both OMMD-I and OMMD-II are effective in the non-convex setting . Moreover , in both simulation and multi-task deep experiments , OMMD-II yields better performance than OMMD-I , which verifies the theoretical superiority of the regularized min-norm method over the vanilla min-norm method . In summary , in this paper , we give the first systematic study of multi-objective online learning , which includes novel framework , algorithm design and theoretical analysis . We believe that our paper paves the way for future research on multiple-objective optimization and multi-task learning . 2 PRELIMINARIES . In this section , we briefly review the necessary background knowledge of online convex optimization and multi-objective optimization . 2.1 ONLINE CONVEX OPTIMIZATION . Online Convex Optimization ( OCO ) ( Zinkevich , 2003 ; Hazan , 2019 ) is the most commonly adopted framework for designing online learning algorithms . It can be viewed as a structured repeated game between a learner and an adversary . At each round t ∈ { 1 , . . . , T } , the learner is required to generate a decision xt from a convex compact set X ⊂ Rn . Then the adversary replies the learner with a convex function ft : X → R and the learner suffers the loss ft ( xt ) . The goal of the learner is to minimize the regret with respect to the best fixed decision in hindsight , i.e. , RS ( T ) = T∑ t=1 ft ( xt ) − min x∗∈X T∑ t=1 ft ( x ∗ ) . Note that the above regret is the static regret ( Hall and Willett , 2013 ) , which compares the learner ’ s cumulative loss with that of a fixed decision . There is another version of regret , namely the dynamic regret ( Hall and Willett , 2013 ; Zhang et al. , 2018 ) , which compares the learner ’ s cumulative loss with that of a sequence of changing decisions , i.e. , RD ( T ) = T∑ t=1 ft ( xt ) − T∑ t=1 min x∗t∈X ft ( x ∗ t ) . Any meaningful regret is required to be sublinear in T , i.e. , limT→∞ RS/D ( T ) /T = 0 , which implies that when T is large enough , the learner can perform as well as the best fixed decision in hindsight ( for static regret ) or the changing optimal decisions at each round ( for dynamic regret ) . Online Mirror Descent ( OMD ) ( Hazan , 2019 ) is a classic first-order online learning algorithm . At each round t ∈ { 1 , . . . , T } , OMD yields its decision using the following formula xt+1 = argmin x∈X η⟨∇ft ( xt ) , x⟩+BR ( x , xt ) , where η is the step size , R : X → R is the regularization function , and BR ( x , x′ ) = R ( x ) − R ( x′ ) − ⟨∇R ( x′ ) , x − x′⟩ is the Bregman divergence induced from R. As a generic algorithm , by instantiating different regularization functions , OMD can induce two important algorithms , i.e. , Online Gradient Descent ( Zinkevich , 2003 ) and Online Exponentiated Gradient ( Hazan , 2019 ) . Several papers work on the dynamic regret of online mirror descent ( Jadbabaie et al. , 2015 ; Shahrampour and Jadbabaie , 2017 ) . 2.2 MULTI-OBJECTIVE OPTIMIZATION . Multiple-objective optimization ( MOO ) is concerned with solving the problems of optimizing multiple objective functions simultaneously ( Zitzler and Thiele , 1999 ; Sener and Koltun , 2018 ) . In general , since different objectives may conflict with each other , there is no single solution that can optimize all the objectives at the same time . Instead , MOO seeks to find solutions that achieve Pareto optimality . In the following , we exposit Pareto optimality and related definitions more formally using a vector-valued loss H = ( h1 , . . . , hm ) ⊤ as objectives , where m ≥ 2 and hi : K → R , i ∈ { 1 , . . . , m } , K ⊂ R , is the i-th loss function . Definition 1 ( Pareto optimality ) . ( a ) For any two solutions x , x′ ∈ K , we say that x dominates x′ , denoted as x ≺ x′ or x′ ≻ x , if hi ( x ) ≤ hi ( x′ ) for all i , and there exists one i such that hi ( x ) < hi ( x′ ) ; otherwise , we say that x does not dominate x′ , denoted as x ⊀ x′ or x′ ⊁ x . ( b ) A solution x∗ ∈ K is called Pareto optimal if it is not dominated by any other solution in K. There may exist multiple Pareto optimal solutions . For example , it is easy to show that the optimizer of any single objective , say , x∗1 ∈ argminx∈K h1 ( x ) , is Pareto optimal . Different Pareto optimal solutions reflect different trade-offs among the objectives ( Sener and Koltun , 2018 ; Lin et al. , 2019 ) . Definition 2 ( Pareto front ) . ( a ) All Pareto optimal solutions form the Pareto set , denoted as PK ( H ) . ( b ) The image of PK ( H ) constitutes the Pareto front , denoted as P ( H ) = { H ( x ) | x ∈ PK ( H ) } . Now that we ’ ve established the notion of optimality in MOO , we proceed to introduce the metrics that measure the discrepancy of an arbitrary solution x ∈ K from being optimal . Recall that , in the single-objective setting with merely one loss function h : Q → R , where Q ⊂ R , for any z ∈ Q , the loss gap h ( z ) − minz′′∈Q h ( z′′ ) is directly the discrepancy measure . However , in MOO with more than one loss , for any x ∈ K , the loss gap H ( x ) − H ( x′′ ) , where x′′ ∈ PK ( H ) , is a vector . Intuitionally , the desired discrepancy metric shall scalarize the vector-valued loss gap and yield the value 0 for any Pareto optimal solution . In general , there are two commonly used discrepancy metrics in MOO , namely Pareto suboptimality gap ( PSG ) ( Turgay et al. , 2018 ) and Hypervolume ( HV ) ( Bradstreet , 2011 ) . As HV is a volume-based metric , it is very difficult to optimize or analyze via iterative algorithms ( Zhang and Golovin , 2020 ) . Hence in this paper , we adopt PSG , which has been extensively used in multi-objective bandits ( Turgay et al. , 2018 ; Lu et al. , 2019a ) . Definition 3 ( Pareto suboptimality gap ) . For any x ∈ K , the Pareto suboptimality gap to a given comparator set K∗ ⊂ K , denoted as ∆ ( x ; K∗ , H ) , is defined as the minimal scalar ϵ ≥ 0 that needs to be subtracted from all entries of H ( x ) , such that H ( x ) − ϵ1 is not dominated by any point in K∗ , where 1 denotes the all-one vector in Rm , i.e. , ∆ ( x ; K∗ , H ) = inf ϵ≥0 ϵ s.t . ∀x′′ ∈ K∗ , ∃i ∈ { 1 , . . . , m } , hi ( x ) − ϵ < hi ( x′′ ) . Clearly , PSG is a distance-based discrepancy metric that motivated from a purely geometric viewpoint . In practice , the comparator set K∗ is often set to be the Pareto set PK ( H ) ( Turgay et al. , 2018 ) . Then for any x ∈ K , its PSG is always non-negative and equals to zero if and only if x ∈ PK ( H ) . Multiple Gradient Descent Algorithm ( MGDA ) is an offline first-order algorithm for MOO ( Fliege and Svaiter , 2000 ; Désidéri , 2012 ) . At each iteration l ∈ { 1 , . . . , L } , where L is the maximum number of iterations , it first computes the gradient ∇hi ( xl ) for each objective i ∈ { 1 , . . . , m } , and then derive the composite gradient gl = ∑m i=1 λ i l∇hi ( xl ) as the convex combination of these multiple gradients ; it applies the composite gradient to execute the gradient descent step to update the decision , i.e. , xl+1 = xl − ηgl , where η is the step size . The core module of MGDA is the determination of the weights λl = ( λ1l , . . . , λ m l ) for the gradient composition , which is given as λl = argmin λl∈∆m ∥ m∑ i=1 λil∇hi ( xl ) ∥22 , where ∆m = { λ ∈ Rm | λi ≥ 0 , i ∈ { 1 , . . . , m } , and ∑m i=1 λ i = 1 } denotes the probabilistic simplex in Rm . This is a min-norm solver which finds the weights in the simplex that yields the minimum L2 norm of the composite gradient . Thus MGDA is also called the min-norm method . Existing works ( Désidéri , 2012 ; Sener and Koltun , 2018 ) have shown that MGDA is guaranteed to decrease all the objectives simultaneously until it reaches a Pareto optimal decision ( under the convex setting where all hi are convex functions ) .

Summary: The paper formulates a novel framework for multi-objective online convex optimization. The novel framework, similarly to the single-objective online convex optimization framework, can be viewed as a two players repeated game where at each round the online learner selects a point $x_t$ and the (possibly adversarial) environment selects a vector valued loss function $F_t (\cdot )$. To extend the notion of regret to the multi-objective setting at each round the suboptimality of $x_t$ is measured using the Pareto suboptimality gap (PSG). Two algorithms (OMMD-I and OMMD-II) which upper bound the extension of the dynamic regret in the multi-objective setting are designed. The two algorithms can be viewed as extensions of the min-norm method for offline multi-objective optimization in the online setting. In both algorithms a composite gradient, which is a convex combination of the descent directions for every objective, is calculated. This composite gradient is used to select, together with a Bregman reguralization, the descent direction and consequently the point $x_{t+1}$. The main difference between the two algorithms OMMD-I and OMMD-II is that in the latter the composition of the descent directions in two consecutive rounds will possibly be "more similar" (due to the regularization term $|| \lambda - \lambda_{t-1}||_1$).