Pareto Navigation Gradient Descent: a First Order Algorithm for Optimization in Pareto Set

Many modern machine learning applications , such as multi-task learning , require finding1 optimal model parameters to trade-off multiple objective functions that may conflict with2 each other . The notion of the Pareto set allows us to focus on the set of ( often infinite number3 of ) models that can not be strictly improved . But it does not provide an actionable procedure4 for picking one or a few special models to return to practical users . In this paper , we5 consider optimization in Pareto set ( OPT-in-Pareto ) , the problem of finding Pareto models6 that optimize an extra reference criterion function within the Pareto set . This function can7 either encode a specific preference from the users , or represent a generic diversity measure8 for obtaining a set of diversified Pareto models that are representative of the whole Pareto9 set . Unfortunately , despite being a highly useful framework , efficient algorithms for OPT-10 in-Pareto have been largely missing , especially for large-scale , non-convex , and non-linear11 objectives in deep learning . A naive approach is to apply Riemannian manifold gradient12 descent on the Pareto set , which yields a high computational cost due to the need for eigen-13 calculation of Hessian matrices . We propose a first-order algorithm that approximately14 solves OPT-in-Pareto using only gradient information , with both high practical efficiency15 and theoretically guaranteed convergence property . Empirically , we demonstrate that our16 method works efficiently for a variety of challenging multi-task-related problems.17 1 INTRODUCTION18 Although machine learning tasks are traditionally framed as optimizing a single objective . Many modern19 applications , especially in areas like multitask learning , require finding optimal model parameters to minimize20 multiple objectives ( or tasks ) simultaneously . As the different objective functions may inevitably conflict21 with each other , the notion of optimality in multi-objective optimization ( MOO ) needs to be characterized by22 the Pareto set : the set of model parameters whose performance of all tasks can not be jointly improved.23 Focusing on the Pareto set allows us to filter out models that can be strictly improved . However , the Pareto24 set typically contains an infinite number of parameters that represent different trade-offs of the objectives.25 For m objectives ` 1 , . . . , ` m , the Pareto set is often an ( m− 1 ) dimensional manifold . It is both intractable26 and unnecessary to give practical users the whole exact Pareto set . A more practical demand is to find some27 user-specified special parameters in the Pareto set , which can be framed into the following optimization in28 Pareto set ( OPT-in-Pareto ) problem:29 Finding one or a set of parameters inside the Pareto set of ` 1 , . . . , ` m that minimize a reference criterion F .30 Here the criterion function F can be used to encode an informative user-specific preference on the objectives31 ` 1 , . . . , ` m , which allows us to provide the best models customized for different users . F can also be an32 non-informative measure that encourages , for example , the diversity of a set of model parameters . In this33 case , optimizing F in Pareto set gives a set of diversified Pareto models that are representative of the whole34 Pareto set , from which different users can pick their favorite models during the testing time.35 OPT-in-Pareto provides a highly generic and actionable framework for multi-objective learning and opti-36 mization . However , efficient algorithms for solving OPT-in-Pareto have been largely lagging behind in deep37 learning where the objective functions are non-convex and non-linear . Although has not been formally studied,38 a straightforward approach is to apply manifold gradient descent on F in the Riemannian manifold formed by39 the Pareto set ( Hillermeier , 2001 ; Bonnabel , 2013 ) . However , this casts prohibitive computational cost due40 to the need for eigen-computation of Hessian matrices of { ` i } . In the optimization and operation research41 literature , there has been a body of work on OPT-in-Pareto viewing it as a special bi-level optimization42 problem ( Dempe , 2018 ) . However , these works often heavily rely on the linearity and convexity assumptions43 and are not applicable to the non-linear and non-convex problems in deep learning ; see for examples in Ecker44 & Song ( 1994 ) ; Jorge ( 2005 ) ; Thach & Thang ( 2014 ) ; Liu & Ehrgott ( 2018 ) ; Sadeghi & Mohebi ( 2021 ) ( just45 to name a few ) . In comparison , the OPT-in-Pareto problem seems to be much less known and under-explored46 in the deep learning literature.47 In this work , we provide a practically efficient first-order algorithm for OPT-in-Pareto , using only gradient48 information of the criterion F and objectives { ` i } . Our method , named Pareto navigation gradient descent49 ( PNG ) , iteratively updates the parameters following a direction that carefully balances the descent on F and50 { ` i } , such that it guarantees to move towards the Pareto set of { ` i } when it is far away , and optimize F in a51 neighborhood of the Pareto set . Our method is simple , practically efficient and has theoretical guarantees.52 In empirical studies , we demonstrate that our method works efficiently for both optimizing user-specific53 criteria and diversity measures . In particular , for finding representative Pareto solutions , we propose an54 energy distance criterion whose minimizers distribute uniformly on the Pareto set asymptotically ( Hardin55 & Saff , 2004 ) , yielding a principled and efficient Pareto set approximation method that compares favorably56 with recent works such as Lin et al . ( 2019 ) ; Mahapatra & Rajan ( 2020 ) . We also apply PNG to improve the57 performance of JiGen ( Carlucci et al. , 2019b ) , a multi-task learning approach for domain generalization , by58 using the adversarial feature discrepancy as the criterion objective.59 Related Work There has been a rising interest in MOO in deep learning , mostly in the context of multi-task60 learning . But most existing methods can not be applied to the general OPT-in-Pareto problem . A large body61 of recent works focus on improving non-convex optimization for finding some model in the Pareto set , but62 can not search for a special model satisfying a specific criterion ( Chen et al. , 2018 ; Kendall et al. , 2018 ; Sener63 & Koltun , 2018 ; Yu et al. , 2020 ; Chen et al. , 2020 ; Wu et al. , 2020 ; Fifty et al. , 2020 ; Javaloy & Valera , 2021 ) .64 One exception is Mahapatra & Rajan ( 2020 ) ; Kamani et al . ( 2021 ) , which searches for models in the Pareto65 set that satisfy a constraint on the ratio between the different objectives . The problem they study can be66 viewed as a special instance of OPT-in-Pareto . However , their approaches are tied with special properties of67 the ratio constraint and do not apply to the general OPT-in-Pareto problem.68 There has also been increasing interest in finding a compact approximation of the Pareto set . Navon et al.69 ( 2020 ) ; Lin et al . ( 2020 ) use hypernetworks to approximate the map from linear scalarization weights to70 the corresponding Pareto solutions ; these methods could not fully profile non-convex Pareto fronts due71 to the limitation of linear scalarization ( Boyd et al. , 2004 ) , and the use of hypernetwork introduces extra72 optimization difficulty . Another line of works ( Lin et al. , 2019 ; Mahapatra & Rajan , 2020 ) approximate73 the Pareto set by training models with different user preference vectors that rank the relative importance74 of different tasks ; these methods need a good heuristic design of preference vectors , which requires prior75 knowledge of the Pareto front . Ma et al . ( 2020 ) leverages manifold gradient to conduct a local random walk76 on the Pareto set but suffers from the high computational cost . Deist et al . ( 2021 ) approximates the Pareto set77 by maximizing hypervolume , which requires prior knowledge for choosing a good reference vector.78 Multi-task learning can also be applied to improve the learning in many other domains including domain79 generalization ( Dou et al. , 2019 ; Carlucci et al. , 2019a ; Albuquerque et al. , 2020 ) , domain adaption ( Sun80 et al. , 2019 ; Luo et al. , 2021 ) , model uncertainty ( Hendrycks et al. , 2019 ; Zhang et al. , 2020 ; Xie et al. , 2021 ) ,81 adversarial robustness ( Yang & Vondrick , 2020 ) and semi-supervised learning ( Sohn et al. , 2020 ) . All of82 those applications utilize a linear scalarization to combine the multiple objectives and it is thus interesting to83 apply the proposed OPT-in-Pareto framework , which we leave for future work.84 2 BACKGROUND ON MULTI-OBJECTIVE OPTIMIZATION85 We introduce the background on multi-objective optimization ( MOO ) and Pareto optimality . For notation,86 we denote by [ m ] the integer set { 1 , 2 , .... , m } , and R+ the set of non-negative real numbers . Let Cm =87 { ω ∈ Rm+ , ∑m i=1 ωi = 1 } be the probability simplex . We denote by ‖·‖ the Euclidean norm.88 Let θ ∈ Rn be a parameter of interest ( e.g. , the weights in a deep neural network ) . Let ` ( θ ) =89 [ ` 1 ( θ ) , . . . , ` m ( θ ) ] be a set of objective functions that we want to minimize . For two parameters θ , θ′ ∈ Rn,90 we write ` ( θ ) ` ( θ′ ) if ` i ( θ ) ≥ ` i ( θ′ ) for all i ∈ [ m ] ; and write ` ( θ ) ` ( θ′ ) if ` ( θ ) ` ( θ′ ) and91 ` ( θ ) 6= ` ( θ′ ) . We say that θ is Pareto dominated ( or Pareto improved ) by θ′ if ` ( θ ) ` ( θ′ ) . We say that θ is92 Pareto optimal on a set Θ ⊆ Rn , denoted as θ ∈ Pareto ( Θ ) , if there exists no θ′ ∈ Θ such that ` ( θ ) ` ( θ′ ) .93 The Pareto global optimal set P∗∗ : = Pareto ( Rn ) is the set of points ( i.e. , θ ) which are Pareto optimal on94 the whole domain Rn . The Pareto local optimal set of ` , denoted by P∗ , is the set of points which are Pareto95 optimal on a neighborhood of itself:96 P∗ : = { θ ∈ Rn : there exists a neighborhood Nθ of θ , such that θ ∈ Pareto ( Nθ ) } . The ( local or global ) Pareto front is the set of objective vectors achieved by the Pareto optimal points , e.g.,97 the local Pareto front is F∗ = { ` ( θ ) : θ ∈ P∗ } . Because finding global Pareto optimum is intractable for98 non-convex objectives in deep learning , we focus on Pareto local optimal sets in this work ; in the rest of the99 paper , terms like “ Pareto set ” and “ Pareto optimum ” refer to Pareto local optimum by default.100 Pareto Stationary Points Similar to the case of single-objective optimization , Pareto local optimum implies a notion of Pareto stationarity defined as follows . Assume ` is differentiable on Rn . A point θ is called Pareto stationary if there must exists a set of non-negative weights ω1 , . . . , ωm with ∑m i=1 ωi = 1 , such that θ is a stationary point of the ω-weighted linear combination of the objectives : ` ω ( θ ) : = ∑m i=1 ωi ` i ( θ ) . Therefore , the set of Pareto stationary points , denoted by P , can be characterized by P : = { θ ∈ Θ : g ( θ ) = 0 } , g ( θ ) : = min ω∈Cm || m∑ i=1 ωi∇ ` i ( θ ) ||2 , ( 1 ) where g ( θ ) is the minimum squared gradient norm of ` ω among all ω in the probability simplex Cm on [ m ] .101 Because g ( θ ) can be calculated in practice , it provides an essential way to access Pareto local optimality.102 Finding Pareto Optimal Points A main focus of the MOO literature is to find a ( set of ) Pareto optimal103 points . The simplest approach is linear scalarization , which minimizes ` ω for some weight ω ( decided , e.g.,104 by the users ) in Cm . However , linear scalarization can only find Pareto points that lie on the convex envelop105 of the Pareto front ( see e.g. , Boyd et al. , 2004 ) , and hence does not give a complete profiling of the Pareto106 front when the objective functions ( and hence their Pareto front ) are non-convex.107 Multiple gradient descent ( MGD ) ( Désidéri , 2012 ) is an gradient-based algorithm that can converge to a Pareto local optimum that lies on either the convex or non-convex parts of the Pareto front , depending on the initialization . MGD starts from some initialization θ0 and updates θ at the t-th iteration by θt+1 ← θt − ξvt , vt : = arg max v∈Rn { min i∈ [ m ] ∇ ` i ( θt ) > v − 1 2 ‖v‖2 } , ( 2 ) where ξ is the step size and vt is an update direction that maximizes the worst descent rate among all108 objectives , since ∇ ` i ( θt ) > v ≈ ( ` i ( θt ) − ` i ( θt − ξv ) ) /ξ approximates the descent rate of objective ` i when109 following direction v. When using a sufficiently small step size ξ , MGD ensures to yield a Pareto improvement110 ( i.e , decreasing all the objectives ) on θt unless θt is Pareto ( local ) optimal ; this is because the optimization in111 Equation ( 2 ) always yields mini∈ [ m ] ∇ ` i ( θt ) > vt ≤ 0 ( otherwise we can simply flip the sign of vt ) .112 Using Lagrange strong duality , the solution of Equation ( 2 ) can be framed into vt = m∑ i=1 ωi , t∇ ` i ( θt ) , where { ωi , t } mi=1 = arg min ω∈Cm ‖∇θ ` ω ( θt ) ‖ . ( 3 ) It is easy to see from Equation ( 3 ) that the set of fixed points of MDG ( which satisfy vt = 0 ) coincides with113 the Pareto stationary set P∗.114 A key disadvantage of MGD , however , is that the Pareto point that it converges to depends on the initialization115 and other algorithm configurations in a rather implicated and complicated way . It is difficult to explicitly116 control MGD to make it converge to points with specific properties.117 3 OPTIMIZATION IN PARETO SET118 The Pareto set typically contains an infinite number of points . In the optimization in Pareto set ( OPT-in-119 Pareto ) problem , we are given an extra criterion function F ( θ ) in addition to the objectives ` , and we want to120 minimize F in the Pareto set of ` , that is,121 min θ∈P∗ F ( θ ) . ( 4 ) For example , one can find the Pareto point whose loss vector ` ( θ ) is the closest to a given reference point122 r ∈ Rm by choosing F ( θ ) = ‖ ` ( θ ) − r‖2 . We can also design F to encourages ` ( θ ) to be proportional to r,123 i.e. , ` ( θ ) ∝ r ; a constraint variant of this problem was considered in Mahapatra & Rajan ( 2020 ) .124 We can further generalize OPT-in-Pareto to allow the criterion F to depend on an ensemble of Pareto points125 { θ1 , ... , θN } jointly , that is,126 min θ1 , ... , θN∈P∗ F ( θ1 , ... , θN ) . ( 5 ) For example , if F ( θ1 , . . . , θN ) measures the diversity among { θi } Ni=1 , then optimizing it provides a set of diversified points inside the Pareto set P∗ . An example of diversity measure is F ( θ1 , . . . , θN ) = E ( ` ( θ1 ) , . . . , ` ( θN ) ) , with E ( ` 1 , . . . , ` N ) = ∑ i 6=j ‖ ` i − ` j‖−2 , ( 6 ) where E is known as an energy distance in computational geometry , whose minimizer can be shown to give127 an uniform distribution asymptotically when N →∞ ( Hardin & Saff , 2004 ) . This formulation is particularly128 useful when the users ’ preference is unknown during the training time , and we want to return an ensemble of129 models that well cover the different areas of the Pareto set to allow the users to pick up a model that fits their130 needs regardless of their preference . The problem of profiling Pareto set has attracted a line of recent works131 ( e.g. , Lin et al. , 2019 ; Mahapatra & Rajan , 2020 ; Ma et al. , 2020 ; Deist et al. , 2021 ) , but they rely on specific132 criterion or heuristics and do not address the general optimization of form Equation ( 5 ) .133 Manifold Gradient Descent One straightforward approach to OPT-in-Pareto is to deploy manifold gradient134 descent ( Hillermeier , 2001 ; Bonnabel , 2013 ) , which conducts steepest descent of F ( θ ) in the Riemannian135 manifold formed by the Pareto set P∗ . Initialized at θ0 ∈ P∗ , manifold gradient descent updates θt at the t-th136 iteration along the direction of the projection of∇F ( θt ) on the tangent space T ( θt ) at θt in P∗,137 θt+1 = θt − ξProjT ( θt ) ( ∇F ( θt ) ) . By using the stationarity characterization in Equation ( 1 ) , under proper regularity conditions , one can138 show that the tangent space T ( θt ) equals the null space of the Hessian matrix ∇2θ ` ωt ( θt ) , where ωt =139 arg minω∈Cm ‖∇θ ` ω ( θt ) ‖ . However , the key issue of manifold gradient descent is the high cost for calculating140 this null space of Hessian matrix . Although numerical techniques such as Krylov subspace iteration ( Ma141 et al. , 2020 ) or conjugate gradient descent ( Koh & Liang , 2017 ) can be applied , the high computational cost142 ( and the complicated implementation ) still impedes its application in large scale deep learning problems . See143 Section 1 for discussions on other related works.144 4 PARETO NAVIGATION GRADIENT DESCENT FOR OPT-IN-PARETO145 We now introduce our main algorithm , Pareto Navigating Gradient Descent ( PNG ) , which provides a practical146 approach to OPT-in-Pareto . For convenience , we focus on the single point problem in Equation ( 4 ) in the147 presentation . The generalization to the multi-point problem in Equation ( 5 ) is straightforward . We first148 introduce the main idea and then present theoretical analysis in Section 4.1.149 Main Idea We consider the general incremental updating rule of form θt+1 ← θt − ξvt , where ξ is the step size and vt is an update direction that we shall choose to achieve the following desiderata150 in balancing the decent of { ` i } and F :151 i ) When θt is far away from the Pareto set , we want to choose vt to give Pareto improvement to θt , moving it152 towards the Pareto set . The amount of Pareto improvement might depend on how far θt is to the Pareto set.153 ii ) If the directions that yield Pareto improvement are not unique , we want to choose the Pareto improvement154 direction that decreases F ( θ ) most.155 iii ) When θt is very close to the Pareto set , e.g. , having a small g ( θ ) , we want to fully optimize F ( θ ) .156 We achieve the desiderata above by using the vt that solves the following optimization : vt = arg min v∈Rn { 1 2 ‖∇F ( θt ) − v‖2 s.t . ∇θ ` i ( θt ) > v ≥ φt , ∀i ∈ [ m ] } , ( 7 ) where we want vt to be as close to ∇F ( θt ) as possible ( hence decrease F most ) , conditional on that the157 decreasing rate ∇θ ` i ( θt ) > vt of all losses ` i are lower bounded by a control parameter φt . A positive φt158 enforces that ∇θt ` i ( θ ) > vt is positive for all ` i , hence ensuring a Pareto improvement when the step size is159 sufficiently small . The magnitude of φt controls how much Pareto improvement we want to enforce , so we160 may want to gradually decrease φt when we move closer to the Pareto set . In fact , varying φt provides an161 intermediate updating direction between the vanilla gradient descent on F and MGD on { ` i } :162 i ) If φt = −∞ , we have vt = ∇F ( θt ) and it conducts a pure gradient descent on F without considering { ` i } .163 ii ) If φt → +∞ , then vt approaches to the MGD direction of { ` i } in Equation ( 2 ) without considering F .164 In this work , we propose to choose φt based on the minimum gradient norm g ( θt ) in Equation ( 1 ) as a surrogate indication of Pareto local optimality . In particular , we consider the following simple design : φt = { −∞ if g ( θt ) ≤ e , αtg ( θt ) if g ( θt ) > e , ( 8 ) where e is a small tolerance parameter and αt is a positive hyper-parameter . When g ( θt ) > e , we set φt to be165 proportional to g ( θt ) , to ensure Pareto improvement based on how far θt is to Pareto set . When g ( θt ) ≤ e,166 we set φt = −∞ which “ turns off ” the control and hence fully optimizes F ( θ ) .167 In practice , the optimization in Equation ( 7 ) can be solved efficiently by its dual form as follows.168 Theorem 1 . The solution vt of Equation ( 7 ) , if it exists , has a form of vt = ∇F ( θt ) + m∑ t=1 λi , t∇ ` i ( θt ) , ( 9 ) with { λi , t } mt=1 the solution of the following dual problem max λ∈Rm+ −1 2 ||∇F ( θt ) + m∑ i=1 λt∇ ` i ( θt ) ||2 + m∑ i=1 λiφt . ( 10 ) The optimization in Equation ( 10 ) can be solved efficiently for a small m ( e .. g , m ≤ 10 ) , which is the case169 for typical applications . We include the details of the practical implementation in Appendix B.170 4.1 THEORETICAL PROPERTIES171 We provide a theoretical quantification on how PNG guarantees to i ) move the solution towards the Pareto172 set ( Theorem 2 ) ; and ii ) optimize F in a neighborhood of Pareto set ( Theorem 3 ) . To simplify the result and173 highlight the intuition , we focus on the continuous time limit of PNG , which yields a differentiation equation174 dθt = −vtdt with vt defined in Equation ( 7 ) , where t ∈ R+ is a continuous integration time.175 Assumption 1 . Let { θt : t ∈ R+ } be a solution of dθt = −vtdt with vt in Equation ( 7 ) ; φk in Equation ( 8 ) ; 176 e > 0 ; and αt ≥ 0 , ∀t ∈ R+ . Assume F and ` are continuously differentiable on Rn , and lower bounded177 with F ∗ : = infθ∈Rn F ( θ ) > −∞ and ` ∗i : = infθ∈Rn ` i ( θ ) > −∞ . Assume supθ∈Rn ‖∇F ( θ ) ‖ ≤ c.178 Technically , dθt = −vtdt is a piecewise smooth dynamical system whose solution should be taken in the179 Filippov sense using the notion of differential inclusion ( Bernardo et al. , 2008 ) . The solution always exists180 under mild regularity conditions although it may not be unique . Our results below apply to all solutions.181 Pareto Optimization on ` We now show that the algorithm converges to the vicinity of Pareto set quantified by a notion of Pareto closure . For ≥ 0 , let P be the set of Pareto -stationary points : P = { θ ∈ Rn : g ( θ ) ≤ } . The Pareto closure of a set P , denoted by P is the set of points that perform no worse than at least one point in P , that is , P : = ∪θ∈P { θ } , { θ } = { θ′ ∈ Rn : ` ( θ′ ) ` ( θ ) } . Therefore , P is better than or at least as good as P in terms of Pareto efficiency.182 Theorem 2 ( Pareto Improvement on ` ) . Under Assumption 1 , assume θ0 6∈ Pe , and te is the first time when θte ∈ Pe , then for any time t < te , d dt ` i ( θt ) ≤ −αtg ( θt ) , min s∈ [ 0 , t ] g ( θs ) ≤ mini∈ [ m ] ( ` i ( θ0 ) − ` ∗i ) ∫ t 0 αsds . Therefore , the update yields Pareto improvement on ` when θt 6∈ Pe and αtg ( θt ) > 0.183 Further , if ∫ t 0 αsds = +∞ , then for any > e , there exists a finite time t ∈ R+ on which the solution enters184 P and stays within P afterwards , that is , we have θt ∈ P and θt ∈ P for any t ≥ t .185 Here we guarantee that θt must enter P for some time ( in fact infinitely often ) , but it is not confined in P .186 On the other hand , θt does not leave P after it first enters P thanks to the Pareto improvement property.187 Optimization on F We now show that PNG finds a local optimum of F inside the Pareto closure P in an188 approximate sense . We first show that a fixed point θ of the algorithm that is locally convex on F and ` must189 be a local optimum of F in the Pareto closure of { θ } , and then quantify the convergence of the algorithm.190 Lemma 1 . Under Assumption 1 , assume θt 6∈ Pe is a fixed point of the algorithm , that is , dθtdt = −vt = 0,191 and F , ` are convex in a neighborhood θt , then θt is a local minimum of F in the Pareto closure { θt } , that is,192 there exists a neighborhood of θt in which there exists no point θ′ such that F ( θ′ ) < F ( θt ) and ` ( θ′ ) ` ( θt ) .193 On the other hand , if θt ∈ Pe , we have vt = ∇F ( θt ) , and hence a fixed point with dθtdt = −vt = 0 is an194 unconstrained local minimum of F when F is locally convex on θt.195 Theorem 3 . Let > e and assume g : = supθ { g ( θ ) : θ ∈ P } < +∞ and supt≥0 αt < ∞ . Under Assumption 1 , when we initialize from θ0 ∈ P , we have min s∈ [ 0 , t ] ∥∥∥∥dθsds ∥∥∥∥2 ≤ F ( θ0 ) − F ∗t + 1t ∫ t 0 αs ( αsg + c √ g ) ds . In particular , if we have αt = α = const , then mins∈ [ 0 , t ] ‖dθs/ds‖ 2 = O ( 1/t+ α √ g ) .196 If ∫∞ 0 αγt dt < +∞ for some γ ≥ 1 , we have mins∈ [ 0 , t ] ‖dθs/ds‖ 2 = O ( 1/t+√g /t1/γ ) .197 Combining the results in Theorem 2 and 3 , we can see that the choice of sequence { αt : t ∈ R+ } controls how198 fast we want to decrease ` vs. F . Large αt yields faster descent on ` , but slower descent on F . Theoretically,199 using a sequence that satisfies ∫ αtdt = +∞ and ∫ αγt dt < +∞ for some γ > 1 allows us to ensure that200 both mins∈ [ 0 , t ] g ( θs ) and mins∈ [ 0 , t ] ‖dθ/ds‖ 2 converge to zero . If we use a constant sequence αt = α , it201 introduces an O ( α√g ) term that does not vanish as t→ +∞ . However , we can expect that g is small when202 is small for well-behaved functions . In practice , we find that constant αt works sufficiently well.203 5 EMPIRICAL RESULTS204 We introduce three applications of OPT-in-Pareto with PNG : Singleton Preference , Pareto approximation and205 improving multi-task based domain generalization method . We also conduct additional study on how the206 learning dynamics of PNG changes with different initialization and hyper-parameters ( αt and e ) , which are207 included in Appendix C.3 . Other additional results that are related to the experiments in Section 5.1 and 5.2208 and are included in the Appendix will be introduced later in their corresponding sections.209 5.1 FINDING PREFERRED PARETO MODELS210 We consider the synthetic example used in Lin et al . ( 2019 ) ; Mahapatra & Rajan ( 2020 ) , which consists of211 two losses : ` 1 ( θ ) = 1− exp ( −‖θ − η‖2 ) and ` 2 ( θ ) = 1− exp ( −‖θ + η‖2 ) , where η = n−1/2 and n = 10212 is dimension of the parameter θ.213 Ratio-based Criterion We first show that PNG can solve the search problem under the ratio constraint of214 objectives in Mahapatra & Rajan ( 2020 ) , i.e. , finding a point θ ∈ P∗ ∩Ω with Ω = { θ : r1 ` 1 ( θ ) = r2 ` 2 ( θ ) =215 ... = rm ` m ( θ ) } , given some preference vector r = [ r1 , ... , rm ] . We apply PNG with the non-uniformity216 score defined in Mahapatra & Rajan ( 2020 ) as the criterion , and compare with their algorithm called exact217 Pareto optimization ( EPO ) . We show in Figure 1 ( a ) - ( b ) the trajectory of PNG and EPO for searching models218 with different preference vector r , starting from the same randomly initialized point . Both PNG and EPO219 converge to the correct solutions but with different trajectories . This suggests that PNG is able to achieve220 the same functionality of finding ratio-constraint Pareto models as Mahapatra & Rajan ( 2020 ) ; Kamani et al.221 ( 2021 ) do but being versatile to handle general criteria . We refer readers to Appendix C.1.1 for more results222 with different choices of hyper-parameters and the experiment details.223 Other Criteria We demonstrate that PNG is able to find solutions for general choices of F . We consider224 the following designs of F : 1 ) weighted ` 2 distance w.r.t . a reference vector r ∈ Rm+ , that is , Fwd ( θ ) =225 i=1 ( ` i ( θ ) − ri ) 2/ri ; and 2 ) complex cosine : in which F is a complicated function related to the cosine226 of task objectives , i.e. , Fcs = − cos ( π ( ` 1 ( θ ) − r1 ) /2 ) + ( cos ( π ( ` ( θ2 ) − r2 ) ) + 1 ) 2 . Here the weighted ` 2227 distance can be viewed as finding a Pareto model that has the losses close to some target value r , which can be228 viewed as an alternative approach to partition the Pareto set . The design of complex cosine aims to test whether229 PNG is able to handle a very non-linear criterion function . In both cases , we take r1 = [ 0.2 , 0.4 , 0.6 , 0.8 ] and230 r2 = 1− r1 . We show in Fig 1 ( c ) - ( d ) the trajectory of PNG . As we can see , PNG is able to correctly find the231 optimal solutions of OPT-in-Pareto . We also test PNG on a more challenging ZDT2-variant used in Ma et al.232 ( 2020 ) and a larger scale MTL problem ( Liu et al. , 2019 ) . We refer readers to Appendix C.1.2 and C.1.3 for233 the setting and results.234 5.2 FINDING DIVERSE PARETO MODELS235 Setup We consider the problem of finding diversified points from the Pareto set by minimizing the energy236 distance criterion in Equation ( 6 ) . We use the same setting as Lin et al . ( 2019 ) ; Mahapatra & Rajan ( 2020 ) .237 We consider three benchmark datasets : ( 1 ) MultiMNIST , ( 2 ) MultiFashion , and ( 3 ) MultiFashion+MNIST.238 For each dataset , there are two tasks ( classifying the top-left and bottom-right images ) . We consider LeNet239 with multihead and train N = 5 models to approximate the Pareto set . For baselines , we compare with linear240 scalarization , MGD ( Sener & Koltun , 2018 ) , and EPO ( Mahapatra & Rajan , 2020 ) . For the MGD baseline,241 we find that naively running it leads to poor performance as the learned models are not diversified and thus we242 initialize the MGD with 60-epoch runs of linear scalarization with equally distributed preference weights and243 runs MGD for the later 40 epoch . We refer the reader to Appendix C.2.1 for more details of the experiments.244 Metric and Result We measure the quality of how well the found models { θ1 , . . . , θN } approximate the245 Pareto set using two standard metrics : Inverted Generational Distance Plus ( IGD+ ) ( Ishibuchi et al. , 2015 ) 246 and hypervolume ( HV ) ( Zitzler & Thiele , 1999 ) ; see Appendix C.2.2 for their definitions . We run all the247 methods with 5 independent trials and report the averaged value and its standard deviation in Table 1 . We248 report the scores calculated based on loss ( cross-entropy ) and accuracy on the test set . The bolded values249 indicate the best result with p-value less than 0.05 ( using matched pair t-test ) . In most cases , PNG improves250 the baselines by a large margin . We include ablation studies in Appendix C.2.3 and additional comparisons251 with the second-order approach proposed by Ma et al . ( 2020 ) in Appendix C.2.4.252 5.3 APPLICATION TO MULTI-TASK BASED DOMAIN GENERALIZATION ALGORITHM253 JiGen ( Carlucci et al. , 2019b ) learns a domain generalizable model by learning two tasks based on linear254 scalarization , which essentially searches for a model in the Pareto set and requires choosing the weight of255 linear scalarization carefully . It is thus natural to study whether there is a better mechanism that dynamically256 adjusts the weights of the two losses so that we eventually learn a better model . Motivated by the adversarial257 feature learning ( Ganin et al. , 2016 ) , we propose to improve JiGen such that the latent feature representations258 of the two tasks are well aligned . This can be framed into an OPT-in-Pareto problem where the criterion is259 the discrepancy of the latent representations ( implemented using an adversarial discrepancy module in the260 network ) of the two tasks . PNG is applied to solve the optimization . We evaluate the methods on PACS ( Li261 et al. , 2017 ) , which covers 7 object categories and 4 domains ( Photo , Art Paintings , Cartoon , and Sketches ) .262 The model is trained on three domains and tested on the rest of them . Our approach is denoted as JiGen+PNG263 and we also include JiGen + adv , which simply adds the adversarial loss as regularization and two other264 baseline methods ( D-SAM ( D ’ Innocente & Caputo , 2018 ) and DeepAll ( Carlucci et al. , 2019b ) ) . For the three265 JiGen based approaches , we run 3 independent trials and for the other two baselines , we report the results in266 their original papers . Table 2 shows the result using ResNet-18 , which demonstrates the improvement by the267 application of the OPT-in-Pareto framework . We also include the results using AlexNet in the Appendix . We268 refer readers to Appendix C.4 for the additional results and more experiment details.269 6 CONCLUSION270 This paper studies the OPT-in-Pareto , a problem that has been studied in operation research with restrictive271 linear or convexity assumption but largely under-explored in deep learning literature , in which the objectives272 are non-linear and non-convex . Applying algorithms such as manifold gradient descent requires eigen-273 computation of the Hessian matrix at each iteration and thus can be expensive . We propose a first-order274 approximation algorithm called Pareto Navigation Gradient Descent ( PNG ) with theoretically guaranteed275 descent and convergence property to solve OPT-in-Pareto.276 REFERENCES277 Isabela Albuquerque , Nikhil Naik , Junnan Li , Nitish Keskar , and Richard Socher . Improving out-of-278 distribution generalization via multi-task self-supervised pretraining . arXiv preprint arXiv:2003.13525,279 2020.280 Mario Bernardo , Chris Budd , Alan Richard Champneys , and Piotr Kowalczyk . Piecewise-smooth dynamical281 systems : theory and applications , volume 163 . Springer Science & Business Media , 2008.282 Silvere Bonnabel . Stochastic gradient descent on riemannian manifolds . IEEE Transactions on Automatic283 Control , 58 ( 9 ) :2217–2229 , 2013.284 Stephen Boyd , Stephen P Boyd , and Lieven Vandenberghe . Convex optimization . Cambridge university press,285 2004.286 Fabio M. Carlucci , Antonio D ’ Innocente , Silvia Bucci , Barbara Caputo , and Tatiana Tommasi . Domain287 generalization by solving jigsaw puzzles . In Proceedings of the IEEE/CVF Conference on Computer Vision288 and Pattern Recognition ( CVPR ) , June 2019a.289 Fabio M Carlucci , Antonio D ’ Innocente , Silvia Bucci , Barbara Caputo , and Tatiana Tommasi . Domain290 generalization by solving jigsaw puzzles . In Proceedings of the IEEE/CVF Conference on Computer Vision291 and Pattern Recognition , pp . 2229–2238 , 2019b.292 Zhao Chen , Vijay Badrinarayanan , Chen-Yu Lee , and Andrew Rabinovich . Gradnorm : Gradient normalization293 for adaptive loss balancing in deep multitask networks . In International Conference on Machine Learning,294 pp . 794–803 . PMLR , 2018.295 Zhao Chen , Jiquan Ngiam , Yanping Huang , Thang Luong , Henrik Kretzschmar , Yuning Chai , and Dragomir296 Anguelov . Just pick a sign : Optimizing deep multitask models with gradient sign dropout . In H. Larochelle,297 M. Ranzato , R. Hadsell , M. F. Balcan , and H. Lin ( eds . ) , Advances in Neural Information Processing298 Systems , volume 33 , pp . 2039–2050 . Curran Associates , Inc. , 2020 . URL https : //proceedings.299 neurips.cc/paper/2020/file/16002f7a455a94aa4e91cc34ebdb9f2d-Paper.pdf.300 Timo M Deist , Monika Grewal , Frank JWM Dankers , Tanja Alderliesten , and Peter AN Bosman.301 Multi-objective learning to predict pareto fronts using hypervolume maximization . arXiv preprint302 arXiv:2102.04523 , 2021.303 Stephan Dempe . Bilevel optimization : theory , algorithms and applications . TU Bergakademie Freiberg,304 Fakultät für Mathematik und Informatik , 2018.305 Jean-Antoine Désidéri . Multiple-gradient descent algorithm ( mgda ) for multiobjective optimization . Comptes306 Rendus Mathematique , 350 ( 5-6 ) :313–318 , 2012.307 Qi Dou , Daniel C Castro , Konstantinos Kamnitsas , and Ben Glocker . Domain generalization via model-308 agnostic learning of semantic features . arXiv preprint arXiv:1910.13580 , 2019.309 Antonio D ’ Innocente and Barbara Caputo . Domain generalization with domain-specific aggregation modules.310 In German Conference on Pattern Recognition , pp . 187–198 . Springer , 2018.311 Joseph G Ecker and Jung Hwan Song . Optimizing a linear function over an efficient set . Journal of312 Optimization Theory and Applications , 83 ( 3 ) :541–563 , 1994.313 Christopher Fifty , Ehsan Amid , Zhe Zhao , Tianhe Yu , Rohan Anil , and Chelsea Finn . Measuring and314 harnessing transference in multi-task learning . arXiv preprint arXiv:2010.15413 , 2020.315 Yaroslav Ganin and Victor Lempitsky . Unsupervised domain adaptation by backpropagation . In International316 conference on machine learning , pp . 1180–1189 . PMLR , 2015.317 Yaroslav Ganin , Evgeniya Ustinova , Hana Ajakan , Pascal Germain , Hugo Larochelle , François Laviolette,318 Mario March , and Victor Lempitsky . Domain-adversarial training of neural networks . Journal of Machine319 Learning Research , 17 ( 59 ) :1–35 , 2016 . URL http : //jmlr.org/papers/v17/15-239.html.320 DP Hardin and EB Saff . Discretizing manifolds via minimum energy points . Notices of the AMS , 51 ( 10 ) :321 1186–1194 , 2004.322 Dan Hendrycks , Mantas Mazeika , Saurav Kadavath , and Dawn Song . Using self-supervised learning can323 improve model robustness and uncertainty . In H. Wallach , H. Larochelle , A. Beygelzimer , F. d'Alché-Buc,324 E. Fox , and R. Garnett ( eds . ) , Advances in Neural Information Processing Systems , volume 32 . Cur-325 ran Associates , Inc. , 2019 . URL https : //proceedings.neurips.cc/paper/2019/file/326 a2b15837edac15df90721968986f7f8e-Paper.pdf.327 Claus Hillermeier . Generalized homotopy approach to multiobjective optimization . Journal of Optimization328 Theory and Applications , 110 ( 3 ) :557–583 , 2001.329 Hisao Ishibuchi , Hiroyuki Masuda , Yuki Tanigaki , and Yusuke Nojima . Modified distance calculation in330 generational distance and inverted generational distance . In International conference on evolutionary331 multi-criterion optimization , pp . 110–125 . Springer , 2015.332 Adrián Javaloy and Isabel Valera . Rotograd : Dynamic gradient homogenization for multi-task learning . arXiv333 preprint arXiv:2103.02631 , 2021.334 Jesús M Jorge . A bilinear algorithm for optimizing a linear function over the efficient set of a multiple335 objective linear programming problem . Journal of Global Optimization , 31 ( 1 ) :1–16 , 2005.336 Mohammad Mahdi Kamani , Rana Forsati , James Z Wang , and Mehrdad Mahdavi . Pareto efficient fairness in337 supervised learning : From extraction to tracing . arXiv preprint arXiv:2104.01634 , 2021.338 Alex Kendall , Yarin Gal , and Roberto Cipolla . Multi-task learning using uncertainty to weigh losses for339 scene geometry and semantics . In Proceedings of the IEEE conference on computer vision and pattern340 recognition , pp . 7482–7491 , 2018.341 Pang Wei Koh and Percy Liang . Understanding black-box predictions via influence functions . In International342 Conference on Machine Learning , pp . 1885–1894 . PMLR , 2017.343 Da Li , Yongxin Yang , Yi-Zhe Song , and Timothy M. Hospedales . Deeper , broader and artier domain344 generalization . In Proceedings of the IEEE International Conference on Computer Vision ( ICCV ) , Oct345 2017.346 Da Li , Yongxin Yang , Yi-Zhe Song , and Timothy Hospedales . Learning to generalize : Meta-learning for347 domain generalization . In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32,348 2018a.349 Ya Li , Xinmei Tian , Mingming Gong , Yajing Liu , Tongliang Liu , Kun Zhang , and Dacheng Tao . Deep350 domain generalization via conditional invariant adversarial networks . In Proceedings of the European351 Conference on Computer Vision ( ECCV ) , pp . 624–639 , 2018b.352 Xi Lin , Hui-Ling Zhen , Zhenhua Li , Qingfu Zhang , and Sam Kwong . Pareto multi-task learning . arXiv353 preprint arXiv:1912.12854 , 2019.354 Xi Lin , Zhiyuan Yang , Qingfu Zhang , and Sam Kwong . Controllable pareto multi-task learning . arXiv355 preprint arXiv:2010.06313 , 2020.356 Shikun Liu , Edward Johns , and Andrew J Davison . End-to-end multi-task learning with attention . In357 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp . 1871–1880,358 2019.359 Zhengliang Liu and Matthias Ehrgott . Primal and dual algorithms for optimization over the efficient set.360 Optimization , 67 ( 10 ) :1661–1686 , 2018.361 Xiaoyuan Luo , Shaolei Liu , Kexue Fu , Manning Wang , and Zhijian Song . A learnable self-supervised task362 for unsupervised domain adaptation on point clouds . arXiv preprint arXiv:2104.05164 , 2021.363 Pingchuan Ma , Tao Du , and Wojciech Matusik . Efficient continuous pareto exploration in multi-task learning.364 In International Conference on Machine Learning , pp . 6522–6531 . PMLR , 2020.365 Debabrata Mahapatra and Vaibhav Rajan . Multi-task learning with user preferences : Gradient descent with366 controlled ascent in pareto optimization . In International Conference on Machine Learning , pp . 6597–6607.367 PMLR , 2020.368 Aviv Navon , Aviv Shamsian , Gal Chechik , and Ethan Fetaya . Learning the pareto front with hypernetworks.369 arXiv preprint arXiv:2010.04104 , 2020.370 Javad Sadeghi and Hossein Mohebi . Solving optimization problems over the weakly efficient set . Numerical371 Functional Analysis and Optimization , pp . 1–33 , 2021.372 Ozan Sener and Vladlen Koltun . Multi-task learning as multi-objective optimization . In S. Bengio , H. Wallach,373 H. Larochelle , K. Grauman , N. Cesa-Bianchi , and R. Garnett ( eds . ) , Advances in Neural Information Pro-374 cessing Systems , volume 31 . Curran Associates , Inc. , 2018 . URL https : //proceedings.neurips.375 cc/paper/2018/file/432aca3a1e345e339f35a30c8f65edce-Paper.pdf.376 Nathan Silberman , Derek Hoiem , Pushmeet Kohli , and Rob Fergus . Indoor segmentation and support377 inference from rgbd images . In European conference on computer vision , pp . 746–760 . Springer , 2012.378 Kihyuk Sohn , David Berthelot , Nicholas Carlini , Zizhao Zhang , Han Zhang , Colin A Raffel , Ekin Do-379 gus Cubuk , Alexey Kurakin , and Chun-Liang Li . Fixmatch : Simplifying semi-supervised learning380 with consistency and confidence . In H. Larochelle , M. Ranzato , R. Hadsell , M. F. Balcan , and381 H. Lin ( eds . ) , Advances in Neural Information Processing Systems , volume 33 , pp . 596–608 . Cur-382 ran Associates , Inc. , 2020 . URL https : //proceedings.neurips.cc/paper/2020/file/383 06964dce9addb1c5cb5d6e3d9838f733-Paper.pdf.384 Yu Sun , Eric Tzeng , Trevor Darrell , and Alexei A Efros . Unsupervised domain adaptation through self-385 supervision . arXiv preprint arXiv:1909.11825 , 2019.386 Phan Thien Thach and TV Thang . Problems with resource allocation constraints and optimization over the387 efficient set . Journal of Global Optimization , 58 ( 3 ) :481–495 , 2014.388 Sen Wu , Hongyang R. Zhang , and Christopher Ré . Understanding and improving information transfer389 in multi-task learning . In International Conference on Learning Representations , 2020 . URL https:390 //openreview.net/forum ? id=SylzhkBtDB.391 Sang Michael Xie , Ananya Kumar , Robbie Jones , Fereshte Khani , Tengyu Ma , and Percy Liang . In-n-out : Pre-392 training and self-training using auxiliary information for out-of-distribution robustness . In International393 Conference on Learning Representations , 2021 . URL https : //openreview.net/forum ? id=394 jznizqvr15J.395 Junfeng Yang and Carl Vondrick . Multitask learning strengthens adversarial robustness . 2020.396 Tianhe Yu , Saurabh Kumar , Abhishek Gupta , Sergey Levine , Karol Hausman , and Chelsea Finn . Gra-397 dient surgery for multi-task learning . In H. Larochelle , M. Ranzato , R. Hadsell , M. F. Balcan , and398 H. Lin ( eds . ) , Advances in Neural Information Processing Systems , volume 33 , pp . 5824–5836 . Cur-399 ran Associates , Inc. , 2020 . URL https : //proceedings.neurips.cc/paper/2020/file/400 3fe78a8acf5fda99de95303940a2420c-Paper.pdf.401 Linfeng Zhang , Muzhou Yu , Tong Chen , Zuoqiang Shi , Chenglong Bao , and Kaisheng Ma . Auxiliary training:402 Towards accurate and robust models . In Proceedings of the IEEE/CVF Conference on Computer Vision403 and Pattern Recognition , pp . 372–381 , 2020.404 Eckart Zitzler and Lothar Thiele . Multiobjective evolutionary algorithms : a comparative case study and the405 strength pareto approach . IEEE transactions on Evolutionary Computation , 3 ( 4 ) :257–271 , 1999.406 A THEORETICAL ANALYSIS407 Theorem 1 [ Dual of Equation ( 7 ) ] The solution vt of Equation ( 7 ) , if it exists , has a form of vt = ∇F ( θt ) + m∑ i=1 λi , t∇ ` i ( θt ) , with { λi , t } mi=1 the solution of the following dual problem max λ∈Rm+ −1 2 ∥∥∥∥∥∇F ( θt ) + m∑ i=1 λt∇ ` i ( θt ) ∥∥∥∥∥ 2 + m∑ i=1 λiφt , where Rm+ is the set of nonnegative m-dimensional vectors , that is , Rm+ = { λ ∈ Rm : λi ≥ 0 , ∀i ∈ [ m ] } .408 Proof . By introducing Lagrange multipliers , the optimization in Equation ( 7 ) is equivalent to the following409 minimax problem:410 min v∈Rn max λ∈Rm+ 1 2 ‖∇F ( θt ) − v‖2 + m∑ i=1 λi ( φt −∇ ` i ( θt ) > v ) . With strong duality of convex quadratic programming ( assuming the primal problem is feasible ) , we can exchange the order of min and max , yielding max λ∈Rm+ { Φ ( λ ) : = min v∈Rn 1 2 ‖∇F ( θt ) − v‖2 + m∑ i=1 λi ( φt −∇ ` i ( θt ) > v ) } . It is easy to see that the minimization w.r.t . v is achieved when v = ∇F ( θt ) + ∑m i=1 λi∇ ` i ( θt ) . Correspond-411 ingly , the Φ ( λ ) has the following dual form:412 max λ∈Rm+ −1 2 ∥∥∥∥∥∇F ( θt ) + m∑ i=1 λi∇ ` i ( θt ) ∥∥∥∥∥ 2 + m∑ i=1 λiφt . This concludes the proof.413 Theorem 2 [ Pareto Improvement on ` ] Under Assumption 1 , assume θ0 6∈ Pe , and te is the first time when θte ∈ Pe , then for any time t < te , d dt ` i ( θt ) ≤ −αtg ( θt ) , min s∈ [ 0 , t ] g ( θt ) ≤ mini∈ [ m ] ( ` i ( θ0 ) − ` ∗i ) ∫ t 0 αsds . Therefore , the update yields Pareto improvement on ` when θt 6∈ Pe and αtg ( θt ) > 0.414 Further , if ∫ t 0 αsds = +∞ , then for any > e , there exists a finite time t ∈ R+ on which the solution enters415 P and stays within P afterwards , that is , we have θt ∈ P and θt ∈ P for any t ≥ t .416 Proof . i ) When t < te , we have g ( θt ) > e and hence d dt ` i ( θt ) = −∇ ` i ( θt ) > vt ≤ −φt = −αtg ( θt ) , ( 11 ) where we used the constraint of∇ ` i ( θt ) > vt ≥ φt in Equation ( 7 ) . Therefore , we yield strict decent on all the417 losses { ` i } when αtg ( θt ) > 0.418 ii ) Integrating both sides of Equation ( 11 ) : min s∈ [ 0 , t ] g ( θs ) ≤ ∫ t 0 αsg ( θs ) ds∫ t 0 αsds ≤ ` i ( θ0 ) − ` i ( θt ) ∫ t 0 αsds ≤ ` i ( θ0 ) − ` ∗∫ t 0 αsds . This yields the result since it holds for every i ∈ [ m ] .419 If ∫∞ 0 αtdt = +∞ , then we have mins∈ [ 0 , t ] g ( θs ) → 0 when t → +∞ . Assume there exists an > e,420 such that θt never enters P at finite t. Then we have g ( θt ) ≥ for t ∈ R+ , which contradicts with421 mins∈ [ 0 , t ] g ( θs ) → 0.422 iii ) Assume there exists a finite time t′ ∈ ( t , +∞ ) such that θt′ 6∈ P . Because > e and g is continuous , Pe423 is in the interior of P ⊆ P . Therefore , the trajectory leading to θt′ 6∈ P must pass through P \Pe at some424 point , that is , there exists a point t′′ ∈ [ t , t′ ) , such that { θt : t ∈ [ t′′ , t′ ] } 6∈ Pe . But because the algorithm can425 not increase any objective ` i outside of Pe , we must have ` ( θt′ ) ` ( θt′′ ) , yielding that θt′ ∈ { θt′′ } ⊆ P ,426 where { θt′′ } is the Pareto closure of { θt′′ } ; this contradicts with the assumption.427 Lemma 1 Under Assumption 1 , assume θt 6∈ Pe is a fixed point of the algorithm , that is , dθtdt = −vt = 0,428 and F , ` are convex in a neighborhood θt , then θt is a local minimum of F in the Pareto closure { θt } ,429 that is , there exists a neighborhood of θt in which there exists no point θ′ such that F ( θ′ ) < F ( θt ) and430 ` ( θ′ ) ` ( θt ) .431 Proof . Note that minimizing F in { θt } can be framed into a constrained optimization problem : min θ F ( θ ) s.t . ` i ( θ ) ≤ ` i ( θt ) , ∀i ∈ [ m ] . In addition , by assumption , θ = θt satisfies vt = ∇F ( θt ) + ∑m i=1 λi , t∇ ` i ( θt ) = 0 , which is the KKT432 stationarity condition of the constrained optimization . It is also obvious to check that θ = θt satisfies the433 feasibility and slack condition trivially . Combining this with the local convexity assumption yields the434 result.435 Theorem 3 [ Optimization of F ] Let > e and assume g : = supθ { g ( θ ) : θ ∈ P } < +∞ and supt≥0 αt < ∞ . Under Assumption 1 , when we initialize from θ0 ∈ P , we have min s∈ [ 0 , t ] ∥∥∥∥dθsds ∥∥∥∥2 ≤ F ( θ0 ) − F ∗t + 1t ∫ t 0 αs ( αsg + c √ g ) ds . In particular , if we have αt = α = const , then mins∈ [ 0 , t ] ‖dθs/ds‖ 2 = O ( 1/t+ α √ g ) .436 If ∫∞ 0 αγt dt < +∞ for some γ ≥ 1 , we have mins∈ [ 0 , t ] ‖dθs/ds‖ 2 = O ( 1/t+√g /t1/γ ) .437 Proof . i ) The slack condition of the constrained optimization in Equation ( 7 ) says that λi , t ( ∇ ` i ( θt ) > vt − φt ) = 0 , ∀i ∈ [ m ] . ( 12 ) This gives that ‖vt‖2 = ( ∇F ( θt ) + m∑ i=1 λi , t∇ ` i ( θt ) ) > vt = ∇F ( θt ) > vt + m∑ i=1 λi , tφt //plugging Equation ( 12 ) . ( 13 ) If θt 6∈ Pe , we have φt = αtg ( θt ) and this gives d dt F ( θt ) = −∇F ( θt ) > vt = −‖vt‖2 + m∑ i=1 λi , tφt = − ∥∥∥∥dθtdt ∥∥∥∥2 + m∑ i=1 λi , tαtg ( θt ) If θt is in the interior of Pe , then we run typical gradient descent of F and hence has d dt F ( θt ) = −‖vt‖2 = − ∥∥∥∥dθtdt ∥∥∥∥2 . If θt is on the boundary of Pe , then by the definition of differential inclusion , dθ/dt belongs to the convex hull of the velocities that it receives from either side of the boundary , yielding that d dt F ( θt ) = − ∥∥∥∥dθtdt ∥∥∥∥2 + β m∑ i=1 λi , tαtg ( θt ) ≤ − ∥∥∥∥dθtdt ∥∥∥∥2 + m∑ i=1 λi , tαtg ( θt ) , where β ∈ [ 0 , 1 ] . Combining all the cases gives d dt F ( θt ) ≤ − ∥∥∥∥dθtdt ∥∥∥∥2 + m∑ i=1 λi , tαtg ( θt ) . Integrating this yields min s∈ [ 0 , t ] ∥∥∥∥dθsds ∥∥∥∥2 ≤ 1t ∫ t 0 ∥∥∥∥dθsds ∥∥∥∥2 ds ≤ F ( θ0 ) − F ∗t + 1t ∫ t 0 m∑ i=1 λi , sαsg ( θs ) ds ≤ F ( θ0 ) − F ∗ t + 1 t ∫ t 0 αs ( αsg + c √ g ) ds , where the last step used Lemma 2 with φt = αtg ( θt ) : m∑ i=1 λi , tαtg ( θt ) ≤ α2t g ( θt ) + cαt √ g ( θt ) ≤ α2t g + cαt √ g , and here we used g ( θt ) ≤ g because the trajectory is contained in P following Theorem 2.438 The remaining results follow Lemma 4.439 A.0.1 TECHNICAL LEMMAS440 Lemma 2 . Assume Assumption 1 holds . Define g ( θ ) = minω∈Cm ‖ ∑m i=1 ωi∇ ` i ( θ ) ‖ 2 , where Cm is the probability simplex on [ m ] . Then for the vt and λi , t defined in Equation ( 7 ) and Equation ( 10 ) , we have m∑ i=1 λi , tg ( θt ) ≤ max ( φt + c √ g ( θt ) , 0 ) . Proof . The slack condition of the constrained optimization in Equation ( 7 ) says that441 λi , t ( ∇ ` i ( θ ) > vt − φt ) = 0 , ∀i ∈ [ m ] . Sum the equation over i ∈ [ m ] and note that vt = ∇F ( θt ) + ∑m i=1 λi , t∇ ` i ( θt ) . We get∥∥∥∥∥ m∑ i=1 λi , t∇ ` i ( θt ) ∥∥∥∥∥ 2 + ( m∑ i=1 λi , t∇ ` i ( θt ) ) > ∇F ( θ ) − m∑ i=1 λi , tφt = 0 . ( 14 ) Define xt = ∥∥∥∥∥ m∑ i=1 λi , t∇ ` i ( θt ) ∥∥∥∥∥ 2 , λ̄t = m∑ i=1 λi , t , gt = g ( θt ) = min ω∈Cm ∥∥∥∥∥ m∑ i=1 ωi∇ ` i ( θt ) ∥∥∥∥∥ 2 . Then it is easy to see that xt ≥ λ̄2t gt . Using Cauchy-Schwarz inequality , ∣∣∣∣∣∣ ( m∑ i=1 λi , t∇ ` i ( θ ) ) > ∇F ( θt ) ∣∣∣∣∣∣ ≤ ‖∇F ( θt ) ‖ ∥∥∥∥∥ m∑ i=1 λi , t∇ ` i ( θ ) ∥∥∥∥∥ ≤ c√xt , where we used ‖∇F ( θt ) ‖ ≤ c by Assumption 1 . Combining this with Equation ( 14 ) , we have∣∣xt − λ̄tφt∣∣ ≤ c√xt . Applying Lemma 3 yields the result.442 Lemma 3 . Assume φ ∈ R , and x , λ , c , g ∈ R+ are non-negative real numbers and they satisfy |x− λφ| ≤ c √ x , x ≥ λ2g . Then we have λg ≤ max ( 0 , φ+ c√g ) .443 Proof . Square the first equation , we get f ( x ) : = ( x− λφ ) 2 − c2x ≤ 0 , where f is a quadratic function . To ensure that f ( x ) ≤ 0 has a solution that satisfies x ≥ λ2g , we need to have f ( λ2g ) ≤ 0 , that is , f ( λ2g ) = ( λ2g − λφ ) 2 − c2λ2g ≤ 0 . This can hold under two cases:444 Case 1 : λ = 0 ; 445 Case 2 : |λg − φ| ≤ c√g , and hence φ− c√g ≤ λg ≤ φ+ c√g.446 Under both case , we have λg ≤ max ( 0 , φ+ c√g ) . 447 Lemma 4 . Let { αt : t ∈ R+ } ⊆ R+ be a non-negative sequence with A : = ( ∫∞ 0 αγt dt ) 1/γ < ∞ , where γ ≥ 1 , and B = supt αt < ∞ . Then we have 1 t ∫ t 0 ( α2s + αs ) ds ≤ ( B + 1 ) At−1/γ . Proof . Let η = γγ−1 , so that 1/η + 1/γ = 1 . We have by Holder ’ s inequality , ∫ t 0 αsds ≤ ( ∫ t 0 αγsds ) 1/γ ( ∫ t 0 1ηds ) 1/η ≤ At1/η = At1−1/γ . and hence 1 t ∫ t 0 ( α2s + αs ) ds ≤ B + 1 t ∫ t 0 αsds ≤ ( B + 1 ) At−1/γ . 448 Algorithm 1 Pareto Navigating Gradient Descent 1 : Initialize θ0 ; decide the step size ξ , and the control function φ in Equation ( 8 ) ( including the threshold e > 0 and the descending rate { αt } ) . 2 : for iteration t do θt+1 ← θt − ξvt , vt = ∇F ( θt ) + m∑ i=1 λi , t∇ ` i ( θt ) , ( 15 ) where λi , t = 0 , ∀i ∈ [ m ] if g ( θt ) ≤ e , and { λi , t } mt=1 is the solution of Equation ( 10 ) with φ ( θt ) = αtg ( θt ) when g ( θt ) > e. 3 : end for B PRACTICAL IMPLEMENTATION449 Hyper-parameters Our algorithm introduces two hyperparameters { αt } and e over vanilla gradient descent.450 We use constant sequence αt = α and we take α = 0.5 unless otherwise specified . We choose e by451 e = γe0 , where e0 is an exponentially discounted average of 1m ∑m i=1 ‖∇ ` i ( θt ) ‖ 2 over the trajectory so that452 it automatically scales with the magnitude of the gradients of the problem at hand . In the experiments of this453 paper , we simply fix γ = 0.1 unless specified.454 Solving the Dual Problem Our method requires to calculate { λi , t } mt=1 with the dual optimization problem455 in Equation ( 10 ) , which can be solved with any off-the-shelf convex quadratic programming tool . In this456 work , we use a very simple projected gradient descent to approximately solve Equation ( 10 ) . We initialize457 { λi , t } mt=1 with a zero vector and terminate when the difference between the last two iterations is smaller than458 a threshold or the algorithm reaches the maximum number of iterations ( we use 100 in all experiments ) .459 The whole algorithm procedure is summarized in Algorithm 1.460 C EXPERIMENTS461 C.1 FINDING PREFERRED PARETO MODELS462 C.1.1 RATIO-BASED CRITERION463 The non-uniformity score from ( Mahapatra & Rajan , 2020 ) that we use in Figure 1 is defined as FNU ( θ ) = m∑ t=1 ˆ̀ t ( θ ) log ( ˆ̀ t ( θ ) 1/m ) , ˆ̀t ( θ ) = rt ` t ( θ ) ∑ s∈ [ m ] rs ` s ( θ ) . ( 16 ) We fix the other experiment settings the same as Mahapatra & Rajan ( 2020 ) and use γ = 0.01 and α = 0.25464 for this experiment reported in the main text . We defer the ablation studies on the hyper-parameter α and γ to465 Section C.3.466 C.1.2 ZDT2-VARIANT467 We consider the ZDT2-Variant example used in Ma et al . ( 2020 ) with the same experiment setting , in468 which the Pareto set is a cylindrical surface , making the problem more challenging . We consider the469 same criteria , e.g . weighted distance and complex cosine used in the main context with different choices470 of r1 = [ 0.2 , 0.4 , 0.6 , 0.8 ] . We use the default hyper-parameter set up , choosing α = 0.5 and r = 0.1.471 For complex cosine , we use MGD updating for the first 150 iterations . Figure 2 shows the trajectories,472 demonstrating that PNG works pretty well for the more challenging ZDT2-Variant tasks.473 C.1.3 GENERAL CRITERIA : THREE-TASK LEARNING ON THE NYUV2 DATASET474 We show that PNG is able to handle large-scale multitask learning problems by deploying it on a three-475 task learning problem ( segmentation , depth estimation , and surface normal prediction ) on NYUv2 dataset476 ( Silberman et al. , 2012 ) . The main goal of this experiment is to show that : 1 . PNG is able to handle477 OPT-in-Pareto in a large-scale neural network ; 2 . With a proper design of criteria , PNG enables to do478 targeted fine-tuning that pushes the model to move towards a certain direction . We consider the same479 training protocol as Liu et al . ( 2019 ) and use the MTAN network architecture . Start with a model trained480 with equally weighted linear scalarization and our goal is to further improve the model ’ s performance481 on segmentation and surface normal estimation while allowing some sacrifice on depth estimation . This482 can be achieved by many different choices of criterion and in this experiment , we consider the following483 design : F ( θ ) = ( ` seg ( θ ) × ` surface ( θ ) ) / ( 0.001 + ` depth ( θ ) ) . Here ` seg , ` surface and ` depth are the loss functions484 for segmentation , surface normal prediction and depth estimation , respectively . The constant 0.001 in the485 denominator is for numeric stability . We point out that our design of criterion is a simple heuristic and might486 not be an optimal choice and the key question we study here is to verify the functionality of the proposed487 PNG . As suggested by the open-source repository of Liu et al . ( 2019 ) , we reproduce the result based on the488 provided configuration . To show that PNG is able to move the model along the Pareto front , we show the489 evolution of the criterion function and the norm of the MGD gradient during the training in Figure 3 . As we490 can see , PNG effectively decreases the value of criterion function while the norm of MGD gradient remains491 the same . This demonstrates that PNG is able to minimize the criterion by searching the model in the Pareto492 set . Table 3 compares the performances on the three tasks using standard training and PNG , showing that493 PNG is able to improve the model ’ s performance on segmentation and surface normal prediction tasks while494 satisfying a bit of the performance in depth estimation based on the criterion.495 C.2 FINDING DIVERSE PARETO MODELS496 C.2.1 EXPERIMENT DETAILS497 We train the model for 100 epochs using Adam op-498 timizer with batch size 256 and 0.001 learning rate.499 To encourage diversity of the models , following the500 setting in Mahapatra & Rajan ( 2020 ) , we use equally501 distributed preference vectors for linear scalarization502 and EPO . Note that the stochasticity of using mini-503 batches is able to improve the performance of Pareto504 approximation for free by also using the intermedi-505 ate checkpoints to approximate P . To fully exploit506 this advantage , for all the methods , we collect check-507 points every epoch to approximate P , starting from508 epoch 60.509 C.2.2 EVALUATION METRIC DETAILS510 We introduce the definition of the used metric for511 evaluation . Given a set P̂ = { θ1 , . . . , θN } that we512 use to approximate P , its IGD+ score is defined as:513 IGD+ ( P̂ ) = ∫ P∗ q ( θ , P̂ ) dµ ( θ ) , q ( θ , P̂ ) = min θ̂∈P̂ ∥∥∥∥ ( ` ( θ̂ ) − ` ( θ ) ) + ∥∥∥∥ , where µ is some base measure that measures the importance of θ ∈ P and ( t ) + : = max ( t , 0 ) , applied on514 each element of a vector . Intuitively , for each θ , we find a nearest θ̂ ∈ P̂ that approximates θ best . Here515 the ( · ) + is applied as we only care the tasks that θ̂ is worse than θ . In practice , a common choice of µ can516 be a uniform counting measure with uniformly sampled ( or selected ) models from P . In our experiments,517 since we can not sample models from P , we approximate P by combining P̂ from all the methods , i.e.,518 P ≈ ∪m∈ { Linear , MGD , EPO , PNG } P̂m , where P̂m is the approximation set produced by algorithm m.519 This approximation might not be accurate but is sufficient to compare the different methods,520 The Hypervolume score of P̂ , w.r.t . a reference point ` r ∈ Rm+ , is defined as521 HV ( P̂ ) = µ ( { ` = [ ` 1 , ... , ` m ] ∈ Rm | ∃θ ∈ P̂ , s.t . ` t ( θ ) ≤ ` t ≤ ` rt ∀t ∈ [ m ] } ) , where µ is again some measure . We use ` r = [ 0.6 , 0.6 ] for calculating the Hypervolume based on loss and522 set µ to be the common Lebesgue measure . Here we choose 0.6 as we observe that the losses of the two tasks523 are higher than 0.6 and 0.6 is roughly the worst case . When calculating Hypervolume based on accuracy , we524 simply flip the sign.525 C.2.3 ABLATION STUDY526 We conduct ablation study to understand the effect of α and γ using the Pareto approximation task on527 Multi-Mnist . We compare PNG with α = 0.25 , 0.5 , 0.75 and γ = 0.01 , 0.1 , 0.25 . Figure 4 summarizes the528 result . Overall , we observe that PNG is not sensitive to the choice of hyper-parameter.529 C.2.4 COMPARING WITH THE SECOND ORDER APPROACH530 We give a discussion on comparing our approach with the second order approaches proposed by Ma et al.531 ( 2020 ) . In terms of algorithm , Ma et al . ( 2020 ) is a local expansion approach . To apply Ma et al . ( 2020 ) ,532 in the first stage , we need to start with several well distributed models ( i.e. , the ones obtained by linear533 scalarization with different preference weights ) and Ma et al . ( 2020 ) is only applied in the second stage to534 find the neighborhood of each model . The performance gain comes from the local neighbor search of each535 model ( i.e . the second stage ) .536 In comparison , PNG with energy distance is a global search approach . It improves the well-distributedness537 of models in the first stage ( i.e . it ’ s a better approach than simply using linear scalarization with different538 weights ) . And thus the performance gain comes from the first stage . Notice that we can also apply Ma et al.539 ( 2020 ) to PNG with energy distance to add extra local search to further improve the approximation.540 In terms of run time comparison . We compare the wall clock run time of each step of updating the 5 models541 using PNG and the second order approach in Ma et al . ( 2020 ) . We calculate the run time based on the542 multi-MNIST dataset using the average of 100 steps . PNG uses 0.3s for each step while Ma et al . 2020 uses543 16.8s . PNG is 56x faster than the second order approach . And we further argue that , based on time complexity544 theory , the gap will be even larger when the size of the network increases.545 C.3 UNDERSTANDING PNG DYNAMICS546 We draw more analysis to understand the training dynamics of PNG.547 Different Staring Points We give analysis on PNG with different initializations showing that PNG is548 more robust to the initialization than other approaches such as Lin et al . ( 2019 ) . We consider the Pareto set549 approximation tasks and reuse synthetic example introduced in Section 5.1 . We consider learning 5 models to550 approximate the Pareto front staring from two different bad starting points . Specifically , in the upper row of551 Figure 4 , we consider initializing the models using linear scalarization . Due to the concavity of the Pareto552 front , linear scalarization can only learns models at the two extreme end of the Pareto front . The second row553 uses MGD for initialization and the models is scattered at an small region of the Pareto front . Different from554 the algorithm proposed by Lin et al . ( 2019 ) which relies on a good initialization , using the proposed energy555 distance function , PNG pushes the models to be equally distributed on the Pareto Front without the need of556 any prior information of the Pareto front even with extremely bad starting point.557 Trajectory Visualization with Different Hyper-parameters We also give more visualization on the PNG558 trajectory when using different hyper-parameters . We reuse synthetic example introduced in Section 5.1559 for studying the hyper-parameters α and γ . We fix α = 0.25 and vary γ = 0.1 , 0.05 , 0.01 , 0.1 ; and fix560 γ = 0.01 and vary α = 0.1 , 0.25 , 0.5 , 0.75 . Figure 5 plots the trajectories . As we can see , when γ is properly561 chosen , with different α , PNG finds the correct models with different trajectories . Different α determines the562 algorithm ’ s behavior of balancing the descent of task losses or criterion objectives . On the other hand , with563 too large γ , the algorithm fails to find a model that is close to P∗ , which is expected.564 C.4 IMPROVING MULTITASK BASED DOMAIN GENERALIZATION565 We argue that many other deep learning problems also have the structure of multitask learning when multiple566 losses presents and thus optimization techniques in multitask learning can also be applied to those domains.567 In this paper we consider the JiGen ( Carlucci et al. , 2019b ) . JiGen learns a model that can be generalized to568 unseen domain by minimizing a standard cross-entropy loss ` class for classification and an unsupervised loss569 ` jig based on Jigsaw Puzzles:570 ` ( θ ) = ( 1− ω ) ` class ( θ ) + ω ` jig ( θ ) . The ratio between two losses , i.e . ω , is important to the final performance of the model and requires a571 careful grid search . Notice that JiGen is essentially searching for a model on the Pareto front using the linear572 scalarization . Instead of using a fixed linear scalarization to learn a model , one natural questions is that573 whether it is possible to design a mechanism that dynamically adjusts the ratio of the losses so that we can574 achieve to learn a better model.575 We give a case study here . Motivated by the adversarial feature learning ( Ganin et al. , 2016 ) , we propose576 to improve JiGen such that the latent feature representations of the two tasks are well aligned . Specifically,577 suppose that Φclass ( θ ) = { φclass ( xi , θ ) } ni=1 and Φjig ( θ ) = { φjig ( xi , θ ) } ni=1 is the distribution of latent feature578 representation of the two tasks , where xi is the i-th training data . We consider FPD as some probability metric579 that measures the distance between two distributions , we consider the following problem:580 min θ∈P∗ FPD [ Φclass ( θ ) , Φjig ( θ ) ] . With PD as the criterion function , our algorithm automatically reweights the ratio of the two tasks such that581 their latent space is well aligned.582 Setup We fix all the experiment setting the same as Carlucci et al . ( 2019b ) . We use the Alexnet and Resnet-18583 with multihead pretrained on ImageNet as the multitask network . We evaluate the methods on PACS ( Li et al.,584 2017 ) , which covers 7 object categories and 4 domains ( Photo , Art Paintings , Cartoon and Sketches ) . Same to585 Carlucci et al . ( 2019b ) , we trained our model considering three domains as source datasets and the remaining586 one as target . We implement FPD that measures the discrepancy of the feature space of the two tasks using587 the idea of Domain Adversarial Neural Networks ( Ganin & Lempitsky , 2015 ) by adding an extra prediction588 head on the shared feature space to predict the whether the input is for the classification task or Jigsaw task.589 Specifically , we add an extra linear layer on the shared latent feature representations that is trained to predict590 the task that the latent space belongs to , i.e.,591 FPD ( Φclass ( θ ) , Φjig ( θ ) ) = min w , b 1 n n∑ i=1 log ( σ ( w > φclass ( xi , θ ) ) ) + log ( 1− σ ( w > φclass ( xi , θ ) ) ) . Notice that the optimal weight and bias for the linear layer depends on the model parameter θ , during the592 training , both w , b and θ are jointly updated using stochastic gradient descent . We follow the default training593 protocol provided by the source code of Carlucci et al . ( 2019b ) .594 Baselines Our main baselines are JiGen ( Carlucci et al. , 2019b ) ; JiGen + adv , which adds an extra domain595 adversarial loss on JiGen ; and our PNG with domain adversarial loss as criterion function . In order to run596 statistical test for comparing the methods , we run all the main baselines using 3 random trials . We use the597 released source code by Carlucci et al . ( 2019b ) to obtained the performance of JiGen . For JiGen+adv , we use598 an extra run to tune the weight for the domain adversarial loss . Besides the main baselines , we also includes599 TF ( Li et al. , 2017 ) , CIDDG ( Li et al. , 2018b ) , MLDG ( Li et al. , 2018a ) , D-SAM ( D ’ Innocente & Caputo,600 2018 ) and DeepAll ( Carlucci et al. , 2019b ) as baselines with the author reported performance for reference.601 Result The result is summarized in Table 5 with bolded value indicating the statistical significant best methods602 with p-value based on matched-pair t-test less than 0.1 . Combining Jigen and PNG to dynamically reweight603 the task weights is able to implicitly regularizes the latent space without adding an actual regularizer which604 might hurt the performance on the tasks and thus improves the overall result.605

Summary: This paper proposes an algorithm to optimize in a Pareto set in order to reach a solution or solutions that can minimize an extra criterion. The proposed algorithm is highly relevant in the context of multitask learning. The proposed algorithm takes both 1) optimizing the extra criterion w/o considering the original objectives and 2) multiple gradient descent (MGD) as its special cases. It has the flexibility to switch between two cases depending on the magnitude of the gradients w.r.t. the original tasks. The authors provide strong theoretical analysis for the proposed algorithm along with some empirical results to show its effectiveness.