Learning the Pareto Front with Hypernetworks

We describe an approach to PFL implemented using HyperNetworks , which we term Pareto HyperNetworks ( PHNs ) . PHN learns the entire Pareto front simultaneously using a single hypernetwork , which receives as input a desired preference vector and returns a Pareto-optimal model whose loss vector is in the desired ray . The unified model is runtime efficient compared to training multiple models and generalizes to new operating points not used during training . We evaluate our method on a wide set of problems , from multi-task regression and classification to fairness . PHNs learn the entire Pareto front at roughly the same time as learning a single point on the front and at the same time reach a better solution set . PFL opens the door to new applications where models are selected based on preferences that are only available at run time . 1 INTRODUCTION . Multi-objective optimization ( MOO ) aims to optimize several possibly conflicting objectives . MOO is abundant in machine learning problems , from multi-task learning ( MTL ) , where the goal is to learn several tasks simultaneously , to constrained problems . In such problems , one aims to learn a single task while finding solutions that satisfy properties like fairness or privacy . It is common to optimize the main task while adding loss terms to encourage the learned model to obtain these properties . MOO problems have a set of optimal solutions , the Pareto front , each reflecting a different trade-off between objectives . Points on the Pareto front can be viewed as an intersection of the front with a specific direction in loss space ( a ray , Figure 1 ) . We refer to this direction as a preference vector , as it represents a single trade-off between objectives . When a direction is known in advance , it is possible to obtain the corresponding solution on the front ( Mahapatra & Rajan , 2020 ) . However , in many cases , we are interested in more than one predefined direction , either because the trade-off is not known before training , or because there are many possible trade-offs of interest . For ∗Equal contributor †Equal contributor example , network routing optimization aims to maximize bandwidth , minimize latency , and obey fairness . However , the cost and trade-off vary from one application running in the network to another , or even continuously change in time . The challenge remains to design a model that can be applied at inference time to any given preference direction , even ones not seen during training . We call this problem Pareto front learning ( PFL ) . Although several recent studies ( Yang et al. , 2019 ; Parisi et al. , 2016 ) suggested to learn the trade-off curve in MOO problems , there is no existing scalable and general-purpose MOO approach that provides Pareto-optimal solutions for numerous preferences in objective space . Classical approaches , like genetic algorithms , do not scale to modern high-dimensional problems . It is possible in principle to run a single-direction optimization multiple times , each for a different preference , but this approach faces two major drawbacks : ( i ) Scalability – the number of models to be trained to cover the objective space grows exponentially with the number of objectives ; and ( ii ) Flexibility – the decision maker can not switch freely between preferences unless all models are trained and stored in advance . Here we put forward a new view of PFL as a problem of learning a conditional model , where the conditioning is over the preference direction . During training , a single unified model is trained to produce Pareto optimal solutions while satisfying the given preferences . During inference , the model covers the Pareto front by varying the input preference vector . We further describe an architecture that implements this idea using HyperNetworks . Specifically , we train a hypernetwork , termed Pareto Hypernetwork ( PHN ) , that given a preference vector as an input , produces a deep network model tuned for that objective preference . Training is applied to preferences sampled from the m−dimensional simplex where m represents the number of objectives . We evaluate PHN on a wide set of problems , from multi-class classification , through fairness and image segmentation to multi-task regression . We find that PHN can achieve superior overall solutions while being 10 ∼ 50 times faster ( see Figure 5 ) . PHN addresses both scalability and flexibility . Training a unified model allows using any objective preference at inference time . Finally , as PHN generates a continuous parametrization of the entire Pareto front , it could open new possibilities to analyze Pareto optimal solutions in large-scale neural networks . Our paper has the following contributions : ( 1 ) We define the Pareto-front learning problem – learn a model that at inference time can operate on any given preference vector , providing a Pareto-optimal solution for that specified objective trade-off . ( 2 ) We describe Pareto Hypernetworks ( PHN ) , a unified architecture based on hypernetworks that addresses PFL and shows it can be effectively trained . ( 3 ) Empirical evaluations on various tasks and datasets demonstrate the ability of PHNs to generate better objective space coverage compared to multiple baseline models , with significant improvement in training time . 2 MULTI-OBJECTIVE OPTIMIZATION . We start by formally defining the MOO problem . An MOO is defined by m losses ` i : Rd → R+ , i = 1 , . . . , m , or in vector form ` : Rd → Rm+ . We define a partial ordering on the loss space Rm+ by setting ` ( θ1 ) ` ( θ2 ) if for all i ∈ [ m ] , ` i ( θ1 ) ≤ ` i ( θ2 ) . We define ` ( θ1 ) ≺ ` ( θ2 ) if ` ( θ1 ) ` ( θ2 ) and for some i ∈ [ m ] , ` i ( θ1 ) < ` i ( θ2 ) . We say that a point θ1 ∈ Rd dominates θ2 ∈ Rd if ` ( θ1 ) ≺ ` ( θ2 ) . If a point θ1 dominates θ2 , then θ1 is clearly preferable , because it improves some objectives and is not worse on any other objective . Otherwise , solutions present a certain trade-off and selecting one specific solution requires additional information about the user preferences . A point that is not dominated by any other point is called Pareto optimal . The set of all Pareto optimal points is called the Pareto front . Since many modern machine learning models , e.g. , deep neural networks , use non-convex optimization , one can not expect global optimality . We call a point local Pareto optimal if it is Pareto optimal in some open neighborhood of the point . The most straightforward approach to MOO is linear scalarization ( LS ) , where one defines a new single loss ` r ( θ ) = ∑ i ri ` i ( θ ) given a vector r ∈ Rm+ of weights . One can then apply standard , single-objective optimization algorithms . LS has two major limitations . First , it can only reach the convex part of the Pareto front ( Boyd et al. , 2004 , Chapter 4.7 ) , as shown empirically in Figure 1 . Second , if one wishes to target a specific ray in loss space , specified by a preference vector , it is not clear which linear weights lead to that desired Pareto optimal point . In the context of the current work , we highlight two properties that an MOO optimization procedure should possess : scale to many objectives and control which solution on the Pareto front is obtained . Lin et al . ( 2019 ) described Pareto multi-task learning ( PMTL ) , an algorithm that splits the loss space into separate cones based on the selected reference rays , and returns a solution per cone using a constrained version of Fliege & Svaiter ( 2000 ) . This approach allows the user to target several points on the Pareto front ; However , it scales poorly with the number of cones and does not converge to the exact desired ray on the Pareto front . Convergence to the desired ray in loss space can be achieved using Exact Pareto Optimal ( EPO ) ( Mahapatra & Rajan , 2020 ) . To find the intersection of the Pareto front with a given preference ray r , EPO balances two goals : Finding a descent direction towards the Pareto front and approaching the desired ray . EPO searches for a point in the convex hull of the gradients , known by Désidéri ( 2012 ) to include descent directions , that has a maximal angle with a vector dbal which pulls the point to the desired ray . EPO combines gradient descent and controlled ascent enabling it to reach an exact Pareto optimal solution if one exists , or the closest Pareto optimal solution . 3 RELATWED WORK . Multitask learning . In multitask learning ( MTL ) we simultaneously solve several learning problems while sharing information among tasks ( Zhang & Yang , 2017 ; Ruder , 2017 ) . In some cases , MTL-based models outperform their single task counterparts in terms of per-task performance and computational efficiency ( Standley et al. , 2019 ) . MTL approaches map the loss vector into a single loss term using a fixed or dynamic weighting scheme . The most frequently used approach is Linear Scalarization ( LS ) , in which each loss term ’ s weight is chosen apriori . A proper set of weights is commonly selected using grid search . Unfortunately , such an approach scales poorly with the number of tasks . Recently , MTL methods propose dynamically balance the loss terms using gradient magnitude ( Chen et al. , 2018 ) , the rate of change in losses ( Liu et al. , 2019 ) , task uncertainty ( Kendall et al. , 2018 ) , or learning non-linear loss combinations by implicit differentiation ( Navon et al. , 2020 ) . However , those methods seek a balanced solution and are not suitable for modeling task trade-offs . Multi-objective optimization . The goal of Multi-Objective Optimization ( MOO ) is to find Pareto optimal solutions corresponding to different trade-offs between objectives ( Ehrgott , 2005 ) . MOO has a wide variety of applications in machine learning , spanning Reinforcement Learning ( Van Moffaert & Nowé , 2014 ; Pirotta & Restelli , 2016 ; Parisi et al. , 2014 ; Pirotta et al. , 2015 ; Parisi et al. , 2016 ; Yang et al. , 2019 ) , neural architecture search ( Lu et al. , 2019 ; Hsu et al. , 2018 ) , fairness ( Martínez et al. , 2020 ; Liu & Vicente , 2020 ) , and Bayesian optimization ( Shah & Ghahramani , 2016 ; Hernández-Lobato et al. , 2016 ) . Genetic algorithms ( GA ) are a popular approach for small-scale multi-objective optimization problems . GAs are designed to maintain a set of solutions during optimization . Therefore , they can be extended in natural ways to maintain solutions for different rays . In this area , leading approaches include NSGA-III ( Deb & Jain , 2013 ) and MOEA/D ( Zhang & Li , 2007 ) . However , these gradient-free methods scale poorly with the number of parameters and are not suitable for training large-scale neural networks . Sener & Koltun ( 2018 ) proposed a gradient-based MOO algorithm for MTL , based on MDGA ( Désidéri , 2012 ) , suitable for training large-scale neural networks . Other recent works include ( Lin et al. , 2019 ; Mahapatra & Rajan , 2020 ) detailed in Section 2 . Another recent approach that aims at a more complete view of the Pareto front is Ma et al . ( 2020 ) . They extend a given Pareto optimal solution in its local neighborhood . In a concurrent work , Lin et al . ( 2020 ) extends Lin et al . ( 2019 ) for approximating the entire Pareto front . They train a single hypernetwork by constantly changing the reference directions in the PMTL objective . The proposed method is conceptually similar to our approach , but since it builds on PMTL , it may not produce an exact mapping between the preference and the corresponding solution . Similarly , Dosovitskiy & Djolonga ( 2019 ) proposed learning a single model conditioning on the objective weight vector . The method uses feature-wise linear modulation Perez et al . ( 2017 ) , and dynamically weighted LS loss criterion . Hypernetworks . Ha et al . ( 2017 ) introduced the idea of hypernetworks ( HNs ) inspired by the genotype-phenotype relation in cellular biology . HN presents an approach of using one network ( hypernetwork ) to generate weights for a second network ( target network ) . In recent years , HN are widely used in various domains such as computer vision ( Klocek et al. , 2019 ; Ha et al. , 2017 ) , language modeling ( Suarez , 2017 ) , sequence decoding ( Nachmani & Wolf , 2019 ) , continual learning ( Oswald et al. , 2020 ) , federated learning ( Shamsian et al. , 2021 ) and hyperparameter optimization ( Mackay et al. , 2019 ; Lorraine & Duvenaud , 2018 ) . HN dynamically generates models conditioned on a given input , obtaining a set of customized models using a single learnable network .

Summary: This paper tracks the problem of learning the entire Pareto front to allow the user to select a desired Pareto optimal solution by one inference procedure without retraining the model. The high-level idea is to learn the entire Pareto front simultaneously using a single hyper network, which receives as input the desired preference vector and returns a Pareto-optimal solution whose loss vector is in the desired direction. The paper gives an early trial to build a toolbox to allow users to get a desired solution by a single inference procedure.