Learning the Pareto Front with Hypernetworks
Authors: Aviv Navon
Aviv Shamsian
Ethan Fetaya
Gal Chechik
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-objective optimization
multi-task learning

Abstract: Multi-objective optimization (MOO) problems are prevalent in machine learning. These problems have a set of optimal solutions, called the Pareto front, where each point on the front represents a different trade-off between possibly conflicting objectives. Recent MOO methods can target a specific desired ray in loss space however, most approaches still face two grave limitations: (i) A separate model has to be trained for each point on the front; and (ii) The exact trade-off must be known before the optimization process. Here, we tackle the problem of learning the entire Pareto front, with the capability of selecting a desired operating point on the front after training. We call this new setup Pareto-Front Learning (PFL).

We describe an approach to PFL implemented using HyperNetworks, which we term Pareto HyperNetworks (PHNs). PHN learns the entire Pareto front simultaneously using a single hypernetwork, which receives as input a desired preference vector and returns a Pareto-optimal model whose loss vector is in the desired ray. The unified model is runtime efficient compared to training multiple models and generalizes to new operating points not used during training. We evaluate our method on a wide set of problems, from multi-task regression and classification to fairness. PHNs learn the entire Pareto front at roughly the same time as learning a single point on the front and at the same time reach a better solution set. PFL opens the door to new applications where models are selected based on preferences that are only available at run time.

Published as a conference paper at ICLR 2021
LEARNING THE PARETO FRONT WITH HYPERNETWORKS
Aviv Navon"
Aviv Shamsian"
Bar-Ilan University, Israel wwt.v.avvo@@bu.cco
Bar-Ilat University, Israel aviv. chamsian@biu.ac.
Ethan Fetaya!
Gal Chechik
Bar-Ilan University, Israel ethan. wwt.yaebiaaaac....mm
Bar- Bar-Ilan University, Israel
NVIDIA, Israel gal. www.hikkbiu..uu...... awt.iiiuucc.....
ABSTRACT
Multi-objective optimization (MOO) problems are prevalent in machine learning. These problems have of optimal solutions, called the Pareto front, where each point on the front represents a different trade-off between possibly conflicting objectives. Recent MOO methods can target specific desired ray in loss space however, most approaches still face two grave limitations: (i) A separate model has be trained for each point on the front; and (ii) The exact trade-off must be known before the optimization process. Here, we tackle the problem of learning the entire Pareto front, with the capability of selecting a desired operating point on the front after training. We call this new setup Pareto-From Learning (PFL).
We describe an approach to PFL implemented using HyperNetworks, which we term Pareto HyperNerworks (PHNs). PHN learns the entire Pareto front simultane- ously using a single hypernetwork, which receives as input a desired preference vector and returns a Pareto-optimal model whose loss vector The unified model is runtime efficient compared to training multiple models and generalizes to new operating points not used during training. We evaluate our method on a wide set of problems, from multi-task regression and classification fairness. PHNs learn the entire Pareto front at roughly the same time as learning single point the front and at the same time reach better solution set. PFL opens the door to new applications where models are selected based on preferences that only available
the desired ray.
time.
INTRODUCTION
Multi-objective optimization (MOO) aims optimize several possibly conflicting objectives. MOO is abundant in machine learning problems, from multi-task learning (MTL), where the goal is to learn several tasks simultaneously, to constrained problems. In such problems, one aims to learn a single task while finding solutions that satisfy properties like faimess or privacy. It is common to optimize the main task while adding loss terms to encourage the learned model to obtain these properties.
MOO problems have a set of optimal solutions, the Pareto froní, each reflecting a different trade-off between objectives. Points on Pareto front can viewed as an intersection of the front with a specific direction in loss space (a ray, Figure D We refer to this direction preference vector, represents a single trade-off between objectives. When direction is known advance, is possible to obtain the corresponding solution the front (Mahapatra & Rajan 2020).
However, in many cases, we are interested in more than one predefined direction, either because the trade-off is not known before training, because there are many possible trade-offs of interest. For
"Equal contributor I Equal contributor

---

Published as a conference paper at ICLR 2021
PHN
EPO
PMTL
NSGA-III
LS
12
Peesible solutions
1.0
1.0
1.0
1.0
1.0
0.6
0.6
0.6
0.6
0.2
0.2
Poreta Front
0.2
0.6
1.0
1
0.2
0.5
1.0
0.2
0.6
1.0
2
0.2
0.6
1.0
0.2
0.6
1.0
demonstrating the relation between Pareto front, preference rays, and solutions. Pareto front (black solid line) for 2D loss space and several rays (colored dashed lines) which represent various possible preferences. Left: A single PHN-EPO model learns the entire Pareto front, mapping any given preference ray to its corresponding solution on the front. The data and task are detailed in Section 5.
Figure 1: Illustrative example using the popular task of Fonseca 7995;:
example, network routing optimization aims to maximize bandwidth, minimize latency, and obey fairness. However, the cost and trade-off vary from one application running in the network to another, or even continuously change in time. The challenge remains to design a model that can be applied at inference time to any given preference direction, even ones not seen during training. We call this problem Pareto front learning (PFL).
Although several recent studies (Yang et al. 2019; Parisi et al. 2016) suggested to learn the trade-off curve in MOO problems, there is no existing scalable and general-purpose MOO approach that provides Pareto-optimal solutions for numerous preferences in objective space. Classical approaches, like genetic algorithms, do not scale to modern high-dimensional problems. It is possible in principle to run sinle-directton optimization multiple times, each for a different preference, but this approach faces two major draw backs: (i) Scalability the number of models to be trained to cover the objective space grows exponentially with the number of objectives; and (ii) Flexibility the decision maker cannot switch freely between preferences unless all models are trained and stored in advance.
Here put forward new view of PFL as a problem of learning conditional model, where the conditioning is over the preference direction. During training, a single unified model is trained to produce Pareto optimal solutions while satisfying the given preferences. During inference, the model covers the Pareto front by varying the input preference vector. We further describe an architecture that implements this idea using HyperNetworks. Specifically, we train hypernetwork, termed Pareto Hypernetwork (PHN), that given preference vector as an input, produces a deep network model tuned for that objective preference. Training is applied to preferences sampled from the m--dimensional simplex where represents the number of objectives.
We evaluate PHN on wide set of problems, from multi-class classification, through fairness and image segmentation to multi-task regression. We find that PHN can achieve superior overall solutions while being 10 50 times faster (see Figure 5). PHN addresses both scalability and flexibility. Training unified model allows using any objective preference at inference time. Finally, PHN generates a continuous parametrization the entire Pareto front, could open new possibilities to analyze Pareto optimal solutions in large-scale neural networks.
Our paper has the following contributions: (1) We define the Pareto-fron learning problem - learn a model that at inference time operate any given preference vector, providing Pareto-optimal solution for that specified objective trade-off. (2) We describe Pareto Hypernetworks (PHN), a unified architecture based on hypemeeworkss that addresses PFL and shows it can be effectively trained. (3) Empirical evaluations on various tasks and datasets demonstrate the ability of PHNs to generate better objective space coverage compared to multiple baseline models, with significant improvement in training time.
2 MULTI-OBJECTIVE OPTIMIZATION
We start by formally defining the MOO problem. An MOO is defined by m losses & Rd + R.+ i 1, 1,.... m, in vector form L: Rd R We define a partial ordering on the loss space
2

---

Published as a conference paper at ICLR 2021
TT by setting 2(01) = e(02) if for all E [ml, (((1)) I (((2)). We define 2(01) ((01) = {(02) and for some i E (00)) < (02)). We say that point 01 R dominates 8g E I (())) < L(02).
e(02) if
If point 0, dominates 02. then 81 is clearly preferable, because it improves some objectives and is not worse on any other objective. Otherwise, solutions present certain trade-off and selecting one specific solution requires additional information about the user preferences. A point that is not dominated by any other point is called Pareto optimal. The set of all Pareto optimal points is called the Pareto front. Since many modern machine learning models, e.g., deep neural networks, use non-convex optimization, one cannot expect global optimality. We call a point local Pareto optimal if it is Pareto optimal in some open neighborhood of the point.
The most straightforward approach to MOO is linear scalarization (LS), where one defines new single loss (0) = Σ, (0) given a vector r E RT of weights. One can then apply standard, single-objective optimization algorithms. LS has two major limitations. First, it can only reach the convex part of the Pareto front (Boyd et al., 2004 Chapter 4.7), as shown empirically in Figure [] Second, if one wishes to target specific ray in loss space, specified by preference vector, it is not clear which linear weights lead to that desired Pareto optimal point.
In the context of the current work, we highlight two properties that an MOO optimization procedure should possess: scale to many objectives and control which solution on the Pareto front is obtained. Lin et al. (2019) described Pareto multi-tast learning (PMTL), algorithm that splits the loss space into separate cones based on the selected reference rays, and returns a solution per cone using a constrained version Fliege & Svaiter (2000). This approach allows the user to target several points on the Pareto front; However, it scales poorly with the number of cones and does not converge to the exact desired ray on the Pareto front.
Convergence to the desired ray in loss space can be achieved using Exact Pareto Optimal (EPO) (Mhaapatra & Rajan 2020). To find the intersection of the Pareto front with a given preference ray r, EPO balances two goals: Finding descent direction towards the Pareto front and approaching the desired ray. EPO searches for a point in the convex hull of the gradients, known by DDsidéri (2012) to include descent directions, that has maximal angle with vector daat which pulls the point to the desired ray. EPO combines gradient descent and controlled ascent enabling it to reach an exact Pareto optimal solution if one exists, or the closest Pareto optimal solution.
RELATWED WORK
Multitask learning. In multitask learning (MTL) we simultaneously solve several learning prob- lems while sharing information among tasks (Zhang & Yang. 2017| Ruder 2017). In some cases, MTL-based models outperform their single task counterparts terms of per-task performance and computational efficiency (Standley et al. 2019). MTL approaches map the loss vector into a single loss term using fixed dynamic weighting scheme. The most frequently used approach is Linear Scalarization (IS), in which each loss term's weight is chosen apriori. A proper set of weights commonly selected using grid search. Unfortunately, such an approach scales poorly with the number of tasks, Recently, MTL methods propose dynamically balance the loss terms using gradient magnitude (Chen et al.) 2018), the rate of change in losses (Liu et al.) 2019), task uncertainty (Kendall et al. 2018)), or learning non-linear loss combinations by implicit differentiation (Navon et al. (2220). However, those methods seek a balanced solution and are not suitable for modeling task trade-offs.
Multi-objective optimization. The goal of Multi-Objective Optimization (MOO) is find Pareto optimal solutions corresponding to different trade-offs between objectives (Ehrgoti 2005). MOO has a wide variety of applications in machine learning, spanning Reinforcement Learning Van Mof faert & Nowē, 2014; [Pirotta& & Restelli, 2016: Parisi et al.) 2014, Pirotta et al., 2015 Parisi et al. 2016 Yang et al. 2019),, neural architecture search (Lu et al. 2019; Hsu et al. 2018). fairness
(Martinez et al. 2020 Liu & Vicente 2020). and Bayesian optimization Shah & Ghahramani 2016
BeenánndezLooaat et al. 2016). Genetic algorithms (GA) are a popular approach for amall-scal multi-objective optimization problems. GAs designed to maintain set of solutions during optimization. Therefore, they can be extended in natural ways to maintain solutions for different rays. In this area, leading approaches include NSGA-III (Deb & Jain 2013) and MOEA/D (Zhang & Li
3

---

Published as a conference paper at ICLR 2021
2007). However, these gradient-free free methods scale poorly with the number of parameters and not suitable for training large-scale neural networks.
Sener & Koltun (2018) proposed gradient-based MOO algorithm for MTL, based MDGA
(Désidéri 2012), suitable for training large- neural networks. Other recent works include (Lin
et al., 2019] Mahapatra & Rajan 2020) detailed in Section 2 Another recent approach that aims
a more complete view of the Pareto front Ma et al.) (2020). They extend given Pareto optimal
solution in its local neighborhood. In concurrent work. Lin al. (2020) extends [in et al. (2019) for approximating the entire Pareto front. They train single hypernetwork by constantly changing the reference directions the PMTL objective. The proposed method is conceptually similar to our approach, but since it builds on PMTL, it may not produce an exact mapping between the preference and the corresponding solution. Similarly, Dosovitskiy Djolonga (2019) proposed learning single model conditioning on the objective weight vector. The method uses feature- wise linear modulation Perez al. 2017), and dynamically weighted L. loss criterion.
Hypernetworks. Ha et al. (017)) introduced the idea of hypernetworks (HNs) inspired by the genotype- phenotype relation cellular biology. HN presents an approach of using one network (hypernetwork) to generate weights for second network (target network). In recent years, HN are widely used in various domains such as computer vision (Klocek al. 2019 Ha et al. (017)) language modeling (Suarez) 2017), sequence decoding (Nachmani & Wolf 2019), continual learning (Oswald cct al. 2020), federated learning (Shamsian et al. (022)) and hyperparameter optimization (Mackay al. 2 Lorraine & [ HN dynamically generates models conditioned on given input, obtaining a set of customized models using a single learnable network.
PARETO HYPERNETWORKS
Algorithm 1 PHN
while not converged do
Dir(a) O(ø, r) h(r;;
Sample mini-batch (I1, v1), (EB,YB)
:(:;0)
Or
f(;)))
ISS then
+ B Eig Vold 0()
EPO then
8 EEO r l,
e lm
+ B Eng 3. V
B1, 0(,
+ - 79
Figure 2: A PHN h(:: receives input preference ray and outputs the correspond- ing model weights 0,.
return &
We start by describing the basic of hypernetworks architecture (Figure 2b. Hypernetworks deep models that generate the weights of another deep model as their output, named the target network. The weights produced by the hypernetwork depend on its input, and one view the training of a hypernetwork as training a family of target networks simultancously. conditioned on the input. Let h(: ; Φ) denote the hypernetwork with parameters & E IR". and let denote the target network whose parameters we denote by 0. In our implementation, we parametrize using a feed-forward network with multiple heads. Each head outputs different weight tensor of the target network. More precisely, the input is first mapped to higher dimensional space using an MLP, to construct shared features. These features are passed through fully connected layers to create a weight matrix per layer in the target network.
Consider an m dimensional MOO problem with an objective vector E. Let r (ri..... F'm E Sm denote a preference vector representing desired rafe-off over the objectives. Here Sm fr I Σ, r, 1, 2 0} is the m-dimensional simplex. Our PHN model takes as input the preference vector r to produce the weights 6, = 0( r) = h(r: Φ) of the target network. The target network is then applied to the training data in usual manner. Throughout the training, randomly sample r at each iteration from a Dirichlet distribution with parameter a E R" At inference time, a given preference direction r, is provided as input the HN ht which outputs the corresponding Pareto
4

---

Published as a conference paper at ICLR 2021
optimal solution, 8r, namely, set of model weights for specific given We train @ to be Pareto optimal and corresponds to the preference vector. For that purpose, we explore two alternatives.
The first, named PHN-LS, uses linear scalarization with the preference vector loss weights, i.e the loss for imput r is E, rile While linear scalarization has some theoretical limitations, it is a straightforward and fast approach that tends to work well in practice. One can think of the (x. y, 8( r)]] where we sample mini-batch and weighting r. We note, direct result of the implicit function theorem, the following:
PHN-LS:
PHN-LS approach as optimizing the loss (r,s(() = a, 12,((2
weights P. i.e. ( VJ, 0) 00 and assume the Hessian E. has full rank at 0(F). There exists a neighborhood U of f and a smooth mapping 0(r) such that Vr E U Eip T. Vl, Bg. 0(r)) 0.
Proposition 1. Let 0(F) be a local Pareto optimal point corresponding
This shows that under the full rank assumption the connection between the LS term T and the optimal model weights 0(r) is smooth and can be learned using universal approximation (Cybenko 1989). We also observe that if for all r, á( r) is a local Pareto optimal point corresponding to T, i.e., Eij r. V My, ((r)) = 0, then (ol = O and this is stationary point for fLS- However, there could exists stationary points and even local minima such that V (Ø) = 0 but in general Lig r2 VE, My, (()) # O and the output of our hypernetwork is not local Pareto optimal point. While these bad local minima exist, as in standard deep neural networks, we empirically find that PHNS avoid these solutions and achieve good results.
Our second approach treats the preference r as a ray in loss space and trains ((d, r) to reach a Pareto optimal point on the inverse ray r-1 namely, T1 -1 =.... I'm Lm- This achieved by using the EPO update direction for each output of the Pareto hypernetwork. The recent EPO algorithm can guarantee convergence to a local Pareto optimal point on a specified ray r, addressing the theoretical limitations of LS. Since the EPO descent direction is convex combination of the gradients, we use the EPO algorithm to compute this adaptive weighting and backpropagate the reweighted losses. See pseudo-code in Alg. [
PHN-EPO:
The EPO however, unlike L.S, cannot be seen as optimizing a certain objective. Therefore, we cannot consider the PHN-EPO optimization, which uses the EPO descent direction, as optimizing specific objective. We also note that since the EPO needs to solve a linear programming step at each iteration, the runtime of EPO and PHN-EPO are longer than the LS and PHN-LS counterparts (sce Figure 5).
PHN Scalability: Using a naive implementation of HNs, their size grows linearly with the target network's s size; this may restrict the scalability of PHNs to large target networks. Fortunately, our approach can be casily extended to handle large target networks. First, one can learn only part of the target network parameters using the HN (Luan et al. 2018). or learn preference-speccffic normalization layers (Huang & Belongie| 2017 Perez et al. 2017) as in Dosovitskiy & Djolongal (2019). Furthermore, one can apply "chunking" ": generate different parts of the target network, conditioned on trainable chunk descriptor input (Oswald et al. (2020). Similarly, one can reduce the dimension of the shared preference representation to achieve a similar effect (see Appendix B.5 The experiments in the main paper used the vanilla version of PHN.
5 EXPERIMENTS
We evaluate Pareto HyperNetworks (PHNs) on a set of diverse multi-objective problems. The experiments show the superiority of PHN over previous MOO methods. We make our source code publicly available at: https::// / tithub. wwm/cv/////oo/......... Avi www.m//vviavvoo///oooaaaooaaaaae
hypernetwors
Compared methods: We compare the following approaches: (1) Our proposed Pareto HyperNet-
works (PHN-LS and PHN-EPO) algorithm described in Section 4 (2) Linear scalarization (LS); (3)
Pareto MTL (PMTL) Lin et al. (0019); (4) Exact Pareto optimal (EPO) Mahapatra & Rajan (2020) ;
(5) Continuous Pareto MTL (CPMTL) Ma et al. (020)) All competing methods train one model per preference vector to generate Pareto solutions. As these models optimize each ray separately, do not treat them as a baseline but as "gold standard" in term of performance. Their runtime, on the other hand, scales linearly with the number of rays.
5

---

Published as a conference paper at ICLR 2021
Multi-Fashion + MNIST
Multi-MNIST
14-
PHN EPO (ouns)
PHN (sun))
Taskk 2 loss
11.5
CINTT
RPQ
0.66
0.7
2MTL
11.3 *
0.55
0166
0.2
00.3 C.E 0.5 0.5
0.4 L0
Task
loss
Figure 3: Mulii-MNIST: Test losses produced by the competing method three Multi-MNIST datasets. Colors correspond to different preference vectors, using the same color map as in Figure 1. PHN generates continuous Pareto front (solid/dashed line for PHN-EPO/PHN--S) while mapping any given preference to its corresponding solution (circles for PHN-EPO and x's for PHN-LS).
Genetic algorithms are perhaps the most popular approaches for MOO. Two leading approaches in that area, that also use preference information are NSGA-III (Deb et al. 2002) and MOEA/D (hhag & Li 2007). Unfortunately, while these methods can be applied to small networks (like the toy example Figure 1, which gave sub-par results even after 100,000 generations), they failed to converge to meaningful solutions in our larger-scale experiments. Therefore, we do not compare PHN with GAs on large-scale setups.
Evaluation metrics: A common metric for evaluating sets of MOO solutions is the Hypervolume
indicator (HV) (Zitzler Thiele 1999). It measures the volume in the loss space of points dominated by solution in the evaluated set (see Appendix C As this volume is unbounded, it is restricted to the volume in rectangle defined by the solutions and a selected reference point. All solutions need to be bounded by the reference point, therefore select different reference points for each experiment. In addition, we evaluate Uniformity, proposed by Mahapatra & Rajan (2020). This metric is designed to quantify how well the loss vector L(O) is aligned with the input ray r. Formally, for any point 0 in the solution space and reference r we define the non unt-uumiifotttyyoom yer (e(O)) = Dx (éll /mm. where l = TyL E. We define the uniformity as - Myy- The main metric, HV, measures the quality of our solutions, while the uniformity measures how well the output model fits the desired preference ray. We stress that the baselines are trained and evaluated for uniformity using the same rays.
Training Strategies: In all experiments, our target network shares the same architecture as the baseline models. For our hypernetwork, we use an MLP with hidden layers and linear heads. We set the hidden dimension 100 for the Multi-MNIST and NYU experiment, and 25 for the Fairness and SARCOS datasets. The Dirichlet parameter a in Alg. Tis is set to 0.2 for all experiments, as we empirically found this value to perform well across datasets and tasks. We provide a more detailed analysis on the selection of the HN's hidden dimension and hyperparameter a in Appendix B.5) Extensive experimental details provided in Appendix A
Hyperparameter tuning: For PHN, we select hyperparameters based on the HV computed on a validation set. Selecting hyperparameters for the baselines challenging because there are no clear criteria that can be computed quickly with many rays. Selecting HPs based on HV requires to train each baseline multiple times on all rays. Therefore, we select hyperparameters based on a single ray and apply the selected hyperparameters to all rays. Specifically, we collect all models trained using all hyperparameter configurations, and remove dominated solutions. Finally, we select the combination of HPs with the highest uniformity.
5.1 ILLUSTRATIVE EXAMPLE
We evaluate PHN on an illustrative MOO problem whose Pareto front is known Fonseca 1995). Here,
(0) exp { - 110 1/vd|(). (2 exp { 110 1/va 21

---

Published as a conference paper at ICLR 2021
Table 1: Evaluation of the different methods on three Multi-MNIST datasets.
Multi-aashion-MNIST
Multi-Fashion
Multi-MNIST
HV f
Unif.
HV 0 Unif. ft
Unif, ft
Run-time
(min., Tesla V100)
LS
2.70
0.849
2.14
0.835
2.85
0.846
9.0 45
CPMIL
2.76
2.16
2,88
10.2 x 5 51
PMTL
2.67
0.776
2.13
0.192
2.86
0.793
17.0 85
EPO
2.67
0.892
2.15
0.906
2.85
0.918
23.6
5 118
PHN-LS (ours)
2.75
0.894
2.19
0.900
2.90
0.901
PHN-EPO (ours)
2.78
0,952
2.19
0,921
2.78
0,920
27
with parameters I E Rd and d 100. This problem has a non-convex Pareto front in R2. We com- pared PHN with EPO, PMTI., MGDA, and two genetic algorithms (GAs) NSGA-III and MOEA/D. The results are presented in Figure [ Linear scalarization (L.S) fails to generate solutions in the concave part of the Pareto front, theoretically shown in [Boyd et al. (2004). PMTL and EPO generate solutions along the entire PF, but the solutions generated by PMTL are only optimized to be the vicinity of the preference vector r. Results for MGDA and MOEA/D algorithms are shown in Appendix B.I Our PHN-EPO generates a continuous cover of the PF in a single run. In addition, PHN-EPO returns exact Pareto optimal solutions, mapping an input preference to its corresponding point on the front. We note that similar to the EPO method, PHN-EPO can learn non-convex Pareto fronts. Additional synthetic experiments with non-convex, known Pareto fronts are presented in Appendix B.]
5.2
IMAGE CLASSIFICATION
We evaluate all methods on three MOO benchmark datasets: (1) Multi-MNIST (Sabour et al. 2017);
(2) Multi-Fashion, and (3) Multi-Fashion MNIST. In each dataset, two instances sampled uniformly at random from the MNIST (LeCun et al., 1998) or Fashion-MNIST (Xiao et al. 2017) datasets. The two images are merged into new image by placing one the top-left corner and the other at the bottom-right corner. The sampled images can be shifted up to four pixels in each direction. Each dataset consists of 120,000 training examples and 20,000 test set examples. We allocate 10% of each training set for constructing validation sets. We train 5 models for each baseline, using evenly spaced rays. The results are presented in Figure and Table The HV was calculated with reference point (2, 2). PHN outperforms all other methods in terms of HV and uniformity, with significant improvement in run time. To better understand the behavior of PHN while varying the input preference vector, visualize the per-pixel prediction contribution in Appendix B.3
5.3
FAIRNESS
Adult
Default
Bank
0:30
2.20
0.30
0:38
0.72
035
Main oprimal
PP aptind
PN optimal
5.01
0:45
0.03
0.00
Xain lone
FP lose
Main hex
FP hue
FN Jne
Main
FN lowe
Figure 4: Accuracy-Fairness trade-off:: The trade-off between the main classification loss and the two fairness losses, on the test sets of the three faimess datasets. Each line corresponds to a different solution along the Pareto front, generated using PHN-EPO.
Fairness has become a popular topic in machine learning in recent years Mehrabi et al. 2019). and numerous approaches have been proposed for modeling and incorporating fairness into MI.
7

---

Published as a conference paper at ICLR 2021
Table 2: Comparison on three Fairness datasets. Run-time is evaluated on the Adult dataset.
Adult
Default
Bank
HV ft Unif. ft
Unif. fh HV ft Unif. ft
Run-time
(min.. Tosia V100)
LS
0.628
0.109
0.522
0.009
0.704
0.215
0.9 = 9.0
PMIL
0.614
0.458
0.548
0.471
0.638
0.497
2.7 x 10 26.6
EPO
0.608
0.734
0.537
0.533
0.713
0.881
1.6
10 15.5
PIN IS (ours)
0
0.289
0.551
0.108
0.730
0.615
1.1
PHN-EPO (ours)
0.648
0.701
0.548
0.359
0.748
0.821
systems (Dwork et al. 2018; Kamishima et al. 2012: Zafar et al., 2019). Here, we adopt the approach of [Bechavod & Liget| (2017). They proposed a 3-dimensional optimization problem, with classification objective and two fairness objectives: False Positive (FP) fairness, and False Negative (FN) fairness. We compare PHN with EPO, PMTL, and LS on three commonly used fairness datasets: Adult (Dua & Graff 2017). Default (Ych & Lien 2009) and Bank (Moro et al. 2014).
Each dataset is divided into train/validation/tess sets of sizes 70%/10%/20% respectively. All baselines trained and evaluated on 10 preference rays. We present the results in Table 2 HV calculated with reference point (1, 1.). PHN achieves the best HV performance across all datasets, with reduced training time compared to baselines. EPO performs best in terms of uniformity, however, recall it is evaluated on its training rays. We visualize the accuracy-fairness trade-off achieved by PHN in Figure We also tested GAs on this task but the runtime was extremely high (two days, compared with runtime of minutes for gradient- based methods) with poor performance.
5.4 PIXEL-WISE CLASSIFICATION AND REGRESSION
Image segmentation and depth estimation are key problems in computer vision with various appli- cations (Minaee al., 2020; Bhoi 2019). We compare PHN the other methods on the NYUv2 dataset (Silberman et al. 2012). NYUv2 is challenging indoor scene dataset that consists of 1449 RGBD images and dense per-pixel labeling. We use this dataset as a multitask learning benchmark for semantic segmentation and depth estimation tasks. We split the dataset into train/validation/tss sets of sizes 70% 10%%0% respectively, and use an ENet-based architecture for the target network (Paszke) al. 2016)) Each baseline method is trained five times using five evenly spaced rays. The results are presented in Table 3 HV is calculated with reference point (3, 3). PHN-EPO achieves the best HV and uniformity, while also being faster than the baselines.
Table 3: Comparison on NYUv2.
Table 4: Comparison on SARCOS dataset.
NYUv2
SARCOS
Unif.
Ran-time (hours, Testa V100)
HV ft
Unil.
Run- time
(hours, Tesla V100)
LS
3,550
0.666
0.58 x 5 2.92
0.688
-0.008
0.17
20 3.3
PMTL
3.554
0.679
0.96 x 5 4.79
CPMTI
3,570
0.71 5 3.55
EPO
0.637
0.548
0.5
20 10
EPO
3.266
0.728
1.02 x 5.11
PHN-LS LLS (ours)
0.693
0.020
0.2
PIIN EPO (ours)
0.728
0.258
0.8
PHN-LS (ours)
3,546
0.798
0.67
PHN-EPO (ours)
3.589
0.820
1.04
5.5 MULTITASK REGRESSION
When learning with many objectives using preference-specific models, generating a good coverage of the solution space becomes too expensive in computation and storage space, as the number of rays needed grows with the number of dimensions. As a result, PHN provides major improvement in training time and solution space coverage (measured by HV). To evaluate PHN in such settings, we use the SARCOS dataset (Vijayakumar). commonly used dataset for multitask regression (Zhang & Yang 2017). SARCOS presents an inverse dynamics problem with 7 regression tasks and 21 explanatory variables. We omit PMTL from this experiment because its training time is significantly
8

---

Published as a conference paper at ICLR 2021
longer compared to other baselines, with relatively low performance. For LS and EPO, train 20 preference-speciffe models. The results are presented in Table HV calculated with reference point (I,..., 1). As before, PHN performs best in terms of HV, while reducing the overall runtime.
6 THE QUALITY-RUNTIME TRADE-OFF
While PHN learns the entire front in a single model, the competing methods need to train multiple models to cover the Pareto front. As a result, these methods have a clear trade-off between their performance and their run time. To better understand this trade-off, Figure shows the HV and run time (in log-scale) of LS and EPO when training a various number models. For Fashion-MNIST trained 25 models, and at the inference time, selected subsets of various sizes (I ray, 2 rays, etc) and computed their HV. The shaded area in Figure the variance over different selections of ray subsets. We similarly trained 30 EPO/LS models for Adult and 40 for SARCOS. PHN-EPO achieves superior or comparable hypervolume while being an order of magnitude faster. PHN-LS is even faster, but this may come the expense of lower hypervolume.
To further understand the source of PHN performance gain, we evaluate the HV on the same rays used for the competing methods (see Appendix [B.4). In many cases, PHN outperforms models trained on specific rays, when evaluated on those exact rays, We attribute the performance gain to the inductive bias effect that PHN induces on model by sharing weights across rays.
(a) Multi Fashion MNIST
(b) Adult
(c) SARCOS
2.8
0.65
2.8
2.7
0.60
2.6
2.5
EPO
0.55
2.6
2.5
2.4
PHN EPO
PHNLS
2.4
100
700
1
10 Runtime (min)
10
100
1000
Runtime (min)
Runtime (min)
Figure 5: HV vs runtime (min.) comparing PHN with preference-specifie LS and EPO. For LS and EPO, each value was computed by evaluating with a subset of models. PHN achieves higher HV significantly faster (x-axis is in log scale). Shaded area marks the variance across selected subsets.
7 CONCLUSION
We put forward novel MOO setup which we term Pareto Front Learning (PFL). Learning the Pareto front using single unified model that can be applied to specific objective preference inference time. We propose PHN, a model for this setup based on a hypernetwork, and evaluate it on a large and diverse set of tasks and datasets. Our experiments show significant gains in both runtime and performance on all datasets. This novel approach provides users with the flexibility of selecting the operating point inference time and opens new possibilities where user preferences can be tuned and changed at deployment.
ACKNOWLEDGEMENT
This study was funded by a grant to GC from the Israel Science Foundation (ISF 737/2018), and by an equipment grant to GC and Bar-Ilan University from the Israel Science Foundation (ISF 2332/18). AS and AN were funded by grant from the Israeli Innovation Authority, through the AVATAR consortium.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yahav Bechavod and Katrina Ligett. Penalizing unfairness in binary classification. arXiv preprint arXiv:1707.000%0 2017.
Amlaan Bhoi. Monocular depth estimation: A survey, ArXiv, abs/1901.09402, 2019.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference Machine Learning, pp. 794 803. PMLR, 2018.
George Cybenko. Approximation by superpositions of sigmoidal function. Math. Control. Signals Syst., pp. 303-314, 1989.
Indraneel Das and John E Dennis. pareto surface in nonlinear multicriteria optimization problems. SIAM journal on optimization, (3):631-657, 1998.
intersection: A new method for generating the
Kalyanmoy Deb and Himanshu Jain. An evolutionary many-objective optimization algorithm using reference-poiin constraints. IEEE transactions evolutionary computation, 18(4):577-6011 2013.
nondominated sorting approach, part i: solving problems with box
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Compul., pp. 182-197, 2002.
Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In International Conference on Learning Representations, 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt::t/ttchhhhttee
ics. uci .edummldumm
C. Dwork, Nicole Immorlica, A. Kalai, and Mark D. M. Leiserson. Decoupled classifiers for group-fair and efficient machine learning. In FAT, 2018
Jean-Antoine Désidéri. Multiple-gradien descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350:313-31 318, 2012.
Matthias Ehrgott. Multicriteria optimization, volume 491. Springer Science & Business Media, 2005.
Yu G Evtushenko and Mikhail Anatol evich Posypkin. Nonuniform covering method as applied to multicriteria optimization problems with guaranteed accuracy. Computational Mathematics and Mathematical Physics, 53(2):144-14 2013.
Jörg Fliege and Benar Fux Svaiter: Steepest descent methods for multicriteria optimization. Math. Methods Oper. Res., 51(3):479-494 2000.
Carlos Manuel Mira da Fonseca. Multiobjectiive genetic algorithms with application control engineering problems. PhD thesis, University of Sheffield, 1995.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. ArXiv, abs/1609 609.09106, 2017.
Daniel Hernández: Mormato Jose Hrmnandez-Lobatee Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective bayesian optimization. International Conference on Machine Learning, pp. 1492-1501, 2016.
Chi-Hung Hsu, Shu-Huan Chang, Jhao-Hong Liang, Hsin-Ping Chou, Chun-Hao Liu, Shih-Chieh Chang, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, and Da-Cheng Juan. Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806. 10332, 2018.
X. Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1510-1519, 2017.
10

---

Published as a conference paper at ICLR 2021
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness aware classifier with prejudice remover regularizer, In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 35-50. Springer, 2012.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491. 2018.
Diederik P. Kingma and Jimmy Ba. abs/1412.69800, 2015.
Adam: A method for stochastic optimization. CORR,
Sylwester Klocek, Lukasz Maziarka, Maciej Wolezyk, J. Tabor, J. Nowak, and M. Smieja. Hypernet- work functional image representation. In ICANN, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-22) 232224,,, 1998.
X. Lin, Zhiyuan Yang, Q. Zhang, and Kwong. Controllable pareto multi-task learning. ArXiv, abs/2010.06313, 2020.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-tas learning. In Advances in Neural Information Processing Systems. pp. 12060-12070, 2019.
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1871-18800 2019.
Suyun Liu and Luis Nunes Vicente. Accuracy and fairness trade-offs in machine learning: A stochastic multi-objective approach. arXiv preprint arXhv:2008.01 2020.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. arXiv preprint arXiv:1802.088
2018.
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419-427, 2019.
Shangzhen Luan, Chen Chen, Baochang Zhang, Jungong Han, and Jianzhuang Liu. Gabor convolu- tional networks. IEEE Transactions Image Processing, 27(9):4357-4366 2018.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4765 4774. Curran Associates, Inc., 2017. moneeeeinnnmmmlllemee proodee--peeiittioooo pdf 7062 a- uni Fied- www.ffieeeiieeeeee approach ttw.fiiiieppprrd------t-eteeeeee
nips. ccpppc/ppppcpccppp///ppp/
Pingchuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration in multi-task
learning. arXiv preprint arXiv:2006. I066666644 16434, 2020.
Matthew Mackay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured hest-response functions. In International Conference on Learning Representations, 2019. URL /
net/Eozum?id rte.ridrrreeE
D. Mahapatra and V. Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. 2020.
N. Martínez, Martín Bertrán, and G. Sapiro. Minimax pareto faimess: A multi objective perspective. 2020.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and A. Galstyan. A survey on bias and fairness in machine learning. ArXiv, 004/1908.09635.. 2019.
Shervin Minaee, Yuri Boykov, F. Porikli, Antonio J. Plaza, N. Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. ArXiv, 0ix/2001.05566 2020.
11

---

Published as a conference paper at ICLR 2021
Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22-31, 2014.
Eliya Nachmani and L. Wolf.
decoders for block codes. In NeurlPS, 2019.
Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. arXiv preprint #rXiv:207700262/ 2020.
J. Oswald, C. Henning, J. Sacramento, and Benjamin F. Grewe. Continual learning with hypernet- works. ArXiv, abs/1906.00669 00095, 2020.
Simone Parisi, Matteo Pirotta, Nicola Smacchia, Luca Bascetta, and Marcello Restelli, Policy gradient approaches for multi-objective sequential decision making. In 2014 International Joint Conference on Newral Networks (IJCNN), pp. 2323-2330. IEEE, 2014.
Simone Parisi, Matteo Pirotta, and Marcello Restelli. Multi-objective reinforcement learning through continuous pareto manifold approximation. Journal of Artificial Intelligence Research, 57:187-227,
2016.
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and E. Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. ArXiv, abs/1606.021477, 2016.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. arXiv preprint 170900770.. 2017.
Matteo Pirotta and Marcello Restelli. Inverse reinforcement learning through policy gradient mini- mization. In AAAI Conference on Artificial Intelligence, pp. 1993-1999, AAAI Press, 2016.
Matteo Pirotta, Simone Parisi, and Marcello Restelli. Multi-objective reinforcement learning with continuous pareto frontier approximation. In 29th AAAI Conference Artificial Intelligence, AAAI 2015 and the 27th Innovative Applications of Artificial Intelligence Conference, IAAI 2015, pp. 2928-2934 AAAI Press, 2015.
Sebastian Ruder. An overview of multi-task learning deep neural networks, 2017.
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objeccivv optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018.
Amar Shah and Zoubin Ghahramani. Pareto frontier learning with expensive correlated objectives. In International Conference on Machine Learning, pp. 1919-1927, 2016.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks. arXiv preprint arXiv:2103.04628. 2021.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. European conference on computer vision, pp. 746-760. Springer, 2012.
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? arXiv preprint arXiv:1905.07553 2019.
Suarez. Character-level language modeling with recurrent highway hypernetwrkks. 2017.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, (5(1):3483-351) 2014.
Sethu Vijayakumar. The sarcos dataset, 2000. URL http://www. gaussianprocess. org/gpmi/data.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv: 2017.
12

---

Published as a conference paper at ICLR 2021
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems, pp. 14636-14647, 2019.
I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2): 2473-2480, 2009.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez- RodrizzRd and Krishna P Gummadi. Fairness constraints: A flexible approach for fair classification. J. Mach. Learn. Res., 20(75): 2019.
Qingfu Zhang and Hui Li. Moea/d: A multiobjecttiee evolutionary algorithm based decomposition. IEEE Transactions on evolutionary computation, 11()):12-731, 2007.
Yu Zhang and Qiang Yang. A survey on multi-task learning, arXiv preprint arXiv:1707.081 14, 2017.
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257-271, 1999.
Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation, 872):173-195, 2000.
13

---

Published as a conference paper at ICLR 2021
Appendix: Learning the Pareto Front with Hypernetworks
A EXPERIMENTAL DETAILS
For NYUv2 we performed hyperparameters search over the number of epochs and learning rates (le - 3, 5e 4, le 1ee44,, 4J. We train each method for 150 epochs with Adam (Kingma & Ba 2013) optimizer and select an optimal set of learning rate and
Pixelwise Classification and Regression:
number of epochs according to Sec. 5 We evaluate PHN over 25 evenly spaced rays.
LeNet-based target network (Le Cun et al. 1998) with two task-specific heads. We train all methods using an Adam optimizer with learning rate Ie for 150 epochs and batch size 256. We then choose the best number of epochs for each method using the validation set. We evaluate PHN over 25 rays.
Image classification: For the Multi-MNIST, experiments
Fairness:
We use 3-layer feed-forve feeward target network with hidden dimensions [40, 20). We use learnable embeddings for all categorical variables. We train all methods for 35 epochs using an Adam optimizer with learning rates {1e 3, 5e 4, le 4]. We choose the best hyperparameter configuration using the validation set. Further details on the datasets are provided in Table We evaluate PHN over sample of 150 rays.
Multitask regression: For the SARCOS experiment, we use a 4 layer ferd-forward target network with 256 hidden dimension. All categorical features are passed through embedding layers. We train all methods for 1000 epochs using an Adam optimizer and learning rates {le- 3, 5e - 4, - 4). We choose the best hyperparameter configuration base the validation set. We use 40,036 training examples, 4,448 validation examples, and 4, 4,449 test examples. We evaluate PHN over a sample of 100 rays.
Table 5: Fairness datasets.
Adult
Default
Bank
Train
32559
21600
29655
Validation
3618
2400
3295
Test
9045
6000
8238
B ADDITIONAL EXPERIMENTS
MOEA/D
MGDA
22
42
Fosible solutions
1.0
1.0
0.6
0.6
0.2
0.2
Pareto Front
(1 ray
0.2
0.6
1.0
L
0.2
0.6
1.0
Figure 6: Illustrative example using the popular task of Fonseca 1995 MOEA/D fails to reach the exact pareto front after 100, 000 generations. MGDA reaches Pareto optimal solutions, but not suitable for generating preference-specific solutions and only covers small fraction of the pareto front.
The genetic algorithm
14

---

Published as a conference paper at ICLR 2021
PHN
EPO
LS
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
Pareto Front
0.1
0.2
0.3
1
0.1
0.2
0.3 L
0.1
0.2
0.3 1
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25 0.50 0.75
L
0.25 0.50 0.75
1
0.25 0.50 0.75 L
Figure 7: Known Pareto front: Top: Evaluation on Problem 2 from Evtushenko & Posypkin (0013). Bottom: Evaluation on ZDT2 (Zitzler et al. 2000). PHN converges to the entire PF, mapping a ray to its corresponding solution on the front. Each circle in the PHN plots denotes solution corresponds to the baseline rays.
B. KNOWN PARETO FRONT
We provide two additional experiments popular MOO benchmarks whose Pareto front is known. In addition Figure 6 we provide additional results for MGDA (Sener & Koltun 2018 and
MOEA/D (Zhang & Li 2007) the illustrative example from Section 5.1
We evaluate PHN on two MOO problems with known, non-convex Pareto front: ZDT2 ZZtzler et al. 2000) and problem two from Evttsheusee & Posypkin (2013), henceforth Problem2. In both problems, the density of solutions across the Pareto-optimal region is non-uniform.
For solution 0 (2), [0, 1) Problem2 is given by, (0) ((01 1)63 + 1)/3, (2) 01,
and ZDT2 given by,
(0) = 0,, (2) = - 0
902
The results are presented in Figure PHN-EPO converges to the entire PF using a single HN model. in contrast with PHN-EPO, EPO fails to converge the PF for all rays in ZDT2.
B.2 GENERALIZATION TO UNSEEN RAYS
When the number of objectives is large, the curse of dimensionality makes it likely that rays encoun- tered in test time are "far" from rays encountered during training. Although we evaluated PHN on unseen raasethhoooe through all experiments, here would like to better examine the generalization of PHN in that scenario. We conducted controlled experiment on SARCOS dataset with 7 tasks. During training, sample rays from predefined grid the dimmeension simplex. We the structured approach of Das & Dennis (1998) for generating well-spaced points. During inference, we measure HV and Uniformity on two sets of rays: Randomly sampled train rays (ii) Randomly sampled test rays from a shifted grid, so they are most distant from the train grid. In both cases, test samples differ
With probability.
15

---

Published as a conference paper at ICLR 2021
from training samples. We find that testing with unseen test rays only causes minor decrease in both HV and Uniformity compared with train rays. The HV reduced from 0.7337 to 0.7336 and uniformity is reduced from 0.163 to 0.158. This experiment indicates that PHN generalizes well to unseen rays, even in problems with many objectives. We note that the HV results presented here are reported in Section 5.3 We attribute this inconsistency change of preference
higher than the vectors distribution used during training. While throught the paper we sample preference rays from Dirichlet distribution with 0.2, here we sample from a fixed, evenly spaced grid. We provide analysis on the robustness to the choice of the Dirichlet distribution parameter a in Appendix B.5
B.3
MNIST PHN INTERPRETATION
Preference
Image
0
1
2
3
4
5
6
7
8
9
7
(0.99, 0.01)
B
8
8
8
8
8
8
8
8
(0.50, 0.50)
B
2
2
2
6
6
8
(0.01, 0.99)
B
8
B
8
B
8
8
Figure 8: PHN interpretation: PHN provided with a sample from the Multi-MNIST dataset, where image of and an image 7 are overlayed. We show the effect that conditioning different preference rays (left) has on the decision made by the model. Red pixels denote increased contribution the prediction. See text for details.
To better understand the behaviour of PHN while varying the input preference vector, Figure 8 shows the per-pixel contribution to prediction. We use the SHAP framework (Lundberg & Lee) 2017) for visualization. Red pixels denote increased probability for given class, while blue pixels denote a decreased probability. The top left and bottom right digits are labeled as {7, 3) respectively. In the first row (ray: (0.99, 0.01)). the model focused on task I (with label 7), most of the red pixels come from the top left digit in column 7, as expected. Similarly, in the third row (ray: (0.01, 0.99)) most of the red pixels are focused on the bottom right digit in column 3.
B.4 SAME-RAY EVALUATION
In Section 5 of the main text we evaluate PHN on multiple evenly spaced rays. For completeness, we report here the HV and Uniformity achieved by PHN, while evaluated on the same rays used for training and evaluating the baseline methods. The results are provided in Table Surprisingly, in many cases PHN achieves better performance. We attribute this to the inductive bias effect of sharing weights across all preference rays.
Table 6: Evaluation of PHN on baselines' ray.
FasHion+MNIST Unit.
Multi-Fashion Cnif.
Mulii-MNIST
Adult
Default
Bank
NYUv2
SARCOS
Unif.
Unit.
Unit.
Unit.
Unif.
HV
Unif.
PHN-LS 2.740
0.852
2.190
0.838
2.900
0.840
0.624
0.282
0.538
0.106
0.695 0.603
0.683
0.693
-0.051
PHN EPO 2,760
0.908
2,180
0,860
2,760
0.857
0.631 0.742
0,531 0.395
0.717 0.833
3.565 0,701
0.728 0.190
B.5 DESIGN CHOICE ANALYSIS
In Section 5 we used fixed values of the hidden dimension and the Dirichlet parameter (see Alg. T. Here we examine the effect of different choices for these hyperparameters. We use the Multi-Fashion
+ MNIST dataset, and the PHN-EPO variant of our method. We first fix the hidden dimension to 100, and vary from 0.1 to 1 (uniform sampling). The results are presented in Figure 9Ia). at least for this dataset, PHN is quite robust to changes in a. Next, we set a 0.2 and examine the effect of changing the dimension of the hidden layers in hypernetwork. We present the results in Figure (b). Here, the effect on the HV is stronger, with best HV achieved using hidden dimension of 100.
a). It appears that
16

---

Published as a conference paper at ICLR 2021
(a)
(b)
2.75
HV
2.00
0.3
0.2
0.5
1.0
50
100
250
Haldon dam.
Figure 9: Hyperparameters analysis: Performance analysis the Multi-Fashion MNIST dataset, for different architecture designs and sampling choices.
B.6
ADDITIONAL RUNTIME ANALYSIS
(a) Multi-Fashion
(b) Multi-MNIST
(c) NYUv2
2.00
2.1
2.85
200
2,80
3.5-
3.1
2,70
US
2.65
1.7
PHN BPG
2.60
2.55
3.0
10
10
Buntine [inin]
Rantime [mcin.]
Puntime jnit..]
Figure 10: Runtime analysis for NYUv2, and two additional Multi-MNIST dataset.
We extend the runtime analysis from Section 5.5 Figure 10 provides additional runtime analysis on the Multi-MNIST, Multi-Fashion, and NYUv2 datasets. For NYUv2 we omit the EPO method, since was outperformed by LS in both HV and runtime.
C HYPERVOLUME
Figure 11: Hypervolume illustration: The colored area represents the hypervolume obtained by Pareto
set (PI, Pz, Ps} with respect to the reference point.
The hypervolume metric defined as follows: Given a set of points S C RR and a reference point R the hypervolume of S is measured by the region of non-dominated points bounded above by p.
HV(S) VOL Rd E S: x)
17

---

Published as a conference paper at ICLR 2021
where VOL is the euclidean volume. We can interpret hypervolume the measure of the union of the boxes created by the non-dominated points.
11 P Pi pil PES
HV(S) VOL
P=P
As measure of quality for MOO solutions, it takes into account both the quality of individual solutions, which is the volume they dominate, and also the diversity, measured in the overlap between dominated regions. See example in Fig. II
18


Reviews:
[{"review_id": "mTLHGIO13Ge", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for the efforts of the authors. After reading the response and other reviews,  I raise my score from 5 to 6. However, the proposed algorithm lacks analysis. So I cannot improve my score further. \n___________________________________________________\n\nSummary: \nThe paper proposes using a hypernetwork to learn the entire Pareto front of a multi-objective optimization.  It develops two approaches using linear scalarization and exact Pareto optimal (EPO), respectively. Experiments on several multi-task learning tasks show its advantage in terms of hypervolume and uniformity.\n\nStrengths: \n+ It is very interesting and useful that the paper tries to learn the entire Pareto front directly.\n+ The paper proposes using hypernetworks to learn the Pareto front for multi-task learning. \n\nWeaknesses: \n- Algorithm 1 is the main contribution. The pseudocode and its descriptions are not clear. For example, it is better to use two algorithms to descript PHN-LS and PHN-EPO, respectively. r is the weights for LS, while it is the ray for EPO. However, r should be pre-given. How and why PHN could optimize the multi-objective optimization for any r? What does “Dir(\\alpha)” mean?\n-PHN is a solving method for MOO problems. So the paper should verify that it could learn the entire Pareto front. The paper may test with MOO problems with known Pareto fronts. The results in Figure 2 cannot testify to this claim. This is more important than the experiments on the multi-task learning.\n\nMinor comments:\n1.\tThe legend in Figure2 should be colored.\n2.\tThe captions of the subfigures in Figure 4 should be under them. \n3.\tTypos:\nsampled “form” the m-dimensional-> sampled “from” the m-dimensional, “minimas”->”minima”, longer “then”-> longer “than”, “follwoing\" ->“following\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hQluav46dvD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tracks the problem of learning the entire Pareto front to allow the user to select a desired Pareto optimal solution by one inference procedure without retraining the model. The high-level idea is to learn the entire Pareto front simultaneously using a single hyper network, which receives as input the desired preference vector and returns a Pareto-optimal solution whose loss vector is in the desired direction. The paper gives an early trial to build a toolbox to allow users to get a desired solution by a single inference procedure.\n\nStrength:\n1.\tThis paper is an early trial to use a hyper network to directly approximate the Pareto Optimal front, allowing practitioners to flexibly choose Pareto solutions conditioned on different preference vectors.\n2.\tThe paper is well-written and a substantial number of experiments are conducted, promising good results of the proposed method. \n\nFeedbacks:\n1.\tMy major concern is that there is a similar work [1] that shares the same spirit with the work.  The authors might want to clarify the differences or conduct some experimental comparisons.\n2.\tIt is better to attach more details about the EPO algorithm.\n3.\tThere are some typos or grammar issues, such as 1) Page 6, the title of the subsection ‘Hyperparamter tuning’ should be ‘Hyperparameter tuning’. 2) Page 6, ‘We therefor’ should be ‘We therefore’. 3) Page 6, ‘as follow’ should be ‘as follows’. 4) Page 7, ‘are train and evaluate’ should be ‘are trained and evaluated’.\n\n[1] X. Lin, Z. Yang, Q. Zhang, and S. Kwong, “Controllable Pareto multi-task learning,” arXiv preprint arXiv: 2010.06313, 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "M0ohlA8T8X", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel algorithm for MOO, which combines preference-based methods with hypernetworks, in order to encapsulate the preferences in the network input.\nThe paper is well written, the proposed method is clear, and the experiments are sufficient. \nMy concerns are the following.\n\nFirst, in the abstract you say that\n\"Recent optimization algorithms can target a specific desired ray in loss space,\nbut still face two grave limitations: (i) A separate model has to be learned for\neach point on the front; and (ii) The exact trade-off must be known prior to the\noptimization process. \"\n\nThis is not entirely true. In Reinforcement Learning (RL) manifold-based approach such as\n\nParisi et al, \"Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation\"\nYang et al, \"A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation\"\n\ndo not need neither a separate model, nor to know the trade-off a priori. The work of Parisi et al learns a parametrization producing infinitely many solutions at the same time using a specific loss, while Yang et al train a neural network which takes preferences over the objectives as input, and generalizes over them. \nDespite being tested only on MORL, these algorithms can easily be extended to MOO.\nTherefore, I suggest you to rephrase your abstract and introduction to mention these methods.\n\nIn particular, the work of Yang et al is very reminiscent of your algorithm, since both include the preference vector in the network input. \n\nNonetheless, I feel the paper has enough novely thanks to the use of hypernetworks, EPO, and its application to MOO.\nThe evaluation is sufficient, but I would suggest to move the evaluation of evolutionary algorithms to the main section. Evolutionary algorithms are extremely popular in MOO and thanks to a wide variety of fitness functions can learn any frontier, even though they may require longer run time (as your experiments clearly show).\n\nFinally, a bit more analysis on concave frontier would be beneficial. In Figure 1 you show that LS fails in finding concave frontiers, but it seems to me that all experiments have convex frontier. To my understanding, EPO should address this limitation (\"The recent EPO algorithm can guarantee convergence to a local Pareto optimal point on a specified ray r, addressing the theoretical limitations of LS\"). Am I correct? A bit more discussion and experiments with concave frontiers would be a nice addition to the paper.\n\nOverall, I am leaning to accept it but the authors should address the above issues.\n\n**EDIT **\nThe authors have addressed my concerns, and I have increased my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "K5tA3-pFp-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for multi-objective optimization. The key idea is to learn the entire Pareto front at once by training a hypernetwork that takes preference vector as an inputs and outputs network parameters, which corresponds to a point on the Pareto set with the desired trade-off specified by the preference vector. Specifically, the hypernetwork is a multi-head network where each head outputs a weight tensor of a module in the target network. The method improves HV from the baselines, in several multi-task learning problems, including image classification, regression and, mixed classification and regression.\n\n+) The main contribution of this work is to learn a continuous function that maps a preference vector to network parameters that corresponds to the desired trade-off. The trained hypernetwork generalizes to preference vectors unseen during training so that the required training time for getting models of an arbitrary trade-off is reduced.\n\n+) Compared to prior work, CPMTL, which starts from a Pareto optimal point and extends Pareto front locally around the point, the proposed method attempts to train a single hypernetwork that represents the entire Pareto front.\n\n+) Pareto front provides insights on the trade-off relation between tasks but it usually requires repetitive training of models under different preference settings. I think this method can be useful when understanding the relations among tasks by significantly reducing the training time to obtain the Pareto front.\n\n-) One concern is the scalability to the number of tasks. The amount of reduce in the training time of the proposed method over the base methods depends on how much the method generalize to the preference vectors unseen during training. For example, ideally, the model trained using the preference vectors [0,1] and [1,0] generalizes to [0.5,0.5]. In other words, it is desired that a hypernetwork is trained using less number of samples while generalizes well to arbitrary trade-offs, relying on the smoothness of the hypernetwork. Otherwise, the training requires more number of iterations to sample sufficient number of preference vectors that covers the whole preference vector space to match the original performance, where its size grows exponentially to the number of tasks. Current manuscript lacks experimental or theoretical analysis on this generalization performance.\n\n-) Relating to the above point, I think evaluation measure HV and uniformity are not sufficient to evaluate the overall behavior. HV measures the quality of exploration, but higher HV does not necessarily mean that every model of method A dominates method B. For example, in the left figure of Figure 2, some points of EPO dominates PHN-EPO though HV is higher for PHN-EPO (Table 1). This implies that the proposed method reduces training time at cost of possible degrading movement of Pareto front. In other words, depending on the hyperparameter alpha used for the Dirichlet distribution, under the comparable training time to the baseline, the baseline method may dominate its PHN counter part at certain preferences.\n\n-) And therefore, I think one needs to investigate the accuracy plot (as Figure 2) together with HV and uniformity to better understand the behavior of the method. Some are missing in the current manuscript (e.g. NYUv2)\n\n-) Another concern is the scalability to the target model capacity. In the current manuscript, all the experiments are performed using architectures with small number of parameters (up to \\~0.37M of ENNet), which is much smaller than popular architectures such as ResNet50 (\\~26M). In addition, the required size of the hypernetwork grows at least proportional to the target model capacity,\nand the amount of training data and the training time likely increase accordingly.\n\n-) HyperNetwork require some overhead of memory and computation cost during the test time.\n\n-) Comparison with important prior work CPMTL is necessary.\n\nI would recommend 'accept'. Though I have concerns about the scalability and lack of analysis, I think this work has some advantages stated above.\n\nQuestions:\n1. Discussion on the generalization performance to unseen preference vectors\n2. Comparison with CPMTL\n3. Accuracy plot for NYUv2\n\n___\nThanks for the response. After reading the authors' response and other reviews, I would like to keep my recommendation.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a hyper-net method for multi-objective optimization, which trains a neural network that maps preference vector to the corresponding Pareto solution. The proposed idea is interesting and useful, although the evaluation of the work is not overwhelming convincing. The writing of the work can be further improved. 

Also, the basic idea of the work is the almost the same as a concurrent work "Lin et al 2020. controllable pareto multi-task learning" which is also submitted to this conference. The paper cited that paper briefly, "... The proposed method is conceptually similar to our approach...",  which is too vague and brief. We urge the author to provide a through discussion on the detailed difference and similarity of the works, including empirical comparisons when necessary. 
Decision: Accept (Poster)