Profiling Pareto Front With Multi-Objective Stein Variational  Gradient Descent

1 Introduction . Many scientific and engineering problems involve optimizing multiple conflicting objectives [ 5 , 28 , 4 , 24 ] , including , for example , designing wireless sensors [ 12 ] , building electric power systems [ 30 ] , and training neural networks with multiple tasks [ 35 ] . With multiple conflicting objectives , it is impossible to find a single solution that optimizes all the objectives simultaneously . Instead , it is essential to find a set of diverse solutions in the Pareto front that represent different preferences on the different objective functions , so that the users can have a global view of how the optimal trade-off of the different objectives look like and select the solution according to their own preference . Unfortunately , profiling the Pareto fronts casts a key computational challenge , especially for high dimensional problems [ 5 , 12 , 28 ] . Traditionally , a large literature has been devoted to developing black-box , derivative-free algorithms that are suitable for black-box optimization , such as these based on evolutionary algorithms [ 8 ] and Bayesian optimization [ 25 , 1 , 39 ] . However , the black-box algorithms tend to be expensive and can only be applied to small scale problems due to the lack of gradient information . Gradient-based MOO algorithms have been catching attention only recently , which include mainly multiple gradient descent ( MGD ) based methods [ 10 , 22 , 27 ] . However , ∗Code is available at https : //github.com/gnobitab/MultiObjectiveSampling . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the results of these methods are still dissatisfying in finding diverse and evenly distributed Pareto solutions in complex problems . We introduce Stein variational gradient descent ( SVGD ) [ 20 , 19 , 21 ] and Langevin dynamics [ 37 ] as efficient approaches for profiling Pareto fronts . These methods iteratively evolve a group of particles to represent the target distribution . The main difference of the two algorithms is how they distribute points . The Langevin dynamics uses stochastic noises to perturb the particle trajectories , so they can visit different areas of the Pareto front . On contrast , SVGD is a deterministic sampling algorithm that pushes the particles to high probability regions using gradient information , while enforcing diversity between the particles using a repulsive force . In this work , we propose to a simple approach to integrate SVGD and Langevin Dynamics with MGD to draw samples from the Pareto front . Theoretical analyses are provided for both algorithms to understand their limiting distributions and their convergence speed . One challenge with MGD based sampling is that the limiting distributions do not admit explicit formulations . This is mainly because the forcing from MGD is in general not the gradient of any function . However , we can show its non-gradient component is orthogonal to a large class of functions . Assuming each objective function is strongly convex and regular , we can also show the limiting distributions concentrate on the Pareto front . Moreover , we can show the two algorithms converge to good solutions of MOO with O ( 1/t ) and linear rate . We test our methods on a variety of tasks , ranging from low-dimensional optimization to multi-task neural network optimization . On all the tasks tested , our method can obtain diverse and high quality Pareto solutions that distribute evenly on the Pareto front , without predefined preference vectors . Quantitatively , we substantially outperform PF-SMG and EPO with respect to the hypervolume metric . 2 Background . In Multi-objective optimization ( MOO ) , we are interested in minimizing a vector-valued loss function F ( x ) = ( f1 ( x ) , f2 ( x ) , . . . , fm ( x ) ) ∈ Rm , where x ∈ Rd and fi : Rd → R , i ∈ [ m ] is the i-th scalar-valued objective function . Here we use notation [ m ] = { 1 , 2 , . . . , m } . Obviously , we can not fully optimize all the objective functions simultaneously because they may be conflicting with each other . Instead , we are interested in finding the points which can not be improved simultaneously for the objective functions , yielding the notion of Pareto optimality . Definition 1 ( Pareto Optimality ) . For x1 , x2 ∈ Rd , We say that x1 is dominated by x2 iff fi ( x2 ) ≤ fi ( x1 ) , ∀i ∈ [ m ] , and F ( x1 ) 6= F ( x2 ) . A point x∗ is called globally Pareto optimal on Rd iff it is not dominated by any other x′ ∈ Rd . A point x∗ is called locally Pareto optimal iff there exists an open neighborhood N ( x∗ ) of x∗ , such that x∗ is not dominated by any x ∈ N ( x∗ ) . The collection of globally ( resp . locally ) Pareto optimal points are called the global ( resp . local ) Pareto set . The collection of function values F ( x∗ ) of all the Pareto points x∗ is called the Pareto front . Our goal is to find diverse and representative solutions { xi } ni=1 from the Pareto set , so that their function values { F ( xi ) } ni=1 covers different preferences on different objectives . This would allow the end-users to have a global view on the optimal trade-off between the different objective functions and decide which solution based on their own preference . Linear Scalarization One standard approach to solve MOO is using a preference vector λ = [ λ1 , . . . , λm ] from the probability simplex on [ m ] , i.e. , S = { λ : ∑m i=1 λi = 1 , λi ≥ 0 , i ∈ [ m ] } . Each λ ∈ S leads to a weighted objective function fλ ( x ) = ∑m i=1 λifi ( x ) and its minimizer x∗λ = arg minx fλ ( x ) . Then as we take λ in a grid of S , we hope that the corresponding x∗λ gives a rough sketching of the Pareto front . Although this strategy is simple and easy to implemented , it suffers from the a number of weaknesses . A key problem is that F ( x∗λ ) can only lie on the convex envelope of the Pareto front , and hence it only works in cases when the Pareto front is convex . In addition , a uniform grid of λ on S does not necessarily yield uniformly distributed points on the Pareto front . Multiple Gradient Descent ( MGD ) MGD is a natural extension of the single-objective gradient descent to finding a Pareto point [ 10 ] , which , unlike linear scalarization , works for non-convex Pareto fronts . The idea is to iteratively update the variable x along a direction that ensures that all the objectives are decreased simultaneously ( which is called Pareto improvement ) . Let gi ( x ) = ∇fi ( x ) be the gradient of the i-th objective . Suppose we update the variable by x′ ← x− g∗ ( x ) , where g∗ ( x ) is a vector field to be determined and is a small step size . By Taylor approximation , we have 〈gi , g∗〉 ≈ − ( fi ( x′ ) − fi ( x ) ) / , which represents the decreasing rate of fi when we update x along direction g∗ ( x ) . In MGD , g∗ is chosen to maximize the slowest decreasing rate among all the objectives , that is , g∗ ( x ) ∝ arg max g∈Rd { min i∈ [ m ] 〈g , gi ( x ) 〉 , s.t . ‖g‖ ≤ 1 } . ( 1 ) Therefore , g∗ ( x ) is encouraged to have positive inner products with all gi ( x ) . If this is impossible to achieve , { gi ( x ) } mi=1 will contain conflicting directions , and we would have g∗ ( x ) = 0 which terminates the algorithm . Using Lagrangian duality , we can show that the optimal solution to ( 1 ) is g∗ ( x ) ∝ ∑m i=1 λ ∗ i ( x ) gi ( x ) , where { λ∗i ( x ) } mi=1 is the solution of min { λi } ∥∥∥∥∥ m∑ i=1 λigi ( x ) ∥∥∥∥∥ s.t . m∑ i=1 λi = 1 , λi ≥ 0 , ∀i . ( 2 ) This optimization has a simple closed-form solution when m = 2 , and a fast algorithm is offered by [ 35 ] for m > 2 . By construction , when the step size is small , MGD monotonically decreases all the objectives simultaneously and will terminate when it arrives a local Pareto point ; in this case we have g∗ ( x ) = ∑m i=1 λ ∗ i ( x ) gi ( x ) = 0 , suggesting that the zero vector 0 is inside the convex hull of { gi ( x ) } mi=1 . The MGD direction g∗ provides a natural notion of gradient for multiple objectives . A point x is said to be Pareto stationary if g∗ ( x ) = 0 . Similar to the case of differentiable single-objective optimization , every Pareto local optimal is a Pareto stationary point . See [ 10 , 35 , 22 ] for more discussion . However , the vanilla MGD suffers from several key weaknesses . Although it promises to return a point on the ( local ) Pareto set , and it is difficult to explicitly control which Pareto point it will converge to . The convergence point of MGD is implicitly determined by the initialization and the other hyper-parameters of the algorithm ( e.g. , step size ) in a complicated way . It can be harder for MGD to converge to some Pareto points than the others . In fact , assume the Pareto front is an open set in Rd , then the vanilla MGD , when initialized from outside of Pareto front , will terminate when it reaches the boundary of the Pareto front , and hence never reach the interior points . Below is a simple example to demonstrate this . Example 1 . Consider f1 ( x ) = x2 and f2 ( x ) = ( x − 1 ) 2 for x ∈ R as shown in Figure 1 . Then the Pareto set is the interval [ 0 , 1 ] . However , when initialized outside of [ 0 , 1 ] , MGD will converge to either x = 0 or x = 1 and can not reach the interior points of the Pareto set unless we add a diversity-promoting mechanism , such as random noise or deterministic repulsive force . See figure 1 . 3 Profiling Pareto front with Particle Dynamics . We improve MGD by integrating it with sampling algorithms for approximating distributions to obtain diverse solutions that cover the Pareto front more evenly . In this section , we first introduce our main algorithm , which integrates MGD with Stein variational gradient descent ( SVGD ) , yielding an interacting particle system that evolves diverse Pareto solutions with deterministic repulsive forces ( Section 3.1 ) . We also introduce a related idea of combining MGD with Langevin dynamics to obtain diverse solutions by injecting diffusion noises , which is found less favorable in practice than SVGD-based method but is of independent interest ( Section 3.2 ) .

Summary: This paper proposes a method to learn multiple Pareto solutions for multi-objective optimization (MOO). Specifically, the method iteratively updates a set of points by Stein variational gradient descent (SVGD) to encourage the diversity between the solutions. Another counterpart method based on Langevin dynamics is also provided. The methods are theoretically guaranteed to converge to the Pareto front. The experimental results demonstrate the superiority of the proposed SVGD type of method compared with EPO. The superiority is shown visually and by the hypervolume of the solutions.