Title: Neural Multi-Objective Combinatorial Optimization with Diversity Enhancement
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural heuristic

diversity enhancement

deep reinforcement learning

multi-objective combinatorial optimization
Abstract: Most of existing neural methods for multi-objective combinatorial optimization (MOCO) problems solely rely on decomposition, which often leads to repetitive solutions for the respective subproblems, thus a limited Pareto set. Beyond decomposition, we propose a novel neural heuristic with diversity enhancement (NHDE) to produce more Pareto solutions from two perspectives. On the one hand, to hinder duplicated solutions for different subproblems, we propose an indicator-enhanced deep reinforcement learning method to guide the model, and design a heterogeneous graph attention mechanism to capture the relations between the instance graph and the Pareto front graph. On the other hand, to excavate more solutions in the neighborhood of each subproblem, we present a multiple Pareto optima strategy to sample and preserve desirable solutions. Experimental results on classic MOCO problems show that our NHDE is able to generate a Pareto front with higher diversity, thereby achieving superior overall performance. Moreover, our NHDE is generic and can be applied to different neural methods for MOCO.

Neural Multi-Objective Combinatorial Optimization with Diversity Enhancement

Abstract

Most of existing neural methods for multi-objective combinatorial optimization (MOCO) problems solely rely on decomposition, which often leads to repetitive solutions for the respective subproblems, thus a limited Pareto set. Beyond decomposition, we propose a novel neural heuristic with diversity enhancement (NHDE) to produce more Pareto solutions from two perspectives. On the one hand, to hinder duplicated solutions for different subproblems, we propose an indicatorenhanced deep reinforcement learning method to guide the model, and design a heterogeneous graph attention mechanism to capture the relations between the instance graph and the Pareto front graph. On the other hand, to excavate more solutions in the neighborhood of each subproblem, we present a multiple Pareto optima strategy to sample and preserve desirable solutions. Experimental results on classic MOCO problems show that our NHDE is able to generate a Pareto front with higher diversity, thereby achieving superior overall performance. Moreover, our NHDE is generic and can be applied to different neural methods for MOCO.

1 Introduction

Multi-objective combinatorial optimization (MOCO) has been extensively studied in the communities of computer science and operations research [1, 2]. It also commonly exists in many industries, such as transportation [3], manufacturing [4], energy [5], and telecommunication [6]. MOCO features multiple conflicting objectives based on NP-hard combinatorial optimization (CO), practical yet more complex. Rather than finding an optimal solution like in the single-objective optimization, MOCO pursues a set of Pareto-optimal solutions, called Pareto set , to trade-off the multiple objectives. In general, a decent Pareto set is captured by both desirable convergence (optimality) and diversity.

Since exactly solving MOCO may require exponentially increasing computational time [7], the heuristic methods [8] have been favored in practice over the past few decades, which aim to yield an

HGA

Model

𝜽

f

(

x

5

)

f

(

x

4

)

f

(

x

3

)

f

(

x

2

)

0

f

1

f

2

r

f

(

x

1

)

f

(

x

3

)

f

(

x

2

)

0

f

1

f

2

r

f

(

x

1

)

0

f

1

f

2

r

Instance

s

Weight

𝝀

1

Diversity factor

w

1

Points

r

Points

ℱ 1

{ڂ

𝒓}

Points

ℱ 2

{ڂ

𝒓}

MPO

…

Pareto front

ℱ 1

HGA

Model

𝜽

Instance

s

Weight

𝝀

2

Diversity factor

w

2

MPO

Pareto front

ℱ 2

𝝀

1

𝝀

2

approximate Pareto set. Despite a relatively high efficiency, heuristic methods need domain-specific knowledge and massive iterative search. As neural CO methods based on deep reinforcement learning (DRL) recently achieved notable success in CO problems such as routing, scheduling, and bin packing [9-12], a number of neural MOCO methods have also been accordingly developed [13-16]. Typically, parameterized as a deep model, Neural MOCO is able to automatically learn a heuristic (or policy), so as to directly construct near-optimal solutions in an end-to-end fashion, which takes much less computational time than the traditional ones.

Existing neural MOCO methods mostly decompose an MOCO problem into a series of singleobjective CO subproblems and derive a Pareto set by solving them. However, while enjoying a favorable efficiency for neural MOCO, the sole decomposition is less effective in finding as many diverse solutions as possible, since constructing an optimal solution for the decomposed subproblems is always carried out independently, causing repetitive or duplicated ones for different subproblems.

To tackle this issue, we propose a neural heuristic with diversity enhancement (NHDE), as illustrated in Figure 1. Distinguished from existing neural methods, NHDE couples the decomposition with a comprehensive indicator to learn a policy that can produce diverse solutions across the subproblems while further improving the performance. Besides, for a given subproblem, multiple relevant solutions, rather than a single optimal solution with respect to the scalar objective, are found based on a proposed multiple Pareto optima (MPO) strategy, so as to further strengthen the diversity.

Our contributions are summarized as follows. (1) We propose an indicator-enhanced DRL method. To encourage the deep model to generate different yet diverse solutions for decomposed subproblems, NHDE inputs the Pareto front composed of the preceding solutions and introduces an indicator comprehensively measuring convergence and diversity in the reward. (2) We design a heterogeneous graph attention model to effectively capture the correlation between an instance graph and its Pareto front graph. (3) We present a multiple Pareto optima (MPO) strategy to further identify more relevant solutions in the neighborhood of each subproblem and efficiently update the Pareto front. (4) We deploy our NHDE with two different types of neural MOCO methods to demonstrate its versatility. Experimental results based on various MOCO problems show that our NHDE outperforms state-of-the-art neural baselines, especially with significant improvement in the diversity.

2 Related works

Exact and heuristic methods for MOCO. Exact methods [7, 17] for MOCO can attain the accurate Pareto set, but their computational time may grow exponentially, rendering them less practical. As an alternative, heuristic methods such as multi-objective evolutionary algorithms (MOEAs) have gained widespread attention in practice. Dominance-based NSGA-II [18], decomposition-based MOEA/D [19], and indicator-based SMS-EMOA [20] are three typical paradigms of MOEAs. In these MOEAs, local search, a crucial technique specialized to the target CO, is usually employed [21-23].

Neural CO. In the past few years, neural construction methods [24-26] were proposed to rapidly yield high-quality solutions in an end-to-end fashion. A well-known representative is Attention Model (AM) [27], which was developed based on the Transformer architecture [28]. Then, AM inspired a large number of subsequent works [29-35], which further boosted the performance. Among them,

the policy optimization with multiple optima (POMO) [36], leveraging the solution and problem symmetries, is recognized as a prominent approach. Besides, the other line of works, known as neural improvement methods [37-41], exploited DRL to assist the iterative improvement process from an initial but complete solution, following a learn-to-improve paradigm.

Neural MOCO. Decomposition is a mainstream scheme in learning-based methods for multiobjective optimization [42-44]. An MOCO problem can be decomposed into a series of singleobjective CO problems and then solved by neural construction methods to approximate the Pareto set. A couple of preliminary works trained multiple deep models with transfer learning [13, 45], where each deep model coped with one subproblem. Evolutionary learning [46, 47] was introduced to evolve deep models to further improve the performance. Instead of training multiple deep models for preset weights, preference-conditioned multi-objective combinatorial optimization (PMOCO) [14] and Meta-DRL (MDRL) [15], both of which only trained one deep model, were more flexible and practical. For a given weight vector, the former used a hypernetwork to derive the decoder parameters to solve the corresponding subproblem, while the latter rapidly fine-tuned a pre-trained meta-model to solve the corresponding subproblem. However, those solely decomposition-based neural MOCO methods are limited in the diversity with respect to the solutions of the Pareto set, since some subproblems may lead to duplicated solutions, especially when they are solved independently.

3 Preliminary

3.1 MOCO

An MOCO problem with M objectives can be expressed as min x ∈X f ( x ) = ( f 1 ( x ) , f 2 ( x ) , . . . , f M ( x )) , where X is a set of discrete decision variables.

Definition 2 (Pareto optimality). A solution x ∗ ∈ X is Pareto-optimal if it is not dominated by any other solution, i.e., ∄ x ′ ∈ X such that x ′ ≺ x ∗ .

Definition 1 (Pareto dominance). Asolution x 1 ∈ X dominates another solution x 2 ∈ X ( x 1 ≺ x 2 ), if and only if f i ( x 1 ) ≤ f i ( x 2 ) , ∀ i ∈ { 1 , . . . , M } and ∃ j ∈ { 1 , . . . , M } , f j ( x 1 ) < f j ( x 2 ) .

Definition 3 (Pareto set/front). MOCO aims to uncover a Pareto set, comprising all Pareto optimal solutions P = { x ∗ ∈ X | ∄ x ′ ∈ X : x ′ ≺ x ∗ } . The Pareto front F = { f ( x ) | x ∈ P} corresponds to the objective values of Pareto set, with each f ( x ) referred to as a point in the objective space.

3.2 Decomposition

For MOCO, decomposition [19] is a prevailing scheme due to its flexibility and effectiveness. An MOCO problem can be decomposed into N subproblems with N weights. Each subproblem is a single-objective CO problem via scalarization g ( x | λ ) with a weight λ ∈ R M satisfying λ m ≥ 0 and ∑ M m =1 λ m = 1 . The Pareto set then can be derived by solving the N subproblems.

The simplest yet effective decomposition approach is the weighted sum (WS). It uses the linear scalarization of M objectives, which hardly raises the complexity of the subproblems, as follows,

3.3 Indicator

Hypervolume (HV) is a mainstream indicator to measure performance, as it can comprehensively assess the convergence and diversity without the exact Pareto front [48]. For a Pareto front F in the objective space, HV r ( F ) with respect to a fixed reference point r ∈ R M is defined as follows, where µ is the Lebesgue measure, i.e., M -dimensional volume, and [ f ( x ) , r ] is a M -dimensional cube, i.e., [ f ( x ) , r ] = [ f 1 ( x ) , r 1 ] ×··· × [ f M ( x ) , r M ] .

A 2-dimensional example with 5 points in the objective space is depicted in Figure 1, where F = { f ( x 1 ) , f ( x 2 ) , f ( x 3 ) , f ( x 4 ) , f ( x 5 ) } . HV r ( F ) is equal to the size of the gray area, and finally normalized into [0 , 1] . All methods share the same reference point r for a problem (see Appendix A).

4 Methodology

Our neural heuristic with diversity enhancement (NHDE) exploits indicator-enhanced DRL to produce diverse solutions across different subproblems and leverages a multiple Pareto optima (MPO) strategy to find multiple neighbor solutions for each subproblem, as illustrated in Figure 1. Specifically, an MOCO problem is decomposed into N single-objective subproblems with N weights, which are solved dependently by a unified heterogeneous graph attention (HGA) model θ . For each subproblem i , its features together with the current Pareto front F ( points in the objective space) constituted by preceding solutions are input to model θ , which is guided by the scalar objective with the HV indicator. Then, MPO is utilized to sample multiple solutions and efficiently update F .

4.1 Indicator-enhanced DRL

Given a problem instance s , we sequentially solve its subproblem i ∈ { 1 , . . . , N } , each associated with weight λ i . Let π i = { π i 1 , . . . , π i T } denote the obtained solution 2 at step i , and let F i be the Pareto front yielded by solutions from subproblem 1 to i . In each step i , we select up to K top points f ( π ) ∈ F i -1 from the Pareto front at step i -1 with the ranking determined by the scalar objective g ( π | s, λ i ) with respect to the new given weight λ i . The corresponding scalar objective and the induced surrogate landscape ˜ F i -1 ⊆ F i -1 based on those selected solutions are treated as the policy network inputs (see Figure 2), so as to construct a new solution π i and yield a new F i .

We would like to note that our NHDE is generic, and can directly integrate the base model θ with the existing decomposition-based neural MOCO methods. We demonstrate this property by applying it to two state-of-the-art methods, PMOCO [14] and MDRL [15], denoted as NHDE-P and NHDE-M, respectively. Given λ i and w i as inputs, NHDE-P uses a hypernetwork to generate the decoder parameters of the model θ ( λ i , w i ) , while NHDE-M fine-tunes the pre-trained meta model θ meta with a few steps to address the corresponding subproblem. More details are presented in Appendix B.

The construction of the solution π i with length T for each subproblem i can be cast as a Markov decision process. In particular, 1) the state includes the weight λ i , user-defined diversity factor w i ∈ R 2 satisfying w i 1 , w i 2 ≥ 0 and w i 1 + w i 2 = 1 , partial solution π i 1: t -1 , instance s , and ˜ F i -1 r , where ˜ F i -1 r = ˜ F i -1 ∪ { r } incorporates the aforementioned surrogate landscape at step i and the given reference point r ; 2) the action is to add a node π i t into π i 1: t -1 ; 3) the state transition transforms π i 1: t -1 to π i 1: t , denoted as π i 1: t = { π i 1: t -1 , π i t } ; 4) the reward is defined as R i = -w i 1 × g ( π i | s, λ i )+ w i 2 × HV r ( ˜ F i -1 ∪{ f ( π i ) } ) , where we introduce the hypervolume HV r ( ˜ F i -1 ∪ { f ( π i ) } ) to guide the search; and 5) the stochastic policy generating the solution π i is expressed as P ( π i | s, ˜ F i -1 r , λ i , w i ) = ∏ T t =1 P θ ( π i t | π i 1: t -1 , s, ˜ F i -1 r , λ i , w i ) , with the probability of node selection P θ ( π i t | π i 1: t -1 , s, ˜ F i -1 r , λ i , w i ) parameterized by a deep model θ .

4.2 Heterogeneous graph attention

To effectively solve subproblem i , the model should jointly capture the representations of both the instance's node graph and the Pareto front's point graph. Based on the encoder-decoder structure, we thus design a heterogeneous graph attention (HGA) model to correlate the two heterogeneous graphs as depicted in Figure 2. In the following section, we omit superscript i for better readability.

Encoder. Given an instance graph s containing n nodes with Z -dimensional features and a Pareto front graph ˜ F i -1 r containing k points ( k ≤ K +1 ) with M -dimensional features, their initial embeddings h (0) 1 , . . . , h (0) n ∈ R d and g (0) 1 , . . . , g (0) k ∈ R d are yielded by a linear projection with trainable parameters W h and W g , respectively, and d is empirically set to 128. The eventual embeddings h ( L ) 1 , . . . , h ( L ) n and g ( L ) 1 , . . . , g ( L ) k are derived by further passing through L = 6 attention layers.

Linear Projection

×

L

Layers

×

T

Steps

HGA

Add & Norm

Feed Forward

Node Features

ℎ 1

(𝑙)

...

ℎ 𝑛

(𝑙)

෠

ℎ 1

(𝑙)

...

෠

ℎ 𝑛

(𝑙)

Add & Norm

ℎ 1

(𝐿)

...

ℎ 𝑛

(𝐿)

Compatibility

Softmax

𝑝1

𝑡

...

𝑝𝑛

𝑡

𝜋𝑡

ℎ𝑐

Decoder

Encoder

Linear Projection

𝑔 1

(𝑙)

...

𝑔 𝑘

(𝑙)

ො

𝑔 1

(𝑙)

...

ො

𝑔 𝑘

(𝑙)

𝑔 1

(𝐿)

...

𝑔 𝑘

(𝐿)

Point Features

HGA

ℎ 1

(𝑙)

...

ℎ 𝑛

(𝑙)

𝑔 1

(𝑙)

...

𝑔 𝑘

(𝑙)

Q

K

V

V

K

Q

P2N Attn

N2P Attn

N2N Attn

Add

෨

ℎ 1

(𝑙)

...

෨

ℎ 𝑛

(𝑙)

෤

𝑔 1

(𝑙)

...

෤

𝑔 𝑘

(𝑙)

Pareto Front Graph

Instance Graph

Heterogeneous Graph Attention

𝝀

Surrogate Landscape

Each attention layer is composed of a multi-head HGA layer with Y = 8 heads and a fully connected feed-forward sublayer. For layer l ∈ { 1 , . . . , L } , HGA computes the representations ˜ h ( l ) u and ˜ g ( l ) u across the heterogeneous graphs, which are used to update the embeddings h ( l ) u and g ( l ) u . Skip-connection [49] and batch normalization [50] are both adopted in each sublayer, as follows,

Decoder. The decoder, composed of a multi-head HGA layer and a compatibility layer, autoregressively constructs a solution according to the probability distribution with T steps. At decoding step t ∈ { 1 , ..., T } , the glimpse q c of the context embedding h c (see Appendix C) is computed by the HGA layer. Then, the compatibility α is calculated as follows,

where C is set to 10 [27]. Finally, softmax is employed to calculate the selection probability distribution P θ ( π | s, ˜ F i -1 r , λ i , w i ) for nodes, i.e., P θ ( π t | π 1: t -1 , s, ˜ F i -1 r , λ i , w i ) = Softmax ( α ) .

HGA. The HGA layer in the encoder captures three key relations between the node graph and the point graph. The first, node-to-node α hh uv , indicates each node's attention towards others within the same instance to construct a promising solution. The second, node-to-point α hg uv , suggests each node's attention to points , guiding the constructed solutions distinct from the existing ones in the current Pareto front. The third, point-to-node α gh uv , indicates each point 's attention to nodes, facilitating the learning of the mapping from a solution to its objective values. We disregard the less meaningful point-to-point attention. Concretely, Eq. (6) defines the above three attention scores, which are separately normalized as ˜ α hh uv , ˜ α hg uv , and ˜ α hg uv by softmax. Then, ˜ h u and ˜ g u are computed by Eq. (7).

Finally, as for the multi-head attention, ˜ h u and ˜ g u are further computed as follows,

where ˜ h u,y and ˜ g u,y for head y ∈ { 1 , ..., Y } are obtained according to Eq. (7). In the multihead HGA, W Q h , W K h , W V h , W O h , W Q g , W g K , W V g , and W O g are independent trainable parameters. Similarly, in the decoder, the glimpse q c is calculated by the context embedding h c with the addition of the context-to-node and context-to-point attention, i.e., replacing h u with h c in the Eq. (6-8).

Algorithm 1 Training algorithm of NHDE-P

4.3 Multiple Pareto optima strategy

Contrary to single-objective problems that focus on a single optimal solution, MOCO problems involve a series of Pareto-optimal solutions. In light of this, we introduce a multiple Pareto optima (MPO) strategy to uncover multiple solutions for each subproblem by leveraging the Pareto optimality.

When solving subproblem i , more than one solution can be attained by sampling, e.g., sampling with multiple start nodes as did in POMO [36]. In this case, F i = MPO( F i -1 ∪ G i ) , where G i contains all the candidate points (find by sampling) to be introduced in the new Pareto front and MPO( · ) is an operator that updates the Pareto front. However, as the complexity of MPO( F i -1 ∪ G i ) is O (( |F i -1 | + |G i | ) |G i | ) , it may take a relatively long time, especially when there are thousands of points in F i -1 and G i . Thus, we suggest an efficient update mechanism executed on the surrogate Pareto fronts, F i = MPO( ˜ F i -1 ∪ ˜ G i ) , where ˜ G i ⊂ G i includes at most J (usually setting J > K ) best points selected from f ( π ) ∈ G i according to g ( π | s, λ i ) . The complexity of MPO( ˜ F i -1 ∪ ˜ G i ) is then reduced to O (( K + J ) J ) , which is able to curtail the overall solving time in practice.

4.4 Training and inference

Our NHDE can be applied to different decomposition-based DRL methods, e.g., PMOCO [14] and MDRL [15], and the training algorithm is easy to adapt with slight adjustments, where the one for NHDE-P is presented in Algorithm 1. The key points include three aspects. (1) Multiple weights are sampled to train the same instance (Line 6), since the solving processes for those subproblems are dependent. (2) The HV indicator is adopted in the reward (Line 10). (3) Multiple Pareto-optimal solutions are preserved via MPO (Line 15). Note that when training with a batch, the point sets with different sizes of the instances are padded with repetitive reference points, which are masked in the attention, based on the maximum size. The training algorithm of NHDE-M is given in Appendix E.

In the inference phase, for N given weights and diversity factors, the well-trained model is used to sequentially solve N corresponding subproblems, as shown in Figure 1. Moreover, instance augmentation [14] can be also brought into our MPO for each subproblem, i.e., the points of all sampled solutions from an instance and its augmented instances are included in G i . Our NHDE can achieve desirable performance by using only parts of instance augmentation (see Appendix D), since it can already deliver more diverse solutions.

5 Experiments

Problems. We evaluate the proposed NHDE on three typical MOCO problems that are commonly studied in the neural MOCO literature [13-15], namely the multi-objective traveling salesman problem (MOTSP) [51], multi-objective capacitated vehicle routing problem (MOCVRP) [3], and multi-objective knapsack problem (MOKP) [52]. More specifically, we solve bi-objective TSP (Bi-TSP), tri-objective TSP (Tri-TSP), the bi-objective CVRP (Bi-CVRP) and the bi-objective KP (Bi-KP). For the M -objective TSP with n nodes, each node has M sets of 2-dimensional coordinates, where the m -th objective value of the solution is calculated with respect to the m -th coordinates. Bi-CVRP consists of n customer nodes and a depot node, with each node featured by a 2-dimensional coordinate and each customer node associated with a demand. Following the literature, we consider two conflicting objectives in Bi-CVRP, i.e., the total tour length and the makespan (that is the length of the longest route). Bi-KP is defined by n items, with each taking a weight and two separate values. The m -th objective is to maximize the sum of the m -th values but not exceed the capacity. Three sizes of these problems are considered, i.e., n =20/50/100 for MOTSP and MOCVRP, and n =50/100/200 for MOKP. The coordinates, demands, and values are uniformly sampled from [0 , 1] 2 , { 1 , . . . , 9 } , and [0 , 1] , respectively. The vehicle capacity is set to 30/40/50 for MOCVRP20/50/100. The knapsack capacity is set to 12.5/25/25 for MOKP50/100/200.

Hyperparameters. We train NHDE-P with 200 epochs, each containing 5,000 randomly generated instances. We use batch size B =64 and the Adam [53] optimizer with learning rate 10 -4 ( 10 -5 for MOKP) and weight decay 10 -6 . During training, N ′ =20 weights are sampled for each instance. During inference, we generate N =40 and N =210 uniformly distributed weights for M =2 and M =3 , respectively, which are then shuffled so as to counteract biases. The diversity factors linearly shift through the N subproblems from (1,0) to (0,1), which implies a gradual focus from achieving convergence (scalar objective) with a few solutions to ensuring comprehensive performance with a multitude of solutions. We set K =20 and J =200 . See Appendix F for the settings of NHDE-M.

Baselines. We compare NHDE with two classes of state-of-the-art methods. (1) The neural methods, including PMOCO [14], MDRL [15], and DRL-based multiobjective optimization algorithm ( DRLMOA ) [13], all with POMO as the backbone for single-objective CO subproblems. Our NHDE-P and NHDE-Meach train a unified model with the same gradient steps as PMOCO and MDRL, respectively, while DRL-MOA trains 101 (105) models for M = 2 ( M = 3 ) with more gradient steps, i.e., the first model with 200 epochs and the remaining models with 5-epoch per model via parameter transfer. (2) The non-learnable methods, including the state-of-the-art MOEA and strong heuristics. Particularly, PPLS/D-C [23] is a specialized MOEA for MOCO with local search techniques, including a 2opt heuristic for MOTSP and MOCVRP, and a greedy transformation heuristic [52] for MOKP, implemented in Python. In addition, LKH [54, 55] and dynamic programming (DP), are employed to solve the weighted-sum (WS) based subproblems for MOTSP and MOKP, denoted as WS-LKH and WS-DP , respectively. All the methods use WS scalarization for fair comparisons. All the methods are tested with an RTX 3090 GPU and an Intel Xeon 4216 CPU. Our code is publicly available 3 .

Metrics. We use hypervolume (HV) and the number of non-dominated solutions ( | NDS | ). A higher HV means better overall performance in terms of convergence and diversity, while | NDS | reflects the diversity when HVs are close. The average HV, gaps with respect to NHDE, and total running time for 200 random test instances are reported. The best (second-best) and its statistically insignificant results at 1% significance level of a Wilcoxon rank-sum test are highlighted in bold (underline).

5.1 Main results

All results of NHDE-P and the baselines are displayed in Table 1. Given the same number of weights (wt.), NHDE-P significantly surpasses PMOCO for all problems and sizes in terms of HV and | NDS | , which indicates that NHDE-P has the potential to discover diverse and high-quality solutions. When instance augmentation (aug.) is equipped, NHDE-P achieves the smallest gap among the methods in most cases, except Bi-TSP100 and Bi-CVRP100 where WS-LKH and DRL-MOA perform better. However, WS-LKH consumes much longer runtime than NHDE-P due to iterative search (2.7 hours vs 5.6 minutes), and DRL-MOA costs much more training overhead to prepare multiple models for respective weights. Besides, another reason why NHDE-P is inferior to DRL-MOA on Bi-CVRP100 might be that the hypernetwork (inherited from PMOCO) could be hard to cope with the objectives

with imbalanced scales. Considering this drawback of PMOCO, we also apply our method to MDRL, i.e., NHDE-M, and demonstrate that NHDE-M outperforms DRL-MOA on Bi-CVRP100 in Table 2. Also, NHDE-M outperforms MDRL in all cases. More results of NHDE-M are given in Appendix G.

Regarding the inference efficiency, NHDE-P generally takes (tolerably) more runtime than the other learning based methods. To be fair, we further enhance the state-of-the-art PMOCO by adjusting the number of the weights, so that it takes similar or even more runtime compared with NHDE-P. The results in the last two lines for each problem shows that NHDE-P still attains the smaller gaps. We also observe that | NDS | hardly grows along with the increase of number of weights for PMOCO, since numerous solutions to different subproblems could be repetitive. In contrast, NHDE-P is able to produce more diverse solutions with much fewer weights.

5.2 Generalization study

To assess the generalization capability of NHDE-P, we compare the trained models (from neural methods) for Bi-TSP100 and the other baselines on 200 random Bi-TSP instances with larger sizes, i.e., Bi-TSP150/200. Three commonly used benchmark instances developed from TSPLIB [56],

i.e., KroAB100, KroAB150, and KroAB200, are also tested. The comparison results and Pareto fronts are demonstrated in Table 3 and Figure 3, respectively. As shown, NHDE-P outperforms the state-of-the-art MOEA (i.e., PPLS/D-C) and other neural methods significantly, in terms of HV and | NDS | , which means a superior generalization capability. The figure again verifies that NHDE-P generates a more extended Pareto front than PMOCO, showing a better diversity. Although PPLS/D-C finds a large number of solutions, they are inferior in terms of the optimality, with a biased distribution (e.g., crowded in certain regions). In contrast, NHDE-P generates a more well-distributed Pareto front with stronger convergence. More results on generalization are given in Appendix H.

10

20

30

40

50

f

1

10

20

30

40

50

f

2

NHDE-P (40 wt. aug.)

PMOCO (40 wt. aug.)

PPLS/D-C

WS-LKH (40 wt.)

14

19

18

23

(a)

20

40

60

80

f

1

10

20

30

40

50

60

70

80

f

2

NHDE-P (40 wt. aug.)

PMOCO (40 wt. aug.)

PPLS/D-C

WS-LKH (40 wt.)

27

35

17

25

(b)

20

40

60

80

100

f

1

20

40

60

80

100

f

2

NHDE-P (40 wt. aug.)

PMOCO (40 wt. aug.)

PPLS/D-C

WS-LKH (40 wt.)

36

46

18

28

(c)



















































(a)

 

























(b)

5.3 Ablation Study

To analyze the effect of the indicator-enhanced DRL, we compare NHDE-P with decomposition-based DRL without indicator (NHDE w/o I) and indicator-based DRL without decomposition (NHDE w/o D). To verify the valid design of HGA, three attention-based variants, which are formed by removing point-to-node attention (NHDE w/o P2N), removing node-to-point attention (NHDE w/o N2P), and adding point-to-point attention (NHDE w P2P), are involved for comparison. To assess the impact of MPO, NHDE w/o MPO is also evaluated. More details about these variants are presented in Appendix I. As seen from Figure 4(a). The performance of NHDE-P is significantly impaired when any of the components is ablated. Instead, NHDE w P2P degrades a bit, which reveals that the extra point-to-point attention may bring noises into the model. Moreover, we evaluate the effectiveness of the efficient update of MPO and partial instance augmentation in Figure 4(b), which shows that either of them saliently diminishes solving time with only a little sacrifice of the performance.

6 Conclusion

This paper proposes a novel NHDE for MOCO problems. NHDE impedes repetitive solutions from different subproblems via indicator-enhanced DRL with a HGA model, and digs more solutions in the neighborhood of each subproblem with an MPO strategy. Our generic NHDE can be deployed to different neural MOCO methods. The experimental results on three classic MOCO problems showed the superiority of NHDE, especially with regard to the diversity of Pareto set. A limitation is that the HV calculation would expend additional computational time, which might hinder the scalability of NHDE for solving much larger problems with many objectives. In the future, we will explore alternative schemes like the HV approximations [57, 58] to further promote the training efficiency of NHDE, and we also intend to apply it to tackle real-world MOCO problems.

Acknowledgments and disclosure of funding

This work is supported by the National Natural Science Foundation of China (62072483), and the Guangdong Basic and Applied Basic Research Foundation (2022A1515011690, 2021A1515012298).

References

Naveen Saini and Sriparna Saha. Multi-objective optimization techniques: a survey of the state-of-the-art and applications: Multi-objective optimization techniques. The European Physical Journal Special Topics , 230(10):2319-2335, 2021.

Qi Liu, Xiaofeng Li, Haitao Liu, and Zhaoxia Guo. Multi-objective metaheuristics for discrete optimization problems: A review of the state-of-the-art. Applied Soft Computing , 93:106382, 2020.

Sandra Zajac and Sandra Huber. Objectives and methods in multi-objective routing problems: a survey and classification scheme. European Journal of Operational Research , 290(1):1-25, 2021.

Alper Türkyılmaz, Özlem ¸ Senvar, ˙ Irem Ünal, and Serol Bulkan. A research survey: heuristic approaches for solving multi objective flexible job shop problems. Journal of Intelligent Manufacturing , 31:1949-1983, 2020.

Yunfei Cui, Zhiqiang Geng, Qunxiong Zhu, and Yongming Han. Multi-objective optimization methods and application in energy saving. Energy , 125:681-704, 2017.

Zesong Fei, Bin Li, Shaoshi Yang, Chengwen Xing, Hongbin Chen, and Lajos Hanzo. A survey of multi-objective optimization in wireless sensor networks: Metrics, algorithms, and open problems. IEEE Communications Surveys & Tutorials , 19(1):550-586, 2017.

Matthias Ehrgott, Xavier Gandibleux, and Anthony Przybylski. Exact methods for multi-objective combinatorial optimisation. Multiple criteria decision analysis: State of the art surveys , pages 817-850, 2016.

Arne Herzel, Stefan Ruzika, and Clemens Thielen. Approximation methods for multiobjective optimization problems: A survey. INFORMS Journal on Computing , 33(4):1284-1299, 2021.

Cong Zhang, Yaoxin Wu, Yining Ma, Wen Song, Zhang Le, Zhiguang Cao, and Jie Zhang. A review on learning to solve combinatorial optimisation problems in manufacturing. IET Collaborative Intelligent Manufacturing , 5(1):e12072, 2023.

Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research , 134:105400, 2021.

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

Yimo Yan, Andy HF Chow, Chin Pang Ho, Yong-Hong Kuo, Qihao Wu, and Chengshuo Ying. Reinforcement learning for logistics and supply chain management: Methodologies, state of the art, and future opportunities. Transportation Research Part E: Logistics and Transportation Review , 162:102712, 2022.

Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE Transactions on Cybernetics , 51(6):3103-3114, 2021.

Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinatorial optimization. In International Conference on Learning Representations , 2022.

Zizhen Zhang, Zhiyuan Wu, Hang Zhang, and Jiahai Wang. Meta-learning-based deep reinforcement learning for multiobjective optimization problems. IEEE Transactions on Neural Networks and Learning Systems , 34(10):7978-7991, 2023.

Yaoxin Wu, Wen Song, Zhiguang Cao, Jie Zhang, Abhishek Gupta, and Mingyan Lin. Graph learning assisted multi-objective integer programming. In Advances in Neural Information Processing Systems , 2022.

David Bergman, Merve Bodur, Carlos Cardonha, and Andre A Cire. Network models for multiobjective discrete optimization. INFORMS Journal on Computing , 34(2):990-1005, 2022.

Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation , 6(2):182-197, 2002.

Qingfu Zhang and Hui Li. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation , 11(6):712-731, 2007.

Nicola Beume, Boris Naujoks, and Michael Emmerich. Sms-emoa: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research , 181(3):1653-1669, 2007.

Andrzej Jaszkiewicz. Genetic local search for multi-objective combinatorial optimization. European Journal of Operational Research , 137(1):50-71, 2002.

Jialong Shi, Qingfu Zhang, and Jianyong Sun. PPLS/D: Parallel pareto local search based on decomposition. IEEE transactions on cybernetics , 50(3):1060-1071, 2020.

Jialong Shi, Jianyong Sun, Qingfu Zhang, Haotian Zhang, and Ye Fan. Improving pareto local search using cooperative parallelism strategies for multiobjective combinatorial optimization. IEEE Transactions on Cybernetics , 2022.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems , 2015.

Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. In International Conference on Learning Representations , 2017.

Mohammadreza Nazari, Afshin Oroojlooy, Martin Takáˇ c, and Lawrence V. Snyder. Reinforcement learning for solving the vehicle routing problem. In Advances in Neural Information Processing Systems , 2018.

Wouter Kool, Herke Van Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference on Learning Representations , 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Multi-decoder attention model with embedding glimpse for solving vehicle routing problems. In AAAI Conference on Artificial Intelligence , pages 12042-12049, 2021.

Yeong-Dae Kwon, Jinho Choo, Iljoo Yoon, Minah Park, Duwon Park, and Youngjune Gwon. Matrix encoding networks for neural combinatorial optimization. In Advances in Neural Information Processing Systems , 2021.

Minsu Kim, Junyoung Park, and Jinkyoo Park. Sym-NCO: Leveraging symmetricity for neural combinatorial optimization. In Advances in Neural Information Processing Systems , 2022.

Jieyi Bi, Yining Ma, Jiahai Wang, Zhiguang Cao, Jinbiao Chen, Yuan Sun, and Yeow Meng Chee. Learning generalizable models for vehicle routing problems via knowledge distillation. In Advances in Neural Information Processing Systems , 2022.

Jinbiao Chen, Huanhuan Huang, Zizhen Zhang, and Jiahai Wang. Deep reinforcement learning with twostage training strategy for practical electric vehicle routing problem with time windows. In International Conference on Parallel Problem Solving from Nature , 2022.

Zizhen Zhang, Hong Liu, MengChu Zhou, and Jiahai Wang. Solving dynamic traveling salesman problems with deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems , 34(4): 2119-2132, 2023.

Jianan Zhou, Yaoxin Wu, Wen Song, Zhiguang Cao, and Jie Zhang. Towards omni-generalizable neural methods for vehicle routing problems. In the 40th International Conference on Machine Learning , 2023.

Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. POMO: Policy optimization with multiple optima for reinforcement learning. In Advances in Neural Information Processing Systems , 2020.

Xinyun Chen and Yuandong Tian. Learning to perform local rewriting for combinatorial optimization. In Advances in Neural Information Processing Systems , 2019.

Hao Lu, Xingwen Zhang, and Shuang Yang. A learning-based iterative method for solving vehicle routing problems. In International Conference on Learning Representations , 2020.

Yaoxin Wu, Wen Song, Zhiguang Cao, Jie Zhang, and Andrew Lim. Learning improvement heuristics for solving routing problems. IEEE Transactions on Neural Networks and Learning Systems , 33(9):5057-5069, 2022.

Yining Ma, Jingwen Li, Zhiguang Cao, Wen Song, Le Zhang, Zhenghua Chen, and Jing Tang. Learning to iteratively solve routing problems with dual-aspect collaborative transformer. In Advances in Neural Information Processing Systems , 2021.

Yining Ma, Jingwen Li, Zhiguang Cao, Wen Song, Hongliang Guo, Yuejiao Gong, and Yeow Meng Chee. Efficient neural neighborhood search for pickup and delivery problems. In International Joint Conferences on Artificial Intelligence , 2022.

Xi Lin, Zhiyuan Yang, Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multiobjective optimization. In Advances in Neural Information Processing Systems , 2022.

Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypernetworks. In International Conference on Learning Representations , 2021.

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In Advances in neural information processing systems , 2019.

Hong Wu, Jiahai Wang, and Zizhen Zhang. MODRL/D-AM: Multiobjective deep reinforcement learning algorithm using decomposition and attention model for multiobjective optimization. In ISICA 2019 , volume CCIS 1205, pages 575-589, 2020.

Yinan Shao, Jerry Chun-Wei Lin, Gautam Srivastava, Dongdong Guo, Hongchun Zhang, Hu Yi, and Alireza Jolfaei. Multi-objective neural evolutionary algorithm for combinatorial optimization problems. IEEE Transactions on Neural Networks and Learning Systems , 34(4):2133-2143, 2023.

Yongxin Zhang, Jiahai Wang, Zizhen Zhang, and Yalan Zhou. MODRL/D-EL: Multiobjective deep reinforcement learning with evolutionary learning for multiobjective optimization. In International Joint Conference on Neural Networks , 2021.

Charles Audet, Jean Bigeon, Dominique Cartier, Sébastien Le Digabel, and Ludovic Salomon. Performance indicators in multiobjective optimization. European Journal of Operational Research , 292(2):397-422, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , volume 37, pages 448-456, 2015.

Thibaut Lust and Jacques Teghem. The multiobjective traveling salesman problem: A survey and a new approach. In Advances in Multi-Objective Nature Inspired Computing , pages 119-141, 2010.

Hisao Ishibuchi, Naoya Akedo, and Yusuke Nojima. Behavior of multiobjective evolutionary algorithms on many-objective knapsack problems. IEEE Transactions on Evolutionary Computation , 19(2):264-283, 2015.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Keld Helsgaun. An effective implementation of the lin-kernighan traveling salesman heuristic. European Journal of Operational Research , 126(1):106-130, 2000.

Renato Tinós, Keld Helsgaun, and Darrell Whitley. Efficient recombination in the lin-kernighan-helsgaun traveling salesman heuristic. In International Conference on Parallel Problem Solving from Nature , pages 95-107, 2018.

Gerhard Reinelt. TSPLIB-a traveling salesman problem library. ORSA Journal on Computing , 3(4): 376-384, 1991.

Jim Boelrijk, Bernd Ensing, and Patrick Forré. Multi-objective optimization via equivariant deep hypervolume approximation. In International Conference on Learning Representations , 2023.

Ke Shang, Weiyu Chen, Weiduo Liao, and Hisao Ishibuchi. Hv-net: Hypervolume approximation based on deepsets. IEEE Transactions on Evolutionary Computation , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel neural heuristic approach, called NHDE, for Multi-Objective Combinatorial Optimization (MOCO) problems. The proposed approach enhances the diversity of Pareto solutions by introducing a multiple Pareto optima (MPO) strategy and an indicator-enhanced Deep Reinforcement Learning (DRL) method. Experimental results on classic MOCO problems demonstrate that NHDE outperforms state-of-the-art neural baselines, achieving superior overall performance with a higher diversity of Pareto solutions.\n\nSTRENGTHS:\n1.\tThe paper proposes a novel algorithm to generate diverse solutions for combinatorial optimization (CO) problems.\n2.\tThe proposed method is able to enhance existing methods to achieve improvements on diversity metrics including hypervolumes.\n\nWEAKNESSES:\n1.\tThe solving time of proposed solver is relatively slow compared to other RL-based CO solvers. Besides, one of the curious results is that the solving time of LKH is much slower than the time in previous papers. (see question 1)\n2.\tThe authors do not discuss the scope of solvers of proposed diversity enhancement (see question 2).\n\nQUESTIONS:\n1.\tAs far as I know, LKH is a pretty fast heuristic solver for TSP problem. However, the reported running time of LKH in experiments of TSP is extremely slow and much slower than that reported in previous papers (e.g. [2]). Could authors explain this?\n2.\tFrom my understanding, the proposed method is able to conduct two tasks at a same time: (1) solve the CO problem (2) generate diverse solutions. The supported CO problems includes TSP, VRP. What is the scope of CO problems that the proposed diverse enhancement is able to handle? Does the diverse enhancement able to enhance all RL-based solvers? What about other neural solvers that do not use RL (e.g. [3])?\n3.\tHow does other recent neural solvers perform on solution diversity? (e,g. [4,5])\n4. in Line 118, when does the sub-problem required, for the CO solving itself, or diversity enhancement for multiple evaluation metrics? (Why does there N sub-problems instead of M?)\n\n[1] Kwon Y D, Choo J, Kim B, et al. Pomo: Policy optimization with multiple optima for reinforcement learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 21188-21198.\n\n[2] Xin L, Song W, Cao Z, et al. NeuroLKH: Combining deep learning model with Lin-Kernighan-Helsgaun heuristic for solving the traveling salesman problem[J]. Advances in Neural Information Processing Systems, 2021, 34: 7472-7483.\n\n[3] Wang R, Yan J, Yang X. Combinatorial learning of robust deep graph matching: an embedding based approach[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.\n\n[4] Kwon Y D, Choo J, Kim B, et al. Pomo: Policy optimization with multiple optima for reinforcement learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 21188-21198.\n\n[5] Qiu R, Sun Z, Yang Y. DIMES: A Differentiable Meta Solver for Combinatorial Optimization Problems[C]//Advances in Neural Information Processing Systems.\n\nLIMITATIONS:\nIt could be better if the authors could discuss potential limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a number of techniques to improve the diversity of solutions in neural multi-objective combinatorial optimization (MOCO): 1) solving preference-conditioned subproblems dependently by taking a subset of the current Pareto front as input to a graph attention-based policy network; 2) using the hypervolume indicator as the reward; and 3) sampling multiple solutions from the policy for each subproblem to update the Pareto front. Previous decomposition-based MOCO methods are augmented with these techniques, and empirically evaluated on multi-objective TSP, CVRP, and KP benchmark problems. Results show that there is improvement in average hypervolume compared to previous work, and ablations show that each design choice has positive contribution to average performance.\n\nSTRENGTHS:\nOriginality: This paper's ideas of using a reward based on the hypervolume and using the current Pareto front as part of the state input to the policy are novel (to my knowledge). Doing so ensures a stationary environment for RL, in contrast to previous methods that use distance-based rewards without observing the global solution space that leads to nonstationarity for RL.\n\nQuality: The paper covers most of the important details in regards to problem motivation, background and related work, description of methods and analysis of results.\n\nClarity: The technical writing and notation is clear and concise.\n\nSignificance: The topic of multi-objective CO is an important one that arises in practice and there is increasing work on learning-based approaches, so this is a relevant and significant paper for the conference.\n\nWEAKNESSES:\nThe abstract claims that the proposed techniques are intended to \"hinder duplicated solutions for different subproblems.\" Even though the results show an increase in average diversity, as measured by the hypervolume, the paper does not provide concrete and direct evidence of this specific claim of reducing number of duplicates.\n\nThe method relies on at least three hyperparameters---$N$, the number of subproblems and preferences used in training; $K$, the number of top solutions from the current Pareto front to use as input to the policy; and $J$, the number of new solutions to be sampled for updating the Pareto front. The main paper does not provide any sensitivity study or guidance on how to choose them in practice, so it is unclear whether the proposed techniques are robust or brittle.\n\nThe results in Table 1, Table 2, and Figure 4 only report the average performance, but do not show the variance of results. It is not possible to determine whether the results are statistically significant.\n\nThe proposed techniques rely on a base method, which has some specific limitations:\n1. The overall approach relies on the base method of sequentially solving subproblems, each determined by a preference. It is not clear how the ordering of subproblems impacts the performance of the method.\n2. The policy is a graph attention network, so it requires the CO problem instances to be defined on a graph, so it's not clear the method generalizes to other CO problems where there is no natural graph structure.\n\nIn Section 4.1, where the paper defines the state space, it is not clear why the preferences are not part of the state, if the policy is expected to act differently for different subproblems with different preferences.\n\nOther minor issues to be fixed or clarified:\n- Lines 32-33 and the claim on lines 35-36 need citations.\n- There are some typographical and grammatical issues: e.g.\n  - line 19-20: \"..., practical yet more complex\"\n  - line 233 is an incomplete sentence\n  - line 257 \"detials\"\n- In line 119, the subscripts $1,\\dotsc,T$ appear without explanation. Readers have to guess that it denotes the sequence representation of the solution.\n- From line 147, one can guess that each solution is represented by its objective values as an $M$-dimensional vector, but this should be explicitly stated.\n- In the results, the best $|\\text{NDS}|$ should also be in bold.\n\nQUESTIONS:\nThe authors should:\n- show that results are statistically significant\n- provide evidence for the claim that the techniques reduce the number of duplicate solutions to subproblems\n- explain how to choose the three hyperparameters and show sensitivity results\n\nLIMITATIONS:\nThe use of a graph attention model limits the method to problems with a graph.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes NHDE, a novel neural heuristic with diversity enhancement method, to solve multi-objective combinatorial optimization (MOCO) problems. The main contribution of NHDE is to overcome the crucial duplicated solution issue for the widely-used decomposition-based method, and generate a set of diverse Pareto solutions in a reasonable runtime. The proposed method has two major components: 1) an indicator-enhanced model structure with heterogeneous graph attention (HGA) to generate diverse Pareto solutions, and 2) a simple yet powerful multiple Pareto optima (MPO) strategy to identify more Pareto solutions for each decomposed subproblems.\n\nThe proposed NHDE method is flexible and can be incorporated into different neural MOCO approaches. Experimental results show that NHDE can achieve promising performances on various MOCO problems, such as multi-objective traveling salesman, capacitated vehicle routing, and knapsack problems.\n\nSTRENGTHS:\n+ This work is generally well-written and easy to follow.\n\n+ Multi-objective combinatorial optimization (MOCO) is important for many real-world applications, and the decomposition-based method is a widely-used approach for MOCO. It is promising to see the proposed NHDE can efficiently tackle the crucial duplicated solution issue for the decomposition-based neural method.\n\n+ NHDE can be flexibly incorporated into different decomposition-based methods, and it obtains promising performance on various MOCO problems.\n\nWEAKNESSES:\nI have the following concerns for the proposed method: \n\n**1. Solution Distribution**\n\nNHDE is a promising approach to tackle the duplicated solution issue for the decomposition-based method. Another important issue for the decomposition method is the shape of generated Pareto front, where the solution distribution will be very sensitive to the scalarization method, especially for problems with more than two objectives. In real-world applications, it is often hard to choose the proper scalarization method for a new problem before optimization. One concrete example can be found in Appendix D.5 in the PMOCO[1] for neural MOCO. Can NHDE also (partially) tackle this issue? It is interesting to know the shape of generated Pareto front by NHDE on the tri-TSP with different scalarization methods. \n\n[1] Pareto Set Learning for Neural Multi-Objective Combinatorial Optimization, ICLR 2022.\n\n**2. Batch Inference and Subproblem Order**\n\nIt seems that NHDE needs to sequentially generate the solutions for different subproblems one by one (even for a single instance) since it takes the current Pareto front as input, while PMOCO and MDRL can generate solutions in batch. Is it possible to design a batch version for NHDE? Will the order of subproblems significantly affect NHDE's performance, especially for problems with more than two objectives?\n\n**3. Runtime Analysis**\n\nNHDE requires longer runtime than PMOCO and MDRL for the same number of preferences. It is interesting to know NHDE's performance with a limited runtime budget (e.g., a few seconds) similar to PMOCO. What is the runtime cost of the indicator-enhanced inference and MPO in NHDE separately?\n\n**4. Effect of MPO**\n\nThe ablation study in Figure 4 is very helpful, and I believe it is for Bi-TSP50. According to the results, it seems that the performance of NHDE w/o MPO (0.6335) is similar to the PMOCO (0.6331) but requires a longer runtime. Does the promising performance of NHDE mainly come from MPO but not the indicator-enhanced inference? What is the performance of PMOCO(600 pref.) with MPO?\n\nQUESTIONS:\n- Please address the concerns raised in Weaknesses.\n\n- It is better to also report the results of NHDE + MDRL (NHDE-M) in the main paper.\n\nLIMITATIONS:\nThe limitation of the proposed method is discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper focus on solving multi-objective combinatorial optimization by neural methods. The authors observe that decomposition-based neural MOCO methods may acquire similar solutions for neighboring subproblems. Therefore, to enhance the diversity of the non-dominated solution set, they introduce the hypervolume indicator to reward design and propose a heterogeneous graph attention mechanism that learns the interaction between the Pareto Front and the nodes of the instance. Besides, a multiple optimal strategy is proposed to sample and preserve more solutions. Experimental results show that the proposed NHDE generates more diverse solutions and achieves better hypervolume.

STRENGTHS:
1. Hypervolume Indicator is firstly utilized to learn neural MOCO methods and this idea would inspire researchers to develop novel MOCO solvers beyond existing pure decomposition-based methods.
2. The diversity of the non-dominated solution set is a key point of MOCO problems. It is quite meaningful to enhance diversity.
3. Sufficient experiments are conducted, and the proposed method achieves better convergence and diversity.

WEAKNESSES:
The main concern is about the design of HGA. See the questions below.

QUESTIONS:
In lines 164-167, the authors claim that HGA can guide the constructed solutions distinct from existing ones. However, the proposed HGA learns the interaction between the node features (which represent the instance) and the points of Pareto Front (which are objective values), which could not represent the structure of any solutions. Could you explain why HGA can guide the diverse solution construction without explicitly considering the structure of the existing and current solutions?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08