Title: Pareto Frontier Approximation Network (PA-Net) Applied to Multi-objective TSP
Authors: Ishaan Mehta
Sajad Saeedi
Venue: ICLR 2022
Keywords: Robotics
planning
TSP
RL
Multi Objective Optimization
Pareto Optimality

Abstract: Multi-objective optimization is used in various areas of robotics like control, planning etc. Their solutions are dependent on multiple objective functions, which can be conflicting in nature. In such cases, the optimality is defined in terms of Pareto optimality. A set of these Pareto Optimal solutions in the objective space form a Pareto front (or frontier). Each solution has its own trade off. For instance, the travelling salesman problem (TSP) is used in robotics for task/resource allocation. Often this allocation is influenced by multiple objective functions and is solved using Multi-objective travelling salesman problem (MOTSP). In this work, we present PA-Net, a network that generates good approximations of the Pareto front for the multi-objective optimization problems. Our training framework is applicable to other multi-objective optimization problems; however, in this work, we focus on solving MOTSP.  Firstly, MOTSP is converted into a constrained optimization problem. We then train our network to solve this constrained problem using the Lagrangian relaxation and policy gradient. With PA-Net we are able to generate better quality Pareto fronts with fast inference times as compared to other learning based and classical methods. Finally, we present the application of PA-Net to find optimal visiting order in coverage planning.


Review discussion:
# Pros and Cons
## Pros
- The targeting problem (MOO and the approximation of the PF) is of importance when discussing optimization problems in the real world.
- The proposed formulation (e.g., the surrogate optimization problems (up to $K$), training by the actor-critic approach) enables the user to estimate the Pareto front efficiently.
- The estimation performance outperforms the existing methods.
## Cons
- Evaluation problem class is limited (MOTSP with possibly artificial objective functions).
- I guess that other general methods (NSGA, MOEA/D) are developed as a general solver, but PA-Net based on the TSP-Net is a specialized approximator for the MOTSP.
- Some notations or explanations are missing or confusing to follow the detail of the paper.
# Comments
To clearly understand the paper and clarify its score, I would like to clarify the following point.
(I) Is the proposed approach (surrogate opt. as in Eq.(6)-(7)) general or problem-specific? As far as I read, this PA-Net consists of two ideas; (1) conversion to the set of surrogated CO problems for each preference vector and (2) a generalization of the TSP-Net for efficient learning. As far as I understand, we can solve other MO problems using the proposed approach with appropriate solvers.
This point is important to clarify because other counterparts (i.e., NSGA, MOEA/D) seem to be general approaches, but the PA-Net is based on the TSP-Net.
That is, it is a problem-specific approach for the MOTSP. In that sense, the performance of the PA-Net is possible to expect (e.g., Concorde is much faster than a general solver like Gurobi).
(II) The following points should be clarified to increase the readability of the paper.
(II-1): It is better to define the problem of 'approximating the Pareto fronts' in the paper.
As stated, a Pareto front is a set of points (in Sec.2.1; A set of all such points form a Pareto frontier).
So, we have many Pareto fronts (i.e., many sets of solutions), but we should select a better one in terms of something (e.g., HV).
So, a mathematical definition of this problem could clarify the purpose of the paper.
(II-2): How to set the hyperparameter $K$?
(II-3): Eq(16): the suffix $i$ of $f_i$ is confusing due to the sigma on $i$ at the second term.
(II-4): Table3: Why HVs of NSGA-II and MOEA/D only for 3-obj are much fewer than others for 200-City?
For example, in 5-obj, it was roughly 70% of the PA-5, but 3-obj showed apparent different behaviors of others only for 200-city. What happened?
(II-5): I cannot follow the problem details of the dimension (size). What do 120x4 and 120x6 mean? in Experiments (above Table 1, below Eq.(16))
(II-6): Notation $\\{100, 91\text{ and }, 40\\}$ is hard to interpret (above Table 1, below Eq.(16)).
(II-7): No definition of $\nabla_{\phi} D_\mathit{CR}(\phi)$ used in the critic update.
(II-8): The definition of $S_k$ and $A_k$ should be updated (modified) because $A_k = \\{\pi_k\in A_k \mid \dots\\}$ seems to be a circular definition (i.e.
when defining $A_k$, $A_k$ is used to write $\pi_k$). The definition of $S_k$ has the same problem.
(II-9): Minor bug fix or confusing parts for me:
- Eq.(12) $L(\pi,\vec{w}_k)$ -> $L_k(\pi,\vec{w}_k)$ or not? ($L$ without the suffix $k$ is not defined?)
- Eq.(13) phi -> $\phi$
NO.
Dear Reviewer,
Thank you for your comments. We are working on revising our manuscript based on your comments and will upload the new version in coming days.
In the meantime, please find some responses below:
- In our current work, the focus is on solving MOTSP. The core idea of PA-Net is to learn to approximately solve the problem in Eq. (6) for various preferences in the objective space.
So we believe that this methodology can be adopted to solve other multi-objective problems in a deep RL setting, where a network can be trained to generalize to various preferences in objective space.
Although each network would solve a specific problem;
however, it will  provide performance gains, as in our case.
-  We will revise the manuscript based on comment (II). Response to some sub-comments is given below:
- Regarding Anomaly in data in Table 3: It seems like there was an error in reporting the data for evolutionary algorithms in that table. Thank you for pointing out. We will update the Manuscript with correct data. The correct data will be updated in the manuscript
- Regarding parameter $K$ : Ideally, larger value of $K$ would lead to better generalization performance.
However, this also increases the cost of training of the network.
- Regarding details on dimension: PA-2 and PA-3 were trained on $120$ city MOTSP instances where each city has $4$ features.
Similarly PA-5 was trained on $120$ city MOTSP instances where each city has $6$ features. We will correct the language in our manuscript to avoid any confusion.
- Regarding Eq(12) and Eq(13): Yes, these are typos in equation which will be fixed.
**Strengths:**
+ The studied problem, MOTSP, is important and could be useful for many real-world applications. The learning-based approach for multi-objective combinatorial optimization is a timely research topic and could be interesting to many researchers in the community.
+ The proposed method can generate different approximate Pareto solutions with a single model. It also has a good overall performance and training times on different MOTSP instances, but also with many concerns.
**Weakness:**
*Method:*
**1. Problem Formulation:**
A key step in the proposed method is to convert the multi-objective optimization problem into different constrained single-objective optimization problems.
However, many conversion methods have already been proposed [1-3] which can tackle the non-convex Pareto frontier. None of them are discussed in this work. Is the proposed problem formulation new or adopted from other works?
What are its advantages (and disadvantages) over other methods?
What is J(\pi_k) in (7)?
**2. Finding the Whole Pareto Front:**
Theorem 1 is a crucial motivation of this work to find the whole Pareto front.
However, the theoretical property of Theorem 1 is relatively weak. It can only guarantee that the optimal solution of the constrained single-objective problem can dominate other solutions that satisfy the same constraint.
Therefore, the obtained solution is not guaranteed to be Pareto optimal (can be dominated by other feasible solutions out of the constrained set).
In addition, even assuming all the single-objective problems can be successfully optimized, there is also no guarantee that all Pareto solutions can be found by the proposed method.
It is also confusing why the obtained solutions can exactly lie on the unit vector in Figure 1. In my understanding, the proposed method can find a solution that satisfies the preference-based constraint while minimizing the L2 norm of all objectives. Since the problem has an inequality constraint, the obtained solution should be close to the unit preference vector but not guaranteed to be exactly on the unit vector.
**3. The Model Structure:**
To incorporate the preference into the TSP-Net model, the proposed method simply adds the preference embedding to each node (city) embedding. According to the ablation study in Appendix, the preference embedding network can be a simple single-layer FC. The rest model is the standard TSP-Net where the whole transformer-based encoder can be totally replaced by a single 1-D convolution layer. Why can this simple structure work so well for MOTSP? Will it significantly hurt the model performance of each individual preference compared to a single objective model?
More concerns on the results are listed below in the experiment part.
**4. Other Problems and Related Work:**
One important advantage of the learning-based method is its flexibility to solve different problems [4].
Although this work focuses on MOTSP, I believe it could have a larger impact by showing its ability to solve other problems.
Many works on deep multi-task learning and multi-objective RL have been cited and discussed multiple times in the related work section, while they are not the most related work to this paper. I think it is better to shorten this part into a single paragraph, and leave more space to discuss the related work on learning-based solvers (e.g., [4]) and other approaches for MOTSP.
*Experiments:*
**5. Competitive Baselines:**
According to the experimental results, the learning-based solvers are much better than the heuristic-based solvers.
However, for the single objective TSP, the SOTA heuristic-solver (e.g., Concorde) usually has the best performance. Since the obtained Pareto front is not highly non-convex (as in Figure 2), the results for linear scalarization + Concorde should be included for a better comparison.
**6. Unfair Comparison:**
The comparison to other learning-based solvers is questionable.
First of all, one contribution of DRL-MOA is the transfer learning based training method. The required training epoch (and time) is far less than those reported in this paper.
Hence the results reported in Table 1 are misleading. It seems that all the PA-Nets are trained on the 120-city instances for 2,3 and 5 objective MOTSP.
However, the original DRL-MOA is only trained on the 40-city MOTSP. As reported in this work, "The trained model of bi-objective TSP for DRL-MOA (Li et al.,2020) is used.", will it lead to unfair comparison?
The TSP-Net is more powerful than the Ptr-Net used in DRL-MOA, so it is expected that it can have better performance.
However, it only moderately outperforms DRL-MOA on the 2 and 5 objective problems, and has worse performance on the 3-obj problems. A more reasonable baseline is to use the TSP-Net in DRL-MOA to check whether the preference-based model could lead to worse performance.
To calculate the hypervolume, this work reports the results with {100,500,500} preferences for the 2, 3, and 5 objective MOTSP instances, while it only reports the results of {100, 91, 40} networks for DRL-MOA.
Although it is understandable that one advantage of PA-Net is to generate a dense approximation, the results of {91, 40} preferences should also be reported for a clear comparison. Since PA-Net (with 500 preferences) are already outperformed by DRL-MOA (with only 91 models) for the 3-objective MOTSP, will it be significantly outperformed by DRL-MOA with the same number of solutions?
**7. Missing Experiment Settings:**
How many training samples and epochs are used to train the PA-Net and DRL-MOA? How many test instances are used to measure the performance for different methods? If there is only one instance for each kind of problem, the results are not convincing. What is the run time (inference) for each method to solve the MOTSP instance with different numbers of objectives and cities?
The details for Hypervolume calculation (e.g., the reference points for different instances) are missing.
**8. Surprising Results on the Ablation Studies:**
It is quite surprising that the transformer-based encoder is not needed for the TSP-Net, and only a simple one-layer FC is needed for preference embedding. With this setting, the whole PA-Net should have a much smaller scale. What are the total numbers of parameters for these ablation settings (1, 2, and 3)? With the 1-D Conv encoder, the ablation 3 model should be even smaller than the Prt-Net. Why it still needs ~12 hrs to train the model? Why can this model still outperform the Ptr-Net model on the 2-objective MOTSP?
**Reference:**
[1] Das, Indraneel, and John E. Dennis. "Normal-boundary intersection: A new method for generating the Pareto surface in nonlinear multicriteria optimization problems." SIAM journal on optimization 8, no. 3: 631-657, 1998.
[2] Mavrotas, George. Effective implementation of the ε-constraint method in multi-objective mathematical programming problems. Applied mathematics and computation 213, no. 2: 455-465, 2009.
[3] Miettinen, Kaisa. Nonlinear multiobjective optimization. Vol. 12. Springer Science & Business Media, 2012.
[4] Kool, Wouter, Herke van Hoof, and Max Welling. Attention, Learn to Solve Routing Problems!. ICLR 2019.
NO.
Dear Reviewer,
Thank you for your comments. We are working on revising the manuscript based on your comments and running additional experiments, which will be uploaded in the coming days.
In the meantime, please find some responses below:
- Regarding Problem Formulation: Thank you for pointing out these works. It seems that our approach is very similar in spirit to [1] in the sense that both methods divide the objective space using unit vectors and find the solutions on Pareto fronts along the unit vectors.
However, there are subtle differences in formulation of the problem. Their method requires a global minimizer for each objective apriori to generate a convex hull and then unit vector is required for finding the Pareto points.
While our method only needs a unit vector to solve the constrained problem in Eq (6).
On the other hand, the $\phi$-constraint methods, as in [2], have different formulation. These methods optimize one of the objectives while using the other objectives as constraints. This method has been used to solve bi-objective combinatorial optimization problem in [3]. A major disadvantage of this approach is scalability with larger problems with more objectives. We will make sure to incorporate these relevant works in our updated manuscript.
- Regarding Finding the whole Pareto front: Theorem 1 is a crucial motivation for our work. It certainly only provides guarantee for dominant solutions in the constrained set given in Eq (7).
But the whole idea of PA-Net is to learn to solve the constrained problem in Eq (6) for various preferences in objective space. At the query time, a dense set of solution can be generated using the large number of preferences. An approximation of Pareto Front can be extracted from this dense set.
It is true that there is no guarantee of finding all the Pareto solutions, but such a guarantee is  hard to make, particularly in an NP-Hard problems like MOTSP.
- Regarding confusion in Fig 1: The preferences that we use are unit vectors.
Further, in our formulation, we assume that objective functions are strictly positive.
As a result, the equation (7) in a way becomes an equality constraint.
So with an ideal solver the optimal solution would lie on the preference vector.
However, in practice the solution is usually close to the unit vector.
- Regarding Model structure: Yes, it seems to be a possibility, that removing the transformer can lead to a relatively more efficient network. We are performing some additional experiments, and we will attempt to answer this in coming days.
- Regarding related works: We will edit the manuscript based on your suggestion.
- Regarding Competitive baselines: We are working on generating results with solvers like Concorde. We will report the results in coming days.
- Regarding training times: The training time for DRL-MOA is based on our experiments on training on V100. In their  [code](https://github.com/kevin031060/RL_TSP_4static) even with the parameter transfer strategy, each network is trained for a fixed number of epochs, batch size and training iteration, as a result for each preference the training time is same. We will attempt to train TSP-Net in DRL-MOA strategy and report back the results.
Further, we will also add test results on 91 and 100 preferences.
- Regarding ablation studies: We will add the details of the parameters. The training time of ablation 3 is still 12 hrs because the bottleneck in training comes from generating a batch of MOTSP tours for each preference.
Refrences:
[1] Das, Indraneel, and John E. Dennis. "Normal-boundary intersection: A new method for generating the Pareto surface in nonlinear multicriteria optimization problems." SIAM journal on optimization 8, no. 3: 631-657, 1998.
[2] Mavrotas, George. Effective implementation of the ε-constraint method in multi-objective mathematical programming problems. Applied mathematics and computation 213, no. 2: 455-465, 2009.
[3] Bérubé JF, Gendreau M, Potvin JY. An exact ϵ-constraint method for bi-objective combinatorial optimization problems: Application to the Traveling Salesman Problem with Profits. European journal of operational research. 2009 Apr 1;194(1):39-50.
The extension of neural combinatorial optimization towards multi-objective problems is timely and a relevant topic to bring the subfield forward.
However, the contribution of this paper towards the field seems to be small, being mostly the different formulation of the problem as a constrained optimization problem, while being otherwise based on the methodology from the literature.
At the same time, the focus is only on the TSP problem, which is one of the favorite problems in this subfield, but not very strongly constrained and not expressive enough by itself. The paper would benefit a lot from also considering other problems.
Regarding the experiments, I would be interested in a more precise evaluation of the effect of the preference vector. How specifically can individual points on the Pareto front be identified and selected without enumerating over many solutions?
It also looks like that PA-Net does not find as extreme solutions as DRL-MOA (Figure 2). Could the authors comment on that?
Why did you not consider an exact solver, e.g. with a linearized objective, as an additional baseline?
NO.
Dear Reviewer,
Thank you for your comments. We are working on revising our manuscript based on your comments and will upload the new version in coming days.
In the meantime, please find some responses below:
- Regarding extension to other problems: The focus of this work was MOTSP but we do believe that our framework can be extended to other multi-objective tasks. Due to limited time and space, it would be hard to incorporate other problems, but we will try our best to provide some results in the limited time frame.
- Regarding the effect of preferences: The strength of our network is that it can generalize to various preferences in objective space.
So at test time, we rather generate a set of preferences in the objective space and then infer a result for each preference one by one.
So a user can input their desired preference in the objective space and PA-Net can find a solution corresponding to that preference. The preferences are captured via the constraint in Eq (7). The network may not exactly find a solution corresponding to a particular preference (due to combinatorial nature of the problem) but in practice our network performs reasonably well.
For instance, in the 200 city MOTSP problem, the constraint violation is almost zero.
In case of 3-Objective, it is 0.0031, and in case of 5-Objective it is 0.08. We will put these results in the appendix.
- Regarding finding extreme points: We believe that this might be due to the preferences on which the network is trained. For PA-2 we have trained our network for preferences generated from unit vectors for angles in range {$10-85^{\circ}$} and the same applies to the inference. We can attempt to see the effect of using a broader range and report back the results.
- Regarding using exact solver: In the coming days, we will to use an exact solver to generate comparative results and report back to you.
Strengths:
1. The authors have provided a good background on the problem with adequate review of literature.
2. Motivation to determining the Pareto optimal solution as a constrained programming for the MOTSP is well written and provides good readability.
3. The authors had explained the experimental design clearly in Section 4 and well-written.
4. The authors have also presented the convergence of the proposed method to Concave Pareto Front.
Need some minor improvements
1. Need supportive evidence for making a strong statements such as "These classical methods to solve MOTSP are typically handcrafted and don’t generalize well for different problem instances." (Section 1 Page 1).
2. Rationale of leveraging Monte-carlo sampling to calculate HV needs to be explored in detail.
3.
Although the proposed methodology yields better results under the Hyper volume and computation time, the authors are obliged to provide the rationing of the performance against DRL-MOA, NSGA-II and MOEA/D.
4. The authors had taken good care in explaining and conducting the experiments in detail but the performance analysis of the different algorithms provided were inadequate.
5. The authors had the scope to explain why the classical algorithms are not well performing in larger instances and deep learning based models performing well in those scenarios but missed to detailed out in the discussion section.
Typo:
Appendix section - First paragraph as a typo. Please correct it.
NO.
Dear Reviewer,
Thank you for your comments. We are working on revising our manuscript based on your comments and will upload the new version in coming days.
In the meantime, please find some responses below:
* Response for unsupported statement in Introduction: We agree with you that, that particular sentence is a strong statement to make. In those statements, we wanted to convey that a lot of the existing methods such as evolutionary algorithms  suffer in performance and computation time, with an increase in the scale of the problem. This was demonstrated in our experiments. We will suitably amend that statement in our revised manuscript.
So "These classical methods to solve MOTSP are typically handcrafted and don’t generalize well for different problem instances. Often these methods generate poor quality of solutions and are computationally expensive (Coello et al., 2009)." will be changed to "In practice these evolutionary based methods suffer in performance and computation time with an increase in the scale of the problem as shown in our experiments. "
* Answer to comments regarding use of Monte Carlo sampling for Hypervolume (HV) calculation: HV is a hybrid indicator that is used to evaluate the quality of Pareto front in terms of the dominated objective space.  HV can be visualized as the grey region in this [figure](https://imgur.com/DEnJhMY), which is taken from [1]. We use a sampling based approach to calculate HV. This is done by sampling a large number of points from a hypercube in objective space which is formed using a reference point ([shown here](https://imgur.com/DEnJhMY)).
Then the hypervolume is calculated based on the number of these points which are dominated by the provided Pareto set. This method of calculating the HV was primarily selected because it's straightforward and can generate the values quickly.
* Response to comments 3,4 and 5 :
- From our experimental studies it can be clearly seen that deep learning based methods outperform the evolutionary methods both in terms of HV and computation time. One explanation for this could be that deep learning based methods explore the solution space much better and due to the generalizability power of the deep networks, they are able to perform consistently well on larger problem instances.
On the other hand, evolutionary methods significantly underperform as the scale of the problem increases. This is likely because these algorithms are unable to explore the solution space well for larger problems. Running these algorithms for more iterations could potentially improve their performance in terms of HV, but this comes with an additional computational cost.
- PA-Net performs relatively better than DRL-MOA in most of the cases in terms of HV and computation time. The main reason DRL-MOA lags is that its solutions do not necessarily span the complete objective space.
For instance, you could see gaps in the Pareto front of DRL-MOA. The main reasons for this are that there are no constraints in DRL-MOA that binds its solution to a specific region in objective space.
For example, our network is trained to find solutions along specific unit vectors in objective space.
Further, for DRL-MOA more networks would have to be trained with different scalar preferences to generate a dense Pareto Front which would be expensive.
References:
[1] Audet, Charles, et al. "Performance indicators in multiobjective optimization." European journal of operational research (2020).

Meta-review:
role: PC
data: comment: The paper introduces a method to approximate a Pareto front for multi-objective TSP. The proposed method first converts the MOTSP into a set of constrained single-objective optimization problems with different preference-based constraints. Then it builds a modified TSP-Net with preference augmentation to solve all the constrained problems. The method is empirically compared with multi-objective genetic algorithms and a DRL-based approach, showing to be competitive in the approximation of the Pareto front.

After reading the authors' feedback and discussing their concerns, the reviewers reached a consensus and they think that the paper is still not ready for publication. The authors need to improve their experimental evaluation in order to make it more robust and fair.

Decision: Reject