SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinbiao Chen<sup>1</sup>, Zizhen Zhang<sup>1</sup>, Zhiguang Cao<sup>2</sup>, Yaoxin Wu<sup>3</sup>, Yining Ma<sup>4</sup>, Te Ye<sup>1</sup>, and Jiahai Wang<sup>1,5,6,*</sup>
$^{2}$ School of Computing and Information Systems, Singapore Management University, Singapore
$^{3}$ Department of Industrial Engineering & Innovation Sciences, Eindhoven University of Technology, Netherlands
$^{4}$ Department of Industrial Systems Engineering & Management, National University of Singapore, Singapore
<sup>5</sup>Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, Sun Yat-sen University, P.R. China
$^{6}$ Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, P.R. China
chenjb69@mail2.sysu.edu.cn,zhangzzh7@mail.sysu.edu.cn
zgcao@smu.edu.sg, y.wu2@tue.nl, yiningma@u.nus.edu
yete@mail2.sysu.edu.cn, wangjiah@mail.sysu.edu.cn
Most of existing neural methods for multi-objective combinatorial optimization (MOCO) problems solely rely on decomposition, which often leads to repetitive solutions for the respective subproblems, thus a limited Pareto set. Beyond decomposition, we propose a novel neural heuristic with diversity enhancement (NHDE) to produce more Pareto solutions from two perspectives. On the one hand, to hinder duplicated solutions for different subproblems, we propose an indicator-enhanced deep reinforcement learning method to guide the model, and design a heterogeneous graph attention mechanism to capture the relations between the instance graph and the Pareto front graph. On the other hand, to excavate more solutions in the neighborhood of each subproblem, we present a multiple Pareto optima strategy to sample and preserve desirable solutions. Experimental results on classic MOCO problems show that our NHDE is able to generate a Pareto front with higher diversity, thereby achieving superior overall performance. Moreover, our NHDE is generic and can be applied to different neural methods for MOCO.
Multi-objective combinatorial optimization (MOCO) has been extensively studied in the communities of computer science and operations research [1, 2]. It also commonly exists in many industries, such as transportation [3], manufacturing [4], energy [5], and telecommunication [6]. MOCO features multiple conflicting objectives based on NP-hard combinatorial optimization (CO), practical yet more complex. Rather than finding an optimal solution like in the single-objective optimization, MOCO pursues a set of Pareto-optimal solutions, called Pareto set, to trade-off the multiple objectives. In general, a decent Pareto set is captured by both desirable convergence (optimality) and diversity.
Since exactly solving MOCO may require exponentially increasing computational time [7], the heuristic methods [8] have been favored in practice over the past few decades, which aim to yield an
Figure 1: The framework of NHDE. For subproblem  $i$ , the heterogeneous graph attention (HGA) model takes instance  $s$ , points  $\mathcal{F}^{i-1} \cup \{\boldsymbol{r}\}$ , weight  $\lambda^i$ , and diversity factor  $w^i$  as inputs, and generates solutions to optimize the scalar objective and hypervolume (size of the gray area). More solutions are sampled and the Pareto front  $\mathcal{F}^i$  is then efficiently updated based on multiple Pareto optima (MPO).
approximate Pareto set. Despite a relatively high efficiency, heuristic methods need domain-specific knowledge and massive iterative search. As neural CO methods based on deep reinforcement learning (DRL) recently achieved notable success in CO problems such as routing, scheduling, and bin packing [9-12], a number of neural MOCO methods have also been accordingly developed [13-16]. Typically, parameterized as a deep model, Neural MOCO is able to automatically learn a heuristic (or policy), so as to directly construct near-optimal solutions in an end-to-end fashion, which takes much less computational time than the traditional ones.
Existing neural MOCO methods mostly decompose an MOCO problem into a series of single-objective CO subproblems and derive a Pareto set by solving them. However, while enjoying a favorable efficiency for neural MOCO, the sole decomposition is less effective in finding as many diverse solutions as possible, since constructing an optimal solution for the decomposed subproblems is always carried out independently, causing repetitive or duplicated ones for different subproblems.
To tackle this issue, we propose a neural heuristic with diversity enhancement (NHDE), as illustrated in Figure 1. Distinguished from existing neural methods, NHDE couples the decomposition with a comprehensive indicator to learn a policy that can produce diverse solutions across the subproblems while further improving the performance. Besides, for a given subproblem, multiple relevant solutions, rather than a single optimal solution with respect to the scalar objective, are found based on a proposed multiple Pareto optima (MPO) strategy, so as to further strengthen the diversity.
Our contributions are summarized as follows. (1) We propose an indicator-enhanced DRL method. To encourage the deep model to generate different yet diverse solutions for decomposed subproblems, NHDE inputs the Pareto front composed of the preceding solutions and introduces an indicator comprehensively measuring convergence and diversity in the reward. (2) We design a heterogeneous graph attention model to effectively capture the correlation between an instance graph and its Pareto front graph. (3) We present a multiple Pareto optima (MPO) strategy to further identify more relevant solutions in the neighborhood of each subproblem and efficiently update the Pareto front. (4) We deploy our NHDE with two different types of neural MOCO methods to demonstrate its versatility. Experimental results based on various MOCO problems show that our NHDE outperforms state-of-the-art neural baselines, especially with significant improvement in the diversity.
Exact and heuristic methods for MOCO. Exact methods [7, 17] for MOCO can attain the accurate Pareto set, but their computational time may grow exponentially, rendering them less practical. As an alternative, heuristic methods such as multi-objective evolutionary algorithms (MOEAs) have gained widespread attention in practice. Dominance-based NSGA-II [18], decomposition-based MOEA/D [19], and indicator-based SMS-EMOA [20] are three typical paradigms of MOEAs. In these MOEAs, local search, a crucial technique specialized to the target CO, is usually employed [21-23].
Neural CO. In the past few years, neural construction methods [24-26] were proposed to rapidly yield high-quality solutions in an end-to-end fashion. A well-known representative is Attention Model (AM) [27], which was developed based on the Transformer architecture [28]. Then, AM inspired a large number of subsequent works [29-35], which further boosted the performance. Among them,
the policy optimization with multiple optima (POMO) [36], leveraging the solution and problem symmetries, is recognized as a prominent approach. Besides, the other line of works, known as neural improvement methods [37-41], exploited DRL to assist the iterative improvement process from an initial but complete solution, following a learn-to-improve paradigm.
Neural MOCO. Decomposition is a mainstream scheme in learning-based methods for multi-objective optimization [42-44]. An MOCO problem can be decomposed into a series of single-objective CO problems and then solved by neural construction methods to approximate the Pareto set. A couple of preliminary works trained multiple deep models with transfer learning [13, 45], where each deep model coped with one subproblem. Evolutionary learning [46, 47] was introduced to evolve deep models to further improve the performance. Instead of training multiple deep models for preset weights, preference-conditioned multi-objective combinatorial optimization (PMOCO) [14] and Meta-DRL (MDRL) [15], both of which only trained one deep model, were more flexible and practical. For a given weight vector, the former used a hypernetwork to derive the decoder parameters to solve the corresponding subproblem, while the latter rapidly fine-tuned a pre-trained meta-model to solve the corresponding subproblem. However, those solely decomposition-based neural MOCO methods are limited in the diversity with respect to the solutions of the Pareto set, since some subproblems may lead to duplicated solutions, especially when they are solved independently.
3.1 MOCO
An MOCO problem with  $M$  objectives can be expressed as  $\min_{\boldsymbol{x} \in \mathcal{X}} \boldsymbol{f}(\boldsymbol{x}) = (f_1(\boldsymbol{x}), f_2(\boldsymbol{x}), \ldots, f_M(\boldsymbol{x}))$ , where  $\mathcal{X}$  is a set of discrete decision variables.
Definition 1 (Pareto dominance). A solution  $\pmb{x}^1 \in \mathcal{X}$  dominates another solution  $\pmb{x}^2 \in \mathcal{X} (\pmb{x}^1 \prec \pmb{x}^2)$ , if and only if  $f_i(\pmb{x}^1) \leq f_i(\pmb{x}^2), \forall i \in \{1, \dots, M\}$  and  $\exists j \in \{1, \dots, M\}, f_j(\pmb{x}^1) < f_j(\pmb{x}^2)$ .
Definition 2 (Pareto optimality). A solution  $\pmb{x}^{*} \in \mathcal{X}$  is Pareto-optimal if it is not dominated by any other solution, i.e.,  $\not\exists x' \in \mathcal{X}$  such that  $\pmb{x}' \prec \pmb{x}^*$ .
Definition 3 (Pareto set/front). MOCO aims to uncover a Pareto set, comprising all Pareto optimal solutions  $\mathcal{P} = \{\pmb{x}^* \in \mathcal{X} \mid \not\exists \pmb{x}' \in \mathcal{X} : \pmb{x}' \prec \pmb{x}^*\}$ . The Pareto front  $\mathcal{F} = \{\pmb{f}(\pmb{x}) \mid \pmb{x} \in \mathcal{P}\}$  corresponds to the objective values of Pareto set, with each  $\pmb{f}(\pmb{x})$  referred to as a point in the objective space.
For MOCO, decomposition [19] is a prevailing scheme due to its flexibility and effectiveness. An MOCO problem can be decomposed into  $N$  subproblems with  $N$  weights. Each subproblem is a single-objective CO problem via scalarization  $g(\pmb{x}|\pmb{\lambda})$  with a weight  $\lambda \in \mathcal{R}^M$  satisfying  $\lambda_{m}\geq 0$  and  $\sum_{m = 1}^{M}\lambda_{m} = 1$ . The Pareto set then can be derived by solving the  $N$  subproblems.
The simplest yet effective decomposition approach is the weighted sum (WS). It uses the linear scalarization of  $M$  objectives, which hardly raises the complexity of the subproblems, as follows,
$$
\min  _ {\boldsymbol {x} \in \mathcal {X}} g _ {\mathrm {w s}} (\boldsymbol {x} | \boldsymbol {\lambda}) = \sum_ {m = 1} ^ {M} \lambda_ {m} f _ {m} (\boldsymbol {x}). \tag {1}
$$
3.3 Indicator
Hypervolume (HV) is a mainstream indicator to measure performance, as it can comprehensively assess the convergence and diversity without the exact Pareto front [48]. For a Pareto front  $\mathcal{F}$  in the objective space,  $\mathrm{HV}_r(\mathcal{F})$  with respect to a fixed reference point  $r\in \mathcal{R}^M$  is defined as follows,
$$
\operatorname {H V} _ {\boldsymbol {r}} (\mathcal {F}) = \mu \left(\bigcup_ {\boldsymbol {f} (\boldsymbol {x}) \in \mathcal {F}} [ \boldsymbol {f} (\boldsymbol {x}), \boldsymbol {r} ]\right), \tag {2}
$$
where  $\mu$  is the Lebesgue measure, i.e.,  $M$ -dimensional volume, and  $[f(\pmb{x}), r]$  is a  $M$ -dimensional cube, i.e.,  $[f(\pmb{x}), r] = [f_1(\pmb{x}), r_1] \times \dots \times [f_M(\pmb{x}), r_M]$ .
A 2-dimensional example with 5 points in the objective space is depicted in Figure 1, where  $\mathcal{F} = \{\pmb {f}(\pmb{x}^1),\pmb {f}(\pmb{x}^2),\pmb {f}(\pmb{x}^3),\pmb {f}(\pmb{x}^4),\pmb {f}(\pmb{x}^5)\}$ .  $\mathrm{HV}_r(\mathcal{F})$  is equal to the size of the gray area, and finally normalized into [0, 1]. All methods share the same reference point  $\pmb{r}$  for a problem (see Appendix A).
Our neural heuristic with diversity enhancement (NHDE) exploits indicator-enhanced DRL to produce diverse solutions across different subproblems and leverages a multiple Pareto optima (MPO) strategy to find multiple neighbor solutions for each subproblem, as illustrated in Figure 1. Specifically, an MOCO problem is decomposed into  $N$  single-objective subproblems with  $N$  weights, which are solved dependently by a unified heterogeneous graph attention (HGA) model  $\theta$ . For each subproblem  $i$ , its features together with the current Pareto front  $\mathcal{F}$  (points in the objective space) constituted by preceding solutions are input to model  $\theta$ , which is guided by the scalar objective with the HV indicator. Then, MPO is utilized to sample multiple solutions and efficiently update  $\mathcal{F}$ .
4.1 Indicator-enhanced DRL
Given a problem instance  $s$ , we sequentially solve its subproblem  $i \in \{1, \dots, N\}$ , each associated with weight  $\lambda^i$ . Let  $\pi^i = \{\pi_1^i, \dots, \pi_T^i\}$  denote the obtained solution at step  $i$ , and let  $\mathcal{F}^i$  be the Pareto front yielded by solutions from subproblem 1 to  $i$ . In each step  $i$ , we select up to  $K$  top points  $\pmb{f}(\pmb{\pi}) \in \mathcal{F}^{i-1}$  from the Pareto front at step  $i-1$  with the ranking determined by the scalar objective  $g(\pmb{\pi}|s, \pmb{\lambda}^i)$  with respect to the new given weight  $\lambda^i$ . The corresponding scalar objective and the induced surrogate landscape  $\tilde{\mathcal{F}}^{i-1} \subseteq \mathcal{F}^{i-1}$  based on those selected solutions are treated as the policy network inputs (see Figure 2), so as to construct a new solution  $\pi^i$  and yield a new  $\mathcal{F}^i$ .
The construction of the solution  $\pi^i$  with length  $T$  for each subproblem  $i$  can be cast as a Markov decision process. In particular, 1) the state includes the weight  $\lambda^i$ , user-defined diversity factor  $\pmb{w}^i \in \mathcal{R}^2$  satisfying  $w_1^i, w_2^i \geq 0$  and  $w_1^i + w_2^i = 1$ , partial solution  $\pi_{1:t-1}^i$ , instance  $s$ , and  $\tilde{\mathcal{F}}_r^{i-1}$ , where  $\tilde{\mathcal{F}}_r^{i-1} = \tilde{\mathcal{F}}^{i-1} \cup \{r\}$  incorporates the aforementioned surrogate landscape at step  $i$  and the given reference point  $r$ ; 2) the action is to add a node  $\pi_t^i$  into  $\pi_{1:t-1}^i$ ; 3) the state transition transforms  $\pi_{1:t-1}^i$  to  $\pi_{1:t}^i$ , denoted as  $\pi_{1:t}^i = \{\pi_{1:t-1}^i, \pi_t^i\}$ ; 4) the reward is defined as  $R^i = -w_1^i \times g(\pi^i | s, \pmb{\lambda}^i) + w_2^i \times \mathrm{HV}_r(\tilde{\mathcal{F}}^{i-1} \cup \{f(\pmb{\pi}^i)\})$ , where we introduce the hypervolume  $\mathrm{HV}_r(\tilde{\mathcal{F}}^{i-1} \cup \{f(\pmb{\pi}^i)\})$  to guide the search; and 5) the stochastic policy generating the solution  $\pi^i$  is expressed as  $P(\pi^i | s, \tilde{\mathcal{F}}_{r}^{i-1}, \pmb{\lambda}^i, \pmb{w}^i) = \prod_{t=1}^{T} P_{\pmb{\theta}}(\pi_t^i | \pi_{1:t-1}^i, s, \tilde{\mathcal{F}}_{r}^{i-1}, \pmb{\lambda}^i, \pmb{w}^i)$ , with the probability of node selection  $P_{\pmb{\theta}}(\pi_t^i | \pi_{1:t-1}^i, s, \tilde{\mathcal{F}}_{r}^{i-1}, \pmb{\lambda}^i, \pmb{w}^i)$  parameterized by a deep model  $\pmb{\theta}$ .
We would like to note that our NHDE is generic, and can directly integrate the base model  $\theta$  with the existing decomposition-based neural MOCO methods. We demonstrate this property by applying it to two state-of-the-art methods, PMOCO [14] and MDRL [15], denoted as NHDE-P and NHDE-M, respectively. Given  $\lambda^i$  and  $w^i$  as inputs, NHDE-P uses a hypernetwork to generate the decoder parameters of the model  $\theta (\lambda^i,w^i)$ , while NHDE-M fine-tunes the pre-trained meta model  $\theta_{\mathrm{meta}}$  with a few steps to address the corresponding subproblem. More details are presented in Appendix B.
4.2 Heterogeneous graph attention
To effectively solve subproblem  $i$ , the model should jointly capture the representations of both the instance's node graph and the Pareto front's point graph. Based on the encoder-decoder structure, we thus design a heterogeneous graph attention (HGA) model to correlate the two heterogeneous graphs as depicted in Figure 2. In the following section, we omit superscript  $i$  for better readability.
Encoder. Given an instance graph  $s$  containing  $n$  nodes with  $Z$ -dimensional features and a Pareto front graph  $\tilde{\mathcal{F}}_r^{i-1}$  containing  $k$  points ( $k \leq K+1$ ) with  $M$ -dimensional features, their initial embeddings  $h_1^{(0)}, \ldots, h_n^{(0)} \in R^d$  and  $g_1^{(0)}, \ldots, g_k^{(0)} \in R^d$  are yielded by a linear projection with trainable parameters  $W_h$  and  $W_g$ , respectively, and  $d$  is empirically set to 128. The eventual embeddings  $h_1^{(L)}, \ldots, h_n^{(L)}$  and  $g_1^{(L)}, \ldots, g_k^{(L)}$  are derived by further passing through  $L = 6$  attention layers.
In this sub-section, we consider the construction of only one solution in each step for better readability; however, we note that multiple solutions can be sampled and our formulation would work in a similar manner.
Figure 2: Illustration of the proposed heterogeneous graph attention (HGA) model.
Each attention layer is composed of a multi-head HGA layer with  $Y = 8$  heads and a fully connected feed-forward sublayer. For layer  $l \in \{1, \dots, L\}$ , HGA computes the representations  $\tilde{\pmb{h}}_u^{(l)}$  and  $\tilde{\pmb{g}}_u^{(l)}$  across the heterogeneous graphs, which are used to update the embeddings  $\pmb{h}_u^{(l)}$  and  $\pmb{g}_u^{(l)}$ . Skip-connection [49] and batch normalization [50] are both adopted in each sublayer, as follows,
$$
\boldsymbol {h} _ {u} ^ {(l)} = \operatorname {B N} \left(\hat {\boldsymbol {h}} _ {u} + \operatorname {F F} \left(\hat {\boldsymbol {h}} _ {u}\right)\right), \hat {\boldsymbol {h}} _ {u} = \operatorname {B N} \left(\boldsymbol {h} _ {u} ^ {(l - 1)} + \tilde {\boldsymbol {h}} _ {u} ^ {(l)}\right), \quad \forall u \in \{1, \dots , n \}, \tag {3}
$$
$$
\boldsymbol {g} _ {u} ^ {(l)} = \operatorname {B N} \left(\hat {\boldsymbol {g}} _ {u} + \operatorname {F F} \left(\hat {\boldsymbol {g}} _ {u}\right)\right), \hat {\boldsymbol {g}} _ {u} = \operatorname {B N} \left(\boldsymbol {g} _ {u} ^ {(l - 1)} + \tilde {\boldsymbol {g}} _ {u} ^ {(l)}\right), \quad \forall u \in \{1, \dots , k \}. \tag {4}
$$
Decoder. The decoder, composed of a multi-head HGA layer and a compatibility layer, autoregressively constructs a solution according to the probability distribution with  $T$  steps. At decoding step  $t \in \{1, \dots, T\}$ , the glimpse  $\mathbf{q}_c$  of the context embedding  $\mathbf{h}_c$  (see Appendix C) is computed by the HGA layer. Then, the compatibility  $\alpha$  is calculated as follows,
$$
\alpha_ {u} = \left\{ \begin{array}{l l} - \infty , & \text {n o d e} u \text {i s m a s k e d} \\ C \cdot \tanh  \left(\frac {\boldsymbol {q} _ {c} ^ {T} \left(W ^ {K} \boldsymbol {h} _ {u} ^ {(L)}\right)}{\sqrt {d / Y}}\right), & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
where  $C$  is set to 10 [27]. Finally, softmax is employed to calculate the selection probability distribution  $P_{\theta}(\pi |s,\tilde{\mathcal{F}}_{r}^{i - 1},\pmb{\lambda}^{i},\pmb{w}^{i})$  for nodes, i.e.,  $P_{\theta}(\pi_t|\pi_{1:t - 1},s,\tilde{\mathcal{F}}_r^{i - 1},\pmb {\lambda}^i,\pmb {w}^i) = \mathrm{Softmax}(\pmb {\alpha})$
HGA. The HGA layer in the encoder captures three key relations between the node graph and the point graph. The first, node-to-node  $\alpha_{uv}^{hh}$ , indicates each node's attention towards others within the same instance to construct a promising solution. The second, node-to-point  $\alpha_{uv}^{hg}$ , suggests each node's attention to points, guiding the constructed solutions distinct from the existing ones in the current Pareto front. The third, point-to-node  $\alpha_{uv}^{gh}$ , indicates each point's attention to nodes, facilitating the learning of the mapping from a solution to its objective values. We disregard the less meaningful point-to-point attention. Concretely, Eq. (6) defines the above three attention scores, which are separately normalized as  $\tilde{\alpha}_{uv}^{hh}$ ,  $\tilde{\alpha}_{uv}^{hg}$ , and  $\tilde{\alpha}_{uv}^{hg}$  by softmax. Then,  $\tilde{h}_u$  and  $\tilde{g}_u$  are computed by Eq. (7).
$$
\alpha_ {u v} ^ {h h} = \frac {\left(W _ {h} ^ {Q} \boldsymbol {h} _ {u}\right) ^ {T} \left(W _ {h} ^ {K} \boldsymbol {h} _ {v}\right)}{\sqrt {d / Y}}, \alpha_ {u v} ^ {h g} = \frac {\left(W _ {h} ^ {Q} \boldsymbol {h} _ {u}\right) ^ {T} \left(W _ {g} ^ {K} \boldsymbol {g} _ {v}\right)}{\sqrt {d / Y}}, \alpha_ {u v} ^ {g h} = \frac {\left(W _ {g} ^ {Q} \boldsymbol {g} _ {u}\right) ^ {T} \left(W _ {h} ^ {K} \boldsymbol {h} _ {v}\right)}{\sqrt {d / Y}}. \tag {6}
$$
$$
\tilde {\boldsymbol {h}} _ {u} = \sum_ {v = 1} ^ {n} \tilde {\alpha} _ {u v} ^ {h h} W _ {h} ^ {V} \boldsymbol {h} _ {v} + \sum_ {v = 1} ^ {k} \tilde {\alpha} _ {u v} ^ {h g} W _ {g} ^ {V} \boldsymbol {g} _ {v}, \tilde {\boldsymbol {g}} _ {u} = \sum_ {v = 1} ^ {n} \tilde {\alpha} _ {u v} ^ {g h} W _ {h} ^ {V} \boldsymbol {h} _ {v}. \tag {7}
$$
Finally, as for the multi-head attention,  $\tilde{\pmb{h}}_u$  and  $\tilde{\pmb{g}}_u$  are further computed as follows,
$$
\tilde {\boldsymbol {h}} _ {u} = W _ {h} ^ {O} \operatorname {C o n c a t} \left(\tilde {\boldsymbol {h}} _ {u, 1}, \dots , \tilde {\boldsymbol {h}} _ {u, Y}\right), \quad \tilde {\boldsymbol {g}} _ {u} = W _ {g} ^ {O} \operatorname {C o n c a t} \left(\tilde {\boldsymbol {g}} _ {u, 1}, \dots , \tilde {\boldsymbol {g}} _ {u, Y}\right), \tag {8}
$$
where  $\tilde{h}_{u,y}$  and  $\tilde{g}_{u,y}$  for head  $y\in \{1,\dots,Y\}$  are obtained according to Eq. (7). In the multihead HGA,  $W_{h}^{Q},W_{h}^{K},W_{h}^{V},W_{h}^{O},W_{g}^{Q},W_{g}^{K},W_{g}^{V}$ , and  $W_{g}^{O}$  are independent trainable parameters. Similarly, in the decoder, the glimpse  $\pmb{q}_c$  is calculated by the context embedding  $\pmb{h}_c$  with the addition of the context-to-node and context-to-point attention, i.e., replacing  $\pmb{h}_u$  with  $\pmb{h}_c$  in the Eq. (6-8).
Algorithm 1 Training algorithm of NHDE-P
1: Input: weight distribution  $\Lambda$ , diversity-factor distribution  $\mathcal{W}$ , instance distribution  $\mathcal{S}$ , number of training steps  $E$ , number of sampled weights per step  $N'$ , batch size  $B$ , instance size  $n$   
2: Initialize the model parameters  $\theta$   
3: for  $e = 1$  to  $E$  do  
4:  $s_i \sim \text{SampleInstance}(\mathcal{S}) \forall i \in \{1, \dots, B\}$   
5: Initialize  $\mathcal{F}_i \gets \emptyset \forall i$   
6: for  $n' = 1$  to  $N'$  do  
7:  $\lambda \sim \text{SampleWeight}(\Lambda)$   
8:  $\boldsymbol{w} \sim \text{SampleDiversityFactor}(\mathcal{W})$   
9:  $\pi_i^j \sim \text{SampleSolution}(P_{\theta(\lambda, w)}(|s_i, \tilde{\mathcal{F}}_r, i, \lambda, w)) \forall i \in \{1, \dots, B\} \forall j \in \{1, \dots, n\}$   
10:  $R_i^j \gets -w_1 g(\pi_i^j | s_i, \lambda) + w_2 \text{HV}_r(\tilde{\mathcal{F}}_i \cup \{f(\pi_i^j)\}) \forall i, j$   
11:  $b_i \gets \frac{1}{n} \sum_{j=1}^{n} (-R_i^j) \forall i$   
12:  $\nabla \mathcal{J}(\theta) \gets \frac{1}{Bn} \sum_{i=1}^{B} \sum_{j=1}^{n} [(-R_i^j - b_i) \nabla_{\theta(\lambda, w)} \log P_{\theta(\lambda, w)} (\pi_i^j | s_i, \tilde{\mathcal{F}}_r, i, \lambda, w)]$   
13:  $\theta \gets \text{Adam}(\theta, \nabla \mathcal{J}(\theta))$   
14:  $\mathcal{G}_i \gets \{f(\pi_i^1), \dots, f(\pi_i^n)\} \forall i$   
15:  $\mathcal{F}_i \gets \text{MPO}(\tilde{\mathcal{F}}_i \cup \tilde{\mathcal{G}}_i) \forall i$   
16: end for  
17: end for  
18: Output: The model parameter  $\theta$
4.3 Multiple Pareto optima strategy
Contrary to single-objective problems that focus on a single optimal solution, MOCO problems involve a series of Pareto-optimal solutions. In light of this, we introduce a multiple Pareto optima (MPO) strategy to uncover multiple solutions for each subproblem by leveraging the Pareto optimality.