SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Feature Embedding. Building upon the dual-aspect representation design from [12], we embed state features $^3$ , denoted as  $\psi[\mathcal{G}, \tau_t] = \{\{\varphi_i^t\}_{x_i \in \mathcal{V}}, \{p_i^t\}_{x_i \in \mathcal{V}}\}$  into two separate sets of node embeddings: Node Feature Embeddings (NFEs) to encode  $\psi(\mathcal{G}) = \{\varphi_i^t\}$  and Positional Feature Embeddings (PFEs) to encode  $\psi(\tau_t) = \{p_i^t\}$ . The NFEs, denoted as  $\{h_i^{\mathrm{init}}\}_{x_i \in \mathcal{V}}$ , are a set of  $d$ -dimensional vectors, each of which embeds  $d_h$ -dimensional problem-specific raw features  $\varphi_i^t$  of node  $x_i$  at step  $t$ . To obtain  $h_i^{\mathrm{init}}$ , we upgrade linear projection used in [12] into the MLP with structure of  $(d_h \times \frac{d}{2} \times d)$  for enhanced representation. The PFEs, denoted as  $\{g_i^{\mathrm{init}}\}_{x_i \in \mathcal{V}}$ , are a set of  $d$ -dimensional vectors, each of which embeds positional features  $p_i^t$  of  $x_i$  at step  $t$ , derived from the position of  $x_i$  in the current solution  $\tau_t$ . Following [9], we employ the Cyclic Positional Encoding (CPE) to generate a series of cyclic embeddings in a  $d$ -dimensional space. These CPE embeddings are then used to initialize  $g_i^{\mathrm{init}}$  correspondingly, so as to capture the topological structure of the nodes in the current solution  $\tau_t$ .
Problem-specific raw node features  $\varphi_i^t$ . For TSP, node features  $\varphi_i^t$  of  $x_i$  contains its two-dimensional coordinates (i.e.,  $d_h = 2$ ); For CVRP,  $\varphi_i^t$  contains six features (i.e.,  $d_h = 6$ ) including 1-2) its two-dimensional coordinates, 3) the demand of node  $x_i$ , 4) the sum of demand of the corresponding sub-tour before node  $x_i$  (inclusive), 5) the sum of demand of the corresponding sub-tour after  $x_i$  (exclusive), and 6) an indicator function to signify whether node  $x_i$  is a customer node. When GIRE is applied, we enrich  $\varphi_i^t$  with two Violation Indicator (VI) features, i.e.,  $d_h = 8$ .
Note that the  $\tau_t^{\mathrm{bsf}}$  is leveraged in the reward function and the critic network, but not the policy network.
Stacked encoders. Following the encoders in [9, 12], we use Transformer-styled encoders with Synthesis Attention (Synth-Att) to refine the embeddings  $\{h_i^{\mathrm{init}}\} x_i \in \mathcal{V}$  and  $\{g_i^{\mathrm{init}}\} x_i \in \mathcal{V}$ , where PFEs serve as auxiliary embeddings that bolster the representation learning of NFEs. After the encoding, a unified set of embeddings  $\{h_i\}_{x_i \in \mathcal{V}}$  is obtained, which is then inputted in our recurrent dual-stream (RDS) decoder for k-opt action decoding (see Section 4).
C Training and inference algorithms
C.1 Training algorithm
Input: policy network  $\pi_{\theta}$ , critic network  $v_{\phi}$ , PPO objective clipping threshold  $\vartheta$ , learning rate  $\eta_{\theta}$ ,  $\eta_{\phi}$ , learning rate decay rate  $\varsigma$ , number of PPO inner loops  $\Omega$ , training steps  $T_{\mathrm{train}}$ , number of epochs  $E$ , number of batches per epoch  $B$ , curriculum learning (CL) scalar  $\xi$
Algorithm 1 Reinforcement learning algorithms for NeuOpt
1: for epoch  $= 1$  to  $E$  do   
2: for batch  $= 1$  to  $B$  do   
3: Randomly generate a batch of training instances  $\mathcal{D} = \{\mathcal{G}_i\}$  and their initial solutions  $\{\tau_i\}$    
4: CL: Improve  $\{\tau_i\}$  to  $\{\tau_i^{\prime}\}$  by runing the current policy  $\pi_{\theta}$  for  $T =$  epoch/  $\xi$  steps;   
5: Get initial state  $s_0$  based on  $\{\tau_i^{\prime}\}$  for each instance in the current batch;   
6:  $t\gets 0$    
7: while  $t <   T_{train}$  do   
8: Run policy  $\pi_{\theta}$  on each instance and get  $\{(s_{t'},a_{t'},r_{t'})\}_{t'=t}^{t+n-1}$  where  $a_{t'}\sim \pi_{\theta}(a_{t'}|s_{t'})$    
9:  $t\gets t + n,\pi_{old}\gets \pi_{\theta},v_{old}\gets v_{\phi};$    
10: for  $j = 1$  to  $\Omega$  do   
11:  $\hat{R}_t = v_\phi (s_t)$    
12: for  $t'\in \{t - 1,\dots,t - n\}$  do   
13:  $\hat{R}_{t'}\gets r_{t'} + \gamma \hat{R}_{t' + 1};$    
14:  $\hat{A}_{t'}\gets \hat{R}_{t'} - v_{\phi}(s_{t'})$    
15: end for   
16: Compute RL loss  $\mathcal{L}_{\theta}^{\mathrm{PPO}}$  using Eq. (6) and critic loss  $\mathcal{L}_{\phi}^{\mathrm{Critic}}$  using Eq. (7);   
17:  $\theta \leftarrow \theta +\eta_{\theta}\nabla \mathcal{L}_{\theta}^{\mathrm{PPO}};\phi \leftarrow \phi -\eta_{\phi}\nabla \mathcal{L}_{\phi}^{\mathrm{Critic}};$    
18: end for   
19: end while   
20: end for   
21:  $\eta_{\theta}\gets \varsigma \eta_{\theta},\eta_{\phi}\gets \varsigma \eta_{\phi};$    
22: end for
As detailed in Algorithm 1, we adapt the  $n$ -step proximal policy optimization (PPO) with curriculum learning (CL) strategy used in [9, 12] to train our NeuOpt. For our GIRE scheme, we consider learning separate critics  $v_{\phi}^{\mathrm{orig}}$ ,  $v_{\phi}^{\mathrm{reg}}$ , and  $v_{\phi}^{\mathrm{bonus}}$  to fit the respective reward shaping terms in Eq.(4), so as to better estimate the state values. In light of this, when GIRE is applied, we update Algorithm 1 by: 1) duplicating lines 11, 13-14 to compute  $\hat{R}_{t'}^{\mathrm{orig}}$ ,  $\hat{R}_{t'}^{\mathrm{reg}}$ , and  $\hat{R}_{t'}^{\mathrm{bonus}}$  as well as  $\hat{A}_{t'}^{\mathrm{orig}}$ ,  $\hat{A}_{t'}^{\mathrm{reg}}$ , and  $\hat{A}_{t'}^{\mathrm{bonus}}$  at the same time; 2) updating the line 16 to replace Eq. (6) and Eq. (7) into Eq. (8) and Eq. (9), respectively; and update line 17 accordingly where all the critics share the same learning rate  $\eta_{\phi}$ .
$$
\mathcal {L} _ {\theta} ^ {\mathrm {P P O}} = \frac {1}{n | \mathcal {D} |} \sum_ {\mathcal {D}} \sum_ {t ^ {\prime} = t - n} ^ {t - 1} \min  \left(\frac {\pi_ {\theta} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)}{\pi_ {o l d} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)} \hat {A} _ {t ^ {\prime}}, \operatorname {C l i p} \left[ \frac {\pi_ {\theta} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)}{\pi_ {o l d} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)}, 1 - \vartheta , 1 + \vartheta \right] \hat {A} _ {t ^ {\prime}}\right), \tag {6}
$$
$$
\mathcal {L} _ {\phi} ^ {\text {C r i t i c}} = \frac {1}{n | \mathcal {D} |} \sum_ {\mathcal {D}} \sum_ {t ^ {\prime} = t - n} ^ {t - 1} \max  \left(\left| v _ {\phi} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} \right| ^ {2}, \left| v _ {\phi} ^ {\text {C l i p}} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} \right| ^ {2}\right). \tag {7}
$$
$$
\begin{array}{l} \mathcal {L} _ {\theta} ^ {\text {G I R E}} = \frac {1}{n | \mathcal {D} |} \sum_ {\mathcal {D}} \sum_ {t ^ {\prime} = t - n} ^ {t - 1} \min  \left(\frac {\pi_ {\theta} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)}{\pi_ {o l d} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)} \left(\hat {A} _ {t ^ {\prime}} ^ {\text {o r i g i n}} + \hat {A} _ {t ^ {\prime}} ^ {\text {r e d}} + \hat {A} _ {t ^ {\prime}} ^ {\text {b o u n s}}\right), \right. \tag {8} \\ \operatorname {C l i p} \left[ \frac {\pi_ {\theta} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)}{\pi_ {o l d} \left(a _ {t ^ {\prime}} \mid s _ {t ^ {\prime}}\right)}, 1 - \vartheta , 1 + \vartheta \right] \left(\hat {A} _ {t ^ {\prime}} ^ {\text {o r i g i n}} + \hat {A} _ {t ^ {\prime}} ^ {\text {r e d}} + \hat {A} _ {t ^ {\prime}} ^ {\text {b o u n s}}\right), \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\phi} ^ {\text {G I R E}} = \frac {1}{n | \mathcal {D} |} \sum_ {\mathcal {D}} \sum_ {t ^ {\prime} = t - n} ^ {t - 1} \max  \left(\right. \\ \left| v _ {\phi} ^ {\text {o r i g i n}} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} ^ {\text {o r i g i n}} \right| ^ {2} + \left| v _ {\phi} ^ {\text {r e g}} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} ^ {\text {r e g}} \right| ^ {2} + \left| v _ {\phi} ^ {\text {b o u n s}} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} ^ {\text {b o u n s}} \right| ^ {2}, \tag {9} \\ \left| v _ {\phi} ^ {\text {o r i g i n , c l i p}} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} ^ {\text {o r i g i n}} \right| ^ {2} + \left| v _ {\phi} ^ {\text {r e g , c l i p}} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} ^ {\text {r e g}} \right| ^ {2} + \left| v _ {\phi} ^ {\text {b o u n s , c l i p}} \left(s _ {t ^ {\prime}}\right) - \hat {R} _ {t ^ {\prime}} ^ {\text {b o u n s}} \right| ^ {2}\left. \right). \\ \end{array}
$$
C.2 Inference algorithm
In our previous work [12], the data augmentation was incorporated during inference to boost the search diversity. The rationale is that a specific instance,  $\mathcal{G}$ , can be transformed to different ones, yet still retain the identical optimal solution. These augmented instances can be solved differently by the trained model in parallel, thereby enhancing the diversity of the search for better performance. Specifically, each augmented instance is generated by consecutively executing four predetermined invariant augmentation transformations as listed in Table 9, where the execution order and configurations used for each transformation are randomly determined on the fly following Algorithm 2.
Table 9: Descriptions of the four invariant augmentation transformations (retrieved from [12]).
Transformations	Formulations	Configurations
flip-x-y	(x',y')=(y,x)	perform or skip
1-x	(x',y')=(1-x,y)	perform or skip
1-y	(x',y')=(x,1-y)	perform or skip
rotate	(x'y)/(x cos θ - y sin θ)/x sin θ + y cos θ)	θ∈{0,π/2,π,3π/2}
Algorithm 2 Augmentation (retrieved from [12])
Output: Augmented instance  $\mathcal{G}'$
1:  $\mathcal{G}' \leftarrow \mathcal{G}$
Input: Instance  $\mathcal{G}$
2:  $\mathcal{A}\gets$  RandomShuffle([flip-x-y,1-x,1-y,rotate]);
3: for each augment method  $j \in \mathcal{A}$  do
4:  $\mathfrak{S}(j) \gets \text{RandomConfig}(j)$ ;
5:  $\mathcal{G}' \gets$  perform augment  $j$  on  $\mathcal{G}'$  with config  $\Im(j)$ ;
However, such augmentation is performed only once at the start of the inference, making the augmentation fixed across the search process. We thus propose Dynamic Data Augmentation (D2A). It suggests generating new augmented instances with different augmentation configurations each time when the solver fails to find a better solution within a consecutive maximum of  $T_{\mathrm{D2A}}$  steps (i.e., we consider the search to be trapped in local optima). This allows the model to explicitly solve instances differently once it gets trapped in the local optima, thus promoting an even more diverse search process. Note that the proposed D2A is generic to boost most L2S solvers, and even has the potential to boost L2C solvers when equipped with post-hoc per-instance processing boosters such as EAS [5]. In Algorithm 3, we summarize the D2A procedures where we note that the loops in line 1 and line 7 can be run in parallel during practical implementations. For example, when we use "D2A=5" as per in Table 1, it means that for each instance  $\mathcal{G}$ , there are 5 augmented instances  $\{\mathcal{G}_1, \mathcal{G}_2, \mathcal{G}_3, \mathcal{G}_4, \mathcal{G}_5\}$  being solved simultaneously. Each of these instances has its own counter  $T_i^{\mathrm{stall}}$  and the best-so-far solution  $\tau_i^{\mathrm{bsf}}$  to track whether the search (for the particular instance  $\mathcal{G}_i$ ) has fallen into local optima.
D Additional discussions on GIRE scheme
Separate critic networks. As per Appendix C, we use separate critics in GIRE. We now introduce their detailed architectures. During the critic value estimation, we first upgrade embeddings  $\{h_i\}$
Algorithm 3 Dynamic Data Augmentation (D2A)
Input: Instance  $\mathcal{G}$ , policy network  $\pi_{\theta}$ , inference step  $T$ , number of augments  $D2A$ , maximum number of consecutive steps allowed before considering the search trapped in local optima  $T_{\mathrm{D2A}}$   
Output: Best solution found during solving all the augmented instances  $\mathcal{G}_i$
1: for  $i = 1,\dots ,D2A$  do
2: Get an augmented instance:  $\mathcal{G}_i\gets$  Augmentation  $(\mathcal{G})$
3: Get a random solution  $\tau_{i,0}$  and set it as the best-so-far solution for  $\mathcal{G}_i\colon \tau_i^{\mathrm{bsf}}\gets \tau_{i,0}$
4: Set counter:  $T_{i}^{\mathrm{stall}} \gets 0$
5: end for
6: for  $t = 1, \dots, T$  do
7: for  $i = 1,\dots ,D2A$  do
8: Run one inference step to get a new solution  $\tau_{i,t}$  for  $\mathcal{G}_i$  using policy network  $\pi_{\theta}$ ;
9: if new solution  $\tau_{i,t}$  is a new best-so-far solution for instance  $\mathcal{G}_i$  then
10: Update the best-so-far solution:  $\tau_{i}^{\mathrm{bsf}}\gets \tau_{i,t}$
1: Reset counter:  $T_{i}^{\mathrm{stall}} \gets 0$
2: else
3: Increment counter:  $T_{i}^{\mathrm{stall}} = T_{i}^{\mathrm{stall}} + 1$
4: end if
5: if  $T_{i}^{\mathrm{stall}} \geq T_{\mathrm{D2A}}$  then
6: Get a new augmented instance:  $\mathcal{G}_i\gets$  Augmentation  $(\mathcal{G})$
7: Reset counter  $T_{i}^{\mathrm{stall}}\gets 0$
8: end if
9: end for
20: end for
(from the encoders of  $\pi_{\theta}$ ) to  $\{\hat{h}_i\}_{x_i\in \mathcal{V}}$  using a vanilla multi-head attention layer. The  $\{\hat{h}_i\}$  are then fed into a mean-pooling layer [39], yielding compressed representations as follows,
$$
\hat {y} _ {i} = \hat {h} _ {i} W ^ {\text {L o c a l}} + \text {m e a n} \left[ \left\{\hat {h} _ {i} \right\} _ {x _ {i} \in v} \right] W ^ {\text {G l o b a l}}, \tag {10}
$$
where  $W \in \mathbb{R}^{d \times \frac{d}{2}}$  are trainable matrices, and the mean and max are element-wise operators. All critics  $v_{\phi}^{\mathrm{origin}}$ ,  $v_{\phi}^{\mathrm{reg}}$ , and  $v_{\phi}^{\mathrm{bonus}}$  then share these representations  $\{\hat{y}_i\}$  as parts of their inputs. Specifically,  $v_{\phi}^{\mathrm{origin}}$  is computed via a four-layer MLP in Eq. (11) with structure  $(d + 1, d, \frac{d}{2}, 1)$  which leverages  $f(\tau_t^{\mathrm{bsf}})$  as additional features;  $v_{\phi}^{\mathrm{reg}}$  is computed via a four-layer MLP in Eq. (12) with structure  $(d + 10, d, \frac{d}{2}, 1)$  which leverages  $f(\tau_t^{\mathrm{bsf}})$  and ES features  $\mathcal{I}_t$  as additional features; and  $v_{\phi}^{\mathrm{bonus}}$  is computed via a four-layer MLP in Eq. (13) with structure  $(d + 1, d, \frac{d}{2}, 1)$  which leverages the best-so-far cost w.r.t.  $\epsilon$ - $\mathcal{F}$  regions  $f(\tau_t^{\mathrm{bsf-wrt-\epsilon}})$  as additional features<sup>4</sup>. All MLPs use the ReLU activation function.
$$
v _ {\phi} ^ {\text {o r i g i n}} = \mathrm {M L P} _ {\phi_ {1}} \left(\max  \left[ \left\{\hat {y} _ {i} \right\} _ {x _ {i} \in \mathcal {V}} \right], \operatorname {m e a n} \left[ \left\{\hat {y} _ {i} \right\} _ {x _ {i} \in \mathcal {V}} \right], f \left(\tau_ {t} ^ {\mathrm {b s f}}\right)\right) \tag {11}
$$
$$
v _ {\phi} ^ {\text {r e g}} = \mathrm {M L P} _ {\phi_ {2}} \left(\max  \left[ \left\{\hat {y} _ {i} \right\} _ {x _ {i} \in \mathcal {V}} \right], \operatorname {m e a n} \left[ \left\{\hat {y} _ {i} \right\} _ {x _ {i} \in \mathcal {V}} \right], f \left(\tau_ {t} ^ {\mathrm {b s f}}\right), \mathcal {J} _ {t}\right) \tag {12}
$$
$$
v _ {\phi} ^ {\text {b o u n s}} = \operatorname {M L P} _ {\phi_ {3}} \left(\max  \left[ \left\{\hat {y} _ {i} \right\} _ {x _ {i} \in \mathcal {V}} \right], \operatorname {m e a n} \left[ \left\{\hat {y} _ {i} \right\} _ {x _ {i} \in \mathcal {V}} \right], f \left(\tau_ {t} ^ {\text {b s f - w r t -} \epsilon}\right)\right) \tag {13}
$$
Illustrations of extreme search behaviour and our GIRE efficacy. Recall that our GIRE considers reward-shaping terms to encourage search within more promising feasibility boundaries and regulate extreme search behaviours using an entropy measure of the estimated conditional transforming probabilities  $P_{t}(\mathcal{U}|\mathcal{U}) = P(\tau^{\prime}\in \mathcal{U}|\tau \in \mathcal{U})$  and  $P_{t}(\mathcal{F}|\mathcal{F}) = P(\tau^{\prime}\in \mathcal{F}|\tau \in \mathcal{F})$ . Figure 8 depicts the persistence of extreme search behaviour when RS is absent while validating the efficacy of our reward shaping (RS) in GIRE. We conduct training with and without RS, recording the convergence curves of the objective values (with mean and confidence intervals) in Figure 8(a), the detailed objective values per run in Figure 8(b), the estimated probability  $P_{t}(\mathcal{F}|\mathcal{F})$  in Figure 8(c), and the estimated probability  $P_{t}(\mathcal{U}|\mathcal{U})$  in Figure 8(d). As revealed in Figure 8(a), without RS, the runs show unstable convergence and poorer final objective values. This can be attributed to higher  $P_{t}(\mathcal{F}|\mathcal{F})$  and lower  $P_{t}(\mathcal{U}|\mathcal{U})$  as shown in Figure 8(c) and Figure 8(d), indicating biased search preference towards feasible regions
<sup>4</sup>Given that GIRE uses additional features, the MDP states should be augmented accordingly.
and inefficient exploration in infeasible regions. In the worst case, training may fail to converge to lower objective values due to extremely low  $P_{t}(\mathcal{U}|\mathcal{U})$  (below 0.1 as shown in Figure 8(d)), indicating extreme search behaviour and entrapment in local optima. Conversely, GIRE fosters moderate search behaviours and promotes exploration in infeasible regions (especially for the boundaries), leading to significantly improved training curves with reduced variance and lower objective values.
(a) Objective values.
Figure 8: Training curves of our GIRE with and without RS on CVRP-20 (8 runs).
(b) Objective values.
(c)  $P_{t}(\mathcal{F}|\mathcal{F})$
(d)  $P_{t}(\mathcal{U}|\mathcal{U})$
Influence of  $\epsilon$ . We employ a deterministic rule to decide  $\epsilon$ , i.e.,  $\epsilon = \zeta \times N_{\mathrm{customer}} \times (\bar{\delta} / \Delta)$ , where  $\zeta$  is a coefficient within  $[0,1]$ ,  $\Delta$  is the vehicle capacity, and  $\bar{\delta}$  represents the average customer demands. This formula implies that the total violation corresponds to scenarios where  $\zeta$ -ratio of customers, each with the average demand, breaches the constraint. Empirically, we set  $\zeta = 0.1$ . Figure 9 illustrates the impact of varying  $\zeta$ .
Figure 9: Impact of varying  $\zeta$ .
Influence of  $c_{1}$  and  $c_{2}$ . Recall that in the entropy measure defined by Eq. (5), we introduce two hyper-parameters  $c_{1}$  and  $c_{2}$  to shape the entropy measure pattern  $\mathbb{H}[P]$ , where the measure  $\mathbb{H}[P]$  is used to penalize extreme search behaviours, particularly when values of  $P$  approach 0 or 1. Figure 10 illustrates how hyper-parameters  $c_{1}$  and  $c_{2}$  influence the shape of  $\mathbb{H}[P]$ . When  $c_{2}$  is fixed, a smaller  $c_{1}$  expands the penalty range, while a larger  $c_{1}$  constricts the penalty range. Similarly, when  $c_{1}$  is fixed, a smaller or a larger  $c_{2}$  will also control the shape of the patterns. Empirically, we set the values of  $c_{1} = 0.5, c_{2} = 2.5$  so as to only penalize extreme search behaviour if the feasibility transition probability is outside the [0.25, 0.75] range. In Figure 11, we investigate the influence of  $c_{1}, c_{2}$  on the training stability of our NeuOpt-GIRE approach on CVRP-20. Results show that they may not affect training stability (thus no need for extensive tuning in practical usage).
(a)  $c_{1} = 0.1, c_{2} = 2.5$
(d)  $c_{1} = 0.5, c_{2} = 1.5$
Figure 10: Effects of  $c_{1}$  and  $c_{2}$  on  $\mathbb{H}[P]$  pattern: (a)-(c) fix  $c_{2} = 2.5$  and vary  $c_{1}$ ; (d)-(f) fix  $c_{1} = 0.5$  and vary  $c_{2}$ . Compared to the pattern (b) and (e) used in this paper, varying  $c_{1}$  and  $c_{2}$  either constricts the penalty range, shown in (c) and (f), or expands the penalty range, shown in (a) and (d).
(b)  $c_{1} = 0.5, c_{2} = 2.5$
(e)  $c_{1} = 0.5, c_{2} = 2.5$
(c)  $c_{1} = 1.0, c_{2} = 2.5$
(f)  $c_{1} = 0.5, c_{2} = 3.5$
Figure 11: Influence of constricted and expanded patterns of  $\mathbb{H}[P]$  on the training stability.
Applying GIRE to other constraints. Our GIRE can be viewed as an important early attempt that moves beyond the pure feasibility masking to autonomously explore both feasible and infeasible regions during the search. To adapt GIRE for a particular constraint, we suggest the following:
- For Feature Supplement, simply binary indicator functions can be employed to discern specific constraint violations of each node in the solution, thereby forming the Violation Indicator (VI) features. For example, VI can indicate nodes (customers) in the solution that breach their time window or the pickup/delivery precedence constraints. For the Exploration Statistics (ES) features, they can be retained as originally conceptualized since they are based on historical exploration records, rather than specific constraints.
- For Reward Shaping, it can be retained as originally conceptualized. We suggest characterizing the  $\epsilon$ -feasible regions by a fraction of nodes that do not adhere to the constraints, i.e.,  $\zeta = 0.1$ .
E.1 Details of implementation
Hyper-parameter details. We use  $d = 128$ ,  $T_{\mathrm{D2A}} = 10$ ,  $T_{\mathrm{his}} = 25$  for NeuOpt, and employ 4 attention heads in both the encoders and critics. The ReLU activation function is utilized for MLPs within the network. In line with the training algorithms in [9, 12], we retain their hyper-parameters to ensure an equivalent training amount. Training involves  $E = 200$  epochs and  $B = 20$  batches per epoch, with batch sizes of 512 (TSP) and 600 (CVRP). For TSP, we use  $n = 4$ ,  $T_{\mathrm{train}} = 200$ ; and for CVRP, we use  $n = 5$ ,  $T_{\mathrm{train}} = 250$ . The PPO inner loops is  $\Omega = 3$  and the clip threshold is  $\vartheta = 0.1$ . The Adam optimizer is employed with learning rates  $\eta_{\theta} = 8 \times 10^{-5}$  for  $\pi_{\theta}$ ,  $\eta_{\phi} = 2 \times 10^{-5}$  for  $v_{\phi}$ , as well as a decay rate  $\varsigma = 0.985$ . The reward discount factor  $\gamma$  is set to 0.999 for both problems. Besides, we empirically set approach-specific parameters: the gradient norm of NeuOpt is clipped at 0.05 in all cases, and the curriculum learning scalar  $\xi$  is set as 1, 0.5, and 0.25 for sizes 20, 50, and 100, respectively. The training time varies depending on the specific problem and size, e.g., CVRP requires about 4 days for size 20 (1 GPU), 5 days for size 50 (2 GPUs), and 8 days for size 100 (4 GPUs), which are around half the time taken as reported in DACT [9] and POMO [4] on CVRP100.
Setup details. We follow Kool et al. [3] to sample all training and test instance coordinates within the unit square  $[0,1]\times [0,1]$  uniformly. For CVRP, the demands of customers are uniformly sampled from the set  $\{1,2,\dots,9\}$  and the vehicle capacity is set to 30, 40, and 50 for sizes 20, 50, and 100 respectively (we thus estimate  $\bar{\delta}$  as  $5 / 30$ ,  $5 / 40$ ,  $5 / 50$  in our GIRE, respectively). To facilitate GPU parallelization given the varying lengths of CVRP solutions, we employ the dummy depots design in [9, 39] for length padding, where depots are duplicated during both training and inference. Same as DACT [9], we include 10, 20, and 20 dummy depots for CVRP sizes of 20, 50, and 100 respectively.
Baseline details. We provide details on the compared neural baselines. For the ones with  $\#$ , we found certain issues in their code, which may lead to the underestimation of reported gaps by around  $0.02\%$  (TSP-50) and  $0.04\%$  (TSP-100). We observed that they used 'GEO' (Geographical distance) instead of 'EUC' (Euclidean distance) as the type of edge weights while running the Concorde to determine the optimal solutions. This results in the optimal solutions (optimal in the 'GEO' setting) no longer being optimal in the 'EUC' settings, while the objectives derived by neural solvers are based on Euclidean distances. This discrepancy might be the reason for the reported negative optimality gaps on TSP, which seems anomalous as Concorde is guaranteed to produce optimal solutions.
- GCN+BS [14]#: a classic L2P solver that predicts heatmaps for TSP based on GCN. We run its star version that considers both beam search (1,280 beams) and shortest tour heuristics.