SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\tau_ {1} \sim \pi_ {\theta_ {1}}} \max  _ {i \in \{2, \dots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}) = \max  _ {i \in \{2, \dots , K \}} \left[ R (\tau_ {i}) \right] \mathbb {E} _ {\tau_ {1} \sim \pi_ {\theta_ {1}}} \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}) \\ = \max _ {i \in \{2, \dots , K \}} \left[ R (\tau_ {i}) \right] \mathbb {E} _ {\tau_ {1} \sim \pi_ {\theta_ {1}}} \frac {\nabla_ {\theta_ {1}} \pi_ {\theta_ {1}} (\tau_ {1})}{\pi_ {\theta_ {1}} (\tau_ {1})} \\ = \max  _ {i \in \{2, \dots , K \}} \left[ R (\tau_ {i}) \right] \sum_ {\tau_ {1}} \nabla_ {\theta_ {1}} \pi_ {\theta_ {1}} (\tau_ {1}) \\ = \max  _ {i \in \{2, \dots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} \sum_ {\tau_ {1}} \pi_ {\theta_ {1}} (\tau_ {1}) \\ = \max  _ {i \in \{2, \dots , K \}} \left[ R \left(\tau_ {i}\right) \right] \nabla_ {\theta_ {1}} 1 = 0 \\ \end{array}
$$
Intuitively,  $\max_{i\in \{2,\dots ,K\}}\bigl [R(\tau_i)\bigr ]$  does not depend on the first agent, so this derivation simply shows that  $\max_{i\in \{2,\dots ,K\}}\bigl [R(\tau_i)\bigr ]$  can be used as a baseline for training  $\theta_{1}$
Subtracting this to the quantity obtained in Equation B.1, we have:
$$
\begin{array}{l} \nabla_ {\theta_ {1}} J _ {\mathrm {p o p}} (\theta_ {1}, \theta_ {2}, \ldots , \theta_ {K}) = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \ldots , \tau_ {K}} \max _ {i \in \{1, 2, \ldots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}), \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {2}, \ldots , \tau_ {K}} \mathbb {E} _ {\tau_ {1}} \max _ {i \in \{1, 2, \ldots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}), \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {2}, \dots , \tau_ {K}} \mathbb {E} _ {\tau_ {1}} \left(\max  _ {i \in \{1, 2, \dots , K \}} \left[ R (\tau_ {i}) \right] - \max  _ {i \in \{2, \dots , K \}} \left[ R (\tau_ {i}) \right]\right) \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}), \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {2}, \dots , \tau_ {K}} \mathbb {E} _ {\tau_ {1}} \mathbb {1} _ {i ^ {*} = 1} \left(R (\tau_ {1}) - R (\tau_ {i ^ {* *}})\right) \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}), \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \mathbb {1} _ {i ^ {*} = 1} \left(R \left(\tau_ {i ^ {*}}\right) - R \left(\tau_ {i ^ {* *}}\right)\right) \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}). \\ \end{array}
$$
Equation (B.1) comes from the fact that  $\left(\max_{i\in \{1,2,\dots,K\}}\bigl [R(\tau_i)\bigr ] - \max_{i\in \{2,\dots,K\}}\bigl [R(\tau_i)\bigr ]\right)$  is 0 if the best trajectory is not  $\tau_{1}$ , and  $R(\tau_1) - \max_{i\in \{2,\dots,K\}}\bigl [R(\tau_i)\bigr ] = R(\tau_1) - R(\tau_{i^{**}})$  otherwise.
Finally, for any  $j \in \{1, \dots, K\}$ , the same derivation gives:
$$
\nabla_ {\theta_ {j}} J _ {\mathrm {p o p}} \left(\theta_ {1}, \theta_ {2}, \dots , \theta_ {K}\right) = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \mathbb {1} _ {i ^ {*} = j} \left(R \left(\tau_ {i ^ {*}}\right) - R \left(\tau_ {i ^ {* *}}\right)\right) \nabla_ {\theta_ {j}} \log \pi_ {\theta_ {j}} \left(\tau_ {j}\right).
$$
$$
\nabla_ {\theta} = \sum_ {j = 1} ^ {n} \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \mathbb {1} _ {i ^ {*} = j} \left(R \left(\tau_ {i ^ {*}}\right) - R \left(\tau_ {i ^ {* *}}\right)\right) \nabla_ {\theta_ {j}} \log \pi_ {\theta_ {j}} (\tau_ {j}),
$$
$$
\nabla_ {\theta} = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \left(R \left(\tau_ {i ^ {*}}\right) - R \left(\tau_ {i ^ {* *}}\right)\right) \nabla_ {\theta_ {i ^ {*}}} \log \pi_ {\theta_ {i ^ {*}}} \left(\tau_ {i ^ {*}}\right),
$$
C Baseline Impact Analysis
The results presented in the paper use the same reinforcement learning baseline as POMO [Kwon et al., 2020], the average reward over starting points, as presented in Alg. 2. However, Theorem 1 directly provides an alternative baseline: the reward of the second-best agent. Figures 5-7 compare the performance of Poppy using the baseline from POMO against Poppy using the baseline analytically computed in Theorem 1.
In almost all cases, we observe that the analytical baseline provides better or equal performance than the POMO baseline. We believe that this highlights both that our intuitive "winner takes it all" approach works well even with slightly different choices of baseline, and that indeed our theoretical analysis correctly predicts a modified baseline that can be used in practice with strong performance.
Table 5: TSP results.
Method	Inference (10k instances)	0-shot (1k instances)
n = 100	n = 125	n = 150
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
Concorde	7.765	0.000%	82M	8.583	0.000%	12M	9.346	0.000%	17M
LKH3	7.765	0.000%	8H	8.583	0.000%	73M	9.346	0.000%	99M
POMO 16 (ensemble)	7.790	0.33%	1M	8.629	0.53%	10S	9.435	0.95%	20S
Poppy 16 (POMO baseline)	7.770	0.07%	1M	8.594	0.14%	10S	9.372	0.27%	20S
Poppy 16 (analytical baseline)	7.769	0.05%	1M	8.594	0.13%	10S	9.372	0.28%	20S
Table 6: CVRP results.
Method	Inference (10k instances)	0-shot (1k instances)	0-shot (1k instances)
n = 100	Time	n = 125	Time	n = 150	Time
Obj.	Gap	Obj.	Gap	Obj.	Gap
LKH3	15.65	0.000%	6D	17.50	0.000%	19H	19.22	0.000%	20H
POMO 32 (ensemble)	15.78	0.83%	5M	17.70	1.11%	1M	19.57	1.83%	1M
Poppy 32 (POMO baseline)	15.73	0.52%	5M	17.63	0.71%	1M	19.50	1.43%	1M
Poppy 32 (analytical baseline)	15.72	0.48%	5M	17.63	0.71%	1M	19.50	1.43%	1M
Table 7: Packing problems results.
(a) KP	(b) JSSP
Method	Testing (10k instances)	Method	Testing (100 instances)
n = 100	10 × 10
Obj.	Gap	Time	Obj.	Gap	Time
Optimal	40.437	-	OR-Tools (optimal)	807.6	0.0%	37S
Greedy	40.387	0.1250%	Single (16 samples)	866.0	7.2%	30M
POMO 16 (ensemble)	40.429	0.021%	2M	Poppy 16 (POMO baseline)	857.7	6.2%	30M
Poppy 16 (analytical baseline)	40.437	0.0005%	2M	Poppy 16 (analytical baseline)	852.2	5.5%	30M
D Comparison to Active Search
We implement a simple sampling strategy to give a sense of the performance of Poppy with a larger time budget. Given a population of  $K$  agents, we first greedily rollout each of them on every starting point, and evenly distribute any remaining sampling budget across the most promising  $K$  (agent, starting point) pairs for each instance with stochastic rollouts. This two-step process is motivated by the idea that is not useful to sample several times an agent on an instance where it is outperformed by another one. For environments without starting points like JSSP, we stick to the simplest approach of evenly distributing the rollouts across the population, although better performance could likely be obtained by selectively assigning more budget to the best agents.
Setup For TSP, CVRP and JSSP, we use the same test instances as in Tables 1, 2 and 3b. For TSP and CVRP, we generate a total of  $200 \times 8 \times N$  candidate solutions per instance (where 8 corresponds to the augmentation strategy by Kwon et al. [2020] and  $N$  is the number of starting points), accounting for both the first and second phases. We evaluate our approach against POMO [Kwon et al., 2020], EAS [Hottung et al., 2022] with the same budget, and SGBS [Choo et al., 2022], as well as against the supervised methods GCN-BS [Joshi et al., 2019], CVAE-Opt [Hottung et al., 2021], and DPDP [Kool et al., 2021]. As EASand SGBS have different variants, we compare against those that do not require backpropagating gradients through the network, similarly to our approach; thus, it should match Poppy's compute time on the same hardware. We additionally compare against the evolutionary approach HGS [Vidal et al., 2012, Vidal, 2022]. For JSSP, we use the same setting as EAS [Hottung et al., 2022], and sample a total of 8,000 solutions per problem instance for each approach. For a proper comparison, we reimplemented EAS with the same model architecture as Poppy.
Results Tables 8, 9 and 10 show the results for TSP, CVRP and JSSP respectively. With extra sampling, Poppy reaches a performance gap of  $0.002\%$  on TSP100, and establishes a state-of-the-art for general ML-based approaches, even when compared to supervised methods. For CVRP, adding sampling to Poppy makes it on par with DPDP and EAS, depending on the problem size, and it is only outperformed by the active search approach EAS, which gives large improvements on CVRP. As the two-step sampling process used for Poppy is very rudimentary compared to the active search method described in Hottung et al. [2022], it is likely that combining the two approaches could further boost performance, which we leave for future work.
Table 8: TSP results (active search)
Method	Inference (10k instances)	0-shot (1k instances)
n = 100	n = 125	n = 150
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
Concorde	7.765	0.000%	82M	8.583	0.000%	12M	9.346	0.000%	17M
LKH3	7.765	0.000%	8H	8.583	0.000%	73M	9.346	0.000%	99M
SL	GCN-BS	7.87	1.39%	40M*	-	-	-	-	-	-
CVAE-Opt	-	0.343%	6D*	8.646	0.736%	21H*	9.482	1.45%	30H*
DPDP	7.765	0.004%	2H*	8.589	0.070%	31M*	9.434	0.94%	44M*
RL	POMO (200 samples)	7.769	0.056%	2H	8.594	0.13%	20M	9.376	0.31%	32M
SGBS	7.769	0.058%	9M*	-	-	-	9.367	0.22%	8M
EAS	7.768	0.048%	5H*	8.591	0.091%	49M*	9.365	0.20%	1H*
Poppy 16 (200 samples)	7.765	0.002%	2H	8.584	0.009%	20M	9.351	0.05%	32M
Table 9: CVRP results (active search)
Method	Inference (10k instances)	0-shot (1k instances)	0-shot (1k instances)
n = 100	n = 125	n = 150
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
HGS	15.56	0.00%	3D	17.37	0.00%	12H	19.05	0.00%	16H
LKH3	15.65	0.53%	6D	17.50	0.75%	19H	19.22	0.89%	20H
SL	CVAE-Opt	-	1.90%	11D*	17.87	2.85%	36H*	19.84	4.16%	46H*
DPDP	15.63	0.40%	23H*	17.51	0.82%	3H*	19.31	1.38%	5H*
RL	POMO (200 samples)	15.67	0.71%	4H	17.56	1.08%	43M	19.43	1.98%	1H
SGBS	15.66	0.61%	10M*	-	-	-	19.43	1.96%	4M*
EAS	15.62	0.39%	8H*	17.49	0.75%	80M*	19.36	1.62%	2H*
Poppy 32 (200 samples)	15.62	0.39%	4H	17.49	0.65%	42M	19.32	1.40%	1H
We here describe the details of the four CO problems we have used to evaluate Poppy, namely TSP, CVRP, KP and JSSP. We use the corresponding implementations from Jumanji [Bonnet et al., 2023]: TSP, CVRP, Knapsack and JobShop. For each problem, we describe below the training (e.g. architecture, hyperparameters) and the process of instance generation. In addition, we show some example solutions obtained by a population of agents on TSP and CVRP. Finally, we thoroughly analyze the performance of the populations in TSP.
E.1 Traveling Salesman Problem (TSP)
Instance Generation The  $n$  cities that constitute each problem instance have their coordinates uniformly sampled from  $[0,1]^2$ .
Architecture We use the same model as Kool et al. [2019] and Kwon et al. [2020] except for the batch-normalization layers, which are replaced by layer-normalization to ease parallel batch processing. We invert the mask used in the decoder computations (i.e., masking the available cities instead of the unavailable ones) after experimentally observing faster convergence rates. The results
Table 10: JSSP
Method	Inference (100 instances)
10 × 10
Obj.	Gap	Time
OR-Tools (optimal)	807.6	0.0%	37S
L2D (Greedy)	988.6	22.3%	20S*
L2D (Sampling)	871.7	8.0%	8H*
EAS-L2D	860.2	6.5%	8H*
Sampling	862.1	6.7%	3H
EAS	858.4	6.3%	3H
Poppy 16	849.7	5.2%	3H
reported for POMO were obtained with the same implementation changes to keep the comparison fair. These results are on par with those reported in POMO [Kwon et al., 2020].
Hyperparameters To match the setting used by Kwon et al. [2020], we use the Adam optimizer [Kingma and Ba, 2015] with a learning rate  $\mu = 10^{-4}$ , and an  $L_{2}$  penalization coefficient of  $10^{-6}$ . The encoder is composed of 6 multi-head self-attention layers with 8 heads each. The dimension of the keys, queries and values is 16. Each attention layer is composed of a feed-forward layer of size 512, and the final node embeddings have a dimension of 128. The decoders are composed of 1 multi-head attention layer with 8 heads and 16-dimensional key, query and value.
The number of starting points  $P$  is 50 for each instance. We determined this value after performing a grid-search based on the first training steps with  $P \in \{20, 50, 100\}$ .
Example Solutions Figure 5 shows some trajectories obtained from a 16-agent population on TSP100. Even though they look similar, small decisions differ between agents, thus frequently leading to different solutions. Interestingly, some agents (especially 6 and 11) give very poor trajectories. We hypothesize that it is a consequence of specializing since agents have no incentive to provide a good solution if another agent is already better on this instance.
Figure 4: Upper left: Proportion of instances that each agent solves best among the population for Poppy 16 on TSP100. Colors indicate the number of agents in the population giving the same solution for these sets of instances. Upper right: The mean performance of 1,000 randomly drawn sub-populations for Poppy 1, 4, 8 and 16. Bottom left: Proportion of test instances where any number of Poppy 16 agents reaches the exact same best solution. The best performance is reached by only a single agent in  $47\%$  of the cases. Bottom Right: Optimality gap loss suffered when removing any agent from the population using Poppy 16. Although some agents contribute more (e.g. 7, 8) and some less (e.g. 15, 16), the distribution is relatively even, even though no explicit mechanism enforces this behavior
Population Analysis Figure 4 shows some additional information about individual agent performances. In the left figure, we observe that each agent gives on average the best solution for  $20\%$  of the instances, and that for around  $4\%$  it gives the unique best solution across the population. These numbers are generally evenly distributed, which shows that every agent contributes to the whole population performance. Furthermore, we observe the performance is quite evenly distributed
across the population of Poppy 16; hence, showing that the population has not collapsed to a few high-performing agents, and that Poppy benefits from the population size, as shown in the bottom figure. On the right is displayed the performance of several sub-populations of agents for Poppy 4, 8 and 16. Unsurprisingly, fixed size sub-populations are generally better when sampled from smaller populations: one or two agents randomly sampled from Poppy 4 perform better than when sampled from Poppy 8 or Poppy 16. This highlights the fact that agents have learned complementary behaviors which might be sub-optimal if part of the total population is missing, and this effect is stronger for large populations.
total length 7.581
total length 7.578
total length 7.771
total length 7.800
total length 7.573
total length 7.587
Figure 5: Example TSP trajectories given by Poppy for a 16-agent population from one starting point (red).
E.2 Capacitated Vehicle Routing Problem (CVRP)
Instance Generation The coordinates of the  $n$  customer nodes and the depot are uniformly sampled in  $[0,1]^2$ . The demands are uniformly sampled from the discrete set  $\left\{\frac{1}{D}, \frac{2}{D}, \dots, \frac{9}{D}\right\}$  where  $D = 50$  for CVRP100,  $D = 55$  for CVRP125, and  $D = 60$  for CVRP150. The maximum vehicle capacity is 1. The deliveries cannot be split: each customer node is visited once, and its whole demand is taken off the vehicle's remaining capacity.
Architecture We use the same model as in TSP. However, unlike TSP, the mask is not inverted; besides, it does not only prevent the agent from revisiting previous customer nodes, but also from
visiting the depot if it is the current location, and any customer node whose demand is higher than the current capacity.
Hyperparameters We use the same hyperparameters as in TSP except for the number of starting points  $P$  per instance used during training, which we set to 100 after performing a grid-search with  $P \in \{20, 50, 100\}$ .
Example Solutions Figure 6 shows some trajectories obtained by 16 agents from a 32-agent population on CVRP100. Unlike TSP, the agent/vehicle performs several tours starting and finishing in the depot.
12 routes, distance 16.04
12 routes, distance 16.16
12 routes, distance 16.63
13 routes, distance 16.23
12 routes, distance 15.86
12 routes, distance 16.20
12 routes. distance 16.16
12 routes, distance 16.05
12 routes, distance 16.24
12 routes, distance 16.56
13 routes, distance 16.17
12 routes, distance 16.02
Figure 6: Example CVRP trajectories given by Poppy for 16 agents from a 32-agent population. The depot is displayed as a black square. The edges from/to the depot are omitted for clarity.
13 routes, distance 16.16
12 routes, distance 15.93
12 routes. distance 16.08
E.3 0-1 Knapsack (KP)
Problem Description Given a set of items, each with a weight and a value, the goal is to determine which items to include in a collection so that the total weight is less than or equal to a given limit and the total value is as large as possible.
Instance Generation Item values and weights are both uniformly sampled in  $[0,1]$ . The bag capacity is fixed at 25.
Training For KP, and contrary to the other three environments, training an agent is lightning-fast as it only takes a few minutes. In this specific case, we noticed it was not necessary to train a single decoder first. Instead, (i) we directly train a population in parallel from scratch, and (ii) specialize the population exactly as done in the other environments.
Architecture We use the same model as in TSP. However, the mask used when decoding is not inverted, and the items that do not fit in the bag are masked together with the items taken so far.
Hyperparameters We use the same hyperparameters as in TSP except for the number of starting points  $P$  used during training, which we set to 100 after performing a grid-search with  $P \in \{20,50,100\}$ .
E.4 Job-Shop Scheduling Problem (JSSP)
Problem Description We consider the problem formulation described by Zhang et al. [2020] and also used in Bonnet et al. [2023], in the setting of an equal number of machines, jobs and operations per job. A job-shop scheduling problem consists in  $N$  jobs that all have  $N$  operations that have to be scheduled on  $N$  machines. Each operation has to run on a specific machine for a given time. The solution to a problem is a schedule that respects a few constraints:
- for each job, its operations have to be processed/scheduled in order and without overlap between two operations of the same job,
- a machine can only work on one operation at a time,
- once started, an operation must run until completion.
The goal of the problem is to determine a schedule that minimizes the time needed to process all the jobs. The length of the schedule is also known as its makespan.
Instance Generation We use the same generation process as Zhang et al. [2020]. For each of the  $N$  jobs, we sample  $N$  operation durations uniformly in  $[1,99)$ . Each operation is given a random machine to run on by sampling a random permutation of the machine IDs.
Transition Function To leverage JAX, we use the environment dynamics implemented in Ju manji [Bonnet et al., 2023] which differs from framing proposed by L2D [Zhang et al., 2020]. However, the two formulations are equivalent, therefore our results on the former would transfer to the latter. Our implementation choice was purely motivated by environment speed.
Architecture We use the actor-critic transformer architecture implemented in Jumanji which is composed of an attention layer on the machines' status, an attention layer on the operation durations (with positional encoding) and then two attention layers on the joint sequence of jobs and machines. The network outputs N marginal categorical distributions for all machines, as well as a value for the critic. The actor and critic networks do not share any weights.
Hyperparameters Like in Zhang et al. [2020], we evaluate our algorithm with  $N = 10$  jobs, operations and machines, i.e.  $10 \times 10$  instances. We use REINFORCE with the critic as a baseline (state-value function). Since episodes may take a long time (for an arbitrary policy, the lowest upper bound on the horizon is  $98 \times N^3$ ), we use an episode horizon of 1250 and give a reward penalty of two times the episode horizon when producing a makespan of more than this limit.
F Time-performance Tradeoff
We present in figure 7 a comparison of the time-performance Pareto front between Poppy and POMO as we vary respectively the population size and the amount of stochastic sampling. Poppy consistently provides better performance for a fixed number of trajectories. Strikingly, in almost every setting, matching Poppy's performance by increasing the number of stochastic samples does not appear tractable.
(a) TSP100
(b) TSP125
(c) TSP150
(d) CVRP100
(e) CVRP125
(f) CVRP150
(g) KP100
(h) JSSP100
Figure 7: Comparison of the time-performance Pareto front of Poppy and POMO, for each problem used in the paper. The x-axis is the number of trajectories sampled per test instance, while the y-axis is the gap with the optimal solution for TSP, KP and JSSP, and the gap with LKH3 for CVRP.
G Larger instances
We display in Table 11 results for POMO and Poppy on much larger TSP instances with 1000 cities. Baselines are taken from Qiu et al. [2023].
Table 11: TSP results (TSP 1000)
Method	Type	Obj.	Gap	Time
Concorde	OR	23.12	0.000%	6.65H
LKH3	OR	23.12	0.000%	2.57H
EAS (Tab)	RL w/ search	49.56	114%	63H
Dimes	RL w/ search	23.69	2.46%	5H
POMO (16 samples)	RL w/o search	50.80	120%	39M
Poppy 16	RL w/o search	39.70	71.7%	39M