SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training One training step corresponds to computing policy gradients over a batch of 64 (TSP, CVRP, KP) or 128 (JSSP) randomly generated instances for each agent in the population. Training time varies with problem complexity and training phase. For instance, in TSP with 100 cities, Phase 1 takes 4.5M steps (5 days), whereas Phase 2 takes 400k training steps and lasts 1-4 days depending on the population size. We took advantage of our JAX-based implementation by running all our experiments on a v3-8 TPU.
4.1 Routing Problems (TSP, CVRP)
Tasks We consider two routing tasks: TSP and CVRP. Given a set of  $n$  cities, TSP consists in visiting every city once and coming back to the starting city while minimizing the total traveled distance. CVRP is a variant of TSP where a vehicle with limited capacity departs from a depot node and needs to fulfill the demands of the visited nodes. The vehicle's capacity is restored when the depot is visited.
Setup We use an encoder-decoder architecture based on that by Kool et al. [2019] and Kwon et al. [2020] (see Appendix E for details). During Phase 1, an architecture with a single decoder is trained; then, the decoder is cloned for each agent in the population and trained according to Algorithm 1. Following Kwon et al. [2020], we generate multiple solutions for each instance  $\rho$  by considering a set of  $P \in [1, N]$  starting points, where  $N$  is the number of instance variables, and use the same reinforce baseline at training. We refer the reader to Appendix A.2 for details.
The training instances are of size  $n = 100$  in both tasks. The testing instances for  $n = 100$  are due to Kool et al. [2019] and Kwon et al. [2020] for TSP and CVRP, respectively, whereas those for  $n \in \{125, 150\}$  are due to Hottung et al. [2022].
Baselines We use the exact TSP-solver Concorde Applegate et al. [2006] and the heuristic solver HGS [Vidal et al., 2012, Vidal, 2022] to compute the optimality gaps for TSP and CVRP, respectively. The performance of LKH3 [Helsgaun, 2017] is also reported for TSP. We highlight that the performance of HGS is not necessarily optimal.
In both TSP and CVRP, we evaluate the performance of the RL methods MDAM [Xin et al., 2021] with and without beam search, LIH [Wu et al., 2021], and POMO [Kwon et al., 2020] in three settings: (i) with greedy rollouts, (ii) using  $r$  stochastic rollouts, and (iii) using an ensemble of  $r$  decoders trained in parallel. The value of  $r$  corresponds to the largest tested population (i.e., 16 in TSP and 32 in CVRP). Setting (ii) is run to match Poppy's runtime with its largest population, while Setting (iii) performs Phase 2 without the population objective. We also report the performance of the RL methods Att-GCRN+MCTS [Fu et al., 2021] and 2-Opt-DL [de O. da Costa et al., 2020] in TSP, and NeuRewriter [Chen and Tian, 2019] and NLNS [Hottung and Tierney, 2020] in CVRP.
Results Tables 1 and 2 display the results for TSP and CVRP, respectively. The columns show the average tour length, the optimality gap, and the total runtime for each test set. The baseline performances from Fu et al. [2021], Xin et al. [2021], Hottung et al. [2022] and Zhang et al. [2020] were obtained with different hardware (Nvidia GTX 1080 Ti, RTX 2080 Ti, and Tesla V100 GPUs, respectively) and framework (PyTorch); thus, for fairness, we mark these times with  $*$ . As a comparison guideline, we informally note that these GPU inference times should be approximately divided by 2 to get the converted TPU time.
In both TSP and CVRP, Concorde and HGS remain the best algorithms as they are highly specialized solvers. In relation to RL methods, Poppy reaches the best performance across every performance metric in just a few minutes and, remarkably, performance improves as populations grow.
In TSP, Poppy outperforms Att-GCRN+MCTS despite the latter being known for scaling to larger instances than  $n = 100$ , hence showing it trades off performance for scale. We also emphasize that specialization is crucial to achieving state-of-the-art performance: Poppy 16 outperforms POMO 16 (ensemble), which also trains 16 agents in parallel but without the population objective (i.e. without specializing to serve as an ablation of the objective).
In CVRP, we observe Poppy 32 has the same runtime as POMO with 32 stochastic rollouts while dividing by 2 the optimal gap for  $n = 100$ .<sup>2</sup> Interestingly, this ratio increases on the generalization instance sets with  $n \in \{125, 150\}$ , suggesting that Poppy is more robust to distributional shift.
Analysis Figure 3 illustrates the resulting behavior from using Poppy on TSP100. We display on the left the training curves of Poppy for three population sizes: 4, 8, 16. Starting from POMO, the population performances quickly improve. Strikingly, Poppy 4 outperforms POMO with 100 stochastic samples, despite using 25 times fewer rollouts. The rightmost plot shows that whilst the population-level performance improves with population size, the average performance of a random agent from the population on a random instance gets worse. We hypothesize that this is due to agent specialization: when each agent has a narrower target sub-distribution, it learns an even more specialized policy, which is even better (resp. worse) for problem instances in (resp. out) of the target sub-distribution. This is in contrast with a simple policy ensemble, for which the average agent performance would remain the same regardless of the population size. Additional analyses are made in Appendix E.1, where we show that every agent contributes to the population performance. Overall, this illustrates that Poppy agents have learned complementary policies: though they individually
${}^{2}$  A fine-grained comparison between POMO with stochastic sampling and Poppy is in Appendix F.
Table 1: TSP results.
Method	Inference (10k instances)	0-shot (1k instances)
n = 100	n = 125	n = 150
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
Concorde	7.765	0.000%	82M	8.583	0.000%	12M	9.346	0.000%	17M
LKH3	7.765	0.000%	8H	8.583	0.000%	73M	9.346	0.000%	99M
MDAM (greedy)	7.93	2.19%	36S*	-	-	-	-	-	-
MDAM (beam search)	7.79	0.38%	44M*	-	-	-	-	-	-
Att-GCRN+MCTS	-	0.037%	15M*	-	-	-	-	-	-
2-Opt-DL	7.83	0.87%	41M*	-	-	-	-	-	-
LIH	7.87	1.42%	2H*	-	-	-	-	-	-
POMO	7.796	0.41%	5S	8.635	0.61%	1S	9.439	0.99%	1S
POMO (16 samples)	7.785	0.27%	1M	8.619	0.42%	10S	9.423	0.81%	20S
POMO 16 (ensemble)	7.790	0.33%	1M	8.629	0.53%	10S	9.435	0.95%	20S
Poppy 4	7.777	0.15%	20S	8.605	0.26%	3S	9.389	0.46%	5S
Poppy 8	7.772	0.09%	40S	8.598	0.18%	6S	9.379	0.35%	10S
Poppy 16	7.770	0.07%	1M	8.594	0.14%	10S	9.372	0.27%	20S
Table 2: CVRP results.
Method	Inference (10k instances)	0-shot (1k instances)	0-shot (1k instances)
n = 100	n = 125	n = 150
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
HGS	15.56	0.000%	3D	17.37	0.000%	12H	19.05	0.000%	16H
LKH3	15.65	0.53%	6D	17.50	0.75%	19H	19.22	0.89%	20H
MDAM (greedy)	16.40	5.38%	45S*	-	-	-	-	-	-
MDAM (beam search)	15.99	2.74%	53M*	-	-	-	-	-	-
LIH	16.03	3.00%	5H*	-	-	-	-	-	-
NeuRewriter	16.10	3.45%	66M*	-	-	-	-	-	-
NLNS	15.99	2.74%	62M*	18.07	4.00%	9M*	19.96	4.76%	12M*
POMO	15.87	2.00%	10S	17.82	2.55%	2S	19.75	3.65%	3S
POMO (32 samples)	15.77	1.31%	5M	17.69	1.82%	1M	19.61	2.94%	1M
POMO 32 (ensemble)	15.78	1.36%	5M	17.70	1.87%	1M	19.57	2.73%	1M
Poppy 4	15.80	1.54%	1M	17.72	2.01%	10S	19.61	2.95%	20S
Poppy 8	15.77	1.31%	2M	17.68	1.78%	20S	19.58	2.75%	40S
Poppy 32	15.73	1.06%	5M	17.63	1.47%	1M	19.50	2.33%	1M
perform worse than the single agent baseline POMO, together they obtain better performance than an ensemble, with every agent contributing to the whole population performance.
4.2 Packing Problems (KP, JSSP)
Tasks To showcase the versatility of Poppy, we evaluate our method on two packing problems: KP and JSSP. In KP, the goal is to maximize the total value of packed items. JSSP can be seen as a 2D packing problem in which the goal is to minimize the schedule makespan under a set of constraints.
Setup For KP, we employ the same architecture and training as for TSP and CVRP. For JSSP, we use an attention-based actor-critic architecture proposed by Bonnet et al. [2023] and use REINFORCE with a critic as a baseline to train our agents. Furthermore, akin to the other problems, we first train the agent until convergence and then duplicate the heads to make our population (see Appendix E for more details). In both problems, the evaluation budget consists of 1,600 samples per instance distributed over the population.
Baselines We evaluate our method on KP against several baselines including the optimal solution based on dynamic programming, a greedy heuristic, and POMO [Kwon et al., 2020]. For JSSP, we compare Poppy against the optimal solution obtained by Google OR-Tools [Perron and Furnon, 2019], L2D [Zhang et al., 2020] with greedy rollouts and stochastic sampling (consisting of 8,000 samples
Figure 3: Analysis of Poppy on TSP100. Left: Starting from POMO clones, training curves of Poppy for three different population sizes (4, 8, 16). Right: With the population objective, the average performance gets worse as the population size increases, but the population-level performance improves.
per problem instance) and our own implementation of MDAM using the same network architecture and budget as Poppy.
Results Table 3 shows the results for KP (3a) and JSSP (3b). The columns indicate the final performance on the test set, which is the average values of items packed for KP and the average schedule duration for JSSP, the optimality gap and the total runtime of these evaluations. The L2D results were obtained on an Intel Core i9-10940X CPU and a single Nvidia GeForce 2080Ti GPU, and thus, their runtimes are marked with  $*$ .
The results from both packing problems demonstrate that Poppy outperforms other RL baselines. For KP, we observe that Poppy leads to improved performance with a population of 16 agents, dividing the optimality gap with respect to POMO sampling and ensemble by a factor of 12 and 42 for the exact same runtime. We emphasize that although these gaps seem small, these differences are still significant: Poppy 16 is strictly better than POMO in  $34.30\%$  of the KP instances, and better in  $99.95\%$ . For JSSP, we show that Poppy significantly outperforms L2D in both greedy and sampling settings, and closes the optimality gap by  $14.1\%$  and  $1.8\%$ , respectively. It should be noted that Poppy outperforms L2D sampling despite requiring 5 times fewer samples. Additionally, to further demonstrate the advantages of training a population of agents, we compare Poppy with a single attention-based model (method named Single) evaluated with 16 stochastic samples. The results show that given the same evaluation budget, Poppy outperforms the single agent and closes the optimality gap by a further  $1\%$ .
Table 3: Packing problems results.
(a) KP
Method	Testing (10k instances)
n = 100
Obj.	Gap	Time
Optimal	40.437	-
Greedy	40.387	0.1250%
POMO (sampling)	40.435	0.0060%	2M
POMO (ensemble)	40.429	0.021%	2M
Poppy 16	40.437	0.0005%	2M
Method	Testing (100 instances)
10 × 10
Obj.	Gap	Time
OR-Tools (optimal)	807.6	0.0%	37S
L2D (Greedy)	988.6	22.3%	20S*
L2D (Sampling)	871.7	8.0%	8H*
MDAM	875.0	8.3%	40M
Single (16 samples)	866.0	7.2%	30M
Poppy 16	857.7	6.2%	30M
Poppy is a population-based RL method for CO problems. It uses an RL objective that incurs agent specialization with the purpose of maximizing population-level performance. Crucially, Poppy does
not rely on handcrafted notions of diversity to enforce specialization. We show that Poppy achieves state-of-the-art performance on four popular NP-hard problems: TSP, CVRP, KP and JSSP.
This work opens the door to several directions for further investigation. Firstly, we have experimented on populations of at most 32 agents; therefore, it is unclear what the consequences of training larger populations are. Whilst even larger populations could reasonably be expected to provide stronger performance, achieving this may not be straightforward. Aside from the increased computational burden, we also hypothesize that the population performance could eventually collapse once no additional specialization niches can be found, leaving agents with null contributions behind. Exploring strategies to scale and prevent such collapses is an interesting direction for future work.
Secondly, our work has built on the current state-of-the-art RL for CO approaches in a single- or few-shot inference setting to demonstrate the remarkable efficacy of a population-based approach. However, there are other paradigms that we could consider. For example, active-search methods allow an increased number of solving attempts per problem and, in principle, such methods for inference-time adaption of the policy could be combined with an initially diverse population to further boost performance. Indeed, we investigate the performance of Poppy with a larger time budget in Appendix D and find that Poppy combined with a simple sampling procedure and no fine-tuning matches, or even surpasses, the state-of-the-art active search approach of Hottung et al. [2022].
Finally, we recall that the motivation behind Poppy was dealing with problems where predicting optimal actions from observations is too difficult to be solved reliably by a single agent. We believe that such settings are not strictly limited to canonical CO problems, and that population-based approaches offer a promising direction for many challenging RL applications. Specifically, Poppy is amenable in settings where (i) there is a distribution of problem instances and (ii) these instances can be attempted multiple times at inference. These features encompass various fields such as code and image generation, theorem proving, and protein design. In this direction, we hope that approaches that alleviate the need for handcrafted behavioral markers whilst still realizing performant diversity akin to Poppy, could broaden the range of applications of population-based RL.
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC).
D. Applegate, R. Bixby, V. Chvatal, and W. Cook. Concorde TSP solver, 2006.
Y. Bengio, A. Lodi, and A. Prouvost. Machine learning for combinatorial optimization: A methodological tour d'horizon. European Journal of Operational Research, 290(2):405-421, 2021.
C. Bonnet, D. Luo, D. Byrne, S. Abramowitz, V. Coyette, P. Duckworth, D. Furelos-Blanco, N. Grinsztajn, T. Kalloniatis, V. Le, O. Mahjoub, L. Midgley, S. Surana, C. Waters, and A. Laterre. Jumanji: a Suite of Diverse and Challenging Reinforcement Learning Environments in JAX, 2023. URL https://github.com/instadeepai/jumanji.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, and C. Hesse. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
X. Chen and Y. Tian. Learning to Perform Local Rewriting for Combinatorial Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
J. Choo, Y.-D. Kwon, J. Kim, J. Jae, A. Hottung, K. Tierney, and Y. Gwon. Simulation-guided beam search for neural combinatorial optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
A. Cully and Y. Demiris. Quality and Diversity Optimization: A Unifying Modular Framework. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2018.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. arXiv preprint arXiv:1609.03499, 2016.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
T. Vidal. Hybrid genetic search for the cvrp: Open-source implementation and swap* neighborhood. Computers & Operations Research, 140:105643, 2022.
T. Vidal, T. G. Crainic, M. Gendreau, N. Lahrichi, and W. Rei. A hybrid genetic algorithm for multidepot and periodic vehicle routing problems. Operations Research, 60(3):611-624, 2012.
O. Vinyals, M. Fortunato, and N. Jaitly. Pointer Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
A. Z. Wagner. Refuting conjectures in extremal combinatorics via linear programming. Journal of Combinatorial Theory, Series A, 169:105-130, 2020.
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
Y. Wu, W. Song, Z. Cao, J. Zhang, and A. Lim. Learning Improvement Heuristics for Solving Routing Problems. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2021.
L. Xin, W. Song, Z. Cao, and J. Zhang. Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
C. Zhang, W. Song, Z. Cao, J. Zhang, P. S. Tan, and C. Xu. Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
A Additional Details on Poppy 15
A.1 Number of Parameters (TSP) 15
A.2 Training Details 15
B Mathematical Elements 16
B.1 Gradient derivation 16
C Baseline Impact Analysis 17
D Comparison to Active Search 18
E Problems 19
E.1 Traveling Salesman Problem (TSP) 19
E.2 Capacitated Vehicle Routing Problem (CVRP) 21
E.3 0-1 Knapsack (KP) 22
E.4 Job-Shop Scheduling Problem (JSSP) 23
F Time-performance Tradeoff 23
G Larger instances 24
Algorithm 2 Poppy training with starting points
1: Input: problem distribution  $\mathcal{D}$ , number of starting points per instance  $P$ , number of agents  $K$ , batch size  $B$ , number of training steps  $H$ , a pretrained encoder  $h_{\psi}$  and decoder  $q_{\phi}$   
2:  $\phi_1, \phi_2, \ldots, \phi_K \gets \text{CLONE}(\phi)$  {Clone the pre-trained decoder parameters  $K$  times.}  
3: for step 1 to  $H$  do  
4:  $\rho_i \gets \text{Sample}(\mathcal{D}) \forall i \in 1, \ldots, B$   
5:  $\alpha_{i,1}, \ldots, \alpha_{i,P} \gets \text{SelectStartPoints}(\rho_i, P) \forall i \in 1, \ldots, B$   
6:  $\tau_{i,p}^{k} \gets \text{Rollout}(\rho_i, \alpha_{i,p}, h_{\psi}, q_{\phi_k}) \forall i \in 1, \ldots, B, \forall p \in 1, \ldots, P, \forall k \in 1, \ldots, K$   
7:  $b_i^k \gets \frac{1}{P} \sum_p R(\tau_{i,p}^k)$   
8:  $k_{i,p}^* \gets \arg \max_{k \leq K} R(\tau_{i,p}^k) \forall i \in 1, \ldots, B, \forall p \in 1, \ldots, P$  {Select the best agent per (instance, starting point).}  
9:  $\nabla L(h_{\psi}, q_{\phi_1}, q_{\phi_2}, \ldots, q_{\phi_K}) \gets -\frac{1}{BP} \sum_{i,p} (R(\tau_{i,p}^{k_*}) - b_i^{k_*}_{i,p}) \nabla \log p_{\psi, \phi_{k_*i,p}^*} (\tau_{i,p}^{k_*})$  {Propagate gradients through these only.}  
10:  $(h_{\psi}, q_{\phi_1}, q_{\phi_2}, \ldots, q_{\phi_K}) \gets (h_{\psi}, q_{\phi_1}, q_{\phi_2}, \ldots, q_{\phi_K}) - \alpha \nabla L(h_{\psi}, q_{\phi_1}, q_{\phi_2}, \ldots, q_{\phi_K})$
A Additional Details on Poppy
A.1 Number of Parameters (TSP)
Table 4 shows the total number of parameters of our models as a function of the population size when experimenting on TSP. Since the decoder represents less than  $10\%$  of the parameters, scaling the population size can be done efficiently. For instance, a population of 16 agents roughly doubles the model size. This observation transfers to CVRP and KP whose encoder-decoder architectures are similar to TSP. The architecture for JSSP is slightly different but the observation remains that the decoder represents a smaller part of the model ( $18\%$ ) and thus the population can be efficiently scaled.
Table 4: Number of parameters for different population sizes.
Encoder	Decoder	Population size
1	4	8	16	32
Parameters	1,190,016	98,816	1,288,832	1,585,280	1,980,544	2,771,072	4,352,128
Extra parameters	-	-	0%	23%	54%	115%	238%
In Section 3.2 (see "Training Procedure"), we described that Poppy consists of two phases. In a nutshell, the first phase consists of training our model in a single-agent setting (i.e., an encoder-decoder model with a single decoder head), whereas the second phase consists of keeping the encoder and cloning the previously trained decoder  $K$  times (where  $K$  is the number of agents) and specialize them using the population objective. Algorithm 2 shows the low-level implementation details of the training of the population (i.e., Phase 2) for environments where POMO [Kwon et al., 2020] uses several starting points; namely, given  $K$  agents and  $P$  starting points,  $P \times K$  trajectories are rolled out for each instance, among which only  $P$  are effectively used for training.
Every approach was trained until convergence. Given TSP, CVRP, KP and JSSP, the single-agent baselines were trained for 5 days, 7 days, 1 day, and 1 day respectively, while the largest populations we provide were trained for 4 days, 5 days, 3 days, and 3 days respectively. We emphasize that Poppy is already performant with less training budget; indeed, Figure 3 (left) shows that the optimality gap of Poppy decreases very quickly at the beginning, although it takes time to reach convergence. Concretely, only a few hours of training is sufficient for Poppy 4 to reach the performance of POMO with 100 samples.
B Mathematical Elements
B.1 Gradient derivation
We recall that the population objective for  $K$  is defined as:
$$
J _ {\mathrm {p o p}} (\theta_ {1}, \theta_ {2}, \ldots , \theta_ {n}) \doteq \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1} \sim \pi_ {\theta_ {1}}, \ldots , \tau_ {K} \sim \pi_ {\theta_ {K}}} \max \left[ R (\tau_ {1}), \ldots , R (\tau_ {K}) \right].
$$
Theorem (Policy gradient for the population objective). The gradient of the population objective is given by:
$$
\nabla J _ {\mathrm {p o p}} (\theta_ {1}, \theta_ {2}, \ldots , \theta_ {n}) = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1} \sim \pi_ {\theta_ {1}}, \ldots , \tau_ {K} \sim \pi_ {\theta_ {K}}} \big (R (\tau_ {i ^ {*}}) - R (\tau_ {i ^ {* *}}) \big) \nabla \log p _ {\theta_ {i ^ {*}}} (\tau_ {i ^ {*}}),
$$
$$
i ^ {*} = \arg \max  \left[ R \left(\tau_ {1}\right), \dots , R \left(\tau_ {K}\right) \right], \quad (i n d e x o f t h e b e s t t r a j e c t o r y)
$$
$$
i ^ {* *} = \operatorname {a r g s e c o n d} \max  \left[ R (\tau_ {1}), \dots , R (\tau_ {K}) \right], \quad (i n d e x o f t h e s e c o n d b e s t t r a j e c t o r y)
$$
Proof. We first derive the gradient with respect to  $\theta_{1}$  for convenience. As all the agents play a symmetrical role in the objective, the same procedure can be applied to any index.
$$
\begin{array}{l} \nabla_ {\theta_ {1}} J _ {\mathrm {p o p}} (\theta_ {1}, \theta_ {2}, \ldots , \theta_ {K}) = \nabla_ {\theta_ {1}} \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1} \sim \pi_ {\theta_ {1}}, \ldots , \tau_ {K} \sim \pi_ {\theta_ {K}}} \max _ {i \in \{1, 2, \ldots , K \}} \left[ R (\tau_ {i}) \right] \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \max _ {i \in \{1, 2, \dots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} \log p (\tau_ {1}, \dots , \tau_ {K}) \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \max _ {i \in \{1, 2, \dots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} \log (\pi_ {\theta_ {1}} (\tau_ {1}) \dots \pi_ {\theta_ {K}} (\tau_ {K})) \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \max _ {i \in \{1, 2, \dots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} (\log \pi_ {\theta_ {1}} (\tau_ {1}) + \dots + \log \pi_ {\theta_ {K}} (\tau_ {K})) \\ = \mathbb {E} _ {\rho \sim \mathcal {D}} \mathbb {E} _ {\tau_ {1}, \dots , \tau_ {K}} \max _ {i \in \{1, 2, \dots , K \}} \left[ R (\tau_ {i}) \right] \nabla_ {\theta_ {1}} \log \pi_ {\theta_ {1}} (\tau_ {1}), \\ \end{array}
$$
We also have for any problem instance  $\rho$  and any trajectories  $\tau_{2},\ldots ,\tau_{K}$  ..