# Cooperative Multi-Robot Task Allocation with Reinforcement Learning
## 5 Experiments
### Reinforcement Learning  
We test our model in five types of environments. We set the number of robots, the map size, and the workload to 5, 100, and 20, respectively. The tasks are randomly distributed in \(100\times 100\) continuous 2D space and the initial workload for each robot is sampled uniformly in a set range \([1,20]\). We tested 10, 20, 30, 40, and 50 tasks to compare the performances  
Figure 6: Makespan for 4 algorithms in the environment with 5 robots and 30 tasks. The initial positions are random and workload is bounded by 20. We plot 100 sample results in light colors and the average in thick lines. The \(X\)-axis represents the real-time for 1, 2, 5, 10, 20, and 50 s. The lower the value, the better performance. of thg algorithms on various difficulty levels. We trained IG, SG, RD, GA, and our model for 100 problem instances. All the problem instances are simulated separately using a single Intel(R) Xeon(R) Gold 6226 @ 2.70 GHz CPU for meta-heuristics and RL-based model. RL-based model is optimized using NVIDIA Quadro RTX 6000 GPU. We trained the models for an hour and compared the minimum makespan. One of the most important hyperparameters of the deep learning model is the number of hidden dimensions. We add the experimental results with different numbers of hidden dimensions for 20 samples in Appendix A.2. We plot the minimum makespan for 50 tasks in Figure 7. The other number of tasks can be found in Appendix A.1. In 50 tasks, IG, SG, and RD models converged early and the performance does not improve much, whereas the performance of GA improves over time. GA and RD find better solutions in the early steps than the RL-based search because we trained the model starting from random initialization, with time needed to train the initial weights. As training goes on, the proposed method (PPO-based) dominates the other models and the makespan decreases continuously. Unlike GA, whose makespan decreases mostly in the early phase (second \(<\) 500), the performance of our method improves smoothly and does not converge during the hour of training. Moreover, the variance of 100 samples is greater for GA than for our method. Therefore, we conclude that the solution search ability of our model improves over time and that it is not biased to a few problem instances. Although our model architecture (encoder-decoder) can be generalized to any number of robots and tasks, we do not pre-train the model, as the cooperative MRTA is more complex than TSP or VRP and because pre-training requires an additional training algorithm. We leave this as a future research project. In Table 1, we present the minimal makespan found for 20, 40, and 60 min for all the models tested with the various number of tasks. As a result, GA performs the best among the meta-heuristic baselines (IG, SG and RD). In the case of 10 tasks, GA shows similar performance (132.0 makespan) with our model (132.2 makespan). In the case of 20 tasks, GA performs best at 20 min (199.4 makespan), but our model shows better most of the time. When the number of tasks exceeds 30, our model shows significantly better performance than other models. As a result, we conclude that our model is highly effective for more complex problem. With regard to the statistical results, we compare two result groups, PPO and GA for 100 samples of 50 tasks. The \(p\)-value of two-sided \(T\)-test is \(1.7\times 10^{-45}\) which suggests that two samples have different average values. In addition, we measure  
Figure 7: Minimum makespan until the indexed time for 100 problem instances. The environment has 5 robots and 50 tasks. Thin lines indicate each instance problem and the bold lines indicate the average of all samples. Best viewed in color. the difference in performance for each sample, as shown in in Figure 8. This highlights the comparable performance in outcomes in a sample-wise manner. This difference in the makespan for each sample suggests that the proposed method finds better solutions than other meta-heuristics for all 100 samples.