# Tensor Implementation of Monte-Carlo Tree Search for Model-Based Reinforcement Learning
## 5 Conclusions  
In this paper, we proposed a parallel implementation of MCTS that efficiently evaluates large number of MCTS trees at once. It utilizes tensor operations and is fully implemented on GPU. We show that the atomic nature of MCTS operations can be transformed into vector operations suitable for GPU. We demonstrated this capability using the example of MuZero model-based RL agent in Atari game domain. Model-based RL agents often combine DNN and MCTS approaches to improve action selection. During the offline training, these RL agents requires to process a huge amount of observations in parallel. These observations are represented by unique root nodes in MCTS. We compared our implementation with approaches based on the Werner and EfficientZero implementations. We show that the proposed approach gives the best results and scales well with the number of observations and number of simulations. We tested two scenarios of tree formations: random action and constant action. For both scenarios the proposed implementations yield the best results. In experiments without DNN, the proposed implementation is 34.8 times faster for the random action and 16 times faster for the constant action scenario than the second best performing CPUGPU_P implementation (\(C_{T}=750\) and \(C_{S}=50\)). We further investigated the effect of DNN in RL agent. In experiments with DNN and for the random action, the proposed implementation is 4.7 times faster than the second best performing CPUGPU_P for the value of parameters \(C_{T}=750\) and \(C_{S}=50\). In the case \begin{table}
\begin{tabular}{c c c c c c c} \hline \multirow{2}{*}{\(C_{T}\)} & \multirow{2}{*}{**GPU**} & \multicolumn{5}{c}{**CPUGPU\_P**} & \multirow{2}{*}{**CPUGPU\_S**} \\ \cline{3-3} \cline{5-7}  & & **2** & **5** & **10** & **25** & \\ \hline \multirow{2}{*}{50} & 1.909 & 7.942 & 5.332 & 5.296 & 7.309 & 33.371 \\  & \(\pm 0.008\) & \(\pm 0.011\) & \(\pm 0.023\) & \(\pm 0.019\) & \(\pm 0.028\) & \(\pm 0.05\) \\ \hline \multirow{2}{*}{100} & 2.074 & 14.182 & 8.426 & 7.849 & 9.201 & 67.45 \\  & \(\pm 0.007\) & \(\pm 0.022\) & \(\pm 0.028\) & \(\pm 0.023\) & \(\pm 0.026\) & \(\pm 0.105\) \\ \hline \multirow{2}{*}{250} & 2.507 & 32.975 & 16.207 & 14.054 & 14.231 & 168.301 \\  & \(\pm 0.005\) & \(\pm 0.028\) & \(\pm 0.028\) & \(\pm 0.035\) & \(\pm 0.033\) & \(\pm 0.142\) \\ \hline \multirow{2}{*}{500} & 3.232 & 65.08 & 29.884 & 24.283 & 22.527 & 338.874 \\  & \(\pm 0.004\) & \(\pm 0.053\) & \(\pm 0.035\) & \(\pm 0.035\) & \(\pm 0.048\) & \(\pm 0.423\) \\ \hline \multirow{2}{*}{750} & 3.984 & 97.403 & 43.776 & 34.896 & 30.752 & 501.425 \\  & \(\pm 0.004\) & \(\pm 0.092\) & \(\pm 0.041\) & \(\pm 0.04\) & \(\pm 0.046\) & \(\pm 0.351\) \\ \hline \end{tabular}
\end{table} 
Table 6: Effect of number of observations on computation time. Results of experiments for constant action with DNN model (\(C_{S}=50\)). \begin{table}
\begin{tabular}{c c c c c c c} \hline \multirow{2}{*}{\(C_{T}\)} & \multirow{2}{*}{**GPU**} & \multicolumn{5}{c}{**CPUGPU\_P**} & \multirow{2}{*}{**CPUGPU\_S**} \\ \cline{3-3} \cline{5-7}  & & **2** & **5** & **10** & **25** \\ \hline \multirow{2}{*}{50} & 0.383 & 1.962 & 2.269 & 3.285 & 5.634 & 6.091 \\  & \(\pm 0.001\) & \(\pm 0.007\) & \(\pm 0.008\) & \(\pm 0.016\) & \(\pm 0.047\) & \(\pm 0.004\) \\ \hline \multirow{2}{*}{100} & 0.539 & 3.053 & 3.091 & 3.919 & 6.442 & 10.001 \\  & \(\pm 0.001\) & \(\pm 0.006\) & \(\pm 0.008\) & \(\pm 0.015\) & \(\pm 0.037\) & \(\pm 0.005\) \\ \hline \multirow{2}{*}{250} & 0.946 & 6.909 & 5.574 & 5.757 & 7.868 & 21.864 \\  & \(\pm 0.001\) & \(\pm 0.012\) & \(\pm 0.014\) & \(\pm 0.021\) & \(\pm 0.026\) & \(\pm 0.009\) \\ \hline \multirow{2}{*}{500} & 1.714 & 12.607 & 9.144 & 8.787 & 10.254 & 41.61 \\  & \(\pm 0.002\) & \(\pm 0.019\) & \(\pm 0.044\) & \(\pm 0.023\) & \(\pm 0.03\) & \(\pm 0.018\) \\ \hline \multirow{2}{*}{750} & 2.466 & 18.185 & 11.857 & 11.47 & 12.571 & 62.345 \\  & \(\pm 0.003\) & \(\pm 0.024\) & \(\pm 0.055\) & \(\pm 0.039\) & \(\pm 0.031\) & \(\pm 0.016\) \\ \hline \end{tabular}
\end{table}