# Tensor Implementation of Monte-Carlo Tree Search for Model-Based Reinforcement Learning
## 4 Experiments
### Results  
We report results for two sets of experiments. In the first set, we measured the performance of MCTS adjusted for the effect of DNN model used by MuZero. Architecture of DNN utilized by MuZero is strongly dependent on the application. Therefore we first provide performance of compared implementations without the DNN model. Second set of experiments shows results on the use case of MuZero with DNN model described earlier. Results for experiments without DNN model are shown in Tables 1 and 2. We fixed the number of simulation \(C_{S}\) to 50. Batch of observations size \(C_{T}\) was set to 50, 100, 250, 500 and 750. In the case of CPUGUP_P implementation, we report results for number of processes set to 2, 5, 10 and 25. For CPU implementation, we report results with number of processes set to 5, 10 and 15. In both cases, a larger and smaller number of processes resulted in higher computation time. In both random action and constant action scenarios we see a similar pattern. Proposed GPU implementations is the fastest for all tested values of the \(C_{T}\) parameter. The difference in speed increases with increasing number of observations. In Figure 5 we show \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{\(C_{T}\)} & \multirow{2}{*}{**GPU**} & \multicolumn{6}{c}{**CPUGUP\_P**} & \multirow{2}{*}{**CPUGUP\_S**} & \multicolumn{3}{c}{**CPU**} \\ \cline{3-3} \cline{5-10}  & & **2** & **5** & **10** & **25** & **5** & **10** & **15** \\ \hline \multirow{2}{*}{50} & 1.686 & 7.519 & 5.194 & 5.095 & 7.096 & 35.175 & 14.941 & 14.49 & 17.266 \\  & \(\pm\)0.008 & \(\pm\)0.012 & \(\pm\)0.028 & \(\pm\)0.019 & \(\pm\)0.028 & \(\pm\)0.064 & \(\pm\)0.119 & \(\pm\)0.143 & \(\pm\)0.127 \\ \hline \multirow{2}{*}{100} & 1.704 & 13.575 & 7.825 & 7.24 & 8.812 & 69.282 & 28.769 & 28.142 & 47.295 \\  & \(\pm\)0.006 & \(\pm\)0.012 & \(\pm\)0.03 & \(\pm\)0.018 & \(\pm\)0.033 & \(\pm\)0.116 & \(\pm\)0.044 & \(\pm\)0.035 & \(\pm\)0.12 \\ \hline \multirow{2}{*}{250} & 1.732 & 31.876 & 15.257 & 13.037 & 12.975 & 170.135 & 72.939 & 72.173 & 147.788 \\  & \(\pm\)0.005 & \(\pm\)0.027 & \(\pm\)0.031 & \(\pm\)0.028 & \(\pm\)0.033 & \(\pm\)0.163 & \(\pm\)0.067 & \(\pm\)0.152 & \(\pm\)0.395 \\ \hline \multirow{2}{*}{500} & 1.766 & 63.233 & 28.529 & 22.916 & 20.886 & 340.716 & 144.271 & 140.942 & 374.888 \\  & \(\pm\)0.004 & \(\pm\)0.041 & \(\pm\)0.033 & \(\pm\)0.044 & \(\pm\)0.037 & \(\pm\)0.474 & \(\pm\)0.205 & \(\pm\)0.088 & \(\pm\)0.753 \\ \hline \multirow{2}{*}{750} & 1.787 & 94.143 & 41.317 & 31.937 & 28.512 & 500.181 & 218.951 & 207.96 & 685.497 \\  & \(\pm\)0.003 & \(\pm\)0.087 & \(\pm\)0.042 & \(\pm\)0.031 & \(\pm\)0.046 & \(\pm\)0.479 & \(\pm\)0.395 & \(\pm\)0.137 & \(\pm\)1.611 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Effect of number of observations on computation time. Results of experiments for constant action without DNN model (\(C_{S}=50\)). \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{\(C_{T}\)} & \multirow{2}{*}{**GPU**} & \multicolumn{6}{c}{**CPUGUP\_P**} & \multirow{2}{*}{**CPUGUP\_S**} & \multicolumn{3}{c}{**CPU**} \\ \cline{3-3} \cline{5-10}  & & **2** & **5** & **10** & **25** & **5** & **10** & **15** \\ \hline \multirow{2}{*}{50} & 1.686 & 7.519 & 5.194 & 5.095 & 7.096 & 35.175 & 14.941 & 14.49 & 17.266 \\  & \(\pm\)0.008 & \(\pm\)0.012 & \(\pm\)0.028 & \(\pm\)0.019 & \(\pm\)0.028 & \(\pm\)0.064 & \(\pm\)0.119 & \(\pm\)0.143 & \(\pm\)0.127 \\ \hline \detail comparison between two best performing implementations for both scenarios. For \(C_{T}=750\), proposed GPU implementations is 34.8 times faster for random action scenario and 16 times faster for constant action scenario than the second best performing CPUGPU_P implementations (with 10 and 25 processes respectively). In the constant action scenario (Table 2), the computational time increased for most of the investigated implementations and their settings. This increase is due to a growing trajectories obtained by the phase of the selection in MCTS.