## 5 Experimental Results
### Test Results with Benchmark Functions  
To see how well N-RTS handles noisy optimization problems, compared to other noisy optimization methods, we conducted additional experiments with some benchmark functions used in the study of [15], as listed in Table 6 (the dimensionality was set to 30). We generated three noisy test cases from each of these functions by adding zero-mean Gaussian noise, with the standard deviation corresponding to 5%, 10%, and 20% of its codomain range \(R\), respectively. The codomain range \(R\) of a function was estimated by calculating the \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **PS by N-RTS** & **PS by NTGA** & **OnS** \\ \hline AGV delay (sec) & 16.9 \(\pm\) 1.2 & 18.7 \(\pm\) 0.8 & 19.3 \(\pm\) 2.1 \\ ET waiting (sec) & 293.6 \(\pm\) 59.6 & 370.2 \(\pm\) 71.7 & 771.8 \(\pm\) 69.7 \\ \(t\)-test (AGV delay) & NTGA, OnS & - & - \\ \(t\)-test (ET waiting) & NTGA, OnS & OnS & - \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results obtained by averaging the performances measured on ten test scenarios (significance level: 0.95). \begin{table}
\begin{tabular}{c c c c} \hline \hline \begin{tabular}{c} **Horizon Length** \\ **(min)** \\ \end{tabular} & \begin{tabular}{c} **Objective Weights** \\ **(AGV Delay: ET Waiting)** \\ \end{tabular} & \
\begin{tabular}{c} **AGV Delay** \\ **(sec)** \\ \end{tabular} &
\begin{tabular}{c} **ET Waiting** \\ **(sec)** \\ \end{tabular} \\ \hline \multirow{4}{*}{30} & 50: 1 & 21.4 \(\pm\) 1.9 & 550.3 \(\pm\) 112.3 \\  & 100: 1 & 26.4 \(\pm\) 2.3 & 782.5 \(\pm\) 135.2 \\  & 300: 1 & 20.4 \(\pm\) 2.4 & 547.7 \(\pm\) 89.4 \\  & 500: 1 & 24.1 \(\pm\) 2.5 & 813.5 \(\pm\) 90.2 \\ \hline \multirow{4}{*}{60} & 50: 1 & 23.5 \(\pm\) 1.8 & 284.4 \(\pm\) 82.1 \\  & 100: 1 & 16.8 \(\pm\) 1.4 & 298.2 \(\pm\) 62.4 \\ \cline{1-1}  & 300: 1 & 17.5 \(\pm\) 2.3 & 287.3 \(\pm\) 72.3 \\ \cline{1-1}  & 500: 1 & 24.4 \(\pm\) 2.6 & 361.5 \(\pm\) 100.2 \\ \hline \multirow{4}{*}{90} & 50: 1 & 18.5 \(\pm\) 2.9 & 346.3 \(\pm\) 130.7 \\  & 100: 1 & 21.6 \(\pm\) 3.1 & 338.5 \(\pm\) 149.1 \\ \cline{1-1}  & 300: 1 & 21.3 \(\pm\) 2.2 & 357.9 \(\pm\) 101.2 \\ \cline{1-1}  & 500: 1 & 28.8 \(\pm\) 2.2 & 366.5 \(\pm\) 200.6 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance of PS by N-RTS with different values of objective weight and horizon length. difference between the average of the function values of 100 points randomly generated within the decision space and the optimal (i.e., the minimum) value. For each test case, algorithms to be compared were run 50 times, and their results were averaged. The total number of evaluations (i.e., samples) was fixed to 100,000 and the other parameter settings were the same as those described previously. After each run of an algorithm, the individual with the best sampled evaluation was picked up from the population, and then its noise-free actual value was identified. This actual value was treated as the solution that the algorithm had found. Table 7 shows the results of non-paired \(t\)-tests with a confidence level of 95%, comparing N-RTS against RTS, NTGA, DS, and MDE. Since RTS is not equipped with any noise handling scheme, it adopts the simple resampling strategy, according to which each individual is evaluated, by giving a sufficient number of samples. The number of samples in our experiment was set to 5, which is empirically shown to achieve the best balance between exploration and exploitation. The parameters of DS and MDE were set to the same values as those chosen by [3]. The values under the column heading \(\sigma\) are the noise levels tested. Additionally, '+' indicates that N-RTS performs significantly better than the respective algorithm listed in the leftmost column of the same row and '=' indicates that the performances of the compared algorithms cannot be considered significantly different. As can be seen in the table, the performance of N-RTS easily dominates RTS, DS, and MDE and is better than, or at least similar to, NTGA.