## 5 Experimental Results
### Test Results with Benchmark Functions
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Function** & **Decision Space** \\ \hline Ackley & \(20-20\exp\left(-0.2\sqrt{\frac{1}{n}\sum\limits_{i=1}^{n}x_{i}^{2}}\right)\) & \([-1,1]^{\text{n}}\) \\  & \(+e-\exp\left(\frac{1}{n}\sum\limits_{i=1}^{n}\cos(2\pi\cdot x_{i})\right)\) & \\ Alpine & \(\sum\limits_{i=1}^{n}|x_{i}\sin x_{i}+0.1x_{i}|\) & \([-10,10]^{\text{n}}\) \\ Axis-parallel & \(\sum\limits_{i=1}^{n}ix_{i}^{2}\) & \([-5.12,5.12]^{\text{n}}\) \\ DeJong & \(\left\|x\right\|^{2}\) & \([-5.12,5.12]^{\text{n}}\) \\ Drop wave & \(-\frac{1+\cos(12\sqrt{\left\|x\right\|^{2}})}{\frac{1}{2}\left\|x\right\|^{2} +2}\) & \([-5.12,5.12]^{\text{n}}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Benchmark functions used for testing noise handling performance. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \(\sigma\) & **Ackley** & **Alpine** & **Axis Parallel** & **DeJong** & **Drop Wave** \\ \hline RTS & 5\% & + & + & + & + & + \\  & 10\% & + & + & + & + & + \\  & 20\% & + & + & + & + & + \\  & 5\% & + & + & = & = & = \\ NTGA & 10\% & + & + & = & = & = \\  & 20\% & = & + & = & = & = \\  & 5\% & + & + & + & + & + \\ DS & 10\% & + & + & + & + & + \\  & 20\% & + & + & + & + & + \\  & 5\% & + & + & + & + & = \\ MDE & 10\% & + & + & + & + & + \\  & 20\% & + & + & + & + & + \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results of non-paired \(t\)-tests with a confidence level of 95%, comparing N-RTS against other algorithms by using the benchmark functions of Table 6 with different noise levels. Figure 7 shows the distribution of the probability values observed at the probabilistic comparison step (i.e., the observed values of \(p_{c}\) in the pseudocode of Figure 6) during the search for an optimal solution of the Ackley function (\(\sigma\) = 0.1) by N-RTS. The cases with a probability lower than 0.8 account for 46% of the total, for which the comparison results are not considered decisive. Since the confidence threshold \(\theta\) was set to 0.98 in N-RTS, 11% of the total cases were considered decisive, for which no additional samples were needed to identify the winner. The samples saved in this way contributed to examining more unexplored candidates in the search space, thus enhancing the search performance. Figure 8 summarizes another analysis result of the logs of the survival selection step of N-RTS during the optimization of the Ackley function (\(\sigma\) = 0.1). Recall that the individuals involved in the survival selection were a newly born \(y\) and a competitor \(x\), chosen from the population (Figure 6). While N-RTS gives additional samples to the better-looking one among \(x\) and \(y\) only when the judgement is not decisive, NTGA unconditionally gives additional samples to \(x\). The white bars in Figure 8 show the probabilities that the better-looking one is really a winner (when judged by noise-free fitness). We can see that this probability increases as the value of \(p_{c}\) (confidence of comparison) increases. The shaded bars in the figure show the probabilities that \(x\) (the one NTGA chooses) is a real winner, which are consistently lower than the white bars. The additional samples invested in the losers were dumped and not accumulated, to contribute to making the evaluations more reliable. What would happen if \(\theta\) is lowered to, for example, 0.95? Figure 7 shows that additional samples can be saved for up to 21% of the survival selection cases. The cost that we pay for this is that the real winning probability of the better-looking individual can be reduced to 0.70 from 0.75, as indicated in Figure 8. Additionally, \(\theta\) must be determined carefully, considering this tradeoff. Figure 7: Distribution of the values of \(p_{c}\) during the search for an optimal solution of the Ackley function (\(\sigma\) = 0.1) by N-RTS.