## 5 Experimental Results
### Performance Comparison  
Table 4 shows the results of the experiments, comparing the performances of PS by N-RTS, PS by NTGA, and OnS. Since the evolutionary algorithms are stochastic, the policies resulting from different search sessions are different, showing different performances. Therefore, we collected ten dispatching polices by running N-RTS ten times and another ten by running NTGA ten times. Then, the performance of PS by N-RTS, for example, on a test scenario, was obtained by applying each of the ten policies to the scenario and averaging the observed performances. To make our performance comparison more reliable, we used ten test scenarios and compared the test results by using a non-paired \(t\)-test with a significance level of 0.95. Notice that every number in Table 4 is, thus, an average of one hundred numbers. We can see that PS by N-RTS clearly dominates not only PS by NTGA but also OnS in terms of both AGV delay and ET waiting time. Another important advantage of using the dispatching policy is that it can be applied with a very small amount of computational overhead. Applying OnS to a single test scenario takes a CPU time of longer than one and a half days, in contrast to less than a few seconds in the cases of applying the policy. After some calculations, we roughly estimated that applying OnS  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **OnS** & **PS** \\ \hline Length of horizon & 30 min. & 60 min. \\ Rescheduling interval & 15 min. & After each job \\ No. of evaluations for search & 800/rescheduling & - \\ Weights for objectives & 300: 1 & 100: 1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Setting for the experiments.  
to a real terminal with 22 storage blocks, for example, would require more than three dedicated CPUs.  
While PS can reschedule almost immediately after each job, OnS cannot afford to do this, due to the heavy computational overhead. The shorter the rescheduling interval, the less likely it becomes that the result of the schedule execution deviates from the expected outcome in an uncertain environment where the operations of ETs, AGVs, and cranes are not deterministic. Another advantage of PS is that it allows a longer scheduling horizon than OnS (Table 3), due to the small computational overhead of applying the policy. However, longer horizons are not always better. Table 5 shows the effects of horizon length on the search for a dispatching policy by N-RTS. The table also shows the effects of different combinations of objective weights. The policies are learned from the same training scenarios as those used in the above experiments (Table 4). The test results given in Table 5 were obtained by applying the policies to ten separately generated test scenarios. The AGV delay and ET waiting times are the average results of applying ten policies, obtained by running N-RTS ten times. Increasing the horizon length from 30 to 60 min resulted in an overall performance improvement, regardless of the objective weight combination. However, a further increase leads to a deterioration in performance. Regarding the objective weight combinations, we see that a hundred times higher weight to AGV than to ET gives rise to the best performance. The performance degrades when the weight to AGV is either too high or too low.