## 6 Results
The better results among \(\text{COA}[\tilde{w}_{\Theta}]\) and \(\text{COA}[\tilde{w}_{\text{bl}}]\) are printed bold. Additionally, as the standard deviations w.r.t. the optimality gaps are quite large, shown in Figure 6, we have also applied a one-sided Wilcoxon signed-rank test to determine for each group, whether the difference in optimality gaps is significant or not. Instance groups for which the Wilcoxon test has assessed at a 95% confidence interval that either \(\text{COA}[\tilde{w}_{\Theta}]\) or \(\text{COA}[\tilde{w}_{\text{bl}}]\) has produced better optimality gaps are marked with an asterisk. The table shows that, especially for the CSS and MAN instances, \(\text{COA}[\tilde{w}_{\Theta}]\) generates significantly better results at almost all interaction levels \(\psi\) than \(\text{COA}[\tilde{w}_{\text{bl}}]\). While, for the EVC instances, the average optimality gaps w.r.t \(\text{COA}[\tilde{w}_{\Theta}]\) are lower than the average optimality gaps w.r.t. \(\text{COA}[\tilde{w}_{\text{bl}}]\), there are instance groups for which no significant difference between the optimality gaps can be determined. However, there is no instance group for which \(\text{COA}[\tilde{w}_{\text{bl}}]\) produced significantly better results than \(\text{COA}\) with \(\tilde{w}_{\Theta}\) over all user interaction thresholds. It can be observed that, at very low levels of user interaction, \(\text{COA}[\tilde{w}_{\Theta}]\) and \(\text{COA}[\tilde{w}_{\text{bl}}]\) seem to be equally strong. However, as the amount of collected of user feedback increases, \(\text{COA}[\tilde{w}_{\Theta}]\) quite quickly outperforms \(\text{COA}[\tilde{w}_{\text{bl}}]\). Figure 6 gives a visual comparison between \(\text{COA}[\tilde{w}_{\Theta}]\) and \(\text{COA}[\tilde{w}_{\text{bl}}]\) for selected instance groups and not only shows average optimality gaps but also respective standard deviations around the mean values as shaded areas. The figure confirms that the average gaps produced by \(\text{COA}[\tilde{w}_{\Theta}]\) are generally lower than those of \(\text{COA}[\tilde{w}_{\text{bl}}]\) but also shows that the standard deviations are quite large in general for both approaches. However, as COA progresses and the quality of the solutions improves, the standard deviations decrease, as well. To further investigate the learning capabilities of the surrogate function, we now look at the mean squared error (MSE) of \(\tilde{w}_{\Theta}\) with respect to the known exact values \(w\). The left plots in Figure 7 show the development of this MSE calculated over all suitability values that are not known yet by COA for all instance groups. It can be seen that the MSE is generally small and approaches zero rather quickly. The reason for such small values can be found in the matrix factorization model used in the EC, which adds a bias for unknown suitability values towards zero as users typically only have a small number of locations with positive suitability values for each of their SPRs. Consequently, the MSE is distorted by the large number suitability values that are zero. Therefore, the plots on the right side of Figure 7 show average mean squared errors calculated only over all _positive_ suitability values that are not known yet; we denote this error by MSE+. This measure gives a clearer picture on how the surrogate function continuously improves in all cases with an increasing amount of gained knowledge. At the start of the algorithm the MSEs of the benchmark groups are between 0.2 and 0.3 on average and go towards zero almost linearly with the interaction level. Note that neither the size of an instance nor the given budget seem to have a significant impact on the size of the errors. Additionally, the figure also highlights how the instance parameters \(\sigma_{v}\) and \(\sigma_{r}\) impact the similarity of user preferences as the MSEs for instances with \(\sigma_{v}=5\) and \(\sigma_{r}=0.15\) are generally larger than the MSEs for instances with \(\sigma_{v}=3\) and \(\sigma_{r}=0.03\).