# Evolutionary System Design with Answer Set Programming
## 6 Experiments
### Experimental Evaluation  
We begin by highlighting which routing and similarity techniques produce solutions to the instances. Table 1 shows how many instances were completely solved (sat), i.e., the Pareto front was found, how many solutions were found but the timeout was reached (sat+to), i.e., an approximate Pareto front was found, and how many were unsatisfiable (unsat). The remaining instances either could not be grounded or solving yielded no solution. We select the specific configuration for each class of similarity techniques, viz. strategies (s),  
preferences (P), heuristics (H), that were able to produce the most solutions and compare them with the baseline configurations. The baseline configurations are highlighted in red. Overall, we clearly see that the approximate routing techniques drastically increase the performance compared with arb. The xyz routing performs the best, but it is only slightly better than bou. This is encouraging, as xyz is the least flexible routing and therefore might discard high-quality solutions. Among the similarity techniques, we observe the expected phenomena that strategies lead to unsatisfiability and are merely able to produce results for the least instances. For the full routing, the proof of unsatisfiability finished for 9 instances, while with the advanced routing techniques, 18 instances proved to be unsatisfiable, which is a bit more than half of the instance set. For the advanced routing techniques, strategies actually fully explored more instances than any other configuration. Because strategies exclude solutions, the Pareto front of the fully explored instances might be of worse quality. On the other hand, strategies could be a fast technique with guaranteed similarity to satisfiable instances; we return to this below. Preferences performed worse than the baseline. This might be due to the fact that a more complex Pareto optimization is used that includes similarity. For all routing techniques, heuristics supply the best results in terms of instances with solutions, and they are on par with the baseline regarding fully explored instances. In the following, we use the average rank to compare the different techniques. In essence, each instance induces an ordering among the 87 configurations regarding the measurements of \(\epsilon\)-dominance and (maximum and average) Hamming distance, along with the product of the two. In the case of a tie, the next entry that is worse receives the rank that it would have been at if no tie occurred; for instance, if two configurations are tied for first, the next configuration gets rank three. These ranks are averaged to provide insights into which configurations performed well overall. Configurations that did not yield any solution to an instance are punished with the worst possible rank. Furthermore, due to the vast amount of configurations, we only present the top 50 results plus the baseline configurations (Less than 53 results are shown whenever the baseline configurations are among the top 50). Next, we analyze the first solution that was found with respect to similarity and quality. Our hypothesis is that similarity techniques immediately yield not only similar but better solutions compared with the baseline, as the parent implementation is of high quality. Table 2 shows from left to right the top 50 configurations for Hamming distance (HD), \(\epsilon\)-dominance (\(\epsilon\)D), and their product (HD\(\times\epsilon\)D). \begin{table}
\begin{tabular}{l l l l} \hline \hline  & **sat** & **sat+to** & **unsat** \\ \hline arb & 1 & 14 & 0 \\ arb-S & 1 & 11 & 9 \\ arb-P & 1 & 6 & 0 \\ arb-H & 1 & 22 & 0 \\ \hline bou & 4 & 23 & 0 \\ bou-S & 5 & 12 & 18 \\ bou-P & 4 & 13 & 0 \\ bou-H & 4 & 27 & 0 \\ \hline xyz & 6 & 27 & 0 \\ xyz-S & 9 & 8 & 18 \\ xyz-P & 6 & 19 & 0 \\ xyz-H & 6 & 28 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Best results regarding instances solved for routing and similarity techniques.