# Fast Performance Modeling across Different Database Versions Using Partitioned Co-Kriging
## 5 Experimental Evaluation
### Evaluation of Prediction Accuracy

Table 3: MRE comparison among different approaches for PostgreSQL (version 9.3–11.0). Figure 4: MRE improvement of PCK compared with six baseline algorithms. \begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline
**Sample Size** & **1n** & **2n** & **3n** & **4n** & **5n** & **6n** & **7n** & **8n** & **9n** & **10n** & **11n** & **12n** & **13n** & **14n** & **15n** \\ \hline DeepPerf & 9.109 & 2.846 & 2.249 & 1.945 & 2.007 & 2.007 & 1.857 & 1.917 & 1.841 & 1.799 & 1.736 & 1.711 & 1.606 & 1.544 & 1.56 \\ CART & 1.692 & 1.793 & 1.932 & 1.686 & 1.795 & 1.575 & 1.623 & 1.593 & 1.516 & 1.632 & 1.573 & 1.578 & 1.535 & 1.541 & 1.522 \\ Finetune & 2.39 & 2.271 & 1.876 & 1.722 & 1.801 & 1.692 & 1.754 & 1.66 & 1.64 & 1.67 & 1.539 & 1.584 & 1.547 & 1.517 & 1.567 \\ DDC & 5.452 & 2.873 & 2.286 & 2.058 & 1.819 & 1.926 & 1.91 & 1.806 & 1.673 & 1.713 & 1.861 & 1.633 & 1.692 & 1.628 & 1.705 \\ Model-shift & 1.593 & 1.352 & 1.724 & 1.471 & 1.708 & 1.519 & 1.61 & 1.57 & 1.596 & 1.565 & 1.655 & 1.572 & 1.598 & 1.55 & 1.577 \\ Ottertune & 8.804 & 8.768 & 8.621 & 7.754 & 7.501 & 6.283 & 6.271 & 5.755 & 5.509 & 5.161 & 4.445 & 3.944 & 3.664 & 2.718 & 2.09 \\ PCK & **1.208** & **1.101** & **1.058** & **1.052** & **1.028** & **1.033** & **1.012** & **1.024** & **1.010** & **1.016** & **1.000** & **0.997** & **1.002** & **1.001** & **0.999** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: MRE comparison among different approaches for SQLite (version 3.3.1–3.36.0). DeepPerf tackles the performance prediction problem directly by training a performance model with DNN. Further, Finetune and DDC are two DNN-based transfer learning approaches. The above experimental results show that the MRE results of PCK are better than these three DNN-based performance modeling methods. It is because that training a high-performance DNN often requires a large amount of training data, but we can only provide a small sample due to the high cost of performance measurement. In another aspect, we find out that the two DNN-based transfer learning methods in most cases perform higher prediction accuracy than simple DNN-based performance prediction model in five database systems. Similarly, the Model-shift approach transfers the CART-based performance model in the source to the target using linear regression models, and it also achieves better MRE than CART. This observation verifies the effectiveness of transfer learning on performance prediction task. The validity of above-mentioned transfer learning methods proves that there exists a correlation between the source and target. Our proposed PCK method aims to take advantage of this correlation to build a high-performance prediction model in target domain. In the experiment results, the PCK achieves higher prediction accuracy compared with other four transfer learning method under almost all the sample sizes. From another point of view, PCK requires a much smaller number of target samples to reach the same standard of prediction accuracy, compared with other baseline approaches. It means that PCK outperforms six baseline algorithms not only in terms of prediction accuracy, but also in terms of measurement effort. To further verify our conclusion, we conduct the similar experiments across different version changes. The MRE comparison among different approaches for MySQL and Redis under different version change scenarios (version 5.5-5.7, 5.7-8.0 for MySQL, and version 4.0.1-5.0, 5.0.0-6.0 for Redis) are listed in Tables 7-10, respectively. Experimental results confirm that the PCK outperforms the state-of-the-art baselines in terms of prediction accuracy and measurement cost in almost all experimental settings in this paper.