# Fast Performance Modeling across Different Database Versions Using Partitioned Co-Kriging
## 5 Experimental Evaluation
### Trade-Off on Choosing \(K\)  
In this part, we will discuss the trade-off on choosing the number of clusters (\(K\)), and illustrate the influence of different \(K\) values on the prediction accuracy. In order to explore this problem, we systematically vary the value of \(K\) from 1 to 10 in each subject system and we measure the prediction accuracy in each case. Taking MySQL (version 5.5-8.0) as an example, the experiment results are shown in Table 11.  
Our results, in Table 11, indicate that PCK achieves the highest prediction accuracy for almost all sample sizes when \(K\) equals to 3 in MySQL. The \(MRE\) of PCK decreases when \(K\) increases appropriately. This demonstrates that clustering different smooth regions could help boost the performance of prediction model. However, the prediction accuracy will not continue to grow when \(K\) exceeds a certain threshold, such as 3 in the above example. This is due to the insufficient target samples. If a large \(K\) is chosen in PCK, the available target sample in each region will be too little to learn a reliable performance model. Consequently, the choice of \(K\) is the key to whether the PCK method can achieve high prediction accuracy. In order to further verify the necessity of PCK, we compare the \(MRE\) in different clusters (\(K=3\)) with the \(MRE\) without clustering (\(K=1\)) in MySQL, the result is shown in Figure 5. We observe that the \(MRE\) without clustering is higher than the \(MRE\) in three different clusters in almost all sample sizes. Finally, the optimal prediction accuracy is achieved in this case (\(K=3\)). In addition, PCK achieves the best prediction performance in Redis (version 4.0.1-6.0.5) and PostgreSQL (version 9.3-11.0) when \(K=2\), and \(K=1\) in Cassandra (version 2.1.0-3.11.6), \(K=4\) in SQLite (version 3.31.1-3.36.0), respectively. The reason for the small \(K\) in Cassandra is its larger configuration space, thus the insufficient measurements may lead to inaccurate partitioning. \begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline
**Sample Size** & **1n** & **2n** & **3n** & **4n** & **5n** & **6n** & **7n** & **8n** & **9n** & **10n** & **11n** & **12n** & **13n** & **14n** & **15n** \\ \hline DeepPerf & 10.168 & 4.634 & 4.533 & 3.752 & 2.413 & 2.436 & 2.092 & 1.805 & 1.87 & 1.879 & 1.718 & 1.732 & 1.601 & 1.641 & 1.612 \\ CART & 1.266 & 1.391 & 1.259 & 1.416 & 1.349 & 1.311 & 1.298 & 1.32 & 1.308 & 1.31 & 1.334 & 1.257 & 1.271 & 1.323 & 1.362 \\ Finetune & 1.781 & 1.588 & 1.495 & 1.457 & 1.534 & 1.528 & 1.451 & 1.339 & 1.407 & 1.303 & 1.347 & 1.306 & 1.335 & 1.317 & 1.272 \\ DDC & 8.53 & 6.708 & 3.56 & 2.529 & 2.61 & 1.861 & 2.113 & 1.907 & 1.973 & 1.838 & 1.654 & 1.717 & 1.61 & 1.646 & 1.601 \\ Model-shift & 1.196 & 1.425 & 1.41 & 1.406 & 1.372 & 1.163 & 1.243 & 1.271 & 1.378 & 1.302 & 1.346 & 1.264 & 1.337 & 1.283 & 1.33 \\ Ottertune & 27.598 & 12.593 & 9.313 & 9.703 & 8.023 & 7.651 & 7.166 & 6.636 & 6.768 & 6.597 & 5.941 & 5.592 & 5.433 & 5.01 & 4.565 \\ PCK & **0.711** & **0.648** & **0.641** & **0.64** & **0.636** & **0.636** & **0.635** & **0.634** & **0.635** & **0.634** & **0.635** & **0.633** & **0.632** & **0.631** \\ \hline \hline \end{tabular}
\end{table} 
Table 10: MRE comparison among different approaches for Redis (version 5.0.0–6.0.5).