# NewSQL Databases Assessment: CockroachDB, MariaDB Xpand, and VoltDB
## 5 Experimental Evaluation
### OSSpal Evaluation  
Using the OSSpal methodology, we first define the weight of each of the five categories: 'Functionality', 'Operational Software Characteristics', 'Documentation', 'Support and Service', and 'Software and Technology Attributes'. Table 2 shows the weights assigned to each category, making a total of 100%. These weights were based on several successful studies, such as [14; 15; 16; 17]. However, we had to make some adjustments since MariaDB Xpand is not open source. Given this difference, we recall that two categories were removed: 'Community and Adoption' and 'Development Process'. The category 'Functionality' is the most important, which was given the highest weight (35%). The next category, 'Operational Software Characteristics', is defined with 25%, since it is related to scalability, user interface, setup, and reliability. The third and fourth categories used in this assessment are 'Documentation' and 'Support and Service' with weights of 15% because a suitable database must have good documentation to help with installation and configuration, and feedback from internal or external sources. Finally, the category 'Software Technology Attributes' was assigned with 10% because it measures the use of third-party tools and support languages. The next step is to define and evaluate the different characteristics that involve NewSQL solutions to analyze the 'Functionality' category. The features chosen to evaluate the tools were based on [36]. At this stage, the characteristics that suit both types of tools were included. For this, a score was assigned to each feature on the following scale: 1--slightly important, 2--important, 3--very important. Table 3 shows the weights and \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **CockroachDB** & **MariaDB Xpand** & **VoltDB** \\ \hline SQL Support & Yes & Yes & Yes \\ \hline Transaction Properties & ACID & ACID & ACID \\ \hline \multirow{2}{*}{Implemented Language} & Go, Typescript, Starlark, and & & \\  & Yacc & C & Java, C++ \\ \hline \multirow{3}{*}{Supported Languages} & C\#, C++, Clojure, Go, Java, JavaScript (Node.js), PHP, Python, Ruby, and Rust & C, C++, Java, Javascript, Python & Java, Python, Ruby, Scala, \\  & Python, Ruby, and Rust & Python & Javascript \\ \hline \hline \end{tabular}
\end{table} 
Table 1: _Cont._ \begin{table}
\begin{tabular}{c c} \hline \hline
**Categories** & **Weight** \\ \hline Functionality & 35\% \\ \hline Operational Software Characteristics & 25\% \\ \hline Documentation & 15\% \\ \hline Support and Service & 15\% \\ \hline Software Technology Attributes & 10\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Assigning weights to categories. marks assigned to each metric according to what we consider to be the most important features of a NewSQL database. After assigning the weight's attribution to all categories, each tool evaluation is performed to assess which database earns the highest score. The final evaluation in each category, besides 'Functionality', is based on the following formula:  
\[\text{Final weight}=(\text{weight assigned }\times\text{metric weight}/100\%) \tag{1}\]  
The results of the evaluation are presented in Table 4.  
From the analysis of Table 4, it is possible to conclude:  
* In the 'Functionality' category, MariaDB Xpand and VoltDB stand out with scores of 4, which means that they have almost all features that we considered important in evaluating NewSQL databases;
* For 'Operational Software Characteristics', CockroachDB has the highest score; \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Categories** & **Weight** & **CockroachDB** & **MariaDB Xpand** & **VoltDB** \\ \hline Functionality & 35\% & 3 & 4 & 4 \\ \hline Operational Software & 25\% & 4 & 3.75 & 3.5 \\ Characteristics & 15\% & 5 & 4.65 & 4.65 \\ \hline Documentation & 15\% & 5 & 5 & 5 \\ \hline Support and Services & 15\% & 5 & 5 & 5 \\ \hline Software Technology & 10\% & 4.35 & 3.75 & 3.75 \\ Attributes & & & & \\ \hline Total & 3.99 & 4.16 & 4.10 \\ \hline \hline \end{tabular}
\end{table}