# Experimental Comparison of Editor Types for Domain-Specific Languages
## 4 Results
### Controlled Experiment  
For the experiment comparing an advanced text editor and a form-based editor, the collected results are displayed textually in Table 1 and graphically in Figure 5.  
The median time spent on the tasks in the Xtext group was 381 s, while in the Google Forms group it was 390.5 s (2.43% faster). The Google Forms group had a longer upper tail, so the difference in means is larger: 382.95 s for an advanced text editor and 432.40 s for a form-based editor (11.44%).  
The computed \(p\)-value was 0.1054, which is more than \(\alpha\) (0.05). Therefore, the difference between the groups was not statistically significant.  
The results of the experiment comparing a projectional and form-based editor can be seen in Table 2 and Figure 6.  
\begin{table}
\begin{tabular}{c c c} \hline
**ID** & **Xtext (s)** & **Google Forms (s)** \\ \hline  
td. Dev.** & 112.62 & 133.33 \\ \hline \end{tabular}
\end{table}
Table 1: Results of the XText vs. Google Forms experiment.  
Figure 5: Results of the XText vs. Google Forms experiment.  
In the MPS group, the median time to complete the tasks was 393 s. The Google Forms group achieved a median of 378 s, which is 3.82% faster. We observed a much larger variance in the MPS group, especially in the upper tail. Therefore, the means are 450.65 s for MPS compared to 385.41 s in the Google Forms group. The relative difference of means was 14.48%.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**ID** & **MPS (s)** & **Google Forms (s)** \\ \hline
1 & 243 & 217 \\
2 & 276 & 277 \\
3 & 323 & 387 \\
4 & 260 & 462 \\
5 & 428 & 276 \\ \hline
6 & 387 & 378 \\
7 & 393 & 326 \\
8 & 559 & 374 \\
9 & 303 & 381 \\
10 & 726 & 576 \\ \hline
11 & 841 & 461 \\
12 & 395 & 411 \\
13 & 365 & 330 \\
14 & 707 & 389 \\
15 & 613 & 674 \\ \hline
16 & 572 & 370 \\
17 & 270 & 263 \\ \hline
**Median** & 393.00 & 378.00 \\
**Mean** & 450.65 & 385.41 \\
**Std. Dev.** & 184.69 & 113.46 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results of the MPS vs. Google Forms experiment.  
Figure 6: Results of the MPS vs. Google Forms experiment.  
In this second experiment, we computed a p-value of 0.1350, which means the difference between groups was not significant.  
After combining the results, the best times were measured for the Xtext editor followed by Google Forms and finally MPS. However, the differences were not statistically significant, so we cannot reject the null hypotheses (\(\mathbf{H_{X0}}\), \(\mathbf{H_{M0}}\)) with enough confidence.  
This does not mean, however, that the three editor approaches do not differ in editing efficiency in general. Negative or inconclusive results relate only to the specific context that was tested [9]. In our case, the most probable cause of the obtained results was the simplicity of the language on one side and the proficiency of the subjects on the other side. The participants were master's degree computer science students, often with professional programming experience, while DSLs are usually targeted at non-programming domain experts. Subjects without any previous programming experience often fail to cope with concepts that are obvious to developers, such as nested parentheses [10]. Therefore, we could expect different results for other participant populations.