# Datamator: An Authoring Tool for Creating Datamations via Data Query Decomposition
## 7 Evaluation
### User Study  
To verify that the datamations generated by Datamator can effectively convey insights, we reperformed the experiment (Study-I) described in [1]. In particular, 40 lab students (18 males and 22 females, mean age 25.1, SD 1.77) from a design college with a background in visual communication design were invited to take part in our study. They were divided into two groups (i.e., \(G_{1},G_{2}\)), where each group had 20 members. A between-subject study was performed, in which two groups of people were asked to answer the same question  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Models** & **Exact Match** & **SARI** \\ \hline \(\mathcal{B}_{1}\) & 25.64\% & 0.759 \\ \(\mathcal{B}_{2}\) & 38.39\% & 0.812 \\ \(\mathcal{D}\) & **82.23\%** & **0.876** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Our model (\(\mathcal{D}\)) outperforms the two baseline models (\(\mathcal{B}_{1}\) and \(\mathcal{B}_{2}\)).  
by exploring the data via datamations generated by Datamator and the static charts (the last frames of the datamations), respectively. The study sessions lasted about an hour and participants were compensated with USD 15.  
**Procedure and Tasks** During the study, the participants were asked to resolve a paradox as shown in Figure 6: it seems people with a higher education level (Ph.D.) have a lower average income (Figure 6c), but when we take the work fields into consideration, we obtain the opposite result, i.e., people with a higher education level have a higher income in both industry and academia (Figure 6d). The truth is that people with a lower education level tended to work in industry, where a higher salary is usually paid, and these people outnumbered the people with a higher education level by a large margin. We visualized the two opposite sides of the paradox in a pair of datamations generated by Datamator as illustrated in Figure 6a,b. We showed these datamations to the participants in \(G_{1}\) and showed the last frames as static charts to the participants in \(G_{2}\). These participants were asked to read these visual representations carefully and then select the truth of the paradox from eight potential choices (seven were distractors) provided by us (refer to [1] for details about the choices). Our hypothesis was that \(G_{1}\) tends to have higher accuracy than \(G_{2}\).  
**Results** As a result, only 40% of participants in \(G_{2}\) correctly found out the truth of the paradox, while 75% of participants in \(G_{1}\) did so. A chi-squared test showed that \(G_{1}\) performed significantly better (\(\chi^{2}(1,N=40)=5.01\), \(p<0.05\), one-tailed) than \(G_{2}\) in the experiment. The gap between \(G_{1}\) and \(G_{2}\) was 35%, which was a sizeable difference with [PERSON]'s h-value equal to 0.68.  
**Discussion** This outcome serves to provide empirical validation for our hypothesis. Specifically, the participants enrolled in Group \(G_{1}\) exhibited a notable trend towards achieving enhanced levels of accuracy when compared to their counterparts within Group \(G_{2}\). Building upon the implications of this finding, we conducted a validation of the efficacy of the datamation generated by Datamator in successfully imparting insightful information. Furthermore, these findings confirm the effectiveness of the visualization actions we designed for a variety of analytical operations.  
Figure 6: In the user study, the participants were asked to resolve a paradox shown in this figure: case (**a**) people with a higher education level have lower average income (**c**); case (**b**), however, when we take the work field into consideration, people with higher education have higher average income (**d**), therefore the contrary is true.