# Active Pattern Classification for Automatic Visual Exploration of Multi-Dimensional Data
## 7 User Study
### Subjective Scores  
Figure 14 shows participants' scoring results. Almost all participants gave the highest scores on efficiency. Participants agreed that the tool could calculate visualizations' indicator scores in real-time. Moreover, they considered exploring a whole exploring space as helpful in improving IDE efficiency.  
**Insight 4**: The higher efficiency scores derive from the low computational complexity of the indicator. The RBVE framework thus enables the analyst to quickly identify abnormal samples from a batch of visualizations generated on the fly.  
Effectiveness receives relatively high scores. All participants acknowledged that most visualizations with low indicator scores contain new patterns. However, they also found a few exceptions with relatively high scores, but new patterns, which affects scores of this aspect. Moreover, two participants said they could not use a constant threshold during the data exploration. They sometimes had to change the threshold to find abnormal samples.  
**Insight 5**: Our method can guarantee that most low-scoring visualizations contain new patterns. However, a few exceptions are inevitable due to the inherent uncertainty of deep learning. The RBVE framework thus depends on the analyst to choose new patterns from visualizations with low indicator scores. The positive aspect is that the retraining frequency will gradually decrease as patterns covered by the PC increase, as in Figure 13.  
**Insight 6**: It is impossible to set a uniform threshold that is applicable for all cases. Our indicator can find a significant proportion of abnormal samples at lower values (e.g., <0.2; see Figure 10). However, after the PC has identified most data patterns, almost no low-scoring samples exist. At that time, analysts need to interactively increase the threshold to find abnormal samples in a broader range.  
Figure 14: Scoring results on four aspects.  
Figure 13: Number and time of PC retrainings for each participant.  
Scores of ease-of-use are high, and the score distribution is more concentrated. Participants agreed that they could easily understand the RBVE framework and grasp the usage of the tool. Several participants also pointed out a few limitations of the proof-of-concept tool and gave us suggestions to improve the system. For example, they suggested including more PCs to enable the analysts to explore multiple kinds of data patterns simultaneously. On the other hand, a few participants felt that it was inconvenient to start external programs to modify the PC structure and hoped the tool could support modifying the PC structure interactively.  
**Insight 7**: The tool only integrates core components to verify the feasibility of the RBVE framework. According to the experiment results, we believe implementing the functions suggested by the participants is feasible. For example, we can integrate more PCs into the tool. Analysts thus can explore multiple types of data patterns at the same time. High classification efficiency and low computation complexity ensure that the tool with multiple PCs can respond to user interactions in real-time. We plan to implement the functions suggested by the participants in the future.  
Participants' scores on favorability varied widely. Some of the participants agreed that our method could improve the IDE efficiency and scored this aspect high. However, participants who are more familiar with visual analytics pointed out that our approach is very different from the relevant research on authoring visualizations [3; 4; 70]. They do not know whether our method is better.  
**Insight 8**: Our method is not an improvement of existing methods that have already shown their usability in the demonstrated cases. Instead, we focus on a new IDE problem, i.e., identifying patterns from the generated visualizations, which has not received much attention. It is possible to use PCs and other techniques together in IDE.