# Active Pattern Classification for Automatic Visual Exploration of Multi-Dimensional Data
## 2 Related Work
### Active Sample Recommendation  
Deep learning has developed rapidly and received widespread attention in the past decade. Obtaining sufficient training samples is a prerequisite for building any deep-learning model. Model developers, therefore, have to examine a large number of instances and assign each instance a label, which is time-consuming and costly.  
Active learning [14] is an approach to recommending samples that are most likely to improve the model's prediction accuracy. This type of learning provides indicators to quantify samples' classification uncertainty. Model developers thus should select samples with higher uncertainty to label, which are more likely to be misclassified [15]. Many active-learning indicators are calculated based on the distribution of the classifier's penultimate layer (each value indicates the probability of a sample belonging to a class), such as least significant confidence [16], smallest margin [17], entropy-based sampling [18], Simpson diversity [19], etc. Another type of indicator follows the ensemble strategy, constructing multiple weak classifiers and selecting samples with the lowest consistency of their classification results in these weak classifiers. Representative multi-model based indicators include vote entropy [20; 21] and Kullback-Leibler average [22]. Most active-learning indicators select samples that can significantly improve models' classification accuracy. In contrast, we need an indicator to reveal how likely a sample is to not belong to any existing class. None of the existing active-learning indicators are for this purpose.  
Interactive labeling provides model developers with visual interfaces to identify, select, and label interesting instances to improve the labeling flexibility [10]. Many visualization tools integrate new workflows to reduce the samples to be labeled [23; 24; 25; 26]. [PERSON] et al. [11] utilized the 1-nearest neighbor algorithm, which can automatically propagate the labels to unlabeled samples. Model developers thus only need to label a small number of samples. [PERSON] et al. [12] proposed a cluster-clean-label workflow to iteratively optimize labeled clusters' purity. [PERSON] et al. [27] summarized a series of strategies for interactive labeling. However, relying solely on visual and interactive interfaces without any quantitative reference may produce sub-optimal sample selection [28].  
Many methods combine the strengths of active learning and visualization to balance the efficiency and flexibility of labeling [29; 30]. A common strategy is using active-learning indicators to select informative samples and visual interfaces to control the labeling process [28; 31; 32]. However, these methods target objects with fixed classes, and are inapplicable for the scenarios of pattern classification. [PERSON], [PERSON], and [PERSON] [33] proposed exploratory labeling, which allows model developers to customize and refine labels over time. However, the labels depend on the developer-specified \"seed terms\", reflecting their subjective intent. Instead, our method can automatically identify samples with new pattern classes to update the classifier, avoiding unnecessary human intervention.