# CFP-AL: Combining Model Features and Prediction for Active Learning in Sentence Classification
## 5 Analysis  
To analyze the fact that Entropy Sampling or CAL has relatively large performance deviations for each experiment and even for each iteration, we show the distribution of the feature score and final scores suggested by CAL and CFP-AL in Figure 6. As analyzed in Section 3.1, during the training, the data become clustered, and there are very few data points near the decision boundary based on the labeled data pool. As a result, among the data sampled in each iteration, there are very few data points with high scores.  
Therefore, while the scoring method of CAL, which aims to sample data near the decision boundary, is certainly effective, it is meaningful for an extremely small number of data points. In contrast, CFP-AL considers the anisotropy problem of BERT features and strives to maximize the differences in features through Equation (1), resulting in relatively uniform differences in feature scores. Consequently, the final CFP-AL score also shows a relatively even distribution. This demonstrates that CFP-AL is not highly dependent on the classifier's decision boundary state and has discriminative standards even for data that are not near the decision boundary. Even among data that are dense and have almost similar output probabilities, presenting classification criteria through features was novel in improving performance by making the differences between the data more visible.  
Through prior research and preliminary experiments presented in Figure 3, we empirically discovered that feature is more crucial in the early iterations as the model focuses on clustering. Conversely, once a sufficient amount of data has been clustered and stabilized, uncertainty-based sampling becomes more important. Therefore, CFP-AL achieved good results by setting \(\alpha\) to give more weight to features in the early iterations and more weight to model predictions in the later iterations. However, since this was determined empirically, we conducted additional experiments to observe the effects of different \(\alpha\).  
We conducted experiments by fixing \(\alpha\) at 0.5 to give equal weight to features and prediction in each iteration. Additionally, we set \(\alpha\) to 1-0.1\(c\), giving higher weight to model predictions in the early iterations and higher weight to features in the later iterations.  
Figure 6: Scores of data sampled from iterations 1, 4, 7 of the QQP dataset. The x-axis represents the number of data points sampled in each iteration, and the y-axis represents the scaled score. Green is CAL’s final score, blue is CFP-AL’s feature score, and red is CFP-AL’s final score. In all datasets, the charts are almost similar.  
As expected, the approach of weighting features more in the early iterations and predictions more in the later iterations demonstrated the most consistent and superior performance, as shown in Table 4. This confirms our previous observations and insights, including the degree of clustering during the model's training, the small amount of data near the decision boundary, and the fact that various methods showed similar performance in the early iterations, but uncertainty-based acquisition functions performed better in the later iterations.