# CFP-AL: Combining Model Features and Prediction for Active Learning in Sentence Classification
###### Abstract  
Active learning has been a research area conducted across various domains for a long time, from traditional machine learning to the latest deep learning research. Particularly, obtaining high-quality labeled datasets for supervised learning requires human annotation, and an effective active learning strategy can greatly reduce annotation costs. In this study, we propose a new insight, CFP-AL (Combining model Features and Prediction for Active Learning), from the perspective of feature space by analyzing and diagnosing methods that have shown good performance in NLP (Natural Language Processing) sentence classification. According to our analysis, while previous active learning strategies that focus on finding data near the decision boundary to facilitate classifier tuning are effective, there are very few data points near the decision boundary. Therefore, a more detailed active learning strategy is needed beyond simply finding data near the decision boundary or data with high uncertainty. Based on this analysis, we propose CFP-AL, which considers the model's feature space, and it demonstrated the best performance across six tasks and also outperformed others in three Out-Of-Domain (OOD) tasks. While suggesting that data sampling through CFP-AL is the most differential classification standard, it showed novelty in suggesting a method to overcome the anisotropy phenomenon of supervised models. Additionally, through various comparative experiments with basic methods, we analyzed which data are most beneficial or harmful for model training. Through our research, researchers will be able to expand into the area of considering features in active learning, which has been difficult so far. NLP (Natural Language Processing); supervised fine-tuning; active learning; sentence classification +
Footnote †: journal: Applied Sciences  
0
Footnote 0: _Article_  
0
Footnote 0: _Article_  
exploration. It is well known that training on data with high uncertainty can easily improve performance [5, 6, 7], but paradoxically, there is insufficient analysis explaining why this leads to better results. The limited research on active learning in text classification can be attributed to the Anisotropy phenomenon, where the representations of BERT-like models become overly concentrated in a narrow space [8, 9, 10], making it difficult to distinguish between data points meaningfully. As a result, simple uncertainty-based sampling methods tend to achieve the best performance effortlessly. However, as mentioned earlier, active learning has also been used in various studies involving LLMs for different purposes, and there is more to explore in text classification beyond simply achieving the highest performance with minimal data. In our research, we build on previous efforts to select data near the decision boundary [11] by analyzing the visualized feature space to propose a more effective sampling method. We fine-tune a pretrained model cumulatively using a subset of data sampled through various active learning strategies and compare changes in the feature space, task performance, and robustness as measured by performance on OOD (Out-Of-Domain) tasks. To find a good active learning strategy and improve existing methods for selecting data near the decision boundary, it is essential to start by understanding the supervised fine-tuning process. Since the introduction of deep learning model architectures like Transformers and BERT [12, 13], fine-tuning them for downstream tasks has proven to be both simple and highly effective [14]. While there has been extensive research on the changes that occur during the fine-tuning process [15, 16], ironically, there has been relatively little research on active learning from the perspective of these changes. In text classification, active learning research has primarily focused on data informativeness, uncertainty, and diversity, with many methods being explored.