# CFP-AL: Combining Model Features and Prediction for Active Learning in Sentence Classification
## 3 Active Learning
### CFP-AL (Combining Model Features and Prediction for Active Learning)  
#### 3.2.1 Definition  
Based on the above experimental results, we can observe that learning data with features that are extremely similar, based on Euclidean distance or cosine similarity, is not effective. In addition, we can see that sampling methods based on the model's classifier prediction have no choice but to sample data that are not actually high in uncertainty or are not near the decision boundary. And surprisingly, we can see that among data whose features are similar, there are data that are more helpful for learning. Therefore, we propose a sampling method that additionally reflects feature quality in addition to the sampling method based on the model's prediction uncertainty, which has previously shown good results.  
In particular, to overcome the anisotropy problem, which makes it difficult to use feature representations due to their small differences, we maximize the differences using both Euclidean distance and cosine similarity. Intuitively, as the Euclidean distance increases and the cosine similarity decreases, the differences between features become larger. Therefore, we define the feature score as shown in Equation (1).  
\[S_{F}(x)=\sum_{i=1}^{k}\left(\frac{d\big{(}\Phi(x),\Phi(x^{(i)})\big{)}}{\cos \_sim\big{(}\Phi(x),\Phi(x^{(i)})\big{)}}\right) \tag{1}\]  
where \(k\) is a hyperparameter and represents the number of nearest neighbors. Also, sampling methods based on the model's output prediction uncertainty have been proven over time to achieve the best performance [22; 23]. We define the prediction score by calculating the differences in the model's output probabilities for the data using KL divergence [11], as shown in Equation (2).  
\[S_{P}(x)=\sum_{i=1}^{k}\text{KL}\big{(}p(y|x^{(i)})||p(y|x)\big{)} \tag{2}\]  
To combine the feature score and prediction score, we align the scales using the simple Min-Max Scaling presented in Equation (3).  
\[\frac{\text{score}-\text{min}(\text{score})}{\text{max}(\text{score})-\text {min}(\text{score})} \tag{3}\]  
then adjust the ratio through a weighted \(\alpha\). The \(\alpha\) is empirically adjusted based on experience gained from various experiments: In the early iterations, the confidence in the uncertainty of the model prediction is low due to the small amount of data, so a relatively high weight is given to the feature score. As the iterations increase, the features of data become denser and more robust, and the confidence in the uncertainty of the model's predictions increases, so a higher weight is given to the prediction score. Therefore, we adjust the weights using the \(\alpha\) value so that more is given to the feature score in early  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Iteration**} \\ \cline{2-7}
**Dataset** & **1** & **2** & **3** & **4** & **5** & **6** & **7** \\ \hline QNLI & 70.42 & 91.23 & 94.41 & 94.92 & 92.24 & 92.74 & 95.29 \\ AGnews & 97.68 & 96.26 & 95.40 & 92.46 & 85.69 & 81.43 & 64.66 \\ IMDB & 94.25 & 88.51 & 95.89 & 92.27 & 95.07 & 86.32 & 95.71 \\ SST-2 & 66.72 & 95.91 & 92.42 & 96.52 & 95.21 & 93.65 & 94.92 \\ QQP & 86.57 & 62.79 & 81.85 & 55.25 & 92.27 & 93.82 & 82.60 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Ranking (%) of data sampled by the CAL method in each iteration. The order of ranking is the sum of Euclidean distances from the 10 nearest neighbor data.