# TrendFlow: A Machine Learning Framework for Research Trend Analysis
## 5 Results
### Quantitative Evaluation Results
#### 5.1.3 Keyphrase Generation Evaluation  
The training and validation results for KeyBART-adapter and KeyBART are illustrated in Figures 4 and 5, respectively. In Figure 4, the best configuration (Ir = 7.3 \(\times\) 10\({}^{-5}\)  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Adjusted Rand Index** & **Silhouette Score** & **Calinski-Harabasz Index** \\ \hline K-means + PCA & 0.6944 & 0.09061 & **18.410** \\ GMM + PCA & 0.7065 & **0.09064** & 18.402 \\ DBSCAN + PCA & - & - & - \\ Agglomerative + PCA & 0.6209 & 0.0841 & 17.212 \\ Spectral + PCA & **0.7087** & 0.0900 & 18.395 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Clustering evaluation results for the clustering algorithms used in TrendFlow framework with dimensionality reduction.  
hidden = 32) is highlighted in bold blue, and we only display a subset of five configurations for clarity. These selected configurations have the same learning rate and only vary from hidden dimensions. In Figure 5, the best configuration (lr = 2.9 \(\times\) 10\({}^{-5}\)) is highlighted in bold sky-blue, and we only display a subset of five configurations for clarity. The complete set of results is available on wandb ([[https://api.wandb.ai/links/leoxiang66/uo9](https://api.wandb.ai/links/leoxiang66/uo9) odosw]([https://api.wandb.ai/links/leoxiang66/uo9](https://api.wandb.ai/links/leoxiang66/uo9) odosw), accessed on 5 June 2023).  
Following the training and fine-tuning of hyperparameters, we have identified the optimal configurations for both KeyBART-adapter and KeyBART, as detailed below:  
1. KeyBART-adapter: A learning rate of 7.3 \(\times\) 10\({}^{-5}\) was used, with a hidden dimension of 32 for the adapter layers. The model was trained for 34 epochs on the training set.
2. KeyBART: A learning rate of 2.9 \(\times\) 10\({}^{-5}\) was applied, and the model was trained for 45 epochs on the training set.  
Subsequently, we load the model weights for these two configurations and assess their performance on the test set. The evaluation results are presented in Table 3. The KeyBART-adapter outperforms KeyBART in terms of F-score@M (33.25% vs. 32.75%) and Precision@M (40.33% vs. 32.92%), indicating that the KeyBART-adapter model generates more accurate and relevant keyphrases. However, when it comes to Recall@M, KeyBART demonstrates a slightly higher value (33.08%) compared to the KeyBART-adapter (30.07%). This suggests that KeyBART is better at capturing a larger portion of relevant keyphrases within the top M generated keyphrases, albeit with a lower precision.  
Figure 4: Training and validation results for KeyBART-adapter, with running average smoothing applied for better visualization.  
Figure 5: Training and validation results for KeyBART, with running average smoothing applied for better visualization.  
In summary, the KeyBART-adapter exhibits an overall superior performance in keyphrase generation, particularly in terms of F-score@M and Precision@M. Although KeyBART achieves a marginally higher Recall@M, it is essential to consider the trade-off between precision and recall when selecting the most suitable model for a specific application. In the realm of research trend analysis, scholars are likely to prioritize a smaller number of precise trends over a larger collection of relevant but less accurate trends. This preference stems from the desire to focus on the most critical issues and developments within a given field, enabling researchers to receive targeted guidance and make more informed decisions. As such, placing emphasis on precision would be a prudent approach.