# TrendFlow: A Machine Learning Framework for Research Trend Analysis
## 3 Framework Design
### Keyphrase Generator Module  
Once we have \(k\) clusters \(C_{1}\),..., \(C_{k}\), the keyphrase generator generates keyphrases for each cluster \(C_{i}\). In this work, we use KeyBART [4] as the keyphrase generator. KeyBART is a task-specific language model that learns rich representation of keyphrases from text documents by using different masking strategies for pre-training transformer language models. It achieves state-of-the-art performance on keyphrase generation and near state-of-the-art performance on other NLP tasks such as named entity recognition, relation extraction, question answering, and summarization. In order to enhance KeyBART's comprehension and generation of research trends, we manually annotate an RTA dataset of 90 samples, each containing an abstract and its corresponding research trend, for the purpose of fine-tuning. To be more specific, the first author examined the most recent papers on Papers with Code ([[https://paperswithcode.com/](https://paperswithcode.com/)]([https://paperswithcode.com/](https://paperswithcode.com/)), accessed on 5 June 2023) and Arxiv ([[https://arxiv.org/](https://arxiv.org/)]([https://arxiv.org/](https://arxiv.org/)), accessed on 5 June 2023), providing annotations for the identified research trends. The research trends follow the concatenated keyphrase pattern, as shown in Figure 2. The majority of the chosen papers belong to the fields of machine learning and deep learning, which align with the first author's area of expertise. Subsequently, we partitioned the dataset into training (50 samples), validation (20 samples), and test (20 samples) subsets. We made this dataset public on Huggingface. Given the small size of our dataset, fully fine-tuning KeyBART (consisting of 560 M parameters) may result in a loss of previously acquired knowledge. Consequently, we investigate the integration of the adapters technique [6] with KeyBART. By adding adapter layers between KeyBART's decoder layers and freezing all original parameters, we train only the newly introduced adapter layers. This approach aims to retain KeyBART's original knowledge while enhancing its understanding of research trends. More specifically, we follow the methodology presented in [45]. Each adapter layer comprises a layer normalization layer, succeeded by a down-projection linear layer, a ReLU activation function, an up-projection linear layer, and a residual connection. This configuration enables compression. The input and output dimensions of the adapter layers are adjusted to match KeyBART's decoder layer, while the middle hidden layer's dimension serves as a hyperparameter. In this work, we experiment with hidden dimensions of 32, 64, 128, 256, and 512. We refer to this model architecture as KeyBART-adapter-d, where \(d\) denotes the hidden dimension. In our implementation, KeyBART-adapter-32 encompasses 0.8 M trainable parameters, while KeyBART-adapter-512 comprises 12 M trainable parameters. The considerable decrease in parameter number implies that the fine-tuning process could potentially be much more efficient. The detailed structure of the adapter layer can be found in Figure 3.  
Assuming cluster \(C_{i}\) comprises \(m\) pieces of literature, the keyphrase generator generates research trends for each literature within the cluster, yielding \(m\) predicted research trend strings. The final step in TrendFlow involves grouping similar keyphrases. For example, the keyphrase \"Natural Language Processing\" should be recognized as identical to \"natural language processing\" or \"Natural Language Processing (NLP).\"  
Initially, we split the \(m\) raw research trend strings using a semicolon delimiter to obtain a list of keyphrases. Subsequently, we apply the union-find algorithm [46]--with the Ratcliff/Obershelp algorithm [47] serving as the similarity measurement--to group keyphrases similar in token level. We set the similarity threshold at 0.8, which implies that only keyphrases with similarity scores exceeding 0.8 will be grouped together.