The rapid advancements in large language models (LLMs) have resulted in an exponen- tial growth of survey papers. To manage and understand the evolving taxonomy of these surveys, this project employs graph representation learning including with clas- sical Machine learning algorithms. By treat- ing survey topics and their interrelationships as a graph, we explore the evolving taxon- omy in the context of large language mod- els. Our study highlights trends in survey papers, providing a comprehensive under- standing of their distribution across various research domains. The results reveal a clear trajectory of increasing specialization, em- phasizing the role of LLMs in areas such as prompt engineering, multimodal models, and application domains like education and finance. Graph-based analysis allows us to capture these trends more effectively and en- ables a better understanding of the survey landscape 1 Introduction AI techniques have been widely applied to var- ious domains, such as images (He et al., 2016; Dosovitskiy, 2020), texts (Vaswani et al., 2017; De- vlin et al., 2018), and graphs (Kipf and Welling, 2016; Zhuang and Al Hasan, 2022). As a critical subset of AI techniques, Large Language Mod- els (LLMs) have gained significant attention in recent years (Radford et al., 2018, 2019; Brown et al., 2020; Achiam et al., 2023; Bai et al., 2022; Team et al., 2023). Especially, more and more new beginners are interested in the research top- ics about LLMs. To learn the recent progress in this field, new beginners commonly will read sur- vey papers about LLMs. Therefore, to facilitate their learning, numerous survey papers on LLMs have been published in the last two years. How- ever, a large amount of these survey papers can be overwhelming, making it challenging for newbeginners to read them efficiently. To embrace this challenge, in this project, we aim to explore and analyze the metadata of LLMs survey papers, pro- viding insights to enhance their accessibility and understanding (Zhuang and Kennington, 2024). Specifically, we aim to employ graph represen- tation learning to analyze the taxonomy of survey papers on LLMs, focusing on both topical coverage and temporal trends. By leveraging this method- ology, we aim to provide insights into the current state of the field and how it is evolving, helping researchers and practitioners navigate the complex landscape of LLM-related literature. Overall, our contributions can be summarized as follows: •We introduce a graph-based framework to model and analyze the taxonomy of survey pa- pers on LLMs, providing a scalable approach for understanding the structure of research topics. •We offer a detailed temporal analysis of the growth in survey paper publications, highlight- ing key periods of increased activity and shift- ing research focus. •Through our analysis, we identify significant trends in LLM-related research, including emerging areas such as "Prompting Science" and the application of LLMs in "Finance" and "Education." •By constructing and visualizing the intercon- nections between survey topics, we provide a clear understanding of the relationships be- tween different research domains, enabling better navigation of the literature. 2 Methodology In this section, we begin by outlining the data col- lection process, followed by an analysis of the metadata. Next, we describe how three types of attributed graphs are constructed and explain how graph representations are learned using graph neu- ral networks. 2.1 Data Exploration The data for this project consists of survey papers published between July 2021 and January 2024, fo- cusing on large language models. We curated this dataset from publicly available research databases, extracting relevant metadata such as publication date, topics covered, and keywords associated with each paper. This information was used to create the graph structure for our analysis, with nodes representing topics and edges representing their relationships based on co-occurrence in survey pa- pers. 2021-7 2022-82022-122023-2 2023-3 2023-4 2023-5 2023-6 2023-7 2023-8 2023-92023-10 2023-11 2023-122024-1 Year-Month0.02.55.07.510.012.515.017.520.0Number of Surveys Released Figure 1: Trends of survey papers about large language models since 2021. Numbers reflect the year and month (e.g., 2023-3 is March 2023). 2.2 Data Manipulation In this phase, we focus on preparing the dataset by creating three graph types: text, co-author, and co-category. We build a TF-IDF matrix based on word frequency and distinctiveness for the titles and abstracts. To enrich the data, one-hot encoding is applied to arXiv categories, and this information is combined to generate the final feature matrix used for graph-based classification tasks The re- sults from our graph-based analysis provide sev- eral key insights into the taxonomy and trends of survey papers on LLMs: The number of surveys released has grown consistently from mid-2021 to late 2023, reflecting the rapidly expanding interest in LLM research. A sharp increase was observed from early 2022, peaking in mid-2023, as shown in the monthly publication trends (Figure from file [7]). This suggests an accelerated pace of develop- ment in the field, driven by both academic research and practical applications. The categorization of survey papers shows a diverse set of research ar- eas, with certain topics such as "Prompting Sci-ence" and "Evaluation" being particularly promi- nent (Figure from file [6]). This suggests that much of the current research is focused on understanding the best practices for prompting LLMs and eval- uating their performance across various domains. Beyond core areas, we observed emerging inter- est in application-specific domains like "Finance," "Law," and "Education." These areas reflect the expanding use cases of LLMs in industry-specific applications, indicating a shift toward more practi- cal, applied research. Trustworthy ComprehensivePromptingScience RecSys & IR Multi-modal & Pre-trainingEvaluation Software EngineeringAdaptation TuningRoboticsGraphs OthersLaw FinanceEducation Hardware Architecture T axonomy0510152025Number of Papers Figure 2: Distribution of classes in the taxonomy The graph representation allowed us to explore the interconnections between topics. For exam- ple, surveys discussing "Multimodal Models" were often linked to "Pre-training" methodologies, high- lighting the importance of foundational techniques in supporting new model architectures. Similarly, cross-disciplinary areas like "Software Engineer- ing" and "Hardware Architecture" frequently in- tersected with core LLM research, underlining the multidisciplinary nature of the field. 3 Result and Discussion The number of surveys released has grown consis- tently from mid-2021 to late 2023, reflecting the rapidly expanding interest in LLM research. A sharp increase was observed from early 2022, peak- ing in mid-2023, as shown in the monthly publica- tion trends (Figure 2). This suggests an accelerated pace of development in the field, driven by both academic research and practical applications. The categorization of survey papers shows a diverse set of research areas, with certain topics such as "Prompting Science" and "Evaluation" being par- ticularly prominent (Figure 1). This suggests that much of the current research is focused on under- standing the best practices for prompting LLMs and evaluating their performance across various domains. Beyond core areas, we observed emerg- ing interest in application-specific domains like "Finance," "Law," and "Education." These areas re- flect the expanding use cases of LLMs in industry- specific applications, indicating a shift toward more practical, applied research. The graph represen- tation allowed us to explore the interconnections between topics. For example, surveys discussing "Multimodal Models" were often linked to "Pre- training" methodologies, highlighting the impor- tance of foundational techniques in supporting new model architectures. Similarly, cross-disciplinary areas like "Software Engineering" and "Hardware Architecture" frequently intersected with core LLM research, underlining the multidisciplinary nature of the field. 4 Limitations and Future Scope 4.1 Limitations Despite the insights gained through this study, there are a few limitations to acknowledge: Our analysis is dependent on the availability and accessibility of survey papers. Some recent or domain-specific surveys might not be included due to dataset lim- itations. While graph representation learning ef- fectively captures relationships between topics, the complexity of research topics can lead to oversim- plification. Certain interdisciplinary areas may not be adequately captured by our clustering approach. The graph-based methodology is specifically tuned to analyze LLM-related survey papers. Its appli- cability to other fields of research remains to be tested and may require adjustments. 4.2 Future Scope This project opens up several avenues for future work: Future studies can expand the dataset by in- cluding more diverse sources of survey papers, par- ticularly those published in niche domains or non- English publications. Incorporating time-evolving graph models could help capture the dynamic evo- lution of research areas, enabling real-time updates of the taxonomy. Applying advanced natural lan- guage processing (NLP) techniques could allow for automated extraction of topics and relationships from survey papers, reducing manual curation ef- forts. The graph-based approach developed in this project could be adapted for analyzing survey pa- pers in other fast-evolving fields such as artificial intelligence, computer vision, or bioinformatics. 5 Conclusion In conclusion, our graph-based approach to under- standing survey paper taxonomy reveals a complexand evolving landscape of LLM research. The re- sults not only showcase the growth of the field but also provide a structured way to navigate the broad spectrum of topics, helping both researchers and practitioners identify key trends and emerging ar- eas of interest. A APPENDIX A.1 Data Collection: The dataset consists of survey papers focused on large language models (LLMs) published between July 2021 and January 2024. The metadata for each paper includes the title, au- thors, publication date, keywords, and arXiv categories. Data sources: Survey papers were retrieved from publicly available databases such as arXiv and other research repositories • •A.2 Graph Construction: Three types of attributed graphs were con- structed. Text Graph: Built using a TF-IDF matrix based on the title and abstract of each paper. This matrix was generated by calculat- ing word frequency and distinctiveness. Co- Author Graph: Nodes represent authors, and edges indicate co-authorship. Co-Category Graph: Nodes represent arXiv categories, with edges drawn between categories that fre- quently appear together in the same paper. One-Hot Encoding: Applied to arXiv cate- gories to generate categorical features •A.3 Evaluation Metrics: Classification accuracy, F1 score, and preci- sion were the key metrics used to evaluate the model’s performance. The evaluation was conducted through 5-fold cross-validation to ensure generalizability.