As the landscape of research on Large Lan- guage Models (LLMs) rapidly evolves, un- derstanding the taxonomy of these models becomes increasingly crucial for researchers. This study explores the automatic classifica- tion of survey papers related to LLMs, utilizing graph representation learning and various data manipulation techniques. By collecting and an- alyzing metadata from 144 literature reviews, we construct co-category graphs to evaluate and compare the effectiveness of different classifica- tion paradigms, including pre-trained language models and graph neural networks. Our find- ings indicate that leveraging graph structures significantly enhances taxonomy classification performance compared to traditional language models. Additionally, we demonstrate the ad- vantages of using weak labels generated from smaller models, revealing new insights into weak-to-strong generalization. This research not only contributes to the understanding of LLM taxonomy but also provides a framework for future explorations in the field, highlighting the importance of innovative evaluation meth- ods in navigating the complexities of LLM re- search. 1 Introduction AI techniques have been widely applied to var- ious domains, such as images (He et al., 2016; Dosovitskiy, 2020), texts (Vaswani et al., 2017; De- vlin et al., 2018), and graphs (Kipf and Welling, 2016; Zhuang and Al Hasan, 2022). As a critical subset of AI techniques, Large Language Models (LLMs) have gained significant attention in recent years (Radford et al., 2018, 2019; Brown et al., 2020; Achiam et al., 2023; Bai et al., 2022; Team et al., 2023). Especially, more and more new be- ginners are interested in the research topics about LLMs. To learn the recent progress in this field, new beginners commonly will read survey papers about LLMs. Therefore, to facilitate their learn- ing, numerous survey papers on LLMs have beenpublished in the last two years. However, a large amount of these survey papers can be overwhelm- ing, making it challenging for new beginners to read them efficiently. To embrace this challenge, in this project, we aim to explore and analyze the metadata of LLMs survey papers, providing in- sights to enhance their accessibility and understand- ing (Zhuang and Kennington, 2024). Specifically, we aim to develop a comprehensive framework for analyzing and exploring the data to uncover under- lying patterns, manipulating the data to enhance its quality and relevance, and evaluating the results to ensure robust and actionable insights. After analyz- ing the data, our plan includes the following key components: Exploration: Conducting an in-depth analysis to identify trends and anomalies within the dataset. Manipulation: Applying techniques to clean, transform, and prepare the data for analysis, en- suring its integrity and suitability for our required objectives. Evaluation: Assessing the effectiveness of our methods and results through statistical measures and validation techniques to confirm their reliabil- ity and significance. Overall, our contributions can be summarized as follows which will be covered more in detail under the methodology section: • Data Exploration • Data Manipulation • Data Evaluation 2 Related Work This study builds on the emerging body of research focused on automating the classification of sur- vey papers in the rapidly evolving field of Large Language Models, leveraging graph representation learning to enhance taxonomy assignment (Zhuang and Kennington, 2024). 3 Methodology 3.1 Data Exploration In this section, we delve into the data exploration process conducted on the survey dataset to uncover insights into research trends, taxonomy distribu- tion, author contributions, and thematic content of papers. The dataset comprises various attributes, including release dates, taxonomy categories, au- thors, and summaries, which facilitate a compre- hensive analysis. A. Dataset Loading and Overview: The dataset was loaded using the Pandas library. The initial step involved importing the CSV file containing survey data, which included critical columns necessary for subsequent analyses. This dataset serves as the foundation for understanding trends in survey papers, categorization, and author contributions over time. B. Trends Over Time: To analyze the release trends of survey papers, we converted the ’Re- lease Date’ column to a datetime format and sub- sequently grouped the data by year and month. This process allowed us to visualize the number of papers released each month. As a result, a line plot was generated to illustrate the trends in sur- vey papers over time. Each data point was marked for clarity, enabling easy identification of peaks and troughs in the publication rate. The findings revealed interesting patterns in research activity, which may correlate with emerging topics or ad- vancements in the field as shown in Figure 1. Ad- ditionally, we calculated the mean number of sur- veys released per month, which provided a quanti- tative measure of research activity during the study period. The analysis indicated that the average number of surveys published monthly was approxi- mately 9.6, suggesting sustained interest and pro- ductivity in the field. C. Taxonomy Distribution: The distribution of taxonomy categories assigned to survey papers was evaluated by counting the frequency of each category. A bar chart was created to visually repre- sent this distribution, highlighting the diversity of topics explored in the surveys as shown in Figure 2. Among the categories, the "Trustworthy" classi- fication was particularly noteworthy, with a total of 26 papers assigned to this category, showcasing a significant focus on reliability in research findings. D. Further Exploration: 2021-07 2022-08 2022-12 2023-02 2023-03 2023-04 2023-05 2023-06 2023-07 2023-08 2023-09 2023-10 2023-11 2023-12 2024-01 Year-Month0.02.55.07.510.012.515.017.520.0Number of Surveys Released Number of Survey Papers Over TimeFigure 1: Survey Trends Trustworthy ComprehensivePromptingScience RecSys & IR Multi-modal & Pre-trainingEvaluation Software EngineeringAdaptation TuningRoboticsGraphs OthersLaw Finance Education Hardware Architecture T axonomy0510152025Number of PapersDistribution of T axonomy Categories Figure 2: Taxonomy Distribution Bar Chart •Histogram of Release Dates To gain insights into the annual distribution of survey papers, a histogram of the release years was plotted. This visualization demon- strated the concentration of papers published in certain years, providing context to research trends which can be observed from the Figure 3. 2021 2022 2023 2024 Year020406080100120140Number of PapersDistribution of Survey Papers by Release Year Figure 3: Release Year Histogram •Category Distribution Pie To further explore research topics, a pie chart was generated to illustrate the distribution of research categories. Each category was rep- resented proportionally, with external labels detailing the percentage share of each cate- gory within the dataset as shown in Figure 4. Figure 4: Category Distribution Pie Chart •Author Contributions A violin plot was used to visualize the con- tributions of the top authors in terms of the number of papers published. This plot high- lighted disparities in author output, revealing key contributors in the field as depicted from the Figure 5. 2 3 4 5 6 7 8 Number of PapersPhilip S. Yu Enhong Chen Yufei Wang Xin Wang Yan Liu Caiming Xiong Lei Li Yixin Liu Wensheng Gan Yue Zhang Jian Pei Jindong Wang Xing Xie Ji-Rong Wen Lichao Sun Zishan Guo Bing Qin Roberta Raileanu Tianming Liu Yang LiuAuthorsT op 20 Authors by Number of Papers (Violin Plot with Jitter) Figure 5: Contribution of Authors •Word Cloud Analysis Finally, a word cloud was generated from the summaries of the papers, providing a visual representation of the most frequently men- tioned terms. This technique helps in under- standing prevalent themes and topics in the research literature which could be observed from the Figure 6. Word Cloud of Paper SummariesFigure 6: Word Cloud 3.2 Data Manipulation 1.Building the Feature Matrix: The first step in data manipulation was to create a feature matrix from the dataset using the appropriate function. This function transforms three key features: Title, Summary, and Categories. •Title and Summary Vectorization: The TfidfVectorizer from the sklearn.feature_extraction.text module was used to convert the text data in the ’Title’ and ’Summary’ columns into numerical format. This transformation creates a Term Frequency-Inverse Document Frequency (TF-IDF) representation, where each unique word is represented by a vector. Thestop_words=’english’ parameter is employed to exclude common English words that do not contribute significant meaning (e.g., "the", "is"). •One-Hot Encoding of Categories: The ’Categories’ column was pro- cessed by splitting the string of cate- gories into separate columns using the str.split method. Following that, thepd.get_dummies() function was uti- lized to perform one-hot encoding, gen- erating binary columns for each category. This allows the model to recognize the presence of each category as a separate feature. •Combining Features: The feature matrices from the TF-IDF transformations and the one-hot encoded categories were combined into a single feature matrix using pd.concat() . This results in a comprehensive feature set that encapsulates information from both textual data and categorical labels. 2.Normalizing the Data: Normalization was applied to the feature matrix using the MinMaxScaler from the sklearn.preprocessing module. This pro- cess rescales the feature values to a range of [0, 1], ensuring that all features contribute equally to the model training. Normalization is particularly important in machine learning as it improves convergence during training and mitigates the influence of features with larger ranges. 3.Encoding the Labels: The categorical labels in the ’Categories’ col- umn were encoded using the LabelEncoder . This encoding converts the string labels into integers, making them suitable for classifica- tion tasks. Each unique category is assigned a distinct integer value, which facilitates the training of machine learning algorithms. 4.Splitting the Dataset: The dataset was di- vided into training and testing sets using thetrain_test_split function from the sklearn.model_selection module. A test ratio of 0.4 was specified, meaning that 40% of the data would be allocated for testing while the remaining 60% would be used for training the model. The random state was set to 42 in order to ensure reproducibility of the split. 5.Verification of Dataset Shapes: After split- ting the dataset, the shapes of the resulting training and testing feature matrices and la- bels were printed to verify the integrity of the operation. The shapes confirm that the dataset was correctly divided, with 86 samples in the training set and 58 samples in the testing set. Output: Training features shape: (86, 3547); Testing features shape: (58, 3547); Training labels shape: (86,); Testing labels shape: (58,) The data manipulation steps outlined above effectively prepare the dataset for machine learning modeling by creating a robust fea- ture matrix, normalizing features, encoding labels, and splitting the dataset into training and testing sets. This preparation is crucialfor ensuring that the subsequent modeling pro- cess is accurate and efficient. 3.3 Data Evaluation In this section, we evaluate and compare the per- formance of two classification models: Logistic Regression and Support Vector Machine (SVM). The models were applied to a specific dataset, and their performances were measured using key evalu- ation metrics, including accuracy, precision, recall, and F1 score. 1.Model Training and Predictions: •Logistic Regression: LR is a statisti- cal method used for binary classification that models the probability of a categor- ical dependent variable based on one or more predictor variables using the lo- gistic function. The model was trained using the training data, and predictions were made on the test set. •Support Vector Machine (SVM): SVM is a supervised machine learning algo- rithm used for classification and regres- sion tasks that finds the optimal hyper- plane to separate data points of different classes in high-dimensional space. This model was trained in a similar manner, using the training data and making pre- dictions on the test set. Model Accuracy Precision Recall F1 Score Logistic Regression 0.4655 0.2790 0.4655 0.3209 SVM 0.4310 0.2179 0.4310 0.2744 Table 1: Comparison of Evaluation Metrics for Logistic Regression and SVM models. 2.Confusion Matrix Visualization: Confusion matrices were plotted for both models to pro- vide a visual representation of their perfor- mance. The confusion matrices show the dis- tribution of true positives, true negatives, false positives, and false negatives, which helps identify specific areas where each model may struggle. Both the confusion matrices can be observed through Figures 7 and 8. 3.Performance Comparison: A bar graph was created to compare the evaluation metrics of both models visually. The analysis revealed that - Logistic Regression outperformed SVM in all evaluation metrics, including accuracy, cs.AI cs.AI, cs.CL cs.AI, cs.CL, cs.CY, cs.MA cs.AI, cs.CL, cs.IR cs.AI, cs.CL, cs.LG cs.AI, cs.LG cs.AR, cs.CL, cs.LG cs.CL cs.CL, cs.AI cs.CL, cs.AI, cs.CR cs.CL, cs.AI, cs.CV cs.CL, cs.AI, cs.CV, cs.HC, cs.MA cs.CL, cs.AI, cs.CV, cs.MM cs.CL, cs.AI, cs.CY cs.CL, cs.AI, cs.CY, cs.LG cs.CL, cs.AI, cs.LG cs.CL, cs.CR, cs.LG cs.CL, cs.IR cs.CL, cs.LG cs.CR cs.CV cs.CV, cs.AI cs.CV, cs.AI, cs.CL, cs.LG cs.CV, cs.CL cs.CY cs.CY, cs.AI, cs.CL, cs.CV, cs.LG cs.CY, cs.AI, cs.CL, cs.LG cs.DL, cs.CL cs.DL, cs.CL, cs.CY cs.DL, cs.CL, cs.CY, cs.SI cs.HC cs.IR, cs.AI cs.IR, cs.AI, cs.CL cs.IR, cs.AI, cs.SE cs.LG cs.LG, cs.AI cs.LG, cs.AI, cs.CL cs.LG, cs.AI, cs.DC, cs.PF cs.LG, cs.AI, cs.SI cs.LG, cs.CL, cs.SI cs.NE, cs.AI, cs.CL cs.RO, cs.AI cs.SE cs.SE, cs.AI cs.SE, cs.AI, cs.CL, cs.PL cs.SE, cs.HC nan Predicted Labelcs.AI cs.AI, cs.CL cs.AI, cs.CL, cs.CY, cs.MA cs.AI, cs.CL, cs.IR cs.AI, cs.CL, cs.LG cs.AI, cs.LG cs.AR, cs.CL, cs.LG cs.CL cs.CL, cs.AI cs.CL, cs.AI, cs.CR cs.CL, cs.AI, cs.CV cs.CL, cs.AI, cs.CV, cs.HC, cs.MA cs.CL, cs.AI, cs.CV, cs.MM cs.CL, cs.AI, cs.CY cs.CL, cs.AI, cs.CY, cs.LG cs.CL, cs.AI, cs.LG cs.CL, cs.CR, cs.LG cs.CL, cs.IR cs.CL, cs.LG cs.CR cs.CV cs.CV, cs.AI cs.CV, cs.AI, cs.CL, cs.LG cs.CV, cs.CL cs.CY cs.CY, cs.AI, cs.CL, cs.CV, cs.LG cs.CY, cs.AI, cs.CL, cs.LG cs.DL, cs.CL cs.DL, cs.CL, cs.CY cs.DL, cs.CL, cs.CY, cs.SI cs.HC cs.IR, cs.AI cs.IR, cs.AI, cs.CL cs.IR, cs.AI, cs.SE cs.LG cs.LG, cs.AI cs.LG, cs.AI, cs.CL cs.LG, cs.AI, cs.DC, cs.PF cs.LG, cs.AI, cs.SI cs.LG, cs.CL, cs.SI cs.NE, cs.AI, cs.CL cs.RO, cs.AI cs.SE cs.SE, cs.AI cs.SE, cs.AI, cs.CL, cs.PL cs.SE, cs.HC nanTrue Label0000500000000000000000000 0100000000000000000000000 0000000001000000000000000 0000010000000000000000000 00001200000000000000000000 00000110000000000000000000 0000010000000000000000000 0000010000000000000000000 0000010000000000000000000 0000020002000000000000000 0000000001000000000000000 0000300000000000000000000 0000010000000000000000000 0000010000000000000000000 0000100000000000000000000 0000100000000000000000000 0000100000000000000000000 0000010000000000000000000 0000100000000000000000000 0000010000000000000000000 0000010000000000000000000 0000200000000000000000000 0000010000000000000000000 0000010000000000000000000 0000200000000000000000001LR Confusion Matrix 024681012 Figure 7: LR Confusion Matrix precision, recall, and F1 score which can be observed from the Figure 9. 4.Overall Performance: The average scores for each model were calculated to determine overall performance which showed that Logis- tic Regression’s average ccore was 0.3828 and that of SVM’s was 0.3740. The results indi- cate that Logistic Regression performed better overall, suggesting it may be a more suitable model for this dataset and classification task. This evaluation underscores the importance of using appropriate metrics to assess model performance and the potential for further im- provements through hyperparameter tuning and exploring additional algorithms. 4 Conclusion In this analysis, we thoroughly explored and ma- nipulated the dataset to derive meaningful insights for classification tasks. We employed two machine learning models: Logistic Regression and Support Vector Machine (SVM), to evaluate their perfor- mance in predicting outcomes based on textual fea- tures extracted from the data. The data pre-processing steps included trans- forming the text data into numerical representa- tions using TF-IDF, effectively capturing the sig- nificance of terms within the documents. Our eval- uation metrics — accuracy, precision, recall, and F1 score provided a comprehensive assessment of cs.AI cs.AI, cs.CL cs.AI, cs.CL, cs.CY, cs.MA cs.AI, cs.CL, cs.IR cs.AI, cs.CL, cs.LG cs.AI, cs.LG cs.AR, cs.CL, cs.LG cs.CL cs.CL, cs.AI cs.CL, cs.AI, cs.CR cs.CL, cs.AI, cs.CV cs.CL, cs.AI, cs.CV, cs.HC, cs.MA cs.CL, cs.AI, cs.CV, cs.MM cs.CL, cs.AI, cs.CY cs.CL, cs.AI, cs.CY, cs.LG cs.CL, cs.AI, cs.LG cs.CL, cs.CR, cs.LG cs.CL, cs.IR cs.CL, cs.LG cs.CR cs.CV cs.CV, cs.AI cs.CV, cs.AI, cs.CL, cs.LG cs.CV, cs.CL cs.CY cs.CY, cs.AI, cs.CL, cs.CV, cs.LG cs.CY, cs.AI, cs.CL, cs.LG cs.DL, cs.CL cs.DL, cs.CL, cs.CY cs.DL, cs.CL, cs.CY, cs.SI cs.HC cs.IR, cs.AI cs.IR, cs.AI, cs.CL cs.IR, cs.AI, cs.SE cs.LG cs.LG, cs.AI cs.LG, cs.AI, cs.CL cs.LG, cs.AI, cs.DC, cs.PF cs.LG, cs.AI, cs.SI cs.LG, cs.CL, cs.SI cs.NE, cs.AI, cs.CL cs.RO, cs.AI cs.SE cs.SE, cs.AI cs.SE, cs.AI, cs.CL, cs.PL cs.SE, cs.HC nan Predicted Labelcs.AI cs.AI, cs.CL cs.AI, cs.CL, cs.CY, cs.MA cs.AI, cs.CL, cs.IR cs.AI, cs.CL, cs.LG cs.AI, cs.LG cs.AR, cs.CL, cs.LG cs.CL cs.CL, cs.AI cs.CL, cs.AI, cs.CR cs.CL, cs.AI, cs.CV cs.CL, cs.AI, cs.CV, cs.HC, cs.MA cs.CL, cs.AI, cs.CV, cs.MM cs.CL, cs.AI, cs.CY cs.CL, cs.AI, cs.CY, cs.LG cs.CL, cs.AI, cs.LG cs.CL, cs.CR, cs.LG cs.CL, cs.IR cs.CL, cs.LG cs.CR cs.CV cs.CV, cs.AI cs.CV, cs.AI, cs.CL, cs.LG cs.CV, cs.CL cs.CY cs.CY, cs.AI, cs.CL, cs.CV, cs.LG cs.CY, cs.AI, cs.CL, cs.LG cs.DL, cs.CL cs.DL, cs.CL, cs.CY cs.DL, cs.CL, cs.CY, cs.SI cs.HC cs.IR, cs.AI cs.IR, cs.AI, cs.CL cs.IR, cs.AI, cs.SE cs.LG cs.LG, cs.AI cs.LG, cs.AI, cs.CL cs.LG, cs.AI, cs.DC, cs.PF cs.LG, cs.AI, cs.SI cs.LG, cs.CL, cs.SI cs.NE, cs.AI, cs.CL cs.RO, cs.AI cs.SE cs.SE, cs.AI cs.SE, cs.AI, cs.CL, cs.PL cs.SE, cs.HC nanTrue Label0000500000000000000000000 0100000000000000000000000 0000100000000000000000000 0000010000000000000000000 00001200000000000000000000 00000110000000000000000000 0000010000000000000000000 0000010000000000000000000 0000010000000000000000000 0000030001000000000000000 0000000001000000000000000 0000300000000000000000000 0000010000000000000000000 0000010000000000000000000 0000100000000000000000000 0000100000000000000000000 0000100000000000000000000 0000010000000000000000000 0000100000000000000000000 0000010000000000000000000 0000010000000000000000000 0000200000000000000000000 0000010000000000000000000 0000010000000000000000000 0000300000000000000000000SVM Confusion Matrix 024681012 Figure 8: SVM Confusion Matrix Accuracy Precision Recall F1 Score Metrics0.00.10.20.30.4ScoreComparison of Evaluation Metrics: Logistic Regression vs SVM Logistic Regression SVM Figure 9: LR vs SVM Model Comparison model performance. The results indicated that Lo- gistic Regression outperformed SVM across all metrics, achieving an accuracy of 0.4655, precision of 0.2790, and recall of 0.4655, thereby highlight- ing its suitability for this particular classification problem. The insights gained from this analysis under- score the importance of selecting the appropriate model based on performance metrics tailored to the specific dataset and task. Future work may focus on optimizing model parameters, experimenting with additional features, and exploring advanced algorithms to further enhance predictive accuracy. Overall, this exploration and evaluation process il- lustrates the critical role of data manipulation and modeling in deriving actionable insights from com- plex datasets. A APPENDIX