With the rapid development of research on large language models (LLMs), the number of re- lated survey papers is continuously increasing, posing challenges for scholars and researchers navigating this field. This study aims to apply machine learning techniques to effectively clas- sify survey papers on LLMs, thereby providing researchers with better literature retrieval and analysis tools. I first constructed a diverse fea- ture matrix by integrating text data and class la- bels from different datasets. Using preprocess- ing methods such as TF-IDF vectorization and one-hot encoding, I prepared for subsequent model training. In the experiments, I imple- mented a random forest classifier to analyze the relationship between features and classification labels. Preliminary results indicate that my clas- sification model achieved an accuracy of 26% without parameter tuning, which improved to 31% after hyperparameter optimization. De- spite challenges such as class imbalance and feature correlation, my research provides an ef- fective method for the automatic classification of survey papers. Future work will focus on further improving classification accuracy and exploring other machine learning algorithms to expand the applicability of such tasks. 1 Introduction AI techniques have been widely applied to var- ious domains, such as images (He et al., 2016; Dosovitskiy, 2020), texts (Vaswani et al., 2017; De- vlin et al., 2018), and graphs (Kipf and Welling, 2016; Zhuang and Al Hasan, 2022). As a critical subset of AI techniques, Large Language Models (LLMs) have gained significant attention in recent years (Radford et al., 2018, 2019; Brown et al., 2020; Achiam et al., 2023; Bai et al., 2022; Team et al., 2023). Especially, more and more new be- ginners are interested in the research topics about LLMs. To learn the recent progress in this field, new beginners commonly will read survey papersabout LLMs. Therefore, to facilitate their learn- ing, numerous survey papers on LLMs have been published in the last two years. However, a large amount of these survey papers can be overwhelm- ing, making it challenging for new beginners to read them efficiently. To embrace this challenge, in this project, we aim to explore and analyze the metadata of LLMs survey papers, providing in- sights to enhance their accessibility and understand- ing (Zhuang and Kennington, 2024). Specifically, we aim to build a robust feature matrix from a di- verse dataset, preprocess the data, and utilize a Ran- dom Forest classifier to analyze the relationships between various features and their corresponding labels. This involves vectorizing textual data using TF-IDF, applying one-hot encoding for categorical features, and normalizing the feature matrix to en- sure uniformity in model training. We will evaluate the model’s performance through various metrics to assess its effectiveness and identify potential areas for improvement. Overall, our contributions can be summarized as follows: •Construct a feature matrix that integrates both textual and categorical data, enhancing the model’s input representation. •Preprocess the data through normalization and encoding, preparing it for effective machine learning model training. •Implement and evaluate a Random Forest clas- sifier, providing insights into the classification performance and identifying challenges, such as class imbalance, that may affect the results. 2 Related Work Data preprocessing is a crucial step in the machine learning pipeline, as it directly impacts the quality and effectiveness of the models. In this project, I leverage various techniques from the pandas (pan- das) library to preprocess our dataset effectively. The use of pandas for data manipulation and clean- ing is well-documented in its official documenta- tion. In this study, I utilized key tools from scikit-learn (scikit-learn). for preprocess- ing, feature extraction, model training, and evaluation. The TfidfVectorizer from the feature_extraction module was used to con- vert paper titles and summaries into numerical TF- IDF vectors. I normalized the feature matrix using MinMaxScaler from the preprocessing module and split the dataset with train_test_split from themodel_selection module. I applied RandomForestClassifier from the ensemble module for classification and used GridSearchCV to fine-tune the model’s hyperparameters. For evaluation, accuracy_score , classification_report , and confusion_matrix from the metrics module were employed to assess accuracy and classification performance. (scikit-learn). In terms of data visualization, I utilized the Mat- plotlib (Matplotlib, 2021). library. Specifically, I used line plots and bar charts to illustrate the trends in survey data and the distribution of proposed tax- onomies. The library’s customizable features en- hanced the clarity of the visual results. Additionally, I used the NumPy (NumPy, 2024). and SciPy (SciPy, 2024). libraries for numerical computations and data processing. NumPy pro- vides essential functionalities for handling arrays and performing mathematical operations, while the sparse module of SciPy allows me to efficiently manage and combine sparse matrices. These li- braries facilitated the creation of a comprehensive feature matrix for my classification model. The de- tailed documentation of NumPy and SciPy offers valuable insights into their respective methods and applications, aiding me in effectively implementing my analysis. 3 Methodology 3.1 Data Exploration The dataset used in this project was loaded using the pandas library, which enabled efficient manipu- lation of the data for further analysis. The dataset consisted of survey papers with relevant features such as release dates and taxonomy classifications. First, I examined the temporal trends in the dataset. The "Release Date" field was converted into a datetime format, which allowed me to extractmonthly and yearly trends. By grouping the data by month, I plotted the number of surveys released over time. The figure revealed distinct publishing trends across different months, highlighting peri- ods of increased survey paper production, shown in figure 1. 2021-072022-082022-122023-022023-032023-042023-052023-062023-072023-082023-092023-102023-112023-122024-010.02.55.07.510.012.515.017.520.0Number of Surveys Released Figure 1: The number of survey papers published each month Next, I explored the distribution of paper classi- fications, referred to as "Taxonomy." Using a bar chart, I visualized the number of papers belonging to each taxonomy category. This provided insight into which categories had the most significant focus within the dataset, shown in figure 2. Trustworthy ComprehensivePromptingScience RecSys & IR Multi-modal & Pre-trainingEvaluation Software EngineeringAdaptation TuningRoboticsGraphs OthersLaw FinanceEducation Hardware Architecture T axonomy0510152025Number of Papers Figure 2: The distribution of the proposed taxonomy In addition to these foundational analyses, fur- ther exploration involved grouping the data by both "Year" and "Taxonomy" to examine trends across different taxonomies over time. The results were visualized in a multi-line plot, revealing how the prominence of each taxonomy evolved across years, shown in figure 3. Through these explorations, I gained a better un- derstanding of the dataset’s structure and trends, which laid the groundwork for subsequent model- ing and analysis. 3.2 Data Manipulation In this section, I focused on building and prepro- cessing a feature matrix from the dataset. The process started with text-based features, such as the "Title" and "Summary" columns, which were 2021 2022 2023 2024 Year05101520Number of Papers Trends of Paper T axonomy Over Years T axonomy Adaptation Tuning Comprehensive Education Evaluation Finance Graphs Hardware Architecture Law Multi-modal & Pre-training Others Prompting RecSys & IR Robotics Science Software Engineering TrustworthyFigure 3: Trends of Paper Taxonomy Over Years vectorized using the Term Frequency-Inverse Doc- ument Frequency (TF-IDF) method. This allowed me to represent the textual data as numerical val- ues, capturing the importance of specific words across documents. The TF-IDF vectorizer was ap- plied separately to both the "Title" and "Summary" columns, producing sparse matrices that mapped words to their corresponding TF-IDF scores. Next, I applied one-hot encoding to the "Cate- gories" column. Since each paper could belong to multiple categories, I split the categories into individual values and created binary indicators for each unique category. This transformation resulted in a matrix where each category was represented as a separate feature with a value of 1 or 0, indicating its presence or absence. After obtaining these individual feature matri- ces, I concatenated them to form a comprehensive feature matrix that incorporated both the TF-IDF features and the one-hot encoded categories. This matrix represented all the necessary features for further analysis. To preprocess the feature matrix, I normalized the data using the MinMaxScaler, ensuring that all features were scaled to a common range. This step was crucial for training machine learning mod- els, as it prevented any feature from disproportion- ately influencing the model due to its scale. After that I encoded the "Taxonomy" labels using a label encoder, transforming the categorical labels into numerical values for classification tasks. Finally, the dataset was split into training and testing sets using a 60/40 ratio, ensuring that themodel could be evaluated on unseen data. This step prepared the dataset for model training and evaluation. 3.3 Data Evaluation To evaluate the effectiveness of the feature ma- trix in classifying papers into their respective tax- onomies, I employed a Random Forest Classifier as our machine learning model. In my first attempt, the model was trained using default hyperparame- ters, resulting in an accuracy of 0.26. This initial accuracy indicates that the model’s performance was relatively low, suggesting that it struggled to effectively classify the test set. To enhance the model’s performance, I used hy- perparameter tuning using Grid Search in my sec- ond attempt. This process involved defining a pa- rameter grid that varied the number of estimators, maximum depth, minimum samples split, and min- imum samples per leaf. After optimizing the model with these configurations, the accuracy improved to 0.31, marking a 5% increase from the first attempt, shown in table 1. The increase in accuracy can be attributed to the model’s better tuning, which allowed it to learn from the training data more ef- fectively. Despite this improvement, the accuracy of 0.31 still reflects a relatively low performance. This indicates that there are still significant areas for enhancement in my model’s ability to classify the papers correctly. To assess the model’s performance, I employed standard evaluation metrics, including precision, recall, and F1-score. Additionally, I generated a classification report to provide detailed insights into the model’s performance across various cat- egories. A confusion matrix was also created to visualize the distribution of correct and incorrect predictions, helping to identify specific categories with higher misclassification rates. Future work will focus on further refining the model, potentially exploring different algorithms, additional feature engineering, and employing more advanced techniques to improve classifica- tion accuracy. Attempt Accuracy First attempt 0.26 Second attempt 0.31 Table 1: Model Accuracy Comparison 3.4 Additional Contributions of Sentiment Analysis In this study, I further assessed sentiment trends to complement the analysis of the paper dataset. To achieve this, I utilized the text classification pipeline from Hugging Face’s Transformers li- brary (HuggingFace, 2024) to perform sentiment classification on the titles of the papers, shown in figure 4. By analyzing each title, I was able to quickly capture the overall sentiment trends in the research field, providing valuable insights for sub- sequent research. The classification results were organized into a dataframe, and a pie chart was created to display the proportions of positive and negative labels. This approach not only improved the efficiency of the analysis but also added more dimensions to the research. 4 Conclusion In this report, I described the process of apply- ing machine learning techniques to classify survey papers, emphasizing the importance of data prepro- cessing, feature engineering, and model evaluation. By constructing a comprehensive feature matrix that integrates text and categorical data, I laid the foundation for effective model training. Prelimi- nary results showed that the accuracy of the random forest classifier reached 26%, highlighting the need for further improvement. By implementing hyperparameter tuning, I im- proved the model’s accuracy to 31%. While this is a step forward, it still underscores the challenges POSITIVE71.5%NEGATIVE 28.5%Proportion of Positive and Negative LabelsFigure 4: Sentiment Distribution of Paper Titles of class imbalance and feature relevance. My eval- uation metrics, including precision, recall, and F1 score, provided valuable insights into the model’s performance across different classification cate- gories. Future work will focus on addressing the identi- fied challenges, including updating datasets, evalu- ating new methods (such as prompting techniques in large language models), and solving class im- balance problems. By exploring other machine learning algorithms and incorporating more com- plex preprocessing methods, I hope to further en- hance the model’s effectiveness and reliability for better performance in classifying survey papers. Ultimately, this research contributes to the grow- ing field of data science, providing insights and methods applicable to similar classification tasks. A APPENDIX A.1 Experimental Environment Setup •Operating System: Windows 11 Home, 64-bit, x64-based processor •Processor: 11th Gen Intel(R) Core(TM) i7- 1165G7 @ 2.80GHz 2.80 GHz • Memory: 16.0 GB (15.7 GB usable) • Python Version: 3.12.5 • Library Versions Used: –pandas 2.2.2 –scikit-learn 1.5.2 –numpy 2.1.1 –matplotlib 3.5.3 –SciPy 1.14.1 A.2 Hyperparameter Settings The following hyperparameter settings were used during model training and tuning: • Random Forest Classifier Hyperparameters: –n_estimators : 100, 200 –max_depth : None, 10, 20 –min_samples_split : 2, 5 –min_samples_leaf : 1, 2 •Cross-Validation: 5-fold cross-validation was used for model tuning. A.3 Feature Matrix Construction and Preprocessing The following preprocessing steps were performed during feature matrix construction: •TF-IDF was applied to vectorize the "Title" and "Summary" fields, using up to 1000 fea- ture words per document. •One-hot encoding was applied to the "Cate- gories" field and integrated into the feature matrix. •The feature matrix was normalized using MinMaxScaler , mapping feature values to the [0, 1] range. A.4 Dataset Splitting The dataset was split as follows: • Training Set Proportion: 60% • Test Set Proportion: 40% • Random Seed: random_state=None A.5 Model Evaluation The following metrics were used to evaluate the model’s performance: • Accuracy • Precision, Recall, and F1-score • Confusion Matrix