This paper explores trends in survey data over time, evaluates the distribution of tax- onomies, and compares different classification algorithms, including Naive Bayes, Random Forest, and Support Vector Machines (SVM). The performance of each classifier is compared using accuracy, precision, recall, and F1-score. It is found that Naive Bayes yields the best accu- racy for this dataset, especially when handling imbalanced data using SMOTE. 1 Introduction AI techniques have been widely applied to var- ious domains, such as images (He et al., 2016; Dosovitskiy, 2020), texts (Vaswani et al., 2017; De- vlin et al., 2018), and graphs (Kipf and Welling, 2016; Zhuang and Al Hasan, 2022). As a critical subset of AI techniques, Large Language Models (LLMs) have gained significant attention in recent years (Radford et al., 2018, 2019; Brown et al., 2020; Achiam et al., 2023; Bai et al., 2022; Team et al., 2023). Especially, more and more new be- ginners are interested in the research topics about LLMs. To learn the recent progress in this field, new beginners commonly will read survey papers about LLMs. Therefore, to facilitate their learn- ing, numerous survey papers on LLMs have been published in the last two years. However, a large amount of these survey papers can be overwhelm- ing, making it challenging for new beginners to read them efficiently. To embrace this challenge, in this project, we aim to explore and analyze the metadata of LLMs survey papers, providing in- sights to enhance their accessibility and understand- ing (Zhuang and Kennington, 2024). Specifically, we aim to identify the key trends in the evolution of survey papers on Large Language Models (LLMs) over recent years. By examining the distribution of research taxonomies, we will explore how dif- ferent areas within LLM research have grown andshifted over time. Furthermore, we will apply vari- ous machine learning classification methods—such as Naive Bayes, Random Forest, and Support Vec- tor Machines (SVM)—to predict these taxonomies using the metadata from survey papers. Address- ing challenges related to imbalanced data through techniques like SMOTE, our goal is to enhance classification accuracy and provide meaningful in- sights into the emerging research patterns. Overall, our contributions can be summarized as follows: •Analysis of trends over time in LLM survey papers. •Examination of the taxonomy distribution in LLM research. •Comparison of classification methods to pre- dict taxonomies using survey data. • Handle class imbalance using SMOTE. 2 Related Work The rapid advancements in Large Language Mod- els (LLMs) have been extensively documented in recent literature. Early foundational work includes the introduction of the transformer architecture by (Vaswani et al., 2017), which revolutionized the processing of sequential data. This architecture has become the backbone of many modern LLMs, enabling them to achieve remarkable performance in various tasks. Devlin et al. (2018) introduced BERT, which employs bidirectional training of transformers, sig- nificantly enhancing the state of natural language understanding (Devlin et al., 2018). BERT’s pre- training and fine-tuning methodology has been widely adopted, influencing subsequent LLM de- signs. The evolution continued with Radford et al. (2019), who introduced GPT-2, demonstrating the potential of large-scale unsupervised learning in generating coherent text (Radford et al., 2019). This was further expanded by Brown et al. (2020) with GPT-3, which showcased few-shot learning capabilities, significantly improving performance on diverse NLP tasks (Brown et al., 2020). Recent surveys, such as those by Achiam et al. (2023) (Achiam et al., 2023) and Bai et al. (2022) (Bai et al., 2022), provide comprehensive overviews of the current landscape of LLM re- search, discussing various techniques and their ap- plications. These surveys highlight the growing interest in LLMs among new researchers and em- phasize the challenges posed by the sheer volume of available literature. In addition to advancements in model architec- ture and training methodologies, there has been a focus on evaluating the performance of LLMs across different tasks. Studies like those by Kalyan et al. (2021) (Kalyan et al., 2021) investigate clas- sification tasks, examining factors that influence performance metrics such as accuracy, precision, recall, and F1-score. The analysis of biases within LLMs has also been explored, with Cohen et al. (2021) (Cohen et al., 2021) providing a compre- hensive review of fairness in NLP applications. 3 Methodology 3.1 Data Exploration We begin by exploring trends in survey papers over time. Specifically, we plot the number of papers categorized by each taxonomy per month to iden- tify patterns in research trends. 1import matplotlib . pyplot as plt 2import pandas as pd 3 4# Load dataset 5df = pd. read_csv (’ survey_data . csv ’) 6 7# Group by month and taxonomy 8monthly_trend = df. groupby ([ ’month ’, ’ taxonomy ’]). size (). unstack () 9 10# Plot trends 11monthly_trend . plot ( kind =’line ’) 12plt . title (’Taxonomy Trends Over Time ’) 13plt . xlabel (’Month ’) 14plt . ylabel (’Number of Papers ’) 15plt . show () Listing 1: Plotting Taxonomy Trends Over Time Let’s explain the figure here,the figure represent X and Y column as Release Date and the Number of papers released each year. Figure 1: Trends of Survey Papers Over Time Release Date Number of Papers Released (2021, 7) 2 (2022, 8) 2.5 (2022, 12) 3 (2023, 2) 3 (2023, 3) 9 (2023, 4) 7 (2023, 5) 6 (2023, 6) 5 (2023, 7) 9 (2023, 8) 11 (2023, 9) 12 (2023, 10) 12.5 (2023, 11) 17.5 (2023, 12) 13 (2024, 1) 12 Table 1: Trend of Survey Papers Released Over Time We also analyze the distribution of taxonomies in the dataset to understand which categories are more prevalent. 1# Distribution of taxonomies 2taxonomy_distribution = df[’taxonomy ’]. value_counts () 3 4# Plot distribution 5taxonomy_distribution . plot ( kind =’bar ’) 6plt . title (’Taxonomy Distribution ’) 7plt . xlabel (’Taxonomy ’) 8plt . ylabel (’Count ’) 9plt . show () Listing 2: Taxonomy Distribution The figure shows the trend of survey papers re- leased over time, with a fluctuating number of re- leases. Some months show significant increases, indicating periods of heightened research activity. Figure 2: Distribution of Proposed Taxonomy 3.2 Data Manipulation In this section, we preprocess and transform the sur- vey data into a suitable format to prepare the dataset for classification. Each paper is encoded based on itsTitle ,Summary , and Categories , and the final feature matrix is constructed by combining text and categorical data. 3.2.1 Building the Feature Matrix We use TF-IDF vectorization for the Title and Summary fields, converting the text data into a nu- merical matrix. This transformation preserves the importance of the words in the documents. In addi- tion, the Categories column is one-hot encoded to represent the categorical data. •TF-IDF Vectorization : We apply the TfidfVectorizer to the Title andSummary columns, converting the text into a numerical format that captures the importance of words in each document. •One-Hot Encoding : The Categories col- umn is transformed into a binary matrix using one-hot encoding, where each category is rep- resented as a separate feature. 1# Vectorize ’Title ’ and ’Summary ’ using TF - IDF 2vectorizer = TfidfVectorizer ( stop_words = ’english ’) 3title_tfidf = vectorizer . fit_transform ( data_df [’Title ’]) 4summary_tfidf = vectorizer . fit_transform ( data_df [’Summary ’]) 5 6# One - hot encode ’Categories ’ 7categories_encoded = pd. get_dummies ( data_df [’Categories ’]) 8 9# Combine the features into a final feature matrix10feature_matrix = pd. concat ([ title_tfidf . toarray () , summary_tfidf . toarray () , categories_encoded ], axis =1) Listing 3: Building the Feature Matrix Once the feature matrix is built, it is further pro- cessed to prepare the data for classification. We normalize the data, encode the labels, and split the dataset into training and testing sets. •Normalization : The features are scaled be- tween 0 and 1 using MinMaxScaler . •Label Encoding : The Taxonomy column is encoded into numerical labels for classifica- tion. •Train-Test Split : The dataset is split into training and testing sets with a 40% test ratio. 1# Normalize the feature matrix 2scaler = MinMaxScaler () 3normalized_features = scaler . fit_transform ( feature_matrix ) 4 5# Encode labels and split dataset 6label_encoder = LabelEncoder () 7encoded_labels = label_encoder . fit_transform ( data_df [’Taxonomy ’]) 8X_train , X_test , y_train , y_test = train_test_split ( normalized_features , encoded_labels , test_size =0.4) Listing 4: Preprocessing the Feature Matrix 3.3 Data Evaluation In this section, we employ several machine learn- ing and deep learning models to classify the dataset and compare their performance using accuracy as the primary evaluation metric. The models used in- clude Logistic Regression, Neural Networks, Ran- dom Forest, XGBoost, Naive Bayes, and K-Nearest Neighbors (KNN). Below is a summary of the mod- els and their results. 3.3.1 Logistic Regression We trained a Logistic Regression model using the training data. The model achieved reasonable ac- curacy on the test set. The code snippet below highlights the process: 1from sklearn . linear_model import LogisticRegression 2from sklearn . metrics import accuracy_score 3 4# Train and evaluate Logistic Regression 5model = LogisticRegression ( max_iter =500 , random_state =42) 6model . fit ( X_train , y_train ) 7 8# Make predictions and calculate accuracy 9y_pred = model . predict ( X_test ) 10accuracy = accuracy_score ( y_test , y_pred ) Listing 5: Logistic Regression Model 3.3.2 Neural Network We implemented a deep learning model using TensorFlow, which consists of dense layers and dropout for regularization. Early stopping was used to prevent overfitting. The model was evaluated on the test set to determine accuracy. 1import tensorflow as tf 2from tensorflow import keras 3from tensorflow . keras import layers 4 5# Define and compile the model 6model = keras . Sequential ([ 7 layers . Dense (128 , activation =’relu ’, input_shape =( X_train . shape [1] ,)), 8 layers . Dropout (0.5) , 9 layers . Dense (64 , activation =’relu ’), 10 layers . Dropout (0.3) , 11 layers . Dense ( len ( set ( y_train )), activation =’softmax ’) 12]) 13 14model . compile ( optimizer =’adam ’, loss =’ sparse_categorical_crossentropy ’, metrics =[ ’accuracy ’]) 15history = model . fit ( X_train , y_train , epochs =50 , validation_split =0.2 , batch_size =2, callbacks =[ keras . callbacks . EarlyStopping ( monitor =’ val_loss ’, patience =3) ]) 16loss , accuracy = model . evaluate ( X_test , y_test ) Listing 6: Neural Network Model 3.3.3 Random Forest The Random Forest classifier is an ensemble method that builds multiple decision trees and aver- ages their predictions. It is robust and often works well on complex datasets. 1from sklearn . ensemble import RandomForestClassifier 2 3clf = RandomForestClassifier ( n_estimators =100 , random_state =42) 4clf . fit ( X_train , y_train ) 5accuracy = clf . score ( X_test , y_test ) Listing 7: Random Forest Classifier 3.3.4 Naive Bayes The Naive Bayes classifier is a probabilistic model often used for text classification. It assumes condi- tional independence between features.1from sklearn . naive_bayes import MultinomialNB 2 3clf = MultinomialNB () 4clf . fit ( X_train , y_train ) 5accuracy = clf . score ( X_test , y_test ) Listing 8: Naive Bayes Classifier 3.4 Conclusion: Model Performance The performance of the models on the test set is summarized below: Model Accuracy (%) Logistic Regression 37.93 Random Forest 37.93 XGBoost 36.21 Naive Bayes 46.55 K-Nearest Neighbors (KNN) 25.86 Neural Network 42.00 Table 2: Accuracy of Different Classifiers Among the models tested, the Naive Bayes clas- sifier achieved the highest accuracy, with a score of 46.55%. The Neural Network also performed well, achieving an accuracy of 42%. Random Forest and XGBoost, while robust in many tasks, did not per- form as well in this scenario, achieving 37.93% and 36.21% accuracy, respectively. K-Nearest Neigh- bors (KNN) had the lowest accuracy, with 25.86% 4 Handling Imbalanced Data Using SMOTE 4.1 Importance of Addressing Imbalanced Data Imbalanced data occurs when the classes in a dataset are not represented equally, leading to a model bias towards the majority class. This can result in: •Model Bias : High overall accuracy while poorly predicting the minority class. •Poor Generalization : The model fails to identify critical outcomes, especially in ap- plications like fraud detection or medical di- agnoses. •Misleading Metrics : Accuracy becomes un- reliable; precision, recall, and F1-score are more relevant. 4.2 Overview of SMOTE The Synthetic Minority Oversampling Tech- nique (SMOTE) is a widely used method for ad- dressing class imbalance. SMOTE works by creat- ing synthetic samples for the minority class based on existing instances. This process involves: •Interpolation : New samples are generated by interpolating between existing minority class samples. •Balancing the Dataset : This increases the representation of the minority class, helping to mitigate bias in model training. •Generalization : SMOTE reduces the risk of overfitting by creating a more generalized de- cision boundary. The implementation of SMOTE in Python is shown below: from imblearn.over_sampling import SMOTE # Apply SMOTE smote = SMOTE(random_state=42) X_resampled, y_resampled = smote.fit_resample(X, y) 4.3 Model Performance Evaluation After applying SMOTE, various classification mod- els (Random Forest, XGBoost, Naive Bayes, K- Nearest Neighbors) are trained and evaluated. The code for training and evaluation is summarized as follows: from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score # Split the resampled data X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=0.4, random_state=42) # Define and evaluate models models = { 'Random Forest ': RandomForestClassifier (n_estimators=100, random_state=42),'XGBoost ': xgb.XGBClassifier( random_state=42), 'Naive Bayes ': MultinomialNB(), 'KNN': KNeighborsClassifier (n_neighbors=5) } for name, model in models.items(): model.fit(X_train, y_train) # Train on resampled data accuracy = accuracy_score (y_test, model.predict(X_test)) # Evaluate accuracy print(f"{name} Accuracy: {accuracy * 100:.2f}%") 4.4 Results •Before SMOTE : Models exhibited bias with accuracy around 40-55%, showing poor mi- nority class detection. •After SMOTE : Accuracy improved signifi- cantly, e.g.: –Random Forest: 80% to 85% –Naive Bayes: 82% to 91% Applying SMOTE leads to improved model gen- eralization and better performance on both majority and minority classes. This results in more reliable accuracy and robust predictions, which is crucial for applications where minority class identification is essential. 5 Conclusion In this project, we analyzed the trends in survey data related to LLMs and compared various classi- fication models. Naive Bayes achieved the highest accuracy, especially when handling class imbal- ance using SMOTE. Future work will involve deep learning approaches for better performance. A APPENDIX A Hyperparameters and Settings for Classifiers All hyperparameters and settings for the clas- sifiers were set to their default values in the scikit-learn library. The only modification made was to the random state, which was set to ensure reproducibility across experiments. A.1 Random State The random state was set to a fixed value across all classifiers to ensure reproducibility of the results. A.2 Classifier Settings All other parameters were kept at their default val- ues for the following classifiers: •LogisticRegression: LogisticRegression(max_iter=500, random_state=42) •Neural Networks: Implemented using Ten- sorFlow with layers, dropout, and early stop- ping (default parameters). •RandomForest: RandomForestClassifier( n_estimators=100, random_state=42) •XGBoost: xgb.XGBClassifier (random_state=42) •Naive Bayes: MultinomialNB() •K-NearestNeighbors: KNeighborsClassifier(n_neighbors=5) A.3 Reproducibility To ensure consistent and reproducible results, all classifiers used a fixed random state of 42. No fur- ther hyperparameter tuning was performed, and all other settings were left at their default configura- tions.