## 3 Material and Methods
### Methodology  
This research encompasses several key stages to enhance the classification efficiency and robustness of the models. * Data Processing: We opted for minimal data preprocessing steps, such as basic text normalization and removing non-textual characters, instead of extensive preprocessing like tokenization, stop-word removal, or stemming, as these extensive steps may lead to the loss of important contextual information. Advanced word embedding techniques like BERT and RoBERTa are designed to handle raw text inputs effectively [5; 6]. This approach preserves contextual information and reduces computational overhead, aligning with recent findings emphasizing the efficiency of using raw data with sophisticated models [45]. Modern learning models, particularly those involving transformer architectures, are highly robust and capable of learning from raw data [46]. * Data Splitting and Augmentation: To address the severe class imbalance in the dataset, data splitting was conducted using specific numbers for each class to ensure a balanced representation. The Missouri State Highway Patrol dataset was divided into training and testing sets based on a fixed number of instances per class for testing. The remaining samples were used for training. Data augmentation techniques were applied exclusively to the training set to upsample the minority classes by generating synthetic data. This step increased the representation of the minority classes, \"Fatal\" and \"Property Damage\" classes, thereby balancing the class distributions. * Cross-Validation: To ensure the robustness of our results and mitigate the risk of overfitting, 5-fold cross-validation was employed during the training process. This approach validates the model's performance across different subsets of the data, providing a comprehensive evaluation. * Feature Extraction Techniques and Classifiers: We investigate multiple feature extraction techniques and machine learning classifiers. The classifiers used in this study include random forest (RF) [47], naive Bayes (NB) [48], and XGBoost (XGB) [49], each paired with various word embedding techniques such as TF-IDF [3], Word2 Vec [4], and BERT [5]. Additionally, two state-of-the-art pre-trained language models, BERT [5] and RoBERTa [6], were fine-tuned with a Bidirectional Long Short-Term Memory (Bi-LSTM) classification model [50]. These selections are based on their demonstrated efficiency in various NLP tasks [30; 31; 32; 35; 44; 51]. * Model Training: In the model training phase, we utilized a combination of traditional machine learning models and advanced deep learning techniques. Specifically, we paired three machine learning models--XGBoost (XGB) [49], naive Bayes (NB) [48], and random forest (RF) [47]--with three-word embedding techniques: TF-IDF [3], Word2 Vec [4], and BERT [5]. Additionally, we fine-tuned two state-of-the-art pre-trained language models, BERT [5] and RoBERTa [6], with a Bidirectional Long Short-Term Memory (Bi-LSTM) classification model [50]. These models were trained separately on both the original and augmented datasets to comprehensively evaluate performance differences and the impact of data augmentation on model robustness and accuracy. * Ensemble Hard Voting: In this study, we utilized a hard voting ensemble method to combine the predictions of three top-performing models: TF-IDF/XGB, BERT-LSTM, and RoBERTa-LSTM. This technique aggregates the predictions from each model \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Personal Injury** & **Fatal** & **Property Damage** & **Total** \\ \hline Training set (original dataset) & 5295 & 428 & 21 & 5744 \\ Training set (augmented dataset) & 5295 & 1000 & 1000 & 7295 \\ Testing set & 588 & 48 & 20 & 656 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Distribution of training and testing datasets before and after augmentation. and selects the majority vote as the final prediction, enhancing overall classification accuracy and robustness.