## 4 Experiments
### Ensemble Model with Hard Voting
#### 4.3.2 Hard Voting Model  
To enhance the crash severity classification, we employed three different models: BERT (DistilBERT), RoBERTa, and XGBoost (with TF-IDF). Each model was fine-tuned and configured with specific parameters to maximize performance. This approach aimed to improve performance by leveraging the strengths of each model on both the original and augmented datasets. Figure 11 shows an overview of the ensemble model with the top-performing classifiers. Table 2 provides an overview of the main settings and hyperparameters used for each model.  
Figure 10: XGB-TFIDF performance on augmented dataset (5-fold cross-validation).  
The BERT model (DistilBERT) was fine-tuned with a Bidirectional Long Short-Term Memory (Bi-LSTM) layer, using the DistilBERT tokenizer with a maximum token length of 200. The model was optimized with the Adam optimizer and trained using a sparse categorical crossentropy loss function, with a batch size of 16 for 5 epochs. Uniform class weights were applied, and training dynamics were managed using ModelCheckpoint, EarlyStopping, and ReduceLROnPlateau callbacks. Similarly, the RoBERTa model was fine-tuned with a Bi-LSTM layer, employing the RobertaTokenizer with a maximum token length of 200. It shared the same optimizer and loss function as BERT but was trained for 10 epochs to ensure more extensive learning. Class weights were significantly adjusted to address class imbalance, particularly for the minority class, and the same callbacks as BERT were used. The XGBoost model, combined with TF-IDF vectorization for feature extraction, utilized a TF-IDF vectorizer configured to consider a maximum of 5000 features. The XGBoost classifier operated with default parameters and the m'logloss' evaluation metric, focusing on leveraging the TF-IDF vectorized features for classification without the need for the same type of preprocessing or callbacks required by the transformer models.  
The ensemble method involved aggregating the predictions from each model into a 2D array and applying the mode function from scipy.stats to determine the most frequent  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model Name** & **Parameters** & **BERT (DistilBERT)** & **RoBERTa** & **XGB (TF-IDF)** \\ \hline \multirow{2}{*}{Model Architecture} & DistilBERT with & \multirow{2}{*}{RoBERTa with Bi-LSTM} & \multirow{2}{*}{XGBoost with TF-IDF} \\  & Bi-LSTM & & \\ \multirow{2}{*}{Tokenizer} & DistilBERTTokenizer & RobertaTokenizer & TfidfVectorizer \\ \multirow{2}{*}{Max Length} & 200 & 200 & N/A \\  & Adam & Adam & Default (XGBoost) \\ \multirow{2}{*}{Loss Function} & Sparse Categorical & Sparse Categorical & \multirow{2}{*}{Mlogloss} \\  & Crossentropy & Crossentropy & \\ \multirow{2}{*}{Batch Size} & 16 & 16 & N/A \\ \multirow{2}{*}{Epochs} & 5 & 10 & N/A \\  & \{0: 1.0, 1: 1.0, 2: 1.0\} & \{0: 1.0, 1: 1.0, 2: 10.0\} & N/A \\ \multirow{2}{*}{Callbacks} & ModelCheckpoint, & \multirow{2}{*}{ModelCheckpoint, \multirow{2}{*}{EarlyStopping, \multirow{2}{*}{ReduceLROnPlateau}} & \multirow{2}{*}{N/A} \\  & EarlyStopping, & & \\ \multirow{2}{*}{Preprocessing} & ReduceLROnPlateau & & \\  & Tokenization, Attention & \multirow{2}{*}{T-IDF Vectorization} & \multirow{2}{*}{(max\_features = 5000)} \\ \multirow{2}{*}{Preprocessing} & \multirow{2}{*}{Masks} & \multirow{2}{*}{Masks} & \multirow{2}{*}{T-IDF Vectorization} \\  & & & \\ \end{tabular}
\end{table}
Table 2: Main settings and hyperparameters for BERT, RoBERTa, and XGBoost models.  
Figure 11: Hard voting model.  
prediction for each instance. By adopting the majority voting principle, where the class label predicted by at least two out of the three classifiers was selected as the final classification, the hard voting ensemble effectively combined the individual strengths of TF-IDF/XGB, BERT-LSTM, and RoBERTa-LSTM. This method improved the overall classification performance, particularly for the minority \"Property Damage\" class, by providing a more robust and accurate classification than any single model.