## 4 Performance Evaluation Through a Case Study
### Training and Testing Results  
Model training results are shown in Table 2 The loss had good performance for the LLM, LSTM, and MLP-based ENA models in the order of M4.2 (0.0534), M4.1 (0.0587), M2.2 (0.0851), and M1.2 (0.0852). The difference in loss improvement between M4.2 and M2.2 models was 0.0317, which shows a relatively large difference considering that the  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**ENA Model** & **Batch** & **Epochs** & **Layer Architecture** & **Normalization and Activation** & **Learning Rate** & \multicolumn{2}{c}{**Pre-Trained Model**} \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{**ID**} & **Size** & & & **and** & **Activation** & **Rate** & **Embedding** & **Transformers** \\ \hline M1.1. MLP & 32 & 150 & [14]-[128; 64; 32]-[10] & & 0.001 & - & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M1.2. MLP} & 32 & 150 & [14]-[64; 128; 64]-[10] & ReLU, batch normal, dropout & 0.001 & - & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M1.3. MLP} & 32 & 150 & [14]-[64; 128; 64; 32]-[10] & & 0.001 & - & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M1.4. MLP} & 32 & 150 & [14]-[32; 64; 32]-[10] & & 0.001 & - & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M2.1. LSTM} & 32 & 150 & [14]-LSTM[128]-[10] & & 0.001 & - & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M2.2. LSTM} & 32 & 150 & [14]-LSTM[128]-[64; 32]-[10] & Dropout & 0.001 & - & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M2.3. LSTM} & 32 & 150 & [14]-LSTM[256]-[128; 64]-[10] & & 0.001 & - & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M3.1. Transformers} & 32 & 300 & [320; 512]-ENC[512]-[64]-[10] & & 1.00 \(\times\) 10\({}^{-5}\) & BERT-base-uncased & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M3.2. Transformers} & 64 & 300 & [320; 512]-ENC[512]-[64]-[10] & Multi-head attention, layer normal, dropout & 1.00 \(\times\) 10\({}^{-5}\) & BERT-base-uncased & - \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M4.1. LLM} & 32 & 150 & [293]-EMB-ENC-[10] & & 1.00 \(\times\) 10\({}^{-5}\) & BERT-base-uncased & BERT \\ \cline{1-2} \cline{6-7} \multicolumn{1}{c}{M4.2 LLM} & 32 & 300 & [293]-EMB-ENC-[10] & & 1.00 \(\times\) 10\({}^{-5}\) & BERT-base-uncased & BERT \\ \hline \hline \end{tabular}
\end{table}
Table 1: ENA model hyperparameters.  
average loss improvement value between models except M1.4 and M3.1 \(\sim\) 3, which had very poor performance, was 0.0051. The LLM-based model was fast at converging the solution. For reference, at epoch = 10, the loss of the MLP-based M1.1 model was 0.1685, but the loss was 0.074 for M4.1. The LLM-based ENA model showed a quick and stable exploration of optimal solutions. The prediction accuracy for the test dataset was in the order of M1.3 (0.9544), M4.2 (0.9531), M1.2 (0.9519), and M4.1 (0.9507). For reference, the Transformers-based ENA model tended to converge very late for training. The pre-trained LLMs had good training performances. However, the LLM-based model used many GPU resources (M4 model size = 427,783 KB), and the calculation speed was slow (M4 model training time = average 5 h 35 min).  
The loss and accuracy of the MLP-based model showed high performance under the GPU resources used, and the pre-trained LLM-based models showed fast solution exploration.  
In Figure 8, the earthwork classification result of M3.2 is incorrectly predicted because it contains the bottom of the pavement layer. In contrast, M4.1 is under the ground line and has a sequence containing a part of the ground and the structure line, so it can be seen that the results are well predicted. Figure 9 shows some of the predictions for input sequences for each model. Here, the model output \(\hat{y}\) of each sequence is displayed on the left side of the sequence bar. The right side shows the legend of the earthworks and earth lines marked by color.