## 4 Performance Evaluation Through a Case Study
### Quantity Takeoff Classification (QTC) Performance Results  
The ENA model's QTC performance comparison considered 3 + 600 to 4 + 600 stations, a crossing section containing various cross-sections in the dataset. A performance evaluation was performed on the earthwork quantity items included here (811 items in total) (Table 3). The QTCacc performance of the models was high in the following order (QTCacc Rank): LLM (M4.1 = 97.17%, M4.2 = 96.74%), LSTM (M2.2 = 82.10%), and MLP (M1.3 = 81.61%).  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**ENA Model ID** & **Train No** & **Loss** & **Accuracy** & \
\begin{tabular}{c} **Model Size** \\ **(kb)** \\ \end{tabular} &
\begin{tabular}{c} **Time Performance** \\ **(min)** \\ \end{tabular} \\ \hline M1.1. MLP & 1650 & 0.0870 & 0.9494 & 61 & 0:02:34 \\ \hline M1.2. MLP & 1714 & 0.0852 & 0.9519 & 84 & 0:02:36 \\ \hline M1.3. MLP & 1716 & 0.0882 & 0.9544 & 93 & 0:02:42 \\ \hline M1.4. MLP & 1718 & 0.1322 & 0.9507 & 30 & 0:02:25 \\ \hline M2.1. LSTM & 1730 & 0.0889 & 0.9408 & 812 & 0:02:13 \\ \hline M2.2. LSTM & 1732 & 0.0851 & 0.9420 & 850 & 0:02:17 \\ \hline M2.3. LSTM & 1734 & 0.0886 & 0.9408 & 3312 & 0:02:00 \\ \hline M3.1. Transformers & 2003 & 0.3533 & 0.7744 & 74,557 & 0:11:06 \\ \hline M3.2. Transformers & 2014 & 0.3551 & 0.7719 & 74,557 & 0:07:48 \\ \hline M3.3. Transformers & 2021 & 0.3596 & 0.7423 & 74,557 & 0:06:25 \\ \hline M4.1. LLM & 0103 & 0.0587 & 0.9507 & 427,783 & 3:12:05 \\ \hline M4.2. LLM & 2334 & 0.0534 & 0.9531 & 427,783 & 7:59:00 \\ \hline \hline \end{tabular}
\end{table}
Table 2: ENA models’ performances.  
Figure 8: ENA models loss graph (x = epoch, y = loss).  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**ENA Model ID**} & \multicolumn{2}{c}{**Ground True**} & \multicolumn{2}{c}{**Ground False**} & \multicolumn{2}{c}{**Accuracy**} & \multirow{2}{*}{**QTCacc Rank**} \\ \cline{2-2} \cline{5-8}  & **A1 (m\({}^{2}\))** & **C1** & **A2 (m\({}^{2}\))** & **C2** & **P1** & **P2** & **QTCacc (\%)** & \\ \hline M1.1. MLP & 9723.32 & 758 & 4300.33 & 53 & 69.34\% & 93.46\% & 81.40\% & 8 \\ \hline M1.2. MLP & 9716.86 & 759 & 4306.78 & 52 & 69.29\% & 93.59\% & 81.44\% & 7 \\ \hline M1.3. MLP & 9729.45 & 761 & 4294.20 & 50 & 69.38\% & 93.83\% & 81.61\% & 5 \\ \hline M1.4. MLP & 9726.16 & 760 & 4297.48 & 51 & 69.36\% & 93.71\% & 81.54\% & 6 \\ \hline M2.1. LSTM & 9762.06 & 762 & 4261.59 & 49 & 69.61\% & 93.96\% & 81.79\% & 4 \\ \hline M2.2. LSTM & 9868.09 & 761 & 4155.55 & 50 & 70.37\% & 93.83\% & 82.10\% & 3 \\ \hline M2.3. LSTM & 9512.40 & 756 & 4511.24 & 55 & 67.83\% & 93.22\% & 80.53\% & 9 \\ \hline M3.1. Transformers & 847.35 & 133 & 13,176.29 & 678 & 6.04\% & 16.40\% & 11.22\% & 12 \\ \hline M3.2. Transformers & 3347.50 & 40 & 10,676.14 & 771 & 23.87\% & 4.93\% & 14.40\% & 11 \\ \hline M3.3. Transformers & 7317.13 & 73 & 6706.51 & 738 & 52.18\% & 9.00\% & 30.59\% & 10 \\ \hline M4.1. LLM & 13,643.05 & 787 & 380.59 & 24 & 97.29\% & 97.04\% & 97.17\% & 1 \\ \hline M4.2. LLM & 13,592.18 & 783 & 431.46 & 28 & 96.92\% & 96.55\% & 96.74\% & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 3: ENA QTC performances (Area ratio = T1/(T1 + F1). QTC ratio = T2/(T2 + F2). ENA ratio = (A1 + A2)/2).  
Figure 9: ENA model’s prediction test results (data sequence bar. \(\mathbf{\check{\prime}}\): y = \(\mathbf{\check{y}}\)).  
To visually compare accuracy, the model prediction results for cross-sectional views showing large performance gaps between ENAs are shown in Figure 10. Looking at these results, it can be seen that there was a large difference in prediction accuracy between the M4.1 (LLM), M2.2 (LSTM), and M1.3 (MLP) models. An LLM-based model can better predict the area of earthworks. This shows that the LLM-based ENA, which is mainly used in NLP, can learn the earthwork quantity takeoff feature of the unstructured cross-sectional drawing dataset better than other models.