## 4 Experiment and Results
### Benchmark LLMs' Performance on Structured Query Generation
#### 4.1.1 Benchmark LLMs Selection  
Three LLMs (ERNIE 3.0, Llama-13B, SQLCoder) are utilized as the benchmark of the experiment. ERNIE represents a large model optimized for Chinese, with a focus on knowledge-enhanced pre-training strategies, making it suitable for tasks such as structured knowledge extraction and instruction understanding in the BIM Chinese context. Llama-13B is a general-purpose, open-source English model that has not been specifically optimized for SQL or structured tasks, and is thus suitable for comparing the adaptability of general models in specialized domain tasks (baseline). SQLCoder represents the state-of-the-art fine-tuned model for structured query generation, having undergone reinforcement training for SQL-related tasks, and is specifically designed for the NL2 SQL task. These models differ significantly in terms of architecture, pre-training objectives, and domain adaptability, making them typical representatives for a comprehensive comparative analysis. The ERNIE 3.0, Llama-13B, and SQLCoder models have all fine-tuned on the collected dataset (Section 3.1), enabling them to convert user queries into SQL queries in JSON format for BIM. The unique architecture and knowledge-enhancement mechanism of ERNIE 3.0 require a larger batch size to stabilize the update of knowledge-related parameters, as well as a more significant number of epochs to ensure effective adaptation of the knowledge structure to domain-specific tasks. These factors highlight the need for substantial computational resources and meticulous tuning during the training process to achieve optimal performance in domain adaptation. The fine-tuning parameters for the three models are in Table 5.  
Table 5 presents the key hyperparameter settings used to fine-tune the three benchmark models--ERNIE 3.0, LLaMA-13B, and SQLCoder--within the scope of this study. The final hyperparameter configurations were determined through a series of exploratory experiments, during which multiple combinations of training epochs and learning rates were tested to balance model convergence, training efficiency, and computational feasibility. 1. Epoch (Number of Training Epochs) \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameters** & **ERNIE3.0** & **Llama-13B** & **SQLCoder** \\ \hline Epoch & 15 & 5 & 5 \\ Learning Rate & 0.000003 & 0.000001 & 0.000001 \\ Batch Size & 16 & 1 & 1 \\ Sequence Length & 4096 & 4096 & 4096 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Fine-tuning Parameters for the Benchmark Models. For ERNIE 3.0, training epoch values of 5, 10, and 15 were initially evaluated. The model demonstrated underfitting at lower epoch settings, while 15 epochs consistently yielded better convergence in domain adaptation scenarios, owing to its knowledge-enhanced architecture and deeper parameter structure. Thus, 15 epochs were adopted as the final setting to ensure sufficient model learning without overfitting. In contrast, for LLaMA-13B and SQLCoder, experiments with 3, 5, and 10 epochs showed that performance improvements plateaued after 5 epochs. Considering their more concise architectures and pre-training on code-related or structured language tasks, 5 epochs provided a good trade-off between accuracy and training time. 2. Learning Rate Regarding the learning rate, a grid search was conducted over values ranging from \(1\times 10^{5}\) to \(3\times 10^{6}\) for ERNIE 3.0, and from \(5\times 10^{6}\) to \(1\times 10^{6}\) for LLaMA-13B and SQLCoder. The optimal learning rate for ERNIE 3.0 was determined to be \(3\times 10^{6}\) (0.000003), which ensured stable fine-tuning over its large-scale parameters while preventing gradient explosion. For LLaMA-13B and SQLCoder, a more conservative learning rate of \(1\times 10^{6}\) (0.00001) was found to yield better generalization and reduce the risk of overfitting on the relatively limited domain-specific training data. 3. Batch Size The batch size selection was primarily constrained by hardware limitations.