## 5 Experiments
### Experimental Results and Discussions  
Table 3 presents the experimental results using Chinese prompts to query multiple LLMs with Chinese Ship Handling and Navigation Theory MCQs. A total of 706 MCQs were employed, with GPT-4o achieving the best performance, attaining an accuracy of 60.76% and the lowest time consumption. It is important to note that the time reported here does not refer to the absolute inference time of the LLM, as network latency from API access can influence the time statistics. In our prompt, we instructed the tested models to output only the letter corresponding to the correct option, without any additional explanations or symbols, akin to actual human theoretical tests. However, some models still generated explanations beyond the letter, resulting in a significant increase in output tokens. This indicates that certain LLMs need to improve their understanding of prompts. Furthermore, the variability in accuracy and time consumption among the models highlights differences in their architectures and training methodologies. For instance, models like GPT-3.5-turbo and GPT-4o not only provided high accuracy but also demonstrated efficient processing times, suggesting their robustness in understanding and responding to Chinese prompts. On the other hand, while models such as Meta-Llama-3-70B and Qianfan-Chinese-Llama-2-70B exhibited competitive accuracy, their higher time consumption could be attributed to more complex processing requirements or network-related delays. This suggests a trade-off between accuracy and computational efficiency that should be considered based on specific application needs. Additionally, the significant differences in the number of output tokens across models suggests variations in their adherence to prompt instructions. For example, Qianfan-Chinese-Llama-2-70B and Meta-Llama-3-70B generated a large number of output tokens, indicating a propensity to provide additional explanations beyond the required answer letter. This behavior could be detrimental in scenarios where concise responses are crucial. Moreover, the models developed by Chinese companies, such as Qwen-turbo and ERNIE-4.0-8k, demonstrated promising results. Their performance was comparable to GPT-4o, which achieved the best results. In Table 4, we present the results of using English prompts to test the same set of Chinese MCQs, providing an evaluation of the impact of language on the accuracy of large language models. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Ques.**} & \multirow{2}{*}{**\# Corr.**} & \multirow{2}{*}{**Acc.**} & \multirow{2}{*}{**Time (s)**} & \multicolumn{2}{c}{**\# Total Tokens**} \\ \cline{5-6}  & & & & & **\# Input Tokens** & **\# Output Tokens** \\ \hline Qwen-turbo & 706 & 423 & 59.92\% & 636.6 & 247,105 & 739 \\ \hline ERNIE-4.0-8k & 706 & 412 & 58.36\% & 2921.65 & 214,026 & 6925 \\ \hline GPT-3.5-turbo & 706 & 316 & 44.76\% & 415.86 & 415,338 & 711 \\ \hline CPT-4 & 706 & 389 & 55.10\% & 529.42 & 415,338 & 733 \\ \hline GPT-4o & 706 & 429 & 60.76\% & 339.01 & 292,675 & 706 \\ \hline GLM-3-turbo & 706 & 340 & 48.16\% & 940.27 & 239,117 & 2134 \\ \hline GLM-4-Air & 706 & 352 & 49.86\% & 995.76 & 230,267 & 2122 \\ \hline GLM-4 & 706 & 371 & 52.55\% & 1110.49 & 230,267 & 2133 \\ \hline Qianfan-Chinese-Llama-2-7B & 706 & 273 & 38.67\% & 2520.14 & 230,108 & 2,010 \\ \hline Qianfan-Chinese-Llama-2-13B & 706 & 317 & 44.90\% & 6994.99 & 230,108 & 111,678 \\ \hline Qianfan-Chinese-Llama-2-70B & 706 & 398 & 56.37\% & 5510.65 & 230,108 & 93,757 \\ \hline Meta-Llama-3-8B & 706 & 283 & 40.08\% & 2235.81 & 230,108 & 709 \\ \hline Meta-Llama-3-70B & 706 & 313 & 44.33\% & 9015.19 & 230,108 & 116,630 \\ \hline Gemma-7B-it & 706 & 282 & 39.94\% & 2779.97 & 230,108 & 30,907 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Using Chinese prompts to query multiple LLMs with Chinese ship-handling theory MCQs.