## 5 Experiments
### Experimental Results and Discussions
Figure 4 compares two datasets, showing that the nine models developed by Meta, OpenAI, and Google exhibited a slight advantage when using English prompts over Chinese prompts. Conversely, the five LLMs developed by Chinese companies, such as Qwen-turbo, ERNIE-4.0-8k, and ChatGLM, demonstrated better performance with Chinese prompts. This difference may be attributed to variations in the corpora used by different companies in training their base models. Additionally, we observed that the number of parameters in the tested models exhibited a linear relationship with accuracy. As the number of model parameters increased, accuracy improved. For instance, GPT-4o, with its higher parameter count, consistently outperformed other models in both scenarios. The results underscore the significant influence of language on model performance. Models like GPT-4o and GPT-3.5-turbo demonstrated high adaptability, maintaining robust performance across both English and Chinese prompts, which is essential for applications requiring multilingual support. However, certain models exhibited a marked preference for prompts in their native language. For example, Qwen-turbo achieved accuracy of 59.92% with Chinese prompts but dropped to 47.17% when using English prompts. This suggests that these models may have been predominantly trained on Chinese corpora, optimizing their performance for Chinese prompts. Time consumption data reveal that models like GPT-4o not only provided high accuracy but also demonstrated efficient processing times, particularly with English prompts. The significant differences in the number of output tokens generated by the models suggest variations in their adherence to prompt instructions. While some models, such as Qianfan-Chinese-Llama-2-70B, generated excessive tokens when using English prompts, indicating the inclusion of unnecessary explanations, others like GPT-4o adhered strictly to the prompt requirements, thereby enhancing their overall efficiency. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Ques.**} & \multirow{2}{*}{**\# Corr.**} & \multirow{2}{*}{**Acc.**} & \multirow{2}{*}{**Time (s)**} & \multicolumn{2}{c}{**\# Total Tokens**} \\ \cline{5-6}  & & & & & **\# Input Tokens** & **\# Output Tokens** \\ \hline Qwen-turbo & 706 & 333 & 47.17\% & 631.79 & 232,279 & 734 \\ \hline ERNIE-4.0-8k & 706 & 398 & 56.37\% & 3129.01 & 218,968 & 8784 \\ \hline GPT-3.5-turbo & 706 & 315 & 44.62\% & 442.89 & 260,724 & 713 \\ \hline CPT-4 & 706 & 380 & 53.82\% & 498.75 & 260,724 & 799 \\ \hline GPT-4o & 706 & 435 & 61.61\% & 329.97 & 237,607 & 706 \\ \hline GLM-3-turbo & 706 & 336 & 47.59\% & 1519.96 & 232,763 & 2375 \\ \hline GLM-4-Air & 706 & 356 & 50.42\% & 1387.82 & 223,913 & 2116 \\ \hline GLM-4 & 706 & 367 & 51.98\% & 1391.58 & 223,913 & 2365 \\ \hline Qianfan-Chinese-Llama-2-7B & 706 & 312 & 44.19\% & 1980.27 & 224,460 & 4461 \\ \hline Qianfan-Chinese-Llama-2-13B & 706 & 337 & 47.73\% & 4285.56 & 224,460 & 113,071 \\ \hline Qianfan-Chinese-Llama-2-70B & 706 & 396 & 56.09\% & 5289.31 & 224,460 & 93,379 \\ \hline Meta-Llama-3-8B & 706 & 289 & 40.93\% & 1843.25 & 224,460 & 706 \\ \hline Meta-Llama-3-70B & 706 & 359 & 50.85\% & 8680.55 & 224,460 & 116,637 \\ \hline Gemma-7B-it & 706 & 294 & 41.64\% & 2,803.91 & 224,460 & 31,128 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Using English prompts to query multiple LLMs with Chinese ship-handling theory MCQs. In Table 5, we present the results of testing 814 English MCQs using English prompts. These data allowed us to evaluate the performance of LLMs across different languages and question types. All models showed significant improvements in accuracy when queried with English prompts. Among them, GPT-4o achieved an outstanding accuracy rate of over 85%, making it the only LLM likely to pass the test for ship handling, watchkeeping, and navigation theory.