# An Investigation of Applying Large Language Models to Spoken Language Learning
## 3 Experiments and Results
### Analysis of Advanced Prompting Methods  
In Section 3.2, we utilized an empirical value of \(k=3\) as the number of shots for large-scale experiments. In this section, we explore the performance of different prompting methods for several representative LLMs with different numbers of shots. **Few-shot and CoT prompting:** Figure 6 presents the results of direct few-shot and CoT few-shot prompting using different numbers of shots. Based on the results, we can see that increasing the number of examples improved performance to a limited extent, and increasing the examples of the reasoning chain had a more significant and stable effect on models above 70 B (LLaMA2-chat, GPT-4). However, for smaller models, these prompts exceeded their capabilities, resulting in a degradation in performance. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Model** & **A** & **B** & **C** & **D** & **Acc.** & \(\mathbf{p}\)**-Value** \\ \hline GPT-4 & 77 & 84 & 69 & 71 & 74.6\% & \(5\times 10^{-1}\) \\ GPT-3.5-turbo & 76 & 91 & 70 & 64 & 53.1\% & \(2\times 10^{-1}\) \\ LLaMA2-70B-chat & 64 & 87 & 100 \(\mathbf{\mathrm{\SIUnitSymbolMicro}}\) & 50 \(\mathbf{\mathrm{\SIUnitSymbolMicro}}\) & 44.3\% & \(2\times 10^{-3}\) \\ Flan-UL2-20B & 93 \(\mathbf{\mathrm{\SIUnitSymbolMicro}}\) & 56 \(\mathbf{\mathrm{\SIUnitSymbolMicro}}\) & 70 & 82 & 41.6\% & \(4\times 10^{-3}\) \\ Flan-T5-XXL-11B & 93 \(\mathbf{\mathrm{\SIUnitSymbolMicro}}\) & 67 & 80 & 61 & 44.8\% & \(6\times 10^{-2}\) \\ \hline
**No. of Labels** & **74** & **76** & **80** & **71** & & \\ \hline \hline \multicolumn{7}{l}{ Averaged using \(\{0,3,5,8,10,12,15,20,25,30\}\) shots and 1\(\sim\)9-shot CoT results. Part of the experimental data (i.e., CoT) come from Section 3.3.} \\ \end{tabular}
\end{table} 
Table 4: Frequency of predictions and labels. The underestimated predictions are marked with \(\mathbf{\mathrm{\SIUnitSymbolMicro}}\), whereas the overestimated predictions are marked with \(\mathbf{\mathrm{\SIUnitSymbolMicro}}\) (\(\pm 20\%\) of the label frequency). The last column indicates the \(p\)-value of the \(\chi^{2}\) test for the null hypothesis _“The answer distribution produced by LLM equals that of the ground-truth”_. Figure 6: Number of correctly answered questions (total 301) on SLIQ-LL (Application Questions subset) using \(k=1\ldots 9\) shots. The developed examples used in the previous experiments were manually written, whereas the current ones were sampled from the dataset. The thought chain was generated by GPT-4 with correctly generated answers (the thought chain is not guaranteed to be correct). **In-domain prompts v.s. out-of-domain prompts:** We used prompts from different domains for two models capable of responding to CoT prompts. For the in-domain cases, we used domain-specific prompts for each type of question. In Figure 7, it is evident that the in-domain approach has significant advantages compared to the out-of-domain one, where the examples were selected from the more common MMLU dataset [17]. **Self-consistency:** Although the answers to the SLI questions did not have as many reasoning paths as mathematical reasoning questions, we found that self-consistency improved the performance of the GPT-3.5 model (as shown in Table 5). However, for LLaMA2-70B-chat, its occasional cleverness was offset by multiple generated errors. **Augmented language models:** On the Internet, individuals actively share their language learning experiences. By effectively using this external knowledge, LLMs can improve credibility and mitigate hallucination issues. In this study, we examined the LLMs' ability to use external knowledge to improve SLI by allowing auxiliary tools to be used by GPT-3.5. Experiments on the zero-shot setting showed that GPT-3.5 called Google and Wikipedia searches 351 and 131 times, respectively, when answering the application questions. However, its performance was not improved compared to the case without using external tools, with both of them achieving an accuracy of 49.1%.