# An Investigation of Applying Large Language Models to Spoken Language Learning
## 2 Methodology  
### Dataset  
Even though benchmarks for general abilities and some professional fields are widely used, there is still a scarcity of evaluation datasets on SLI for language learning. We introduce a new dataset called **SLIQ-LL (Spoken Language Intelligence Questions for Language Learning)** (available on Github: [[https://github.com/vocaliodmiku/SLI-LL](https://github.com/vocaliodmiku/SLI-LL)]([https://github.com/vocaliodmiku/SLI-LL](https://github.com/vocaliodmiku/SLI-LL)), accessed on 11 August 2023). This dataset covers the topics of phonetics, phonology, and second language acquisition, which are frequently addressed in language learning education. The dataset consists of two subsets:  
* **Knowledge and Concept**: We designed a set of concept-related questions to test the large models' knowledge of spoken languages, such as _\"what is language transfer\"_ and _\"how many classifications of consonants are there for the manner of articulation?\"_ These questions were mainly sampled from the exercises at the end of each chapter of the book [42] and then manually adjusted to the needs of the research.
* **Application Questions**: To address the ever-changing nature of personalized problems, it is necessary to utilize knowledge of phonetics or linguistics to make complex reasoning. For example, different contexts call for different appropriate stress patterns, which is fundamental to an automatic personalized language learning system. Therefore, based on language teaching practices involving pronunciation, pauses, stress, and intonation, we manually designed a series of representative questions. For example, regarding stress, we formulated questions about the placement of word stress within words, phrase stress when conveying specific meanings, and sentence stress. For the issue of word stress within words, we considered having GPT determine the positions of stressed syllables in both regular and irregular words (e.g., the word \"present\" has different stress locations depending on the part of speech). For the issue of intonation, we designed questions that would allow the LLMs to determine the intended meanings of sentences with different positions of rising/falling tone or vice versa, i.e., allowing LLMs to determine the proper intonation for sentences with specific meanings. Overall, we aimed to increase the breadth of questions and enhance their practical relevance as much as possible. An example about pronunciation is shown in Figure 1.  
We collected and designed a total of 445 questions, including 144 knowledge and concept-related questions and 301 application questions. Figure 2 (left) shows the type distribution of the application questions. In the context of language learning, during our initial preparations, we found that intermediate-level learners need to focus more on fundamental pronunciation issues, as opposed to suprasegmental features like stress, break, and intonation. To present such data, we formulated multiple-choice questions, each with only one correct answer. We uniformly assigned their answers, and Figure 2 (right) shows the statistics for these questions.