# An Investigation of Applying Large Language Models to Spoken Language Learning
## 1 Introduction
### Related Works  
**Large Language Models (LLMs):** Language models have revolutionized NLP in recent years. Researchers have found that enlarging a Pre-trained Language Model (PLM) (e.g., model parameters) often leads to better performance [10; 11]. A number of studies have explored pushing the limits of performance by training ever larger PLMs (e.g., the 175B-parameter GPT-3 [12] and the 540B parameter PaLM [13]). A remarkable success story in LLMs is ChatGPT ([[https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/)]([https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/)), accessed on 11 August 2023), developed by OpenAI. It adapts LLMs from the GPT series for dialogue, presenting amazing conversation abilities with humans. This triggered community-wide enthusiasm, driving the construction of various imaginative and fantastic applications. Microsoft AI scientists have recently explored the capabilities of OpenAI's GPT-4 [14], a more powerful large language model, claiming that GPT-4 demonstrates \"sparks\" of human-level intelligence or artificial general intelligence (AGI) [15]. There have been a number of attempts to assess ChatGPT and other LLMs from different angles, including factors such as natural language tasks, reasoning, robustness, trustworthiness, ethical considerations, and specific applications (e.g., natural science, social science, engineering, medical application, education, etc.) [16; 17; 18; 19; 20; 21; 22]. The broad applicability of LLMs underscores the evaluation of emerging intelligence within expert domains. **Spoken Language Intelligence (SLI):** Our world is inherently multimodal, and we engage with our environment through a variety of mediums, including text, images, sounds, and sensory experiences. There exist numerous multimodal methods that exhibit exceptional problem-solving capabilities across various scenarios, with text-based semantics frequently serving as the key agent, either explicitly or implicitly [23; 24]. For example, we aspire to empower machines to comprehend an image or video and articulate its meaning in words [25]. Conversely, we create specialized text prompts to generate images with the creativity of professional painters [26]. In the realm of speech, automatic speech recognition (ASR) technology is capable of extracting corresponding text from speech signals, even in complex speaking scenarios. Conversely, Text-to-Speech (TTS) systems can realistically produce speech sounds that correspond to a given piece of text, mimicking the nuances of human speech. The powerful capabilities of LLMs make it easy to integrate these systems, for example, to build voice assistants. Owing to the unified Transformer structure, in addition to simple cascading, they can be completely integrated in an end-to-end manner [27]. LLMs have the ability to replace the LM module in ASR, decode discretized representations directly, or even replace the intricate TTS text frontend, resulting in more expressive speech generation [28]. Recently, AudioPaLM was created [29] by fusing text- and speech-based language models. It inherited the capability to preserve paralinguistic information and intonation from AudioLM [30] and the linguistic knowledge present in text LLMs of PaLM-2 [31]. It outperformed current systems in speech translation tasks and demonstrated the capability to conduct zero-shot speech-to-text translation across numerous languages. **Prompt Engineering:** Prompts provide a natural and intuitive way for humans to interact with LLMs, allowing users to design and supply tailored prompts to guide LLMsin producing preferred responses or accomplishing specific tasks. A typical prompting method is In-Context Learning (ICL) [12], which utilizes natural language text to formulate task descriptions and/or demonstrations, enabling LLMs to recognize and perform new tasks by learning from a few given examples. Furthermore, to further improve ICL, chain-of-thought (CoT) prompting [32] incorporates a sequence of intermediate reasoning steps into prompts.