Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey (2024)

Abstract: Large Language Models (LLMs) demonstrate significant value in domain-specific applications, benefiting from their fundamental capabilities. Nevertheless, it is still unclear which fundamental capabilities contribute to success in specific domains. Moreover, the existing benchmark-based evaluation cannot effectively reflect the performance of real-world applications. In this survey, we review recent advances of LLMs in domain applications, aiming to summarize the fundamental capabilities and their collaboration. Furthermore, we establish connections between fundamental capabilities and specific domains, evaluating the varying importance of different capabilities. Based on our findings, we propose a reliable strategy for domains to choose more robust backbone LLMs for real-world applications.

heading: 1 Introduction
text: In the current research and application of artificial intelligence, the abundant acquisition of big data, breakthroughs in high-performance computing technology, and innovations in algorithm design have jointly promoted the development and deployment of LLMs (Li, 2022). LLMs are also considered to have strong potential value in specific domains, with an increasing number of industries embracing LLMs and already demonstrating outstanding performance (Gururangan et al., 2020; Ling et al., 2023b; Kaddour et al., 2023).
However, applying LLMs in specific domains has encountered a series of challenges. These challenges mainly stem from the inherent characteristics of domain tasks and data, such as the diversity of data sources, the complexity of domainspecific knowledge, and the specificity of application goals and constraints. To enable LLMs to be better applied in specific domains and address the challenges they face in these areas, this paper
* Corresponding author.
summarizes two key issues that need to be resolved when applying LLMs in specific domains:
Issue 1: Fundamental capabilities of LLMs and their interactions. LLMs exhibit outstanding performance in comprehending and addressing complex tasks, thus demonstrating great potential in specific domain applications. Numerous studies broadly summarize the core capabilities of LLMs as robust comprehension and generation (Huang and Chang, 2023; Ling et al., 2023a), yet fall short of assisting us in aligning the LLMs’ fine-grained capabilities with the intricate requirements of realworld scenarios. Consequently, elucidating the inherent fundamental capabilities manifested by LLMs in domain-specific scenarios and the dynamics among these capabilities becomes essential.
Issue 2: The Capabilities Assessment of LLMs in Specific Domain. Due to the disparity between the capabilities evaluated in benchmarks and those required in real-world domains (Ling et al., 2023a; Kaddour et al., 2023), the excellent performance of LLMs in benchmarks may not necessarily translate to actual applications in specific domains. Therefore, conducting capabilities assessments of LLMs to establish a bridge between benchmarks and real-world domains is crucial.
Based on the above two issues, this survey aims to systematically summarize the fundamental capabilities of LLMs and clarify the capabilities assessment of LLMs. The key contributions of this survey paper are summarized below.
1. This paper summarizes the fundamental capabilities of LLMs in domain applications, including memorization, reasoning, generalization, and diversification. It provides detailed descriptions of each capability and how they collaborate to accomplish specific applications.
11116
2. This paper summarizes the applications of LLMs in nine specific domains from the perspective of real scenarios. In addition, this paper summarizes the importance of fundamental abilities corresponding to each domain, addressing the issue of strong performance on benchmarks not necessarily translating to domain scenarios, and providing users with clear model selection strategies.

heading: 2 Fundamental Capabilities and Interactive Capabilities
text: The human brain’s information processing has been extensively studied, revealing five core modules: natural language interaction, knowledge, memorization, reasoning, and generalization (Xi et al., 2023). Some research suggests that LLMs exhibit a similar information processing mechanism to the human brain (Toneva and Wehbe, 2019; Caucheteux and King, 2022). Consequently, we categorize LLM information processing into four fundamental capabilities including memorization, reasoning, generalization, and diversification. As illustrated in Figure 1, LLMs utilize short-term memory to understand task instructions and longterm memory to retrieve historical data (Cowan, 2008; Norris, 2017; Zhu et al., 2020; Zhang et al., 2023g; Cheng et al., 2016; Davis and Marcus, 2015; Dawid and LeCun, 2023). This data is processed through the reasoning module, which performs logical, commonsense, and symbolic reasoning to generate outputs (Lu et al., 2023b; Bursztyn et al., 2022; Yan et al., 2023; Banerjee et al., 2021; Hamilton et al., 2022). Throughout this process, the generalization enables LLMs to manage information across varying lengths, structures, and tasks (Davis and Marcus, 2015; Lake and Baroni, 2023; Zhao et al., 2021a), while diversification allows for tailored results (Sun et al., 2024).
The four fundamental capabilities of the LLM work together to complete complex domain applications. In the following section, we will introduce each of the fundamental capabilities in detail.

heading: 2.1 Memorization Capabilities
text: The memorization capabilities of LLMs play a pivotal role in their effectiveness and performance across various domains. Memory, in the context of LLMs, refers to the capacity to retain and access information over time. It can be broadly categorized into two types: long-term memory and short-term memory. Long-term memory refers to
the LLM’s ability to store and recall knowledge, facts, and concepts acquired during training and previous experiences. It encompasses the model’s understanding of world knowledge, implicit encoding of substantial information, and its capacity to leverage both internal and external knowledge sources. On the other hand, short-term memory focuses on the LLM’s in-context learning capabilities and its ability to retain and utilize information within a limited temporal context. Enhancements in short-term memory aim to overcome the contextual limitations of LLMs and enable them to generate coherent content over longer stretches.

heading: 2.1.1 Long-term Memory
text: The long-term memory of LLMs is intimately connected to their scale, with LLMs showcasing broader knowledge capacity and diversity. Benchmarks like KoLA critically evaluate LLMs’ world knowledge across numerous tasks (Yu et al., 2023c), while studies by Tirumala et al. (2022) reveal that model size is crucial for efficient memorization. LLMs acting as knowledge bases, as reviewed by AlKhamissi et al. (2022), encode substantial information implicitly, and innovations like REALM employ latent retrievers to augment this knowledge (Guu et al., 2020). Petroni et al. (2019) suggest LLMs can serve as effective knowledge bases even without fine-tuning. Together, these studies underscore the remarkable potential of LLMs to use both internal and external knowledge sources to enhance their long-term memory, applicable across various fields.
Addressing the challenge of preserving the longterm memory of LLMs during continual learning is crucial for their application in specialized fields. Luo et al. (2023c) propose a novel framework, SCCL, which mitigates catastrophic forgetting—a common obstacle to maintaining long-term knowledge—by employing adaptive classification strategies alongside memory replay and distillation techniques. Complementing this, Luo et al. (2023d) suggests that initial training on general linguistic tasks and the adoption of a hybrid continual learning strategy can substantially reduce the loss of long-term syntactic and semantic knowledge.

heading: 2.1.2 Short-term Memory
text: Short-term memory in LLMs has been a focus area to enhance their in-context learning (ICL) capabilities. ICL is a capability that allows LLMs to understand and execute tasks based on the immedi-
ate context provided within the input text (Brown et al., 2020; Dong et al., 2022). This skill set eliminates the need for extensive retraining or finetuning across different tasks. By analyzing a few examples included in their input, LLMs can infer the task requirements and apply learned patterns to generate accurate responses (Zhao et al., 2021b). This in-context learning ability showcases LLMs’ adaptability, making them highly efficient for a broad spectrum of applications with minimal setup (Mosbach et al., 2023). Despite its effectiveness, this approach has limitations in terms of the depth and complexity of understanding it can provide, which is directly influenced by the model’s design and the richness of the context provided (Lu et al., 2022; Zhang et al., 2022b). Meanwhile, the working mechanism of ICL is also a widely open question and has been investigated a lot by the commuity (Min et al., 2022; Liu et al., 2021; Olsson et al., 2022; Bhattamishra et al., 2023).

heading: 2.2 Reasoning Capabilities
text: The reasoning capabilities of LLMs refer to logically process information, draw conclusions, and make decisions based on available data and knowledge (Qiao et al., 2023). The reasoning capabilities of LLMs have greatly enhanced their application across various industries. For example, they apply commonsense reasoning to user interactions in
customer service and healthcare, providing contextually relevant responses. Additionally, advances in symbolic reasoning allow LLMs to support software development and mathematical fields with increased accuracy and clarity. These developments mark significant progress toward more sophisticated AI systems capable of augmenting human tasks. In this section, we summarize the recent advances in the reasoning capabilities of LLMs.

heading: 2.2.1 Compositional Reasoning Capabilities
text: Recent advancements demonstrate that augmenting LLMs with specialized modules or training approaches enhances their compositional reasoning capabilities, surpassing traditional methods in diverse and complex tasks. Lu et al. (2023c) augments LLMs with modules for complex reasoning, achieving significant accuracy improvements on multi-modal tasks. Chen et al. (2023c) propose a novel prompting strategy, skills-in-context (SKiC), enabling LLMs to exhibit compositional reasoning by solving unseen, complex problems through the innovative composition of pre-existing skills, achieving groundbreaking success on compositional tasks. Ma et al. (2023) introduces new benchmarks for evaluating GVLMs’ compositional reasoning, with a novel metric to reduce morphological bias. Compositional Task Representations (CTR), a new prompt-free approach, is proposed
to learn compositional codes, surpassing promptbased methods in zero-shot learning (SHAO et al., 2023). An LLM trained on the PLANE benchmark shows strong capacities in compositional entailment, leveraging subword representations(Bertolini et al., 2022).

heading: 2.2.2 Complex Task Decomposition
text: A variety of new prompting techniques such as ADaPT, chain of thought, zero-shot CoT, iterative context-aware, least-to-most, decomposed, and successive prompting have been proposed to enable LLMs to decompose and tackle complex tasks more effectively (Zhou et al., 2022; Drozdov et al., 2023; Khot et al., 2022; Dua et al., 2022). Prasad et al. (2023) introduces ADaPT, which enhances LLMs’ decision-making by planning and decomposing tasks as needed, significantly improving performance on complex tasks. Wei et al. (2022) demonstrates that "chain of thought" prompting boosts LLMs’ complex reasoning, achieving stateof-the-art results on the GSM8K benchmark. Kojima et al. (2022) presents Zero-shot-CoT, using simple prompts to unlock LLMs’ underlying reasoning capabilities, achieving substantial gains across diverse reasoning tasks. Wang et al. (2022a) proposes an iterative prompting framework that progressively extracts PLMs knowledge for multi-step reasoning, overcoming the limitations of traditional prompting methods.

heading: 2.2.3 Commonsense Reasoning
text: Recent work in commonsense reasoning explores innovative approaches like integrating LLMs with search algorithms, conducting comprehensive surveys, and applying code generation models to outperform traditional methods, while also exposing the limitations of LMs in truly understanding commonsense knowledge without specific supervision. Bhargava and Ng (2022) surveys recent tasks in commonsense reasoning and generation, evaluating the capabilities and limitations of state-of-the-art pre-trained models. Zhao et al. (2023d) demonstrates that combining LLMs with MCTS for task planning leverages commonsense knowledge to enhance reasoning and efficiency in complex tasks. Madaan et al. (2022) proposes using code generation LMs for structured commonsense reasoning tasks, outperforming traditional LMs in natural language processing. Li et al. (2022) conducts a zero-shot and few-shot evaluation of LMs’ commonsense knowledge, revealing limitations and the
insufficiency of larger models to reach human-level performance.

heading: 2.2.4 Symbolic Reasoning
text: Recent work on symbolic reasoning highlights the effectiveness of novel prompting techniques and hybrid frameworks combining LLMs with symbolic solvers or distillation methods. Gaur and Saunshi (2023) explores symbolic reasoning in math word problems using LLMs, introducing a selfprompting method that aligns symbolic reasoning with numeric answers, enhancing interpretability and accuracy. Wei et al. (2022) demonstrates that chain of thought prompting significantly improves LLMs’ performance on complex reasoning tasks including symbolic reasoning. Pan et al. (2023a) introduces Logic-LM, a framework that combines LLMs with symbolic solvers, resulting in substantial improvements in logical reasoning tasks. Gaur and Saunshi (2022) shows that GPT-3’s performance on symbolic math word problems can be enhanced with specific prompting techniques that encourage the model to describe its reasoning process. Li et al. (2023g) reveals that even smaller models can benefit from chain-of-thought prompting through Symbolic Chain-of-Thought Distillation from larger models, leading to improved reasoning performance.

heading: 2.3 Generalization Capabilities
text: Generalization refers to a model’s ability to apply learned knowledge from past experiences to new, unseen situations (Elangovan et al., 2021; Shen et al., 2021). This capability is essential for realworld applications where models encounter various data. In this section, we will explore the generalization capabilities of LLMs, focusing on three key aspects: length, structural, and across-task generalization.

heading: 2.3.1 Length Generalization
text: Length generalization in LLMs, which refers to the model’s capacity to extend acquired skills to longer problem instances outside the training range, is crucial for addressing complex problems with extensive descriptions (Anil et al., 2022). Improving length generalization is key to enhancing the practical use of LLMs in diverse real-world situations.
Theoretical insights, such as Anil et al. (2022) and Xiao and Liu (2023), have examined the length generalization capabilities of transformer-based
models and identified conditions for length generalization in reasoning tasks. For arithmetic tasks, Jelassi et al. (2023) introduced train set priming to improve generalization while the innovative LMInfinite (Han et al., 2023), RASP-Generalization Conjecture (Zhou et al., 2023a) and Attention Bias Calibration (ABC) (Duan and Shi, 2023) employ different variant of attention mechanism for longer text generation. On the other hand, Awasthi and Gupta (2023) focused on multitask training with task hinting to address length generalization.
These studies collectively present a range of strategies, from practical methodologies like task hinting to theoretical frameworks, enhancing LLMs’ ability to manage longer input sequences. They mark significant progress in overcoming the challenges of length generalization, paving the way for more capable and adaptable LLMs.

heading: 2.3.2 Structure Generalization
text: Structure generalization of LLMs refers to the capability to process and interpret complex data structures, such as graphs and tables even though the models are trained on text-only datasets. This ability is crucial for applications extending beyond traditional text-based tasks, spanning various domains including bioinformatics and social network analysis.
Numerous studies have aimed at enhancing the capabilities of LLMs to process and generate diverse data forms beyond traditional text, including graphs (Tang et al., 2023b; Guo et al., 2023a; Pan et al., 2023b; Zhang et al., 2023h; Liu et al., 2023b; Zhang et al., 2023f; Wang et al., 2023b,b), tables (Zhao et al., 2023a), and visualization charts (Wang et al., 2023c). This expansion into handling various data types is particularly notable in fields such as healthcare (Thirunavukarasu et al., 2023), recommendation (Wang et al., 2023c), question answering (Pan et al., 2023b; Jiang et al., 2023a), and biomedical science (Wang et al., 2023b; Qian et al., 2023), significantly broadening the practical applications of LLMs.
These studies collectively underscore the expanding versatility of LLMs in handling structured data, revealing a trend toward more sophisticated AI models capable of complex reasoning and diverse applications.

heading: 2.3.3 Generalization Across Tasks
text: Task generalization in Large Language Models (LLMs) refers to their ability to manage a wide
range of tasks, especially those not seen during training. This capacity allows the models to tackle a variety of novel and unexpected challenges, showcasing their flexibility, efficiency, and versatility.
To enhance task generalization in LLMs, two prevalent strategies are employed. Firstly, fine tuning approaches, such as multi-task (Sanh et al., 2022), instruction tuning (Wei et al., 2021), and meta tuning (Zhong et al., 2021), fine-tune language models across various NLP tasks to augment their comprehension of instructions, thereby achieving significant zero-shot learning capabilities. These methods highlight LLMs’ potential in managing an array of tasks through enhanced instruction understanding. However, fine-tuning parameters of large language models can be resourceintensive. In response, Ye et al. (2023) and Brown et al. (2020) investigate few-shot or in-context learning mechanisms. By providing a few task examples, LLMs can infer the task’s requirements and format, allowing them to address new tasks effectively. This approach circumvents the need for extensive fine-tuning, instead leveraging examples to foster the models’ abilities.
In summary, these studies highlight the generalization of LLMs through diverse methodologies, ranging from fine-tuning to prompting strategies. The overarching objective is to enhance models that not only perform proficiently on familiar tasks but also demonstrate remarkable adaptability to novel challenges, thereby creating more adaptable and intelligent systems.

heading: 2.4 Diversification Capabilities
text: The concept of diversification in LLMs pertains to their capability to produce unique content tailored to various contexts. This diversification arises during the inference process, where a model generates a new token, yt, based on the previously generated tokens, yt−1, and a specific condition, x, according to the formula yt ∼ p(yt|y:t−1, x). Notably, the architecture of most LLMs is decoder-only, meaning that conditions such as prompts or in-context examples are incorporated as initial tokens. Thus, we regard these initial inputs as x and the sequence of generated tokens as y:t−1. By manipulating these inputs and conditions, LLMs can produce a wide array of content.
We delve into the diversification of the expanding capabilities of LLMs in terms of role-playing and creativity. These two areas highlight the versatility of LLMs, showcasing their ability to adapt to
diverse scenarios and tasks. Role-playing enhances the dynamism and context-awareness of LLMs, enabling more nuanced interactions across different scenarios by utilizing role profiles as the condition x.Furthermore, creativity plays a crucial role in unlocking the potential of LLMs for generating innovative and valuable content. This is achieved by modifying the generation process, specifically the sequence of previously generated tokens, yt−1.

heading: 2.4.1 Role-playing
text: Role-playing in LLMs represents a significant advancement in the field of natural language processing and artificial intelligence. It involves LLMs assuming specific characters or personas, enabling them to engage in more dynamic, context-rich, and human-like interactions. By embodying different roles, i.e. x we defined before, LLMs can offer tailored responses based on character-specific knowledge and behavior patterns, enhancing the relevance and engagement of user interactions.
Wei et al. (2023a) investigate multi-party conversations, revealing that LLMs can significantly improve group dynamics when trained on datasets like MultiLIGHT. Wang et al. (2023f)’s RoleLLM framework enhances role-playing in LLMs, leading to advancements in English and Chinese models. Shanahan et al. (2023) discusses the importance of role play in understanding dialogue agents’ behaviors, focusing on aspects like deception and selfawareness. Li et al. (2023a) develop ChatHaruhi, demonstrating enhanced role-playing in mimicking anime characters. Personalization in LLMs is the focus of Salemi et al. (2023)’s LaMP benchmark, which improves model outputs by incorporating user profiles. Finally, Li et al. (2023c) explore autonomous cooperation among LLMs through role-playing, showcasing the potential of inception prompting in multi-agent systems.
These studies collectively represent a significant stride in enhancing the role-playing capabilities of LLMs. They demonstrate how role-playing can transform LLMs into more adaptable, engaging, and effective conversational partners, capable of nuanced interactions across various domains by adjusting the condition x.

heading: 2.4.2 Creativity
text: The creativity in LLMs is gaining traction, emphasizing their potential to generate novel and valuable content. Emphasizing creativity in LLMs is key to developing AI systems that not only replicate hu-
man language but also exhibit a degree of ingenuity akin to human creativity.
Recent studies in this area offer diverse insights. Chakrabarty et al. (2023) develop a framework for evaluating the creativity of LLMs, revealing their current limitations compared to human writers. Franceschelli and Musolesi (2023) explore LLMs’ creative writing potential, examining their development through various creativity theories and considering their societal impact. Summers-Stay et al. (2023) demonstrate that LLMs can enhance their creativity by mimicking human brainstorming techniques. Swanson et al. (2021) introduce tools to assist creative writers in leveraging LLMs’ capabilities, while Sinha et al. (2023) propose a model to balance creativity with factual accuracy in LLM outputs. Bhavya et al. (2023) focus on creative analogy mining using PLMs, underscoring the role of LLMs in augmenting human creativity.

heading: 2.5 Interactive Capabilities
text: In addition to the four fundamental capabilities, LLMs also possess strong interactive capabilities during domain applications. Interactive capabilities refer to the capacity of LLMs to enhance performance by acquiring external information, planning and making decisions regarding the environment, and utilizing external tools (Xi et al., 2023). For example, integrating specialized tools can overcome limitations of LLM in domain tasks (Qin et al., 2023b; Patil et al., 2023), while interaction with environments such as web pages, communities, and databases expands application domains (Yao et al., 2022; Team, 2023a). Appendix A provides a detailed overview of the interactive capabilities of large models.

heading: 3 The Capabilities Assessment of LLMs in Specific Domains
text: LLMs have different applications in different domain scenarios. For instance, in the medical and ledge domains, they may function as domain experts engaged in dialogues or summarizing documents (Shi et al., 2023; Tang et al., 2023a; Choi et al., 2023; Pettinato Oltz, 2023). Systematically summarizing the application methods of LLMs in various domains facilitates to combine these models with specific scenarios more efficiently. However, some research is often classified and summarized from the perspective of NLP tasks (Ling et al., 2023a; Kaddour et al., 2023). Kaddour et al.
(2023) classify applications in medical scenarios into medical question answering and comprehension, and medical information retrieval. However, LLMs may participate in medical diagnosis, diagnostic assistance, and other scenarios. The differences make it difficult for research results to be directly applied to real scenarios. We enumerates articles from nine domains we have summarized in Figure 3, including medicine, law, computational biology, finance, social sciences and psychology, computer programming and software engineering, robots and agents, AI for disciplines, and creative work. We will provide a detailed summary of the real-world applications and roles played by LLMs in these domains in the future.

heading: 3.1 Fundamental Capabilities Assessment and Application
text: The performance of LLMs in specific domains is closely related to their fundamental capabilities. However, we often evaluate LLMs based on their performance on benchmarks, but their strong performance on benchmarks may not necessarily translate to domain scenarios (Guo et al., 2023c; Zhou et al., 2023b). For example, while InstructBLIP exhibits outstanding performance in image caption tests, its performance significantly diminishes in online interactive evaluations closer to real-world scenarios (Dai et al., 2023).
Guo et al. (2023c) highlights that the range of model capabilities assessed by different benchmarks varies, leading to discrepancies between benchmarks and the model’s performance in domain scenarios. Therefore, the quantitative assessment of fundamental capabilities within specific domains is crucial for users in choosing the most appropriate benchmarks. We employ a case study approach to conduct a case-by-case statistical anal-
ysis of the articles in Appendix B, deriving quantitative values for the important capabilities in each domain through expert evaluation. Taking the medical field as an example, among the 25 papers categorized for this study, methods involving memorization capabilities are present in 16 papers, reasoning capabilities in 15, generalization capabilities in 12, diversification capabilities in 10, and interactive capabilities in 5. Therefore, memorization capabilities emerge as the most critical in the medical domain, accounting for 64% of the focus. In table 1, we list the number of papers covered in each domain and analyze the fundamental capabilities demonstrated by these papers across various domains. Based on this data, we create radar charts for each domain.
As shown in Figure 2, we constructed radar charts to illustrate the relative importance of different fundamental capabilities in various domains. Based on these radar charts, researchers can quantify the differences between benchmarks and real scenarios. In the following chapters, we will introduce our selection strategy in medical and computer programming domains as examples.

heading: 3.1.1 Medical
text: According to the radar chart, memorization capabilities (64%) and reasoning capabilities (60%) are important fundamental capabilities in the medical domain. In scenarios like Medical Diagnosis and Knowledge Acquisition, LLMs need to engage in dialogues with patients using their medical knowledge. In this context, long-term memory related to the domain knowledge and reasoning capabilities to assist in answering questions are crucial for model performance. In scenarios of Diagnostic Assistance and Medical Report Generation, LLMs typically assist doctors in reading patient case in-
formation and generating treatment plans. In this context, short-term memory capabilities and reasoning capabilities play a crucial role.
Therefore, we recommend applying LLMs that excel in memorization capabilities and reasoning abilities benchmarks to the medical domain. Here, we provide three recommended medical benchmarks, with Table 2 summarizing the performance of different LLMs on these three benchmarks.
MedQA (Jin et al., 2021) is a medical text question and answer dataset in a multiple-choice format. It aims to test the professional knowledge and clinical decision-making abilities of LLMs. The exam-
ination of professional knowledge mainly targets the memorization capabilities of LLMs, while the assessment of clinical decision-making abilities primarily focuses on the reasoning and generalization capabilities of LLMs.
MedMCQA (Pal et al., 2022) is a large-scale multiple-choice question and answer dataset, with data sourced from All India Institute of Medical Sciences (AIIMS) and National Eligibility cum Entrance Test (NEET PG). Different from the MedQA dataset, besides directly examining the memorization capabilities of LLMs, the MedMCQA dataset includes detailed explanations for each answer, requiring LLMs to possess deep language reasoning capabilities.
PubMedQA (Jin et al., 2019) is a biomedical question answering dataset collected from PubMed abstracts. The task involves generating an answer in a multiple-choice format of yes/no given a question. This dataset demands reasoning over biomedical research texts, especially the capabilities to understand and analyze quantitative content, in order to answer questions.
In order to demonstrate the effectiveness of our proposed method for selecting robust backbone LLM, we chose the medical domain as a case study to further illustrate the practicality and effectiveness of our approach.
As outlined in above, we identiy memorization capabilities and reasoning capabilities as the most crucial fundamental capabilities in the medical domain. Based on this, we recommend models that perform well on benchmarks (MedQA, MedMCQA, and PubMedQA) focused on these capabilities. Through this process, we discover that MedPaLM 2 excels in these benchmarks. This finding
is consistent with the industry recognition that the model has received in the medical domain. Specifically, renowned organizations such as HCA Healthcare, BenchSci, Accenture, and Deloitte have deployed the Med-PaLM 2 model across various medical scenarios, validating its value in real-world applications. In contrast, although RobotGPT-30B and jianpeiGPT performed well on CMB benchmark, their performance in real-world applications does not match that of the former, further proving the effectiveness of our selection methods.

heading: 3.1.2 Computer Programming
text: According to the radar chart, reasoning capabilities (91%) are considered the most crucial skill in computer programming and software development, followed by memorization capabilities (52%). Since code is a symbolic, hierarchical, and logic-driven language commonly used for handling complex tasks, the reasoning capabilities of LLMs are applied in various code scenarios. Short-term memory helps LLMs understand requirements and gather contextual information in code generation and automatic program repair.
Evaluation criteria for programming-related tasks evolve from single-type code language and static metrics to multi-type code languages and metrics (Zhang et al., 2023i; Zan et al., 2023). Among these, evaluation standards involving multilanguage, multi-type metrics require models to possess stronger reasoning capabilities. In this paper, we recommend HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) benchmarks as the basis for selecting LLMs for programming-related scenarios. Table 3 presents the performance of some LLMs on these benchmarks.

heading: 4 Conclusion
text: In this paper, we summarize the fundamental capabilities of LLMs in domain applications and illustrate how they collaborate. Simultaneously, we summarize the applications of LLMs in various domains from real-world perspectives. Furthermore, we outline the key capabilities emphasized in different domains, aiding users in more accurately applying LLMs in domain applications.

heading: 5 Limitations
text: LLMs have found extensive applications across various fields. Although we aim to summarize the applications of LLMs in all domains, our work does not claim to exhaustively cover all possible application scenarios. Furthermore, although every attempt was made to provide readers with a comprehensive overview of each domain, the literature cited and discussed in this document does not constitute a fully exhaustive collection.
Additionally, in assessing the fundamental capabilities focused on by LLMs in various domains, we conduct in-depth analyses of each piece of literature to manually identify these capabilities. However, it is possible that some other relevant works were overlooked, potentially leading to inaccuracies in our analysis.

heading: Acknowledgements
text: We extend our sincere gratitude to the anonymous reviewers for their helpful feedback and the diligent efforts of the conference committees. This work has received partial funding from the Major Research Plan of the National Natural Science Foundation of China (Grant No.92370110) and the Joint Funds of National Natural Science Foundation of China (No.U21B2009).

heading: B Summary of Capabilities in Various
text: Domains

heading: Future Work
text: We will provide a detailed summary of the real-world applications and roles played by LLMs in these domains in the future