Venue: NAACL 2024

# Unleashing the Emergent Cognitive Synergy in Large Language Models:

A Task-Solving Agent through Multi-Persona Self-Collaboration

 Zhenhailong Wang1, Shaoguang Mao2, Wenshan Wu2, Tao Ge2\({}^{\dagger}\), Furu Wei2, Heng Ji1

1University of Illinois Urbana-Champaign, 2Microsoft Research Asia

{wangz3,hengji}@illinois.edu

{shaoguang.mao,wenshan.wu,tage,fuwei}@microsoft.com

Work was done when interning at Microsoft Research Asia.Corresponding author.

###### Abstract

Human intelligence thrives on cognitive synergy, where collaboration among different minds yield superior outcomes compared to isolated individuals. In this work, we propose **Solo Performance Prompting (SPP)**, which transforms a single LLM into a **cognitive synergist** by engaging in multi-turn self-collaboration with multiple personas. A cognitive synergistic is an intelligent agent that collaboratively combines multiple minds' strengths and knowledge to enhance problem-solving in complex tasks. By dynamically identifying and simulating different persons based on task inputs, SPP unleashes the potential of cognitive synergy in LLMs. Our in-depth analysis shows that assigning multiple fine-grained personas in LLMs improves problem-solving abilities compared to using a single or fixed number of personas. We evaluate SPP on three challenging tasks: Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, encompassing both **knowledge-intensive** and **reasoning-intensive** types. Unlike previous works, such as Chain-of-Thought, that solely enhance the reasoning abilities in LLMs, experimental results demonstrate that SPP effectively reduces factual hallucination, and maintains strong reasoning capabilities. Additionally, comparative experiments show that cognitive synergy only **emerges** in GPT-4 and does not appear in less capable models, such as GPT-3.5-turbo and Llama2-13b-chat, which draws an interesting analogy to human development. Code, data, and prompts can be found at: [https://github.com/MikeWangWZHL/Solo-Performance-Prompting.git](https://github.com/MikeWangWZHL/Solo-Performance-Prompting.git)

## 1 Introduction

Although large language models (LLMs) have demonstrated impressive performance as general task-solving agents, they still encounter challenges Qin et al. (2023); Bang et al. (2023); OpenAI (2023); Bubeck et al. (2023) in various knowledge-intensive and reasoning-intensive tasks due to factual hallucination Maynez et al. (2020) and a lack of slow-thinking Sloman (1996) capabilities. Unlike humans, who can leverage the power of collaboration and information integration among different cognitive processes and individuals (referred to as _cognitive synergy_Curseau et al. (2015); Goertzel (2009, 2017)), current LLMs are akin to "jack-of-all-trades" with a vast mixture of knowledge and characteristics. Recent advancements, such as Chain-of-Thought (CoT) prompting Wei et al. (2023); Kojima et al. (2022) and Self-refinement Madan et al. (2023); Shinn et al. (2023), have successfully enhanced the reasoning abilities of LLMs by simulating slow-thinking through the generation of intermediate steps or iterative revision. However, factual hallucination remains a major challenge for LLMs on knowledge-intensive tasks.

A cognitive synergism is an intelligent agent

Figure 1: Schematic illustration of Solo Performance Prompting (SPP) and the difference compared to previous prompting methods.



that collaborates with multiple minds to enhance problem-solving and efficacy in complex tasks. In this work, we aim to **create a cognitive synergism based on a single LLM** that can _"split into" multiple personas and engage in self-collaboration to solve both knowledge-intensive and reasoning-intensive tasks_. This idea is heavily inspired by the role of pretend play Piaget (1954); Pellegrini (2009) in cognitive development and recent findings that assigning personas Deshpande et al. (2023); Xu et al. (2023) to LLMs can elicit specific behaviors, improve answer quality, and potentially build an AI society Park et al. (2023); Schick et al. (2022); Li et al. (2023); Cai et al. (2023) with collaborative LLM agents. However, as shown in Table 1, previous works have limitations such as fixed or task-specific personas, the need for additional fine-tuning, and increased inference costs due to multiple LLM instances.

To unleash the potential of cognitive synergy for general task-solving, we propose **Solo Performance Prompting (SPP)**, which _prompts a single LLM to identify, simulate, and collaborate with multiple personas_. Figure 1 provides a high-level overview of SPP. Here, a persona can represent either a domain expert, such as a movie enthusiast, or a target audience, such as a ten-year-old child. Through the dynamic identification of various personas, we empower a single LLM to acquire diverse domain knowledge accurately without additional retrieval systems. By facilitating multi-turn self-collaboration, we enable self-revision and self-feedback from various perspectives without requiring additional agents.

In real-world scenarios, such as those in creative industries, there is often a need to incorporate diverse information from different domains. Figure 2 presents a concrete example of how SPP operates on a challenging task that requires creative integration of information from various domains, such as the Legend of Zelda game, Harry Potter movies, and Jay Chou's albums. Standard prompting fails to generate satisfactory output due to missing essential information and factual errors. In contrast, SPP produces informative and coherent answers by automatically identifying expert personas and engaging in a multi-turn self-collaboration. In this process, the AI Assistant persona iteratively writes

Figure 2: Task-solving example of Solo Performance Prompting (SPP) with GPT-4. The personas of the participants are automatically identified by GPT-4 based on the task input. This example shows that Standard Prompting suffers from factual errors, whereas SPP provides accurate information and a coherent answer. Note that, in real-world applications, the domains can vary not only within entertainment but also encompass history, science, education, healthcare, etc.

 

drafts of the story, solicits feedback from other participants, and revises accordingly.

To explore the prevalence of cognitive synergy in different LLMs, we apply SPP to LLMs with varying scales and capabilities, including GPT-4, GPT-3.5-turbo, and Llama-13b-chat. Comparative results show that cognitive synergy only emerges in GPT-4 and not in less capable models. This draws an interesting analogy to human development, as children typically start engaging in role-playing at the age of 2 to 3 (Piaget, 1954), but not earlier. In summary, the key contributions of this paper are as follows:

* We investigate whether LLMs can leveraging cognitive synergy for general task-solving. We introduce **Solo Performance Prompting (SPP)**, which simulates multi-agent, multi-persona collaboration in a pure zero-shot manner.
* We evaluate SPP across **three challenging tasks**: Trivia Creative Writing, Codenames Collaborative and Logic Grid Puzzle, spanning both knowledge- and reasoning-intensive domains. To our knowledge, SPP is the first zero-shot prompting method that can enhance both knowledge and reasoning abilities on GPT-4.
* We present an intriguing finding regarding the emergent nature of cognitive synergy ability in LLMs, which **only emerges in GPT-4** and not in less powerful models.
* We conduct in-depth analyses of the impact of the identified personas and SPP prompt design, providing insights into why **dynamic, fine-grained personas** are necessary, as opposed to fixed, coarse-grained personas.

## 2 Solo Performance Prompting

To unleash the power of synergizing different personas to tackle complex problems, we propose Solo Performance Promoting (SPP) which instructs a LLM to perform the following the procedure for general task-solving: **(1) Persona Identification**: Identify multiple participants with special personas (including a leader persona: AI Assistant) that are essential for solving the particular task. **(2) Brainstorming**: The participants share knowledge and provide suggestions on how to approach the task based on their own expertise. **(3) Multi-Persona Iterative Collaboration**: The leader persona, AI Assistant, proposes initial solutions, consults the other participants for feedback, and revise the answer iteratively. Figure 2 shows a walking example of SPP during inference. Next, we formally describe the SPP procedure in detail.

Given an input sequence \(x\) and a model \({\cal M}\), let a prompt (including demonstration examples) prepended to the input to be \(p\) and the final output to be \(y\). Denote an intermediate generation before generating the final \(y\) as \(z\). Under this formulation, Standard Prompting and Chain-of-Thought (CoT) Prompting can be described as:

_Standard Prompting:_ \[y={\cal M}(x)\] (1) _CoT Proempting:_ \[y={\cal M}(p_{col}\|x\|\{z_{1},z_{2},...,z_{n}\})\] (2) where \[p_{col}\] is the CoT prompt, e.g., "Solve the task step-by-step" and \[\{z_{1},z_{2}...,z_{n}\}\] are the intermediate steps. In contrast, our proposed Solo Performance Prompting can be described as follows:

_Solo Performance Prompting:_ \[y=\\ {\cal M}(p_{spp}\|x\|z_{p}\|\{z^{1}_{k},z^{2}_{2},...,z^{m}_{b}\}\|\{ z^{0}_{s},z^{1}_{f},...,z^{m}_{f}\}_{j=1..n})\] (3)

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **General task** & **Pure zero-shot** & **Has multiple** & **Fersonas dynamically** & **Has iterative** & **Need only a** \\  & **solving?** & **prompting?** & **persons?** & **identified?** & **refinement?** & **single LLM?** \\ \hline \hline \end{tabular} † Standard Prompting (Brown et al., 2020)

� Chain-of-Thought (Wei et al., 2023)

� Inner Monologue (Huang et al., 2022)

� ReAct (Yao et al., 2022)

� Reflection (Shinn et al., 2023)

� Self-Refine (Madan et al., 2023)

� Tree-of-thought (Yao et al., 2023)

� GPT-Bargaining (Fu et al., 2023)

� Camel (Li et al., 2023)

� ExpertPrompting (Xu et al., 2023)

� See Table 1: High-level comparison with various prompting-based methods. Methods directly comparable to ours are denoted by ‡. Results for the comparison can be found in Section 3. In Section 4, we further design and compare with two variants of Solo Performance Prompting: one adopting fixed personas, as in Camel (Li et al., 2023), and another with additional persona profiles, as proposed in ExpertPrompting (Xu et al., 2023) 

where the SPP prompt (\(p_{spp}\)) includes a high-level instruction and two carefully crafted demonstration examples1 that showcase the expected task-solving procedure of SPP. We describe the design details of the prompt in SSA**A.1**. The corresponding intermediate generations (\(z\)) of SPP are detailed below.

Footnote 1: The tasks we use in the demonstration examples do not overlap with the evaluation tasks.

Persona Identification (\(z_{p}\)).Given an input task, SPP first generates a list of participants with different personas. For example in Figure 2, the model identified a _Jay Chou Fan_ persona to help answer "the last song in the second album by Jay Chou". We let the language model identify the personas dynamically instead of manually defining them. Given only two demonstration examples (detailed in SSA), we observe that a state-of-the-art large language model, e.g., GPT-4 [35], can identify accurate and meaningful personas for diverse tasks. We denote this part of intermediate generation as \(z_{p}\) in Equation 3.

Brainstorming (\(z^{i}_{b}\)).Among the identified participants, "AI Assistant (you)" is treated as a leader persona that initiates the collaboration and generates initial solutions. Before generating the initial answer, the personas brainstorm on how to approach the task from their own perspectives. For example, the _Jay Chou Fan_ points out that the last song in Jay Chou's second album is "An Jing" ("Silence"). We find that the brainstorming phase effectively improves the quality of the initial solution. In Equation 3, the superscript \(i=0\) is used to denote the "AI Assistant" persona, while \(i\geq 1\) represents other dynamically identified personas. The intermediate generations of the brainstorm step are denoted as \(\{z^{1}_{b},z^{2}_{b},...,z^{m}_{b}\}\).

Multi-Persona Iterative Collaboration (\(z^{0}_{s}\), \(z^{i}_{f}\)).Based on the brainstorming remarks, the AI Assistant persona generates an initial solution \(z^{0}_{s}\), then it consults each of the other participants for feedback \(\{z^{i}_{f}\}\). The participants are encouraged to critique the current generation and give revision suggestions. For example, the Jay Chou Fan persona checks whether the song "An Jing" ("Silence") is correctly included in the story. This process can be repeated for multiple times until every participant is satisfied with the current solution. In Equation 3, we denote the intermediate generations of the multi-turn dialogue as \(\{z^{0}_{s},z^{1}_{f},...,z^{m}_{f}\}_{j=1...n}\) where \(n\) is the number of iterations before reaching the final answer. The final answer can be directly read out following user-specified output format.

In summary, SPP instructs an LLM to solve general tasks via multi-persona self-collaboration in a pure zero-shot manner. In contrast, as detailed in Table 1, previous prompting-based methods are either task-specific or require additional mechanism, e.g., searching [67], external tools [68], memory component [9], and fine-tuning [69].

## 3 Experiments

To explore the effectiveness of Solo Performance Prompting (SPP), we adopt an evaluation methodology similar to that of previous work [67]. We carefully design new tasks and select tasks from existing benchmarks [11] that are challenging even for the most capable LLMs [35]. The evaluation aims to cover diverse types of tasks encompassing both _knowledge-intensive_ and _reasoning-intensive_ domains.

Tasks.We invent the **Trivia Creative Writing** task (SS3.1), which requires the model to internally acquire and integrate diverse information from various fields. We observe that even GPT-4 [35] frequently exhibit hallucination and factuality errors in the Trivia Creative Writing task. We also propose the **Codenames Collaborative** task (SS3.2), an extension of the Codenames task from the BigBench [11] that features a two-role collaboration setup. Codenames Collaborative demands creative reasoning across a broad range of related knowledge and challenges the model's theory of mind skills. Lastly, we include a challenging pure-reasoning task, **Logic Grid Puzzle** (SS3.3), from the BigBench [11] which necessitates complex multi-step reasoning.

Baselines.We compare our approach with **Standard Promoting**, **Chain-of-Thought****(CoT)** prompting methods (outlined in SS2) and **SelfRefine**[12]. For CoT, a similar prompt design to [67] is employed, where the model is prompted to generate a plan or a series of steps before producing the final output. For Self-Refine, we follow [12] to design _feedback_ and _refine_ prompts. We perform one self-refine iteration which requires three times more inferences than SPP. Full prompts for the methods can be found in Appendix A.2 

Models.The default model we use is GPT-4 (OpenAI, 2023b). Detailed inference configurations, API versions, and full results can be found in Appendices C and F. In SS3.4, we further investigate the prevalence of cognitive synergy in LLMs with different scales and capabilities, including GPT-3.5-turbo (OpenAI, 2023a) and Llama2-13b-chat (Touvron et al., 2023).

Trivia Creative Writing: A Knowledge-Intensive Task Description.As illustrated in Figure 3, Trivia Creative Writing asks a model to write a coherent story while incorporating the answers to \(N\) trivia questions. Our preliminary experiments (Figure 10) show that a sufficiently large \(N\) can effectively challenge GPT-4 to demonstrate factual knowledge across diverse domains. Thus, we mainly consider two evaluation settings, \(N=5\) and \(N=10\). We built a benchmark with 100 instances for each \(N\), covering a total of 1000 trivia questions2 extracted from the TriviaQA Joshi et al. (2017) dataset. More details can be found in Appendix B.1.

Footnote 2: To select difficult question instances that can pose challenges to GPT-4, we use a smaller open-source LLM, _fastchart_5_3b_Zheng et al. (2023), to obtain preliminary performance on the validation set, and then choose the failure cases as our question selection.

Evaluation Metrics.Evaluating GPT-4 level generation results can be challenging. Our preliminary experiments indicate that, even for humans, it is very difficult to identify which generation is better in terms of overall "quality" of the story from different prompting methods. Thus, instead of focusing on evaluating the coherence of the generation, which can be highly subjective, we employ an automatic metric which focuses on detecting factual hallucinations. As shown in Figure 3, we perform string matching with the ground truth target answers for each question on the output generation. For each question, a match to any of the answer aliases provided by the TriviaQA dataset is considered a correct mention. The metric score is computed as: \(\frac{\text{\# correct answer mentions}}{\text{\# trivia questions}}\).

Results.Table 2 presents the results of the Trivia Creative Writing task. The key observations are as follows: (1) Chain-of-Thought (CoT) does not outperform Standard prompting, indicating that CoT is ineffective in eliciting an LLM's knowledge abilities. Qualitative examples in Figure 8 and 11 illustrate that although CoT generates reasonable plans for task resolution, the final generation still contains factual errors and hallucinations. (2) SelfRefine only brings marginal improvements over iterations. (3) SPP outperforms all baselines significantly. The improvement is more pronounced in the \(N=10\) setting compared to \(N=5\) (10% vs. 7%), suggesting that Solo Performance Prompting is particularly beneficial when the task requires incorporating knowledge from numerous domains.

Codenames Collaborative: A Knowledge+Reasoning Task Description.As illustrated in 4, Codenhanges Collaborative is a collaborative task that challenges a model's knowledge, reasoning, and theory of mind abilities by assigning two player roles: the _Spymaster_ and the _Guesser_. The Spymaster's role is to provide a hint word related to the target words, excluding some other distractor words, while the Guesser's role is to identify the target words based on the given hint and the full list of words. The same LLM (GPT-4 (OpenAI, 2023b)) is used for both roles sequentially, and a dataset with 50 instances is constructed based on BigBench's Srivastava et al. (2022) codenames task data.

Evaluation Metrics.The original Codenames task in the BigBench dataset has limitations due to its focus on the Guesser role and subjectivity in hint

\begin{table}
\begin{tabular}{l|c c|c c||c c||c c} \hline \hline \multirow{2}{*}{**Methods**} & \multicolumn{2}{c|}{**Trivia.C.W (N=5)**} & \multicolumn{2}{c||}{**Trivia.C.W (N=10)**} & \multicolumn{2}{c||}{**Codenames.C**} & \multicolumn{2}{c}{**Logic.G.Puzzle**} \\  & Score (\%) & \(\Delta\) & Score (\%) & \(\Delta\) & Score (\%) & \(\Delta\) & Score (\%) & \(\Delta\) \\ \hline Standard & 74.6 & 0.0\% & 77.0 & 0.0\% & 75.4 & 0.0\% & 57.7 & 0.0\% \\ CoT & 67.1 & \(\downarrow\)10.0\% & 68.5 & \(\downarrow\)11.1\% & 72.7 & \(\downarrow\)3.6\% & 65.8 & \(\uparrow\)14.1\% \\ \hline Self-Refine [iter=0] & 73.8 & & 76.3 & & 75.2 & & 58.8 & \\ Self-Refine [iter=1] & 73.9 & \(\downarrow\)1.0\% & 76.9 & \(\downarrow\)0.1\% & 64.6 & \(\downarrow\)14.6\% & 60.0 & \(\uparrow\)4.0\% \\ \hline
**SPP (ours)** & **79.9** & \(\uparrow\)**7.1\%** & **84.7** & \(\uparrow\)**10.0\%** & **79.0** & \(\uparrow\)**4.8\%** & **68.3** & \(\uparrow\)**18.5\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: GPT-4 results on Trivia Creative Writing (Trivia.C.W), Condenames Collaborative (Codenames.C) and Logic Grid Puzzle (Logic.G.Puzzle). \(\Delta\) indicates the relative gain/loss compared with Standard Prompting (first row). We report the average scores across two individual runs with/without a system message (detailed in Appendix C).

 

words. Our new task, Codenames Collaborative, resolves this by creating a self-contained evaluation setting that accurately measures the model's capability without human annotation. As illustrated in Figure 4, we compute the overlapping ratio between the predicted words from the Guesser and the target words as the metric.

Results.Table 2 shows the results on the Codenames Collaborative task. Similar to the Trivia Creative Writing task, we find that CoT does not bring positive gains compared with the Standard prompting. Interestingly, iterative self-refinement brings negative impact on this task, due to a high tendency changing the initial response even if it is already good. In contrast, SPP brings significant improvements (~5%), which indicates its effectiveness on collaborative tasks that require knowledge, reasoning, and theory of mind skills. Figure 12 provides further qualitative examples illustrating that SPP generates _detailed_ and _interpretable_ intermediate dialogues.

### Logic Grid Puzzle: A Reasoning-Intensive Task Description and Evaluation Metrics

Task Description and Evaluation MetricsWe utilize the Logic Grid Puzzle task from the Bigbench [37] dataset, which comprises 200 instances. Each instance describes a logic puzzle typically involving 2 to 5 houses, with each house inhabited by a person with specific characteristics, such as playing the piano. The objective is to answer questions about house numbers based on given clues, which requires multi-step reasoning and the selection of relevant information. An example input and output of the Logic Grid Puzzle task are illustrated in Figure 5. For evaluation metrics, we calculate the accuracy of the predicted house numbers by comparing them with the ground truth targets provided by the dataset.

Figure 4: Codenames Collaborative task example.

Figure 3: Trivia Creative Writing task example.

 

Results.Table 2 presents the results on Logic Grid Puzzle. In contrast to the previous two tasks, we find that CoT brings significant improvements compared to Standard prompting, verifying the observation from previous work that CoT elicits better reasoning abilities. Furthermore, we discover that SPP also achieves strong performance on this reasoning-intensive task.

### The Emergence of Cognitive Synergy

We further discover that **cognitive synergy can only be fully unleashed in LLMs with a certain level of instruction-following capabilities, akin to that of GPT-4.** This can be intriguingly compared to human development, where children usually begin to participate in role-playing around the ages of 2 to 3 (Piaget, 1954), but not before that age.

As shown in Figure 6, the effectiveness of SPP is not seen in smaller and less capable models like GPT-3.5 and Llama2. Additionally, on Llama2, we identify a unique problem which we refer to as _early-termination_, where the model stops generating after identifying the participants, resulting in exceptionally low performance with SPP. The model behaves as if it were waiting for input from a user instead of following the demonstration examples to generate responses on its own. Detailed discussions and examples on the early-termination problem can be found in Appendix E.

## 4 Analysis

SPP effectively improves both knowledge and reasoning abilities in LLMs.As demonstrated by the results in SS3, Solo Performance Prompting (SPP) not only brings significant improvements to knowledge-intensive tasks such as Trivia Creative Writing and Codenames Collaborative without relying on external knowledge bases, but also achieves strong performance on reasoning-intensive tasks like Logic Grid Puzzle. To our knowledge, SPP is the first zero-shot prompting method that can enhance both knowledge and reasoning abilities on GPT-4.

LLMs can effectively identify useful personas in a zero-shot manner.We are interested in investigating whether the identified personas are highly relevant to the tasks. We visualize the personas automatically identified by SPP using a word cloud for each task in Figure 7a, where a larger font indicates a higher frequency. The key observations include: (1) The identified personas are closely correlated with the particular task. For example, in Logic Grid Puzzle, even though "logic puzzle" is not mentioned in the input, the LLM frequently identifies the persona "Logic Puzzle Expert." (2) On knowledge-intensive tasks, such as Trivia Creative Writing, SPP identifies more diverse and specific personas, while on reasoning-intensive tasks, such as Logic Grid Puzzle, the personas are more homogeneous.

We further investigate whether a detailed profile for each persona is needed for eliciting domain knowledge, as suggested by Xu et al. (2023). To this end, we design a variant of SPP, **SPP-Profile**, which involves generating profiles for each persona during the Persona Identification phase. The results in Figure 7b show that SPP-Profile does not outperform SPP. This suggests that a fine-grained

Figure 5: Logic Grid Puzzle task example.

 

persona name without a detailed description may already be sufficient for eliciting certain domain knowledge.

Dynamic personas v.s. fixed personas.To further investigate the importance of dynamically identifying personas for each task instance instead of fixing a general persona, an ablated variant of SPP, **SPP-Fixed-Persona**, is introduced. For SPP-Fixed-Persona, we modify the prompt (Figure 17) to force the personas to be fixed as an "AI Assistant" and an "Expert". Comparing SPP and SPP-Fixed-Persona in Figure 6(b), we have the following insights: (1) SPP consistently outperforms SPP-Fixed-Persona across all tasks, suggesting that dynamic, fine-grained personas are more effective than fixed, general personas. Qualitative examples in Figure 8 and 13 shows that the fine-grained personas such as "Film Expert" and "Sports Enthusiast" correctly provide the answers, while the fixed persona "Expert" fails. (2) SPP-Fixed-Persona also suffers from the **early-termination** problem as defined in SS3.4, where the LLM stops collaboration before providing the final answer as if it were waiting for external inputs.

Impact of the demonstrations in SPP prompt.To investigate the effectiveness of the hand-crafted demonstration examples in SPP, we conduct an ablation study where we remove the second demo example and preserve the first one, which shows only a two-persona collaboration setting. As shown in Figure 9, we observe that (1) Adding the second example, which requires collaboration of more than two personas, effectively boosts the performance. (2) SPP is fairly robust to the prompt change and show good performance with only the first demo example.

## 5 Related Work

LLMs as role-playing agents.Recent research Deshpande et al. (2023); Xu et al. (2023); Fu et al. (2023); aut (2023); Li et al. (2023) demonstrates that assigning personas or roles to LLMs influences their generation behavior. AI societies with distinct personas or occupations have been explored for collaboration Park et al. (2023); Schick et al. (2022); Li et al. (2023); Cai et al. (2023). However, limitations in persona assignment and multi-agent collaboration include single or fixed persona assignments Xu et al. (2023); Fu et al. (2023); Schick et al. (2022); Li et al. (2023) and the need for multiple LLM instances, increasing inference cost. In contrast, SPP uses a single LLM to dynamically identify useful personas for general tasks. Our discovery on the emergent nature of cognitive synergy also aligns with related work Olausson et al. (2023), which investigates the emergent ability of self-debbugging in code generation.

Enhancing reasoning and factual knowledge in LLMs.LLMs face challenges in complex knowledge-intensive tasks due to hallucination Maynez et al. (2020) and reasoning-intensive tasks due to the lack of human-like slow thinking Sloman (1996); Kahneman (2011). Approaches like Chain-of-Thought (CoT) and Self-Refinement encourage LLMs to solve tasks step by step or iteratively revise their answers Wei et al. (2023); Kojima et al. (2022); Zhang et al. (2022); Fu et al. (2022); Xue et al. (2023); Yao et al. (2023); Madaan et al. (2023); Shinn et al. (2023); Gou et al. (2023);

Figure 6: SPP achieves superior performance only with the most powerful LLM (GPT-4), but not with GPT-3.5 and Llama2-13b. This indicates that cognitive synergy abilities only emerge in LLMs with GPT-4 level capabilities.

 

Chen et al., 2023; Huang et al., 2022; Yao et al., 2022). However, these methods do not necessarily reduce factual hallucination. Retrieval augmented LLMs Borgeaud et al. (2022); Izacard et al. (2022); Wang et al. (2022); Shuster et al. (2021) enhance knowledge acquisition but do not improve reasoning abilities. We propose Solo Performance Prompting (SPP) to elicit both knowledge and reasoning abilities in LLMs, improving factuality while maintaining strong performance on pure-reasoning tasks.

## 6 Conclusion

Solo Performance Prompting unleashes the cognitive synergy abilities within powerful LLMs, significantly reducing factual hallucination while enhancing reasoning. The performance is assessed using newly proposed tasks, e.g., Trivia Creative Writing and Codenames Collaborative, demonstrating superior results compared to Standard, CoT and Self-Refine. The discovery of the emergent nature of cognitive synergy on different LLMs draws interesting analogy to human development.

Figure 8: Qualitative examples on Trivia Creative Writing comparing SPP, CoT and SPP-Fixed-Persona. While CoT provides reasonable intermediate steps, it still struggles with factual hallucination. SPP v.s. SPP-Fixed-Persona reveals that dynamically identified fine-grained personas, such as the “Film Expert,” tend to outperform the fixed general persona of an ”Expert. More examples can be found in Figures 11, 12, and 13.

Figure 7: **(a) Qualitative analysis on the identified persons; (b) Quantitative analysis on two SPP variants.**





### Limitations

Although Solo Performance Prompting exhibits promising improvements in acquiring factually correct knowledge compared to Standard prompting, it has some limitations. For instance, even when a fine-grained persona is assigned, the answer may still be incorrect. It remains unclear to what extent assigning a persona can help enhance domain knowledge in a specific area. Dedicated diagnostic experiments and theoretical efforts are needed to quantify the impact of having a persona or not.

Furthermore, we currently adopt an identical SPP prompt with the same two demonstration examples for any given task inputs, which may be suboptimal. Future work investigating how to find better demonstration examples conditioned on each input could further improve the effectiveness of SPP.

Last but not least, if given sufficient computational budget, a natural variant of SPP could extend to a _multi-agent cognitive synergism_ setup where a leader persona identifies several expert agents and forms a cabinet to collaboratively solve a task. The multi-agent setup allows for leveraging richer computation power, larger local memory, and more flexible human-computer interaction, which could be essential for deploying to real-world applications.

## Acknowledgements

We would like to express our gratitude to the anonymous reviewers for their insightful comments and suggestions. We would also like to thank our colleagues and fellow interns at Microsoft Research Asia for their valuable internal discussions and feedback. Zhenhailong Wang and Heng Ji are partially supported by U.S. DARPA ECOLE Program No. #HR00112390060 and U.S. DARPA ITM Program No. FA8650-23-C-7316. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government.

## References

* Auto-gpt [https://github.com/Significant-Gravitas/Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT).
* Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. _arXiv preprint arXiv:2302.04023_.
* Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trilions of tokens. In _International conference on machine learning_, pages 2206-2240. PMLR.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Bubeck et al. (2023) Sebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzh Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. _arXiv preprint arXiv:2303.12712_.
* Cai et al. (2023) Tianle Cai, Xuezhi Wang, Tengyu Ma, Linyuan Chen, and Denny Zhou. 2023. Large language models as tool makers. _arXiv preprint arXiv:2305.17126_.
* Chen et al. (2023) Xinyun Chen, Maxwell Lin, Nathanaael Scharli, and Denny Zhou. 2023. Teaching large language models to self-debug. _arXiv preprint arXiv:2304.05128_.
* Curseu et al. (2015) Petru L Curseu, Nicoleta Meslec, Helen Pluut, and Gerardus JM Lucas. 2015. Cognitive synergy in groups and group-to-individual transfer of decision-making competencies. _Frontiers in psychology_, 6:1375.
* Deshpande et al. (2023) Ameet Deshpande, Vishvuk Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. _arXiv preprint arXiv:2304.05335_.
* Fu et al. (2023) Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023. Improving language model negotiation with self-play and in-context learning from ai feedback. _arXiv preprint arXiv:2305.10142_.
* Fu et al. (2022) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-based prompting for multi-step reasoning. _arXiv preprint arXiv:2210.00720_.
* Goertzel (2009) Ben Goertzel. 2009. Cognitive synergy: A universal principle for feasible general intelligence. In _2009 8th IEEE International Conference on Cognitive Informatics_, pages 464-468. IEEE.
* Goertzel (2017) Ben Goertzel. 2017. A formal model of cognitive synergy. In _Artificial General Intelligence: 10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15-18, 2017, Proceedings 10_, pages 13-22. Springer

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing.
* Huang et al. (2022) Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022. Inner monologue: Embodied reasoning through planning with language models. _arXiv preprint arXiv:2207.05608_.
* Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. _arXiv preprint arXiv:2208.03299_.
* Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
* Kahneman (2011) Daniel Kahneman. 2011. _Thinking, fast and slow_. macmillan.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _arXiv preprint arXiv:2205.11916_.
* Li et al. (2023) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitri Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large scale language model society. _arXiv preprint arXiv:2303.17760_.
* Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. _arXiv preprint arXiv:2303.17651_.
* Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1906-1919, Online. Association for Computational Linguistics.
* Olaussson et al. (2023) Theo X Olaussson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Demystifying gpt self-repair for code generation. _arXiv preprint arXiv:2306.09896_.
* OpenAI (2023a)OpenAI. 2023a. Gpt-35. [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5).
* OpenAI (2023b)OpenAI. 2023b. GPT-4 technical report.
* Park et al. (2023) Joan Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simularca of human behavior. _arXiv preprint arXiv:2304.03442_.
* Pellegrini (2009) Anthony D Pellegrini. 2009. _The role of play in human development_. Oxford University Press, USA.
* Piaget (1954) Jean Piaget. 1954. The construction of reality in the child.
* Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? _arXiv preprint arXiv:2302.06476_.
* Schick et al. (2022) Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gaudier Izacard, Qingfei You, Christoforos Nampantis, Edouard Grave, and Sebastian Riedel. 2022. Peer: A collaborative language model. _arXiv preprint arXiv:2208.11663_.
* Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. _arXiv preprint arXiv:2303.11366_.
* Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. _arXiv preprint arXiv:2104.07567_.
* Sloman (1996) Steven A Sloman. 1996. The empirical case for two systems of reasoning. _Psychological bulletin_, 119(1):3.
* Srivastava et al. (2022) Anrohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abbubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _arXiv preprint arXiv:2206.04615_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2022) Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, and Heng Ji. 2022. Zemi: Learning zero-shot semi-parametric language models from multiple tasks. _arXiv preprint arXiv:2210.00185_.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
* Xu et al. (2023) Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023. ExpertPromping: Instruction large language models to be distinguished experts. _arXiv preprint arXiv:2305.14688_.

 

Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, and Heng Ji. 2023. Rcot: Detecting and rectifying factual inconsistency in reasoning by reversing chain-of-thought. _arXiv preprint arXiv:2305.11499_.
* Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. _arXiv preprint arXiv:2305.10601_.
* Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. _ArXiv_, abs/2210.03629.
* Zhang et al. (2022) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models.
* Zheng et al. (2023) Linmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Ducheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.

## Appendix A Prompts

### SPP Prompt Design

To prompt an LLM to behave as a cognitive synergistic that follows the expected task-solving procedure as mentioned in SS2, we carefully designed the structure of the SPP prompt as follows. The full prompts can be found in SSA.2.3

Footnote 3: We use the same prompt for any arbitrary tasks.

System Principle.The first part of the prompt contains a high-level instruction: "When faced with a task, begin by identifying the participants who will contribute to solving the task. Then, initiate a multi-turn collaboration process until a final solution is reached. The participants will give critical comments and detailed suggestions whenever necessary."

Demonstration Examples.Then, we include two manually crafted demonstration examples to showcase the expected task-solving behavior. The first example describes a _Game of 24_ task, where we only include two personas: an AI Assistant and a Math Expert. This task aims to provide an example of a _reasoning-intensive task_, where the AI Assistant needs to propose multiple proposals, and the other participants need to give _fine-grained feedback_ on where the current solution went wrong and how to improve it. The second example describes a poem-writing task with _diverse requirements_, including lexical constraints, semantic constraints, and audience awareness. This task aims to provide an example of a _knowledge-intensive task_, where diverse personas are required to collaboratively solve the task. This example also demonstrates a case where it is important to assign a dedicated persona to the audience, e.g., a ten-year-old child.

Task Prefix.The last part of the prompt reminds the model to "identify the participants and collaboratively solve the following task step by step." followed by task-specific format instructions and inputs.

### Full Prompts

Figures 15, 16 and 17 show the full prompts for SPP, SPP-Profile and SPP-Fixed-Persona respectively. Figure 18 shows the prompts for Chain-of



Thought (CoT) prompting. Figure 19 shows the prompts for Self-Refine prompting.

## Appendix B Task Details

### Trivia Creative Writing

Figure 3 shows a detailed illustration of the Trivia Creative Writing task. Additionally, we investigate how the number of the questions (N) and the ordering of the questions would affect the performance on the Trivia Creative Writing task. As shown in Figure 10, with a larger number of questions (N\({\geq}5\)), Trivia Creative Writing effectively challenges GPT-4's performance. While a single question (N=1) yields similar outcomes regardless of the prompting method, SPP approach is notably superior for larger Ns. The ordering of the questions has minimal impact to the task performance.

The topic list is automatically generated by prompting GPT-4 to provide 100 nouns from pop culture4.

Footnote 4: The full prompt for generating the topic list can be found in Figure 20. We performed further human curation to avoid potential harmful content.

## Appendix C Inference Configurations

The main results in Table 2 are obtained from GPT-4. The GPT-4 API version we employ is Azure 2023-3-15-preview.5 The _temperature_ is set to \(0.0\) (most conservative) and _top_p_ to \(1.0\) for all generations to maximize reproducibility. Since even though the temperature is set to \(0.0\) the GPT-4 generation can still be non-deterministic, we conduct additional experiment to investigate its generation consistency under this configuration. As shown in Table 3, we perform three individual runs and compute the mean and standard deviation of the metric score on Trivia Creative Writing. We find that the variance is sufficiently small and Solo Performance Prompting enjoys lower variance than Standard and CoT prompting.

Footnote 5: There are rare cases when a generation triggers the content filter of the API. We exclude those instances from our results.

To evaluate the potential impact of initial persona assignment through a system message, we consider two inference settings: _with_ or _without_ the default system message, "You are an AI assistant that helps people find information". Divergent patterns are observed across various tasks and methods regarding the use of the system message. We report the average metric scores across both inference settings in Table 2. Full GPT-4 results for each setting can be found in Appendix F.

For GPT-3.5 results in Figure 6, we employ the same prompt, hyper-parameters and the best system message setting in terms of SPP's GPT-4 performance. For Llama2, we leverage the Huggingface text-generation pipeline6 with greedy decoding.

Footnote 6: [https://huggingface.co/blog/llama2](https://huggingface.co/blog/llama2)

## Appendix D Additional Qualitative Analysis

Figure 11 presents examples of the Trivia Creative Writing task, illustrating that although CoT can generate plausible plans for task resolution, the final outcomes often contain factual inaccuracies and instances of hallucination. In contrast, SPP elicits precise knowledge with fine-grained personas.

Figure 12 displays examples of the Codenames Collaborative task, illustrating that SPP generates

\begin{table}
\begin{tabular}{l|c c c|c} \hline \hline
**Methods** & **Run 1** & **Run 2** & **Run 3** & **Mean (std)** \\ \hline Standard & 75.6 & 74.4 & 73.1 & 74.4 \(\pm 1.3\) \\ CoT & 68.8 & 69.6 & 70.8 & 69.7 \(\pm 1.0\) \\ SPP & 80.0 & 79.8 & 80.8 & 80.2 \(\pm 0.5\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Investigation on the generation consistency of GPT-4 API. The experiment is performed on the Trivia Creative Task (N=5). We set the inference temperature to 0.0 and top_p to 1.0 as all experiments conducted in the paper. The results show that the GPT-4 generation is fairly consistent with a small variance (\(\sim 1\%\)). We also observe that SPP shows lower variance compared with Standard and CoT prompting across different runs.

Figure 9: Analysis on the impact of the demonstration examples in SPP prompt. We compare the effectiveness of the original SPP prompt with a variant where we remove the second demonstration example, which shows a multi-persona scenario. We observe that (1) SPP is fairly robust to the change in the prompt; (2) adding an additional multi-persona example apart from the single-persona one effectively boosts performance on all three tasks.

 

intermediate dialogues that are both _detailed_ and _interpretable_, leading to superior performance compared to CoT.

Figure 13 shows additional qualitative examples on Solo Performance Prompting vs SPP-Profile.

## Appendix E Early-termination with SPP-Fixed-Persona

Figure 14 shows an example of the early-termination problem (defined in SS 4) where the generation stops before reaching the final solution as if the models is waiting input from an external user.

The problem is particularly severe on certain tasks, e.g., Codenames Collaborative, resulting in unexpectedly low performance as shown in Figure 6(b). The problem can be largely alleviated by removing the system message but cannot be entirely eliminated. Table 4 shows the statistics of the early-termination problem for each task and method. In contrast, we did not observe early-termination on SPP, SPP-Profile, Standard, or CoT prompting with GPT-4.

## Appendix F Full Results

Full results of the three tasks: Trivia Creative Writing, Codenames Collaborative and Logic Grid Puzzle can be found in Tables 5, 6 and 7, respectively.

 


* [10] M. C. D. Lee, and J. S. Kim, "A survey of the population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 6, pp. 89-96, 2000.
* [11] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-118, 2000.
* [12] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-118, 2000.
* [13] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-118, 2000.
* [14] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-118, 2000.
* [15] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-120, 2000.
* [16] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [17] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [18] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [19] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [20] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [21] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [22] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [23] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [24] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [25] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [26] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [27] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [28] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [29] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [30] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [31] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [32] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [33] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [34] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [35] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [36] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [37] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [38] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [39] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [40] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [41] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [42] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [43] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [44] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [45] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 2000.
* [46] A. K. G. L. R. H. W. T. Jones, and P. F. E. Smith, "The population of young populations in a population with a population of 25 years," _Journal of Population Biology: Series B_, vol. 347, no. 1, pp. 105-126, 


* [146] M. C. D'Auria, A. G. Krizhevsky, and R. L. Krause, "The automated computation of deep neural networks for large-scale image recognition," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2098-2103, 2017.
* [147] J. F. Donahue, S. Girshick, and Y. Bengio, "Generative adversarial nets for large scale visual recognition," _arXiv preprint arXiv:1512.00985v1_, 2015.
* [148] P. Dollar, I. Guyon, E. Rabinovich, and V. Ramanan, "Semi-supervised learning with convolutions," in _Advances in Neural Information Processing Systems (NIPS)_, vol. 3, no. 1, p. 103009, 2016.
* [149] T. Darrell, B. Courville, and R. Fergus, "Rectified object detection via convolutional neural networks," _IEEE Transactions on Image Analysis and Machine Intelligence_, vol. 10, no. 1, pp. 1-10, Jan. 2016.
* [150] H. Chen, X. Li, Z. Wang, and W. Sun, "Deep residual network for image classification," _CoT Computers & Graphics_, vol. 10, no. 1, pp. 1-10, Jan. 2016.
* [151] J. Deng, L. Fei-Fei, and Q. Weinberger, "ImageNet: a new approach to image segmentation," _IEEE transactions on pattern analysis and machine intelligence_, vol. 10, no. 1, pp. 1-10, Feb. 2016.
* [152] J. Dong, L. Fei-Fei, and Q. Weinberger, "Imagenet classification by using convolutional neural networks," in _Computer Vision and Pattern Recognition Workshops (CVPR)_, pp. 1-10, Jan. 2016.
* [153] J. Dong, L. Fei-Fei, and Q. Weinberger, "Learning generative adversarial nets for large scale visual recognition," _Journal of Machine Learning Research_, vol. 10, no. 1, pp. 1-10, Jan. 2016.

[MISSING_PAGE_POST]

 

Figure 13: SPP vs SPP-Fixed-Persona qualitative examples on Trivia Creative Writing (N=5). Each example shows one of the trivia questions in the input instance, the identified participants and the provided answer. We observe that the dynamically identified fine-grained personas, such as "Film Expert", "Music Enthusiast" and “Sports Enthusiast”, tend to outperform the fixed general personas, ”Expert”.

 



**Example 1:** The 4 words best associated with the word "film" from the following list:

* [-] _SPP procedure instructions and demonstration examples ..._] Now, identify the participants and collaboratively solve the following task step by step. Remember to present your final solution with the prefix "Final answer:".
* [-] **Task:** (_Write a short and coherent story about Harry Potter that incorporates the answers to the following 5 questions: Who was the man behind The Chipmunks? Which Lloyd Webber musical premiered in the US on 10th December 1993? Who was the next British Prime Minister after Arthur Ballour? Who had a 70s No 1 hit with Kiss You All Over? What claimed the life of singer Kathleen Ferrier?_)

**Generation**

**Participants: Al Assistant (you); Harry Potter Expert; Music Expert; History Expert; Medical Expert Start collaboration!**

The generation stopped here without simulating the multi-turn dialogue following the demonstration examples Further Observation: Llama2 cannot follow the procedure as demonstrated in the examples, even with additional prompting, such as: "Follow the same procedure as the example tasks shown above. Simulate the dialogue of the participants." This indicates that a strong instruction-following ability is a prerequisite for SPP to effectively elicit cognitive synergy abilities.

Figure 14: Examples of the early-termination problem with SPP on Llama2-13b-chat and SPP-Fixed-Persona on GPT-4.





## Appendix BPP Prompt

Figure 15: SPP full prompt.





### SPP-Profile Prompt

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**Scores (N = 5) (\%)**} \\
**Methods** & **w/ system message** & **w/o system message** & **average** & **max** \\ \hline Standard & 75.6 & 73.6 & 74.6 & 75.6 \\ CoT & 68.8 & 65.6 & 67.1 & 68.8 \\ \hline Self-Refine [iter=0] & 74.9 & 72.7 & 73.8 & 74.9 \\ Self-Refine [iter=1] & 75.3 & 72.5 & 73.9 & 75.3 \\ \hline SPP-Fixed-Persona & 66.1 & 79.6 & 72.9 & 79.6 \\ SPP-Profile & 79.8 & 78.3 & 79.1 & 79.8 \\
**SPP** & **80.0** & **79.8** & **79.9** & **80.0** \\ \hline \hline \end{tabular} 
\begin{tabular}{l c c c} \hline \hline
**Methods** & **scores (N = 10) (\%)** & **scores** & **w/ system message** & **average** & **max** \\ \hline Standard & 77.2 & 76.8 & 77.0 & 77.2 \\ CoT & 71.6 & 65.3 & 68.5 & 71.6 \\ \hline Self-Refine [iter=0] & 77.1 & 75.4 & 76.3 & 77.1 \\ Self-Refine [iter=1] & 78.2 & 75.6 & 76.9 & 78.2 \\ \hline SPP-Fixed-Persona & 70.5 & 81.3 & 75.9 & 81.3 \\ SPP-Profile & 82.3 & 83.8 & 83.0 & 83.8 \\
**SPP** & **85.2** & **84.2** & **84.7** & **85.2** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Trivia Creative Writing full results, including two inference settings: with system message and without system message. “average” and ”max” indicating the mean and max score across the two settings. The system message we use is:“You are an AI assistant that helps people find information.."

Figure 16: SPP-Profile full prompt. “[...]” indicates identical parts with SPP. Green text indicates the key difference between SPP-Profile and SPP.





## Appendix B

Figure 17: SPP-Fixed-Persona full prompt. Red text indicates the key difference between SPP-Fixed-Persona and SPP.



* [15] M. C. Dickson, A. K. Sridharan, and R. J. Schuster, "A survey of the internet: a review on the web-based system design," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 2084 of _Lecture Notes in Computer Science_, pages 379-386, 2007.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Methods** & \multicolumn{4}{c}{**Scores (\%)**} \\  & **w/ system message** & **w/o system message** & **average** & **max** \\ \hline Standard & 56.8 & 58.6 & 57.7 & 58.6 \\ CoT & **69.5** & 62.1 & 65.8 & 69.5 \\ \hline Self-Refine [iter=0] & 62.0 & 55.5 & 58.8 & 62.0 \\ Self-Refine [iter=1] & 64.5 & 55.5 & 60.0 & 64.5 \\ \hline SPP-Fixed-Persona & 63.3 & 65.3 & 64.3 & 65.3 \\ SPP-Profile & 65.7 & 64.0 & 64.8 & 65.7 \\
**SPP** & 66.3 & **70.4** & **68.3** & **70.4** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Logic Grid Puzzle full results, including two inference settings: with system message and without system message. “average” and ”max” indicating the mean and max score across the two settings. The system message we use is: ‘‘You are an AI assistant that helps people find information.’