Simulating Expert Discussions with Multi-agent for Enhanced Scientific Problem Solving (2024)

Abstract: Large Language Models (LLMs) have shown remarkable potential across various domains, yet their application in addressing complex scientific problems remains a formidable challenge. This paper presents a novel methodology to augment the problem-solving capabilities of LLMs by assigning them roles as domain-specific experts. By simulating a panel of experts, each LLM is tasked with delivering professional and cautious responses to scientific inquiries. Our approach involves querying multiple LLMs and assessing the consistency of their responses. High agreement among the LLMs suggests greater confidence in the proposed solution, whereas discrepancies prompt a collaborative discussion among the LLMs to reach a consensus. This method emulates realworld scientific problem-solving processes, fostering a more reliable and robust mechanism for LLMs to tackle scientific questions. Our experimental results show that assigning roles to multiple LLMs as domain-specific experts significantly improves their accuracy and reliability in solving scientific problems. This framework has the potential to advance the application of AI in scientific research, enhancing its effectiveness and trustworthiness.

heading: 1 Introduction
text: Large Language Models (LLMs) have achieved remarkable success in a wide range of natural language processing tasks, including text generation (Swanson et al., 2021; Yang et al., 2023), machine translation (Burda-Lassen, 2023; Alves et al., 2023), and text summarization (Laban et al., 2023). Despite their versatility and strong performance across various domains, the application of LLMs to solving complex scientific problems has remained a significant challenge. The primary obstacle lies not in the absence of domain-specific knowledge within these models, but rather in their
*Corresponding Author
limited ability to effectively harness this knowledge when confronted with intricate scientific problems that demand expert-level understanding and reasoning (Addlesee, 2024).
The application of LLMs to scientific problemsolving presents a unique challenge due to the stringent requirements for precision and reliability in research. Minor inaccuracies can have far-reaching consequences, undermining the validity and trustworthiness of results. While LLMs possess extensive knowledge, their current architectures often struggle to consistently apply this knowledge to meet the rigorous demands of scientific inquiry. This limitation underscores the need for innovative approaches to enhance the problem-solving capabilities of LLMs in specialized domains. Improving the performance of LLMs in accurately and reliably
243
solving complex scientific problems could significantly advance their utility in research settings and unlock new potentials for artificial intelligence in science.
In this study, we introduce a novel methodology called Simulating Expert Discussions with Multiagent (SEDM) that enhances the problem-solving capabilities of LLMs by assigning them roles as domain-specific experts, as illustrated in Figure 1. This approach involves simulating a panel of experts, where each LLM is tasked with providing professional and cautious responses to scientific inquiries. By querying multiple LLMs and evaluating the consistency of their responses, we can gauge the confidence in the proposed solutions. High agreement among the LLMs indicates greater reliability, while discrepancies trigger a collaborative discussion among the models to reach a consensus. This method mirrors real-world scientific problemsolving processes, fostering a more dependable mechanism for LLMs to address scientific questions. We evaluate the performance of SEDM on a range of problems across various scientific domains, including physics, chemistry, and mathematics. We use accuracy as the evaluation metric and compare SEDM with baseline methods such as direct LLM usage and few-shot learning.
In summary, our contributions are:
• We propose a novel multi-agent framework that assigns specific expert roles to LLMs, enabling them to collaboratively address scientific problems.
• We develop a discussion architecture for multiagent systems, and experiments have shown that this architecture can effectively enable multiple agents to reach the correct consensus.
• We demonstrate through extensive experiments that our approach significantly improves the accuracy LLMs in scientific problem-solving. For instance, when using the GPT-4 model, SEDM achieves an average accuracy of 57.18% across all subjects, representing an improvement of 32 percentage points compared to direct query and 36 percentage points compared to few-shot learning. These results advance the application of AI in scientific research.

heading: 2 Related Work
text: Large Language Model Reasoning Large language models (LLMs) have demonstrated significant reasoning capabilities, especially when scaled to hundreds of billions of parameters (Ouyang et al., 2022; OpenAI et al., 2024). Various techniques, such as chain-of-thought prompting (Wei et al., 2022; Kojima et al., 2022; Shi et al., 2022) and rationale engineering (Fu et al., 2023; Zhou et al., 2022), have been proposed to further elicit and enhance the reasoning abilities of LLMs. However, despite these advancements, LLMs still struggle with complex reasoning tasks, particularly in the domain of scientific problem-solving (Chen et al., 2023; Wang et al., 2024; Ma et al., 2024). LLMs often struggle to provide reliable and consistent answers to intricate scientific questions (Wang et al., 2024), necessitating the development of novel approaches to improve their reasoning capabilities in this context.
Multi-Model Collaboration and Role-Playing Previous studies have explored the benefits of roleplaying in LLMs, demonstrating that assigning distinct roles can lead to more specialized and accurate outputs (Lu et al., 2024; Guan et al., 2024; Tao et al., 2024; Bhattacharyya et al., 2024). Additionally, collaborative frameworks where multiple models interact and discuss to reach a consensus have shown promise in improving the robustness of the generated solutions (Du et al., 2024; Lu et al., 2024; Sadler et al., 2024; Mehta et al., 2024; Figueras et al., 2023; Xiong et al., 2023). Considering the complexity and rigor of scientific research, more effective methods are needed to stimulate the optimal intelligence of multi-agent systems.
Large Language Models in Solving Scientific Problems Recent studies have explored the potential of LLMs in scientific problem-solving, including theorem proof (Dong et al., 2023; Song et al., 2024), hypothesis generation (Qi et al., 2023; Yang et al., 2024) and scientific discovery (Boiko et al., 2023; AI4Science and Quantum, 2023). However, the understanding and reasoning capabilities of LLMs in fundamental STEM(Science, Technology, Engineering, and Mathematics) domains remain underexplored (Wang et al., 2024; Ma et al., 2024). While LLMs exhibit impressive performance on high-level scientific tasks, their ability to grasp complex scientific concepts, engage in rigorous logical reasoning, and provide reliable solutions to domain-specific problems is
still uncertain. These challenges necessitate a more nuanced approach to harnessing the full potential of LLMs.

heading: 3 Method: Simulating Expert Discussions with Multi-agent
text: We propose a novel approach called Simulating Expert Discussions with Multi-agent (SEDM) to enhance the scientific problem-solving capabilities of large language models (LLMs) by simulating expert discussions. The overall framework of our methodology is illustrated in Figure 2. We assign multiple LLMs with domain-specific expert roles and simulate a panel discussion among these experts on a given scientific problem. By analyzing and evaluating the consistency of the LLM experts’ responses, we derive reliable solutions.

heading: 3.1 Role Assignment
text: In our proposed approach, we assign domainspecific expert roles to multiple LLMs to address a given scientific problem within a particular domain. This assignment of expert roles is motivated by the following rationale:
• Fostering Collaboration and Consensus Scientific progress often relies on collaboration and consensus-building among experts within the same domain. By assigning identical roles, we encourage LLMs to engage in simulated collaborative discussions, challenging each other’s assumptions, reconciling differences, and ultimately converging towards a consensus solution.
• Enhancing Reliability through Ensemble Methods Despite being instances of the same LLM architecture, each individual model may exhibit variations in its outputs due to factors such as random initialization, stochastic sampling, or sensitivity to input perturbations. By employing an ensemble of multiple LLMs with identical roles, we can leverage the collective wisdom of the group, mitigating the impact of individual model instabilities and enhancing the overall reliability of the proposed solutions.
• Exploring Diverse Reasoning Paths While sharing the same domain knowledge and expertise, each LLM may explore different reasoning paths and problem-solving strategies
when presented with the same scientific problem. Assigning identical roles allows us to capture and analyze these diverse reasoning paths, potentially uncovering novel insights or alternative approaches that a single LLM might overlook.
Through this approach, we create a simulated panel of domain-specific experts with shared expertise but diverse reasoning perspectives. This setup emulates the real-world dynamics of scientific discourse, where experts from the same field evaluate and build upon each other’s work, ultimately advancing our understanding of complex scientific problems.

heading: 3.2 Expert Discussion Simulation
text: At the heart of our proposed methodology lies the simulation of a panel discussion among multiple LLMs, each assuming the role of a domain-specific expert within the same scientific field. The overall overview of expert discussion simulation phase is shown in Figure 3. This approach aims to leverage the collective knowledge and diverse perspectives of the LLMs to tackle complex scientific problems effectively. The simulation process encompasses the following key steps:
Problem Presentation The initial step involves presenting a well-defined scientific problem or inquiry to the panel of LLMs.
Individual Responses Upon receiving the problem, each LLM, operating within its assigned expert role, generates an independent response. This response is based on the LLM’s knowledge and understanding of the specific sub-discipline or area of specialization it represents. By providing individual responses, the LLMs contribute their unique perspectives and insights to the problemsolving process, mimicking the diversity of opinions often encountered in real-world scientific discussions.
Response Analysis and Comparison Once all the LLMs have provided their individual responses, the next step involves collecting and analyzing these responses for consistency and complementarity. The analysis focuses on identifying areas of agreement and divergence among the LLMs’ perspectives. High levels of agreement among the responses suggest a strong consensus and increased confidence in the proposed solution. Conversely, divergent viewpoints highlight areas that require further exploration, clarification, or synthesis, open-
ing up opportunities for a more comprehensive understanding of the problem.
Collaborative Discussion and Refinement In cases where the initial responses reveal discrepancies or complementary insights, a collaborative discussion phase is initiated. During this phase, the LLMs engage in a simulated dialogue, exchanging their perspectives, challenging assumptions, and working towards reconciling any differences. This discussion process closely resembles the way experts within the same scientific domain would interact and collaborate in real-world settings, fostering a rigorous and iterative refinement of ideas. Through this iterative process of discussion, the LLMs aim to converge towards a more comprehensive and well-supported solution to the scientific problem at hand.
The simulated expert discussion within a specific domain, as outlined above, harnesses the collective knowledge and diverse perspectives of the LLMs to tackle complex scientific problems. By emulating the rigorous process of scientific inquiry, where ideas are scrutinized, refined, and synthesized through critical discourse among experts, our methodology aims to enhance the problem-solving capabilities of LLMs in scientific domains. This approach not only leverages the strengths of individual LLMs but also promotes a collaborative and iterative problem-solving process, ultimately leading to more reliable and comprehensive solutions to scientific challenges.

heading: 3.3 Consistency Evaluation
text: Upon completion of the expert discussion simulation, we conduct a consistency evaluation of the solutions proposed by the multiple LLMs assuming expert roles. This evaluation process is crucial for ensuring the reliability and robustness of the proposed solutions. Specifically, we employ the following strategies:
Consensus Determination When all experts reach a unanimous agreement during the discussion process, we consider that they have achieved consensus on the given problem. In such cases, we directly adopt the solution unanimously agreed upon by the experts as the final result. The attainment of consensus often indicates that the solution has undergone thorough discussion and argumentation, lending it higher credibility and reliability.
Maximum Discussion Round Limit Recognizing that expert discussions in the real world cannot continue indefinitely, we set a maximum number of discussion rounds for the expert deliberations. This limit serves to prevent the discussion process from entering an endless loop while also encouraging the experts to reach consensus or make decisions within a reasonable timeframe.
If the experts fail to reach complete agreement within the maximum number of discussion rounds, we resort to the following strategy: we select the solution proposed by the expert (agent) who most persistently defended their viewpoint throughout the discussion. The rationale behind this strategy is that
the expert who firmly maintains their stance likely possesses a deeper understanding of the problem and has provided more comprehensive arguments, rendering their proposed solution more convincing and reliable.
Discussion Convergence Analysis Although we establish a maximum number of discussion rounds, empirical evidence suggests that expert discussions often converge to consensus or a few primary viewpoints within a relatively small number of rounds. We conduct a convergence analysis of the discussion process, quantifying the frequency of achieving consensus or converging to main viewpoints at different round thresholds. Through extensive case studies, we observe that expert discussions typically reach consensus or converge to primary viewpoints within 2-3 rounds. This finding aligns with real-world expert discussion scenarios, demonstrating the effectiveness and practicality of our approach.
By employing these consistency evaluation strategies, we effectively synthesize the opinions of multiple experts to derive reliable and robust problem solutions. Moreover, by analyzing the convergence properties of the discussion process, we validate the efficacy of our method.

heading: 4 Experiment

heading: 4.1 Experimental Setup
text: Dataset In our experiment, we used the SciBench dataset (Wang et al., 2024), which is a comprehensive benchmark specifically designed to evaluate the scientific problem-solving ability of Large Language Models (LLMs). SciBench covers university level problems in various scientific disciplines, including mathematics, physics, and chemistry. This dataset includes open-ended questions from textbooks and open-ended questions from undergraduate exams, ensuring a rigorous evaluation of LLM’s reasoning and computational skills. In this experiment, we use open-ended questions as testing. This dataset provides a solid foundation for testing and improving LLM’s ability to solve problems in complex scientific environments.
Baseline To establish a baseline for our proposed multi-expert framework, we conducted experiments using two state-of-the-art LLMs: GPT3.5 and GPT-4. Specifically, we utilized the gpt-3.5turbo-0125 version for GPT-3.5 and the gpt-4-turbo version for GPT-4. For each model, we employed two query methods:
• Direct Querying: The scientific problem was directly presented to the LLM without any additional context or examples.
• Few-Shot Learning (Brown et al., 2020): We provided the LLM with a small set of representative examples of scientific problems and their solutions before presenting the target problem. This approach aims to prime the model with relevant context and improve its performance on the specific task.

heading: 4.2 Main Results
text: We evaluate the performance of our proposed Simulating Expert Discussions with Multi-agent (SEDM) approach against two baselines: direct querying of the LLM and few-shot learning. The experiments are conducted across three scientific domains: physics, chemistry, and mathematics. Each domain is further divided into subdomains, such as thermodynamics and classical mechanics for physics, to assess the model’s performance on a diverse range of scientific problems. In the main experiment, we used the setting of 2 experts and 2 discussion rounds.
Table 1 presents the accuracy scores of the models on the test set. The results demonstrate that SEDM significantly outperforms the baselines across most domains and subdomains. For GPT-3.5, SEDM achieves an average accuracy of 33.25%, markedly higher than the 9.59% for Direct response and 9.60% for few-shot learning. Similarly, for GPT-4, SEDM attains an average accuracy of 57.18%, compared to 25.09% for Direct and 21.46% for few-shot.
However, it is important to note that SEDM does not always achieve the highest scores in every subdomain. For instance, in GPT-4’s performance in statistics domain, the direct querying approach slightly outperforms SEDM. This may be attributed to the nature of statistical problems, which are often more standardized and formulaic compared to other subdomains. Many statistical problems can be solved by applying specific formulas or algorithms, which aligns well with the strengths of language models. Consequently, direct querying may be sufficient to handle these relatively standard problems.
Despite these few exceptions, SEDM consistently demonstrates robust performance improvements across the majority of subdomains, highlighting its effectiveness and adaptability in enhancing the problem-solving capabilities of LLMs. It is worth noting that the performance of few-shot learning is comparable to or slightly worse than direct querying. This may be due to the limited
ability of the selected prompt examples to fully capture the diversity of the domain, leading to a decrease in the performance of few-shot learning.
The results also reveal some variation in performance across subdomains. For instance, in physics, the models achieve higher accuracy in fundamental concepts compared to thermodynamics and classical mechanics. This suggests that the complexity and specificity of the subdomain can influence the model’s performance. Nevertheless, SEDM consistently outperforms the baselines in almost all subdomains, demonstrating its robustness and adaptability.

heading: 4.3 Further Analysis
text: Solution Quality In addition to evaluating the accuracy of the models, we also assess the quality of the generated solutions. We randomly sample 100 problems and evaluate the solutions using LLMs and human evaluation based on three criteria: (1) the correctness of the reasoning steps, (2) the clarity of the explanations, and (3) the appropriateness of the mathematical notations and symbols used. Each criterion was rated on a scale of 1 to 5, with 5 being the highest quality.
For the human evaluation, we employed three expert annotators. To ensure reliability, we calculated the inter-annotator agreement using Fleiss’ kappa (Fleiss, 1971) for each of the three criteria:
• Correctness of reasoning steps: κ = 0.71
• Clarity of explanations: κ = 0.62
• Appropriateness of mathematical notations and symbols: κ = 0.55
The overall average kappa value was 0.63, indicating substantial agreement among the annotators.
The detailed prompts for LLM evaluation and the specific guidelines for human evaluation are provided in Appendix B. Table 2 presents the average quality scores for solutions from GPT-4. Compared to baseline, SEDM consistently achieves higher quality scores in both LLM and human evaluations. The solutions generated by SEDM demonstrate clearer reasoning steps, more coherent explanations, and more precise use of mathematical notations. This suggests that the multi-expert discussion framework not only improves the accuracy of the solutions but also enhances their overall quality and readability.
Number of Experts We investigate the impact of the number of experts in the panel on the performance of SEDM. We vary the number of experts from 2 to 5 and evaluate the accuracy of the generated solutions.Figure 4 shows the relationship between the number of experts and the average accuracy of GPT-3.5 and GPT-4. The results reveal that increasing the number of experts generally leads to higher accuracy. However, the performance gains diminish as the number of experts exceeds 4. This suggests that a panel of 2-4 experts strikes a balance between performance improvement and computational efficiency. Having too many experts may introduce redundancy and increase the computational overhead without significant performance benefits.
Number of Discussion Rounds We also investigate the impact of the number of discussion rounds on the performance of SEDM. We conducte experiments with varying numbers of discussion rounds, ranging from 1 to 5, and measure the accuracy of the generated solutions, as illustrated in Figure 5. The results indicate that increasing the number of discussion rounds generally improves the accuracy, but the performance gains plateau after 3 rounds. This suggests that 2-4 discussion rounds provide a good trade-off between performance and efficiency.
Ablation Study The ablation study results presented in Table 3 demonstrate the effectiveness of each component in our proposed SEDM framework. By comparing the performance of the full SEDM framework with its variants, we can gain insights into the contributions of the expert role assignment and the expert discussion components.
When the expert role assignment is removed, the performance of both GPT-3.5 and GPT-4 drops sig-
nificantly. GPT-3.5 achieves only 11 out of 100 correctly answered questions, while GPT-4 manages to answer 42 out of 100 questions correctly. This substantial decrease in performance highlights the importance of assigning domain-specific expert roles to the LLMs, as it enables them to provide more accurate and reliable responses to scientific inquiries.
Similarly, the removal of the expert discussion component also leads to a notable decline in performance. GPT-3.5 correctly answers 17 out of 100 questions, and GPT-4 achieves 55 out of 100 correct answers. This finding suggests that the collaborative discussion among the LLMs plays a crucial role in reaching a consensus and improving the overall accuracy of the system.
The ablation study provides strong evidence for the effectiveness of our proposed SEDM framework. By assigning domain-specific expert roles to LLMs and facilitating collaborative discussions among them, we can significantly enhance their performance in addressing complex scientific questions. This finding underscores the potential of our approach to advance the application of AI in scientific research, offering a more reliable and trustworthy solution for tackling scientific problems.

heading: 5 Conclusion
text: In this paper, we have introduced a novel approach called Simulating Expert Discussions with Multi-agent (SEDM) to enhance the scientific problem-solving capabilities of LLMs. By assigning domain-specific expert roles to multiple LLMs and simulating a panel discussion, our method leverages the collective knowledge and diverse per-
spectives of these models to tackle complex scientific problems effectively.
The proposed SEDM framework represents a significant step forward in harnessing the potential of LLMs for scientific problem-solving. By simulating expert discussions and leveraging the collective intelligence of multiple models, we can enhance the accuracy, reliability, and robustness of LLM-generated solutions. This approach opens up new avenues for applying artificial intelligence in scientific research, enabling more effective and trustworthy problem-solving.

heading: 6 Limitations
text: While the Simulating Expert Discussions with Multi-agent (SEDM) approach has shown promise in enhancing the scientific problem-solving capabilities of LLMs, several limitations warrant further investigation.
Firstly, the current study is limited to a subset of scientific domains, namely physics, chemistry, and mathematics. Future research should explore the generalizability of SEDM to a broader range of disciplines to assess its adaptability and effectiveness across diverse problem types and domain-specific challenges.
Secondly, the current implementation of SEDM employs fixed LLMs assuming expert roles within a specific domain. Although effective, this approach may not fully capture the complexity of real-world scientific collaborations. Future work could investigate more dynamic role assignment strategies, allowing for the inclusion of interdisciplinary experts to enrich discussions.

heading: Acknowledgments
text: The authors thank all the anonymous reviewers for their valuable comments and constructive feedback. The authors acknowledge financial support from the National Social Science Fund of China (No. 23BTQ102). Xiaoqiu Le is the corresponding author.

heading: A Experiment Prompts
text: The prompts employed in this work for the Direct querying, few-shot learning, and Simulating Expert Discussions with Multi-agent (SEDM) are illustrated in Figures 6, 7, and 8.

heading: B Evaluation Criteria and Prompts for Solution Quality Assessment
text: The prompts used for LLM evaluation and the guidelines provided for human evaluation of solution quality are presented in Figure 9 and Table 4 respectively. These criteria focus on assessing the correctness of reasoning steps, clarity of explanations, and appropriate use of mathematical notations.
Criterion 1 (Poor) 2 (Fair) 3 (Good) 4 (Very Good) 5 (Excellent) Correctness of Reasoning Steps Most steps are incorrect or missing Several major errors in reasoning
Minor errors in reasoning, but overall approach is correct Reasoning is correct with very minor oversights All reasoning steps are perfectly correct and complete
Clarity of Explanations Explanations are confusing or absent Explanations are unclear and difficult to follow Explanations are mostly clear but some points are ambiguous Explanations are clear with minor areas for improvement Explanations are exceptionally clear, concise, and easy to understand Appropriateness of Mathematical Notations and Symbols Incorrect or missing notations and symbols throughout Several major errors in notation and symbol usage
Minor errors in notation and symbol usage, but generally appropriate Notations and symbols are correct with very minor inconsistencies All mathematical notations and symbols are perfectly appropriate and consistently used
Table 4: Human Evaluation Guidelines for Scientific Problem Solutions.

heading: Future Work
text: Future research should explore the generalizability of SEDM to a broader range of disciplines to assess its adaptability and effectiveness across diverse problem types and domain-specific challenges. Secondly, the current implementation of SEDM employs fixed LLMs assuming expert roles within a specific domain. Although effective, this approach may not fully capture the complexity of real-world scientific collaborations