Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation

Authors: biqing  qi
kaiyan  zhang
kai  tian
haoxiang  li
zhang-ren  chen
sihang  zeng
ermo  hua
jin-fang  hu
bowen  zhou
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-19 15:08:59

Abstract: The rapid growth of biomedical knowledge has outpaced our ability to efficiently extract insights and generate novel hypotheses.
Large language models (LLMs) have emerged as a promising tool to revolutionize knowledge interaction and potentially accelerate biomedical discovery. In this paper, we present a comprehensive evaluation of LLMs as biomedical hypothesis generators. We construct a dataset of background-hypothesis pairs from biomedical literature, carefully partitioned into training, seen, and unseen test sets based on publication date to mitigate data contamination. Using this dataset, we assess the hypothesis generation capabilities of top-tier instructed models in zero-shot, few-shot, and fine-tuning settings. To enhance the exploration of uncertainty, a crucial aspect of scientific discovery, we incorporate tool use and multi-agent interactions in our evaluation framework. Furthermore, we propose four novel metrics grounded in extensive literature review to evaluate the quality of generated hypotheses, considering both LLM-based and human assessments. Our experiments yield two key findings:  1) LLMs can generate novel and validated hypotheses, even when tested on literature unseen during training, and 2) Increasing uncertainty through multi-agent interactions and tool use can facilitate diverse candidate generation and improve zero-shot hypothesis generation performance. However, we also observe that the integration of additional knowledge through few-shot learning and tool use may not always lead to performance gains, highlighting the need for careful consideration of the type and scope of external knowledge incorporated. These findings underscore the potential of LLMs as powerful aids in biomedical hypothesis generation and provide valuable insights to guide further research in this area.

Main paper: 
 introduction: 
 Biomedical research is a driving force behind advancements in human health and well-being. However, the exponential growth of biomedical data and literature has made it increasingly difficult for researchers to keep pace with the latest discoveries and generate novel hypotheses. Large language models (LLMs) (Touvron et al., 2023a; b; Jiang et al., 2024) have emerged as a promising solution to this challenge, offering the potential to revolutionize the way we interact with and discover knowledge in the biomedical domain (Wang et al., 2023a; Taylor et al., 2022) . At the core of the knowledge discovery process is the formulation of sound hypotheses (Zhong et al., 2023; Boiko et al., 2023) . Currently, both ChatGPT, GPT-4, 
preliminary: 
 Experimenal Loop of Scientific Discovery We execute the automated data partitioning pipeline, using publication dates as the basis, to mitigate the risk of data contamination.In this section, we delineate the problem of hypothesis generation. Subsequently, we elucidate the process for developing biomedical-specific datasets. We then analyze the constructed datasets, examining aspects of data contamination and the semantic distribution across different data settings.As shown in Figure 2a , hypothesis generation typically transpires following an in-depth analysis of literature and detailed examination of specific phenomena, playing a pivotal role in the scientific discovery process. In order to improve the assessment of the hypothesis proposition capabilities of LLMs, we formalize this process as a text completion task. Given dataset D, an instruction I, and text pairs (X i , Y i ) n i=1 containing background knowledge and corresponding hypotheses extracted from medical papers, our objective is to assess model M by having it generate hypotheses based on the task instruction and background knowledge, i.e., M(I, X i ) = Y i , for each i ∈ 1, ..., n. The objective function is formulated as: y * = arg max y 1 ,...,y n n ∏ t=1 P(y t |y 1 , . . . , y t-1 , I, X).Existing LLMs, including ChatGPT and Llama, encounter challenges in retracing their training data, complicating efforts to ensure non-overlap with the test set. To rigorously assess the hypothesis generation capability of LLMs, we initially create evaluation datasets based on literature, taking into account the publication date.Unlike previous studies that assess data contamination through the correlation between question entropy and accuracy (Hendrycks et al., 2020; Wei et al., 2023b; Zhou et al., 2023) , our method directly controls visibility by considering both the training date and the publication date. This method ensures that the test data remains non-visible, addressing a limitation overlooked in earlier methods. As depicted in Figure 2b , the year 2023 has been established as the cut-off point, coinciding with the publication date of the majority of top-tier LLMs. The training dataset includes literature published before January 2023, while the test dataset contains literature after January 2023. This arrangement creates pairs of data with background knowledge and corresponding hypothesis proposals. We strictly follow the standard pipeline as outlined in Self-Instruct (Wang et al., 2022) for our data generation process: 1) Compose the paper set based on the topic and content of the literature. 2) Utilize ChatGPT and GPT-4 to summarize the literature knowledge. 3) Generate background knowledge-assume pairs. 4) Filter low-quality data by publishers. 5) Split the dataset according to publication time.While this approach allows for efficient dataset creation, we acknowledge that the accuracy of ChatGPT's extractions and any biases introduced in the process warrant further evaluation. Manual annotation of a subset of the data could help assess the quality and reliability of the automated extraction.Adhering to the above pipeline outlined, we ultimately acquired two distinct types of datasets. 1) Seen dataset This dataset comprises 2700 background and hypothesis pairs sourced from literature published before January 2023. This dataset was partitioned into training (2500) and validation ( 200) subsets (as well as seen test set). It is consistent with the corpus that the LLMs have been exposed to. 2) Unseen dataset The unseen dataset consists of 200 pairs extracted from papers published in August 2023, which the LLMs have not encountered during training and are used for testing purposes.More detailed information with respect to publication data and topic distributions of constructed dataset can be seen in Appendix C.1. 
can llms truly generate zero-shot hypotheses?: 
 In this section, we conduct a thorough evaluation of premier LLMs across a range of experimental settings to determine their ability to effectively generate hypotheses and analyze influenced factors.Initially, we present the models being evaluated, outline the experimental settings, and describe the metrics used for evaluation.Models For a fair comparison, we exclusively evaluate LLMs trained on corpora before March 2023 to avoid data contamination. We consider three categories of models: 1) APIbased LLMs: this is mainly ChatGPT. 2) General domain instructed LLMs: These models consist of open-source models that have undergone fine-tuning based on Llama using general domain instructions. We primarily choose the top-tier models based on rankings before Setember 2023 in the Alpaca Eval Leaderboard1 . 3) Specific domain instructed LLMs: These include PMC-LLaMA (Wu et al., 2023) , and MedAlpaca (Han et al., 2023) , which are trained on a variety of sources in medicine domain. We summarize the training data and publication dates for each models in Appendix C.2.To ensure a consistent output format across different models, we create prompts in two formats: zero-shot and few-shot examples. We adopt a 5-shot format, selecting examples from the training set before January 2023 using both randomly sampled and similarity retrieval methods. To assess the hypothesis generation capability beyond zero-shot, we identify the top-performing open-source models through prompting-based evaluation. Finally, we proceed to fine-tune WizardLM-13B-V1.2 with the background and hypothesis pairs for further comparison. We provide more details in Appendix F.Given the inherent uncertainty beyond established ground truth in hypothesis generation, we undertake evaluations with and without predefined golden hypotheses. 1) With ground truth: In evaluations with golden hypotheses, we employ standard text generation metrics, including BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) , to assess word overlap between the generated outputs and the ground truth. 2) Without ground truth: Considering the vastness of the hypothesis space, renders it difficult to comprehensively assess the quality of generated hypotheses using word overlap metrics alone. To provide a more comprehensive evaluation of the generated hypotheses from multiple facets, we devise four metrics for evaluation (Zhong et al., 2023) :• Novelty: Does the hypothesis introduce new information or perspectives?• Relevance: How closely is the hypothesis related to the topic or question?• Significance: What impact does the hypothesis have on understanding or addressing the problem?• Verifiability: Can the hypothesis be tested using existing methods or data?Furthermore, inspired by recent research that highlights LLMs as proficient annotators (Gilardi et al., 2023; Liu et al., 2023) , demonstrating a strong correlation with human ratings, we employ GPT-4 for automated evaluation, where both the generated hypotheses and the provided background are evaluated across these aspects. The scoring scale ranges from 0 to 3, where a higher score indicates superior results. Additionally, we solicit GPT-4 to furnish a step-by-step explanation to substantiate the assigned score. We also conduct human evaluation for the top-tier models identified in the evaluation of GPT-4 in Section 3.4.Figure 3 : This figure displays the BLEU scores on both seen and unseen datasets.This section presents the results of hypothesis generation across various models in both zero-shot and few-shot settings. We primarily analyze the results from two perspectives: the impact of the zero-shot setting and the influence of introducing external knowledge on hypothesis generation.The results presented in Figure 3 and 4 demonstrate the significant impact of zero-shot settings in improving hypothesis generation, particularly in terms of fostering high novelty.Detailed results are provided in Appendix D.1. In this section, we primarily analyze these results from two critical perspectives as outlined below:Zero-shot vs. Few-shot Figure 4 demonstrates that nearly all models, particularly WizardLM series models, and Openchat-v3.2-super, show enhanced novelty capabilities in a zero-shot setting. Concurrently, these models also demonstrate superior verifiability when presented with few-shot literature examples. This indicates a trade-off in the hypothesis generation capacity of LLMs, necessitating careful consideration of specific constraints. Notably, there are divergent results concerning medical domain adaptation in LLMs, which will be further discussed in the subsequent Section 3.2.2.Seen test-set vs. Unseen test-set Despite the inclusion of literature published before 2023 in the pre-training corpus of most LLMs, we have delineated "seen" and "unseen" test sets, with the "unseen" test set being considered for zero-shot analysis. Generally, LLMs are likely to show enhanced performance on the "seen" test set, potentially due to the memorization of knowledge acquired during training, resulting in a superior performance compared to the "unseen" test set. Contrarily, our findings presented in Figure 3 reveal that LLMs tend to exhibit better performance on the "unseen" test set. We hypothesize that the intricacies involved in hypothesis generation may impede LLMs' ability to effectively utilize the parameterized "dark knowledge". Figure 5 : This figure elucidates the correlation between uncertainty and evaluation scores for all models, encompassing both zero-shot and few-shot settings, and incorporating both sampled and similarity retrieval few-shot prompts.We further extend analysis to the influence of external knowledge, including few-shot examples, domain adaptation, and instruction tuning. The findings indicate that these factors do not consistently enhance performance across all metrics.Few-Shot Examples Enhance Verifiability but Decrease Novelty. Regarding word overlap metrics, including BLEU and ROUGE, most models, especially gpt-3.5-turbo and WizardLM series models, show improved performance when provided with in-context examples as shown in Figure 3 . However, it's important to note that these few-shot prompts significantly increase verifiability while simultaneously leading to lower novelty compared to zero-shot results as shown in Figure 4 . We also conduct an analysis of the influence of fine-tuning for biomedical domain adaptation on hypothesis generation. The results obtained from MedAplaca and PMC-LLaMA indicate that domain adaptation can significantly improve word overlap performance. Evaluation derived from GPT-4 imply that domain adaptation improves the capability of LLMs in utilizing few-shot, particularly with related literature, while diminishing their zero-shot ability.In addition to domain adaptation, we also examine the effect of direct fine-tuning of LLMs on specially constructed instruction training sets. The findings reveal that instruction tuning enhances the capacity of LLMs for knowledge retention, leading to improved BLEU and ROUGE scores and heightened verifiability, especially when contrasted with models without fine-tuning. However, this enhanced memorization concurrently diminishes the model's capability to generate novel and significant hypotheses.C: Design Experiments for How t Published as a conference paper at COLM 2024Drawing from the findings in zero-shot settings and the impact of external knowledge, we identify uncertainty as the key differentiator among these scenarios. The few-shot setting tends to enhance the creative capacity of LLMs, yielding content with higher generalization and diversity. In terms of external knowledge, parameterized knowledge and judiciously selected samples narrow the candidate space, thereby reducing the uncertainty of generated content but enhancing verifiability. Conversely, randomly sampled few-shot examples introduce uncertainty. There are many methods for computing uncertainty, including those based on entropy, semantics, and consistency (Kuhn et al.; Xiong et al.) . However, these methods often entail a high computational cost, primarily due to the requirement for multiple sampling iterations, rendering them impractical for our experiments. Given the pioneering nature of our research in applying Large Language Models (LLMs) to knowledge discovery, and to maintain manageability within the scope of this paper, we have chosen to use SelfBLEU (Alihosseini et al., 2019) to evaluate the internal uncertainty of all generated hypotheses. The SelfBLEU scores serve as indicators of uncertainty in the hypotheses generated by LLMs, where lower scores signify greater uncertainty and diversity.As depicted in Figure 5 , we examine the relationship between uncertainty and detailed evaluation scores of various models under different settings, including zero-shot and fewshot settings (the latter comprising sampled and retrieved examples, denoted as 5-shot and 5-shot*, respectively). Our findings reveal that hypotheses generated in the zero-shot setting exhibit higher uncertainty and novelty scores, as illustrated in Figure 5a . Conversely, few-shot settings with augmented hypotheses demonstrate lower uncertainty yet higher verifiability, as indicated in Figure 5b . These observations offer valuable insights for further research into the balance between novelty and verifiability in hypothesis proposing. Motivated by these findings, we propose a LLM-based multi-agent framework to delve into uncertainty exploration involving tools and multi-agents collaboration in Section 4.In this section, we conduct human evaluation to assess the coherence of hypotheses generated by LLMs, comparing these with GPT-4 evaluation scores to guide further automated evaluations.We apply the four metrics from Section 3.1 (novelty, relevance, significance, and verifiability) for manual evaluation and comparison with GPT-4's assessments, where each metric is scored from 0 to 3. Due to cost constraints, human evaluation focuses on the highest-performing LLM based on automatic metrics and GPT-4 evaluations. The selected models and prompts are listed in Table 1 . 100 examples from the unseen test set were evaluated by three experts in the biomedical field.The results in Table 1 show a strong correlation (Pearson and Spearman coefficients above 0.7) between human and GPT-4 evaluations, reinforcing the potential of LLMs in hypothesis evaluation. For further insights, we analyze the correlation between word overlap scores and manual scores in the Appendix D. A case study illustrating various model-generated hypotheses and ChatGPT evaluations is available in Appendix E. Notably, we have collaborated with experts in medicine and biology and adopted a 5% sampling rate for expert evaluation. Given the constraints of manpower and costs, a full-scale evaluation was not feasible. However, we believe that a 5% sampling rate provides a statistically representative overview and effectively reflects the overall quality of the data. We appreciate your understanding and support in this matter.Subsequently, we present the role design and the tools use in this context. Finally, we present preliminary validated results of the multi-agent system on the constructed dataset. 
multi-agent framework: 
 Inspired by the discovery process in Figure 2a , we introduce a comprehensive framework tailored for hypothesis formulation. This framework encapsulates a multi-agent system where each agent assumes a distinct role, mirroring the collaborative nature of scientific endeavors. Through a symbiotic and iterative process, these agents collaborate to craft hypotheses that are not only grounded in existing knowledge but also pave the way for novel insights. By emulating the essence of scientific discovery, our framework strives to produce hypotheses that are both innovative and scientifically robust. As depicted in Figure 6a , we partition the framework into five components, encompassing four automated agents and the option for human involvement within the loop.In our proposed multi-agent framework, each component plays a distinct and pivotal role. The Analyst serves as the foundation, meticulously extracting and defining core elements from the research background. Its primary objective is to interpret the literature, distilling it into keywords or topics that subsequently guide the Engineer's search efforts. The Engineer, leveraging these keywords, embarks on a mission to retrieve and organize pertinent information. They meticulously plan and execute detailed searches, ensuring that the findings are compiled in a structured manner. This organized materials then lands in the domain of the Scientist, whose objective is to weave together the Engineer's findings with the original research background. Through careful interpretation, the Scientist crafts a hypothesis that is both grounded in existing knowledge and offers a fresh perspective. However, before this hypothesis is finalized, it undergoes scrutiny by the Critic. The Critic's role is paramount in ensuring the hypothesis's robustness, coherence, and novelty. They evaluate the hypothesis against the backdrop of the research background, ensuring it stands up to academic rigor. Feedback from the Critic, if necessary, loops back to refine the hypothesis or prompts the Analyst for further insights, creating a cyclical and iterative process of refinement.Tool Use To explore external knowledge beyond the inherent dark knowledge within LLMs, we integrate the Engineer agent with search engines , mainly PubMed2 . Similarly, to control the visibility of the unseen test dataset, we filter and exclude literature published after January 2023 from the search results. We carry out tool use experiments using ReAct (Yao et al., 2023b) and OpenAI function calling. ReAct is a method that extends the concept of Chain of Thought (CoT) (Wei et al., 2022) , involving thinking before taking action and subsequently making observations based on feedback from the environment. In our experiments, we instruct the LLMs to initially contemplate the provided background information and then make a decision regarding whether to utilize tools. Upon receiving feedback from the tools, the LLMs are expected to identify supporting evidence in the results or potentially make further tool requests. The LLMs are responsible for concluding the hypothesis generation process and summarizing the hypotheses independently.Our primary focus is to investigate the impact of tool use and multi-agent collaboration on hypothesis generation. We present the experimental results in Figure 6b . Based on the results, we summarize our findings from the following two perspectives:Results of Tool Use Based on our results, we observe that tool use has minimal impact on improving the hypothesis generation ability of LLMs. This observation aligns with the findings presented in Section 3.2.2 regarding the analysis of external knowledge. Notably, the ReAct-based method performs slightly worse than OpenAI function calling. It is also evident that LLMs struggle to identify useful information and exhibit weaknesses in the thought-action-observation process, even when utilizing the official interface from OpenAI.Hypothesis generation is a demanding task where the balance between certainty and uncertainty plays a crucial role and needs further exploration.In addition to tool use, our findings suggest that the division of labor and interaction among multi-agents can significantly enhance the model's capability to propose hypotheses by introducing uncertainty. This mirrors the dynamics of real-world scientific research, where hypotheses are formulated through iterative discussions and refutations. Additionally, it is worth noting that tool use can further enhance the performance of the multi-agent framework. 
conclusion: 
 This study presents a comprehensive evaluation of large language models (LLMs) as biomedical hypothesis generators, focusing on their zero-shot generalization ability. We constructed a novel biomedical corpus to assess the hypothesis generation capabilities of LLMs, revealing their remarkable ability to generate novel and validated hypotheses even when tested on unseen literature. We introduced a multi-agent framework that leverages LLMs to enhance collaborative analysis in the hypothesis generation process and developed multidimensional metrics to evaluate the quality of generated hypotheses. Our work contributes to the growing research on the application of LLMs in scientific discovery, particularly in the biomedical domain. Future research could explore the integration of domain-specific knowledge, incorporation of diverse data sources, and development of user-friendly interfaces to further enhance the performance and adoption of LLMs in biomedical hypothesis generation. Our study highlights the immense potential of LLMs as catalysts for biomedical discovery, offering a promising avenue for accelerating the pace of scientific progress. 
limitations and future works: 
 Our research reveals the potential of Large Language Models (LLMs) to generate new hypotheses, marking a significant positive finding. However, we must acknowledge certain limitations. Firstly, a thorough investigation is required into the impact of factual hallucinations inherent in LLMs on hypothesis formulation. Additionally, evaluating and analyzing knowledge discovery in LLMs demands in-depth consideration from various perspectives, necessitating further exploration.Currently, our focus is on the hypothesis generation stage, utilizing tools such as Web search and PubMed. Introducing more specialized tools into the knowledge discovery cycle could prove invaluable. In future work, we plan to integrate bioinformatics analysis tools (Zhang et al., 2024) to enhance our research efforts.Moving forward, we intend to incorporate knowledge graphs to aid in generating higherquality hypotheses. Additionally, we will explore the inclusion of more dynamic tools, such as multi-tool collaborations, to further improve the performance of our LLM-based models.To significantly enhance the hypothesis generation capabilities of our models, we aim to optimize strategies that introduce uncertainty via reinforcement learning. This approach will help us address the limitations and unlock the full potential of LLMs in the knowledge discovery process. 
 
 Appendix: Scientific discovery involves key components, each crucial for advancing our understanding of the natural world: data analysis, hypothesis formulation, experiment design, execution, and observation and reflection Jain et al. (2023) as shown in Figure 2a .1) Data Analysis: Foundational in the scientific process, it entails collecting and examining data to discern patterns and anomalies, extracting insights through statistical techniques and visualization. It initiates scientific inquiry, guiding further exploration. 2) Generating Hypotheses: Among these components, hypothesis formulation is pivotal. It entails crafting informed guesses to explain observed phenomena. Hypotheses serve as guiding frameworks, directing and focusing research by articulating specific relationships and outcomes for experimental exploration. 3) Experiment Design: Once a hypothesis is set, designing experiments becomes essential to rigorously test its validity. This involves defining variables, specifying control groups, and outlining methods and procedures. Well-designed experiments ensure objective hypothesis testing and yield meaningful, informative results. 4) Experiment Execution: Meticulous execution of designed experiments and data collection are critical. Researchers adhere precisely to experimental protocols, recording observations, measurements, and unexpected findings. Integrity in execution ensures reliable, reproducible outcomes. 5) Accumulating Observations: After experiments, scientists engage in observation and reflection. They analyze collected data to determine if results support or refute the initial hypothesis. If unsupported, hypotheses may be revised or new ones formulated based on findings. Observation and reflection permit iterative refinement of scientific understanding.Hypothesis Pioneers Pathways: Guiding Knowledge Discovery. While all components are essential, hypothesis formulation holds a unique position. It drives the scientific endeavor, guiding research question selection, experiment design, and data analysis. Well-constructed hypotheses not only provide direction but also lay the foundation for meaningful scientific discoveries by posing rigorously testable questions. Hypothesis formulation serves as the intellectual anchor steering scientific investigation and ultimately advancing knowledge.B.1 Data-Driven Scientific Discovery Data-driven knowledge discovery research within LLM is relatively limited, with the current focus primarily on dataset construction and task-driven design. In this context, (Zhong et al., 2023) proposed a dataset for investigating the transition from goals to discoveries. However, it should be noted that accurate discoveries within this dataset are not recent. (Wang et al., 2023b) introduced a method for automatically collecting and constructing publication data, along with a proposal for a hypothesis generation approach in the natural language processing (NLP) domain. However, this method requires prior human knowledge, explicit context, and is not an automated process. It's worth noting that their data was constructed from literature before 2021 from the ACL collection, implying that the information may already exist in open-source models like chatGPT and LLAMA. Furthermore, (Wang et al., 2023b) focused on integrating computational tools in the field of chemistry, primarily analyzing the capabilities of LLMs in using integrated tools but neglecting the ability for zero-shot generalization in chemistry reactions. (Boiko et al., 2023) delved more into the abilities of LLMs regarding planning and conducting experiments but did not consider proposing new hypotheses. (Yang et al., 2023b) introduced a new task for open-domain hypothesis induction and created a dataset comprising 50 articles from social science journals. Additionally, they developed a multi-module system for exploring feedback mechanisms. However, all of the above-mentioned literature lacks strict guarantees on the visibility of test data to models, thereby limiting our exploration of the zero-shot generalization capability of LLMs through learning from existing knowledge to propose new hypothesis. Unlike existing works, we have designed datasets based on publication dates, which can easily ensure a strict independence between test data and LLMs.Large language models demonstrate exceptional capabilities in tasks such as question answering, program coding, and instruction following. However, they still confront significant challenges related to factual hallucination (Zhang et al., 2023; Rawte et al., 2023) , knowledge outdated (Cao et al., 2023) , and interactions with real-world. To address these challenges, recent research has explored enhancing LLMs by incorporating tools such as search engines (Nakano et al., 2022; Parisi et al., 2022) , calculators (Schick et al., 2023b) , code interpreter (Zhu et al., 2023b) , RESTful APIs (Song et al., 2023; Patil et al., 2023) and others. The integration of LLMs with tool use, also known as LLM-driven autonomous agents (LAAs), has attracted substantial public attention. These agents are equipped with reasoning (Wei et al., 2023a; Yao et al., 2023a) , planning (Shen et al., 2023; Valmeekam et al., 2023) , decision-making (Yang et al., 2023a; Kang et al., 2023) , and long-term memory capabilities (Zhu et al., 2023a; Hu et al., 2023) , and they are constructed upon the foundation of LLMs. LAAs can autonomously plan sub-goals for complex tasks, execute actions, obtain feedback from the environment, and adjust their behaviors to adapt (Yao et al., 2023b; Xi et al., 2023a; Shinn et al., 2023) . LAAs have demonstrated significant potential in addressing complex real-world tasks, including software development (Qian et al., 2023; Hong et al., 2023) , drama creation (Maas et al., 2023) , course design (Chen et al., 2023) , chemistry experiments (Bran et al., 2023) and more. Furthermore, multi-agent collaboration plays a significant role in LAA applications, allowing agents to collaborate and interact to solve problems through various role-playing scenarios (Park et al., 2023; Fu et al., 2023; Gong et al., 2023; Li et al., 2023) . To the best of our knowledge, there is still a dearth of exploration regarding the use of agents, particularly multi-agents, for scientific discovery. In this paper, our objective is to undertake a preliminary effort to enhance the hypothesis proposing capability of LLMs by harnessing tools and multiple agents, along with conducting an analysis of influencing factors.Seen Unseen In this section, we delve into further implementation details of our experiments, including information about the constructed dataset and open-source models.We present the publication dates and topic distributions of the various datasets for comparison, as illustrated in Figure 7 , where we utilize Nomic Atlas3 to visualize the topic distribution of abstracts in both the training and test datasets.As depicted in Figure 8a , we embed the background and hypothesis sentences from both the seen and unseen test sets using OpenAI embeddings , followed by dimensionality reduction through t-SNE (Van der Maaten & Hinton, 2008) . The results indicate minimal differences between the two test sets, despite their derivation from distinct time periods.We present the meta-information of the open-source models used in our experiments, as shown in Table 4 . We have gathered data regarding their pre-training, supervised learning corpus, and release dates to ensure the non-visibility of the unseen test data. The models were selected based on their performance ranking on the Alpaca-Eval leaderboard prior to September 2023.2 0 0 0 2 0 0 1 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6  2 0 0 7 2 0 0 8 2 0 0 9 2 0 1 0 2 0 1 1 2 0 1 2 2 0 1 3 2 0 1 4 2 0 1 5 2 0 1 6 2 0 1 7 2 0 1 8 2 0 1 9 2 0 2 0 2 0 2 1 2 0 2 We present more detailed results of diverse LLMs and prompting scenarios on datasets specifically constructed for ChatGPT and GPT-4 in Table 2 and Table 3 , respectively. Additionally, we provide the outcomes of experiments involving multi-agent interactions and tool use in Table 6 . In addition to Figure 3 , Figure 9 presents the ROUGE scores for both seen and unseen datasets.We have included additional results from human evaluations in Table 5 , primarily focusing on correlation scores between word overlap metrics and manual evaluations. Note that we continue to use the same samples used in human evaluation to compute BLEU and ROUGE-L for a fair comparison. We calculate the Pearson and Spearman coefficients between each automatic metric and the average human score. These results reveal that word overlap metrics, such as BLEU and ROUGE-L, exhibit notably lower correlation with manual scores. While BLEU and ROUGE-L may have a high correlation with relevance metrics, they are weak in providing a comprehensive evaluation of the generations. Conversely, evaluations conducted by ChatGPT demonstrate higher correlation with human evaluations, as illustrated in Table 1 . However, there is still a significant need to explore advanced metrics, particularly automated ones, in the context of scientific discovery. V ic u n a -3 3 b -v 1 .3 (0 -s h o t) V ic u n a -3 3 b -v 1 .3 (5 -s h o t) V ic u n a -3 3 b -v 1 .3 (5 -s h o t) * L la m a -2 -7 0 b -c h a t( 0 -s h o t) L la m a -2 -7 0 b -c h a t( 5 -s h o t) L la m a -2 -7 0 b -c h a t( 5 -s h o t) * W iz a rd L M -1 3 B -V 1 .2 (0 -s h o t) W iz a rd L M -1 3 B -V 1 .2 (5 -s h o t) W iz a rd L M -1 3 B -V 1 .2 (5 -s h o t) * W iz a rd L M -7 0 B -V 1 .0 (0 -s h o t) W iz a rd L M -7 0 B -V 1 .0 (5 -s h o t) W iz a rd L M -7 0 B -VIn this section, we present several generated hypotheses from various models and provide examples of the evaluation process, step by step, using ChatGPT.We compare the generated hypotheses of different LLMs selected in human evaluation. The selected medicine literature was published in August 2023 (Chang et al., 2023) , which proposed the power law index as an early marker of myocardial infarction. As shown in Table 7 , some responses like gpt-3.5-turbo (0-shot) propose this hypothesis from zero, based only on the background. This indicates that LLMs have great potential in proposing highly novel and verifiable hypotheses and could play a significant role in scientific discovery. The case study focuses on MI predictive biomarkers and two-stage power-law rheology. GPT-3.5 identifies a "fundamental biomechanical principle," while Llama-2-70b-chat discusses "changes in the levels of collagen and proteoglycans in the extracellular matrix," and WizardLM-70B-V1.0 addresses "the progression of myocardial Table 6 : Results of individual agents and multi-agent systems, both with and without the use of tools, on the unseen test dataset. The results demonstrate that both multi-agent systems and the utilization of tools enhance the ability of LLMs in hypothesis generation. Among the various types of models, both 2 a and 2 b are evaluated with tool use. The difference between them lies in their implementations: ReAct (Yao et al., 2023b) and OpenAI function calling 4 , respectively.(1) The past 25 years have seen significant development in the field of digital electrophysiology, including advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring.(2) Telemonitoring of CIEDs has been shown to reduce time to detection of clinically actionable events and improve patient outcomes.(3)Wearable devices, such as smartwatches, rings, and fitness trackers, have become more accurate and reliable in monitoring cardiac rhythm using photoplethysmography.(1) The implementation of telemonitoring and the use of wearable devices for cardiac rhythm monitoring will continue to increase in the future.(2) Artificial intelligence can be utilized to better identify actionable alerts and improve patient outcomes in telemonitoring of CIEDs.(3) Smartwatches may have the potential to replace CIEDs in certain cases.(1) The advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates.(2) These advancements achieve these outcomes by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans.(3) This hypothesis is supported by studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes.The refined hypothesis captures the essence of the research background and the collaborative insights, providing a logical and scientifically grounded statement.It can now be further evaluated and tested in future research.(Round 1) Analyst: Based on the provided research background, I have identified the following keywords and topics:Keywords:-Digital electrophysiology -Telemonitoring -Cardiac implantable electronic devices (CIEDs) -Wearable devices -Cardiac rhythm monitoring -PhotoplethysmographyTopics:1. Advancements in digital electrophysiology and telemonitoring of CIEDs 2. Benefits of telemonitoring in improving patient outcomes 3. Accuracy and reliability of wearable devices for cardiac rhythm monitoring using photoplethysmography Preliminary insights:-The field of digital electrophysiology has seen significant development in the past 25 years, particularly in the areas of telemonitoring of CIEDs and the use of wearable devices for cardiac rhythm monitoring.-Telemonitoring of CIEDs has been shown to have positive impacts, such as reducing the time to detect clinically actionable events and improving patient outcomes.-Wearable devices, including smartwatches, rings, and fitness trackers, have become more accurate and reliable in monitoring cardiac rhythm using photoplethysmography.Based on these insights, it can be hypothesized that the advancements in telemonitoring and wearable devices have led to improved cardiac rhythm monitoring and patient outcomes. However, further analysis and collaboration with the Scientist and Critic are needed to refine and validate this hypothesis.(Round 2) Scientist: Based on the insights provided by the Analyst and the research background, I propose the following hypothesis:Hypothesis: The advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes and more accurate detection of clinically actionable events.This hypothesis suggests that the development and implementation of telemonitoring technologies for CIEDs, along with the increasing accuracy and reliability of wearable devices using photoplethysmography, have positively impacted patient outcomes by reducing the time to detect clinically actionable events. By continuously monitoring cardiac rhythm, these advancements have the potential to provide timely interventions and improve overall patient care.The hypothesis is grounded in the existing literature, which highlights the benefits of telemonitoring in improving patient outcomes and the increasing accuracy of wearable devices for cardiac rhythm monitoring. However, further collaboration with the Critic is necessary to evaluate the novelty, coherence, and scientific validity of the hypothesis.(Round 3) Critic: The hypothesis proposed by the Scientist is well-aligned with the research background and the insights provided by the Analyst. It effectively captures the essence of the advancements in telemonitoring of CIEDs and the use of wearable devices for cardiac rhythm monitoring. The hypothesis also highlights the potential positive impact on patient outcomes and the detection of clinically actionable events.However, there are a few areas that could be further refined or expanded upon in the hypothesis:1. **Specific outcomes**: The hypothesis mentions ""improved patient outcomes,"" but it would be beneficial to specify the specific outcomes that are expected to improve. For example, are we referring to reduced hospitalizations, improved quality of life, or decreased mortality rates?2. **Mechanisms**: The hypothesis briefly mentions the potential for timely interventions and improved patient care, but it would be helpful to elaborate on the specific mechanisms through which the advancements in telemonitoring and wearable devices are expected to achieve these outcomes. Are there specific interventions or treatments that can be administered based on the data collected from these devices?3. **Scientific validity**: While the hypothesis is grounded in the existing literature, it would be valuable to provide some references or studies that support the claims made in the hypothesis. This would enhance the scientific validity of the hypothesis and demonstrate that it is not just speculative but based on empirical evidence.To refine the hypothesis, I would recommend the Scientist to consider incorporating these suggestions and providing more specific details and references to support the claims made. This will strengthen the coherence and scientific validity of the hypothesis.Analyst, based on the feedback provided, please re-evaluate the research background and provide any additional insights or interpretations that could further refine the hypothesis.(Round 4) Analyst: Based on the feedback from the Critic, I have re-evaluated the research background and identified additional insights:1. **Specific outcomes**: The research background mentions that telemonitoring of CIEDs has been shown to reduce time to detection of clinically actionable events and improve patient outcomes. These outcomes could include early detection of arrhythmias, reduced hospitalizations due to cardiac events, improved quality of life, and decreased mortality rates. It would be beneficial to specify one or more of these specific outcomes in the hypothesis.2. **Mechanisms**: The advancements in telemonitoring and wearable devices can potentially lead to improved patient outcomes through several mechanisms. For example, continuous monitoring of cardiac rhythm can enable early detection of abnormal rhythms, allowing for timely interventions such as medication adjustments or electrical cardioversion. Additionally, the data collected from these devices can provide valuable insights for personalized treatment plans and remote patient management. Including these specific mechanisms in the hypothesis would enhance its coherence.3. **Scientific validity**: To strengthen the scientific validity of the hypothesis, it would be valuable to reference specific studies or research that support the claims made. For example, citing studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events or improving patient outcomes would provide empirical evidence to support the hypothesis.Incorporating these additional insights and suggestions, the refined hypothesis could be:Hypothesis: The advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates. These advancements achieve these outcomes by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans. This hypothesis is supported by studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes.This refined hypothesis takes into account the specific outcomes, mechanisms, and scientific validity, as suggested by the Critic. It provides a more comprehensive and well-supported statement that captures the essence of the research background and the collaborative insights.(Round 5) Scientist: Based on the additional insights and suggestions provided by the Analyst and the Critic, I propose the following refined hypothesis:Hypothesis: The advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates. These advancements achieve these outcomes by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans. This hypothesis is supported by studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes.This refined hypothesis incorporates the specific outcomes mentioned in the research background, such as reduced hospitalizations, improved quality of life, and decreased mortality rates. It also expands on the mechanisms through which these advancements are expected to achieve these outcomes, including early detection of abnormal rhythms, timely interventions, and personalized treatment plans. Additionally, the hypothesis acknowledges the importance of scientific validity by referencing studies that support the claims made.The refined hypothesis is both innovative and scientifically grounded, as it builds upon the existing literature and incorporates the collaborative insights provided by the Analyst and the Critic. It provides a logical and well-supported statement that can be further evaluated and tested.(Round 6) Critic: The refined hypothesis proposed by the Scientist is well-structured and addresses the feedback provided by the Critic. It incorporates specific outcomes, mechanisms, and scientific validity, making it a comprehensive and scientifically robust statement.Final Answer:(1) The advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates.(2) These advancements achieve these outcomes by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans.(3) This hypothesis is supported by studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes.The refined hypothesis captures the essence of the research background and the collaborative insights, providing a logical and scientifically grounded statement. It can now be further evaluated and tested in future research.Table 8 : This table illustrates a case from multi-agent collaboration on hypothesis proposing.We provide evaluation results conducted by ChatGPT in Table 9 . ChatGPT offers a stepby-step explanation for each metric, ultimately providing independent scores in integer format.Background:(1) The past 25 years have seen significant development in the field of digital electrophysiology, including advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring.(2) Telemonitoring of CIEDs has been shown to reduce time to detection of clinically actionable events and improve patient outcomes.(3)Wearable devices, such as smartwatches, rings, and fitness trackers, have become more accurate and reliable in monitoring cardiac rhythm using photoplethysmography.(1) The implementation of telemonitoring and the use of wearable devices for cardiac rhythm monitoring will continue to increase in the future.(2) Artificial intelligence can be utilized to better identify actionable alerts and improve patient outcomes in telemonitoring of CIEDs.(3) Smartwatches may have the potential to replace CIEDs in certain cases.(1) The advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates.(2) These advancements achieve these outcomes by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans.(3) This hypothesis is supported by studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes.The refined hypothesis captures the essence of the research background and the collaborative insights, providing a logical and scientifically grounded statement.It can now be further evaluated and tested in future research.Explanation: The generated scientific hypothesis builds upon the background information by stating that the advancements in telemonitoring of CIEDs and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes. It further suggests that these advancements achieve these outcomes by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans. This hypothesis goes beyond the background by proposing specific mechanisms through which the advancements in telemonitoring and wearable devices contribute to improved patient outcomes. It also mentions the supporting evidence from studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes. Overall, the hypothesis provides new insights and expands upon the existing knowledge in the field. Score: 2Explanation: The generated scientific hypothesis is strongly related to the given background. The background provides information about the advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring. The hypothesis builds upon this information by stating that these advancements have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates. It further explains that these outcomes are achieved by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans. The hypothesis is supported by the background information, which mentions that telemonitoring of CIEDs has been shown to reduce time to detection of clinically actionable events and improve patient outcomes. Overall, the hypothesis is directly related to the background and provides a logical and scientifically grounded statement. Score: 3Explanation: The generated scientific hypothesis builds upon the background information provided and presents a logical and scientifically grounded statement. It suggests that the advancements in telemonitoring of CIEDs and the use of wearable devices for cardiac rhythm monitoring have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates. The hypothesis further suggests that these advancements achieve these outcomes by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans. This hypothesis is supported by the background information that mentions the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes. Overall, the hypothesis provides significant insights beyond the background and presents a testable statement for future research. Score: 3Explanation: The generated scientific hypothesis is highly verifiable based on the given background. The background provides evidence of significant advancements in telemonitoring of cardiac implantable electronic devices (CIEDs) and the use of wearable devices for cardiac rhythm monitoring. It also states that telemonitoring of CIEDs has been shown to reduce time to detection of clinically actionable events and improve patient outcomes. Additionally, it mentions that wearable devices have become more accurate and reliable in monitoring cardiac rhythm.The hypothesis builds upon this background by suggesting that these advancements have led to improved patient outcomes, such as reduced hospitalizations, improved quality of life, and decreased mortality rates. It further proposes that these outcomes are achieved by enabling early detection of abnormal rhythms, facilitating timely interventions, and providing valuable data for personalized treatment plans.The hypothesis is supported by studies that have demonstrated the effectiveness of telemonitoring in reducing time to detection of clinically actionable events and improving patient outcomes. Therefore, the hypothesis is highly verifiable and has a strong likelihood of being confirmed in future research. Score: 3Table 9 : This table illustrates a case for hypothesis evaluation using ChatGPT.In this section, we offer prompts for prompt LLMs for hypothesis generation and for ChatGPT in the evaluation process.F.1 Zero-and Few-shot PromptsWe provide prompts for hypothesis generation under both zero-shot and few-shot settings, which are detailed in Table 10 and Table 11 respectively. Specially, the latter includes two methods to obtain in-context examples: random sampling and similarity retrieval.You are a researcher. You can give novel hypothesis based on your exist knowledge and the given background.Based on the known existing knowledge, generate new conjectures in the following format:(1) xxx (2) xxx (3) xxx Be sure to use English answers (proper nouns need to be marked in English), statements as concise and academic as possible, do not have too much repetitive information, numerical values using the original numbers, be sure to strictly follow the format, the corresponding content output to xxx.Note: Please respond directly to the multiple hypotheses without adding any extra sentences. Now give hypothesis based on the following background: {user input} Table 10 : Prompt for zero-shot hypothesis generation.We present prompts for each role in multi-agent collaboration in Table 12 , and prompts for environment settings in Table 13 .https://tatsu-lab.github.io/alpaca eval/Can agent collaboration enhance LLMs' zero-shot generalization?In this section, we aim to improve LLMs' hypothesis generation capability by leveraging uncertainty with tool use and multi-agent collaboration. We firstly introduce the conceptual multi-agent system for hypothesis generation, drawing inspiration from scientific research.https://pubmed.ncbi.nlm.nih.gov/https://github.com/nomic-ai/nomichttps://openai.com/blog/function-calling-and-other-api-updates 


Decision: This paper explores the capabilities of LLMs as (biomedical) hypothesis generators. There was consensus amongst reviewers that this i novel task which might inspire follow-up work, and the work here is conducted with reasonable rigor (namely the dataset curation and the range of models evaluated). 

There are some concerns regarding the evaluation, but then this is an inherently difficult sort of thing to evaluate empirically, and this in my view should not preclude research on new (hard to evaluate) applications of LLMs.

[comments from PCs] Figures are not readable. It's critical to improve them.

Reviews: review: paper_summary: paper_summary: This work explores the use of large language models (LLMs) in generating hypotheses for biomedical research. It introduces a novel dataset specifically designed to avoid data contamination and assesses these models across various settings, including zero-shot and few-shot scenarios, using several quantitative metrics to evaluate the quality of the generated hypotheses. Methodologically, it integrates 'tool use' and multi-agent interactions to determine how these approaches can enhance the hypothesis generation capabilities of LLMs. The results indicate that LLMs can, to some extent, generate relevant hypotheses, which could potentially accelerate biomedical research and discovery.
strength_weakness: strength_weakness: Overall, the paper could benefit from a more stricter contextualisation and focus, given the challenging nature of the tasks. 
- First, the work would benefit from a more detailed analysis of the types of hypotheses that LLMs can generate from a biomedical perspective, rather than just assessing the general quality of the hypotheses. This would better demonstrate the models' generalisation across various biomedical research scenarios. 
- The discussion about 'dark knowledge' is vaguely defined and poorly explained, leaving readers uncertain about its relevance and impact on hypothesis generation, and the strength of the supporting evidence for such statements.
- Additionally, the discussion on uncertainty and the rationale behind certain evaluation metrics and frameworks, like SelfBLEU and the agent-based framework, is overly general and not well-supported, limiting the reliability of the evaluation process.
- Lastly, a more elaborated analysis that explicitly compares this contribution with existing literature would better contextualize and strengthen the novelty of the proposed approaches.
paper_summary: paper_summary: This paper proposed a novel dataset with rigorous curation, as well as some interesting/new metrics to evaluate/test the quality of the hypothesis of biofield. The authors used both human assessment and GPT-4 assessment in their evaluation. Many interesting discoveries were revealed and discussed, which will inform the community in the long term.
strength_weakness: strength_weakness: 1. Dataset validation and cleaning. Based on Figure 1, it seems that most of the curation is made by GPT-4, and human intervention is not very much involved. This might somehow bring bias and it might be useful to have human annotators to evaluate the error and bias rate.

2. Minor point: qualitative studies might be included to have a better sense of the alignment of human assessment and LLMs, in the cases where they agree and they disagree.
questions: questions: 1. Most of the baselines of Table 1 focus on not most advanced models. There are many other state of the art, for example, Mistral MOE models.  Is there a reason for this choice?
paper_summary: paper_summary: The authors present a comprehensive evaluation of large language models (LLMs) as biomedical hypothesis generators. By constructing a novel temporal dataset of background-hypothesis pairs and employing innovative experimental setups, they provide valuable insights into the capabilities of LLMs in zero-shot and few-shot settings. The authors also propose a tool-augmented/multi-agent framework and multidimensional evaluation metrics, which seem to contribute to a deeper understanding of the factors influencing LLM-based hypothesis generation. Additionally, they explore the role of uncertainty and demonstrate that increasing uncertainty through tool use and multi-agent interactions can enhance the diversity and quality of generated hypotheses.
strength_weakness: strength_weakness: 1. **Design of evaluation metrics for evaluation without ground truth**: The paper lacks sufficient details and validation regarding the design of the four evaluation metrics.
2. **Tool use**: The authors directly employ the tool sets from ReAct and OpenAI to conclude that tool use does not significantly help with hypothesis generation. However, it is expected that general-domain tools would provide limited contributions.
3. **Role of the multi-agent framework**: It may lack ablation studies to evaluate the individual contributions and necessity of each role within the multi-agent framework.
questions: questions: 1. **Protocol and standards for human evaluation**: The authors conduct human evaluations to assess the coherence of hypotheses generated by LLMs and compare them with GPT-4 evaluation scores. However, the paper would benefit from providing more details regarding the protocol and standards followed during the human evaluation process. This additional information would help readers better understand the methodology and ensure the reproducibility of the human evaluation results.

2. **Analysis of the case study**: The case study section presents a list of tables containing generated hypotheses from various models. While this information is valuable, the lack of detailed analysis accompanying the tables may confuse readers. To improve clarity and enhance the readers' understanding, the authors should provide a more thorough analysis of the case study results.
paper_summary: paper_summary: The paper evaluates the large language model's ability as a biomedical hypothesis generator. It starts by constructing background hypothesis pairs from biomedical literature and splitting the test sets based on publication date to mitigate data leakage. The paper then assesses the hypothesis generation capabilities of the state-of-the-art models in the zero-shot, few-shot, and fine-tuning settings. It conducts both automatic and human evaluations. The experiment shows several impacts of external knowledge. The paper also builds a multi-agent framework for hypothesis generation and refinement. Specifically, the paper constructs four roles including analyst, engineer, scientist, and critic. The paper also tests tool learning.
strength_weakness: strength_weakness: 1. Some details of the paper are unclear. It seems that the assumption and the background pairs are generated by GPT4. While the authors admit that manual annotation of a subset could help reflect the quality of the dataset, the paper fails to provide any manual evaluation of the constructed data. Additionally, the dataset seems to be a little bit small(with 200 pairs for testing). What does the 5shot* mean? Does it mean similarity retrieval-based few-shot? The fonts in Figure 5 are too small. The title of section 3 needs to be updated since it also analyzes few-shot and fine-tuning settings. 
2. While the paper also evaluates the performance of multi-agent and tool use, the analysis part is superficial compared to section 3. The paper claims that adding a tool can further enhance the multi-agent. However, its performance on relevance and verifiability dropped. Additionally, the average score is also worse.
3. The external knowledge part focuses on the few-shot part. It might be more useful to include some external knowledge graph in the experiment to further strengthen this section's analysis.
questions: questions: 1. What are the sources of the dataset?
2. What are the quality of the dataset?

Comments: title: reponse
Response to authors' rebuttal
Thank you for your response
comment: thanks authors for the response, and I remain positive about the paper.
Thanks for your clarification! You have addressed all of my concerns. I have read it carefully and decided to keep my positive score.
I acknowledge that I have read the responses by the authors and I will keep my positive score.