Hypothesis Generation with Large Language Models
Authors: Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, Chenhao Tan


Reviewer paper summaries: This paper focuses on hypothesis generation derived from data and introduces HypoGeniC, a method that utilizes LLMs to generate hypotheses aimed at uncovering unknown knowledge. The process begins by crafting initial hypotheses from a restricted set of examples, then progressively hones them through iterative updates to improve precision. Inspired by the multi-armed bandit problem, it employs a reward function to balance the trade-off between exploitation and exploration. Experiments on synthetic data and three real-world datasets demonstrate the effectiveness of the proposed method compared to few-shot prompting.
Summary of strengths: - This paper addresses hypothesis generation from data, which is an intriguing area of study.
- The core of the proposed method is the hypothesis generation algorithm, which follows the principles of the multi-armed bandit problem to design a reward function that balances exploitation and exploration.
Summary of weaknesses: - Although the authors state, “The primary interest lies in the quality of the hypothesis,” they mainly report the accuracy of different models. What about the quality of hypotheses generated by different models, especially RoBERTa and Llama-2-7B, which show high accuracy?
- The motivation behind using few-shot and in-context learning baselines is unclear. Why not compare with fine-tuned LLMs like Mistral and GPT-3.5-turbo?
- The public datasets used may suffer from potential data leakage issues due to the pre-training of LLMs. Hypotheses generated, particularly using Twitter data, may have been exposed in the literature before the models’ knowledge cutoffs.
Comments, suggestions and typos: - Quantitative quality evaluations are needed to prove that the proposed methods can offer more insightful hypotheses.
Overall assessment scores: 3