Title: Augmenting Research Ideation with Data: An Empirical Investigation in Social Science
Primary area: foundation or frontier models, including LLMs
Keywords: Ideation, Research idea generation, Data-driven, Social science

Abstract: Large Language Models (LLMs) show strong potential for generating novel research ideas, yet such ideas often struggle with feasibility and effectiveness. 
In this paper, we investigate whether augmenting LLMs with relevant data during the ideation process can improve idea quality. 
Our framework integrates data at two stages: (1) incorporating metadata during idea generation to guide models toward more feasible concepts, and (2) introducing an automated preliminary validation step during idea selection to assess the empirical plausibility of hypotheses within ideas.
For the ease of data collection and expert evaluation, we conduct experiments in the social science domain, with a specific focus on climate negotiation topics. Expert evaluation shows that metadata improves the feasibility of generated ideas by 20\%, while automated validation improves the overall quality of selected ideas by 7\%.
Beyond assessing the quality of LLM-generated ideas, we conduct a human study to examine whether these ideas, augmented with related data and preliminary validation, can inspire researchers in their own ideation. Participants report that the LLM-generated ideas and validation are highly useful, and the ideas they propose with such support are proven to be of higher quality than those proposed without assistance.
Our findings highlight the potential of data-augmented research ideation and underscore the practical value of LLM-assisted ideation in real-world academic settings.

TL;DR: We show that augmenting LLMs with data and preliminary validation improves the feasibility and expected effectiveness of research ideas, inspiring higher-quality human ideation in social science.