ICML 2025

Sparse Autoencoders for Hypothesis Generation
Rajiv Movva * 1 Kenny Peng * 2 3 Nikhil Garg 3 Jon Kleinberg 2 Emma Pierson 1
Abstract
We describe HYPOTHESAES, a general method
to hypothesize interpretable relationships between
text data (e.g., headlines) and a target variable
(e.g., clicks). HYPOTHESAES has three steps:
(1) train a sparse autoencoder on text embeddings
to produce interpretable features describing the
data distribution, (2) select features that predict
the target variable, and (3) generate a natural lan-
guage interpretation of each feature (e.g., men-
tions being surprised or shocked) using an LLM.
Each interpretation serves as a hypothesis about
what predicts the target variable. Compared to
baselines, our method better identifies reference
hypotheses on synthetic datasets (at least +0.06
in F1) and produces more predictive hypotheses
on real datasets (∼twice as many significant find-
ings), despite requiring 1-2 orders of magnitude
less compute than recent LLM-based methods.
HYPOTHESAES also produces novel discoveries
on two well-studied tasks: explaining partisan dif-
ferences in Congressional speeches and identify-
ing drivers of engagement with online headlines.
1. Introduction
Large language models (LLMs) show promise as a tool for
hypothesis generation. Discovering relationships between
text data and a target variable is an important and funda-
mental task with diverse applications in economics, political
science, sociology, medicine, and business (Grimmer, 2010;
Rathje et al., 2021; Sun et al., 2022; Gentzkow & Shapiro,
2010; Monroe et al., 2009; Nelson, 2020; Ranard et al.,
2016; Ting et al., 2017). What features of a restaurant re-
view predict a low rating? What features of a social media
post predict whether it will go viral? What features of a
patient’s clinical notes predict if they will develop cancer?
*Equal contribution 1UC Berkeley 2Cornell University 3Cornell
Tech. Correspondence to: Rajiv Movva <rmovva@berkeley.edu>,
Kenny Peng <k...