Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Prior
Authors: Yue Gong
Raul Castro Fernandez
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Hypothesis Assessment
Knowledge Elicitation
Correlation Analysis
Correlation Prediction
Primary area: applications

Abstract: As hypothesis generation becomes increasingly automated, a new bottleneck has emerged: hypothesis assessment. Modern systems can surface thousands of statistical relationships—correlations, trends, causal links—but offer little guidance on which ones are novel, non-trivial, or worthy of expert attention. In this work, we study the complementary problem to hypothesis generation: automatic hypothesis assessment. Specifically, we ask—given a large set of statistical relationships, can we automatically assess which ones are novel and worth further exploration? We focus on correlations as they are a common entry point in exploratory data analysis that often serve as the basis for forming deeper scientific or causal hypotheses.

To support automatic assessment, we propose to leverage the vast knowledge encoded in LLMs' weights to derive a prior distribution over the correlation value of a variable pair. If an LLM's prior expects the correlation value observed, then such correlation is not surprising, and vice versa. We propose the Logit-based Calibrated Prior, an LLM-elicited correlation prior that transforms the model’s raw output logits into a calibrated, continuous predictive distribution over correlation values. We evaluate the prior on a benchmark of 2,096 real-world variable pairs and it achieves a sign accuracy of 78.8%, a mean absolute error of 0.26, and 95% credible interval coverage of 89.2% in predicting Pearson correlation coefficient. It also outperforms a fine-tuned RoBERTa classifier in binary correlation prediction and achieves higher precision@K in hypothesis ranking. We further show that the prior generalizes to correlations not seen during LLM pretraining, reflecting context-sensitive reasoning rather than memorization.

Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Basee Calibrated Prior
Yue Gong" Amazon Web Services yuegongyGamazon.cmm com
Raul Castro Fernandez The University of Chicago raulcf@uchicago edu
Abstract
As hypothesis generation becomes increasingly automated, new bottleneck has emerged: hypothesis assessment. Modern systems surface thousands of sta- tistical
trends, causal links-but offer little guidance on which ones novel, non-trive non-trivial, worthy of expert attention. In this work, we study the complementary problem to hypothesis generation: automatic hypothesis assessment. Specifically, we ask nok-given& large set of statistical relationships, we automatically assess which ones are novel and worth further exploration? We focus correlations they that often serve as the basis for forming deeper scientific or causal hypotheses.
common entry point in exploratory data analysis
To support automatic assessment, we propose to leverage the vast knowledge encoded LLMs' weights to derive a prior distribution over correlation value of variable pair. If an LLM's prior expects the correlation value observed, then such correlation is not surprising, and vice versa. We propose the logit-based Calibrated Prior, an LLM-elicited correlation prior that transforms the model's 's output logits into a calibrated, continuous predictive distribution over correlation values. We evaluate the prior benchmark of 2,096 real-world variable pairs and achieves a sign accuracy of 78.8%, a mean absolute error of 0.26, and 95% credible interval coverage of 89 99.2% in predicting Pearson correlation coefficient. It also outperforms a fine-tuned RoBERTa classifier in binary correlation prediction and achieves higher precision @ K in hypothesis ranking. We further show that the prior generalizes to correlations not seen during LLM pretraining, reflecting context-sensitive reasoning rather than memorization.
Introduction
Generating hypotheses from large data repositories is quickly becoming easier. Modern data discovery systems 2 8 25 21 can enumerate every statistical relationship across datasets, and LLMS can draft thousands of plausible ideas by mining literature and data 33 B1 29] What used to take researcher weeks now happens in minutes. This ease of generation, however, introduces a new bottleneck: assessment. Experts are flooded with machine-suggested relationships-coorelatiosss causal links, trends, anomalies-withow relationships are trivial, redundant, already well known, forcing human experts to sift through long list just to find a few that are novel.
clear signal for which ones merit deeper investigation. Many of these
For example, illustrated in Figure correlation discovery system has surfaced tens of thousands of correlated variable pairs, leaving human experts to manually filter out trivial or expected patterns using their prior knowledge. A strong correlation between daily temperature and ice cream sales, for instance, is intuitive and quickly dismissed. In contrast, a negative correlation between household
Work done at the University of Chicago
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

income and housing prices might appear counterintuitive and warrant further scrutiny. This manual triage must be repeated across thousands of pairs to uncover truly novel or surprising correlations, making the process highly labor-intensive. One might hope that ranking correlations by magnitude could alleviate this burden. However, as shown in Figure (left panel where variable pairs are ranked by (robs)) stronger correlations are not necessarily more surprising in fact, they often reflect well-known or redundant relationships, a trend we further verify in Section
Correlation Discovery
Human Experts Prior Knowledge
LLM Prior Knowledge
@
High Surprising
Systems
"I know daily temperature and ice cream sales are strongly positively correlated. This correlation offers little information
Correlated Variable Pairs
Veriable1
Veriable2
Fabs
"This one counterintuitive, thought household income and housing price are positively correlated. Worth some further investigation."
-1 p(r). Surprising
0.7
Daily Tamp
Jos Cream
0.8
Household income
Housing Prices
0.2
Foos
-0.2
Manually sift corretations using prior knowledge
Tens of thousands of correlations
0.5
1
Figure 1: How Human Experts Assess Correlations Manually and How an LLM Can Help
In this paper, study the complementary problem hypothesis generation: automated hypothesis assessment. Specifically, we ask given a large set of statistical relationships, can we automatically assess which ones are novel and worth further exploration? We focus on correlation relationships as starting point, since they are a common entry point in exploratory data analysis and often serve as seeds for forming deeper scientific causal hypotheses
To tackle this problem, we draw inspiration from how experts reason: they use prior knowledge to form expectations about correlation's 's direction and magnitude. If the observed correlation (Noha) matches expectations,
unsurprising; it deviates, it may signal something worth exploring. In essence, experts apply an implicit prior shaped by their knowledge and the variable context.
Our core idea is to approximate this human prior using the rich, encoded knowledge within LLMs 27 18). Specifically, we define the LLM-elicited correlation prior, PLM(TX.Y Cx.y),, as a predictive distribution over correlation values rx,y between a variable pair X, Y conditioned on their context Cx.y, such as the description for each variable. By prompting the LLM with this context, we elicit its belief about the correlation values, treating these beliefs as a proxy for human expectations.
The LLM-elicited correlation prior helps identify which correlations are novel and worth expert attention. For instance, in Figure 1 (T.M(Daily Temp. Iee Cream Sales | C) centers around 0.7, making an observed value of 0.8 unsurprising. In contrast, a correlation of -0.2 against a prior centered at 0.5 signals high surprise. This surprise-based scoring offers scalable way to surface potentially insightful correlations. In our later evaluation (Section 5). we show that the LLM prior highlights expert-validated hypotheses from noisy urban data [19]
In this work, we propose Logit-based Calibrated Prior, an LLM-elicited correlation prior which transforms the LLM's raw output logits into calibrated, continuous predictive distribution over correlation values (Section[] 2). But how do we evaluate its quality?
First, we assess accuracy: if the prior's mode reliably predicts the sign and magnitude of observed correlations, it suggests alignment with empirical patterns. Second, we evaluate information content. A strong prior should assign high likelihood to observed correlations, reducing their information content relative to an uninformative baseline (c.g., a uniform prior). When applied at scale, this indicates the prior captures real- world patterns, easing the burden analysts. Third, measure cuabratioonwhethee the prior's uncertainty reflects reality-using 95% credible interval coverage. This is crucial for decision- making: overconfident priors exaggerate surprise and risk misdirecting expert attention. Finally, we ask a deeper question: is the prior reasoning from context, or merely recalling memorized correlations based variable names? To probe this, introduce novel evaluation based on contextual contradiction to disentangle these possibilities.
To support these goals, construct benchmark of 2,096 variable pairs with observed correlations. We evaluate predictive quality and information reduction (Section hypothesis discovery utility (Section 5) and whether the prior reflects contextual reasoning memorization (Section

---

Results are promising: our Logit-based absolute error of 0.26 on Pearson correlation coefficients in the range 1-1, 1]. with strong calibration- 95% intervals covering 89.2% of observed values. It also reduces the average information content from 0.69 (uniform prior) to 0.27. Our method outperforms baselines, including uninformative priors, Gaussian priors from LLM-verbllized parameters, and fine-tuned RoBERTa classifier [30]. It also achieves higher precision K when retrieving meaningful correlations in noisy urban data. For instance, it highlights link between bike dock density and community wealth, a hypothesis studied in prior work 77, while down-ranking obvious patterns like library visitors and book circulation. Finally, we show that the prior generalizes beyond correlations seen during pretraining.
based Calibrated Prior achieves 78.8% sign accuracy and a mean
These results show that LLMs encode informative prior beliefs about statistical relationships, demon- strating their potential to serve as proxies for hypothesis assessment-a a task that currently relies on human expertise and is highly labor-intensive. Our work highlights a promising direction for leveraging LLMs to help experts navigate large hypothesis spaces and make novel discoveries.
2 Logit-based Calibrated Prior (LCP): Constructing a Continuous Correlation Prior from LLM Logits
In this section, present the Logit- based Calibrated Prior (LCP), method for constructing the correlation prior, PLM(TX.) Cx.y) -a predictive distribution over correlation values rx,y between variable pair X, Y conditioned on their context Cx,y, such the description for each variable.
One way to elicit a distribution from an LLM is to have parameterize fixed form-e-g., modeling its belief over correlation as Gaussian by providing mean and standard deviation. However, this approach relies on the assumption that the model's internal belief distribution conforms to the chosen parametric form, which is not the case in most cases. To test this, we conducted chi-square goodness-of-fi analysis (28) on the LLM's output distributions for 2,096 correlations. The normality assumption was rejected in 2,095 cases at the 5% significance level, indicating that the LLM's beliefs are poorly approximated by a Gaussian fit (see Appendix A for details). Moreover, this parametric approach introduces additional complexity: While the original goal is to estimate single correlation value, it requires estimating additional parameters whose values are themselves subject to error.
Our approach. Rather than asking the L.M to estimate parameters of a fixed distributional form, we directly prompt it to predict the correlation coefficient (see prompt Appendix H..)) and construct a full distribution over possible correlation values. While one could obtain this distribution by sampling from the model, it would be computationally expensive. To address this, we propose a more efficient strategy by constructing the prior directly from the LLM's logits. This approach does not assume the distribution's 's shape and allows the model to focus on estimating the correlation between the variables.
We begin by constructing a discrete probability distribution from the model's raw token logits (Algorithm ( Without loss of generality, we assume r denotes Pearson's 's correlation coefficient, constrained to the range [-1, 1]. At each decoding step t. the language model produces a real-valued
logit vector l,, where each entry e,") corresponds a token in the vocabulary. These logits are converted into log- prg--obbbbiliies via the softmax function. For a selected token at position t, its
log-probability is given by logp, (v.) = p((x)) log E, exp((("))
We design prompt that elicits a structured scalar response, such / ("coefficien"": "<value>" "). To extract the correlation value, we first identify the start and end positions of "<value>" the output sequence (line 4). Starting from this token position, we extract the top-k tokens at each subsequent decoding step (line 5). A complete numeric response, such as "-0.69" is composed of a valid sequence of tokens-c.g.. sign, integer part, decimal point, and numeric suffix. For instance, the numeric suffix token, the model might assign different probabilities to completions like 69, 60, or 70. To prevent length biases, we ensure that positive and negative numbers use the same number of tokens in their representation (see Appendix B).
We enumerate all token sequences (line 5), concatenate them into strings (line 6), cast them to float values (line 8), and compute their joint prg-probabilities by summing the log-probabilities of each token in the sequence (line 12). We discard any sequences that produce invalid float values or values outside the valid correlation range [-1, 1] (line 10). When multiple token sequences map to the same numeric value (e.g., "0. 65" and " .65"), we aggregate their unnormalized probabilities (line 13-17).
3

---

Finally, we normalize across all valid correlation values using the softmax function to obtain a discrete probability distribution, {(r), p;)) (line 19) where r is decoded correlation value, and Pj its model-assigned probability.
Impact of Single-Path Decoding. Our approach decodes one token time conditioned the most likely token from the previous step. In particular, we first select the most probable sign token (c.g.. + or -) and condition all subsequent decoding on that choice. This introduces an approximation: we compute the joint probability of each numeric string under single sign path, rather than marginalizing over both. While a full multi-beam search would more faithfully capture the true joint distribution by exploring multiple branches at each step, we find that this approximation works well in practice. First, the model achieves high sign accuracy: as shown in Section 4 predicts the correct sign 78.8% of the time, so most sequences are decoded under the correct branch. Second, it exhibits high sign confidence: across 2096 correlation predictions, the median probability gap between the two signs is 99.8% (77.7% on average), indicating that the probability mass of the alternative branch negligible. Finally, our method ensures scalability: a full beam search would increase decoding cost exponentially with sequence length, whereas the single-path strategy scales efficiently to tens of thousands variable pairs while maintaining strong empirical performance.
The discrete distribution is sparse and limited to discrete values determined by the tokenizer and top-k decoding strategy. However, downstream susks-such as computing surprise in Figure require probability density at arbitrary values. To support this, we smooth the distribution using a weighted sum of Gaussian kernels centered each decoded value. Since Pearson correlations lie in [-1. 1]. we truncate and renormalize the distribution to ensure it integrates to one. The final LCP density function f(r) is defined as:
Smoothing to Obtain a Continuous Prior.
1
N
(-(-)))²)
f(r)
ΣEs Ps j=1
exp
[-
7
V2
20²
where a is the standard deviation of each kernel and controls the degree of smoothing, and Z is the normalization constant.
Algorithm 1 "CosttnctDiscreteriiirrrooo
1: Input: Token logits (fo)T=1, structured output template T such /"coefficient": "<value>")
2: Output: Discrete prior (r),,,)),,-
3: Initialize empty map: logp_ mapp_aap t
& Locate start and end positions of the value field 5: for all sequences s (Ptg....,, Dt,) from top-k tokens at each position do
to, t1 + FindVaaleeToxenSSpp(((]]
6: str + concat(s)
7: if is_valid____ float(str) and float(str) [-1,1] then float (str)
8:
9: else
10:
continue
11:
end if
12:
log Et++2 logp!") (12)
13:
i E logp map then
14:
logp_map[r] map[r] <- log happloogp_ap//// + exp(logp.))
15: else
16:
logp_mig pgp_map[r]] + pr 17: end if
18: end for
19: (rr,,,))){ < (ertmax(logp_aap)
Normalize log-probs into valid probability distribution
20: return ((rs)P)))
Selecting an appropriate kernel standard deviation a is critical to ensure the prior reflects realistic uncertainty. If a is too small, the resulting distribution will be overconfident and overly spiky; if too large, will underconfident and overly diffuse. Standard bandwidth selection rules, such as Scott's 2 or Silverman's rule [26). are not applicable in our setting, they assume i.i.d. samples from an underlying distribution. In our case, contrast, the discrete values and their probabilities are derived from LLM output logits and reflect model-specifie beliefs, not empirical frequencies.

---

To address this, we tune o using a held-out validation set by minimizing the average negative log- l-llikiihhodd the observed correlation values:
arg min Bra |- log Per (Nons)]
This objective penalizes priors that assign low probability density to ground-truth correlations, thereby encouraging distributions that place probability mass closer to the observed values. Optimizing o this way calibrates uncertainty to reflect empirical variability and improves downstream reliability. The validation set Dval consists of 300 randomly sampled correlations, disjoint from our evaluation dataset. The optimized value "- = 0.4 is used for LCP.
The kernel standard deviation a does not need be re-tuned long as four key elements remain unchanged: the LLM, the prompting strategy, the task (predicting Pearson correlation coefficients). and the kernel function used. This because a corrects for the systematic bias the model's uncertainty-that is, whether the model tends to be consistently overconfident or underconfident in its predictions. When the model architecture, prompt design, task, and the kernel function remain fixed, this bias remains stable across inputs, even if individual predictions vary. In this setting, single globally tuned a sufficient to calibrate the model's uncertainty across a broad range of variable pairs. We further demonstrate in evaluation (Section 4 S
that the selected generalizes well on the evaluation dataset, demonstrating its robustness. However, if any of these components change-suchs such switching to different model, altering the prompt, targeting different correlation metric, or switching to a different kernel function the structure of the output distribution may shift, and should be re-tuned to ensure proper calibration.
In the evaluation, we compare the Logit-based Calibrated Prior against two baseline methods for constructing correlation priors, highlighting the advantages of avoiding parametric assumptions and applying proper calibration. The first is a Gaussian prior, which assumes the LLM can directly parameterize normal distribution by predicting its mean and standard deviation. The second is an uncalibrated KDE prior, which is similar to our method, but selects the kernel standard deviation using Scott's 's rule based the empirical standard deviation of the discrete probability distribution.
3 Benchmark Construction
We curate a benchmark of 2,096 real-world variable pairs to evaluate correlation priors. Each entry includes two variables, their descriptions, a dataset summary, and the observed Pearson correlation Tobs E [-1, 11, computed from raw data. The benchmark combines variable pairs from the Cause-Effect Pains [13 and Kaggle (30) datasets. have open-sourced our code and data at LtwiataiiiiiLMMLM----------- www.--oorreliiioo stw.rrelaatioonAAsssseetettt Assessment,
The Cause-Effect dataset contains 108 variable pairs with known causal relationships. We retain 96 pairs where the correlation statistically significant (p 0.05). The Kaggle dataset con- sists of correlations between variable pairs extracted from pub- liely available tables on Kaggle. The original dataset provides variable names but lacks variable descriptions. To enrich the
1000
Frequency
750
500
context for variables, we use the Kaggle API to retrieve dataset summaries and employ GPT-4o (see Appendix H.5) to assess whether the variable names are self-descriptivee
250
filter out non-informative names (e.g., single characters or generic iden- tifiers like "Unnamed: 0") and retain only those pairs for which both variables are judged meaningful. This further cleaning al- lows us to isolate and study the model's 's ability to reason about relationships, rather than its ability to interpret metadata.
0.2
[r]
Figure 2: The Bias toward High Correlations in Kaggle dataset
After filtering, we obtain 7045 statistically significant correlations (p < 0.05). To mitigate the bias toward extreme correlations (see Fig. 2). we perform stratified sampling by |r|: divide the range [-1, 1] into 10 equal-width bins and sample 200 correlations per bin, yielding a balanced set of 2,000.
Kaggle API does not support retrieving variable descriptions
5

---

A balanced sample ensures fair evaluation across all correlation strengths, preventing the model's performance from being skewed by overrepresented low or high |r| values.
4 How Well Does LCP Predict Empirical Correlations?
We evaluate LCP by measuring how well it predicts observed correlations. First, we assess predictive accuracy using two metrics: sign accuracy, the fraction where f Tobs > O, and absolute error, [f - - where is the mode the prior. We calculate the mode using grid sampling. Next, evaluate differential information content by computing log p(rom)), adapting Shannon's self- information (24] to the continuous case. For simplicity, we refer to it as information content hereafter. good prior assigns high likelihood to observed values, reducing the information content of the corpus and easing analyst workload. Finally, we assess calibration by 95% credible interval coverage- the fraction of cases where Tobs falls within the prior's 95% credible interval. Calibration is critical: an overconfident prior may exaggerate surprise from small deviations, leading to false positives and misleading experts.
We compare LCP with the following baselines. All methods use GPT-4o (2024-08-06) 17 as the underlying model.
Uniform Prior: non-informative baseline with constant density 0.5 over [-1, 1]. The sign accuracy for it is measured by randomly guessing the sign of the correlation.
Gaussian Prior: We adapt the method from Capstick al. M. which elicits Gaussian priors via LLM-prompted mean and standard deviation, to model a truncated Gaussian prior over correlations
in 1, 1] (see Appendix H.2).
KDE Prior: A kernel density estimation using Gaussian kernels, where the kernel standard deviation is set using weighted version of Scott's rule: = 1.06 ā Rell 1/5 where ā is the weighted standard deviation of {(),,;)), and Reff is the effective sample size.
Results. Fig. reports the average value of each metric across all correlations, positioned in a quadrant plot. Complementarily. Fig. presents the full distributions of absolute error, p(robs), and information content. Fig. 3 shows that LCP achieves the best balance, matching the highest sign accuracy (78.8%) of KDE while providing significantly better calibration (89.2% coverage). In contrast, the uncalibrated KDE and Gaussian priors are overconfident, assigning low likelihood to Tobs and yielding poor coverage (59.99 and 49. %2 respectively). On the other hand, the uniform prior offers high coverage (92.3% but suffers from poor accuracy and high absolute error (f
Tabs = 0.51).
In addition, LCP significantly reduces the average information content of the correlation corpus from 0.69 under a uniform prior to 0.27, indicating that it assigns higher likelihood to observed correlations. In contrast, the Gaussian and KDE priors increase the average information content 4.10 and 1.73, respectively, due to their overconfident predictions. This is reflected in the long tail of low-density values in Fig. 44. As shown Fig. 4 LCP yields more concentrated distributions for both likelihood p(Tots) and information content, highlighting better calibration.
80
78.8%, 59.9%
78.8%, 89.2% | Fobs F0.26, IC=0.27
| Fobs 00266, IC=1.73
Sign Aecuracy 1
70
73.1%, 49.1%
Fobs /0b2 -0.26, IC=4.10
Good accuracy, poor calibration
Good accuracy and calibration Poor accuracy, good calibration
60
Uniform Gaussian
KDE
50
LCP (Ours)
49.0%, 92.3%
| - robs 00551, IC-0.69
50
60
70
80
90
95% Coverage (Calibration) 1
Figure Accuracy vs. Calibration of Correlation Priors (IC=Information Content)
To understand the poor calibration of the Gaussian and KDE priors, we examine their kernel standard deviations. Both produce overly small values, leading to sharply peaked densities. The median is

---

Uniform
Gausiaa
KDE
LCP
1.5
Density pto (Ing seale)
Informution Conteat
1.0
10:
Abolute
0.5
10"
9d2
0.0
(a)
(b)
(c)
Figure 4: Full Distribution of Metrics Different Priors
0.10 for the Gaussian prior and 0.08 for the KDE prior-both much smaller than the fixed o = 0.4 in LCP. Under the Gaussian prior, the LLM returned a 0.1 in 74% (1,552/2,096 of cases. The full distribution of a values shown Appendix C As we further examine in Appendix(D) [D] this behavior sampling variability and implicitly assumes fixed sample
arises because the LLM interprets size of 100-producing a default value of a = 0.1 regardless of context. The predicted captures expected variation from random sampling (aleatoric unceriainty). but fails adjust based on the input context or account for uncertainty arising from limited knowledge (epistemic uncertainty) 12 51.
Figure analyzes LCP's behavior across ten bins of observed correlation Tobs The bias f f- Tobs decreases with Tobs: the model overestimates strong negatives and underestimates strong positives. Sign accuracy is lowest when aabs small, bottoming out near -0.15 and rising sharply beyond |obu Z 0.3, reaching near-perfect accuracy for |rbbl 2 0.7. In Fig. 5e.d, the prior assigns lowest density (i.e., highest information content) moderately negative correlations (Fabs -0.5), indicat- ing weaker estimation. In contrast, strong positives receive the highest density and lowest information content. Overall, the prior shows asymmetric error: it performs best strong positives and struggles with moderate negatives, consistently underestimating correlation magnitude-a reflection of the LLM's conservative predictions without direct data access.
0.5
Sign Accuracy
Deasity Tabs
Info Content
5,0
Fobs
0.5
2.5
0.0
0.0
-1.0
-0.6
-0.2
0.2
0..6
1.0
-1.0
-0.6
-0.2
0.2
01.6
1.0
-1.0
-0.6
0.2
0.2
0.6
1.0
-1.0
-0.6
-0.2
0.2
0.6
1.0
Febs (4)
Tobs
Fobs (c)
Fobs (d)
(b)
Figure 5: Performance across ten bins of the true correlation Tobs
Comparison of LCP and RoBERTa on Binary Correlation Classification. While LCP models a full distribution, BERT- and RoBERTa-based classifiers [14 6 can be adapted for binary correlation deteciionddeeeeining whether pair of variables is correlated based on predefined threshold. We adopt the method from Trummer [30], who fine-tune RoBERTa using labeled pairs to build correlation classifier. LCP is adapted to solve the binary classification task by thresholding its mode, enabling direct comparison with classification- based approaches.
To ensure a fair comparison, we first evaluate RoBERTa a zero-shot setting, matching LCP, which requires no training. We then fine-tune RoBERTa on 20% of the benchmark, following standard practice for applying RoBERTa to downstream tasks. Figure shows performance across different correlation thresholds. Note that RoBERTa must be re-trained for each threshold.
LCP consistently outperforms both baselines terms of accuracy, F1, and MCC across all thresholds- achieving up to 0.84 accuracy, 0.79 F1, and 0.53 MCC-despil being entirely zero-shot. This indicates that our method provides the most balanced predictions overall. Zero-shot RoBERTa behaves like one-class detector: it predicts correlated for every pair, yielding perfect recall but zero MCC and rapidly deteriorating accuracy/precision as the threshold tightens from 0.5 to 0.8. Fine-tuned RoBERTa corrects this imbalance to some extent after seeing 20% of the data, but its gains are greshold-speciffe and require retraining whenever the decision boundary moves. In contrast, By producing a full predictive distribution over LCP naturally adapts to different thresholds.

---

Threshold =0,5
Threshold = 0,6
Threshold 0.7
Threshold 0.8
0.5
live
BA USST
0.0
Recall
Fi
MCC
Zero-shot RoBERTo
RoBERTa 20% training
(Ours)
MCC=0
Figure 6: Classification Performance over Different Correlation thresholds
Sensitivity Analysis of LCP. We conducted a sensitivity analysis to assess the effect of two factors on LCP's robustness: (i) the choice of kernel function used for smoothing, and (ii) the validation set size used to calibrate the smoothing parameter
We compared three alternative kernels- Uniilsmm, Epanechnikov, and Triangle-againn the Gaussian kernel used in our main results. For each kernel, we tuned the bandwidth parameter on a held-out validation set and evaluated performance on 2096 correlated pairs. The results Table Sableessho show that LCP maintains strong sign prediction accuracy and calibration across all kernel choices.
Table I: Sensitivity to kernel function used for smoothing.
Kernel
Sign Accuracy
|f robal
95% Coverage
Uniform
76.4%
0.32
82.1% 1
Fpanechnikoy
78.8%
0.26
89.7%
Triangle
78.8%
0.31
90.9%
Gaussian (reference)
78.8%
0.26
89.2%
These findings suggest that LCP's performance is not sensitive to the specific choice of kernel function, as long as the bandwidth appropri- ately tuned. However, due to differences in ker- nel shape, the optimal should re-selected using validation set when switching kernels.
Table 2: Optimal across validation set sizes.
Set Size
300
500
1000
2000
Opptimal
0.40
0.38
0.38
0.41
We also evaluated how the optimal smoothing bandwidth varies with the size of the held-out validation set. As shown in Table the optimal a remains stable across different validation sizes, indicating that LCP robust sample size variations.
Using LCP to Retrieve Expert-Flagged, Hypothesis-Worte Correlations
Can LCP support hypothesis assessment in noisy, real-world settings? We evaluate on Nexus 8] system designed to help domain experts discover correlations in urban data. Nexus computes 40,538 pairwise correlations from Chicago Open Data [19] by aligning and aggregating numeric attributes from different tables tables-either either temporally (by month) spatially (by census tract). Attributes are summarized (e.g., via mean or sum), joined on shared key, and then correlated. This pipeline introduces real-world challenges: joins across sources, aggregation choices, and missing values-all of which impact the resulting correlations.
Of the full set, 15 correlations were labeled as eypothesis-worthy by human experts in the original Nexus evaluation. For example, a correlation between bike dock density and community wealth suggests stations are more common in affluent areas, hypothesis studied in [7]. We use these expert-flagged examples to evaluate how well LCP retrieves hypothesis-worth correlations in messy, transformed data. Since ground- trutt correlation values unavailable due data aggregation and imputation, we adopt an information retrieval setup: treating the 15 sspert-flagged correlations as targets within pool of 115, formed by adding 100 random samples from the full Nexus corpus.
We compare five ranking strategies: (i) random, (ii) by absolute correlation |r|, (iii) by increasing probability assigned by RoBERTa model fine-tuned on 20% of the benchmark, where lower

---

probability of the "correlated" class indicates higher surprie, (iv) by LLM surprise, where the LLM (GPT-40) given the full metadata (column names, table names, description, and observed correlation) and prompted to classify the correlation as "surprising" or "not surprising", and (v) by increasing prior likelihood p(rots) under I.P,, treating lower likelihood as more surprising. We report Precision@55 @5, @ 10, @ 15, and the average rank of expert-labeled correlations.
Table 3: Retrieval Performance Comparison
Method
Precision@5
Precision@ 10
Precision 15
Average Rank +
Random Ranking
0.13
0.13
0.13
58.0
Ranked by [r]
0
0
0
95.4
Ranked by RoBERTa
0.60
0.60
0.53
30.9
Ranked by LLM Surprise
0.4
0.2
0.13
Ranked by LCP
0.60
0.80
0.60
21.5
As shown in Table ranking correlations by LCP outperforms all baselines-achievinn up 0.80 Precision @ 10 and reducing the average rank of expert-labeled correlations to 21.5, compared to 30.9, 58.0 and 95.4 for the RoBERTa, random and |r|-based rankings, respectively (see Appendix D for derivation of the expected performance under random ranking). Ranking by |r] performs worst, with the highest average rank and zero precision, extreme correlations often reflect trivial redundant relationships (e.g., repeated attributes across years), not meaningful insights in Chicago Open Data.
Using ICP, all four correlations related the expert-laheled hypothesis that bike stations are more likely to be located in wealthier areas are ranked within the top 6. In contrast, an unsurprising correlation-the one between library visitors and library circulation-is ranked much lower at 95th. These results demonstrate that LCP can surface correlations that align with expert judgment, even in the presence of data transformations and noise.
Is LCP Reasoning from Context or Relying on Memorization?
We evaluate whether LCP is reasoning from context simply relying on memorization, crucial distinction for generalization beyond the model's 's pretraining data. This is essential for hypothesis assessment, where many relationships are unseen during training and depend context. To probe this, we introduce an evaluation based on contexiual contradiction. For each variable pair, we construct an alternate context that plausibly reverses the original correlation, simulating counterfactual. We then re-derive the prior by prompting the LLM with this modified context (Appendix H.4). If the model adjusts its belief accordingly, suggests reasoning from context rather than memorization.
Contradictory Context Generation. We use the Cause-Effect Pairs dataset to construct counter- factual scenarios. From this dataset, select 84 variable pairs where the model initially predicts the correct correlation sign. For each pair, we prompt Gemini 2.5 Pro to generate a new context that plausibly reverses the original relationship (see Appendix H.3) All 84 generated contexts are manually reviewed by the authors to ensure the reversal is logically sound and free of explicit cues (e.g., phrases like "therefore there should be a negative correlation"). We assign the negated correlation - -ohh as the new observed value. Since these contexts are synthetic and no real data exists, these new Tobs values serve as approximations.
Result. Table shows the performance of correlation priors on reversed correlations. LCP achieves 100% sign accuracy on the original contexts, dropping slightly to 95.2% under contradictory contexts. Manual inspection reveals that two of the four errors stem from reasoning failures: the model grasps the high-level logic but fails at the final inference step in multi-hop scenarios (see Appendix (F). LCP also maintains strong calibration, with 92.9% coverage at the 95% level, and achieves lower information content (0.25 vs. 0.69) and absolute error (0.30 vs. 0.55) compared to the uniform prior.
Table 4: Performance of correlation priors on correlations with contradictory contexts.
Method
Sign Acc. (1)
| Patal (L)
Information Content (L)
95% Coverage (t)
Uniform
0.464
0.55 0.25
0.69
92.3%
ICP (ours)
0.952
0.30 + 0.28
0.25 + 0.98
92.9 %

---

This experiment shows that LCP is not merely recalling memorized correlations. When given counterfactual contexts, it updates its predictions accordingly, achieving 95.2% sign accuracy with strong calibration and low error. These results suggest that I.CP generalizes beyond pretraining and behaves dynamically, crucial property for real-world hypothesis assessment.
Related Work
Elicit Priors from Human Experts. O'Hagan al. [66 and Gosling [9] introduce the SHELF framework, a structured protocol for cliciting expert judgments and converting them into probability distributions. The process involves training, individual assessments, group discussions, and consensus- building, followed by fitting a statistical distribution to the agreed-upon judgments. This human elicitation process is costly and time-consuming whereas approach exploits the rich knowledge encoded in LLM weights to approximate expert priors automatically.
LLMs for Regression Tasks. Several works exploit the knowledge encoded in LLMs for regression. Choi et al. [3] use LLMs for feature selection by prompting whether a variable is predictive of a given target, while others [20] I0 [ aim to model prior distributions over feature weights. For example, Requeima et al. [20) require training examples to guide the LLM in generating output distributions, and Capstick et al. [T] assume the LLM can directly parameterize distribution by prompting it to output means and standard deviations given feature and target names. contrast, our work focuses on constructing prior distribution over correlation coefficients between variable pairs before observing any data, using raw LLM logits directly-without without requiring the model to parameterize a distribution.
Additional Related Work. We include further discussion data discovery systems and automatic hypothesis generation Appendix
Conclusions
In this paper, propose the Logit- Correlation Prior, LLM-elicited prior that transforms raw output logits into a calibrated, continuous predictive distribution over correlation values-paving the way for automatic hypothesis assessment. Our experiments show (i) LCP achieves the best balance between accuracy and calibration for predicting empirical correlations, outperforming Uniform, Gaussian, and KDE priors; (ii) LCP outperforms fine-tuned RoBERTa classifier on binary correlation classification; (iii) LCP effectively highlights hypothesis-worth correlations flagged by human experts in noisy urban data; and (iv) LCP goes beyond memorizing correlation values from pretraining, performing contextual reasoning.
Limitations
Generating an LCP requires an LLM call per correlation, which can be costly at scale. To im- prove scalability, preprocessing steps-such as filtering out redundant variable pairs across similar datasets-can help reduce the number of required queries. LLMs may also produce false positives or negatives. An LLM may possess knowledge beyond that of human experts, causing to dismiss corre- lations that are actually insightful to the experts (false negatives). It can also misinterpret well-known relationships, incorrectly flagging them surprising (false positives), shown in Appendix F
10
Acknowledgments
We thank the anonymous reviewers for their constructive feedback, which significantly improved the clarity and quality of this work. This work was supported partially by the National Science Foundation (CAREER Award 2340034) and the Data Ecology Research Initiative at the Data Science Institute, University of Chicago.
10

---

References
[1] Alexander Capstick, Rahul G. Krishnan, and Payam Barnaghi. Autoelicit: Using large language models for expert prior elicitation in predictive modelling, 2025.
org/
abs/ abs/2411
[2] Fernando Chirigati, Harish Doraiswamy, Theodoros Damoulas, and Juliana Freire. Data polygamy: The maany-aany relationships among urban spatio-temporal data sets. In Proceedings of the 2016 International Conference on Management of Data, pages 1011-1022 2016.
[3] Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models as task-specific priors. arXiv preprint arXiv:22100 10. /2530, 2022.
[4] MIT Critical Data, Matthieu Komorowski, Dominic Marshall, Justin Salciccioli, and Yves Crutain. Exploratory data analysis. Secondary analysis of electronic health records, pages 185-203, 2016.
[5] Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural Safery, 31(2):105-113 2009.
[6] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLLT), pages 4171-4186, 2019.
[7] Elizabeth Flanagan, Ugo Lachapelle, and Ahmed El-Geneidy. Riding tandem: Does cycling infrastructure investment mirror gentrification and privilege in portland, or and chicago, il? Research in Transportation Economics, 60:14-24, 2016.
[8] Yue Gong, Sainyam Galhotra, and Raul Castro Fernandez. Nexus: Correlation discovery over collections of spatio-temporal tabular data. Proc. ACM Manag. Data, 2(3), May 2024. doi: 10.1 145/36645997 URL RRRLhttps:// //dt:::://do..oo org/10. 11110.....11//11 1145/3654995
[9] John Paul Gosling. Shelf: the sheffield elicitation framework. Elicitation: The science and art of structuring judgement, pages 61-93. Springer, 2017.
[10] Henry Gouk and Boyan Gao. Automated prior elicitation from large language models for bayesian logistic regression. In AutoMI. Conference 2024 (Workshop Track), 2024. URL https://opperevieee wwt.://orrriiiieeellllll....
[11] Saint John Walker. Big data: A revolution that will transform how we live, work, and think, 2014.
[12] Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In NeurlPS, 2017.
[13] Anthony ML Liekens, Jeroen De Knijf, Walter Daelemans, Bart Goethals, Peter De Rijk, and Jurgen Del-Favero. Biograph: unsupervised biomedical knowledge discovery via automated hypothesis generation. Genome biology, 12:1 12:1-12,2 2011.
[14] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1907.1
2019.
[15] Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: methods and benchmarks. Journal of Machine Learning Research, 17(22)11-02. 2016.
[16] Anthony O'Hagan, Caitlin E Buck, Alireza Daneshkhah, J Richard Eiser, Paul Garthwaite, David Jenkinson, Jeremy E Oakley, and Tim Rakow. Uncertain judgements: eliciting experts' probabilities. John Wiley & Sons, 2006.
[17] OpenAl. Gpt-4o. cessed: 2025-05-15.
2024. Ac-

---

[18] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? 1909.0 2019.
arXiv preprint
[19] Chicago Data Portal. Chicago data portal, 2025. ott. org
[20] James Requeima, John Bronskill, Dami Choi, Richard Turner, and David K Duvenaud. Llm processes: Numerical predictive distributions conditioned on natural language. Advances in Neural Information Processing Systems, 17:109909-119677 2024.
[21] Aécio Santos, Aline Bessa, Fernando Chirigati, Christopher Musco, and Juliana Freire. Correla- tion sketches for approximate join-correlation queries. In Proceedings of the 2021 International Conference on Management of Data, pages 1531-1544, 2021.
[22] Aécio Santos, Flip Korn, and Juliana Freire. Efficiently estimating mutual information between attributes across tables. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 193-206, 2024. doi: 10.1109/ICDE00 1e0.00014.2200000 46.2022.0002..
[23] David W Scott. Multivariate density estimation: theory, practice, and visualization. John Wiley & Sons, 2015.
[24] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[25] Shohei Shimizu, Takanori Inazumi,, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Patrik Hoyer. Directlingam: A direct method for learning a linear non-gaussian structural equation model. Journal of Machine Learning Research-JMLR, 12(Apr):
2011.
[26] Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
[27] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972) 2023.
[28] George W Snedecor and Witiiam G Cochran. Statistical methods, 8thedn. Ames: lowa State Univ. Press Iowa, 54:71-88 82, 1989.
[29] Scott Spangler, Angela D. Wilkins, Benjamin J. Bachman, Meena Nagarajan, Tajhal Dayaram, Peter Haas, Sam Regenbogen, Curtis R. Pickering, Austin Comer, Jeffrey N. Myers, loana Stanoi, Linda Kato, Ana Lelescu, Jacques J. Labrie, Neha Parikh, Andreas Martin Lisewski, Lawrence Donehower, Ying Chen, and Olivier Lichtarge. Automated hypothesis generation based on mining scientific literature. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, page 1877-1886, New York, NY, USA. 2014. Association for Computing Machinery. ISBN 978145032956,. doi: / 2623667
10. 10.1145/262333.2262666 URL http::ttt/tiioorrg/ggeett
[30] Immanuel Trummer. Can large language models predict data correlations from column names? Proc. VLDB Endow. 16(13):430-43333 September 2023. ISSN 2150-8097,
doi: 10.14778/
3625054.3225066. URL https:///ol org/10. 14778/3625055 2051///362225533336666666 3625066
[31] Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Improving scientific hypothesis generation with knowledge grounded large language models, 2024. URL https:// :/ap:s://axxxvv wwtrrxivv.oggg/// abs//411.000 02382
[32] Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. Large language models for scientific discovery molecular property prediction. Nature Machine Intelligence, pages 1-11, 2025.
[33] Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. Hypoth- esis generation with large language models. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), page 117-139. Association for Computational Linguistics, 2024. doi: 10. 10.11663///22244.mp44ciiea 1.10. URI. http::/t:///d doi http://........rr//00011661111/0
mlp4science-1.10
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim in the abstract and introduction that knowledge encoded in LLMs support automatic hypothesis assessment. To this end, we contribute the Logit- based Calibrated Prior correlation prior derived from an LLM's output logits. Our evaluation shows that this prior effectively highlights correlations considered hypothesis-wond worthh by human experts. Thus, our main claims accurately reflect the paper's contributions and scope. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all necessary details to enable full reproducibility of our experimen- tal results. Specifically: (1) Section presents a detailed description of our algorithm (see Algorithm 1) (2) SectionsJandd desscrie our benchmark construction and evaluation proce- dures; and (3) we release our code and data at
wtt.gtt..ooo..ee.eeeeeoooooeeeet com/ cool//heDDaaatttion
LLM for-oooo-rrrrreoeeeeeeeeo -ar.aaationnsssssennnne Assessment
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide source code for reproducing our experiments at hhttps:://ithhu)
http//hDaaaataaiionnllle Prior-foor Correlation Assessment The repository includes a README with detailed instructions for installation, contiguration, and running the experiments.
Guidelines;
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
Cobllic/gides/Cee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/pips..ccppulic//uuee http://uiiiiesseeeeee /
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all necessary details for people to understand our results. Section outlines the rationale behind the evaluation metrics and describes the setup for each method in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
15

---

We report the full distribution of evaluation metrics across all runs (Fig. 4). include error bars in the binned analysis (Fig. and provide standard deviations in Table
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the L.M used in Section including its knowledge cutoff date; its usage cost is publicly available [17].
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //te:r/nnsusssee clss.cc/puuu htcc/ppblic//Ehhcccccleleellll.d
Answer: [Yes]
Justification:
The authors have reviewed and adhered to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
16

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss positive impacts in the introduction, highlighting how our approach supports analysts in identifying hypothesis-wond worthy statistical relationships. Potential negative impacts are discussed in Section ( including the risk of false positives and negatives from LLMs. We emphasize that the prior is intended to assist, replace, human experts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
17

---

Justification: All external assets used in this work are properly cited, and we have ensured that their licenses and terms of are fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /daaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide well-documented code and benchmark datasets at https:// github.com//aiie http:////.aaaaaaaaa.o../ooooooor Assssssattiineeeessseeeeetett//
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy //npur:////urraie cc. Cc//oonferencee/22255///// for what should or should not be described.
19

---

A Normality Test via Chi-Square Goodness-of-Fit
To assess whether the LLM's output distribution over correlation values conforms Gaussian shape, we perform chi-square goodness-of fit test. For each correlation prompt, we obtain a discrete probability distribution {(r).P) p5)] 1 where each r; E [-1,
decoded numeric value and Pj the associated model-assigned probability mass, derived from token-level logits.
We convert the probability mass function into of pseudo-counts by assuming nominal sample size M = 1000, yielding observed counts Oj = . Ppj. We then estimate the mean and variance o² of the distribution as follows:
rr 2= Pj-
JJ Pj.
Next, compute the expected count for each support point ry under fitted Gaussian:
(T3 (T, - 11)²
exp
22T22
20²
which we normalize to form a probability distribution ñú - 43/ Σ, gn. and then scale to expected counts F.j M Py- The chi-square test statistic is computed as:
(0 E2)²
E
The null hypothesis is that the observed distribution comes from the fitted Gaussian. We evaluate the p-value corresponding to the computed x² and reject the null at the 5% significance level.
Applied to the 2,096 correlations in our benchmark, the normality hypothesis was rejected in 2,095 cases, indicating that the LLM's output distributions are poorly approximated by parametric Gaussian form. This result justifies our conpaammetticc approach, which avoids imposing fixed distributional shape.
B Preventing Length Biases
In setup that uses the GPT-4o tokenizer, both positive and negative correlation values represented with the same number of tokens. Specifically, both forms include four tokens: sign, integer part, decimal point, and fractional part.
For example, the response:
{
"coefficient"; "0.5"
}
tokenized as ", '0', ,.,, 2.',5 15'.
While:
"coefficient": "-0.6"
tokenized as > "",, '0', '. '6'
our decoding process, we always treat the token immediately following the colon as the sign token, trimming the prefix (* "?) before it. For positive numbers, the sign token can be empty string or ++, while negatives it > As result, negative numbers do not suffer disadvantage due to extra token length.
20

---

C Distribution of Kernel Standard Deviations
Figure shows the distribution of kernel standard deviations used in the Gaussian and KDE priors. Both priors tend to produce small o values, contributing to overconfident and poorly calibrated predictions, The median 0.10 for the Gaussian prior and 0.08 for the KDE prior.
0.6
Kernel std a
0.4
LCP a*
0.2
0.10
0.08
0.0
Gaussian
KDE
Figure 7: Distribution of kernel standard deviations for Gaussian and KDE priors.
D Understanding the LLM's Behavior in Reporting Standard Deviations
The LLM (GPT-4o) favors the value 0.1 when reporting standard deviations. a = 0.1 appears in 74% of cases (1,552 out of 2,096 prompts). To better understand this behavior, conducted targeted analysis of the LLM's internal assumptions when predicting a.
We prompted GPT-4o with 50 column pairs whose names were random strings with no semantic meaning (e.g., abe 123, xzy987). This design removes contextual cues, allowing us to observe the model's default behavior under maximum uncertainty. In all 50 cases, the predicted correlation coefficient was exactly zero, and in 46 out of 50 cases, the predicted standard deviation was 0.1. The strong preference for 0.1 even in the absence of context suggests that, when prompted to express its uncertainty as the standard deviation fixed assumptions -such contextual information.
normal distribution, the LLM may default to implicit sample size-rather than adjusting its estimate based
To investigate why = 0.1 is so commonly predicted, we analyzed the distribution of Sample Pearson's correlation coefficient under the assumption that the true correlation p = When data sampled from bivariate normal distribution with zero correlation, the sampling distribution of has following form:
P VTT
fr(r)
(1
for -
where n is the sample size and I(-) is the gamma function. This distribution is bell-shaped, and its standard deviation decreases as n increases. Specifically, the variance is given by Var[r] = - so the standard deviation is SD(r] ア. When = 100, this yields SD(r] Rs 0.1, which aligns with the value most often returned by the LLM.
To test this hypothesis, we asked GPT-4o to explicitly state the sample size it assumes when estimating uncertainty. In all 50 test cases, it responded with 100, confirming that its predicted standard deviation reflects a fixed assumption about sample size rather than context-specific reasoning.
This result suggests that GPT-4o's predicted a reflects aleatoric uncertainty uncertainty due to random sampling around fixed true correlation. The model assumes fixed value of p and estimates how much empirical values of might vary if repeatedly sampled. However, the type of uncertainty we aim capture in this work is primarily epistemic LLM's lack of knowledge about the relationship.
arising from the
21

---

For example, if X represents altitude and Y represents precipitation, the correlation might be 0.7 in the U.S. and 0.6 in Germany. If the LLM does not know which country the data comes from, the true due to sampling variability, but due to missing contextual information.
correlation is ambiguous This is epistemic uncertainty. Unlike aleatoric uncertainty, can be reduced by providing more metadata or context about the table and dataset.
In summary, GPT-4o's use of 0.1 reveals a key limitation of the Gaussian prior: captures fixed form of uncertainty based on sampling variability but fails to account for uncertainty arising from a lack of knowledge such as ambiguity in context missing information. As a result, it often produces overconfident predictions.
E Expected Performance Under Random Ranking
The random ranking baseline yields constant expected Precision @ of approximately 15/115 115 0.13, since each of the 15 expert correlations is equally likely to appear in any of the 115 positions. The corresponding average rank is 58.0, which is the expected value of a discrete uniform distribution over ranks to 11: E[rank] = 1+115- 58
F Multi-hop Reasoning Failure
We analyze a case involving the variables Horsepower and acceleration time (measured as 0-100 km/h time), which are originally negatively correlated: higher horsepower typically leads to faster acceleration and thus shorter time.
We introduce a counterfactual context: "Imagine a world where cars are powered by highly unstable, experimental engines. More horsepower means greater chance of catastrophic failure, forcing drivers to accelerate slowly and cautiously avoid explosions." " Under this context, we would expect the correlation to flip-hiher horsepower should now lead to longer acceleration time, yielding positive correlation.
However, the LLM still predicts a negative correlation. Its explanation is as follows:
"In this hypothetical world, cars with more horsepower more prone catas- trophic failure due to their unstable engines. As a result, drivers of high-horsepower are likely accelerate more slowly and cautiously to avoid triggering explo- sion. This cautious driving behavior would lead to longer 0-100 km/h acceleration times. Conversely, cars with lower horsepower are less likely experience such failures, allowing drivers to accelerate more quickly. Therefore, there is likely a strong negative correlation between horsepower and acceleration time, higher horsepower results in slower acceleration due to the need for caution."
While the model correctly reasons that higher horsepower leads to slower acceleration, it fails at the final reasoning step-mapping slower acceleration to longer time. This results in an incorrect correlation prediction, highlighting multi-hop reasoning failure.
G Additional Related Work
Data Discovery. Modern data discovery systems 2 25 22 30 efficiently compute statistical relationships such as correlations, causality, and mutual information across datasets. They extend beyond analyzing variables within a single dataset to discovering relationships between variables across multiple datasets by automatically transforming and joining different datasets. Specifically, for correlation discovery, Nexus [8] aligns large repositories of spatio-temporal datasets and identifies correlations, while Trummer [0 use RoBERTa classifier to predict whether two variables are correlated based solely on their names. Data discovery systems surface a large number of potential relationships, but helping analysts identify the ones most relevant to their needs remains a key challenge in this field. Our approach, which uses an LLM-elicited prior to rank relationships, serves stepping stone toward addressing this challenge.
Automatic Hypothesis Generation. While data discovery systems identify statistical relationships from structured data that may lead to new hypotheses, complementary line of work 229 31
22

---

32 13 33
focuses on mining unstructured scientific literature. These methods extract semantic knowledge such as entities, links, and claims-from from text, and store this knowledge for further analysis. Some approaches [29 13 construct knowledge graphs and use graph analysis to suggest hypotheses, while others leverage language models to analyze the knowledge and suggest hypotheses Zhou et al. 33 explores combining literatreederived insights with structured data.
directly 32 31.
H
Prompts
H.1
Correlation Prediction Prompt to Construct Logit-based Calibrated Prior
Correlation Prediction Prompt for LCP
Task:
You are given two attributes from a tabular dataset. Your task is to predict the Pearson's correlation coefficient between the two attributes.
Now, begin to solve the following problem:
Attributes:
[attr1)
[attr2)
Source Table: {table}
Descriptions:
Dataset Description: (tbl_desc)
Attribute Descriptions:
fattrik: (varl_desc)
(attr2): [var2_desc)
Respond with your predictions in the following format:
"coefficient"; "<predicted correlation coofficient>",
H.2
Correlation Prediction Prompt to Construct Gaussian Prior
Correlation Prediction Prompt for LCP
Task: You are given two attributes from a tabular dataset. Your task to predict the Pearson's correlation coefficient between the two attributes and estimate your confidence in the predicted correlation by providing the standard deviation as a measure uncertainty. Note that the standard deviation cannot be zero.
Now, begin solve the following problem:
Attributes:
{attri}
{attr2)
Source Table: {table)
Descriptions:
Dataset Description: (tbl_desc)
Attribute Descriptions:
(attri):
fattr2): (var2_desc)
23

---

Respond with your predictions in the following format:
{
"coefficient"; "<predicted correlation coefficient>",
"standard deviation"; "<predicted uncertainty>",
H.3
Generate Contradictory Context
Counterfactual Context Generation Prompt
Task:
You are given two attributes and the expected correlation between them from tabular dataset. Your task is to invent hypothetical context that flips the expected relationship between these attributes.
For example, on Earth, income and education are positively correlated; in an alternat world where education makes people less capable, income and education would be negatively correlated.
Please provide your context in 2-3 concise sentences, avoiding any explicit mention of the correlation.
Now, solve the following:
Attributes:
(attr1)
{attr2)
Source Table: {table}
Descriptions:
Dataset Description: (tbl_desc)
Attribute Descriptions:
(attri): [varl_ ]
{attr2): (var2_desc)
Expected Correlation: {r_ {b_obs}
Respond in JSON:
"new_context";
H.4 Correlation Prediction with Hypothetical context
Correlation Prediction with Hypothetical context
Task:
Given two attributes from tabular dataset and hypothetical context (which may differ from Earth), predict the Pearson correlation coefficient between them.
Guidelines:
Use the scenario described under Context to inform your reasoning.
Return a single floating-pont value the range [-1, 1].
Now, solve the following:
24

---

Context: [context)
Attributes:
{attr1}
[attr2)
Source Table: {table}
Descriptions:
Dataset Description: [tbl_desc)
Attribute Descriptions:
{attrik: (varl_desc)
(attr2): fvar2_dese
Format your answer
{
"coefficient": "<predicted correlation coefficient>", "explanation: "<explanation of the prediction>" }
H.S
Column Semantics Quality Assessment
Column Semantics Quality Assessment Prompt
You are given column name and the context in which it appears. Your task to judge whether the column name clearly and accurately conveys its meaning.
Column Name: {col_name)
Dataset Name: {dataset_name)
Dataset Description: (dataset_desc)
Please respond in JSON using exactly this format:
"valid": "<yes no>"
}
25


Reviews:
[{"review_id": "2SwiB1xZqA", "reviewer": "Reviewer_AT4t", "summary": "The paper presents a method to construct a prior on correlation between any given variables and uses it to filter novel hypotheses from a tabular dataset. It does so by eliciting a prior distribution from an LLM, represented by a mixture of Gaussian which is calibrated (that is, fitted) to training data. In addition to the method, authors construct a new benchmark consisting of variable pairs from two public datasets, the Pearson correlation values, and some metadata. Experiments on the benchmark suggest that the calibration procedure is necessary to get accurate priors, in terms of low bias and high credible interval coverage. Authors also test the method to find hypothesis-worthy variable pairs from a human-annotated dataset.\n\n---\nAfter the response\n\nAuthor rebuttal sufficiently addresses my concern on missing method details and clarifies the scope of the paper. Hence, I increased the score to 5, Accept.", "questions": "1. Authors should discuss the choice of Pearson’s correlation measure and more generally discuss, why the problem setup of considering two real-valued variables is important and interesting? Does the solution extend to measuring correlation after adjusting for confounders via linear models (a popular analytic approach to hypothesis assessment that generalizes two-variable setup). \n\n2. What are some of the formal assumptions required for the relationship between the correlation discovery system, the LLM, and the tabular dataset? Should the discovery system and the LLM be trained on disjoint data? I imagine that the LCP prior will be biased when the hypothesis generation and assessment are done by the same LLM.\n\n3. The for loop in step 5 over top-k tokens was unclear in Algorithm 1. What are the top-k tokens referring to? Could you give an example?\n\n4. Nexus results indicate LCP aligns with human judgements on hypothesis-worthiness. How is hypothesis-worthiness related to the training (calibration, prompt, LLM) objectives for LCP method? Does the prompt for Nexus experiments ask for alignment with human judgements or novelty?\n\n5. Please provide more details on the types of questions in the benchmark. It will be helpful to assess the domains where the method were tested and where it can generalize.\n\nI am open to increase my score to 5 based upon clarifications to the questions and more discussion on addressing weaknesses.", "limitations": "# Limitations\n\nAuthors should discuss the limitations of the problem setup such as reliance on an LLM that has knowledge of the context behind the dataset and an LLM that outputs token probabilities. The method at present is limited to analyzing two variables at a time.\n\n## Minor (no responses are requested)\n\nPlease discuss in more detail the motivation for the hypothesis assessment step and why it is a bottleneck at present, giving examples.\n\nPlease specify how to compute mode of the fitted prior.\n\nAny reason why GPT 4o and Gemini 2.5 models were used for their respective tasks and wee the results worse if Gemini was used for hypothesis assessment and GPT for counterfactual generation?\n\nPlease discuss ways to formalize novelty of a hypothesis.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. The method is presented clearly, and is straight-forward to understand and appreciate. Authors discuss modeling choices like the checks for Normality and need for calibrating sigma.\n2. Papers main strength is the experiment validation.\n- Benchmark is curated with careful thought keeping examples of different correlation values and from two datasets.\n- Metrics make sense and comprehensively quantify accuracy, uncertainty quantification, and retrieval performance.\n- Results convincingly show the advantage of the calibration step by an appropriate choice of baselines.\n3. Results are presented well and visualized by easy-to-read tables and figures.\n\n# Weaknesses\n\n1. The main concern I have is that the scope of the method is unclear. What types of variables (continuous or discrete, missing) are supported by the method? Can we apply it to build priors when there are more than two variables? \nBased on the datasets used in the benchmark, what domains of datasets can we expect the method to perform well? If I have domain-specific knowledge about the dataset that may not be accessible to the LLM, can it be used to influence the prior. Having an LLM be the sole source of contextual knowledge makes it tricky to understand the limitations of the method. \nAuthors should discuss the scope of the datasets the method has been tested on and discuss ways to provide more contextual information for example via the prompt to the LLM. \n\n2. Some finer-grained details of the results should be provided to show the robustness of the method. For instance, how many retries, k in top-k tokens, and decoding steps are needed to get enough correlation coefficient values in order to construct a reasonably good prior. Does the method seem sensitive to transformations of the variables (change in units, scaling by a multiple or addition)? Did the size and type of correlation pairs in the validation set matter to the sigma = 0.4 value? (I am not looking for extensive new experiments but more details on existing experiments.)\n\n3. The application of the method seems to be at odds to the training. The calibration ensures that the fitted parameters put a high likelihood on the observed correlations, whereas in the application of the method, hypotheses that are given low likelihood or more surprise are chosen. How do we reconcile the two objective and distinguish between improbable vs surprising correlations since both are characterized by low likelihoods in the paper? \n\nThe significance of the paper is unclear to me because of the ambiguity in what is the scope of the domains for the method (point 1), how easy it is to use the method (point 2), and how to formalize the objective in hypothesis assessment that the method is aiming to optimize.", "was_revised": true, "final_justification": "Author rebuttal sufficiently addresses my major concerns on missing methods details and clarifies the scope of the paper. The two-variable setup for Pearson correlation is a good first step and applicable broadly. Datasets are sourced from diverse domains which makes the results more generalizable. Additionally, authors support their algorithmic decisions by additional experiments on robustness of sigma to validation set size, and robustness of the downstream performance to different values of sigma. The paper will benefit from more discussion on how to practically distinguish between surprising and implausible correlations. I would suggest authors to caution the readers against the use of the same LLMs in both hypothesis creation and assessment to avoid circular reasoning.\n \nDue to the clarifying responses to most of my concerns, I increased the score to 5, Accept.", "rebuttal": "Thank you so much for your detailed and constructive comments, and we truly appreciate them. We hope the clarification and additional results provided below address your concerns. \n\n## On the scope and motivation of LCP\n> W1: The main concern I have is that the scope of the method is unclear. ...\n\nLCP is defined for pairs of numeric attributes, producing a predictive distribution over the Pearson correlation coefficient. It supports continuous real‑valued variables directly (Pearson’s r requires both variables to be numeric, but they can be either continuous or discrete). Any missing‑value handling is performed upstream. LCP itself only consumes the descriptions of two variables and never their raw values. \n\nExtending this to more than two variables would require eliciting a multivariate distribution (e.g. a Gaussian copula), which we have not experimented with as it is outside our current scope. In principle, one could apply LCP to each pair in a set of n variables, but inferring a coherent joint distribution would need additional machinery (e.g., ensuring positive definiteness).\n\n> Q1: Authors should discuss the choice of Pearson’s correlation measure and more generally discuss, ...\n\n**Rationale for Focusing on Pearson’s Correlation and the Two-Variable Setup**\n\nAnalyzing correlations between two real-valued variables is a foundational step in data exploration and hypothesis generation, especially when working with large structured datasets. In practice, many analysts start with simple pairwise correlations to uncover relationships and generate initial hypotheses for deeper analysis. For instance, underwriting teams use correlations to evaluate new features, and city planners examine correlations between vaccine coverage and factors like grocery access or public transit. We focus on Pearson’s correlation because it is the most widely used measure of linear association in exploratory data analysis [1]. \n\n[1] \"Thirteen Ways to Look at the Correlation Coefficient.\", The American Statistician\n\n**Extension to Partial Correlation and Higher-Order Relationships**\n\nOur framework is readily extensible to partial correlations or more complex relationships, as it only requires natural-language context and elicits a scalar coefficient from the LLM. To adapt the method for partial correlation (i.e., correlation between X and Y after adjusting for confounders Z), one could augment the prompt: “Given these variable descriptions, *after controlling for* covariates Z, what is the expected Pearson correlation between X and Y?”, The rest of the method would remain unchanged. That said, robustly evaluating LLMs’ ability to reason about partial correlations requires dedicated benchmarks and further empirical study, which is out of the scope of this paper. This paper takes the first step by focusing on zero-order (pairwise) correlation.\n\n## On the domains of datasets used in our evaluation\n\n> W1: Based on the datasets used in the benchmark, what domains of datasets can we expect the method to perform well?\n\n> Q5: Please provide more details on the types of questions in the benchmark. ...\n\nIn the original submission, we provided an anonymous repository with all benchmarks. Below is a summary of the dataset domains:\n\n- Cause–Effect Pairs:  Well-known relationships such as altitude vs. temperature and CO2 emissions vs. energy use, spanning physics, biology, and climate science.\n- Kaggle‑Sourced Pairs: Public datasets including relationships such as horsepower vs. fuel efficiency and BMI vs. blood pressure, across social sciences, economics, healthcare, and sports.\n- Chicago Open Data Pairs: Civic and environmental pairs, e.g., crime rate vs. education, air quality vs. hospital admissions.\n\nThese examples span diverse domains, demonstrating both the breadth of our benchmark and the generalizability of our method. We agree that this clarification is important and will include it in the paper.\n\n## On incorporating domain-specific knowledge \n\n> W1: If I have domain-specific knowledge about the dataset that may not be accessible to the LLM, can it be used to influence the prior. ...\n\nWe would like to clarify that in our method, the LLM is not the sole source of contextual knowledge. Our method defines the prior as $p_{LM}(r_{X,Y} | C_{X, Y})$ where $C_{X, Y}$ is entirely user-provided and can include variable descriptions, table summaries, or any domain-specific knowledge.\n\nIn the Nexus evaluation, where data undergo domain-specific transformations, we explicitly describe these steps in the prompt. This demonstrates how users can incorporate domain-specific knowledge—such as aggregation at the census tract level or spatial joins—that goes beyond the LLM’s pretraining.  LCP can effectively leverage such context to rediscover expert-labeled correlations (Section 5). We further explored this in Section 6 by providing counterfactual contexts, testing whether the LLM relies on explicit prompt context or defaults to memorized knowledge.\n\nThat said, we acknowledge that we did not incorporate explicit human priors into LCP in this work. Our main objective was to develop a scalable, fully automated tool for surfacing interesting correlations from large candidate pools. Incorporating richer user-supplied priors is an excellent direction for future research.\n\n## On finer-grained details about the approach and the experiments\n\n> W2: Some finer-grained details of the results should be provided to show the robustness of the method. ...\n\n> Q3: The for loop in step 5 over top-k tokens was unclear in Algorithm 1. What are the top-k tokens referring to? ...\n\nThank you for pointing this out. We will include finer-grained details in the paper. Meanwhile, here are additional details on the robustness of our method:\n\n**Decoding strategy and parameter choices:**\n\nIn Algorithm 1, the “top‑k tokens” are the k most likely tokens produced by the LLM at each step when generating the correlation value. This value is represented by four tokens: the sign, integer part, decimal point, and fractional part (which may cover up to three digits). For example, \"0.56\" in the response {\"coefficient\": \"0.56\"} is tokenized as ' \"', '0', '.', '56'.  \n\nAt each decoding step, we consider the top 20 tokens (the API’s maximum) and enumerate all valid combinations to form candidate correlation values. We use a single decoding pass (no retries), and with k=20, this yields an average of 65 distinct correlation values per variable pair. As shown in Section 4, this provides sufficient diversity for building a robust prior and achieving strong empirical results. We will clarify these details further in the paper.\n\n**Sensitivity to variable transformations:**\n\nBy construction, Pearson’s r is invariant to affine transformations (scaling and shifting) of the input variables. Empirically, in the Nexus evaluation, where data is aggregated and joined across tables, our method remains effective (Section 5).\n\n**Stability of Sigma Across Validation Set Sizes and Pair Types:**\n\nThe validation set for tuning σ is randomly sampled to represent the overall data distribution, without bias toward specific correlation types. We also explored how varying the validation set size affects the optimal σ through subsampling experiments, summarized below:\n\n|Validation set size|Optimal σ|\n|-|-|\n|300|0.40|\n|500|0.38|\n|1000|0.38|\n|2000|0.41|\n\nAs shown, the optimal σ remains stable across different validation set sizes. This supports our explanation in the paper (lines 151–161) that σ corrects for systematic model bias, which is consistent as long as the architecture, prompt, and task remain unchanged. We will include a more detailed sensitivity analysis in the full paper.\n\n> Q4: Nexus results indicate LCP aligns with human judgements on hypothesis-worthiness. ....\n\nHypothesis‐worthiness in our Nexus evaluation is not a separate objective baked into the prompt. It emerges directly from our calibrated LLM prior. Concretely: \n\n1. Calibration aligns LCP with human expectations: We tune σ so that the LLM’s elicited density places high probability on correlations that people usually expect, making the prior a proxy for human intuition.\n\n2. Surprise scores surface interesting pairs: In Nexus, we simply prompt for the Pearson coefficient and rank observed correlations by their likelihood under LCP.\n\n## On the relationship between calibration and surprise\n\n> W3: The application of the method seems to be at odds to the training. ...\n\nThank you for raising this important point. We will clarify the relationship between calibration and surprise in LCP. In our framework, calibration and surprise are complementary: calibration aligns the LLM prior with broadly shared human expectations by assigning high likelihood to well-known or intuitive correlations (e.g., “altitude vs. temperature,” “horsepower vs. fuel efficiency”). The calibration set is built to reflect these canonical relationships.\n\nAfter calibration, the prior encodes what is expected in the domain. At application time, any correlation assigned low likelihood by the prior is flagged as surprising, meaning it deviates from established patterns. We do not distinguish between “improbable” and “surprising”—both are defined as unlikely under the calibrated prior. Thus, the prior serves both to encode common knowledge (via calibration) and to flag novel or non-obvious patterns (via low likelihood, i.e., high surprise). We will clarify this relationship in the full paper.\n\n> Q2: What are some of the formal assumptions required for ...\n\nTo clarify, our correlation discovery system does not use an LLM for hypothesis generation—it computes empirical correlations directly from tabular data. The LLM is only used for hypothesis assessment, so there is no risk of self-evaluation or circular reasoning. By “hypothesis generation,” we simply refer to interpreting discovered correlations (e.g., inferring that higher crime may lower housing prices from a strong negative correlation)."}, {"review_id": "SAVBa7tUaO", "reviewer": "Reviewer_teNe", "summary": "The paper uses LLMs as priors to assess hypotheses about the correlation between two variables. The authors propose Logit-based Calibrated Prior (LCP), a method that extracts a calibrated prior from LLM using information from the logits. Given contextual information about the correlation to be assessed, the LLM is tasked to predict a correlation coefficient. The top-k values and their corresponding probabilities are directly read from the logits to form a density map. Compared with baselines, the approach achieves better sign accuracy, reduces error and has a density closer to the true observations. When using the method to estimate interesting real-world hypotheses, LCP better aligns with human experts, compared to baselines.", "questions": "1. How do you address the inherent bias of LLMs towards specific numerical values (see weakness 1)?\n\n2. How does varying the value of the kernel standard deviation $\\sigma$ affects the robustness of the method in practice?\n\n3. Can you provide examples of counterfactual assessments (as described in Section 6)?\n\n4. Existing work relies on iterative improvements over a prior hypothesis (e.g. [1,3] above). Have you investigated whether integrating the output prior into the prompt as an initial guess to be critiqued can lead to improvements over future iterations?\n\n5. Have you studied the effect of the temperature hyperparameter on the output density?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed idea is novel and well-motivated: given the large amount of data learned by LLMs and the prevalence of questions of variable correlation in scientific applications, this is an interesting and potentially impactful approach.\n\n2. The method is theoretically sound and the extensive set of experiments support the claim made. I find the comparison with human experts particularly compelling.\n\n3. The approach differs greatly from existing work on automated hypothesis assessment that typically prompts an LLM to generate an hypothesis and uses a second LLM context window to self-critique the hypotheses (e.g. [1,2,3]). Specifically, the use of the logits to provide an uncertainty metric and a non-Gaussian density function in addition to the correlation score is an interesting and more principled approach.\n\n\nWeaknesses:\n\n1. The method asks the LLM to directly provide the correlation score and extracts a distribution over possible values instead of a single output by looking at the top-k token distribution from the logits. However, LLMs are notoriously biased when generating numerical outputs, specific numbers encompassing more weight because they occur in more diverse contexts (e.g. pi or rounded decimals). This issue is not addressed and may affect the reliance on the distribution over the $r_j$ scores (in Algorithm 1).\n\n2. While hypothesis assessment is at the core of the motivations, a small portion of the paper is actually dedicated to this problem (Section 5) and the core of the method mainly focuses on calibration. I think that the paper would gain from either reducing the emphasis on hypothesis assessment and developing more downstream tasks, or including more datasets/experiments on hypothesis assessment. \n\n\n[1] Huang, K., Jin, Y., Li, R., Li, M. Y., Candès, E., & Leskovec, J. (2025). Automated hypothesis validation with agentic sequential falsifications. arXiv preprint arXiv:2502.09858.\n\n[2] Capstick, A., Krishnan, R., & Barnaghi, P. (2025). AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling. In Forty-second International Conference on Machine Learning.\n\n[3] Xiong, G., Xie, E., Shariatmadari, A. H., Guo, S., Bekiranov, S., & Zhang, A. (2024). Improving scientific hypothesis generation with knowledge grounded large language models. arXiv preprint arXiv:2411.02382.", "was_revised": true, "final_justification": "The discussion with the authors resolved most of my concerns and answered my questions:\n-  the effect of temperature and $\\sigma\" hyperparameters has been asnwered\n- the bias of LLMs towards numerical outputs has been resolved\n- the discussion on the use of LCP for CoT or iterative situations answered my questions\n\nOverall, this is a very interesting work and I keep my positive score.", "rebuttal": "We thank the reviewer for the insightful feedback and detailed comments! Your feedback will help us improve the paper significantly.\n\n## On the inherent bias of LLMs towards specific numeric values\n\n> Weakness 1: The method asks the LLM to directly provide the correlation score and extracts a distribution over possible values instead of a single output by looking at the top-k token distribution from the logits. However, LLMs are notoriously biased when generating numerical outputs, specific numbers encompassing more weight because they occur in more diverse contexts (e.g. pi or rounded decimals). This issue is not addressed and may affect the reliance on the distribution over the scores (in Algorithm 1).\n\n> Question 1: How do you address the inherent bias of LLMs towards specific numerical values (see weakness 1)?\n\nThank you for raising this important point. We acknowledge that LLMs exhibit bias toward rounded decimals—for example, in our experiments, among 293 predictions in the range [0.3, 0.4), 282 were exactly 0.30. To mitigate this, we apply kernel smoothing to the decoded distribution, which spreads probability mass across nearby values and avoids sharp spikes at rounded numbers like 0.30. Additionally, the objective of LCP is to capture general human expectations rather than produce exact point estimates. Thus, biases toward rounded values do not significantly impact downstream robustness. For instance, if the LCP prior is centered at 0.30 and the observed correlation is 0.35, kernel smoothing ensures that 0.35 still receives high probability, resulting in low surprise. We will include this clarification in the full paper.\n\n## On the sensitivity of LCP to kernel standard deviation\n\n> Question 2: How does varying the value of the kernel standard deviation affects the robustness of the method in practice?\n\nWe conducted a preliminary analysis to test the sensitivity of LCP to kernel standard deviation. We evaluated three kernel standard deviation ($\\sigma$) values (0.2, 0.4, 0.6) on our benchmark and the Nexus hypothesis‑worthiness task; the result is as follows:\n\n| $\\sigma$    | Sign Acc. | Mean $\\|\\bar{r}-r_{obs}\\|$ | 95% Coverage | Pecision@5  | Pecision@10 | Pecision@15 |\n| ---- | --------- | --------------- | ------------ | ---- | ---- | ---- |\n| 0.2  | 78.8%     | 0.27            | 77.4%        | 0.60 | 0.80 | 0.60 |\n| 0.4  | 78.8%     | 0.26            | 89.2%        | 0.60 | 0.80 | 0.60 |\n| 0.6  | 78.8%     | 0.33            | 91.9%        | 0.60 | 0.80 | 0.60 |\n\nIncreasing $\\sigma$ leads to a smoother prior, which improves coverage of credible intervals (from 77.4% to 91.9%) but also increases the mean absolute error between distribution mean and observed correlations (from 0.26 to 0.33). Sign accuracy remains stable across all settings.\n\nImportantly, while the calibration of the prior is sensitive to $\\sigma$, the relative ranking of surprising correlations, which drives downstream tasks like hypothesis-worthy pair discovery, remains robust across a wide range of $\\sigma$ values. This is because moderate changes in smoothing shift the absolute likelihoods but generally preserve the order of correlations. In the above Nexus evaluation, all tested $\\sigma$ produced the same Precision@k.\n\nChoosing $\\sigma$ involves a trade-off between sharpness and calibration: smaller $\\sigma$ yields narrower, less calibrated intervals, while larger $\\sigma$ offers better calibration but more conservative estimates. We select $\\sigma$ = 0.4 as it achieves strong calibration without impacting discovery performance. Overall, our method’s ranking-based results are robust to changes in $\\sigma$, but proper calibration remains essential for trustworthy probability estimates and credible intervals.\n\n## On counterfactual assessments\n\n> Question 3: Can you provide examples of counterfactual assessments (as described in Section 6)?\n\nWe provide a full example in Appendix E and will further clarify this in the full version. Additionally, all counterfactual contexts are open-sourced in the benchmark folder of our anonymous repository (shared in the original submission). Here are two representative examples:\n\n1. An example where the LLM predicts correctly: “drinking water access” and “infant mortality” are originally negatively correlated.\n\n*Counterfactual context*: “Imagine a world where every source of drinking water is heavily contaminated, so regions with the greatest water access suffer the highest rates of health problems.”\n\n2. An example where the LLM predicts incorrectly (Appendix E): Horsepower and acceleration time (measured as 0–100 km/h time), which are originally negatively correlated. \n\n*Counterfactual context:* “Imagine a world where cars are powered by highly unstable, experimental engines. More horsepower means a greater chance of catastrophic failure, forcing drivers to accelerate slowly and cautiously to avoid explosions.”\n\n## On Iterative approaches\n\n> Question 4: Existing work relies on iterative improvements over a prior hypothesis (e.g. [1,3] above). Have you investigated whether integrating the output prior into the prompt as an initial guess to be critiqued can lead to improvements over future iterations?\n\nUsing the LCP output as a “first guess” in an iterative critique loop is indeed an interesting idea. In the current work we focus solely on single‑pass prior elicitation and calibration, so we haven’t yet experimented with feeding the fitted prior back into the prompt for successive refinements. That said, we want to clarify that across our benchmarks, a single‐pass LCP call already achieves strong performance on empirical correlations. More importantly, LCP is meant to scale over large corpora of variable pairs: requiring only one LLM invocation per pair keeps both latency and cost manageable. We include this discussion in the full paper.\n\n## The effect of temperature on LCP\n\n> Question 5: Have you studied the effect of the temperature hyperparameter on the output density?\n\nWe have not explicitly varied the temperature hyperparameter in our experiments; instead, we use the default (unscaled) logits from the model to compute the output distribution. In our approach, we control the sharpness of the resulting density through the kernel bandwidth parameter sigma during the calibration phase. Conceptually, both temperature and sigma affect the spread of the distribution (but via different mechanisms): lowering the temperature makes the output probabilities more peaked, similar to using a smaller $\\sigma$ for kernel smoothing. Our results show that overly sharp distributions lead to poor calibration. We chose to adjust only the kernel bandwidth because this approach is both more transparent and more convenient: smoothing with $\\sigma$ allows us to tune the density shape directly in a single step, without having to balance two interacting parameters (temperature and bandwidth). Moreover, even if temperature were adjusted, kernel smoothing would still be necessary to produce a continuous density, so controlling everything via $\\sigma$ is simpler and more interpretable.\n\n## On the structure of the paper\n> Weakness 2: While hypothesis assessment is at the core of the motivations, a small portion of the paper is actually dedicated to this problem (Section 5) and the core of the method mainly focuses on calibration. I think that the paper would gain from either reducing the emphasis on hypothesis assessment and developing more downstream tasks, or including more datasets/experiments on hypothesis assessment.\n\nWe agree that hypothesis assessment is a central motivation and appreciate the suggestion to strengthen this aspect. In the current paper, we prioritized the calibration part, as it forms the foundation for reliable hypothesis assessment. We recognize that expanding downstream experiments or including more diverse datasets for hypothesis assessment would further strengthen the paper and provide a more comprehensive evaluation. This is a valuable direction for future work, and we plan to extend our framework and experiments in this area. Thank you for this thoughtful feedback."}, {"review_id": "en7YhDU78C", "reviewer": "Reviewer_YsvF", "summary": "This paper considers a task where an LLM is employed to assess the generated hypothesis by its a priori. To be specific, the Logit-based Calibrated Prior is proposed to find out unexpected correlations. \n\nThe proposed method first extracts top-K tokens as well as their probabilities calculated from logits, then constructs a prior distribution with a Gaussian kernel.\n\nThe paper also constructs a benchmark that consists of about two thousand paired variables as well as their Pearson correlations from real-world sources. Compared with the baseline prior distribution, LCP presents better performance on predicting empirical correlations using GPT-4o. Also, counterfactual experiments are conducted to evaluate the LCP's reliance on memorization.", "questions": "- Would LCP be sensitive to the kernel functions and the sample size used to estimate the distribution? \n- In the prompts, what does `Source Table` mean? Does that mean the tabular data is also a part of the input?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes an interesting setting to incorporate LLMs a priori to analyze the correlations from tabular datasets. \n- The paper presents clear technical details, and the method is intuitive and well-motivated.\n\n**Weaknesses** \n- Such assessment depends on LLMs' own prior. which may not be consistent across different LLMs; and furthermore, it lacks a clear explanation (the prior from LLMs itself is still mysterious ) about why some correlations are selected, and some are not.", "was_revised": true, "final_justification": "The authors' response has sufficiently addressed my concerns.\n- The experiments on kernel functions and sample size have strengthened the solidity of their methods.\n- The response to Weakness 1 is reasonable. I understand the entire solution may be beyond one single paper.\n\nTherefore, I increase my score to 5. Congratulations again for your interesting work.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and valuable suggestions.\n\n> Question 1: Would LCP be sensitive to the kernel functions and the sample size used to estimate the distribution?\n\nThank you for this important question. We conducted a preliminary sensitivity analysis to assess the effect of both kernel function choice and sample size on LCP’s robustness.\n\n**Kernel Function:** \n\nWe compared three common kernel functions (Uniform, Epanechnikov, and Triangle) against the Gaussian kernel used in the main paper. For each kernel, we tuned the bandwidth parameter $\\sigma$ on the same held-out validation set and evaluated on a benchmark of 2,096 correlated pairs:\n\n| Kernel                          | Sign Accuracy |  $\\|\\bar{r} - r_{\\text{obs}}\\|$ | 95% Coverage |\n| ------------------------------- | ------------- | ------------------ | ------------ |\n| Uniform Kernel                  | 76.4%         | 0.32               | 82.1%        |\n| Epanechnikov Kernel             | 78.8%         | 0.26               | 89.7%        |\n| Triangle Kernel                 | 78.8%         | 0.31               | 90.9%        |\n| Gaussian Kernel (for reference) | 78.8%         | 0.26               | 89.2%        |\n\nThe results show that LCP’s performance is stable across different kernel functions, maintaining strong accuracy and calibration. This indicates that LCP is not sensitive to the specific choice of kernel function. However, because each kernel has a different shape, the optimal bandwidth parameter $\\sigma^*$ should be re-tuned using a held-out validation set for best results..\n\n**Sample Size:** We interpret sample size as the number of distinct correlation values collected from top‑k decoding. In our setup (k=20), we enumerate an average of 65 values per variable pair. Section 4 shows this is sufficient for constructing a robust prior and achieving strong empirical results.\n\nAdditionally, we find that the optimal $\\sigma$ for smoothing is stable across different validation set sizes, as shown below:\n\n| Validation set size | Optimal $\\sigma$ |\n| ------------------- | ------------- |\n| 300                 | 0.40          |\n| 500                 | 0.38          |\n| 1000                | 0.38          |\n| 2000                | 0.41          |\n\nWe hope these clarifications are helpful. Please let us know if you would like further details.\n\n> Question 2: In the prompts, what does Source Table mean? Does that mean the tabular data is also a part of the input?\n\n`Source Table` is the name of the table. The tabular data itself is not part of the input. We will clarify this in the final version.\n\n> Weakness 1: Such assessment depends on LLMs' own prior. which may not be consistent across different LLMs; and furthermore, it lacks a clear explanation (the prior from LLMs itself is still mysterious ) about why some correlations are selected, and some are not.\n\nWe appreciate the reviewer’s point about the limitations of relying on LLMs and have discussed this in Section 9. Our primary motivation in this work was to develop LCP as a scalable, fully automated tool for surfacing interesting correlations from large candidate pools, which necessitated a human-out-of-the-loop design. Before applying LCP to identify hypothesis-worthy correlations, we perform a calibration phase to ensure that the LLM prior aligns with broadly shared human expectations, assigning high likelihood to commonly known or intuitive correlations. As shown in Section 4, this calibrated prior achieves strong accuracy and low mean absolute error on widely recognized correlations. At application time, correlations that receive low likelihood under the prior are deemed surprising, representing deviations from these established expectations. While the internal formation of the LLM’s prior remains somewhat opaque, the process by which LCP defines and ranks surprise is transparent to the users."}, {"review_id": "xfxvxe9lgv", "reviewer": "Reviewer_7HLQ", "summary": "The authors present a novel algorithm of \"automated hypothesis assessment\" to automatically detect interesting, e.g. surprising, correlations within complex systems that yield large sets of variables. The described algorithm assumes a prior hypothesis discovery step that, for example, mines possible correlations between any two variables from data. Instead of having a human expert manually filter the discovered correlations, the authors propose to utilize the output logits of a LLM to detect `surprising' correlations values, that deviate from the prior expectation of the LLM. \n\n\nThe authors utilize variable names and context descriptions and let the LLM output the expected correlation strength in a structured format. During generation of the correlation number string, the logits of all tokens that may form valid numerical values are considered to normalize the (summed log) logit values of the most likely generation, giving rise to a distribution over multiple predicted correlational values and their assigned probabilities. The authors smooth the obtained discrete values into a mixture of Gaussians using the obtained correlation values as mean and an fixed sigma value which is computed on a hold-out adjustment set. The authors test their approach on 2096 samples of the well known CauseEffectPairs dataset and several Kaggle datasets used in a previous work. They use GPT-4o as the sole LLM and compare against uniform prior, a Gaussian prior with direct LLM predicted variance and a variance selected via Scott's rule, and measure sign accuracy of the correlation, absolute error and Shannon information. The presented LCP (Logit-based Calibrated Prior) method seems to feature good accuracy and calibration (percentage of samples selected as 'surprising'), while the baselines perform significantly worse in at least one of the metrics. Additionally, the authors furthermore compare the accuracy binary correlation decisions of their method to that of to BERT- and RoBERTa-based classifiers and show that LCP can consistently outperform the baselines. The authors inspect the ability to re-discover 15 expert annotated correlations from Nexus / Chicago open data within a set of 100 an additional random variables with reasonable precision and average ranking. Finally, the authors provide contradicting/counterfactual contexts that inverse predicted directions, to inspect possible effects of memorization. The methods shows good performance with over 95% accuracy.", "questions": "I would kindly ask the authors comment on the following questions regarding the previously mentioned weaknesses:\n\n1) **Narrow Variance Prediction / Naive Baseline:** I would like to ask the authors to comment on whether performance degradations of the baselines are only due to too narrow choice of variance. Do other baselines exists that do now suffer from such problems? Would a naive approach that directly asks for the 'surprise' levels or hypothesis-worthiness provide a more fair baseline?\n2) **Token Probabilities:** Could the authors comment of the impact of a single token sequence on token probabilities. Would a multi-beam search provide more truthful estimates?\n3) **Sequence Length Bias:**  How much is the previously mentioned sequence length bias expected to impact performance, by putting, e.g., negative numbers at a disadvantage due to the additional minus sign. Would the consideration of the average token probability be suited to fix this issue?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "To the best of my knowledge, the paper presents an original and novel algorithm that tackles the problem of automated hypothesis filtering. The approach and experiments are described clearly and different experiments are well setup. The presented approach might be a strong contribution towards fully-automated hypothesis testing pipelines. The main weakness of the work might be the use of rather weak baselines which all seem to suffer from the same problem of variance miscalibration.\n\n\n\n**Strengths**\n\n1) The overall algorithm and individual steps are well described and sound. The presented methodology seems to be novel and shows strong performance across experiments. The utilized datasets are well curated and balanced by cleaning uninformative of variable names, insignificant correlation and resampling to balance for different levels correlations strengths.\n2) A main benefit of the presented algorithm is that it only requires a calibration, but no retraining/fine-tuning step with off-the-shelf LLM, while simultaneously exceeding the performance of fine-tuned baselines. Apart from pure performance considerations, the authors provide further insightful experiments on the re-discovery of expert-annotated correlations and the provision of contradicting contextual descriptions. These experiments give further insights into biases in over- or underestimation of correlation strengths and the role memorization versus contextual reasoning within the LLM, providing a clear frame for use-cases of the proposed method.\n3) The authors are clear about possible benefits and limitations of the presented approach in discussion and through the presented experiments. Specifically the focus on alignment of LLM and human judgments, which eventually is the main concern for the successful use of such systems, is well addressed throughout the whole paper, but also considered specifically in the experiment of section 5.\n\n\n\n**Weaknesses**\n\n1) In section 4, the authors compare against baselines which mostly follow the purpose of constructing a uniform or mixture of Gaussian distributions from LLM predictions. KDE and Gaussian baselines  mainly seem to suffer from choosing a too narrow variance (which is indicated by a 95%-coverage of over 92% of the random/uniform baseline, and with the absolute mean distance errors $|r-r_{obs}|$ of KDE, Gaussian and LCP being the same value, indicating a good overall mean prediction of the LLM). I was wondering whether the most naive way of simply directly prompting 'to which extend a correlation with Pearson coefficient of [CORR] between variables [X] and [Y] within the provided context is surprising/hypothesis-worthy.' might not provide reasonable baseline and eliminate performance degradations due to incorrectly scaled variances.\n2) From algorithm 1 and the description of it Sec. 2 I get the impression that parsing of correlational values is done from a single sequence of token generations with the corresponding captured logits. Wouldn't this procedure immediately skew the probability of alternative token sequences in the second step, as the next tokens are now conditioned on the single chosen token of the first step? Would a branching beam-search for individual paths of token generations capture and normalize probabilities more truthfully?\n3) The authors multiply individual token probabilities to compute the overall sequence probability. This provides a bias towards shorter sequences, by assigning decreasing probabilities for longer sequences. Coincidentally, this puts negative correlations at a disadvantage. Is this something that needs to be corrected for, e.g. by considering the average token probability?\n\n\n\n**Minor**\n\n* The authors attribute the overly narrow sigma predicted by the baselines to be an artifacts of aleatoric- versus limited considerations of epistemic uncertainty. While the authors discussion on the emergence of such values under particular assumptions in Appendix C, I would like to argue that LLM are usually not \"aware\" of such differences (if not explicitly tasked to provide a rationale about it). LLMs are commonly assumed to be 'stochastic parrots' with limited ability to understand the underlying (causal) mechanisms that lead to particular outcomes [1-6]. The values of n=100 and sigma=0.1 might simply be an memorization artifact that might appear in training data, e.g. as exemplary sufficient amount of samples used in many descriptions. While the provided theory is certainly interesting, I would like to recommend to briefly mention the much more simple theory of a mere memorization/dataset bias.\n* While the authors successfully demonstrate the working of their method, they used GPT-4o as the sole model under consideration. While I understand that compute limitation might apply and I do not consider it a downside in case results can not be provided for the rebuttal, general insights on the applicability of the approach and overall strength of the paper might be improved by providing results on other (open) reasoning models such as the recent DeepSeek, Qwen models.\n* The authors perform an experiment in Sec. 5 to retrieve \"expert-flagged, hypothesis-worthy correlations\" and present average precision and rank, what are the top identifies samples and do they intuitively make sense?\n* I would like to recommend to the authors to mention the full name of MCC once in line 253.\n\n\n\n[1] Bender, Emily M., et al. \"On the dangers of stochastic parrots: Can language models be too big?.\" *Proceedings of the 2021 ACM conference on fairness, accountability, and transparency*. 2021.\n\n[2] Zečević, Matej, et al. \"Causal Parrots: Large Language Models May Talk Causality But Are Not Causal.\" *Transactions on Machine Learning Research*. 2023\n\n[3] Jin, Zhijing, et al. \"Can large language models infer causation from correlation?.\" *arXiv preprint arXiv:2306.05836* (2023).\n\n[4] Kiciman, Emre, et al. \"Causal reasoning and large language models: Opening a new frontier for causality.\" *Transactions on Machine Learning Research* (2023).\n\n[5] Liu, Hanmeng, et al. \"Evaluating the logical reasoning ability of chatgpt and gpt-4.\" arXiv preprint arXiv:2304.03439 (2023).\n\n[6] Barman, Kristian Gonzalez, et al. \"Towards a benchmark for scientific understanding in humans and machines.\" Minds and Machines 34.1 (2024): 6.", "was_revised": true, "final_justification": "I consider all of my remarked points on sequence length bias, single token sequence / narrow variance predictions and naive baselines to be answered well and comprehensively. The new results shows that the approach sufficiently sets itself apart from naive approaches and discussion on sequence bias/variance addresses possible limitations sufficiently.", "rebuttal": "Thank you for the constructive feedback! We sincerely appreciate all your comments, which have been invaluable in improving the paper. We hope the following clarifications and additional results address your concerns.\n\n## On Sequence length bias\n\n> Weakness 3: The authors multiply individual token probabilities to compute the overall sequence probability. This provides a bias towards shorter sequences, by assigning decreasing probabilities for longer sequences. Coincidentally, this puts negative correlations at a disadvantage. Is this something that needs to be corrected for, e.g. by considering the average token probability?\n\n> Question 3: Sequence Length Bias: How much is the previously mentioned sequence length bias expected to impact performance, by putting, e.g., negative numbers at a disadvantage due to the additional minus sign. Would the consideration of the average token probability be suited to fix this issue?\n\nWe’d like to first clarify the question regarding the sequence length bias. \n\nIn our setup that uses the GPT-4o tokenizer, both positive and negative correlation values are represented with the same number of tokens. Specifically, both forms include four tokens: sign, integer part, decimal point, and fractional part.\n\nFor example, the response\n\n{\n\n \"coefficient\": \"0.5\"\n\n}\n\nis tokenized as `' \"'`, `'0'`, `'.'`, `'5'`. \n\nwhile\n\n{\n\n \"coefficient\": \"-0.6\"\n\n}\n\nis tokenized as `' \"-'`, `'0'`, `'.'`, `'6'`.\n\nIn our decoding process, we always treat the token immediately following the colon as the sign token, trimming the prefix (`' \"'`) before it. For positive numbers, the sign token can be an empty string or '+', while for negatives it is '-'. As a result, negative numbers do not suffer a disadvantage due to extra token length. \n\nWe will add the above clarification to the paper.\n\n## On the impact of a single token sequence on token probabilities\n\n> Weakness 2: From algorithm 1 and the description of it Sec. 2 I get the impression that parsing of correlational values is done from a single sequence of token generations with the corresponding captured logits. Wouldn't this procedure immediately skew the probability of alternative token sequences in the second step, as the next tokens are now conditioned on the single chosen token of the first step? Would a branching beam-search for individual paths of token generations capture and normalize probabilities more truthfully?\n\n> Question 2: Token Probabilities: Could the authors comment of the impact of a single token sequence on token probabilities. Would a multi-beam search provide more truthful estimates?\n\nThank you for highlighting this important point. We agree that a full multi-beam search would provide a more faithful estimate of the true joint token probability, as it considers all possible token branches at each decoding step. In our current single‑pass approach, we first sample the sign prefix and then condition subsequent decoding steps on that choice, approximating\n\n$p(sign, maginitude)  = p(sign)p(magnitude | sign)$\n\nrather than the true joint. We will clarify this limitation in the paper.\n\nHowever, in practice, we find that this approximation works well for three reasons:\n\n1. **High sign accuracy:** As shown in Section 4, the LLM achieves a sign accuracy of 78.8% across our benchmark. This means that in the vast majority of cases, the single-pass decoding correctly captures the true sign of the correlation, and thus most of the probability mass is assigned to the correct branch.\n2. **High sign confidence**: In addition to being accurate, the LLM is typically very confident in its sign prediction. Across the 2,096 correlations in our benchmark, the median probability gap between the two signs is 99.8% (77.7% on average). This indicates that the model significantly prefers one sign direction, so the probability mass of the alternative branch is negligible. Therefore, the joint probability missed by not performing beam search on the less likely sign branch has minimal practical impact.\n3. **Scalability**: A full branching beam search would require exponentially more decoding paths—and, in practice, significantly more LLM calls—as the beam width increases at each token position. Since LCP is designed to process tens of thousands of variable pairs, such an approach would be expensive. The single‑pass strategy thus provides a practical balance, maintaining strong empirical performance while ensuring computational efficiency.\n\n## On Narrow Variance Prediction and Naive Baselines\n\n> Weakness 1: In section 4, the authors compare against baselines which mostly follow the purpose of constructing a uniform or mixture of Gaussian distributions from LLM predictions. KDE and Gaussian baselines mainly seem to suffer from choosing a too narrow variance (which is indicated by a 95%-coverage of over 92% of the random/uniform baseline, and with the absolute mean distance errors of KDE, Gaussian and LCP being the same value, indicating a good overall mean prediction of the LLM). I was wondering whether the most naive way of simply directly prompting 'to which extend a correlation with Pearson coefficient of [CORR] between variables [X] and [Y] within the provided context is surprising/hypothesis-worthy.' might not provide reasonable baseline and eliminate performance degradations due to incorrectly scaled variances.\n\n> Question 1: Narrow Variance Prediction / Naive Baseline: I would like to ask the authors to comment on whether performance degradations of the baselines are only due to too narrow choice of variance. Do other baselines exists that do now suffer from such problems? Would a naive approach that directly asks for the 'surprise' levels or hypothesis-worthiness provide a more fair baseline?\n\n**On Narrow Variance Prediction** \nThe performance degradations of the KDE and Gaussian baselines are indeed mainly due to their overly narrow variance. As shown in Figure 3, both achieve good accuracy but suffer from poor calibration, resulting in overconfident predictions. Specifically, the Gaussian prior tends to have lower sign accuracy than KDE and LCP because it uses only the verbalized mean prediction from the LLM, ignoring other possible values in the underlying distribution. This makes it more prone to sign errors, especially when the correlation magnitude is small (e.g., < 0.2).\n\n**On Naive “Surprise” Baselines:**\n\nWe thank the reviewer for proposing alternative baselines. To evaluate this, we implemented two new baselines on our benchmark:\n\n1. **Binary Surprise:** The LLM receives all relevant context and the observed correlation coefficient, and is asked to classify the correlation as “surprising” or “not surprising”.\n2. **Level Surprise:** The LLM receives the same context and observed correlation and is asked to rate the surprise level from 1 (least surprising) to 5 (most surprising).\n\nWe then ranked correlations by the reported surprise scores. The results are:\n\n| Method          | Precision@5 | Precision@10 | Precision@15 |\n| --------------- | ----------- | ------------ | ------------ |\n| Binary Surprise | 0.4         | 0.2          | 0.13         |\n| Level Surprise  | **0.6**     | 0.5          | 0.33         |\n| LCP             | **0.6**     | **0.8**      | **0.6**      |\n\nLCP outperforms both naive baselines. The verbalized surprise approaches have two main drawbacks:\n\n1. **Lack of transparency:** The LLM’s “surprise” assessment is opaque and may not consistently align with user expectations. In contrast, LCP is calibrated on a validation set of common-sense correlations, making its definition of “surprise” more interpretable and transparent.\n2. **Limited granularity:** Verbalized surprise produces only coarse ratings, leading to many ties in ranking and degrading retrieval effectiveness. LCP, by constructing a continuous predictive distribution, supports more nuanced and flexible assessment.\n\nMoreover, LCP is more general: users can define discrete surprise levels if desired, but still benefit from the underlying continuous prior.\n\nWe agree with the reviewer that including these baselines is important for improving the paper. We will report the new baseline results in the final paper.\n\n## On the top-ranked correlations\n\n> The authors perform an experiment in Sec. 5 to retrieve \"expert-flagged, hypothesis-worthy correlations\" and present average precision and rank, what are the top identifies samples and do they intuitively make sense?\n\nThe top identified correlations make sense intuitively. For example, one is the strong positive correlation between the number of Divvy bike docks (Divvy is Chicago’s public bike-sharing system) and a community’s socio-economic status, suggesting that bike stations are more concentrated in wealthier areas. This hypothesis has been investigated by domain experts in [1]. We will include a more detailed discussion in the full paper.\n\n[1] \"Riding tandem: Does cycling infrastructure investment mirror gentrification and privilege in Portland, OR and Chicago, IL?\", Research in Transportation Economics."}]

Author rebuttal:

Meta-review: I wanted to thank the authors for such a productive rebuttal phase. Reviewers all reached consensus of accepting the paper, which I also support.
Decision: Accept (poster)