Title: Large Language Bayes
Venue: NeurIPS 2025
Accepted: yes
Keywords: bayesian inference

variational inference
Abstract: Many domain experts do not have the time or expertise to write formal Bayesian models. This paper takes an informal problem description as input, and combines a large language model and a probabilistic programming language to define a joint distribution over formal models, latent variables, and data. A posterior over latent variables follows by conditioning on observed data and integrating over formal models. This presents a challenging inference problem. We suggest an inference recipe that amounts to generating many formal models from the large language model, performing approximate inference on each, and then doing a weighted aver- age. This is justified and analyzed as a combination of self-normalized importance sampling, MCMC, and importance-weighted variational inference. Experimentally, this produces sensible predictions from only data and an informal problem description, without the need to specify a formal model.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces Large Language Bayes (LLB), a method for automating Bayesian modeling from natural language descriptions using a large language model (LLM).\nThe primary objective of the paper is to generate multiple candidate models based on a user's prompt. Then, a final prediction is computed by creating a weighted average of the posteriors from each model. The weight of each model is determined by its marginal likelihood, which measures how well observed data is fitted.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n    - The paper presents an accessible tool for Bayesian modelling using LLMs through a theoretical analysis.\n    - Experiment results are encouraging.\n- Weaknesses\n    - LLM model sampling appears to be highly sensitive to multiple factors, including prompting, sequencing, and model. The manuscript could mitigate practical concerns related to sampling sensitivity with further experiments. (See Questions).\n    - The result in Theorem 2 - addressing the posterior approximation quality - isn’t feasible in practice, given the ample search space of models.\n\nQUESTIONS:\n1. When sampling PPL models, are the output generations individualized, meaning they each have their own “message thread”?\n    * If yes, how did the authors deal with the LLM generating/suggesting the same model multiple times? Any statistics on this? Wouldn’t this behavior possibly be a bias built into the LLM? Do you have any thoughts on how to handle this when writing the model with the PPL?\n    * If not, wouldn’t prior model output influence subsequent ones? That is, creating a chain of biases.\n2. How would this method compare with traditional Bayesian learning, where the optimization problem is to find the model parameters from a distribution that could, potentially, have a prior?\n3. Any thoughts on using the approximate ~p(z|x,m) as a score for dynamically prompting the LLM? In an iterative approach, one could use the posterior approximation as a guide for the LLM prompt template, aiming for more accurate models.\n\nLIMITATIONS:\n- Reproducibility of results is dependent on LLM size/quality and prompting\n- Inference demands limit the method to smaller use-cases", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper proposes a procedure by which to use LLMs alongside Stan or another PPL for posing probabilistic inference problems: prompt the language-model with an informal description of the generative model and the target variables for posterior prediction, have it generate probabilistic programs, and then ensemble those probabilistic programs to predict the target variables. They demonstrate that this can work on some example problems.

STRENGTHS AND WEAKNESSES:
Strength: manages to still research Bayesian inference while making use of the latest computational capabilities

Weakness: model averaging as a weighting strategy can suffer from high variance and unstable weighting in the case of model misspecification (see: https://proceedings.mlr.press/v238/reichelt24a.html), which is almost sure to happen when converting from natural language to probabilistic program via LLMs.  Self-normalized importance sampling similarly has bias and variance issues of its own.

Strength: the authors created a fresh benchmark of probabilistic inference problems instead of using standard ones that could be in training corpora.

QUESTIONS:
Can the authors systematically characterize which varieties of approximate inference they tried out in order to get the scheme we see in this manuscript?  How do the experimental results vary if hyperparameters or the choice of Monte Carlo algorithms are varied?

LIMITATIONS:
I would like to see an answer to my questions above to really get a sense of whether the inference results shown are robust enough to "deploy" to people who are not PPL full-timers.
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-01

comments: SUMMARY:
This paper proposes a simple but powerful way to turn a plain-language problem description into a full Bayesian analysis without hand-coding a model.
An LLM is prompted to emit many candidate formal models in a probabilistic-programming language; each model’s posterior and marginal likelihood are approximated.
The authors give a practical algorithm, theoretical justifications and experiments on synthetic dataset to support their proposed idea.

STRENGTHS AND WEAKNESSES:
## Strength
1. I think the paper is very well-written, it is very easy to follow what the authors wish to express.
2. The paper provides a practical way for LLM users to perform principled Bayesian predictions, which leads to nice practical impact.
3. Theoretically, one only needs to match the target variable, which in my opinion provides strong flexibility.

## Weakness
1. The experiments are done on synthetic data. While I would assume it is not easy to find a real world data in this case, can the authors provide some discussion on how can one apply their method in practice?
2. Typo: line 9 (justified "an" analyzed).

QUESTIONS:
1. Is my understanding of "only need to match target" correct?
2. Can the authors provide some discussion on how can one apply their method in practice?
3. How does the computational cost behave when applying the proposed algorithm?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to make Bayesian inference widely available by automatically\ndeducing the appropriate Bayesian model and performing inference under the model\nwith the user of the system only being required to provide some data and a text\ndescription of the latent variables that they are interested in.\n\nThe paper outlines a very straightforward recipe that can be very easily replicated.\nFirst, use an LLM to draw samples of valid PPLs and perform posterior inference over\nthe latent variable with each PPL. Then use these posterior samples to do variational\ninference of the marignal likelihood of the data given the PPL model. Finally, return \na likelihood-weighted average of the posterior from all the PPL model posteriors.\n\nThe paper also provides other methods where a pure variational inference approach can\nbe used.\n\nThere are some results provided on few examples with a few hundred data points\nincluding examples where the data type of the latent variable is not explicitly \nstated.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength : Novelty\n\nWhile there are other papers on PPL generation including generation of PPLs from LLMs\nthere are none that explicitly perform model averaging as described here in a simple\npractical recipe.\n\nThe authors have also provided a second inference method using a purely variational\napproach which might be a novel.\n\n## Strength : Broad Applicability\n\nThe ability to make Bayesian Inference broadly applicable with a simple text prompt\nhas the possibility to introduce an unprecedented level of rigor in scientific\ndiscourse where probabilistic inference is needed for an accurate interpretation of\ndata but where statistical expertise is generally lacking.\n\n## Weakness : Detached from BMA literature\n\nThis paper is a very natural continuation of the Bayesian Model Averaging literature\nbut it appears to attempt to negate that connection by a rather offhand remark,\n\"However, existing algorithms for Bayesian model averaging (e.g. reversible jump MCMC\n [8, chapter 11]) do not appear to be applicable here, since they involve explicitly \n iterating over all models (here very larger or possibly infinite) and/or using evaluations of p(m|t).\"\n\nThis is not a fair remark. In BMA literature model averaging does use the posterior \nof the model given the data as the weight which is very similar to the current paper.\nCurrent paper draws samples of the model using an LLM and then weights these samples\nusing the likelihood of the data given the model. Now, depending on how one chooses\nto interpret the samples from the LLM this can be considered technically identical.\n\nIn any case, there is a fair amount of literature in BMA including frequentist methods\nwhich could all be considered relevant. It is unclear if the two methods described in\nthis paper are indeed novel unless a thorough comparison is made to BMA.\n\n## Weakness : Scalability\n\nNow, PPL inference is already quite intractable and this paper adds another layer\nof intractability by generating a large number of PPLs and performing inference on \nall of them. This makes the approach appear to be impractical.\n\nThe paper demonstrates results on only very simple examples with a few hundred data\npoints at best. This doesn't provide any reassurance on the scalability of the approach.\n\n## Weakness : Baseline Comparisons\n\nThe experiments don't really compare the generated posteriors to those that would\nbe generated if say an expert was to write a PPLs. It would be fine to compare\nagains the performance of a PPL generated by a non-expert.\n\nThere is no discussion of whether the generated posteriors are in any way calibrated\nwhich makes it hard for the reader to convince themselves that the proposed method\nhere is sound.\n\nQUESTIONS:\nWhy is the proposed method in this paper not been compared to related work in BMA?\n\nWhy has the algorithmic variants in section 5.3 been provided when there are no\nexperiments to back up the claims here?\n\nWhat kinds of models would experts and non-experts have generated for these\nscenarios and how do the highest weight models compare to those?\n\nHow well calibrated are the posteriors?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work deals with a key limitation of probabilistic programming: model specification. Domain experts, such as clinicians, are typically not familiar with the syntax and semantics of probabilistic models, posing a significant barrier to quantitative analysis and requiring a dedicated data scientist or engineer to encode their domain knowledge into the model. Knowledge elicitation is often error-prone and cumbersome. This work addresses this problem using LLMs, allowing experts to describe their problems in English and translating the problem description (t) into a probabilistic program (m) using a decoder-only LLM, such as GPT. The program m describes a model over target variables (z) given observed variables (x), that is, p(z | x, m). Since LLMs are stochastic, the program m can be interpreted as having been sampled from p(m | t). The authors exploit this fact to make the program more robust by defining it as an ensemble, a set of sampled programs, the final density over the target a weighted average of each model’s density. They consider two methods: flat averaging and Bayesian model averaging (BMA). While the former weighs each sampled model equally (that is, proportional to the prior distribution p(m | t)), the latter uses the posterior over the model $p(m | x, t) \\propto p(m | t) p(x | m)$, which has the additional p(x | m) marginal likelihood term. Exact inference is exceedingly difficult. First, the LLM’s distribution over the model m given the text description p(m | t) is usually inaccessible; only samples drawn from it are accessible. Second, the model’s posterior density over the target p(z | x, m) and its marginal likelihood p(x | m) are hard to compute exactly. The former is estimated via NUTS, while the latter is estimated via variational inference, setting the variational distribution to be a Gaussian matching the empirical mean and variance of the samples. The authors demonstrate the efficacy of their proposed method on five newly created domains: the fully Bayesian averaged model outperforms the flat average, especially in challenging domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This framework exploits the complementarity between LLMs and probabilistic programs: LLMs excel at natural language processing but cannot reason; probabilistic programs excel at reasoning but cannot deal with unstructured information.\n2. Empirical evaluation on newly constructed domains\n3. Model-agnostic approximate inference procedure for Bayesian model averaging.\n\nWeaknesses\n1. Generated models failing to compile due to syntactic invalidity seems to be a major limitation. This seems to get worse in complex domains like Gold. This might be addressed by using constrained generation, e.g., Loula et al. (2023)\n2. While the authors construct new domains to avoid leakage with the LLM’s training data, the domains themselves are relatively less complex. It appears that the complexity of the domain, the size of the text description, and its conflicts with the LLM’s embedded information would affect the quality of models.\n\nReferences\n\nLoula, João, et al. \"Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo.\" ICLR, 2025.\n\nQUESTIONS:\n1. Since the models are LLM-generated and the experts are not familiar with probabilistic programming, there seems to be a risk of the model encoding undesirable biases from the LLM’s training data.\n\n2. Section 5.1 says that \"These results underline the importance that the LLM-defined prior be relatively broad.” Can you elaborate on this point? Would imposing syntactic validity constraints on the LLM output make it less likely to assign low p(m | t) to models having high marginal likelihood ?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}