Title: Fisher meets Feynman: score-based variational inference with a product of experts
Venue: NeurIPS 2025
Accepted: yes
Keywords: black-box variational inference

score matching

product of experts

multivariate t-distribution

Feynman identity

Fisher divergence

expressive variational family
Abstract: We introduce a highly expressive yet distinctly tractable family for black-box variational inference (BBVI).  Each member of this family is a weighted product of experts (PoE), and each weighted expert in the product is proportional to a multivariate $t$-distribution.  These products of experts can model distributions with skew, heavy tails, and multiple modes, but to use them for BBVI, we must be able to sample from their densities.  We show how to do this by reformulating these products of experts as latent variable models with auxiliary Dirichlet random variables.  These Dirichlet variables emerge from a Feynman identity, originally developed for loop integrals in quantum field theory, that expresses the product of multiple fractions (or in our case, $t$-distributions) as an integral over the simplex. We leverage this simplicial latent space to draw weighted samples from these products of experts---samples which BBVI then uses to find the PoE that best approximates a target density. Given a collection of experts, we derive an iterative procedure to optimize the exponents that determine their geometric weighting in the PoE. At each iteration, this procedure minimizes a regularized Fisher divergence to match the scores of the variational and target densities at a batch of samples drawn from the current approximation.  This minimization reduces to a convex quadratic program, and we prove under general conditions that these updates converge exponentially fast to a near-optimal weighting of experts.  We conclude by evaluating this approach on a variety of synthetic and real-world target distributions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the variational inference (VI) where the approximate variational distribution is assumed to be a weighted product of experts, and each expert is a multivariate t-distribution. To optimize the expert weights, this paper first use Feynman identity to reformulate the product of t-distributions as a latent variable model where the latent variable (denoted as w) lies in the simplex and the observed variable (denoted as z) is conditionally t-distributed given w. Then propose to minimize the Fisher divergence between the PoE and the target density. Some experiments on synthetic and real-world datasets are conducted to demonstrate the performance of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nOverall, I enjoy reading this paper.\n1. The idea of using the Feynman identity to reformulate the PoE into a continuous mixture of t-distributions is interesting. In this formulation, the mixture weights w are Dirichlet-dsitributed, and the variable z follows t-distribution given w. This makes sampling from the PoE more tractable: one can sample w from a Dirichlet and then sample z from conditional t-distribution.\n\n2. The Fisher div between the PoE and target is optimized using \"batch-and-match\" strategy. Since this requires sampling from PoE for each iteration, the Feynman identity-based reformulation provides a practical advantages by facilitating this sampling process.\n\n3. Experimental results demonstrate that the proposed PoE model captures the heavy tails of the target distributions more effectively than Gaussian approximations, benefiting from the properties of t-distributions.\n\nWeaknesses:\n1. The proposed algorithm only learns the expert weights (\\alpha ), while the set of expert distributions (t-distributions) must be predefined. The parameters of these experts remain fixed during training, which could limit the model’s ability to approximate complex target distributions.\n\n2. The Feynman identity-based reformulation uses the fact that the density of t-distribution is proportional to the inverse of a quadratic from. This property is essential for applying  The Feynman identity, and may not generalize easily to other distribution families. This might limit the broader applicability of the proposed approach.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a novel score-based BBVI procedure based on a Product of Experts (PoE) that models skewed and/or heavy tailed distributions more accurately than state-of-the-art methods.\n\nTo that end, the authors introduce a new family of PoEs, each proportional to a multivariate t-distribution. Notably, they show how to draw samples from this (complex) family and how to estimate its normalizing constant, by transforming the problem via the Feynman identity (Section 2).\n\nWith this PoE (plus importance sampling tricks), the authors derive an iterative score-matching procedure to optimize the PoE's geometric weighting of members, via minimization of a regularized Fisher divergence (Section 3). This minimization is proven to be a convex quadratic problem, hence resulting in rates of convergence to the neighborhood of an optimal weighting of experts.\n\nThe empirical experiments in synthetic and a Bayesian posterior benchmark of Section 4 illustrate how BBVI with this PoE better approximates skewed and/or heavy tailed densities.\n\nSTRENGTHS AND WEAKNESSES:\n**The contributions of this work are two-fold**:\n- The use of the Feynman identity to demonstrate how to draw samples, compute the marginal and estimate quantities of interest from a PoE with experts proportional to multivariate t-distributions (efficiently over the simplex, instead of over the reals)\n- The use of such PoE within BBVI to derive an iterative score-matching procedure for optimization of the geometric weighting of experts for successful approximation to skewed and heavy-tailed distributions\n    \nAdditionally, I would highlight the theoretical analysis of the weighting of experts procedure, resulting in convergence guarantees.\n\nAs the **main weaknesses** of this work, I would identify:\n- The heuristic proposed to select the experts (and the lack of evaluation of its computational cost and impact on performance)\n- The (relatively) complex procedure to attain gains (only?) on skewed and heavy-tailed posteriors\n    - See my questions below on the method's tradeoffs, and whether results in other posteriors are comparable to SOTA\n\nQUESTIONS:\n- Can the authors elaborate on the benefits (and if any, drawbacks) of the proposed PoE based BBVI technique beyond the skewed and heavy-tailed distributions presented in Section 4?\n    - Namely, what is the performance of PoE-BBVI in other posteriors?\n    - Is it better/comparable/worse than alternatives?\n\n- Can the authors elaborate on the computational cost/complexity of computing the PoE within the BBVI framework?\n    - what is the tradeoff between compute time and posterior accuracy?\n    - can these metrics be reported for the evaluations in Section 4?\n\n- Even though it is listed as a limitation, more details on the initialization procedure would be appreciated:\n    - How sensitive are the presented results to a good initialization of the fixed collection of experts selected upfront?\n    - What is the number of experts used in the experiments, and how do results change with respect to varying it?\n    - What's the computational complexity of such initialization procedure?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores a new family of models for black box variational inference, where the models are a weighted product of experts, with each expert being a multivariate t-distribution. Noting a relationship to a Feynman identity of integrals from physics, the authors is how how this model can be written as a latent variable model, where the latent variable is Dirichlet distributed. This enable the authors to define a sampling strategy and a means to compute expectation values. From the standpoint of VI, the authors introduce a heuristic scheme for selecting experts, but also show how using a modifier Fisher divergence for the optimization of the weights of experts can lead to a tractable Quadratic program. A convergence theorem is also proven. On several synthetic experiments, the model is seen to work well.\n\nSTRENGTHS AND WEAKNESSES:\nThis is an interesting paper, introducing a new model and relying on mathematical tools from physics. The paper is clearly written, and includes description of the model, the sampling, the inference strategy, and the convergence.\n\nOne weakness is the method for selection of experts. This is essentially a heuristic method, that requires hill climbing and hand selecting means, inverse scale, and numbers of experts. Without a more automated and optimizable method for selecting experts and estimating their parameters, is is not clear how broadly useful this method can be, especially in anything but low dimensions of the data. Nonetheless, this work provides a good first step towards exploring this family of models.\n\nThe documentation of the experiments is also difficult to assess in the main text, although significantly more useful information is made available in the appendix.\n\nQUESTIONS:\nCould you move more information for both the heuristics for placing experts, and the details of the experiments into the main body? these two aspects seem under described in the text, and the paper would benefit from more details in the main body.\n\nDo you have a possible path to move away from heuristics for selecting experts, and instead optimize their parameters?\n\nDo you a have a notion of how big of a bias is introduced by the empirical estimate for the loss? Do you have any experiments that show if this significantly affects the optimization?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a new family of variational approximations where each member is a weighted product of experts and each expert is proportional to a multivariate t-distribution. Sampling from these distributions is non-trivial and this work presents a Feynman reparameterisation to tackle this issue. This reparameterisation reformulates these PoE distributions into latent variable models with auxiliary Dirichlet random variables. They then propose a Fisher Divergence based objective to fit this distribution to a given target distribution and prove that the corresponding updates converge to the optimal weights for the experts. They appraise this framework's performance by fitting synthetic datasets and datasets from posteriordb in Stan.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This work, to the best of my novel appears to be novel and original. The work seems inspired and geared towards solving present-day problems.\n2. The theoretical foundations are solidly developed and clearly presented. \n\n**Weaknesses**\n1. Though this work motivates the development of this novel variational approximation from the perspective of having expressive and practical approximate posteriors, I do not see strong supporting evidence in the work to conclude either of these. Firstly, a limited evaluation has been conducted here and without more exhaustive testing on practical applications and uncertainty estimation benchmarks, it's hard to ascertain the quality of the posterior inferred.\n2. The proposed approximation has very elegant foundations but, in the authors own words, optimising the parameters is `complicated` and without a strategy to place the experts at the right spots this approach cannot infer the right expert parameters. In this regard I would also like the authors to comment on the case where experts are placed at completely random locations. This seemingly non-negotiable requirement to correctly initialise the experts, shrinks the applicability of the proposed variational approximation.\n3. According to `Result 2.1` text in `l 113- l 115 `, this PoE can be viewed as a continuous mixture of t-distributions. In this regard I would like the authors to comment on relevant approaches in Implicit variational inference [1][2] where continuous mixtures of Gaussians have been proposed as expressive variational approximations. How does the proposed approach compare to such formulations?\n\n\n**Minor**\n1. `l 25` >Each of such gradient evaluation..... . I think this line is a little vague. In my opinion, it really depends on the downstream task or purpose which determines whether a sample or a gradient is more useful. Kindly consider rephrasing this line or adding more information.\n2.  It's hard to draw conclusions from the figures in the appendix, namely figure E.1 and F.2. In F.2 the legend is not readable.\n\n[1] Unbiased Implicit Variational Inference, Titsias et. al. 2019\n[2] Implicit Variational Inference for high dimensional posteriors, Uppal et. al. 2023\n\nQUESTIONS:\nFor most of my primary concerns please refer to the weaknesses section above.\n\nTwo more questions I would like to ask -\n1. What happens when you initialise the mixture components of a GMM at the right spots and only optimise the mixing proportion parameter? In terms of divergences, does the GMM not capture the target distribution accurately? How much better would a PoE be compared to this? When a GMM with a large number of components is initialised for the same target, does standard VI not infer a sparser GMM by driving down mixing proportion values for some of the components?\n2.  From `Figure D.1` it seems that you have to always start with a very high number of experts, off which some are found to be redundant. Does this not lead to an increase in computational cost? Is there any way to be more efficient with this initialisation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-04"}