Title: Squared families are useful conjugate priors
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian inference

conjugate families

Gaussian processes

squared families
Abstract: Squared families of probability distributions have been studied and applied in numerous machine learning contexts. Typically, they appear as likelihoods, where their advantageous computational, geometric and statistical properties are exploited for fast estimation algorithms, representational properties and statistical guarantees. Here, we investigate the use of squared families as prior beliefs in Bayesian inference. We find that they can form helpful conjugate families, often allowing for closed-form and tractable Bayesian inference and marginal likelihoods. We apply such conjugate families to Bayesian regression in feature space using end-to-end learnable neural network features. Such a setting allows for a rich multi-modal alternative to Gaussian processes with neural network features, often called deep kernel learning. We demonstrate our method on few shot learning, outperforming existing neural methods based on Gaussian processes and normalising flows.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the use of squared families as priors in Bayesian inference.\nThis work considers the generalised square family (GSF) of distributions,\nshowing that it is an especially convenient family for Bayesian priors, because\n(under some constraints) (1) the marginal likelihood and posterior predictive\nbecome tractable, (2) the posterior is also GSF. These properties make GSF a good choice\nfor Bayesian regression, and in the experiment section, the authors demonstrate\nthat GSF priors can be used to outperform existing methods (DKT and NGGP) on\nfew-shot learning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n\n- The paper has strong results, and comprehensively demonstrates the benefits of\n  GSF priors in Section 3.\n- The experiments cleanly show benefits of GSF over alternative few-shot\n  learning methods in Table 1 and Figure 4.\n\nWeaknesses:\n\n- The introduction, especially the first subsection of squared families, reads\n  more like a background section, lacking a clear problem statement and\n  motivation for the use of GSF priors.\n- The contributions are not clear in the introduction. Two examples: first, the few-shot setting used\n  in the experiments is not disucssed at all; second, authors say 'we first\n  construct a hierarchy of families, ... (GSF)', which on first reading sounds\n  like a contribution, but then confusingly GSFs appear in the background section.\n- Claims that GSF is a rich and 'convenient to work with' family of\n  distributions (e.g. line 100/101, line 164) is not easily verifable from the\n  main text. Concrete examples would be helpful for readers.\n- The experimental results could be more comprehensive, e.g. by using\n  additional, more modern datasets/methods for comparison\n- Experiments are conducted only with a single layer of MLP. I would imagine DKT\n  and NGGP could be improved with deeper networks, potentially leading the reader to doubt the baselines.\n- Figure 4 could be much clearer, e.g. by labelling rows/columns and axes and/or\n  adding a legend.\n- There are no functioning links in the paper (e.g. figure links, table links, refs, etc.)\n\nQUESTIONS:\n- You say that GSF is a rich family of distributions, but then (correct me if\n  I'm wrong) experiments are only conducted using Gaussian\n  measures/likelihoods. Are you able to try with any alternatives? Specifically,\n  the categorical likelihood would be very useful.\n- Are you able to combine GSF with e.g. deeper feature extractors? 1 layer is a\n  bit artificial. Results would be more convincing if they included a more\n  thorough hyperparameter search.\n- How exactly (algorithmically) are you performing your Bayesian updates and NN\n  updates? An algorithmic description in the experimental section would help\n  ellucidate this.\n\nLIMITATIONS:\nLimitations are adequately discussed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores *Squared Families* as prior distributions and the conjugacy of prior distributions belonging to *Generalized Squared Families*. GSFs are indexed by a parameter, $\\textbf{M}$, a base measure, $\\mu$, and a feature, $\\Phi$. The authors show GSF priors can admit closed form expressions for commonly used objects (e.g. marginal likelihoods, posterior predictive distributions) when, for example, the base measure is conjugate to the likelihood. The paper explores particular settings (base measure + likelihood pairs) and works through results for those settings. The paper concludes with an experiments section comparing against two baselines on a collection of datasets.\n\nSTRENGTHS AND WEAKNESSES:\nThese bullet points present the thoughts I had while reading the paper and more detailed questions / suggestions are contained in \"Questions\". \n\nStrengths:\n- The core idea is unique and as far as I know this particular argument of using squared families as priors is new.\n- The graphics of the paper are well done and help communicate ideas. For example, Figure 4 helps give intuition about the benefit of GSPs' multimodal behavior. \n- The paper's relation to previous work on squared families is present. \n- The presentation contains relevant details and I believe the analytic results are accurate. \n\nWeaknesses:\n- The paper is hard to follow as things are presented without context or implicitly defined (e.g. line 14). \n- The details are sometimes too much and distract from the actual point being made, see suggestion on reworking section 3. \n- The numerical results are limited and make it hard to say there is any benefit to using GSFPs, see suggestion on the experiments section.\n\nQUESTIONS:\n1. The computational benefits of GSPs were presented as one of their sources of value, but section 5's limitations makes me question if this is actually true since the complexity presented on lines 269-271 seem comparable to naive GPs. Can the authors please explain what in particular makes the reported complexities desirable? These seem as restrictive as any kernel based method. \n\n2. The paper presents a substantial amount of material, but it is not as accessible as it could be for the work to have broader impact. For instance, Section 3 could be shortened, as the key point about conjugacy for exponential families is already made immediately after Proposition 4. The content in Sections 3.2 and 3.3 could be streamlined, with technical details moved to the appendix. Currently, both regression and generalized squared family processes are presented with equal detail, but shifting the finer points to the appendix and focusing more on the differences between the two settings would make the exposition more effective\n\n3. In the experiments, the methods are evaluated solely on predictive performance, making it difficult to assess whether the multimodal capabilities of squared families are actually leveraged. Aside from Figure 4, do the authors have additional results that demonstrate the claimed benefits? The paper does not convincingly show that the unimodality of Gaussian processes is a practical limitation or that the proposed method meaningfully addresses it. A clearer argument supporting this claim would help strengthen the paper.\n\n\n4. The experiments are limited in both scope, given the narrow range of datasets and baselines used, and in depth, as only NLL is reported. Additionally, it is difficult to determine whether the observed differences on these small-scale datasets reflect meaningful methodological differences or are the result of implementation issues (e.g., line 258). Do the authors have results on additional baselines or datasets, such as standard UCI regression tasks or pose estimation? Including Gaussian processes as a baseline could help clarify whether the multimodal approach is genuinely beneficial. It would also be helpful to report additional metrics, such as RMSE, to better assess performance.\n\nI think question (1) above could be addressed if I am simply misunderstanding the paper and the authors could correct my thinking. Question (2) is certainly something that could be addressed with simple changes to the writing. Question (3/4) are also addressable but I don't know if this is something that can be done in during the short response period. Addressing 1/2 would likely me bump my score but still lean towards reject, while addressing 3/4 would lead me to lean towards weak accept. Please let me know if my criteria is not clear and I will clarify.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the use of squared families as priors. The authors show that when the base measure and the likelihood are conjugate, the posterior, marginal likelihood, and predictive distribution all admit closed-form expressions. The squared prior is capable of modeling multi-modal distributions, and its parameters can be parameterized by neural networks. This ability to capture multi-modality gives the method an advantage over Gaussian processes, as demonstrated in few-shot learning regression experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The use of squared families as priors is quite novel. The conditions under which closed-form computations are possible are fairly general, making the method potentially applicable to a wide range of modern probabilistic machine learning problems.\n- The paper is well written and easy to follow.\n\nWeaknesses\n- The paper could benefit from a more detailed literature review of recent related work on squared families, including a clearer discussion of their respective settings and how they differ from the current approach. At present, this comparison appears primarily in the early part of the introduction and is somewhat general, which may make it difficult for readers to fully appreciate the distinctions and significance of this work.\n\nQUESTIONS:\n- In GSFP, the base measure is chosen to be a Gaussian with mean $\\text{vec}(\\mathbf{M})$, sharing the parameter $\\mathbf{M}$ with the squared prior. Is this specific parameterization necessary for achieving good performance on regression tasks?\n- For regression tasks, is it possible to adopt other base measures, feature maps, or likelihoods beyond the Gaussian base—for example, those mentioned in Appendix B? How robust is the method to different choices of these components?\n\nLIMITATIONS:\nSee weaknesses and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the use of Generalised Squared Families (GSF) as **priors** for Bayesian linear feature regression problems.\n\nGeneralised squared densities $q(\\omega | \\theta, \\mu, \\psi)$ are defined with respect to measure $\\mu$, proportionally to the squared Euclidean norm of the inner product $\\theta \\psi(\\omega)$. Due to its analytical properties, the density can be written in terms of a ratio between two analytically tractable traces ---as in Equation (1) ---, generalizing mixture models.\n\nThe authors demonstrate in Section 3 that closed-form expressions for the marginal likelihood, posterior parameter and posterior predictive distributions are attainable for GSFs, whenever the base measure of the GSF is itself conjugate to the likelihood of the model.\n\nBased on these analytical properties, the authors tackle Bayesian (linear in features) regression models, and showcase that for Gaussian likelihoods, Bayesian GSF priors over these models (called GSF processes) result in closed form posterior and marginals of interest.\n\nExperiments showcase the benefits of considering a multi-modal prior/posterior family in few-shot learning examples, where better model-fit is achieved when compared to reasonable baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**The main strength** and significant contribution of this submission is to extend Gaussian Process regression with multimodal priors and posteriors via GSFs, and to show that these have closed form solutions.\n    \n- Namely, this work can be seen as a generalization of Gaussian prior with Gaussian likelihood regression models in feature space, via GSF priors, that allow for multimodal expressions of uncertainty.\n\nAdditionally,\n- The background section is a thorough and informative summary of the concepts needed to understand this work\n- Section 3, where the Bayesian linear regression in feature space with GSF priors is introduced, is a math-heavy section that clearly presents all the key assumptions and properties needed to derive the closed form expressions for GSFPs.\n\n**The main weakness**, as acknowledged by the authors themselves, is the computational (and memory) cost associated with training a GSFP.\n\nQUESTIONS:\n- Can the authors elaborate and more clearly explain the computational complexity of GSFPs? What are the key bottlenecks and how can one implement or overcome them?\n\n- I would encourage the authors to discuss (if any) the potential of GSF priors in classical GP regression problems\n    - what are the benefits/drawbacks of this family of priors?\n    - can the authors showcase empirically how these priors would impact regression performance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}