Title: HoT-VI: Reparameterizable Variational Inference for Capturing Instance-Level High-Order Correlations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Probabilistic Models

Variational Inference

Instance-level Correlation

Reparameterization

High-order Correlation
Abstract: Mean-field variational inference (VI), despite its scalability, is limited by the independence assumption, making it unsuitable for scenarios with correlated data instances. Existing structured VI methods either focus on correlations among latent dimensions which lack scalability for modeling instance-level correlations, or are restricted to simple first-order dependencies, limiting their expressiveness. In this paper, we propose High-order Tree-structured Variational Inference (HoT-VI), that explicitly models $k$-order instance-level correlations among latent variables. By expressing the global posterior through overlapping $k$-dimensional local marginals, our method enables efficient parameterized sampling via a sequential procedure. To ensure the validity of these marginals, we introduce a conditional correlation parameterization method that guarantees positive definiteness of their correlation matrices. We further extend our method with a tree-structured backbone to capture more flexible dependency patterns. Extensive experiments on time-series and graph-structured datasets demonstrate that modeling higher-order correlations leads to significantly improved posterior approximations and better performance across various downstream tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of expanding mean-field Variational Inference (VI) to account for instance-level dependencies in a computationally efficient way. By considering a particular form of Gaussian variational family, the authors are able to incorporate instance-level dependency within a predefined and tunable range into the model. Guidelines are provided to confine the iterates within a suitable region of the optimization space. Experiments on real-world data support the method's practical applicability and efficacy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The proposed method is original and its implementation seems to be quite simple. There is a clear way to dial between the expresiveness of the variational family and the model's complexity (however this trade-off is not explored in details, see Weaknesses below).\n* The method empirically outperforms the baselines in a wide variety of real-world scenarios, including anomaly detection (3 datasets), time series forecasting (4 datasets) and clustering (4 datasets).\n\n**Weaknesses**\n* There is no runtime study, Appendix D.4 is not very informative about the computational cost of the method. The trade-off between the gains yielded by more complex models and the added computational cost is not discussed. This is essential for establishing the method's superiority over existing approaches, and should be addressed by the authors.\n* The variational posteriors must be Gaussian, hence the method is not applicable to settings where such approximating families are irrelevant, e.g., directional statistics.\n* In the constrained clustering experiment, it is unclear how the constraints are incorporated into the model (even with the additional details in Appendix D.3).\n* The authors do not discuss how the proposed reparameterization effectively restricts the expressiveness of the variational family.\n\n**Additional comments**\n* Line 77: there are two product signs in the equation.\n* Table 5: typo in “Epchs”\n* Throughout the text: replace \"validness\" with validity, and \"indexes\" with indices.\n\nQUESTIONS:\n* TreeVI appears to be the most similar work to yours. Could you elaborate on the key differences between your method and TreeVI, and highlight the improvements brought by your approach?\n* How does your method compare to concurrent methods like TreeVI in terms of effective runtime?\n* While some information about the datasets used in the experiments is provided in Appendix D.1, it is not said exactly how large these datasets actually are and how many observations they contain. For instance, SMAP is a collection of data products. Could you provide more details on this matter?\n* Lines 80-81: “the method can easily be extended to take the dimensional-level correlation into account”. Can you expand on this?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWithin the context of variational inference (VI) for latent-variable models, this paper proposes a variational structure that builds in dependence between k neighbouring latent indicators. In principle, doing so will allow improved dependence (e.g. temporal) modelling within the latents, and better modelling and predictive abilities on the observed data. The paper uses a Gaussian distribution for the variational approximation, but with some conditioning and reparameterising tricks, the ability to handle this computationally is managed, with reasonable scalability. The simulations show improved performance in a number of modelling tasks compared to existing methods with weaker dependence among latents.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper has seemingly found a way to build structure between latent variables, via a normal variational density, while managing to control the computational overheads for increasing dimension of dependence. This structure not only includes a k-Markov correlation structure, but this can be reworked to encompass a \"tree-structured backbone\" dependency structure.\n\nThe simulation experiments demonstrated that capturing such dependence is (of course) beneficial in a number of time series and clustering tasks.\n\nWeaknesses: It seems that there may be some complexities involved in constructing more detailed correlation structures. I didn't get too much of a sense how hard or involved this was from the paper though.\n\nIt's also not clear what the computational overheads of the proposed method are, and how this scales with increasing k.\n\nQUESTIONS:\nI was looking for some information about the computational overheads needed for this method (given that this is effectively the problem the paper is solving). Both for the presented simulations in Section 4 (comparative times for analysis of each method), but also an understanding (empirical and general) of how these overheads scale as k increases in a given computational task. Is k=3,5,10 the limit that this method can reasonably tackle, or is k=50,100,250 possible too?\n\n\nSome trivial typos (no response required):\n\nFig 1 - should the last node be Z_N?\nl.97 \"connection structure [is] imposed\" - delete \"is\"\nl.102 \"follows\" -> \"follow\"\nl.138 \"for THE ELBO\" insert \"the\"\nl.146 \"above TO BE valid.\" insert \"to be\"\nl.298 and l.300 \"state of THE GRU\" insert \"the\"\nl.319 \"learn AN expressive\" change \"a\" to \"an\"\nl.327 Write L1, L2 as L_1, L_2, delete the extra space before the comma from \"L_2 loss ,\"\nReferences lots of missing capitalisations. E.g. \"lstm\", \"gaussian\", \"wasserstein\", \"bayesian\", \"Journal of machine learning research\" etc.\nReference [57] and [58] look suspiciously similar. Is there a publication racket going on here?\n\nLIMITATIONS:\nYes, but as discussed above, it would be good to have some further discussion on this, as it was only mentioned very briefly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focusses on approximate Bayesian inference in correlated latent variable models using variational inference. The paper proposes a Gaussian variational family with (chain or tree) structured  precision matrices. \n\nThe paper provide a product decomposition theorem (Theorem 2.1) for the density of a multivariate Gaussian, where the precision matrix exhibits k-order connectivity structure. The theorem also works \"the other way\", giving conditions for when the set of marginal form a valid joint density. The theorem enables efficient sampling from the joint distribution in a sequential manner via the set of marginals.  \n\nFor efficiency, an amortized solution is proposed, where the required correlation coefficents of the variatonal distribution are reparametrized via a neural network. Specifically, the correlation matrices are re-parametrized through a combination of (constrained) first order and higher order correlation coefficient to ensure positive definiteness and proves that the resulting correlation matrix is valid (Theorem 2.2 and Corollary 2.3).\n\nThe paper concludes with an empirical evaluation of the proposed method in the context of time series anomaly detection, forecasting and constrained clustering. The results for anomaly detection shows that the proposed method outperforms the included baseline in terms F1 (better outlier detection) and ELBO (better representation of the exact posterior as measured by the KL divergence) and similar strong performance for the forecasting and clustering experiments\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-structured, well-written and easy to follow. The proposed method for bridging the gap between a mean-field posterior and a general Gaussian is interesting. There is a lot of prior work on structured precision matrices in general, but I believe that the proposed idea is novel in the context of variational inference (at least to the best of my knowledge) and will be of interest to the community. \n\nGenerally, the experimental design seems sound and shows a clear benefit relative to the included baselines. However, I am not that familiar with the set of included baselines and therefore, it is no clear to me whether they are strong baseline or easy to beat. Moreover, it is unclear how the baseline methods are configured, to what degree they are tuned etc.\n\nMoreover, some of the implementation details are unclear. For example, what is the likelihood/loss in the VAE? It is a standard Gaussian? How many posterior samples is used for 1) training and 2) evaluation? Finally, the authors do not report uncertainty for any of the evaluation metrics, i.e. no error bars etc.\n\nQUESTIONS:\nI believe the paper could be stronger if  the authors could elaborate on the following points:\n\n- The results generally show improved performance as a function of the order of the structured precision matrix. It is therefore natural to wonder whether the performance would be even better if the author went beyond 10? Did the authors experiment with approximation order beyond 10?\n  \n- Moreover, it would be interesting and informative to understand the effect of increasing the order of the approximation, all the way from mean-field to a fully flexible Gaussian (possible on a smaller dataset) and possibly also comparing to other variational families, e.g. low rank Gaussians and flows.\n  \n- It would also be interesting to investigate the trade in computational complexity when the order is increased from mean-field to a full Gaussian. What happens to the run time? When will it become more efficient to simply use a full Gaussian rather than the structured approximation. \n  \n-  There are no error bars/uncertainties associated with the metrics. How robust are the results?\n  \n- What is the experimental set-up for the baselines? Did the authors do an effort to make them appear as strong as possible or did they use them blindly?\n  \n- What is the likelihood of the VAE? A Gaussian? How many posterior samples were used during training and evaluation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a more accurate variational inference approximation that takes instance level correlation into account. They do this by approximating the N by N covariance matrix with a K by K matrix that takes into account the correlation between $i$th instance upto the $i+k$th instance. The correlation coefficients of this K by K matrix are parametrised through a neural network. The authors then introduce a method to ensure that the resultant covariance matrix is PD as well as an extension to a tree based correlation structure. The authors show improved performance in time series and constrained clustering tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The method is interesting and sound.\n- The experiment are fairly wide ranging and over multiple baselines.\n\n**Weaknesses:**\n- The paper could be a lot clearer:\n     - What is the exact cost of inference with P?\n     - The intuition behind modelling only K correlations (apart from better computational cost) is not clear. What are you losing here? There should also be a discussion on the intuition for choosing this parameter.\n    - The baselines are not sufficiently described. Which of these methods actually model instance level correlation? If they do, what is the difference between the proposed method?\n\nQUESTIONS:\n- What is the intuition behind the K level correlation modelling? Is using a value of 1 effectively a Markov assumption in the posterior?\n- At what level of K should we recover the same correlation structure as the baselines? In Table 1 I would have expected the 1 order method to perform as well as an LSTM-VAE that models the same correlation (apart form the fact that the authors use a GRU). \n- Why is it necessary to directly parametrise the correlation coefficients $R$? If terms of the covariance matrix are parametrised won't this allow for parametrising the cholesky of the covariance matrix which will ensure that the covariance is PD by construction?\n- What kind of neural network is $f_{\\phi}$? Are the different $x_i$ concatenated across the feature dimension or is attention applied across the $K$ dimensions?\n- Is there any relation to the inducing points framework used in sparse GPs? The problem being solved seems similar, here an N by N matrix is too complex to invert, so it is approximated by a K by K matrix with K < N.\n- What is the cost of the sampling procedure laid out in L121?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}