Structured Recognition for Generative Models with Explaining Away
Authors: Changmin Yu
Hugo Soulat
Neil Burgess
Maneesh Sahani
Venue: NeurIPS 2022 Accept (2022)
Keywords: variational inference
Gaussian process
computational neuroscience
graphical models

Abstract: A key goal of unsupervised learning is to go beyond density estimation and sample generation to reveal the structure inherent within observed data. Such structure can be expressed in the pattern of interactions between explanatory latent variables captured through a probabilistic graphical model. Although the learning of structured graphical models has a long history, much recent work in unsupervised modelling has instead emphasised flexible deep-network-based generation, either transforming independent latent generators to model complex data or assuming that distinct observed variables are derived from different latent nodes. Here, we extend amortised variational inference to incorporate structured factors over multiple variables, able to capture the observation-induced posterior dependence between latents that results from “explaining away” and thus allow complex observations to depend on multiple nodes of a structured graph. We show that appropriately parametrised factors can be combined efficiently with variational message passing in rich graphical structures. We instantiate the framework in nonlinear Gaussian Process Factor Analysis, evaluating the structured recognition framework using synthetic data from known generative processes. We fit the GPFA model to high-dimensional neural spike data from the hippocampus of freely moving rodents, where the model successfully identifies latent signals that correlate with behavioural covariates.

Structured Recognition for Generative Models with Explaining Away
Changmin Yu!.2 Hugo Soulat? Neil Burgess'
Maneesh Sahani²
Institute of Cognitive Neuroscience: "Gatsby Computational Neuroscience Unit; UCL, London, United Kingdom
(changmin.yu..o yu. hugo soulat. 19; n. w.buugess @ucl. ac uk;
maneesh@gatsby ucl. ac.uk
Abstract
A key goal of unsupervised learning is to go beyond density estimation and sample generation reveal the structure inherent within observed data. Such structure can be expressed in the pattern of interactions between explanatory latent vari- ables captured through a probabilistic graphical model. Although the learning of structured graphical models has long history. much recent work in unsupervised modelling has instead emphasised flexible deep-network-basse generation, either transforming independent latent generators to model complex data assuming that distinct observed variables are derived from different latent nodes. Here, we extend amortised variational inference to incorporate structured factors over multiple vari- ables, able to capture the observation-induce that results from "explaining away" and thus allow complex observations to depend on multiple nodes of a structured graph. show that appropriately parametrised factors can be combined efficiently with variational message passing in rich graphi- cal structures. We instantiate the framework nonlinear Gaussian Process Factor Analysis, evaluating the structured recognition framework using synthetic data from known generative processes. We fit the GPFA model to high-dimensional neural spike data from the hippocampus of freely moving rodents, where the model successfully identifies latent signals that correlate with behavioural covariates.
posterior dependence between latents
Introduction
central challenge unsupervised learning is to identify and model patterns of statistical dependence in high-dimensional data. One approach to this challenge exploits latent-variable generative models, which capture the statistical structure of data through the modelled influence of the latent variables on observations, and through interactions between the latents themselves. Recent developments in deep learning have enabled deep generative models (DGM), with remarkable success in density estimation and high-fidelity image or text generation [1-5]. However, much of the DGM development has emphasised the expressiveness and accuracy of the ntent-tt-obbserattin generative process, with the latents themselves often assumed independent priori.
Variational learning requires inferring or approximating the posterior distribution of the latents conditioned observations [6]. In a variational autoencoder [VAE; 21, inference for DGM is amortised by training recognition network to return parameters of the variational posterior. The structured variational autoencoder (SVAE; 7] connects the DGM framework with richer models that include structured prior dependence amongst the latent variables. an SVAE the structured latent prior is combined with DGM link between each latent and corresponding observation. Variational inference is amortised using recognition networks that return parameters of a factor associated with each link, which are combined with the structured prior to obtain the full posterior. However, in
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

many generative models of interest, observations depend on more than one latent variable. Such interactions induce joint potentials in the likelihood, coupling latents in the posterior even if they are independent priori, phenomenon sometimes called "explaining away" [8]. To capture such inpervation-iddueee dependence in the DGM setting we develop the structured recognition VAE (SRVAE), where recognition potentials that incorporate joint factors induced by the observations are learnt in a (structured) variational autoencoding framework.
We instantiate the SRVAE framework for range of graphical structures, including non-linear latent Gaussian process (GP) models. the latter case, we bring together the GP-prior VAE [9, 10], and (sparse) Gaussian process factor analysis (GPFA) [11, 12] models, yielding a novel autoencoding model (SR-nIGPFA). Experiments show that SR- BR-nIGPFA outperforms alternatives that lack structured recognition. We apply SR-nIGPFA to data collected from population of neurons in the hippocampal complex, and show that the unsupervised approach automatically captures dimensions underlying neural firing that reflect relevant behavioural correlates.
2 Background
2.1 Variational Inference and Structured Variational Autoencoders
Consider general generative process with latent variables and observations y,
2 ~~ p(200,
y ~ 7())
where 0 and y are the parameters of the prior and conditional likelihood distributions, respectively (and we write è = (8.-) 7)). We assume deterministic 0 for simplicity, but distributions over 0 (with conjugate hyperpriors) can be incorporated by variational Bayes (VB) [13]. The posterior distribution, p(zly, Q) xx p(=[0)p(yl 7). is often analytically intractable. In such cases, a common approach is to seek approximation g(=) constrained to a tractable class by variational inference (VI); that is, by minimising the KI.-divergence to the true posterior distribution or, equivalently, maximising a variational free energy [6] Fvi (log =(0) log (())) q(=)
(2ly, θ) = argmin KL KL(g())llp(zl)) θ)] = argmax ( (logp(y, 2 ()) log g()))
(v1()) (2ly,
(1)
9EQ
In the variational autoencoder (VAE) architecture, the parametric optimisation of (()) implicit in (1)
amortised by recognition network that takes input the observed values y and returns parameters of q. The parameters of this recognition network are then trained jointly with the generative model by stochastic optimisation of the free energy [1. 2].
In both standard and amortised VI, the approximate distribution g is often constrained to factor over the latent variables, a so-called mean-field constraint. While tractable, such mean-field approximations may be too restrictive to capture the complexity of the true posterior [14, 15]. Many approaches have been proposed for improving the expressiveness of variational approximation [16-19]. Here we consider the structured VAE [SVAE; 7]. Unlike standard VAEs, the SVAE assumes the generative prior distribution to be specified by structured probabilistic graphical model (PGM), p(2(0) oc Ileec (Ze),), where (CE correspond to C' clique potentials. In addition, the amortised inference network outputs recognition factors, r(zly, Φ). that approximate the generative likelihood function rather than the full variational posterior. These recognition factors are combined with the structured prior distribution to obtain the amortised variational posterior,
(p( p(=10)p(ul2,7)
gsvAE @svae(zly, +) = argmax
Σ log
x r(2ly.0"))paal (=)(()))), (2)
((2)
qoor(= by ()(()))))) DEy
g(=)
where Ø" minimises the averaged KI. over the data observations V. Johnson et al. [7] considered recognition factors that are local (singleton) evidence potentials, chosen to be conjugate to p(=10). Even in this case, the form of (2) allows the dependency structure established by the prior to be carried over to the variational posterior distribution. Further details of SVAE appear in Appendix A.
2.2 Gaussian Process Factor Analysis
Gaussian Process Factor Analysis (GPFA) ) a model used neural data analysis to infer the dynamic latent structure underlying high-dimensional population spike trains [11, 12]. Standard GPFA assumes

---

the following generative model:
latent functions: (.) (P(m)() (z),)) x))+,))). :)), for h I....,,,
affine embeddings: h.,(.) - Enn C fth())) + dn, for 1, ,,,
(3)
observations: (t) (( for I....,T,
where (-)-) and (....)) parametrise the GP prior, Cjk and d, define an affine mapping from the latent space the observation space (or transform of the observations for general likelihoods), and g(.) is a smooth scalar link function appropriate for the observation distribution. We take m.).) = 0 unless stated otherwise. Correlations between observed neurons captured by dependence on the common set of latents, while temporal correlations in the high-dimensional observations are modelled by the latent temporal correlations of the GPs. The PGM describing the GPFA generative model (with sparse variational approximation, see below) is shown in Fig la.
Although maximum- -ikuul-oooeehoood parameters for GPFA be found using (variational) expectation- maximisation [20, 11] exact GP inference scales cubically in the number of observation times [21]. Sparse variational GP (svGP) inference based on auxilliary inducing points [22] reduces the time complexity learning in GPFA [23], and also facilitates efficient extensions to (pointwise) non- conjugate likelihoods [12].
For 1,..., K, we introduce inducing values uı representing the evaluations of the latent process fk at Mr inducing locations, Zk. For simplicity, assume M1 .... Mx = M unless stated otherwise. The GPFA generative model (3) can be augmented to include these auxiliary variables:
p(u: zx) = (O Kk ) p(f* (-)lm) = (Px( (:). (.( Ba(-)Kk
(.)), (E
(4)
F4z4
where (x(+) = x²( Zb: 3,2, )-1
(K ) is the linear operator that maps un to fh(();; 1g zx) a vector-valued function with xa (I; [nC] (z, 2k1), (z ...)...., ny (z, ZKM,, }}; and KE 2, is the covariance matrix obtained by evaluation of the kernel function x-)(-) at the inducing locations Zk.
Introducing a variational distribution over the inducing points and the latent functions, and utilising the generative model (Fig la), the free energy of the sparse variational GPFA (svGPFA) has the form
[ p(y: ff C. (FFUU (U)
F(O, C, d. 2. σ)
log
(5)
q(F, ()
((F.U:0)
where (with [z]] denoting an array obtained by iterating over the index i), = [vn(t)] E RN
f, = [()] E R
[ E RT'K C= [cnx]] E RNX d = [dn] E IR-V U = [ua] E RKXM
and Z = [2 T E x The svGP approach constrains qF,, U; x) to the form q ((U)
with Gaussian (U) [22]. In the multi-GP case, the distribution is typically also taken to factorise
over processes [23], q(F, U) = I ff (q(ux) with f = [ ((T) e R², g(uc) N(m,, St). Under these constraints on the svGPFA free energy simplifies to
K
FWOPHA = Σ (ly: ply: he)
Σ k=
(L(g(u))llp(u zz)],
where ((h:) = fdu (U) is itself a GP with mean and kernel functions given by
mf = Σ (t) dn and (t,tt')) Σ (aa 3 (1) KZazz (t))))
k=1
Hence the computational cost for sparse variational GP inference reduces from O(T3) down to O(M3 + +TM²).
Structured Recognition and Explaining Away
3.1 Structured Recognition in VAEs
general non-linear likelihood may induce additional latent dependency structure in the posterior beyond that of the prior (a simple illustrative example appears in Appendix C). Although the SVAE
3

---

Max
(aa)
Paus
Fa
Chax d
Clox +d
r(bly, ólp(u) r(bly.olp(uu
Chux +
r(uly,
h.
b
(bly.c)
Y
Yo
(a)
(b)
(c)
Figure 1: Graphical models for sparse variational GPFA models. (a) Generative model of GPFA model with sparse inducing point approximation: (b) Standard GPFA inference with sparse amortised variational approximation (note that C I and for SGP-VAE [10]): (c) Structured recognition potential enables full-covariance variational inference in SR-nIGPFA model.
approach developed by Johnson et al. [7] composed amortised inference with a structured prior distribution, the recognition models these authors discussed all contributed amortised potentials that factored over latent variables. Such approach cannot accurately model posteriors with dependence structure that differs from that of the prior PGM.
Here we adopt recognition network that outputs structured factor potentials over the latents, providing amortised estimates of the additional dependencies induced by "explaining away". We denote the resulting model the Structured Recognition VAE (SRVAB). The SRVAE variational approximation takes the form
gsRvAE 0,),0,0) II rEC
de (2ci 0) II Ec(zely:o),
(6)
eEC.
where C, is the set of recognition factors and the &.s the associated factor potentials. We assume that the E.s are chosen to conjugate to the prior factors unless otherwise stated. Hence the analytical (approximate, if necessary) form of @SRVAE the most general form, we could let E(2(yr Φ) be a single joint factor potential over the complete set of latent variables.
be computed with (variational) message passing. In
The free energy objective takes the same form as the standard autoencoding free energy objective [1, 2]. but with the structured amortised variational approximation, ISRVAE, as the variational distribution.
(S)VAE Φ) (log p(=10) log 2 7) log 9SRvAB y, d, 0)) qssvvE (=)a.o.0)
The training of the model follows standard VAE-style stochastic optimisation to update the parameters of the recognition and generative networks, as well as (optionally) the parameters of the prior PGM.
The following proposition supports the use of SRVAE framework (the proof appears in Appendix E).
Proposition 3.1. The SRVAE objective function provides a lighter lower bound to the free energy than the SVAE objective function.
max (0, 2 max FSRVAE 0) 2 max FS.AE(O,
(8)
Below describe two instantiations of the SRVAE framework, the first based on a latent Gaussian mixture model, and the second on a latent GPFA model, both with DGM outputs. Structured amortised inference facilitates scalable inference of the posterior latent distribution with full covariance structure, allowing more accurate learning than with factored recognition approaches. However, the SRVAE framework is more general-purposed, and can be combined with many different latent variable models. Further examples appear in Appendix E.

---

3.2 Structured Recognition with Latent Gaussian Mixture Model
The first experiments in Section 5 apply SRVAE to the setting of latent Gaussian mixture model (GMM) [24] of the form (Figure 5a)
Ex J (#)e))rical==/m)
h (h (30) Σ (a))
y: (yN (he), INN(h.))
This model was studied by Johnson et al. [7], who used an SVAE with fully factorised recognition potentials and variational message Passing[VMP: 25J obtain the GMM variational posterior (details can be found in Appendix A). However, the neural-network defined conditional model induces posterior correlations between the latent variables beyond those imposed by the prior. Thus here we used an SRVAE model, with a full covariance Gaussian recognition potential on h, combined with the prior factor using VMP.
3.3 Structured Recognition Variational Autoencoding Nonlinear GPFA
The GPFA model (3) incorporates a generalised- linear likelihood, with the link function g()) acting separately each affine embedding value h, We now in position to use the SRVAE framework to extend GPFA to include a DGM likelihood, greatly increasing the expressiveness of the generative model. Consider observations {(r2, ye)) and define
RR and h, E RV to be the corresponding vectors of latent process values and embeddings at Z2- [In neural applications, the input 2, is usually taken be identical to the timestamp 4; see (3). Here consider the more general case of arbitrary time-dependent inputs.] The generative model we consider retains the affine mapping from f, to h, but replaces the link function by a non-linear multivariate DGM: yr p(ylg (h:; 7)). Although the generative affine mapping could, in principle, be subsumed within the general deep network, the embeddings h, will play valuable role parametrising structured recognition.
For amortised svGP inference, our goal to define a variational distribution q(U Φ). Ashman et al. [10] have considered a similar latent GP model called the SGP-VAE There, the variational distribution over the latents was found by combining the prior GP distribution with an amortised approximation of the likelihood function factored over latent processes [22]:
(F, U) II (f Iu (ux))();)) where (uu)) x p(us) II r(uu Ne, Zk),
(10)
The graphical model for SGP-VAE inference corresponds to that of Figure Ib, with C and d 0. While this approach captures induced correlation amongst the inducing points, it fails to capture correlations between latent processes that arise in the posterior through "explaining away", potentially leading to sub-optimal inference and learning.
Our solution is to recast the output of the amortised inference to the likelihood of h rather than U, and propose the following structured variational distribution.
QF.UU)= = II p(F (R)) q(U), where q(U) x / dHp(UU)p(H) U) II (I(h.ll)))
(11)
and r(bxlye) ((h ()) = ((, W with wh diagonal, and H [h(21)..... h(zT)]. Given the Amear-Gaussine relationship between U and h (Fig la), the inte- can can computed closed form. Furthermore, even though the recognition potential on h, r(h y.). ). is assumed to be fully factorised over the dimensions of each h, the variational distri- bution on U includes coupling between latent processes induced by the affine mixing coefficients.
gral (11)
Hence (U) captures the correlations both between the latent processes and through time (through combination with the GP prior under the structured autoencoding formulation).
Specifically, given the general expression of the GPFA generative model and the sparse approximation with inducing points, the linear Gaussian relationship between h and U at any x is given by
p() x) (C(3 d, C(K. (())
(12)
5

---

where
(z)
Kzyz1
((x)=
Ku
3K (x)
K z
with K₂ = diag (n (z. æ)]. Combining this result with the fully factorised recognition potential on h at 24 (11), we obtain the structured variational distribution on U:
((U) = NV(u,, x PUU) II N (CC((2)) w/),, with
(13)
Si = Kut 1 + Σ B ((x)) c² ( mu = Su "(())
d1)
This variational posterior on U leads corresponding posterior on h at each x::
(h(x)) = fau (U)
(C((())) ( + d, C(K, - 3(x) - Ku)
and we write ((((z,)) = (m(m: (t).. Then the complete (reparametrised) Monte Carlo estimate of the free energy objective given mini- batch of data, {(zs, yo) takes the following expression.
B
F,0,7,0, 210,7,0,c.d.7 d.
log logplys/m + L.e.) (L(g(U)|IP(U))
(13)
where E. /N 1), 0 is the set of kernel parameters, L is the lower-triangular Cholesky component of S such that S (L.)).
We note that we have retained the scalability of amortised inference by choosing an amortised diagonal-Gaussinn potential on h. However, the linear- Gaussian relationship between the inducing points U and (2x) (12) leads a full-covariance variational Gaussian approximation for U, allowing amortised inference to capture the observation-induee (commonly referred to as the "explaining away" effect). Thus, the retention of the affine GPFA mapping introduces key extenstion to SGP-VAE. We will refer to the new model as structured recognition non-linear GPFA (SR-nIGPFA). The generative model corresponds to that of svGPFA (Figure la) but with the nonlinearity g(-) generalised to a flexible form modelled by neural network. The complete inference procedure for SR-nIGPFA is graphically illustrated in Figure
posterior correlations between latent processes
The implicit computation of the variational distribution over U given the recognition potentials on h also makes possible carry out svGP inference with changed inducing locations (13). This is particularly useful in situations that require inference over test datasets of different durations to those seen training. particular, may be possible learn an amortised inference network using short sub-sequences drawn from a longer dataset, and then infer a posterior over latent GPs for the complete data sequence efficiently by optimising the placement of inducing points along its full length. See further details on the free-form svGP inference step in Appendix F.2.
4 Related work
Structured Deep Generative Models. number of studies have considered latent graphical structure within the DGM framework [26, 27, 7, 28 28-30). Our work builds on and generalises the prominent SVAE proposal of Johnson et al. [7]. Both SVAE and SRVAE combine the structure of a prior PGM with the flcxibility of neural-network-baee recognition potentials. These were taken to extend over single latent variables in the earlier study. Here, we consider joint potentials that capture dependence induced by the generative likelihood. In closely related study, Lin et al. [28] proposed a structured inference network, which approximates the variational distribution the combination of the recognition potential and separate structured latent distribution independent of the prior, the independent structured latent distribution can have non-conjugate factors improve the expressiveness of the model.
recognition. The difference lies in the form of the
SvGPFA and Extensions. Since its introduction as model to identify linear low-dimensional struction in neural population data [11], GPFA has been extended to incorporate non-linear link

---

5.0
SRUNE-CCIM
4.5
SIN GMM
SIN CMM
84.0
VAEE
VD0HM
Ing recoraaiion
2.5
2.0
(a)
(b)
(c)
(d)
(c)
(f)
Figure 2: Empirical Evaluations Pinwheel Dataset. (a) Sampled data from the pinwheel dataset; Training curves of the (b) variational free energy objective and (c) testing reconstruction MSE through training (all in log-scale); Sampled data given trained model of (d) SRVAE-GMM: (c) variational GMM; (f) SIN-GMM [28]. All evaluations are based on the averages over random seeds.
functions and non-conjugate (notably Poisson count or point-process) noise models, and combined
with sparse variational inference for efficiency [23, 31, 12, 32). Related work has considered GP latents in the context of DGMs, particularly using variational autoencoding [9, 33, 10]. In a sense, SR-nIGPFA combines both approaches. incorporates multiple latent GPs (as in GPFA and some DGM GP models) with an affine mapping that feeds into a nonlinear DGM. However, this structure means that the affine map contributes little the generative process. Instead it provides a target for amortised inference, which combines with sparse variational GP inference to yield full structured posterior on inducing point values. As such, the affine map may arguably be better seen an element of the structured recognition model.
Results
We evaluated SRVAE methods for both latent mixtures, and nIGPFA, comparing to relevant baselines on synthetic and real datasets. Empirical results on the instantiations the SRVAE framework with other latent variable models can be found in Appendix F!.
5.1 Experiments with SRVAE-GMM
To assess the empirical performance of structured amortisation in standard latent-variable model, we evaluated SRVAE G-GMM on the pinwheel dataset of [7] (Figure 2a). We compare SRVAE-GMM with the structured inference network [SIN; 28), which is more flexible instantiation of the SVAE framework. From Figure 2, we observe that SRVAE-GMM the training variational free energy, reconstruction reaa-s error, and generation fidelity. The models are identical apart from the recognition stage, providing support for the idea that by capturing the "explaining away" posterior factors, structured recognition leads to more accurate learnt model.
outperforms SIN-GMM in terms of
Python implementation can be found at Atttps://gtt htt.///ggttttuttt./ hwtp://cto/aabaaaaaaaeae.
structured- ceweidereeoogniiiio tion- wtw.rrewwwonitttoooooeee2e22222
7

---

Synthetic data
EEG
Population spiking
SMSE
NLL
SMSE
NLL
SMSB
SR-nIGPFA
0.31 + 0.02
1.69 + 0.04
0.27 + 0.05
1.81 + 0.21
0.47 = 0.18
SGP-VAE [10]
0.36 + 0.05
1.76 0.07
0.35
2.17 0.17
0.55 = 0.19
Vanilla-VAB [1. 2]
1.05 1 0.02
13.60 3.58
0.57 0.09
3.45 0.87
3.19 = 1.98
SVAE-LDS [7]
0.933 + 0.02
9.37 1.94
3.04 0.38
11.71 2.66
2.52 1.31
Table Quantitative comparison performance between SR-nIGPFA and baseline models on syn- thetic dataset, EEG dataset and population spiking dataset (Section 5.2.2) with respect to standarised mean squared error (SMSE) and negative log-likelihood (NLL). Averages over random seeds.
5.2
Experiments with SR-nIGPFA
5.2.1 Synthetic and EEG Dataset
We compared SR-nIGPFA to methods that do not capture the inter-latent posterior correlation, using synthetic data and small-scale EEG dataset used previously [34, 10]. Unless stated otherwise, we employed exponentiated quadratic kernel, s(<z, x²) À expp( ||m=-'113, where À and the marginal variance and length-scald parameters, respectively.
Baselines Our main baseline model is SGP-VAE [10] (described in Sections 3.3 and 4). We also compared to a "vanilla" VAE with fully factorised Gaussian variational distribution [1, 21, and SVAE with linear dynamical system (SVAE-LDS) latent prior [7]. All models were implemented with the same recognition and generative network architectures (see Appendix G for implementation details).
Synthetic Dataset We generated data from the GPFA generative model (Eq. 3), with g()) = a (+)),. where ((-) represents the functional mapping through a fixed, randomly-initialisen 2-layer MLP with ReLU hidden non non-linearitty and (-) is the sigmoid function.
EEG Dataset We follow the experimental procedure described by Requeima et al. [34], and consider an EEG measurement dataset spanning second at 256 Hz sample frequency, taken during image viewing [35]. Each datapoint consists of the voltage readings from 7 electrodes positioned participant's 's scalp. Here we report results when all data are observed, contrary to the settings in [34, 10] (we include results and further discussion partial observability in Appendix F.2).
the
SR SR-nIGPFA achieved lower standardised mean squared error (SMSE) and lower test negative log- likelihood (NLL) than the alternative methods on both data sets (Table 1), providing evidence for the benefits of structured recognition. These gains come despite the computational complexity of SR-nIGPFA being of the same order SGP-VAE (see appendix F.2).
5.2.2 Population Neuronal Firing Data
The firing of place cells in Hippocampal area CAI and grid cells in the medial Entorhinal Cortex (mEC) is known to be modulated by the animal's ' location [36, 37], speed and direction of locomo- tion [38-40], with the mapping between these behavioural covariates and neural activity expressed in non-linear mixed tuning curves. The behavioural signals are continuous and often mutually depen- dent, and so create temporal and spatial structure in the time series of population activity. We asked whether SR-nIGPFA and related methods would be able to identify and extract this structure without supervision; that is, without direct access the behavioural covariates.
We used single-cell spiking data from neurons in the hippocampal CAI and mEC regions of rats recorded during exploration of a Z-shaped track, as reported by Ólafsdóttir et al. [41]. The data comprised 28 experimental sessions, each spanning 10 minutes. Example neural firing patterns of the population are shown in Figure of the Appendix.
For SR-nIGPFA and SGP-VAE, adopted the GPFA generative model Eq. with DGM non-line con-linearity and Poisson observation likelihood.
N
p (()())((()) = II (x))))())(()))g((((()))
(14)
8

---

where x are discrete times, N is the number of recorded neurons, and g(-) represents the neural- network mapping from the GPFA features, h, the rate of each neuron over contiguous 100 ms bins. SR-nIGPFA obtained a lower SMSE of prediction than SGP-VAE and other methods tested (Table 1).
firing counts for
Computational constraints meant that the SR-nIGPFA model was trained using short batches of data and 64 inducing points per batch. Latent trajectory estimates derived from such batches will not necessarily be continuous at the boundaries between them. Thus, once the model was fit, we performed svGP inference over complete sessions with increased numbers of inducing points (see Section 3.3 and Appendix F.3.2).
To relate these recovered latent time-series behavioural covariates, performed two-dimensional Canonical Correlation Analysis [CCA; 42]. We present results for one session here, emphasising qualitative effects. Further qualitative and quantitative results across sessions can be found in Appendix F.3.2.
Figure 3a shows heatmap of the correlation coefficients between the canonical correlates of the posterior means (1, 2)) and the individual behavioural covariates (distance from one end of the track, speed, direction of travel, head direction, and 'unfolded" position along a full lap of the track). Many correlations are high, indicating the low-dimensional manifold parametrised by the conjunctive set of behavioural correlates can be accurately captured by the posterior latent variables learned with SR-nIGPFA solely from neural spikes. Please refer to Appendix Figure 13 for numerical values of the correlations (and for other sessions).
To see whether the learned latent structure contained decodable information about behavioural covariates, we extracted the direction direction-modullted modulated neurons predicted by the trained model (details in Appendix F.3.2). Figure 3b compares the direction modulation of the model-predicted neurons against that of the rest of the neurons, where direction modulation is defined as the correlation between raw single-cell spike counts and the direction values. Neurons predicted by the model to be direction modulated exhibited significantly greater direction modulation than the other neurons (p-value=5.56 x 10 [43]. Figure 3c shows the firing profile for travel in each direction for the neuron with the strongest predicted direction modulation. The pattern of firing fields exhibits clear directional dependence. Similar comparisons hold for spatial and speed modulation, indicating that SR-nIGPFA is able, in purely unsupervised fashion, to learn latent space that contains linearly decodable information about the behavioural covariates associated with individual neuronal firing, even for neurons whose activity exhibits conjunctive coding.
Figure 3d shows the two canonical correlates obtained from the latent trajectories as a function the animal's spatial location. These refecc both location and direction of movement (colours), indicating the latent dimensions learned by SR-IIGPFA disentangle direction from spatial location a simple linear projection, despite the conjunctive coding mechanism exhibited in the CA1 neurons. A similar plot derived from SGP- VAE (Figure 14b), shows less clear disentanglement.
Figure 3e shows the temporal evolution of the latent CCs for SR-nIGPFA and SGP-VAE, as well as the spatial location of the rat. Both CCX1 and CCX2 of SR-nIGPFA consistently track the dynamics of the distance (with perfect phase alignment and half-cycle offset, respectively), whereas neither CCXI or CCX2 of SGP- VAE could accurately reflect the distance. Moreover, given the additional svGP inference step over the entire trajectory (with increased number of inducing points), latent signals of SR-nIGPFA exhibit significantly greater smoothness than that of SGP-VAE
6 Discussion
We have developed a framework of structured recognition for variational autoencoders, which can be viewed as a strict generalisation of the SVAE [7], capturing posterior correlations induced by the "explaining away" effeect. Correspondingly, the SRVAE framework yields a tighter bound on the log-likelihood than the SVAE free energy objective.
Applying structured recognition and non-linear variational autoencoding to GPFA, we introduce SR-nIGPFA, facilitating scalable free-form variational Gaussian approximation. SR-nIGPFA supports a fully non-linear, non-conjugate extension to standard GPFA [11, 23, 12], which is particularly well-suited to settings where posterior inference must be scaled to additional data. We show that SR-nIGPFA outperforms the bascline on all presented tasks, both quantitatively and qualitatively.

---

Mard
04
0.25
EST moodd protittod
(a)
(
20140012, POTTOR FUEST (SNVECITAL
cox1 Oox1-Pou (5PM)12PM))
OCAGk (BRADDPAA)
GOAL (ASEO)
COG BART
300
100
203
1009
.243
3000
0000
(d)
(f)
Figure Analysis SR-nIGPFA posterior latent structure. (a) Heatmap of the correlation
between the CCs and the behavioural covariates for the selected session; (b) mnooulaatllmoollatiooon comparison between the model-predicted Exemplary firing fields of the model-preeiccerd firection-modulatee neuron during inbound outbound movements; (d) Plot of CCX {1, 2} of SR-nIGPFA posterior latents against spatial location, color indicates direction of movement (yellow: inbound, magenta: outbound); (e-f) Temporal evolution of (standardised values of) (CXX(122 and spatial location for SR-nIGPFA and SGP-VAE.
neurons against other neurons; (c)
Studying population spiking data of hippocampal neurons, we show that SR-nIGPFA is able to identify latent signals that exhibit strong correlation with the behavioural covariates that are well-know to conjunctively encoded by the recorded neurons, but without direct access these covariates. While here we focused on temporal correlation exhibited in time-series data, we note that the sparse amortised variational approximation presented here can also be straightforwardly applied other GP-latent models, such as GP-LVM [44, 45]. While have considered spiking data alone, these could be substituted by or combined with EEG, calcium imaging, or indeed behavioural data recorded during the same session. Such joint analysis of latent structures given different recordings within the same session could potentially facilitate new findings with respect to, e.g., replay, phase coding, etc.
Acknowledgement
We thank the anonymous reviewers for helpful comments and discussions, and Zilong Ji and Dan Bush for helpful discussions regarding the experiments with the neural data. This work is funded by the UKRI, DeepMind, the Gatsby Charitable Foundation, the Simons Foundation and the Wellcome Trust.
10

---

References
[1] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[2] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and upproximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[3] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[4] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462-1471. PMLR, 2015.
[5] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.053328. 2016.
[6] Martin J Wainwright and Michael Irwin Jordan. Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008.
[7] Matthew Johnson, David Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta, Composing graphical models with neural networks for structured representations and fast inference. Advances in neural information processing systems, 29:2946-2954 2016.
[8] Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988.
[9] Francesco Paolo Casale, Adrian Dalca, Luca Saglictti, Jennifer Listgarten, and Nicolo Fusi. Gaussian process prior variational autoencoders. Advances in neural information processing systems, 31, 2018
[10] Matthew Ashman, Jonathan So. Will Tebbutt, Vincent Fortuin, Michael Pearce, and Richard E Turner. Sparse Gaussian process variational autoencoders. arXiv preprint arXiv:2010.10177, 2020.
[11] Byron M Yu. John Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V Shenoy, and Maneesh Sahani. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Advances neural information processing systems, 21, 2008.
[12] Lea Duncker and Maneesh Sahani. Temporal alignment and latent Gaussian process factor inference in population spike trains. Advances in Neural Information Processing Systems, 31, 2018.
[13] Hagai Attias. Inferring parameters and structure of latent variable models by variational bayes. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, pages 21-30, 1999.
[14] Lawrence Saul and Michael Jordan. Exploiting tractable substructures in intractable networks.
Advances in neural information processing systems, 8, 1995.
(15) Matthew D Hoffman and David M Blei. Structured stochastic variational inference. In Artificial Intelligence and Statistics, pages 361-369, 2015.
[16] David M Blei and Michael Jordan. Variational inference dirichlet process mixtures. Bayesian analysis, 1()):121-143, 2006.
[17] Dustin Tran, Rajesh Ranganath, and David M Blei. The variational gaussian process. arXiv preprint arXiv: 2015.
[18] Yingzhen Li, Richard E Turner, and Qiang Liu. Approximate inference with amortised meme. arXiv preprint 1rXiv:1702.08344 2017.

---

[19] Dong Bok Lee, Dongchan Min, Seanie Lee, and Sung Ju Hwang. Meta gmave: Mixture of gaussian vae for unsupervised meta-learning In International Conference on Learning Representations, 2020.
[20] Arthur Dempster, Nan Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-2, 1977.
[21] C. E. Rasmussen and C. K. I. Williams. Gaussian processes for machine learning. MIT Press, 2005.
[22] Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pages 567-574. PMLR, 2009.
[23] Vincent Adam, James Hensman, and Maneesh Sahani. Scalable transformed additive signal decomposition by non-conjugate gaussian process inference. In 2016 IEEE 26th international workshop on machine learning for signal processing (MLSP), pages 1-6, IEEE, 2016.
[24] Christopher M Bishop. Pattern recognition and machine learning, volume 4. Springer, 2006.
[25] John Winn, Christopher M Bishop, and Tommi Jaakkola. Variational message passing. Journal of Machine Learning Research, 6(4), 2005.
[26] Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. arXiv preprint arXiv:1511.0512, 2015.
[27] Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lec, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint 1e1iv::611.0224444 2016.
[28] Wu Lin, Nicolas Hubacher, and Mohammad Emtiyaz Khan. Variational message passing with structured inference networks. arXiv preprint 0rXiv:/803.055000 2018.
[29] Chongxuan Li, Max Welling, Jun Zhu, and Bo Zhang. Graphical generative adversarial networks Advances in neural information processing systems, 31, 2018.
[30] Max Paulus, Dami Choi, Daniel Tarlow, Andreas Krause, and Chris Maddison. Gradient estimation with stochastic softmax tricks. Advances in Neural Information Processing Systems, 33:5691-5704, 2020.
[31] Yuan Zhao and II Memming Park. Variational latent Gaussian process for recovering single-trial dynamics from population spike trains. Neural Computation, 29(5): 2017.
[32] Stephen Keeley, David Zoltowski, Yiyi Yu, Spencer Smith, and Jonathan Pillow. Efficient non- conjugate gaussian process factor models for spike count data using polynomial approximations. In International Conference on Machine Learning, pages 5177-5186. PMLR, 2020.
[33] Alex Campbell and Pietro Liò. tvgp-vae: Tensor- gaussian process prior variational autoencoder. arXiv preprint
2020.
[34] James Requeima, William Tebbutt, Wessel Bruinsma, and Richard E Turner. The gaussian process autoregressive regression model (gpar). In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1860-1869. PMLR, 2019.
[35] Xiao Lei Zhang, Henri Begleiter, Bernice Porjesz, Wenyu Wang. and Ann Litke. Event related potentials during object recognition tasks. Brain research bulletin, 38(6):531-523 1995.
[36] John O'keefe and Lynn Nadel. The hippocampus as a cognitive map. Oxford university press, 1978.
[37] Edvard Moser, Emilio Kropff, and May-Britt Moser. Place cells, grid cells, and the brain's spatial representation system. Annu. Rev. Neurosci., 31:69-89, 89, 2008.
12

---

[38] Bruce L McNaughton, Carol A Barnes, and JJEBR O' Keefe. The contributions of position, direc- tion, and velocity to single unit activity the hippocampus of freely- movingg rats. Experimental brain research, 52(1):41 1983.
[39] Francesca Sargolini, Marianne Fyhn, Torkel Hafting, Bruce L McNaughton, Menno P Witter, May-Britt Moser, and Edvard Moser. Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science, 3 2006.
[40] Emilio Kropff, James E Carmichael, May-Britt Moser, and Edvard Moser. Speed cells in the medial entorhinal cortex. Nature, 523(7561):419-422 2015.
[41] H Freyja Ólafsdóttrr, Francis Carpenter, and Caswell Barry. Coordinated grid and place cell replay during rest. Nature neuroscience, 19(6):992-794, 2016.
[42] David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):2639-28) 2664,
2004.
[43] Francesca Cacucci, Colin Lever, Thomas J Wills, Neil Burgess, and John O' Keeefe Theta- modulated place-by-direction cells in the hippocampal formation in the rat. Journal of Neuro- science, 24(38))8265-8277) 2004.
[44] Neil Lawrence. Gaussian process latent variable models for visualisation of high dimensional data. Advances in neural information processing systems, 16, 2003.
[45] Anqi Wu, Nicholas Roy, Stephen Keeley, and Jonathan W Pillow. Gaussian process based non- linear latent structure discovery in multivariate spike train data. Advances in neural information processing systems, 30, 2017.
[46] Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 2016.
[47] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: /6/ 16:::01/110000 144, 2016.
[48] Garoe Dorta, Sara Vicente, Lourdes Agapito, Neill DF Camphell, and Ivor Simpson. Training vaes under structured residuals. arXiv preprint TrXXvv/804.0005555 2018.
[49] Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. In International Conference on Machine Learning, pages 4277-4285. PMLR, 2018.
[50] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. www.ll cow.uun.oo//exu//iis..... 2, 2010.
[51] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http:: http:://pppprre.e.eiiepepee 9015- 2015-pytorch an- impeiaiiieeeeeeeeeeeeeeeeeeeeeee style high- www.-eeerffoo-eeeeeee www.rrrmwannnneeeeeeeeeeeeeeeeee wwweiiinng-bbbrrryy library. pdf.
[52] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.698. 2014.
[53] Edward H Nich, Manuel Schottdorf, Nicolas W Freeman, Ryan J Low, Sam Lewallen, Sue Ann Koay, Lucas Pinto. Jeffrey L Gauthier, Carlos D Brody, and David W Tank. Geometry of abstract learne knowledge in the hippocampus. Nature, 595(7865):80 84, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "ZGwU7C3Pk5j", "reviewer": "Reviewer_L1xL", "summary": "The paper proposes a way for amortised inference in structured generative models with explaining away. It defines a specific form of variational posterior that is propositional to the product of the prior distribution and the amortised factor potentials instead of a mean-field ones used in existing work. The paper proves that the proposed inference model gives a tighter ELBO.\n\nIt further considers two more structured models with a tree prior or a Gaussian process factor analysis prior and applies the proposed method to construct the variational posterior and perform standard learning as in the original SVAE.\n\nThe paper conducts a toy example to show the promise of the proposed method over SVAE and evaluate the proposed model on the EGG data and Population Neuronal Firing Data. It quantitatively compares with SGPVAE. ", "questions": "See weakness", "limitations": "I did not see a negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The motivation for using a \"more structured\" posterior is clear and theoretically justified.\n\n2. The presentation is easy to follow and related work is properly discussed.\n\nWeakness\n1. I'm not fully convinced by the experiments. It would be much better if the authors can compare the proposed method with SVAE and VMP-SVAE on the commonly adopted models and benchmarks therein. For instance, both SVAE and VMP-SVAE are evaluated with a latent mixture model and a latent state-space model. A direct comparison in such settings can strongly support the motivation and theory.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "94FSXMzXWs", "reviewer": "Reviewer_D7Ke", "summary": "This paper proposes to extend structured variational inference to two settings to enable the modeling intra-latent variable dependencies. The first setting is a variational auto-encoder where the variational distribution and the approximated likelihood function is defined on a set of predefined factors (in this paper, a tree). The second setting is a structured version of svGPFA for which one can compute the full covariance matrix. Experiments are performed on a series of synthetic and real data. ", "questions": "SGPVAE is the only baseline for the AEA-svGPFA, yet SGPVAE itself is not yet published on a major venue. Would this be a valid enough comparison? Have you considered other (possibly simpler) baselines, for example, a vanilla VAE or an SVAE with a linear dynamic system latent at least? ", "limitations": "When the order of dependency increase, tractability would become an issue. For example, on the Tree-structured VAE case, how would you discuss if the graph is loopy (e.g., add an edge from one leaf to another) or if the graph is high order dependency (e.g, if a leaf node also depend on its grand father)?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is able to model more complicated dependency beyond VAEs with mean-field inference model and the original svGPFA\n- In the GPFA setting, the proposed method has the same asymptotic time complexity as the baseline SGPVAE\n- The experiments and visualization on the Population spike dataset seems to be valid\n\nWeaknesses\n\n- On the structured VAE thread\n    - Generally, I am not so convinced on the structured VAE side of the model, as there exist other works that also consider tree-structured or higher-order dependency over the latent variables, and there exist further issues about higher-order dependency.\n    - About existing work considering intra-latent dependencies, for example, [1] introduces structured latent variables over permutations and trees, and [2] considers the joint modeling of continuous and tree-structured latent variables. Both works have more detailed discussions (about aspects like modeling and gradient estimation) and are applied to more complicated structures (Math equations and human language syntax trees) than the Bar dataset used in this work.\n    - The tighter variational bound in equation 5 may also not be view as significant (in the context of 2022), as there are works with exact likelihood [3], also works discussing whether tighter bound are necessarily better[4].\n    - Given the existing literature, the tree-structured VAE on the Bar dataset seems not very significant.\n- On the AEA-svGPFA thread\n    - I am more convinced on this extensions in terms of both model sophistication (to facilitate the modeling of full covariance) and the empirical evaluation (especially the visualized dynamics).\n    - Yet I have to admit that I am not very familiar with the related svGPFA literature, so I would not be very confident to say if the contribution on this thread is significant enough for acceptance. I am open to discussions from the authors about why this work is significant enough from the GPFA perspective (and increase my scores accordingly if convinced).\n\nAdditional minor comments\n\n- The fonts of Figure 1 and 2 are two small to recognize without zooming\n- In equation 5, the objective for SVAE is not defined\n\nReferences\n\n[1] Paulus et. al. NeurIPS 2020. Gradient Estimation with Stochastic Softmax Tricks. \n\n[2] Kim et. al. ACL 2019. Compound Probabilistic Context-Free Grammars for Grammar Induction.\n\n[3] Lou et. al. ICLR 2020. SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models.\n\n[4] Rainforth. et. al. ICML 2018. Tighter Variational Bounds are Not Necessarily Better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3A_A8rK0O-", "reviewer": "Reviewer_9rkF", "summary": "Update after author rebuttal:\nthanks for your changes, I believe that the restructuring makes the paper clearer and easier to read. I have now increased my score to a weak accept.\nI still believe that the idea of extending the SVAE to structured factor potentials is quite obvious, and that the strongest contribution of this work is the AEA-svGPFA model. Similarly to reviewer D7Ke however, I must admit I am not familiar enough with the applications of the AEA-svGPFA model and its related literature to properly assess the impact this work can have in the Neurips community if accepted. I will discuss this with the other reviewers and the area chair.\n\n\n=======================\n\nThis paper introduces an extension to the Structured VAE framework that considers structured factor potential instead of fully factorized ones. This allows the model to better capture observation-induced posterior dependencies. \n\nThe framework is applied 1) to a tree structured model 2) to a GP factor analysis model for neural spiking time series, and outperforms competing methods in a number of synthetic and real-world experiment.  ", "questions": "See comments under \"weaknesses\" above.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*STRENGTHS* \n\n1. The paper proposes an interesting an novel application of the SVAE framework to the svGPFA model, which allows to better model posterior dependencies among variables while keeping the model computationally efficient \n2. The AEA-svGPFA outperforms competing methods on neuronal firing data \n3. The authors perform in depth discussions that show the impact of better modelling posterior correlations with the proposed model \n\n \n\n \n\n \n\n*WEAKNESSES* \n\nMy main concerns for this paper, are the novelty of the idea and the clarity of which one is the main contribution. \n\nI consider the proposed framework that extends the SVAE to structured factor potential as obvious, and as such with very limited novelty (same for proposition 3.1). \nEven after reading the SVAE paper alone, I find it natural that for posterior distributions with more complex dependencies among variables one can use factors over multiple variable instead of a mean field approach. \nThis was also noted by the authors of the SVAE paper as a possible extension of their model, in section 3.3. of \"Structured VAEs: Composing Probabilistic Graphical Models \nand Variational Autoencoders\" by Johnson et al, 2016 (notice that this is a different version of the paper from the one you cited). \n\n \n\nAs stated above, for me the main contribution and novelty of the paper is the AEA-svGPFA model, and the paper should make this clear.  \nI would make the AEA-svGPFA model the main focus of the paper, presenting it as an application of the standard SVAE framework to the svGPFA model. \nDoing this, you can \n1. Move the tree-structured latent PGM and the synthetic bar experiments to the appendix, since they would contribute very little to the main storyline \n2. Reduce the SVAE-related discussions \n3. Add to the main text more important details on the AEA-svGPFA that are now in the appendix, which would make the paper easier to read \n\n \n\nMinor comments: \n* Typo in line 260: is->if? \n* In line 269 you mention Figure 14 that does not exist \n* Figure 4 subplots are way too small to be read on a printed version of the paper ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept