Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a way for amortised inference in structured generative models with explaining away. It defines a specific form of variational posterior that is propositional to the product of the prior distribution and the amortised factor potentials instead of a mean-field ones used in existing work. The paper proves that the proposed inference model gives a tighter ELBO.

It further considers two more structured models with a tree prior or a Gaussian process factor analysis prior and applies the proposed method to construct the variational posterior and perform standard learning as in the original SVAE.

The paper conducts a toy example to show the promise of the proposed method over SVAE and evaluate the proposed model on the EGG data and Population Neuronal Firing Data. It quantitatively compares with SGPVAE. 
strengths_and_weaknesses: Strengths
1. The motivation for using a "more structured" posterior is clear and theoretically justified.

2. The presentation is easy to follow and related work is properly discussed.

Weakness
1. I'm not fully convinced by the experiments. It would be much better if the authors can compare the proposed method with SVAE and VMP-SVAE on the commonly adopted models and benchmarks therein. For instance, both SVAE and VMP-SVAE are evaluated with a latent mixture model and a latent state-space model. A direct comparison in such settings can strongly support the motivation and theory.


questions: See weakness
limitations: I did not see a negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to extend structured variational inference to two settings to enable the modeling intra-latent variable dependencies. The first setting is a variational auto-encoder where the variational distribution and the approximated likelihood function is defined on a set of predefined factors (in this paper, a tree). The second setting is a structured version of svGPFA for which one can compute the full covariance matrix. Experiments are performed on a series of synthetic and real data. 
strengths_and_weaknesses: Strengths:

- The proposed method is able to model more complicated dependency beyond VAEs with mean-field inference model and the original svGPFA
- In the GPFA setting, the proposed method has the same asymptotic time complexity as the baseline SGPVAE
- The experiments and visualization on the Population spike dataset seems to be valid

Weaknesses

- On the structured VAE thread
    - Generally, I am not so convinced on the structured VAE side of the model, as there exist other works that also consider tree-structured or higher-order dependency over the latent variables, and there exist further issues about higher-order dependency.
    - About existing work considering intra-latent dependencies, for example, [1] introduces structured latent variables over permutations and trees, and [2] considers the joint modeling of continuous and tree-structured latent variables. Both works have more detailed discussions (about aspects like modeling and gradient estimation) and are applied to more complicated structures (Math equations and human language syntax trees) than the Bar dataset used in this work.
    - The tighter variational bound in equation 5 may also not be view as significant (in the context of 2022), as there are works with exact likelihood [3], also works discussing whether tighter bound are necessarily better[4].
    - Given the existing literature, the tree-structured VAE on the Bar dataset seems not very significant.
- On the AEA-svGPFA thread
    - I am more convinced on this extensions in terms of both model sophistication (to facilitate the modeling of full covariance) and the empirical evaluation (especially the visualized dynamics).
    - Yet I have to admit that I am not very familiar with the related svGPFA literature, so I would not be very confident to say if the contribution on this thread is significant enough for acceptance. I am open to discussions from the authors about why this work is significant enough from the GPFA perspective (and increase my scores accordingly if convinced).

Additional minor comments

- The fonts of Figure 1 and 2 are two small to recognize without zooming
- In equation 5, the objective for SVAE is not defined

References

[1] Paulus et. al. NeurIPS 2020. Gradient Estimation with Stochastic Softmax Tricks. 

[2] Kim et. al. ACL 2019. Compound Probabilistic Context-Free Grammars for Grammar Induction.

[3] Lou et. al. ICLR 2020. SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models.

[4] Rainforth. et. al. ICML 2018. Tighter Variational Bounds are Not Necessarily Better.
questions: SGPVAE is the only baseline for the AEA-svGPFA, yet SGPVAE itself is not yet published on a major venue. Would this be a valid enough comparison? Have you considered other (possibly simpler) baselines, for example, a vanilla VAE or an SVAE with a linear dynamic system latent at least? 
limitations: When the order of dependency increase, tractability would become an issue. For example, on the Tree-structured VAE case, how would you discuss if the graph is loopy (e.g., add an edge from one leaf to another) or if the graph is high order dependency (e.g, if a leaf node also depend on its grand father)?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Update after author rebuttal:
thanks for your changes, I believe that the restructuring makes the paper clearer and easier to read. I have now increased my score to a weak accept.
I still believe that the idea of extending the SVAE to structured factor potentials is quite obvious, and that the strongest contribution of this work is the AEA-svGPFA model. Similarly to reviewer D7Ke however, I must admit I am not familiar enough with the applications of the AEA-svGPFA model and its related literature to properly assess the impact this work can have in the Neurips community if accepted. I will discuss this with the other reviewers and the area chair.


=======================

This paper introduces an extension to the Structured VAE framework that considers structured factor potential instead of fully factorized ones. This allows the model to better capture observation-induced posterior dependencies. 

The framework is applied 1) to a tree structured model 2) to a GP factor analysis model for neural spiking time series, and outperforms competing methods in a number of synthetic and real-world experiment.  
strengths_and_weaknesses: *STRENGTHS* 

1. The paper proposes an interesting an novel application of the SVAE framework to the svGPFA model, which allows to better model posterior dependencies among variables while keeping the model computationally efficient 
2. The AEA-svGPFA outperforms competing methods on neuronal firing data 
3. The authors perform in depth discussions that show the impact of better modelling posterior correlations with the proposed model 

 

 

 

*WEAKNESSES* 

My main concerns for this paper, are the novelty of the idea and the clarity of which one is the main contribution. 

I consider the proposed framework that extends the SVAE to structured factor potential as obvious, and as such with very limited novelty (same for proposition 3.1). 
Even after reading the SVAE paper alone, I find it natural that for posterior distributions with more complex dependencies among variables one can use factors over multiple variable instead of a mean field approach. 
This was also noted by the authors of the SVAE paper as a possible extension of their model, in section 3.3. of "Structured VAEs: Composing Probabilistic Graphical Models 
and Variational Autoencoders" by Johnson et al, 2016 (notice that this is a different version of the paper from the one you cited). 

 

As stated above, for me the main contribution and novelty of the paper is the AEA-svGPFA model, and the paper should make this clear.  
I would make the AEA-svGPFA model the main focus of the paper, presenting it as an application of the standard SVAE framework to the svGPFA model. 
Doing this, you can 
1. Move the tree-structured latent PGM and the synthetic bar experiments to the appendix, since they would contribute very little to the main storyline 
2. Reduce the SVAE-related discussions 
3. Add to the main text more important details on the AEA-svGPFA that are now in the appendix, which would make the paper easier to read 

 

Minor comments: 
* Typo in line 260: is->if? 
* In line 269 you mention Figure 14 that does not exist 
* Figure 4 subplots are way too small to be read on a printed version of the paper 
questions: See comments under "weaknesses" above.
limitations: Yes
review_title: null

Meta-review: metareview: **Summary**: This paper proposes a specific form of variational approximation for amortized inference that is proportional to the product of the prior and amortized factor potentials. This is similar to the approach proposed in work on Structured Variational Autoencoders (Johnson et al, NeurIPS 2016), but differs in that each factor in the prior is paired with a corresponding amortized factor over the same set of variables, whereas the original work on SVAEs assumes mean-field factors over individual variables. The authors evaluate this approach in the context of SVAEs, but the main intended us case is an application to sparse variational Gaussian process factor analysis (svGPFA) models, resulting in a proposed AEA-svGPFA model. 

**Strengths**: Reviewer *L1xL* found that this submission proposes a good approach with clear motivation and theoretical justification, with a good presentation and discussion of related work.  Reviewers *D7KE* and *9rkF* also appreciated the novel application of SVAE to the svGPFA model, allowing it to better capture posterior dependencies, whilst retaining the same asymptotic time complexity as SGPVAE, with good results for the AEA-svGPFA on neural spike population data. Reviewer *9rkF* further appreciated the analysis of impact of modeling posterior coorelations. 

**Weaknesses**: Reviewers were on the whole not fully convinced by the significance of the application to SVAEs [L1xL, D7KE, 9rKF], and each noted that they felt unqualified to judge the significance of the AEA-svGPFA results due to a lack of familiarity of this model class. 

**Reviewer Author Discussion**: The authors updated their submission to add an instantiation of the AEA on latent GMMs, evaluated on the pinwheel domain (Johnson et al. 2016), compared to SIN (Lin et al 2018), a vanilla VAE, and a varaitional GMM. The also discussed why a tighter bound (Proposition 3.1) aids posterior inference (with additional discussion in Appendix D), and provided clarifications to reviewers *L1xL* and *D7KE*. In response to the author discussion, all 3 reviewers raised their score (5->6). 

**Reviewer AC Discussion**: While all reviewers indicated to the AC that they are leaning towards acceptance, the are also all in agreement that this a paper that is borderline and could also be rejected. The main point of deliberation remains whether the paper would be stronger if it focused on the the AEA-svGPFA and omitted the AEA-SVAE results (which remain somewhat unconvincing to the reviewers). The reviewers also reiterated that they find it difficult to judge the significance of the AEA-svGPFA results. 

**Formatting**: There are minor formatting violations; Table 1 and Figure 2 are about 1 inch too wide, but this is easily addressable. 

**Overall AC Recommendation**: The AC took a quick and very cursory look at the the paper. The AC has no immediate concerns about soundness and clarity, but also finds it difficult to assess significance. Taking everything into account, this paper appears narrowly above the threshold for acceptance, but may have to be cut to make room for other submissions. 
confidence: Less certain
Decision: Accept