Venue: NeurIPS 2022 Workshop SVRHM

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: In this very interesting study, the authors compare representations learned by a collection of hierarchical VAE models with the visual system.

Contributions (- neutral, + strong, ++ very strong):
- (-) like classical deep nets, the upper layer of their hierarchical VAE models learns relevant high-level features, useful for example for texture detection.
- (+) unlike deep nets, but like alternative predictive coding models, their models can also account for inpainting capabilities of the visual system, and illusory edge representations in V1.
- (++) unlike other models, their models can also account for known properties of noise correlations in the early visual system, interpreted here as probabilistic representations of a posterior.
- (+) they distinguish between different types of hierarchical VAE models as models of the visual system by comparing their predictions on learned features and noise correlations.

Comment:
- More explanations on the conceptual motivation and differences between the different models proposed, especially between the bottom up vs top-down posterior factorization approaches, would have been useful to better understand the motivation for comparing these specific models (and not alternatives), and to understand the implications of the study for our understanding of visual processing.

Typo:
- I believe equation in line 72 is missing a term and should probably be p_theta(x|z_1,z_2) = p_theta(x|z_1)p_theta(z_1|z_2)

review_title: Very interesting findings by comparing representations learned by hierarchical VAEs and the visual system

final_score: rating: 6: Marginally above acceptance threshold
review_content: The goal of this work is to investigate the properties and representations of the early visual cortex. To tackle this problem a two layer hierarchical VAE was proposed whose generative part is Markovian. The motivation of choosing a Markovian generative model is inspired from neuroscience which encourages interpret-ability in representation learning and establishes a hierarchy between latent layers. To quantify the efficacy of the proposed method, experiments were performed on natural images with applications such as noise correlation for low level representation learning and image in painting and illusory contours for high level representation learning. Results indicated the TDVAE perform better than Linear, Chain and Skip VAEs.   




Pros.

1. The problem statement is relevant and interesting for the community. It is indeed important to study and analyze the representations of VAEs in order to better understand and establish links between generative modelling and neuroscience. 

2. The proposed method is biologically inspired and the performed experiments (although preliminary) seem to back the claims made to analyze and link the latent representations in generative modeling with the understandings of the visual cortex from neuroscience.

3. The other baselines for comparison such as Linear VAE, Skip VAE and Chain VAE are technically correct from the proof of idea point of view. 

Cons:

There are some terms used in the paper which I feel can be explained in a better way.

1. Lines 19, 20 " Generative models of natural images to understand....." What does Generative model of natural images mean?
2.  If the design choices mentioned in the paper for each VAE can be explained that would have been better. For each application mentioned how will what kind of design choices will be optimal? I am specifically asking about neural network architecture in terms of types of layers, number of layers, depth, filters, etc.

3. In order to scale to more meaningful image datasets for the same set of applications, how would the authors decide to approach?

4. For more clarity of the work, comparisons in terms of experiments with Predictive Coding is highly encouraged. 


 
review_title: The goal of this work is to investigate the properties and representations of the early visual cortex. To tackle this problem a two layer hierarchical VAE was proposed whose generative part is Markovian. The paper reads well and the bio inspired approach seems relevant to the scope of the workshop.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: **Summary**: The authors perform an extensive study of multiple choices of hierarchical VAE architectures inspired by the recent literature, and find a proposed method (TDVAE) which matches many observed properties of biological visual systems. 

**Positives**: This work is a good comparison of multiple hVAE frameworks and specifically an in-depth investigation of which representations they learn. This type of study is somewhat lacking in the literature compared to other generative modeling frameworks which have also been compared with the early visual system (such as ICA). The detailed appendix is very much appreciated for this kind of empirical study as well. Finally, I find the illusory contour experiment to be the most interesting and novel, and commend the authors for devising it.

**Areas for improvement**: The section on noise correlations was a bit dense and challenging to understand for someone not familiar with the relevant neuroscience literature. 

**Questions**: Can the authors describe why they measured image accuracy in the image in-painting experiment rather than using a metric like likelihood? Perhaps I am misunderstanding something about the experiment setup. 

**Conclusions**: Overall I think this is a very relevant paper which attempts to bridge modern generative models used in machine learning with many observations from neuroscience. I strongly recommend acceptance. 
review_title: Review: Study of top-down visual effects with hierarchical VAEs

Decision: Accept(Oral)