Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Incorporating the local learning approaches into the training of the deep generative models can potentially create a new model that has both the capacity for high-dimensional inputs and flexibility for locally changing environments. However, the local learning approach is limited to using only a simple model, because complex models require a large amount of data and extended training time. This paper overcomes this trade-off based on the insight that the real-world dataset often shares some structural similarities between each neighborhood.

Pros:
The paper is well-written. It is easy for the reader to understand. The derivations in the paper are correct.


The motivation is plausible. It is reasonable to expect that each local subspace of the dataset shares some structure since most dataset tends to be governed by the consistent rules of the real world.


The numerical experiments show that the locality enables the model to achieve the disentangled representation for each subspace without any label information.

Cons:
The novelty seems a little straight-forward. The paper just extends the objective function of the VAE to have different parameters for each local subset.

In consideration of extended training time in the complex model, the paper doesn't provide an evaluation of the efficiency of their proposed model.

The paper isn't very polished yet. There were more than a few spelling and grammatical errors, please proofread the work and improve the writing.

This paper in its current form is already fairly good. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Authors of this paper propose to utilize the embedding model for the other local structure as a weak form of supervision based on the insight that the real-world datasets often shares some structural similarity between each neighborhood. The Local VAE is proposed to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning.

Local VAE incorporates local information by using prior distributions of local parameters in VAE. The overall model performs probabilistic inference via the conditional distribution from the meta parameters. There are several concerns:
1. In section 2, authors discussed LLE. It is unclear the purpose of the section 2.1? 
2. LLE does not require W is nonnegative only, and \sum_j W_{i,j}=0 is also contradictory with the nonnegative assumption. 
3. Authors claimed that Local VAE algorithm corresponds to the assumption that the dataset approximately lies on multiple subsets and each subset is generated from different parameters. It is unclear what is the connection of the Local VAE to multi-scale structures of the datasets.
4.  Authors evaluated neighbors by sampling and k-nearest neighbors on latent space. It is unclear why not use the common k-nearest neighbors on the input data. K-nearest search should not be a computational problem for large datasets by using fast approach.

As the motivations of this paper, existing methods require massive amount of data and extended training time. However, authors did not demonstrate these points by comparing the proposed method with existing methods.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposed a local VAE based on the model-agnostic meta-learning concept. Images generated based local VAE are shown to be better than those generated bu \beta-VAE in general in terms of generation quality and disentanglement/compactness/informativeness.

This paper shows an interesting idea of viewing neighbourhood of a data point as a task to adopt the meta-learning concept to  train VAE which gives superior performance.

However, this paper falls short regarding its overall organisation of the paper and the way that different concepts being articulated which make it hard to follow and read. It is not easy to understand the linkage between a task and a neighbourhood being advocated without re-reading the paper for a few times. It seems to me that very substantial effort is needed for the revision to reach the ICLR standard.

The introduction section should be carefully rewritten to explain clearly the notions of locality and structural relationship. As far as I can see in the subsequent sections, the locality/structural relationship seems to be referring to the neighbourhood either in the input space or the latent space. Also, the way of discussing local learning should be elaborated a bit to make it clear what exact notion of local learning  is being presented.


Specific comments:

Page 2: 
“since the most dataset tends to be governed by the consistent rules of the physical world”
->
“since most dataset tends to be governed by the consistent rules of the physical world”

“Such kind of dataset has a multi-scale structure from a local to a global scale.” - some grammatical issue

Page 5:
Line 4: Equation 3 seems not related to meta-learning

review_title: Official Blind Review #3

Meta-review: The paper presents a structured VAE, where the model parameters depend on a local structure (such as distance in feature or local space), and it uses the meta-learning framework to adjust the dependency of the model parameters to the local neighborhood.

The idea is natural, as pointed by Rev#1. It incurs an extra learning cost, as noted by Rev#1 and #2, asking for details about the extra-cost. The authors' reply is (last alinea in first reply to Rev#1): we did not comment (...) because in essence, using neighborhoods in a naive way is not affordable. 
The area chair would like to know the actual computational time of Local VAE compared to that of the baselines.  

More details (for instance visualization) about the results on Cars3D and NORB would also be needed to better appreciate the impact of the locality structure. The fact that the optimal value (wrt Disentanglement) is rather low ($10^{-2}$) would need be discussed, and assessed w.r.t. the standard deviation.  

In summary, the paper presents a good idea. More details about its impacts on the VAE quality, and its computation costs, are needed to fully appreciate its merits. 
Decision: Reject