Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper performs empirical study on the influence of overparameterization to generalization performance of noisy-or networks and sparse coding, and points out overparameterization is indeed beneficial. I find the paper has some drawbacks.

1. Overparameterization is better than underparamterization and exact parameterization is not surprising. The question is how much do we need to overparameterize. As the number of parameters goes to infinity, the model can eventually remember all the training data, and has poor generalization. The real interesting question to ask is how to use an excessive amount of parameters, yet still avoid overfitting.

2. The discussed models are too simple. I am expecting some theoretical analysis for tasks simple as noisy-or and sparse coding, or some experiments for more complicated (deep) models need to be done, to make the paper more solid.

Update
=====

Thank the authors for the response. The authors do address my comment #1. I agree that overparameterization improves recovery is a new finding. However, I still think the "information gain" of this paper is somewhat thin. There could be at least some intuitions on why overparameterization helps noisy-or models. I think the analysis can be more in-depth to make this paper more interesting. 

I would like to raise my score a bit to a "neutral" score, but given the current scoring system I'll just keep my score. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper “aims to be a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings. ” The author’s empirical study is comprehensive, and to my knowledge the most detailed published work on this to date. Specifically, the authors empirically study 
- the ability of networks to recover latent variables
- the effects of extreme overparameterization
- the effects the training method (e.g. batch size)
- latent variable stability over the course of training

In line with the findings for supervised settings, the authors find that overparameterization is often beneficial, and that overfitting is a surprisingly small issue. This is an interesting and useful observation, particularly since it at first sight appears to be in disagreement with some earlier work (the authors suggest explanations for the differing observations). 

As the authors point out (and I agree), the paper constitutes a compelling reason for theoretical research on the interplay between overparameterization and parameter recovery in latent variable neural networks trained with gradient descent methods. 

The authors perform studies on a range of different real-world and synthetic datasets. 

The paper is well-written, well-structured, and easy to follow. Relevant literature has been cited. The appendices contain a wealth of details that will make this work reproducible. 

Decision: weak accept. The paper contains some new insights, but its contributions are not quite as substantial (e.g. lack of precise mathematical statements) or surprising as those in stronger ICLR papers. 

A small gripe: the authors promise “ a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings”. I would argue that “making precise” is too strong for what the paper actually delivers. I suggest rewording this.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper investigates benefit of over-parameterization for latent variable generative model while existing researches typically focus on supervised learning settings. It is experimentally shown that the over-parameterization helps to obtain better optimization, but too much over-parameterization gives performance deterioration. In the numerical experiments, the effect of over-parameterization is investigated from several aspects.

The motivation of this paper is interesting. The writing of the paper is clear, and I could follow the contents easily.

On the other hand, I have the following concerns on the significance of the paper.
- All datasets investigated in this paper are rather small. If there were thorough investigations on more modern deep generative models, then the paper would be stronger. For example, the latent variable model is recently well discussed in the context of disentanglement representation. The generative models to obtain disentanglement representation could be investigated in the frame-work of this paper.
- This is an empirical study, but if there was theory to support the empirical observations, then the paper was more convincing. The problem itself is just a sparse coding problem. Hence, I think what investigated in this paper can be discussed by relating sparse coding theories. However, there is no theoretical justification on the experimental results.
- Summarizing the above arguments, the insight obtained in this paper is a bit weak. More ablation study and more experiments on general models will clarify what is going on in the over-parameterized model for latent generative models.
review_title: Official Blind Review #2

Meta-review: This paper studies over-parameterization for unsupervised learning. The paper does a series of empirical studies on this topic. Among other things the authors observe that larger models can increase the number latent variables recovered when fitting larger variational inference models. The reviewers raised some concern about the simplicity of the models studied and also lack of some theoretical justification. One reviewer also suggests that more experiments and ablation studies on more general models will further help clarify the role over-parameterized model for latent generative models. I agree with the reviewers that this paper is "compelling reason for theoretical research on the interplay between overparameterization and parameter recovery in latent variable neural networks trained with gradient descent methods". I disagree with the reviewers that theoretical study is required as I think a good empirical paper with clear conjectures is as important. I do agree with the reviewers however that for empirical paper I think the empirical studies would have to be a bit more thorough with more clear conjectures. In summary, I think the paper is nice and raises a lot of interesting questions but can be improved with more through studies and conjectures. I would have liked to have the paper accepted but based on the reviewer scores and other papers in my batch I can not recommend acceptance at this time. I strongly recommend the authors to revise and resubmit. I really think this is a nice paper and has a lot of potential and can have impact with appropriate revision.
Decision: Reject