Benefits of Overparameterization in Single-Layer Latent Variable Generative Models
Authors: Rares-Darius Buhai
Andrej Risteski
Yoni Halpern
David Sontag
Venue:  (2020)
Keywords: overparameterization
unsupervised
parameter recovery
rigorous experiments

Abstract: One of the most surprising and exciting discoveries in supervising learning was the benefit of overparameterization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it has been observed that overparameterization can be helpful as early as Dasgupta & Schulman (2007). In this paper, we perform an exhaustive study of different aspects of overparameterization in unsupervised learning via synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (recovering the parameters of the ground-truth model, held-out log-likelihood), sensitivity to variations of the training algorithm, and behavior as the amount of overparameterization increases. We find that, when learning using  methods such as variational inference,  larger models can significantly increase the number of ground truth latent variables recovered.

Under review as a conference paper at ICLR 2020
BENEFITS OF OVERPARAMETERIZZAION IN SINGLE- LAYER LATENT VARIABLE GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
One of the most surprising and exciting discoveries in supervising learning the benefit of overparameterivation (i.e. training very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been under-explored, despite the fact that it has been observed that overparameterization can be helpful early as Dasgupta & Schulman (2007). In this paper, we perform an exhaustive study of different aspects of overparameterization synthetic and semi-synthetic experiments. We discuss benefits to different metrics of success (recovering the parameters of the ground-truth model, held-out log- likelihood), sensitivity to variations of the training algorithm, and behavior the amount of overparameterization increases. We find that, when learning using methods such variational inference, larger models can significantly increase the number of ground truth latent variables recovered.
unsupervised learning via
INTRODUCTION
Unsupervised learning is an area of intense focus in recent years. In the absence of labels, the goal of unsupervised learning can vary. Generative adversarial networks, for example, have shown promise for density estimation and synthetic data generation. In commercial applications, unsupervised learning is often used extract features of the data that are useful in downstream tasks. In the sciences, the goal is frequently to confirm or reject whether particular model fits well or to identify salient aspects of the data that give insight into underlying causes and factors of variation.
Many unsupervised learning problems of interest, cast as finding maximum likelihood model, are computationally intractable in the worst case. Though much theoretical work has been done on provable algorithms using the method of moments and tensor decomposition techniques (Anandkumar
et al. 2014, Arora et al., 2017: Halpern & Sontag 2013), iterative techniques such as variational inference still widely preferred. In particular, variational approximations using recognition networks have become increasingly popular, especially in the context of variational autoencoders (Mnih & Gregor 2014). Intriguingly, it has been observed, e.g. by Yeung al. (2017). that in practice many of the latent variables have low activations and hence not used by the model.
Related phenomena have long been known in supervised learning: it was folklore knowledge among practitioners for some years that training larger neural networks can aid optimization, yet not affect generalization substantially. The seminal paper by Zhhng et al. (2016) thoroughly studied this phenomenon with synthetic and real-life experiments in the supervised setting. In brief, they showed that some neural network architectures that demonstrate strong performance on benchmark datasets are so massively overparameterized that they can "memorize" large image data sets (they can perfectly fit completely random data set of the same size). Subsequent theoretical work provided mathematical explanations of some of these phenomena (Allen-Zhu et al. 2018 Allen-Zhu & Li 2019). In contrast, overparameterrizatoon in the unsupervised case has received much less attention.
This paper aims to be controlled empirical study that measures and disentangles the benefits of overparameterization in unsupervised learning settings. More precisely, we consider the task of fitting two common latent-variable models discrete factor analysis model using noisy-OR parameterization and sparse coding (interpreted as a probabilistic model, see Section 2 Through experiments synthetic and semi-synthetic data sets, we study the following aspects:

---

Under review as a conference paper at ICLR 2020
Latent variable recovery: We show that larger models increase the number of ground truth latent variables recovered, as well as the number of runs in which the all ground truth latent variables recovered. Furthermore, we show that recovering the ground-truth -rrnth latent variables from the overparameterized solutions can be done via a simple filtering step: the optimization tends to converge a solution in which all latent variables that do not match ground truth latent variables can either be discarded (i.e. have low prior probability) or are near-duplicates of other matched latent variables.
Effects of extreme overparameterization We show that while the benefits of adding new latent variables have diminishing returns, the harmful effects of extreme overparameterization are at most minor. Both the number of ground truth recoveries and the held-out log- oogliike do not worsen significantly as the number of latent variables increases. Instead, sometimes the performance continues to increase even with 10 times the true number of latent variables.
Effects of training algorithm: We show that changes to the training algorithm, such as signifi- cantly increasing the batch size or using different variational posterior, do not affect the beneficial effects of overparameterization significantly. For learning noisy-OR networks, we test two algo- rithms based variational learning: one with logistic regression recognition network, and one with a mean-field posterior (e.g. see Wainwright et al. (2008).
One possible explanation for why overpa- rameterization helps is that having more latent variables increases the chances that at least one initialization will be close to each the ground-truth latent variables. (This indeed the idea of Dasgupta & mchulman (2007)). This does not appear to be the dominant factor here. We track the "matching" of the trained latent variables to the ground truth latent variables (matching = minimum cost bipartite matching, with a cost based parameter closeness), and show that this matching changes until relatively late in the training process. This suggests that the benefit of overparameterization being observed is not simply due increased likelihood of initializations close to the ground truth values.
Latent variable stability over the course of training:
In this investigation, our experiments. With more complex, deep generative models, there is lot of latitude in choosing the architecture of the model when overparameterizing, making it more difficult disentangle the importance of the choice of architecture from overparameterization in general.
simple generative models that
exert maximal control over
2 LEARNING OVERPARAMETERIZER GENERATIVE MODELS
We focus on the task of fitting two commonly used latent-variable models: noisy-OR networks and sparse coding. Beyond the ubiquity of these models, the reason we chose them is that they provide the "simplest" architectures for a generative model: single latent layer, albeit with different activation functions. This makes the choice of "overparametterize"" architecture non-controversiall: we simply include more latent variables.
Noisy-OR Networks: A noisy-OR network Peaal 1988) is a bipartite directed graphical model, in
which one layer contains binary latent variables and the other layer contains binary observed variables. Edges are directed from latent variables to observed variables. The model has as parameters a set of prior probabilities π E [0, 11 for the latent variables, a set of noise probabilities E [0, 1]" for the observed variables, and set of weights W E Raxm for the graph. If the latent variables are denoted
as h E (0, 1)mM and the observed variables as {0, 1 1)",, the joint probability distribution specified by the model factorizes as: p(x, h)) (hh) II p(x, [ht, where
m
m
p(h) = II R (1 T.) 1=1
- 1-h,
p(z, = 0/6) = (1 4) II expp(-Wshh)..
is common to refer to exp(- W;;) as the failure probability between h2 and xy (i.e. the probability with which, if h4 1, it "fails to activate" æ,)).
Training algorithm: We optimize an approximation of the likelihood of the data under the model, the evidence lower bound (ELBO). This is necessary because direct maximum likelihood optimization is intractable. If the joint pdf is p(:, h;; 0), we have, using the notation in Mnih Gregor 20144):
log p(z; 0) > E ((())) [log p(x, h;; 0) log q(h|z;; +)] L(x, 0,
2

---

Under review as a conference paper at ICLR 2020
where q(-|2;; () is variational posterior, also known recognition network in this setting. When q( lx; ④) = p(h|x; 0), wh, the inequality becomes equality; however, it is intractable to compute p(h|z; 0). Instead, we will assume that belongs to simpler family of distributions: a logistic regression network parameterized by weights wi E R and bias terms by E RR Vi E (1,..., m). s.t.:
<h|r: Φ) - II [1 w, b.) o(x w, + b,))
Then, we maximize the lower bound by taking gradient steps w.r.t. and c. Furthermore, to improve the estimation of the gradients, we use variance normalization and input-dependent signal centering. as in Mnih & Gregor (2014). For the input-dependent signal centering, we use a two-layer neural network with 100 hidden nodes in the second layer and tanh activation functions.
Extracting the ground-truth truth latent variables: As we are training an overparameterized model, we need to filter the learned latent variables to extract latent variables corresponding to the ground-truth variables. First, we discard all latent variables that are discardable, namely have learned prior probability less than 0.02 or have all learned failure probabilities of the related observable variables greater than 0.8. Then, for every pair of latent variables that are duplicates (measured as having failure probability vectors closer than 4.0 in I distance), we discard the one with lower prior probability - such that for any cluster of duplicate latent variables, only the one with the largest prior probability survives.
Sparse Coding: A sparse coding model is specified by matrix A E m with ||A:112 = 1, Vi (i.e. unit columns). Samples are generated from this model according to x = Ah, with h E RT" ||А]] and ||/|lo k (i.e. sparsity k). The coordinates of the vector h play the role of the latent variables, and the distribution h is generated from is as follows: first, uniformly randomly choose k coordinates of h to be non-zero; next, sample the values for the non-zero coordinates uniformly in [0, 1]: finally, renormalize the non-zero coordinates so they sum to 1.
Training algorithm: We use a simple alternating-minin minimization algorithm given in Li et al. (2016). It starts with a random initialization of A, such that A has unit columns. Then, at each iteration, "decodes" " the latent variables for batch of samples, s.t. a sample x is decoded as h = max(0, Aix ar). for some fixed and the current version of A. After decoding, it takes gradient step toward minimizing the "reconstruction error" || - x: -|,; and then re- re-normalizes the columns of A such that it has unit columns. Here, overparameterizattn means learning a matrix A E RTxs with s >> m, where m is the number of columns of the ground truth matrix.
Extracting the latent variables: Similarly as in the noisy-OR case, we are training an overparame- terized model, so to extract latent variables which correspond to the ground-truth variables we need to filtering step. First, we apply the decoding step h max(0, Aix 0.005) all samples x in the training set, and mark as "present" all coordinates in the support of h. Second, we discard the columns that were never marked "present". The intuition is rather simple: the first step is proxy for the prior in the noisy-OR case (it captures how often a latent variable is "used"). The second step removes the unused latent variables. (Note, one can imagine a softer removal, where one removes the variables used less than some threshold, but this simpler step ends up being sufficient for us.)
3 EMPIRICAL STUDY FOR NOISY-OR NETWORKS
We study the effect of overparameterization in noisy-OR networks using 7 synthetic data sets:
(1) The first, IMG, is based on Singliar& & Hauskrecht (006)) There are latent variables and
64 observed variables. The observed variables represent the pixels of an 8 x 8 image. Thus, the connections of latent variable to observed variables can be represented as an
image (see Figure D.. Latent variables have priors n. 0.25. All failure probabilities different from 1.0 are 0.1.
(2) The second, PLNT, is semi-synthetic: we learn a noisy-OR model from a real-world data set, then sample from the learned model. We learn the model from the UCI plants data set (Lichman jet al., 2013). where each data point represents a plant that grows in North America and the 70 binary features indicate in which states and territories of North America it is found. The data set contains 34,781 data points. The resulting noisy-OR model has latent variables, prior probabilities between 0.05 and 0.20, and failure probabilities either less than 0.5 equal to 1.
3

---

Under review as a conference paper at ICLR 2020
latent variables
observed variables
(flrst eight)
(a)
(a) Configuration of the IMG noisy-OR model. In the first row, each 8 image represents latent variable. Each pixel an 8 image represents the failure probability of the latent variable with the corresponding observed variable (white pixels correspond to failure probabilities different from 1.0). In the second row, each node represents an observed variable; the observed variables corresponding to the first row of the x images are shown. The edges show failure probabilities different from 1.0. (b) Samples of the IMG data set. Each x image represents sample, and each pixel represents an observed variable (white pixels correspond to 1).
Figure 1:
The next three data sets are based on randomly generated models with 8 latent and 64 observed variables (same as IMG):
(3) UNIF: Each latent variable's prior is sampled πū ~ [10 0.3] and it connected to each observation with probability 0.25. If connected, the corresponding failure probability is drawn from 24 [0.05, 0.2]; otherwise it is 1.0.
(4) CON8: म; 1/8 for all i. Each latent variable is connected to exactly 8 observed variables, selected at random. If connected, the failure probability is 0.1; otherwise it is 1.0.
(5) CON24: Same as CONS, but each latent variable is connected to 24 observed variables.
The rationale for the previous two distributions is different densities for the connection patterns.
The final two are intended to assess whether overparameterization continues to be beneficial in the presence of model misspecification, i.c. when the generated data does not truly come from a noisy-OR model, or when there are additional (distractor) latent variables that occur with low probability.
(6) IMG-FLIP: First, generate a sample from the IMG model described above. Then, with probability 10%, flip the value of every fourth observed variable in the sample (i.e. To, F4.
(7) IMG-UNIF: This model has 16 latent variables and 64 observed variables. The first 8 latent variables are those of the IMG model, again with prior probability 0.25. We then introduce 8 more latent variables from the UNIF model, with prior probabilities 0.05 each.
For all models except PLNT, noise probabilities are set to 0.001. PLNT uses the learned noise probabilities. To create each data set, we generate 10, 000 samples from the corresponding model. We split these samples into a training set of 9, 000 samples and validation set of 1, 000 samples. Samples are generated exactly once from the ground truth model and re-used all experiments with that model. For the randomly generated models, generate the ground truth model exactly once.
To count how many ground truth latent variables are recovered, we perform minimum cost bipartite matching between the ground truth latent variables and the recovered latent variables. The cost of matching two latent variables is the las distance between their weight vectors (removing first the variables with prior probability lower than 0.02). After finding the optimal matching, we consider as recovered all ground truth latent variables for which the matching cost is less than 1.0. Note the algorithm may recover the ground truth latent variables without converging solution because we do not require the prior probabilities of the latent variables to match (some of the latent variables may be split into duplicates) and because the matching algorithm ignores the state of the unmatched latent variables. We discuss this in more detail in Section 3
maximum likelihood
Overparameterization Improves Ground Truth Recovery and Log-likelihood: For all data sets, we test the recognition network algorithm using 8 latent variables (i.e. no overparameterizaaionn) 16, 32, 64, and 128. For each experiment configuration, we run the algorithm 500 times with different random initializations of the generative model parameters. We report in Figure the average number of ground truth latent variables recovered and the percentage of runs with full ground truth recovery (i.e. where 8 ground truth latent variables recovered). We that in all data sets,
4

---

Under review as a conference paper at ICLR 2020
IMG
PLNT
UNIF
CON8
CON24
FULL REC REC VARS
4.
8
16
32
64
128
8
16
32
64
128
16
32
64
128
16
32
64
128
16
64
128
100
1010
1001
1100
100
00
00
40
40
40
40
40
20
70
20
32
64 128
32
128
16
64
128
32
128
Figure 2: Performance of the noisy-OR network learning algorithm. The plots show statisties for 500 runs of the algorithm with random initializations on different data sets with different number of latent variables. The y-axis on the lop row denotes the average number of ground truth latent variables recovered and in the bottom the percentage of runs with full ground truth recovery. The 95% confidence intervals are shown in red bars.
overparameterization leads to significantly improved metrics compared to using 8 latent variables. The same trends observed for the held-out hog-liielihood. See the Appendix (E, Table 1) for more detailed numerical results for these experiments.
Minor; Benefits Are Often Significant: The results suggest that there may exist an optimal level of overparameterization for each data set, after which overparametcrization stops conferring benefits (the harmful effect then may appear because larger models are more difficullt to train). The peak happens at 32 latent variables for IMG, at 128 for PLNT, at 64 for UNIF, at 128 for CON8, and at 64 for CON24. Therefore, overparameterization continue confer benefits up very large levels of overparameetrrization. In addition, when 128 latent variables are harmful with respect to lower levels overparamettrizzaion., 128 latent variables lead to significantly improved metries compared to no overparameterization.
Harm of Extreme Overparameterizztion
When the full ground truth is recov- ered in an overparameterized setting, the unmatched latent variables usually fall into two categories: discardable duplicates, as described in Section 2 To test this observation systematically, we use the filtering step described that section.
Unmatched Latent Variables Are Discarded or Duplicates:
We applied the filtering step to all experiments reported in Figure 2 in the runs where the algorithm recovered the full ground truth. In nearly all of these runs, the filtering step keeps exactly latent variables that match the ground truth latent variables. Exceptions are 255 runs out of total of 6929 runs (i.e. 3. 3.68%); in these exception cases, the filtering step tends to keep more latent variables, out of which 8 match the ground truth, and the others have higher failure probabilities (but nonetheless lower than the threshold of 0.80). Note that the solutions containing duplicates are not general equivalent to the ground truth solutions terms of likelihood: we give illustrative example in Appendix (H).
Batch Size Does Not Change the Effect: We also test the algorithm using batch size 1000 instead of 20. Although the performance decreases may be expected given folklore wisdom that stochasticity is helpful in avoiding local minima - overparameterization remains beneficial across all metrics. This shows that the effect of overparamettrizztion is not tied to the stochasticity conferred by a small batch size. For example, on the IMG data set, we recover on average 5.91, 7.17, and 7.32 ground truth latent variables for learning with 8, 16, and 32 latent variables, respectively. See the Appendix (E, Table 2) for detailed results.
To test the effect of the choice of varia- tional distribution, on all data sets, we additionally test the algorithm using a per-sample mean-field variational posterior instead of the logistic regression recognition network. The variational posterior models the latent variables independent Bernoulli. In each epoch, for each sample, the variational posterior is updated from scratch until convergence using coordinate ascent, and then a gradient update is taken w.r.t. the parameters of the generative model. Thus, this approximation is more flexible than the recognition network: the Bernoulli parameters can be learned separately for each sample instead of being outputs of global logistic regression network.
Variational Distribution Does Not Change the Effect:
5

---

Under review as a conference paper at ICLR 2020
25
nll
20
correct matches
15
10
(b)
8
11
4 epochs
3
5
<
(a)
Figure 3: State the optimization process on successful run the noisy-OR network learning algorithm on the IMG data set with 16 latent variables. (a) The blue line (with "x") shows the number of latent variables matched to the same ground truth latent variable as at the end of the optimization. The red line (with "o") is the negative held-out log- likelihood. The graph is truncated 5 epochs. (b) The shapes of the latent variables after 1/9 epochs. 2/9 epochs, and 3/9 epochs. (c) The shape the latent variables after epochs. epochs, and 30 epochs.
Though the specific performance achieved on each data set often differs significantly from the previous results, overparameteriizatoo still leads to clearly improved metrics on all data sets. For example, on the IMG data set, we recover on average 6.72, 7.52, and 7.23 ground truth latent variables for learning with 8, 16, and latent variables, respectively. See the Appendix (E, Table 3).
Matching to Ground-truth Latent Variables Is Unstable: To understand the optimization dy- namics better, we inspect how early the recovered latent variables start to converge toward the ground truth latent variables they match in the end. If this convergence started very early, it could indicate that each latent variable converges to the closest ground truth latent variable then, overparameterization would simply make it more likely that each ground truth latent variable has latent variable close to at initialization.
The story is more complex. First, early (especially within the first epoch), there is significant conflict between the latent variables, and it is difficult to predict which ground truth latent variable they will converge to. We illustrate this in Figure 3 run that recovers the full ground truth on the IMG data set when learning with 16 latent variables. In part (a) of the Figure, at regular intervals in the optimization process, we matched the latent variable to the ground truth latent variables and counted how many pairs are the same as at the end of the optimization process. Especially within the first epoch, the number of such pairs small, suggesting the latent variables are not "locked" to their final state. Part (b) of the Figure pictorially depicts different stages in the first epoch of the run with 16 latent variables, clearly showing that in the beginning there are many latent variables that are in "conflict" for the same ground truth latent variables.
Second, even in the later stages the algorithm, is often the case that the contribution of one ground truth latent variable is split between multiple recovered latent variables, that the contribution of multiple ground truth latent variables is merged into one recovered latent variable. This is illustrated in part (c) (the same successful depicted in Figure (b)), which shows multiple later stages of the optimization process which contain "conflict" between latent variables. See the Appendix (F) for the evolution of the optimization process across more intervals.
Of course, the observations above do not rule out the possibility that closeness at initialization between the latent variables and the ground truth latent variables is important ingredient of the beneficial effect. However, we showed that the optimization process remains complex even in cases with overparameteriizatin..
Effects of Model Mismatch: In the experiments with model mismatch, we still use the noisy-OR network learning algorithm from Section We show that, in both data sets, overparameterization allows the algorithm recover the underlying IMG latent variables more accurately, while modeling the noise with extra latent variables. In general, we think that in misspecified settings the algorithm tends to learn projection of the ground truth model onto the specified noisy-OR model family, and that overparameterizztion often allows more of the noise to be "explained away" through latent variables.

---

Under review as a conference paper at ICLR 2020
For both data sets, the first bump in the recovery metries happens when we learn with 9 latent variables, which allows latent variables to be used for the IMG data set, and the extra latent variable to capture some of the noise. After that, more overparametterization increases the accuracy even further. For example, on the IMG-FLIP data set, we recover on average 4.40, 5.59, and 5.99 ground truth latent variables for learning with 8, 9, and 10 latent variables, respectively. See the Appendix (E, Table 4) for detailed results. Also, see the Appendix (G) for examples of the latent variables recovered in successful runs.
For IMG-FLIP, in successful runs, the algorithm tends to learn a model with an extra latent variable with significant non-zen zon-zero prior probability that approximates the shape of the noise (i.e. latent variable with connections to every fourth observed variable). For IMG-UNIF, the algorithm uses the extra latent variables to capture the tail latent variables. In this case, the algorithm often merges many of the tail latent variables, which is likely due to their smaller prior probabilities.
4 EMPIRICAL STUDY FOR SPARSE CODING
We find that the conclusions for sparse coding are qualitatively the same for the noisy-OR models. Thus, for reasons of space, we only describe them briefly; see Appendix (E, Table 5) for full details.
We again evaluate using synthetic data sets. A is sampled in pairs of columns such that the angle between each pair is a fixed y. Specifically, we first generate 12 random unit columns; then, for each of these columns, we generate another column that is random rotation at angle 7 of the original column. As a result, columns in different pairs have with high probability an absolute value inner product of approximately 3 (i.e., roughly orthogonal), whereas the columns in the same pair have an inner product determined by the angle y. The smaller the angle y, the more difficult it is to learn the ground truth, because it is more difficult to distinguish columns in the same pair. We construct two data sets, the first with y = 5 and the second with 7 = 10
We experimented with learning using matrix with 24 columns (the true number) and with 48 columns (overparameterizeed) To measure how many ground truth columns are recovered, we perform minimum cost bipartite matching between the recovered columns and the ground truth columns. As cost, we use the l2 distance between the columns and consider correct all matches with cost below 0.002. We measure the error between the recovered matrix and the ground truth matrix as the sum of costs in the bipartite matching result (including costs larger than the threshold).
As in the case of noisy-OR networks, overparameterization consistently improves the number of ground truth columns recovered, the number of runs with full ground truth recovery, and the error. For example, for y = 10", learning with 24 columns gives on average 19.77 recovered columns, 17.2% full recoveries, and 2.28 error, while learning with 48 columns gives 23.84 recovered columns, 88.4% full recoveries, and 0.04 average error.
5 RELATED WORK
On the empirical side, a few previous papers have considered overparameterized models for unsu- pervised learning and evaluated using synthetic distributions where assessing ground truth recovery is possible. Singliar & Hauskrecht (2006) and Dikmen & Févotte (011)) observe that, in overpa- rameterized settings, the unnecessary latent variables are discarded and that the log-likelihood does not decrease. However, they do not observe any beneficial effects. In the former case, this is likely because their variational approximation is too weak; in the latter case, is because the ground truth is already recovered without overparameterization overparameterization can lead to better held-out log-likelihood than lower levels of overparameter- ization for some learning algorithms. Separately, they show that some of their overparameterizee learning algorithms recover the ground truth: however, they do not investigate how this ability varies as function of the amount of overparameterization.
Hughes et al. (2015) show that larger levels of
On the theoretical side, to our knowledge, the earliest paper that points out a (simple) benefit of overparameteriization is Dasgupta & Schulman (2007) in the context of recovering the means of well-separated spherical Gaussians given samples from the mixture. They point out that using Ol In k) input points as "guesses" (i.e. initializations) for the means allows to guarantee, by the
7

---

Under review as a conference paper at ICLR 2020
coupon collector phenomenon, that include least one point from each component in the mixture which would not be so we only used k. A filtering step subsequently allows them to recover the k components. We could reasonably conjecture that the benefits in our setting due to similar reason: overparameterization could guarantee at least one trained latent variable close to each ground truth latent variable. However, simple high-dimensional concentration of measure easily implies that the probability of having reasonably highly-correlated initialization (c.g. in terms of inner product) an exponentially low probability event. The results Section also demonstrate a large amount of switching in terms of which trained variable is closest to each ground truth variable which further indicates the way optimization is improved by overparameterization is more complicated.
More recently. Li et al. (2018) explored matrix completion and Xu et al. (2018) mixtures of two Gaussians. Li al. (2018), the authors consider fitting a full-rank matrix to the partially observed matrix, yet prove gradient descent finds the correct, low-rank matrix. This setting is substantially simpler than ours: the authors can leverage linearity in the analysis (in a suitable sense, matrix completion can be viewed noisy version of matrix factorization, for which gradient descent has relatively simple behavior). Noisy-OR has substantive nonlinearities (in fact, this makes analyzing non-gradient descent algorithms involved (Arora et al. 2017)), and sparse coding is complicated by sparsity in the latent variables.
In Xu et al. (2018), the authors prove that when fitting symmetric, equal-weight wwo-component mixture, treating the weights as variables helps EM avoid local minima. (This flies in contrast to the intuition that knowing that the weights
equal, one should incorporate this information directly into the algorithm.) This setting is also much simpler than ours: their analysis does not even generalize to non-symmetric mixtures and relies on the fact that the updates have a simple closed-form.
6 DISCUSSION
The goal of this work was to exhibit the first controlled and thorough study of the benefits of overparameterization in unsupervised learning settings, more concretely noisy-OR networks and sparse coding. The results show that overparameterization is beneficial and impervious to variety of changes in the settings of the learning algorithm. We believe that our empirical study provides strong motivation for program of theoretical research understand the limits of when and how gradient- based optimization of the likelihood (or its approximations) can succeed in parameter recovery for unsupervised learning of latent variable models.
We note that overparameterization is common strategy in practice - though it's usually treated as a recipe develop more fine-grained latent variables (e.g., more specific topics in a topic model). In our paper, in contrast, we precisely document the extent to which it can aid optimization and enable recovering the ground-truth model.
We also note that our filtering step, while appealingly simple, will likely be insufficient in more complicated scenarios - e.g. when the activation priors have a more heavy-tailed distribution. such settings, more complicated variable selection procedures would have to be devised, tailored to the distribution of the priors.
As demonstrated in Section 3 the choice of variational distribution has impact on the performance which cannot be offset by more overparameterization In fact, when using the weaker variational approximation in Singliar Hauskrecht (2006) (which introduced some the datasets we use), we were not able to recover all sources, regardless of the level of overparaneterization. This delicate interplay between the power of the variational family and the level of overparameterization demands more study.
Inextricably linked to our study is precise understanding of the effects of architecture especially so with the deluge of different varieties of (deep) generative models. We leave the task of designing controlled experiments for more complicated settings for future work.
Finally, the work of Zhang et al. (2016) overparaneterrizaton in supervised settings considered data-poor regime: where the number of parameters in the neural networks is comparable or larger than the training set. We did not explore such extreme levels of overparameterizztton in our work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Allen-Zhu, Z. and Li, Y. Can sgd learn recurrent neural networks with provable generalization? arXiv preprint 1Xiv:1902.01022 2019.
Allen-Zhu, Z., Li, Y., and Liang, Y. Learning and generalization in overparameterizze neural networks, going beyond two layers. arXiv preprint arXiv: /8/ 1.04918, 2018.
Anandkumar, A., Ge, R., Hsu, D., Kakade, S. M., and Telgarsky, M. Tensor decompositions for learning latent variable models. The Journal of Machine Learning Research, 15(1):2773-2332 2014.
Arora, S., R., Ma, T., and Risteski, A. Provable learning of noisy-or networks. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pp. 1057-1066. ACM, 2017.
Dasgupta, S. and Schulman, L. A probabilistic analysis of em for mixtures of separated, spherical gaussians. Journal of Machine Learning Research, 8(Fcb)::03-226, 2007.
Dikmen, O and Févotte, C. Nonnegative dictionary learning in the exponential noise model for adaptive music signal representation. In Advances in Neural Information Processing Systems, pp. 2267-2275, 2011.
Halpern, Y. and Sontag, D. Unsupervised learning of noisy-or bayesian networks. arXiv preprint arXiv:1309.6834 2013.
Hughes, M., Kim, D. I., and Sudderth, Reliable and scalable variational inference for the hierarchical dirichlet process. In Artificial Intelligence and Statistics, pp. 370-378, 2015.
Jernite, Y., Halpern, Y., and Sontag, D. Discovering hidden variables in noisy-or networks using quartet tests. Advances in Neural Information Processing Systems, pp. 2355, 2013.
Li, Y., Liang, Y., and Risteski, A. Recovery guarantee of non-negative matrix factorization via alternating updates. In Advances in neural information processing systems, pp. 4987-4995, 2016.
Li, Y., Ma, T., and Zhang, H. Algorithmic regularization in over-parametterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory, pp. 2-47, 2018.
Lichman, M. et al. Uci machine learning repository, 2013.
Mnih, A. and Gregor, K. Neural variational inference and learning in belief networks. In International Conference on Machine Learning, pp. 1791-1799, 2014.
Pearl, Probabilistic reasoning in intelligent systems: Networks of plausible inference. 1988.
Singliar, T. and Hauskrecht, M. Noisy-or component analysis and its application to link analysis. Journal of Machine Learning Research, 7(0ct):2189-2211, 2006.
Wainwright, M. J., Jordan, M. I., et al. Graphical models, exponential families, and variational inference. Foundations and Trends( in Machine Learning, 1(1-2):1-305, 2008
Xu, J., Hsu, D. J., and Maleki, Benefits of over over- preeraaratteiizitionn with em. In Advances Neural Information Processing Systems, pp. 10685-10695, 2018.
Yeung. S., Kannan, A., Dauphin, Y., and Fei-Fei, L. Tackling over-pruning in variational autoencoders. arXiv preprint arv: 036443336003, 2017.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.08 0.05500, 2016.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A LEARNING ALGORITHMS
We give here pseudocode for the learning algorithms, well details that were not provided in the main document.
A.1 NOISY-OR NETWORKS WITH RECOGNITION NETWORK
Recall the evidence lower bound (ELBO):
log p(x; 8) 2 (1s0)) [logp(z, h; 0) log q( Φ) = 6(x,8,0)
One can derive the following gradients (see (Mnih & Gregor 2014
VoL(x, 0, ()()) Vo logp(æ, 0)]
VoL(x, 0) = +=;+) (log h; 0) - log (hh/= o)) x V log ((h|2z; 6)]
Then, we maximize the lower bound by taking gradient steps w.r.t. 0 and Φ. To estimate the gradients, we average the quantities inside the expectations over multiple samples from ((|12; Φ).
See Algorithm D for an update step of the algorithm, without variance normalization and input- dependent signal centering.
The experiments use mini-batch size 20 (unless specified otherwise). and ø are optimized using Adam.
Algorithm 1 Update step for learning noisy-OR networks with recognition network. p(:: 03) is the current noisy-OR network model (with parameters 0), q(11: Φ) is the current recognition network model (with parameters d), and is batch of 20 samples.
function Update(p, 9. 2)) for + to 20 do
h0 (ample(g(1lz();;))
y(() log (((()))) (0) 0) log (z/0)))))))))))))))))))))))))()
end for
Ad < AΦ + 11
20 (())
V log q(h (0100)0000)))
A + A + 17 DU
Ve log (+) xx(0)
end function
A.1.1 VARIANCE NORMALIZATION AND INPUT-DEPENDENT SIGNAL CENTERING
We variance normalization and input-dependent signal centering improve the estimation of VoL(a, 0, Φ). as in (Mnih & Gregor 2014).
The goal of both techniques is reduce the variance in the estimation of VoL(x, 0,)). They based on the observation that (see (Mnih & Gregor 2014)):
V VoL(o, 0, Φ) = E (=n0)) [log h; 0) log q(h 2 Φ)) x V (ogg())z;;)) Bg (logr(1; h;; 0) log g(h|>; )) - c) V log q(1) +)
where e does depend on h. Therefore, is possible reduce the variance in the estimator by using some close to log p(x, h; 0) log ((h z;; Φ).
Variance normalization keeps running averages of the mean and variance of log log h;; 0) log q(b]x;;)))) Φ). Let e be the average mean and v be the average variance. Then variance normal- ization transforms log pax, h; 0) log q(h|a;; () into log lug =
max(1.vo)
Input-dependent signal centering keeps an input-dependent function c(x) that approximates the normalized value of log p(z, h; 0) log (h]a:; Φ). We model e(x) two-layer neural network
10

---

Under review as a conference paper at ICLR 2020
with 100 hidden nodes in the second layer and tanh activation functions. We train c(z) to minimize
log,(z,, 0) - log q(h]z;)) c(z))
E [x;0))
max vT)
and optimize it using SGD. Therefore, our estimator of voL(z, B, Φ) is obtained as: (og h; 0) log q( x: o) d(o)) x V log Vologa(hlz; max (1, vV)
(
Val(x,, 0,6), se (2(()
A.2 NOISY-OR NETWORKS WITH MEAN-FIELD VARIATIONAL POSTERIOR
In the mean-field algorithm, variables are modeled as independent Bernouli..
the variational posterior (hh) 11 9:(h)). That is, the latent
For each data point, we optimize q from scratch (unlike the of the recognition network variational posterior, which is "global"), and then we make gradient update to the generative model.
To optimize the variational posterior q we use coordinate ascent, according to:
q: (ha) (xp( ((s., h (x))
where the expectation is over Mmm])(()-
See Z for an update step of the algorithm. We 5 iterations of coordinate ascent, and use 20 samples to estimate expectations.
Algorithm Update step for learning noisy-OR networks with mean-field variational posterior. :())3)) the current noisy-OR network model (with parameters 0), and x sample.
function Update(p, x) +
for iter f- to do
for k t- to do
E = [0, 0]
for & <- 1 to 20 do
h4g = 0
h(]](()) + sample(g(-))
O <- E((] + 20 log p(, (h, x; 0)
[[]] < [[]] + 20 logp(h + 1x, x; 0)
end for
g x cxp(ETg EE--}
end for end for for
1 20 do h + sample(gg-))
A + A0 +
V 2og
end for end function
A.3 SPARSE CODING
See Algorithm for an update step of the algorithm. We use batch size 20 in the learning algorithm in all experiments.
11

---

Under review as a conference paper at ICLR 2020
Algorithm 3 Alternating minimization algorithm update for sparse coding. A is the current matrix, and æ a batch of 20 samples.
1: function Update(A, x)
2: for i <- 1 to 20 do
3: (1) <- max(O, Aix(1) a)
4: end for
5: < A - 1 20 E [(A 1.(0)
6: normalize columns of A
7:
end function
B IMG DATA SET PROPERTIES
In addition to being easy visualize, the noisy-OR network model of the IMG data set has properties that ensure it not "too easy" to learn. Specifically, 5 out of the 8 latent variable do not have "anchor" observed variables (i.e. observed variables for which single failure probability is different from 1.0). Such anchor observed variables are an ingredient of most known provable algorithms for learning noisy-OR networks. More technically, the model requires subtraction step in the quartet learning approach of (Jernite et al.) 2013).
C PLNT DATA SET CONSTRUCTION
We learn the PLNT model from the UCI plants data set (Lichman et al. 2013). where each data point represents plant that grows in North America and the 70 binary features indicate in which states and territories of North America it is found. The data set contains 34,781 data points.
To learn the data set, we use the learning algorithm described in Section A.I with 20 latent variables. We remove all learned latent variables with prior probability less than 0.01. Furthermore, we transform all failure probabilities greater than 0.50 into 1.00. This transformation is necessary to obtain sparse connections between the latent variables and the observed variables; without it, every latent variable connected to almost every observed variable, which makes learning difficult. The resulting noisy-OR network model has 8 latent variables. Each latent variable has a prior probability between 0.05 and 0.20. By construction, each failure probability different from 1.00 between 0.00 and 0.50.
Figure 4 shows a representation of the latent variables learned the PLNT data set. As observed, the latent variables correspond to neighboring regions in North America.
12

---

Under review as a conference paper at ICLR 2020
Figure 4: Latent variable configuration of the PLNT noisy-OR network model. Each map represents latent variable. The regions in blue represent the observed variables for which the failure probability not 1.00. The fifth latent variable, which seems to contain only Florida, also contains Puerto Rico and Virgin Islands (not shown map).
D DETAILED EXPERIMENT CONFIGURATION
Below give more details the configuration of the experiments.
D.I INITIALIZATION
In all noisy-OR network experiments, the noisy-OR network model is initialized by sampling each prior probability n, each failure probability fzi, and each complement noise probability L as follows: sample
uniform[2.00 4.0]. and then set the parameter to sigmoid(z). Note that sigmoid(2)) (2) is roughly between 0.88 and 0.98, so the noisy-OR network is biased toward having large prior probabilities, large failure probabilities, and small noise probabilities. We found that this biased initialization improves results over one centered around 0.5.
In the recognition network noisy-OR experiments, initialize the recognition network to have all weight parameters and bias parameters uniform [-0.1, 0.1].
In the mean mean-field network noisy-OR experiments, initialize the mean- mean-field Bernoulli random variables to have parameters uniform in 0.8).
In the sparse coding experiments, initialize the matrix by sampling each entry from standard Gaussian, and then normalizing the columns such as to have unit l2 norm.
D.2 HYPERPARAMETERS
We generally tune hyperparameters within factors of 10 for each experiment configuration. Due to time constraints, for each choice of hyperparameters we only test it on 10 runs with random initializations of the algorithm, and then choosing the best performing hyperparameters for the large-scale experiments.
In the recognition network noisy-OR experiments, we tune the step size for noisy-OR network model parameters and the step size for the input-dependent signal centering neural network. The step size for the recognition network model parameters the same as the one for the noisy-OR network model parameters (tuning it independently did not seem to change the results significantly). The variance reduction technique requires a rate for the running estimates; we set this to 0.8.
13

---

Under review as a conference paper at ICLR 2020
In the mean-field noisy-OR experiments, only tune the step size for the noisy-OR network model parameters. For the coordinate ascent method used to optimize the mean-field parameters, we use 5 iterations of coordinate ascent (i.e. 5 iterations through all coordinates), and in each iteration estimate expectations with 20 samples.
In the sparse coding experiments, tune the step size for the updates to the matrix and the variable.
D.3 NUMBER OF EPOCHS
In all experiments, we use large enough fixed number of epochs such that the log-likelihood error measurement does not improve at the end of the optimization in all nearly-all the runs. However, to avoid overfitting, we save the model parameters at regular intervals the optimization process, and report the results from the timestep that achieved the best held-out log- leellikelihood/ error measurement (i.e. we perform post-hoc "early stopping").
E TABLES OF RESULTS
Below we present detailed tables of results for the experiments.
Table shows the noisy-OR network results with recognition network (partially included in the main document).
Table 2
shows the noisy-OR network results with recognition network and larger batch size 1000.
Table shows the noisy-OR network results with mean-field variational posterior.
Table 4 shows the noisy-OR network results with recognition network on the misspecified data sets (IMG-FLIP and IMG-UNIF).
Table shows the sparse coding results.
14

---

Under review as a conference paper at ICLR 2020
Table 1: Performance of the noisy-OR network learning algorithm with recognition network. Each row reports statistics for 500 runs of the algorithm with random initializations. The 95% confidence intervals are included. The first column denotes the number of latent variables used in learning. the second column the average number of ground truth latent variables recovered, the third column percentage of runs with full ground truth recovery, and the fourth column the average held-out negative log-likelihood.
LAT VARS
RECOV VARS
FULL RECOV (%)
NEGATIVE LL (NATS)
IMG
8
6.31 0.11
0.11
29.6
4.0
13.91 + 0.12
16
7.62 = 0.06
73.6
12.65 = 0.05
32
7.75
79.6
1.6 3.5
12.57
64
7.73
77.0
3.7
12.56 0.03
128
7.75 = 0.04
75.6 1 3.8
12.57 = 0.02
PLNT
8
4.71 0.12
0.4
12.36 = 0.07
16
6.83
4.4
11.83 0.07
32
6.11
4.
12.23 0.08
64
4..0 7.10
12.03 = 0.07
128
7.58
3.7
11.96 = 0.05
UNIF
8
0.14
12.6
2. 2.9
12.09 = 0.14
16
7.78 0.05
85.4 3.1
10.10 = 0.03
32
7.87
88.2
10.08 0.01
64
=.90
2.2
10.08 = 0.01
128
7.78 0.08
2.5
10.11 = 0.01
CONS
8
3.70 0.15
1.0
8.42 0.10
16
0.15
3.7
7.15 0.08
32
7.05
4.0
6.31 0.04
64
0.06
81.6
6.38 0.03
128
1.88=0004
92.8 2.3
6.43 + 0.02
CON24
8
2.26 + 0.15
0.6
14.89 + 0.17
16
4.90 - 0.21
17.2
11.17 + 0.13
32
1.2100.1 0.10
4.4
1.01
64
7.60
= 0.06
68.4
11.75
128
7.34 0.08
4..24444
15.61 0.24
15

---

Under review as a conference paper at ICLR 2020
Table 2: Performance of the noisy-OR network learning algorithm with recognition network and batch size 1000. Each row reports statistics for 500 runs of the algorithm with random initializations. The 95% confidence intervals are included. The first column denotes the number of latent variables used in learning, the second column the average number of ground truth latent variables recovered, the third column the percentage of runs with full ground truth recovery, and the fourth column average held-out negative log- log-liieli
LAT
RECOV VARS
FULL RECOV (%)
NEGATIVE LL (NATS)
VARS
IMG
8
5.91 0.13
24.2 + 3.8
14.23 + 0.13
16
= =.08
53.6 4.4
13.01 = 0.10
32
58.6 4.3
12.82 = 0.08
PLNT
8
4.35 = 0.12
0.0
13.74 = 0.05
16
5.69 0.09
5.0
13.07 = 0.05
32
6.18 0.08
15.6
12.76 = 0.06
UNIF
8
0.17
3.9
11.64 = 0.13
16
12200..11
4.
10.35 = 0.05
32
5.84 0.21
49.2
10.30 = 0.04
CONS
8
=
0.0
8.89 + 0.10
16
4.31 0.16
2.1
7.98 + 0.10
32
0.10
66.2
6.25 + 0.06
CON24
8
2.18
1.2
14.67 = 0.20
16
5.67 0.24
51.8 + 4.4
11.63 + 0.24
32
0.19 6.86
4.84
10.49 0.19
16

---

Under review as a conference paper at ICLR 2020
Table 3: Performance of the noisy-OR network learning algorithm with mean-field variational posterior. Each row reports statistics for 500 runs of the algorithm with random initializations. The 95% confidence intervals are included. The first column denotes the number of latent variables used in learning, the second column the average number of ground truth latent variables recovered, the third column the percentage runs with full ground truth recovery, and the fourth column the average held-out negative log-likell li-likkeiioodd
LAT VARS
RECOV VARS
FULL RECOV (%)
NEGATIVE LL (NATS)
IMG
8
6.72 + 0.10
41.8 + 4.3
13.50 + 0.11
16
7.52 0.07
6.14
12.58 0.06
32
7.23 1.:30000 0.09
56.2
12.80 = 0.08
PLNT
8
4.35 = 0.12
0.0
13.74 = 0.05
16
5.69 0.09
5.0
13.07 = 0.05
32
6.18 0.08
15.6
12.76 = 0.06
UNIF
8
0
3.9
11.64 0.13
16
6,20 0.19
4.
10.35 = 0.05
32
5.84 #
49.2
10.30 = 0.04
CONS
8
=
0.0
8.89 + 0.10
16
4.31 0.16
2.0
7.98 + 0.10
32
=.10
66.2
6.25 + 0.06
CON24
8
2.12
1.2
14.67 0.20
16
5.67 + 0.24
51.8 1..4+0 4.4
11.63 + 0.24
32
6.86
0.19
74.4 3.8
10.49 0.19
Table 4: Performance of the noisy-OR network learning algorithm with recognition network on the misspecified data sets (IMG-FLIP and IMG-UNIF). Each row reports statistics for 500 runs of the algorithm with random initializations. The 95% confidence intervals are included. The first column denotes the number of latent variables used in learning, the second column the average number of ground truth latent variables recovered, the third column the percentage of runs with full ground truth recovery, and the fourth column the average held-out negative log- ligeliill
LAT
RECOV
FULL
NEGATIVE
VARS
VARS
RECOV
LL (NATS)
IMG-FLIP
0.2 0.10
17.55 0.10
9
5.59 0.13
1.22
16.49 + 0.11
10
5.99 - 0.13
18.2
15.98
0.10
16
6.88 + 0.09
2.9
15.10 + 0.06
IMG-UNIF
4.95 + 0.12
0.0
21.51 + 0.08
5.44 + 0.12
6.0
20.77 0.09
16
7
4.3
18.05 0.08
32
7.76 0.05
3.5
16.41 0.06
17

---

Under review as a conference paper at ICLR 2020
Table 5: Performance of the sparse coding learning algorithm. Each row reports statistics for 500 runs of the algorithm with random initializations. The 95% confidence intervals are included. The first column denotes the number of latent variables used learning, the second column the average number of ground truth columns recovered, the third column the percentage of runs with full ground truth recovery, and the fourth column the average error.
COLS
RECOV
FULL RECOV (%)
ERROR
COLS
7 = 5
24
8.71 0.18
0.0 + 0.0
9.32 0.18
48
14.54 + 0.37
1.4 + 1.0
3.11 = 0.16
7 10
24
19.77 + 0.29
17.2 + 3.3
2.28 = 0.15
48
23.84 + 0.05
88.4 + 2.8
0.04 = 0.02
F STATE OF THE OPTIMIZATION PROCESS
Figures 5 and 6 show more steps of the optimization process the successful run with 16 latent variables mentioned in Figure 3.
Figure 5: Latent variables on successful run of the noisy-OR OR network learning algorithm on the IMG data set with 16 latent variables. Shown the state of the latent variables after epochs 1/9, 2/9. 3/9, 4/9, 5/9, 6/9, 8 8/9, 8/9, and 1.
<
<
0
O
Figure 6: Latent variables on successful run of the noisy-OR network learning algorithm on the IMG data set with 16 latent variables. Shown is the state of the latent variables after epochs 1, 2, 3, 5, 10, 20, 30, 50, and 100.
G RECOVERED LATENT VARIABLES ON MISSPECIFIED DATA SETS
Figure shows successful recoveries for the IMG-FLIP and IMG-UNIF data sets. As observed, some of the extra latent variables are used to model some of the noise due to misspecification.
18

---

Under review as a conference paper at ICLR 2020
(a)
Figure 7: Latent variables recovered in successful runs (i.e. they recover the IMG latent variables) on the mismatched data sets. Below each 8 x images corresponding to latent variable, there is color corresponding its prior (whiter means prior closer 1.0). (a) Successful run with 16 latent variables on the IMG-FLIP data set. (b) Successful run with 16 latent variables on the IMG-UNIF data set.
H DUPLICATE LATENT VARIABLES ARE NOT EQUIVALENT TO A SINGLE LATENT VARIABLE
We give an example of a model with two latent variables that has identical failure probabilities and is not equivalent a model with one latent variable that has the same failure probabilities (and possibly different priors).
Consider a noisy-OR network model with two latent variables h1 h2 and observed variables T2 Let 1 T2 = 0.25 (prior probabilities), fn = f12 f21 fez = 0.1 (failure probabili- ties), and 1 F(22 = 0) = 0.600625, and P(z) = 0, 22 = 0) = 0.56625625.
= 0.0 (noise probabilities). Then the negative moments are P(21 0) 0.600625,
Consider now noisy-OR network model with one latent variable hi and observed variables f21 0.1 and I1 = 1/2 0.0. Then, to match the first-order negative moments
1/.22. Let f1 = P(11 0) and P(rz = 0) P(222 0), we need zj 0.44375 (prior probability). But then this gives PP(11 0, x. 0) 0.5606875, which does not match [P(x, 0, 22 0).
Therefore, there exists no noisy-OR model with one latent variable and identical failure and noise probabilities that is equivalent to the noisy-OR model with two latent variables.
19


Reviews:
[{"review_id": "r1g54dkzqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper performs empirical study on the influence of overparameterization to generalization performance of noisy-or networks and sparse coding, and points out overparameterization is indeed beneficial. I find the paper has some drawbacks.\n\n1. Overparameterization is better than underparamterization and exact parameterization is not surprising. The question is how much do we need to overparameterize. As the number of parameters goes to infinity, the model can eventually remember all the training data, and has poor generalization. The real interesting question to ask is how to use an excessive amount of parameters, yet still avoid overfitting.\n\n2. The discussed models are too simple. I am expecting some theoretical analysis for tasks simple as noisy-or and sparse coding, or some experiments for more complicated (deep) models need to be done, to make the paper more solid.\n\nUpdate\n=====\n\nThank the authors for the response. The authors do address my comment #1. I agree that overparameterization improves recovery is a new finding. However, I still think the \"information gain\" of this paper is somewhat thin. There could be at least some intuitions on why overparameterization helps noisy-or models. I think the analysis can be more in-depth to make this paper more interesting. \n\nI would like to raise my score a bit to a \"neutral\" score, but given the current scoring system I'll just keep my score. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJledmXCtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates benefit of over-parameterization for latent variable generative model while existing researches typically focus on supervised learning settings. It is experimentally shown that the over-parameterization helps to obtain better optimization, but too much over-parameterization gives performance deterioration. In the numerical experiments, the effect of over-parameterization is investigated from several aspects.\n\nThe motivation of this paper is interesting. The writing of the paper is clear, and I could follow the contents easily.\n\nOn the other hand, I have the following concerns on the significance of the paper.\n- All datasets investigated in this paper are rather small. If there were thorough investigations on more modern deep generative models, then the paper would be stronger. For example, the latent variable model is recently well discussed in the context of disentanglement representation. The generative models to obtain disentanglement representation could be investigated in the frame-work of this paper.\n- This is an empirical study, but if there was theory to support the empirical observations, then the paper was more convincing. The problem itself is just a sparse coding problem. Hence, I think what investigated in this paper can be discussed by relating sparse coding theories. However, there is no theoretical justification on the experimental results.\n- Summarizing the above arguments, the insight obtained in this paper is a bit weak. More ablation study and more experiments on general models will clarify what is going on in the over-parameterized model for latent generative models.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJeivC_0YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper “aims to be a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings. ” The author’s empirical study is comprehensive, and to my knowledge the most detailed published work on this to date. Specifically, the authors empirically study \n- the ability of networks to recover latent variables\n- the effects of extreme overparameterization\n- the effects the training method (e.g. batch size)\n- latent variable stability over the course of training\n\nIn line with the findings for supervised settings, the authors find that overparameterization is often beneficial, and that overfitting is a surprisingly small issue. This is an interesting and useful observation, particularly since it at first sight appears to be in disagreement with some earlier work (the authors suggest explanations for the differing observations). \n\nAs the authors point out (and I agree), the paper constitutes a compelling reason for theoretical research on the interplay between overparameterization and parameter recovery in latent variable neural networks trained with gradient descent methods. \n\nThe authors perform studies on a range of different real-world and synthetic datasets. \n\nThe paper is well-written, well-structured, and easy to follow. Relevant literature has been cited. The appendices contain a wealth of details that will make this work reproducible. \n\nDecision: weak accept. The paper contains some new insights, but its contributions are not quite as substantial (e.g. lack of precise mathematical statements) or surprising as those in stronger ICLR papers. \n\nA small gripe: the authors promise “ a controlled empirical study making precise the benefits of overparameterization in unsupervised learning settings”. I would argue that “making precise” is too strong for what the paper actually delivers. I suggest rewording this.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies over-parameterization for unsupervised learning. The paper does a series of empirical studies on this topic. Among other things the authors observe that larger models can increase the number latent variables recovered when fitting larger variational inference models. The reviewers raised some concern about the simplicity of the models studied and also lack of some theoretical justification. One reviewer also suggests that more experiments and ablation studies on more general models will further help clarify the role over-parameterized model for latent generative models. I agree with the reviewers that this paper is "compelling reason for theoretical research on the interplay between overparameterization and parameter recovery in latent variable neural networks trained with gradient descent methods". I disagree with the reviewers that theoretical study is required as I think a good empirical paper with clear conjectures is as important. I do agree with the reviewers however that for empirical paper I think the empirical studies would have to be a bit more thorough with more clear conjectures. In summary, I think the paper is nice and raises a lot of interesting questions but can be improved with more through studies and conjectures. I would have liked to have the paper accepted but based on the reviewer scores and other papers in my batch I can not recommend acceptance at this time. I strongly recommend the authors to revise and resubmit. I really think this is a nice paper and has a lot of potential and can have impact with appropriate revision.
Decision: Reject