Title: Dynamic Partition Models
Authors: Marc Goessling
Yali Amit
Venue: ICLR 2017

Abstract: We present a new approach for learning compact and intuitive distributed representations with binary encoding. Rather than summing up expert votes as in products of experts, we employ for each variable the opinion of the most reliable expert. Data points are hence explained through a partitioning of the variables into expert supports. The partitions are dynamically adapted based on which experts are active. During the learning phase we adopt a smoothed version of this model that uses separate mixtures for each data dimension. In our experiments we achieve accurate reconstructions of high-dimensional data points with at most a dozen experts.

TL;DR: Learning of compact binary representations through partitioning of the variables

Review discussion:
Could your approach be extended to have level of expertise be temporally dependent? If not in the most general setting of non-stationarity, then even in a special case in which different experts specialize in different time periods (for different variables), or even in a periodic fashion (e.g. "sleeping experts" as in "Putting Bayes to Sleep," Koolen et al., NIPS 2012)?
Thanks for the question. Our approach can directly be applied to discrete-time signals with finitely many time points. If there are T time points and D dimensions then the samples can simply be treated as T*D dimensional data. We are not directly employing spatial or temporal information (in contrast to the mentioned reference). Deciding whether variables, which are close in space or in time, should be explained by the same expert is part of the learning process.
However, a prior that favors such assignments of expertise could in principle be added. Since we do not assume that the expert opinions are already available (learning them is one of the main tasks) it seems difficult to work with online observations (one variable at a time) because the expert opinions and levels of expertise have to be learned at the same time.
However, if one D-dimensional sample at a time is observed then we can use a simple online variant of our EM algorithm.
Could you contrast your work with some other common learning-with-experts approaches like static expert or fixed-share etc.? In example see [M. Herbster and M. K. Warmuth. Tracking the best expert. Machine Learning, 32:151–178,1998] or [Olivier Bousquet and Manfred K. Warmuth. Tracking a small set of experts by mixing
past posteriors. Journal of Machine Learning Research, 3:363–396, 2002] etc.
Thanks for the important references. Their online framework is rather different from our (offline) setup. They focus on sequential predictions while we have all observations available from the beginning. We added a paragraph at the end of section 2 to discuss the differences.
For example, they consider a single sequence of variables with a fixed partitioning into experts supports. In our setup the partitioning changes dynamically depending on the observed sample.
However, the biggest difference to our work is that they do not learn the individual experts opinions but only focus on training the levels of expertise. It is assumed that a stream of predictions from the experts is already available while in our work learning of the expert opinions is one of the main tasks (in addition to learning the expertise).
This paper proposes a new kind of expert model where a sparse subset of most reliable experts is chosen instead of the usual logarithmic opinion pool of a PoE.
I find the paper very unclear. I tried to find a proper definition of the joint model p(x,z) but could not extract this from the text. The proposed “EM-like” algorithm should then also follow directly from this definition. At this point I do not see if such as definition even exists.
In other words, is there is an objective function on which the iterates of the proposed algorithm are guaranteed to improve on the train data?
We also note that the “product of unifac models” from Hinton tries to do something very similar where only a subset of the experts will get activated to generate the input: http://www.cs.toronto.edu/~hinton/absps/tr00-004.pdf
I tried to derive the update rule on top of page 4 from the “conditional objective for p(x|h)” in sec. 3.2 But I am getting something different (apart form the extra smoothing factors eps and mu_o). Does this follow? (If we define R=R_nk, mu-mu_k and X=X_n, I get mu = (XR)*inv(R^TR) as the optimal solution, which then needs to be projected back onto the probability simplex).
The experiments are only illustrative. They don’t compare with other methods (such as an RBM or VAE) nor do they give any quantitative results. We are left with eyeballing some images. I have no idea whether what we see is impressive or not.
Thanks a lot for you comments. You raised some important points.
1) In this work we focus on learning a compact distributed representation. The experts are trained such that the conditional likelihood of the data given the latent state is maximized. Just as for autoencoders or sparse dictionaries we are not explicitly specifying the distribution of the latent variables.
But you are right, in order to obtain a fully generative model this has to be added. In our case P(h) is a relatively low-dimensional distribution with weak dependencies, so there are many ways to model that. We will add some possible options to the discussion.
2) Since we do not have a joint model, maybe we should not have called it an EM algorithm since the proposed algorithm does not formally improve a lower bound.
3) Thanks for your effort in trying to optimize the objective function in 3.2. In your derivation you probably left out the denominator of the gradient (see 6.1.2), which also depends on the expert opinions. We actually do not think that there is an analytic expression for the solution. One option would then be to use numeric optimization, but we discussed the problems with that. Our proposal is a heuristic, which works very well and which is quite stable. The update rule is directly motivated by the exact update that is available in the unsmoothed model, see equation (1).
4) Thanks for the important reference. We will discuss how that approach relates to our work and add it to the paper.
5) In section 4.1 we are comparing our results with autoencoders, sparse dictionaries and restricted Boltzmann machines. On that dataset our model clearly outperforms the other three. Our earlier paper (http://arxiv.org/abs/1412.3708) contains quantitative results for a similar dataset and a similar model, also comparing to autencoders and RBMs. We left this out here due to space constraints.
We substantially revised our paper and think that the exposition is now much clearer.
The goal of this paper is to learn “ a collection of experts that are individually
meaningful and that have disjoint responsibilities.” Unlike a standard mixture model, they “use a different mixture for each dimension d.” While the results seem promising, the paper exposition needs significant improvement.
Comments:
The paper jumps in with no motivation at all. What is the application, or even the algorithm, or architecture that this is used for? This should be addressed at the beginning.
The subsequent exposition is not very clear. There are assertions made with no justification, e.g. “the experts only have a small variance for some subset of the variables while the variance of the other variables is large.”
Since you’re learning both the experts and the weights, can this be rephrased in terms of dictionary learning? Please discuss the relevant related literature.
The horse data set is quite small with respect to the feature dimension, and so the conclusions may not necessarily generalize.
Thanks for your feedback!
1) The introduction was completely reorganized. The first paragraph verbally describes the goal of the paper and the second paragraph formalizes it. We then introduce partition models, which form the basis of our work. The main implication of our paper (that it's not necessary to use multiple experts to explain individual variables in HD data) is stated in the last paragraph.
2) The statement "experts only have a small variance for some subset of the variables while the variance of the other variables is large" was clarified and rephrased in Section 2.2.2.
3) Yes, sparse dictionary learning attempts to do something similar. The main difference is that opinion pools of the form sum_k h(k)*w_k(d) are used (instead of only one expert per variable). We added a reference in Section 3. There is also an experimental comparison in Section 4.1.
4) I'm not sure if I correctly understand your last comment. Do you mean the number of factors of variation in the dataset? In this experiment we show that a very HD dataset can be reduced to a small number of experts (here 20) while still allowing reasonable reconstructions. If there are more factors of variation then more experts will be needed.
The paper addresses the problem of learning compact binary data representations. I have a hard time understanding the setting and the writing of the paper is not making it any easier.
For example I can't find a simple explanation of the problem and I am not familiar with these line of research. I read all the responses provided by authors to reviewer's questions and re-read the paper again and I still do not fully understand the setting and thus can't really evaluate the contributions of these work. The related work section does not exist and instead the analysis of the literature is somehow scattered across the paper. There are no derivations provided. Statements often miss references, e.g. the ones in the fourth paragraph of Section 3. This makes me conclude that the paper still requires significant work before it can be published.
Thanks for your feedback! We substantially reorganized our paper to make it more accesible.
1) The introduction was completely rewritten and now clearly states the goal of the paper, the contributions and provides the necessary framework/background of our work.
2) Comparisons with other work are now bundled in Section 3 rather than being scatterd across the paper.
3) We added more references and tried to justify our statements.

Meta-review:
role: PC
data: comment: This paper is about learning distributed representations. All reviewers agreed that the first draft was not clear enough for acceptance.
 
 Reviewer time is limited and a paper that needed a complete overhaul after the reviews were written is not going to get the same consideration as a paper that was well-drafted from the beginning.
 
 It's still the case that it's unclear from the paper how the learning updates or derived. The results are not visually impressive in themselves. It's also still the case that more is needed to demonstrate that this direction is promising compared to other approaches to representation learning.

Decision: Reject