Title: Hierarchical compositional feature learning
Authors: Miguel Lazaro-Gredilla
Yi Liu
D. Scott Phoenix
Dileep George
Venue: ICLR 2017
Keywords: Unsupervised Learning

Abstract: We introduce the hierarchical compositional network (HCN), a directed generative model able to discover and disentangle, without supervision, the building blocks of a set of binary images. The building blocks are binary features defined hierarchically as a composition of some of the features in the layer immediately below, arranged in a particular manner. At a high level, HCN is similar to a sigmoid belief network with pooling. Inference and learning in HCN are very challenging and existing variational approximations do not work satisfactorily. A main contribution of this work is to show that both can be addressed using max-product message passing (MPMP) with a particular schedule (no EM required). Also, using MPMP as an inference engine for HCN makes new tasks simple: adding supervision information, classifying images, or performing inpainting all correspond to clamping some variables of the model to their known values and running MPMP on the rest. When used for classification, fast inference with HCN has exactly the same functional form as a convolutional neural network (CNN) with linear activations and binary weights. However, HCN’s features are qualitatively very different.

TL;DR: We show that max-product message passing with an appropriate schedule can be used to perform inference and learning in a directed multilayer generative model, thus recovering interpretable features from binary images.

Review discussion:
Does the on-line learning mechanism (Sec. 4.2) affect accuracy?
When on-line learning is used, we can present a new minibatch of images for each inference iteration. In this case, for the same computational cost, the final accuracy is higher in the on-line learning case (more training data has been seen by HCN). If we were to use exactly the same set of images in both on-line and batch mode (which would imply re-feeding the same set of images epoch after epoch to the online version -- an option that would only make sense in a data-constrained scenario), then the final accuracies of both would be comparable.
A comparison of batch and on-line modes for the same computational cost when the on-line method uses fresh data on each iteration can be seen in Figures 6.(c) and 6.(e). In the first case (batch), 30 images are presented for 100 epochs (the same 30 images are used epoch after epoch). In the second case (on-line), minibatches of 5 images are processed and then discarded, until a total of 3000 different images have been presented to HCN. In both cases, the computational cost is the same.
However, 6.(e) shows a more accurate recovery of the features since more images have been seen.
If we want to compare batch and on-line modes when the same number of images are available for both, we can run on-line mode and reuse the same images. This is done in Figure 6.(d). In that case, a total of only 30 images are used (like in batch mode), which are fed in minibatches of size 5 to the network, for 100 epochs (so that the 30 images are presented a total of 100 times). We can see that the quality of the solution in Figure 6.(d) (on-line) is comparable to that of Figure 6.(c) (batch). Again, the computational cost is the same.
However, in this case we observed convergence to be faster for the batch mode.
This paper presents a generative model for binary images.  Images are composed by placing a set of binary features at locations in the image.  These features are OR'd together to produce an image.  In a hierarchical variant, features/classes can have a set of possible templates, one of which can be active.  Variables are defined to control which template is present in each layer.  A joint probability distribution over both the feature appearance and instance/location variables is defined.
Overall, the goal of this work is interesting -- it would be satisfying if semantically meaningful features could be extracted, allowing compositionality in a generative model of images.
However, it isn't clear this would necessarily result from the proposed process.
Why would the learned features (building blocks) necessarily semantically meaningful?  In the motivating example of text, rather than discovering letters, features could correspond to many other sub-units (parts of letters), or other features lacking direct semantic meaning.
The current instantiation of the model is limited.  It models binary image patterns.  The experiments are done on synthetic data and MNIST digits.  The method recovers the structure and is effective at classification on synthetic data that are directly compositional.  On the MNIST data, the test errors are quite large, and worse than a CNN except when synthetic data corruption is added.
Further work to enhance the ability of the method to handle natural images or naturally occuring data variation would enhance the paper.
Thank you for your review. We have modified the paper to address your comments.
Also, see the responses below.
The goal of the paper is to  disentangle the hierarchical features that compose a binary image in an unsupervised manner. We manage to do so in several examples that, as far as we know, cannot be solved by another method. We do not claim to approach the state of the art in image classification. The experiments on MNIST highlight the interesting fact that this type of feature learning provides additional resistance to unseen clutter, i.e., shows the advantages of a strong structured prior.
Of course, in the regular discriminative setting, HCN is not the right tool to address MNIST or any other image classification task.
Even though HCN was designed to analyze the structure of binary data, we believe the extension to real-valued data should be easy. To show this, we added a preprocessing step with a bank of Gabor filters and fed it with grayscale compositional data. We tested the same exact setup with an alternative feature learning technique, the AND-OR templates of Wu et al. The result is that the extended HCN is able to disentangle the features, whereas the baseline cannot. These results are now part of the Appendices.
Regarding the nature of the extracted features, we did not claim in the paper that the features would be "semantically meaningful".
However, the following constraints on the model is likely to produce features that are compositional: a) non-negativity constraints as in NMF b) directed connections that allow for explaining away c) alternating feature detection and pooling as in CNNs.
Discovering the letters is the correct answer for the example shown in the paper because we know how the images were generated. Recovering the true generative model is an important step, and we show that the model constraints and learning algorithm is sufficient to do this. This is irrespective of the contention regarding semantic meaning, which we do not claim.
Recovering the true generative model also means finding the most compressive representation of data.
Even when it comes to recovering "semantically meaningful features", recovering the true generative model for a process that also respects the causal semantics of data generation is likely to be an important aspect. Modeling causal aspects require directed connections as in the HCN, and learning with directed connections has been recognized as a hard problem even on seemingly simple datasets.
This paper presents an approach to learn object representations by composing a set of templates which are leaned from binary images.
In particular, a hierarchical model is learned by combining AND, OR and POOL operations. Learning is performed by using approximated inference with MAX-product BP follow by a heuristic to threshold activations to be binary.
Learning hierarchical representations that are interpretable is a very interesting topic, and this paper brings some good intuitions in light of modern convolutional neural nets.
I have however, some concerns about the paper:
1) the paper fails to cite and discuss relevant literature and claims to be the first one that is able to learn interpretable parts.
I would like to see a discussion of the proposed approach compared to a variety of papers e.g.,:
- Compositional hierarchies of Sanja Fidler
- AND-OR graphs used by Leo Zhu and Alan Yuille to model objects
- AND-OR templates of Song-Chun Zhu's group at UCLA
The claim that this paper is the first to discover such parts should be removed.
2) The experimental evaluation is limited to very toy datasets. The papers I mentioned have been applied to real images (e.g., by using contours to binarize the images).
I'll also like to see how good/bad the proposed approach is for classification in more well known benchmarks.
A comparison to other generative models such as VAE, GANS, etc will also be useful.
3) I'll also like to see a discussion of the relation/differences/advantages of the proposed approach wrt to sum product networks and grammars.
Other comments:
- the paper claims that after learning inference is feed-forward, but since message passing is used, it should be a recurrent network.
- the algorithm and tech discussion should be moved from the appendix to the main paper
- the introduction claims that compression is a prove for understanding. I disagree with this statement, and should be removed.
- I'll also like to see a discussion relating the proposed approach to the Deep Rendering model.
- It is not obvious how some of the constraints are satisfied during message passing.
Also constraints are well known to be difficult to optimize with max product. How do you handle this?
- The learning and inference algorithms seems to be very heuristic (e.g., clipping to 1, heuristics on which messages are run). Could you analyze the choices you make?
- doing multiple steps of 5) 2) is not a single backward pass
I'll reconsider my score in light of the answers
Thank you for your detailed comments.
As a result, we have made several modifications to the paper. We address your concerns below.
- Regarding relevant literature, we have added the necessary ``related work'' section, where we discuss all the papers that you mentioned and a few more (including AND-OR trees, grammars, deep rendering model, sum-product networks, etc).
Despite each of those approaches having different advantages and disadvantages, all of them (with rare exceptions like Noisy-Or component Analysis, which we compare with and the deep rendering model, which we also discuss now), share a common limitation that HCN does not have: they cannot render overlapping features. HCN allows multiple features to overlap, thus creating new compositions.
For instance, if feature H is a centered horizontal line and feature V is a centered vertical line, HCN can create a new feature ``cross'' that combines both, and the fact that both are overlapping and sharing a common active pixel (and many common inactive pixels) is properly handled.
In contrast, previous models cannot overlap features, so they partition the input space and dedicate separate subtrees to each of them, and do so recursively. We can see in Figure 5, top row, how we can generate 25 different cross variations using only two features. This would not be possible with the existing feature learning models mentioned in your review (with the given exception), which would need to span each combination as a separate feature.
Regarding the deep rendering model (DRM), it is to some extent, a continuous counterpart of the present work.
Although DRMs allow for feature overlap, the semantics are different: in HCN the amount of activation of a given pixel is the same whether there are one or many features (causes) activating it, whereas in DRM the activation is proportional to the number of causes. This means that the difference between DRM and HCN is analogous to the difference between principal component analysis and binary matrix factorization: while the first can be solved analytically, the second is hard and not analytically tractable. This results in DRM being more tractable, but less appropriate to handle problems with binary events with multiple causes, such as the ones posed in this paper.
- Regarding the ''false novelty claim'': we have failed to find any existing algorithm capable of finding the parts Figure 1 (left) in the paper like HCN does. Note that the disentangled features are overlapping and binary, so none of the papers mentioned in your review could successfully disentangle them, even from a theoretical perspective.
Furthermore, it can be seen that HCN can solve, as a concrete specialization, binary matrix factorization problems, whereas (for the same reason, lack of a model with multiple causes producing a single binary event, which would require explaining away for inference) none of the models you mentioned can be used for this task. We have therefore not removed the novelty claim.
We implemented the AND-OR templates of Wu et al., and the results show that their method does not disentangle the features like HCN.
Instead, their methods keeps prototypes from the input data and cleans them up. (These results are now part of the Appendices.) Theoretical investigation of the other methods you cited also led us to the same conclusion -- they do not disentangle the features. We would be happy to run any code that you might be aware of on our toy data (Figure 1, left) and report the results.
- HCN was designed to solve binary problems and not for real-valued data.
However, given the interest of multiple reviewers in its applicability to continuous images, we decided to provide a simple extension to test whether it would be able to real-valued data. To this end, we added a preprocessing step with a bank of Gabor filters and fed it with grayscale compositional data. The result is that the extended HCN is able to disentangle the features. These results are now part of the Appendices.
- Regarding VAEs and GANs, they could in principle be applied to learn the HCN model. We are not aware of any work that uses a VAE or GAN with a generative model like HCN and such an option is unlikely to be straightforward, for the reasons that follow.
Most common VAEs rely on the reparameterization trick for variance reduction.
However, this trick cannot be applied to HCN due to the discrete nature of its variables, and alternative methods would suffer from high variance.
Another limitation of VAEs wrt HCN is that they perform a single bottom-up pass and lack explaining away: During learning HCN combines top-down and bottom-up information in multiple passes, isolating the parent cause of a given activation, instead of activating every possible cause.
GANs need to compute the gradient of  D(G_W(eps)) where D() is the discriminative network and G_W(eps) is a generative network parameterized by the features W. In this case, not only W is binary, but also the generated reconstructions at every layer, so the GAN formulation cannot be applied to HCN as-is. One could in principle relax the binary assumption of features and reconstructions and use the GAN paradigm to train a neural network with sigmoidal activations, but it is unclear that the lack of binary variables will still produce proper disentangling.
- Other comments:
+ "the paper claims that after learning inference is feed-forward, but since message passing is used, it should be a recurrent network."
After learning, approximate inference can be performed by a single forward message propagation. Messages are passed on to the next level only once, and the resulting computations can in fact be performed by a convolutional neural network with linear activations, run only once. There is no recurrency in this forward pass. Inference results can be improved with multiple forward and backward passes, in which case the equivalence with a feedforward network disappears.
+ "the algorithm and tech discussion should be moved from the appendix to the main paper"
Done.
+ "the introduction claims that compression is a prove for understanding. I disagree with this statement, and should be removed."
Done.
+ "I'll also like to see a discussion relating the proposed approach to the Deep Rendering model."
Done, included in the ``related work'' section.
+ "It is not obvious how some of the constraints are satisfied during message passing.
Also constraints are well known to be difficult to optimize with max product. How do you handle this?"
Each factors establishes constraints on the variables it is connected to, which in turn define the max-product update equations.
When the max-product updates are iterated, the energy of the whole system is minimized. Because breaking a constraint would be equivalent to infinite energy, its minimization results in most constraints being satisfied. If the global optimum of the energy function was found, every constraint would be satisfied (since by construction there is always a valid solution).
However, max-product is not guaranteed to converge to the optimum of the energy function (or even converge at all) and therefore there is no guarantee of all the constraints being satisfied by simply rounding the beliefs of its solution. The beliefs will be approximately satisfying the solution, though. What we do then is simply round the beliefs (binarize) at the top and propagate top-down, round the beliefs at the next layer and propagate top-down again, and so on. That procedure (like ancestral sampling) is guaranteed to satisfy all constraints.
+ "The learning and inference algorithms seems to be very heuristic (e.g., clipping to 1, heuristics on which messages are run). Could you analyze the choices you make?"
On the contrary, the algorithm is standard max-product with a particular message propagation schedule. We do not clip any messages. After max-product is run in the HCN, the beliefs must be rounded to 0 or 1, which is the standard final step needed after solving any continuous relaxation of a binary problem (like loopy max-product). The 0.5 threshold is not arbitrary, and is standard for binary problems solved using max-product.
Regarding the order in which messages are run, this is a scheduling problem for which we do not know the best answer. Propagating the messages in random order would also work, but would take much longer. The idea is to propagate the information as quickly as possible between the bottom and the top of the network, while proceeding sequentially inside each layer to allow local decisions influence others (for instance, to solve explaining away).
+ "doing multiple steps of 5) 2) is not a single backward pass"
Corrected.
N/A
The paper discusses a method to learn interpretable hierarchical template representations from given data. The authors illustrate their approach on binary images.
The paper presents a novel technique for extracting interpretable hierarchical template representations based on a small set of standard operations. It is then shown how a combination of those standard operations translates into a task equivalent to a boolean matrix factorization. This insight is then used to formulate a message passing technique which was shown to produce accurate results for these types of problems.
Summary:
The paper presents an novel formulation for extracting hierarchical template representations that has not been discussed in that form. Unfortunately the experimental results are on smaller scale data and extension of the proposed algorithm to more natural images seems non-trivial to me.
Quality: I think some of the techniques could be described more carefully to better convey the intuition.
Clarity: Some of the derivations and intuitions could be explained in more detail.
Originality: The suggested idea is reasonable but limited to binary data at this point in time.
Significance: Since the experimental setup is somewhat limited according to my opinion, significance is hard to judge.
Details:
1. My main concern is related to the experimental evaluation.
While the discussed approach is valuable, its application seems limited to binary images at this point in time. Can the authors comment?
2. There are existing techniques to extract representations of images which the authors may want to mention, e.g., work based on grammars.
Thank you for your review. We have made the following modifications to the paper to address your comments:
HCN was designed to analyze the structure of binary data. Noisy-OR component (NOCA) analysis, sigmoid belief networks, the restricted Boltzmann machine and a significant amount of existing machine learning literature deals exclusively with binary data.
However, given the interest of multiple reviewers in its applicability to continuous images, we decided to provide a simple extension to test whether it would be able to handle real-valued data. To this end, we added a preprocessing step with a bank of Gabor filters and fed it with grayscale compositional data.
In addition to testing on HCN, we compared the same exact setup with an alternative feature learning technique, the AND-OR templates of Wu et al. The result is that the extended HCN is able to disentangle the features, whereas the baseline cannot. These results are now part of the Appendices.
We have added a ``related work'' section in which provide further details about the connection with existing literature. There are few directly comparable works, in the sense of modeling capabilities.
For instance, grammars exclude the sharing of sub-parts among multiple objects or object parts in a parse of the scene see [Jin & Geman (2006)], and they limit interpretations to single trees even if those are dynamic [Williams & Adams (1999)]. Our graphical model formulation makes the sharing of lower-level features explicit by using local conditional probability distributions for multi-parent interactions, and allows for MAP configurations (i.e, the parse graphs) that are not trees.
In a nutshell, the difference between in HCN and previous feature learning works (with rare exceptions, like NOCA, with which we compare), is that HCN allows multiple features to overlap, thus creating new compositions.
For instance, if feature H is a centered horizontal line and feature V is a centered vertical line, HCN can create a new feature ``cross'' that combines both, and the fact that both are overlapping and sharing a common active pixel (and many common inactive pixels) is properly handled.
In contrast, previous models cannot overlap features, so they partition the input space and dedicate separate subtrees to each of them, and do so recursively. We can see in Figure 5, top row, how we can generate 25 different cross variations using only two features. This would not be possible with the most existing feature learning models, which would need to span each combination as a separate feature.
[Jin & Geman (2006)]
Ya Jin and Stuart Geman. Context and hierarchy in a probabilistic image model. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pp. 2145–2152. IEEE, 2006.
[Williams & Adams (1999)]
Christopher KI Williams and Nicholas J Adams. DTS: Dynamic trees. Advances in neural information processing systems, pp. 634–640, 1999.

Meta-review:
role: PC
data: comment:  The reviewer's opinions were clear for this paper. Mainly it seems that the fact that this work focuses on binary image patterns limited the ability of reviewers to assess the significance of this work based on the instantiation of the model explored in this work. It was also noted that the writing could have been clearer when describing the intuitions for the approach and that the derivations could have been explained in more detail.

Decision: Reject