Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: A new sparse coding model is introduced that learns features jointly with their transformations. It is found that inference over per-image transformation variables is hard, so the authors suggest tying these variables across all data points, turning them into global parameters, and using multiple transformations for each feature. Furthermore, it is suggested to use a tree of transformations, where each path down the tree generates a feature by multiplying the root feature by the transformations associated with the edges. The one-layer tree model achieves similar reconstruction error as traditional sparse coding, while using fewer parameters.

This is a nice addition to the literature on sparse coding and the literature on learning transformation models. The authors identify and deal with a difficult inference problem that can occur in transformation models. That said, I am skeptical about the usefulness of the general approach. The authors take it as a given that “learning sparse features and transformations jointly” is an important goal in itself, but this is never really argued or demonstrated with experiments. It doesn’t seem like this method enables new applications, extends our understanding of learning what/where pathways in the brain, or improve our ability to model natural images.

The authors claim that the model extracts pose information, but although the model explicitly captures the transformation that relates different features in a tree, at test time, inference is only performed over the (sparse) coefficient associated with each (feature, transformation) combination, just like in sparse coding. It is not clear what we gain by knowing that each coefficient is associated with a transformation, especially since there are many models that do this general “what / where” split.

It would be good to check that the x_{v->b} actually change significantly from their initialization values. The loss surface still looks pretty bad even for tied transformations, so they may actually not move much. Does the proposed model work better according to some measure, compared to a model where x_{v->b} are fixed and chosen from some reasonable range of parameter values (either randomly or spaced evenly)?

One of the conceptually interesting aspects of the paper is the idea of a tree of transformations, but the advantage of deeper trees is never demonstrated convincingly. It looks like the authors have only just gotten this approach to work on toy data with vertical and horizontal bars.

Finally, it is not clear how the method could be extended to have multiple layers. The transformation operators T can be defined in the first layer because they act on the input space, but the same cannot be done in the learned feature space. It is also not clear how the pose information should be further processed in a hierarchical manner, or how learning in a deep version should work.

In summary, I do not recommend this paper for publication, because it is not clear what problem is being solved, the method is only moderately novel and the novel aspects are not convincingly shown to be beneficial. 

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an approach to unsupervised learning based on a modification to sparse coding that allows for explicit modeling of transformations (such as shift, rotation, etc.), as opposed to simple pooling as is typically done in convnets.  Results are shown for training on natural images, demonstrating that the algorithm learns about features and their transformations in the data.  A comparison to traditional sparse coding shows that it represents images with fewer degrees of freedom.

This seems like a good and interesting approach, but the work seems like its still in its early formative stages rather than a complete work with a compelling punch line.  For example one of the motivations is that you'd like to represent pose along with the identity of an object.  While this work seems well on its way to that goal, it doesn't quite get there - it leaves a lot of dots still to be connected.  

Also there are a number of things that aren't clear in the paper:

o The central idea of the paper it seems is the use of a transformational sparse coding tree to make tractable the inference of the Lie group parameters x_k.  But how exactly this is done is not at all clear.  For example, the sentence: "The main idea is to gradually marginalize over an increasing range of transformations," is suggestive but not clear.  This needs to be much better defined.  What do you mean by marginalization in this context?  

 o The connection between the Lie group operator and the tree leaves and weights w_b is not at all clear.   The learning rule spells out the gradient for the Lie group operator, but how this is used to learn the leaves of the tree is not clear.  A lot is left to the imagination here.  This is especially confusing because although the Lie group operator is introduced earlier, it is then stated that its not tractable for inference because there are too many local minima, and this motivates the tree approach instead.  So its not clear why you are learning the Lie group operator.

 o It is stated that "Averaging over many data points, smoothens the surface of the error function."  I don't understand why you would average over many data points.  It seems each would have its own transformation, no?

 o What data do you train on?  How is it generated?  Do you generate patches with known transformations and then show that you can recover them?  Please explain.

The results shown in Figure 4 look very interesting, but given the lack of clarity in the above, difficult to interpret and understand what this means, and its significance.

I would encourage the authors to rewrite the paper more clearly and also to put more work into further developing these ideas, which seem very promising.




review_title: Review of "transformational sparse coding"

final_score: rating: null
review_content: Could you elaborate on the (potential) advantage of using deeper hierarchies / trees? It seems that the T's at different levels could simply be composed into a 1-level hierarchy. The collapsed tree would have fewer parameters.

A technicality: what do you mean by hyperbolic transformation in this context? Are you sure this is the right terminology? As I understand hyperbolic transformations, they are not part of the affine group as claimed in the paper.

Do you think your method could be scaled up to a multi-layer network? How would learning work, given that layer-wise training has not been very successful recently?

Could you explain the relative merits of this work as compared to recent work on learning equivariant representations [1,2,3]

[1] Gens, R., & Domingos, P. (2014). Deep Symmetry Networks. In Advances in Neural Information Processing Systems (NIPS).
[2] Cohen, T. S., & Welling, M. (2016). Group equivariant convolutional networks. In Proceedings of The 33rd International Conference on Machine Learning (ICML).
[3] Dieleman, S., De Fauw, J., & Kavukcuoglu, K. (2016). Exploiting Cyclic Symmetry in Convolutional Neural Networks. In International Conference on Machine Learning (ICML).
review_title: Multiple layers

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper trains a generative model of image patches, where dictionary elements undergo gated linear transformations before being combined. The transformations are motivated in terms of Lie group operators, though in practice they are a set of fixed linear transformations. This is motivated strongly in terms of learning a hierarchy of transformations, though only one layer is used in the experiments (except for a toy case in the appendix).

I like the motivation for this algorithm. The realization seems very similar to a group or block sparse coding implementation. I was disappointed by the restriction to linear transformations. The experiments were all toy cases, demonstrating that the algorithm can learn groups of Gabor- or center surround-like features. They would have been somewhat underpowered five years ago, and seemed extremely small by today's standards.

Specific comments:

Based on common practices in ML literature, I have a strong bias to think of $x$ as inputs and $w$ as network weights. Latent variables are often $z$ or $a$. Depending on your target audience, I would suggest permuting your choice of symbols so the reader can more quickly interpret your model.

nit: number all equations for easier reference

sec 2.2 -- It's weird that the transformation is fixed, but is still written as a function of x.

sec 2.3 -- The updated text here confuses me actually. I had thought that you were using a fixed set of linear transformations, and were motivating in terms of Lie groups, but were not actually taking matrix exponentials in your algorithm. The equations in the second half of this section suggest you are working with matrix exponentials though. I'm not sure which direction I'm confused in, but probably good to clarify the text either way.

BTW -- there's another possible solution to the local minima difficulty, which is the one used in Sohl-Dickstein, 2010. There, they introduce blurring operators matched to each transformation operator, and gradient descent can escape local minima by detouring through coarser (more blurred) scales.

sec 3.2 -- I believe by degrees of freedom you mean the number of model parameters, not the number of latent coefficients that must be inferred? Should make this more clear. Is it more appropriate to compare reconstruction error while matching number of model parameters, or number of latent variables?

I wonder if a convolutional version of this algorithm would be practical / would make it more suited as a generative model of whole images.

====
post rebuttal update

Thank you for taking the time to write the rebuttal! I have read it, but it did not significantly effect my rating.
review_title: review

final_score: rating: null
review_content: How is sparsity defined in Table 1? I'm unsure how to interpret these numbers.

Could you further explain the regularizers in Section 2.3? It was especially unclear to me how to interpret the action of the first regularizer, S_in. (what is the role of the outer square in the definition of S_in?)

Also, what is the form of the regularization penalty for the scaling and hyperbolic transformations?
review_title: definition of sparsity, regularizers

Meta-review: This paper learns affine transformations from images jointly with object features. The motivation is interesting and sound, but the experiments fail to deliver and demonstrate the validity of the claims advanced -- they are restricted to toy settings. What is presented as logical next steps for this work (extending to higher scale multilayer convolutional frameworks, beyond toy settings) seems necessary for the paper to hold its own and deliver the promised insights.
Decision: Reject