Title: Transformational Sparse Coding
Authors: Dimitrios C. Gklezakos
Rajesh P. N. Rao
Venue: ICLR 2017
Keywords: Unsupervised Learning
Computer vision
Optimization

Abstract: 
A fundamental problem faced by object recognition systems is that
objects and their features can appear in different locations, scales
and orientations. Current deep learning methods attempt to achieve
invariance to local translations via pooling, discarding the locations
of features in the process.  Other approaches explicitly learn
transformed versions of the same feature, leading to representations
that quickly explode in size. Instead of discarding the rich and
useful information about feature transformations to achieve
invariance, we argue that models should learn object features
conjointly with their transformations to achieve equivariance.  We
propose a new model of unsupervised learning based on sparse coding
that can learn object features jointly with their affine
transformations directly from images. Results based on learning from
natural images indicate that our approach
matches the reconstruction quality of traditional sparse coding but
with significantly fewer degrees of freedom while simultaneously
learning transformations from data. These results open the door to
scaling up unsupervised learning to allow deep feature+transformation
learning in a manner consistent with the ventral+dorsal stream
architecture of the primate visual cortex.

TL;DR: We extend sparse coding to include general affine transformations. We present a novel technical approach to circumvent inference intractability.

Review discussion:
How is sparsity defined in Table 1? I'm unsure how to interpret these numbers.
Could you further explain the regularizers in Section 2.3? It was especially unclear to me how to interpret the action of the first regularizer, S_in. (what is the role of the outer square in the definition of S_in?)
Also, what is the form of the regularization penalty for the scaling and hyperbolic transformations?
1) Sparsity is defined as the average number of active features (non-zero weights).
2) S_in penalizes the total magnitude of weights within each tree. This is done to balance the weight magnitudes across
trees so that no tree dominates the reconstruction.
3) Scaling and hyperbolic transformations are directly penalized at the parameter level with an L2 penalty.
This is the last term in the full loss function in Section 2.4
We will adjust the paper to make these clear.
Apologies for the mild pedantry, but just to make sure this is fully clear:
1 -- How do you identify zero values? Is your optimizer such that you get exact zeros? A closely related question -- what optimization technique do you use? I don't believe you describe optimization in the text. In the caption or text you should identify that you are presenting an L0 sparsity measure.
2 -- As presented in the text, S_in is the squared sum of individual weight magnitudes. Above you say it is the total magnitude of the weights, which which would not be squared, and which I would typically interpret as the square root of the sum of the squares of each weight value. Is the equation in the text correct? I originally asked because it's a slightly unusual penalty, and I wondered what properties it would encourage.
In the updated version of the paper we have simplified our regularization.
Instead of regularizing leaves, we constrain roots to be of unit
norm and penalize magnitude changing transformations. This way inter- and intra- tree regularization is no longer required. For optimizing weights
(as we have now clarified in the paper) we use the feature sign algorithm introduced by Lee et al. ("Efficient sparse coding algorithms",NIPS 2007).
All results and figures in the paper are current.
Could you elaborate on the (potential) advantage of using deeper hierarchies / trees? It seems that the T's at different levels could simply be composed into a 1-level hierarchy. The collapsed tree would have fewer parameters.
A technicality: what do you mean by hyperbolic transformation in this context? Are you sure this is the right terminology? As I understand hyperbolic transformations, they are not part of the affine group as claimed in the paper.
Do you think your method could be scaled up to a multi-layer network? How would learning work, given that layer-wise training has not been very successful recently?
Could you explain the relative merits of this work as compared to recent work on learning equivariant representations [1,2,3]
[1] Gens, R., & Domingos, P. (2014). Deep Symmetry Networks. In Advances in Neural Information Processing Systems (NIPS).
[2] Cohen, T. S., & Welling, M. (2016). Group equivariant convolutional networks. In Proceedings of The 33rd International Conference on Machine Learning (ICML).
[3] Dieleman, S., De Fauw, J., & Kavukcuoglu, K. (2016). Exploiting Cyclic Symmetry in Convolutional Neural Networks. In International Conference on Machine Learning (ICML).
1) A significant advantage of deeper trees is structure. Deeper trees with additive parameters at each branch explore the parameter space in a more structured way.
When regularized properly, the subtree corresponding to an internal node tends to explore the parameter subspace close to that node. In potential tasks where it is disadvantageous to marginalize completely over transformations, intermediate equivariant representations associated with internal nodes can be used. Additive parameters also provide a way for the model to make global, as well as local decisions, since changing the parameters at a branch can affect the whole subtree. In our experiments using binary trees is slower, but slightly improves the reconstruction error. We are in the process of updating the paper with a small example that demonstrates the above, as well as a formal definition of deeper trees with additive parameters.
2) These transformations are: a) parallel hyperbolic deformation along the x/y axis and b) hyperbolic deformation along the diagonals.
Both belong to the 2D general affine group GA(2). A justification of why these are good candidates for inclusion in a model of visual perception
can be found in:
Dodwell, P. C. (1983). The Lie transformation group model of visual perception. Perception and Psychophysics, 34(1), 1–16.
We will include the proper transformation names and citation in the paper. We will include figures that show how each generator affects a template
in the appendix.
3) Extending our model to a hierarchy is definitely our end-goal and the subject of on-going work. The single-layer model introduced in our paper produces an output similar to the first layer of capsules introduced by Hinton et al. with the advantage that it reuses filters in different poses. This layer converts raw pixel values to a set of features+poses. A hierarchy would combine these features with an additional in-feature pose for complex features.
4) All three papers mentioned are models for supervised learning. The first supports similar transformations. The second supports translations, reflections and 90 degree rotations (not arbitrary rotations). The last supports only translations and rotations. Our method is unsupervised, extends sparse coding and supports all transformations in the GA(2). It can potentially support all transformations described by the associated first order differential equation. We will position our work better within the literature, include the appropriate references and update our submission accordingly.
To further expand the list of prior work, I wonder if you could also briefly elaborate on the connection to
Culpepper, Benjamin, and Bruno A. Olshausen. "Learning transport operators for image manifolds." Advances in neural information processing systems. 2009.
This work is also relevant to a specific claim in your paper. One of the motivations for your approach in Sec. 2.1 is that \partial L / \partial x cannot be computed analytically. I believe the Culpepper+Olshausen approach involves evaluating this gradient analytically.
We added the reference and corrected our claims in the updated version of the paper. Our approach to computing the gradient
can actually be interpreted as a stochastic version of the one used in the reference you provided.
Revision #1
Thank you for your comments and suggestions. We uploaded a new version of the paper to address them:
1) We added all relevant references, corrected our claims and updated our "Relevant Work" section.
2) We added a figure that shows the effects of each transformation.
3) We added a formal definition of deeper trees and added a small example that shows learned structure.
4) We added more figures of learned features.
5) We changed our regularization.
Instead of regularizing derivative features, we constrain root features to be of unit
norm and penalize transformations that change the magnitude. Inter- and intra- tree regularization is no longer required
and we can use the feature-sign algorithm to infer the weights.
6) We removed our parameter distance and magnitude figures, since these metrics depend heavily on the initialization approach of choice
and hence are not very informative.
All results and figures are current.
Revision #2
1) Some transformations change feature magnitudes and that affects sparsity patterns. To make the comparison with traditional sparse coding
clearer, we first run our model and then sparse coding, constraining the norm of the features to be equal to the average norm of our model.
This makes it easier to compare the reconstruction error and the sparsity.
2) We added more results (layouts and sparsity penalties) and figures of learned features. Decreasing the penalties of the magnitude preserving
transformations allows us to learn more diverse dictionaries and larger transformations.
3) Replaced (# of pixels) to (# of pixels - 1) in the degrees of freedom to account for the feature norm constraint.
Revision #3:
1) Some features can get initialized around shallow local minima. These features can become under-used by the model and not contribute a lot to the reconstruction. We modified our learning approach to re-initialize under-used features (defined as the number of data points using the feature in a batch dropping close to zero). We added the details of re-initialization in the relevant section. Re-initialization also gets rid of features that are initialized close or outside the receptive field, creating aliasing effects. Re-initialization also offers a slight improvement in reconstruction quality.
A new sparse coding model is introduced that learns features jointly with their transformations. It is found that inference over per-image transformation variables is hard, so the authors suggest tying these variables across all data points, turning them into global parameters, and using multiple transformations for each feature.
Furthermore, it is suggested to use a tree of transformations, where each path down the tree generates a feature by multiplying the root feature by the transformations associated with the edges. The one-layer tree model achieves similar reconstruction error as traditional sparse coding, while using fewer parameters.
This is a nice addition to the literature on sparse coding and the literature on learning transformation models. The authors identify and deal with a difficult inference problem that can occur in transformation models.
That said, I am skeptical about the usefulness of the general approach. The authors take it as a given that “learning sparse features and transformations jointly” is an important goal in itself, but this is never really argued or demonstrated with experiments. It doesn’t seem like this method enables new applications, extends our understanding of learning what/where pathways in the brain, or improve our ability to model natural images.
The authors claim that the model extracts pose information, but although the model explicitly captures the transformation that relates different features in a tree, at test time, inference is only performed over the (sparse) coefficient associated with each (feature, transformation) combination, just like in sparse coding. It is not clear what we gain by knowing that each coefficient is associated with a transformation, especially since there are many models that do this general “what / where” split.
It would be good to check that the x_{v->b} actually change significantly from their initialization values. The loss surface still looks pretty bad even for tied transformations, so they may actually not move much. Does the proposed model work better according to some measure, compared to a model where x_{v->b} are fixed and chosen from some reasonable range of parameter values (either randomly or spaced evenly)?
One of the conceptually interesting aspects of the paper is the idea of a tree of transformations, but the advantage of deeper trees is never demonstrated convincingly. It looks like the authors have only just gotten this approach to work on toy data with vertical and horizontal bars.
Finally, it is not clear how the method could be extended to have multiple layers. The transformation operators T can be defined in the first layer because they act on the input space, but the same cannot be done in the learned feature space. It is also not clear how the pose information should be further processed in a hierarchical manner, or how learning in a deep version should work.
In summary, I do not recommend this paper for publication, because it is not clear what problem is being solved, the method is only moderately novel and the novel aspects are not convincingly shown to be beneficial.
-The authors take it as a given that “learning sparse features and transformations jointly” is an important goal in itself, but this is never really argued or demonstrated with experiments. It doesn’t seem like this method enables new applications, extends our understanding of learning what/where pathways in the brain, or improve our ability to model natural images.
There are many reasons why learning features and their transformations is interesting, such as:
- Learning to recognize objects in different poses (location, orientation, scale etc.)
- Exploiting pose parameters in recognition and other tasks (ie when recognition depends on finer grained spatial relations between parts of the object).
- Encoding images more efficiently.
The goal of this paper is to show that natural images have a low-level structure that allows for more concise representations using affine transformations
and to provide a framework for learning these transformations efficiently.
-The authors claim that the model extracts pose information, but although the model explicitly captures the transformation that relates different features in a tree, at test time, inference is only performed over the (sparse) coefficient associated with each (feature, transformation) combination, just like in sparse coding. It is not clear what we gain by knowing that each coefficient is associated with a transformation, especially since there are many models that do this general “what / where” split.
The advantage is that we can now express an image patch as a much smaller set of transformed features. This combined with the fact that the model learns the transformation parameters results in an "equivariant" representation of the patch.
As for the claim that there are many models that provide this "what" and "where" split, do you have any specific work/model in mind that does what our model does? We would definitely want to include that in our references.
-It would be good to check that the x_{v->b} actually change significantly from their initialization values. The loss surface still looks pretty bad even for tied transformations, so they may actually not move much. Does the proposed model work better according to some measure, compared to a model where x_{v->b} are fixed and chosen from some reasonable range of parameter values (either randomly or spaced evenly)?
This is an interesting point. The first version of our paper had a figure that showed the l1 norm of the deviation from the initialized values.
When we initialize the parameters with very small variance (with zero mean) the learned values differ significantly from the initial ones. If we initialize the parameters with large variance and use regularization it is unclear whether we should attribute this deviation to regularization or to learning. We decided to err on the side of caution and removed the figure.
-One of the conceptually interesting aspects of the paper is the idea of a tree of transformations, but the advantage of deeper trees is never demonstrated convincingly. It looks like the authors have only just gotten this approach to work on toy data with vertical and horizontal bars.
While investigating deeper trees is certainly the subject of ongoing work, at submission time we had only tested them on toy examples. We included one of these examples to show that learning more structured representations using deeper trees is a possibility.
-Finally, it is not clear how the method could be extended to have multiple layers. The transformation operators T can be defined in the first layer because they act on the input space, but the same cannot be done in the learned feature space. It is also not clear how the pose information should be further processed in a hierarchical manner, or how learning in a deep version should work.
This is a fair point. The model converts raw pixel values to feature weights and pose parameters in a manner consistent with capsules by Hinton et al.
It is true that the same module cannot be directly stacked to form a deep version.
While building a deeper version is definitely our end goal and top priority, it is out of the scope of this paper.
This paper proposes an approach to unsupervised learning based on a modification to sparse coding that allows for explicit modeling of transformations (such as shift, rotation, etc.), as opposed to simple pooling as is typically done in convnets.  Results are shown for training on natural images, demonstrating that the algorithm learns about features and their transformations in the data.  A comparison to traditional sparse coding shows that it represents images with fewer degrees of freedom.
This seems like a good and interesting approach, but the work seems like its still in its early formative stages rather than a complete work with a compelling punch line.
For example one of the motivations is that you'd like to represent pose along with the identity of an object.
While this work seems well on its way to that goal, it doesn't quite get there - it leaves a lot of dots still to be connected.
Also there are a number of things that aren't clear in the paper:
o The central idea of the paper it seems is the use of a transformational sparse coding tree to make tractable the inference of the Lie group parameters x_k.
But how exactly this is done is not at all clear.
For example, the sentence: "The main idea is to gradually marginalize over an increasing range of transformations," is suggestive but not clear.  This needs to be much better defined.  What do you mean by marginalization in this context?
o The connection between the Lie group operator and the tree leaves and weights w_b is not at all clear.   The learning rule spells out the gradient for the Lie group operator, but how this is used to learn the leaves of the tree is not clear.  A lot is left to the imagination here.  This is especially confusing because although the Lie group operator is introduced earlier, it is then stated that its not tractable for inference because there are too many local minima, and this motivates the tree approach instead.
So its not clear why you are learning the Lie group operator.
o It is stated that "Averaging over many data points, smoothens the surface of the error function."  I don't understand why you would average over many data points.  It seems each would have its own transformation, no?
o What data do you train on?  How is it generated?  Do you generate patches with known transformations and then show that you can recover them?  Please explain.
The results shown in Figure 4 look very interesting, but given the lack of clarity in the above, difficult to interpret and understand what this means, and its significance.
I would encourage the authors to rewrite the paper more clearly and also to put more work into further developing these ideas, which seem very promising.
-The central idea of the paper it seems is the use of a transformational sparse coding tree to make tractable the inference of the Lie group parameters x_k.
But how exactly this is done is not at all clear.
For example, the sentence: "The main idea is to gradually marginalize over an increasing range of transformations," is suggestive but not clear.  This needs to be much better defined.  What do you mean by marginalization in this context?
The main idea here is that a tree "reduces" a set of features to a smaller one by factoring out affine transformations. A single layer tree radically reduces a whole set to a single feature. A deeper tree gradually reduces a set of features to a single one, the root. Deeper trees could provide a continuum of intermediate representations, where some transformations are factored out. We understand that we do not have compelling evidence for the usefulness of deeper trees yet and will modify our claims in this section accordingly.
-The connection between the Lie group operator and the tree leaves and weights w_b is not at all clear.   The learning rule spells out the gradient for the Lie group operator, but how this is used to learn the leaves of the tree is not clear.  A lot is left to the imagination here.  This is especially confusing because although the Lie group operator is introduced earlier, it is then stated that its not tractable for inference because there are too many local minima, and this motivates the tree approach instead.
So its not clear why you are learning the Lie group operator.
A single layer tree consists of the root and the leaves. The leaves correspond to transformed versions of the same root. These transformations are derived by a linear combination of the group generators (parametrized by "x") and the matrix exponential. The group generators as fixed and pre-selected to correspond to some interesting basic transformations. The linear combination parameters ("x") define the transformation associated with each leaf as a mixture of such basic transformations and are learned from the data. We suggest that naively performing inference from scratch on each data-point is intractable due to the number of local minima. To counter that, we propose learning common transformations that can be used for all data points. Essentially what we are doing is using affine transformations to extract pose information from a sparse code. We reduce groups of features (leaves) to a single feature (root) by factoring out pose.
-It is stated that "Averaging over many data points, smoothens the surface of the error function."  I don't understand why you would average over many data points.  It seems each would have its own transformation, no?
No. Each patch is modeled as a sparse linear combination of features. These features are modeled as leaves in a forest. Each leaf corresponds to a transformed version of its root. The transformations learned describe how to obtain the leaves.
-What data do you train on?  How is it generated?  Do you generate patches with known transformations and then show that you can recover them?  Please explain.
We train on patches that are randomly extracted from a set of 10 natural images. The dataset is the same as the one used by Lee et al. (2007) (which is also the version of sparse coding that we compare with).
This paper trains a generative model of image patches, where dictionary elements undergo gated linear transformations before being combined. The transformations are motivated in terms of Lie group operators, though in practice they are a set of fixed linear transformations. This is motivated strongly in terms of learning a hierarchy of transformations, though only one layer is used in the experiments (except for a toy case in the appendix).
I like the motivation for this algorithm. The realization seems very similar to a group or block sparse coding implementation. I was disappointed by the restriction to linear transformations. The experiments were all toy cases, demonstrating that the algorithm can learn groups of Gabor- or center surround-like features. They would have been somewhat underpowered five years ago, and seemed extremely small by today's standards.
Specific comments:
Based on common practices in ML literature, I have a strong bias to think of $x$ as inputs and $w$ as network weights. Latent variables are often $z$ or $a$. Depending on your target audience, I would suggest permuting your choice of symbols so the reader can more quickly interpret your model.
nit: number all equations for easier reference
sec 2.2 -- It's weird that the transformation is fixed, but is still written as a function of x.
sec 2.3 -- The updated text here confuses me actually. I had thought that you were using a fixed set of linear transformations, and were motivating in terms of Lie groups, but were not actually taking matrix exponentials in your algorithm. The equations in the second half of this section suggest you are working with matrix exponentials though. I'm not sure which direction I'm confused in, but probably good to clarify the text either way.
BTW -- there's another possible solution to the local minima difficulty, which is the one used in Sohl-Dickstein, 2010. There, they introduce blurring operators matched to each transformation operator, and gradient descent can escape local minima by detouring through coarser (more blurred) scales.
sec 3.2 -- I believe by degrees of freedom you mean the number of model parameters, not the number of latent coefficients that must be inferred? Should make this more clear. Is it more appropriate to compare reconstruction error while matching number of model parameters, or number of latent variables?
I wonder if a convolutional version of this algorithm would be practical / would make it more suited as a generative model of whole images.
post rebuttal update
Thank you for taking the time to write the rebuttal! I have read it, but it did not significantly effect my rating.
-I like the motivation for this algorithm. The realization seems very similar to a group or block sparse coding implementation. I was disappointed by the restriction to linear transformations.
Our intuition says that if we look close enough (smaller patches), basic transformations are appropriate for modeling the input. We expect that these linear transformations combined at multiple levels can result in more complex, non-linear ones.
-The experiments were all toy cases, demonstrating that the algorithm can learn groups of Gabor- or center surround-like features. They would have been somewhat underpowered five years ago, and seemed extremely small by today's standards.
Our goal in this paper is:
1) To show that affine transformations are appropriate for modeling the local structure of images and explain most of the small-scale variation of the input.
2) To provide an efficient learning framework that captures this structure.
Scaling this approach to large images via a convolutional, multi-layer approach is the next step in our work.
-Based on common practices in ML literature, I have a strong bias to think of $x$ as inputs and $w$ as network weights. Latent variables are often $z$ or $a$. Depending on your target audience, I would suggest permuting your choice of symbols so the reader can more quickly interpret your model.
-nit: number all equations for easier reference
We will adjust our submission accordingly.
-sec 2.2 -- It's weird that the transformation is fixed, but is still written as a function of x.
It is written as a function of x, since the transformation is parametrized by x.
Even though the transformations are fixed across data points, their parameters still have to be learned from the data.
-sec 2.3 -- The updated text here confuses me actually. I had thought that you were using a fixed set of linear transformations, and were motivating in terms of Lie groups, but were not actually taking matrix exponentials in your algorithm. The equations in the second half of this section suggest you are working with matrix exponentials though. I'm not sure which direction I'm confused in, but probably good to clarify the text either way.
The model learns fixed linear transformations of sparse coding filters. Depending on which base transformations (i.e. translations, rotations etc.) we allow, the fixed transformations we learn
belong to the associated Lie group.
Specifically, each such transformation is a mixture of translations, rotations etc. To recover any element in the Lie group, we take the matrix
exponential of an element of the Lie algebra of the group, which is a linear combination of the base transformation generators (denoted as G in the paper). The transformation model (and the matrix exponential)
are introduced in Section 2.1. A more thorough discussion on the transformation model can be found in the references provided in that section (Rao & Ruderman (1999), Miao & Rao (2007) and Dodwell (1983)).
-BTW -- there's another possible solution to the local minima difficulty, which is the one used in Sohl-Dickstein, 2010. There, they introduce blurring operators matched to each transformation operator, and gradient descent can escape local minima by detouring through coarser (more blurred) scales.
The blurring approach in Sohl-Dickstein (2010) is indeed very interesting.
However, it is introduced only for single-parameter transformations (single generator, one-dimensional Lie group). In our work
we use multi-dimensional Lie groups to model the learned transformations as a mixture of more basic ones (translations, rotations etc.) whereas Sohl and Dickstein "chain" multiple one-dimensional transformations
together to recover bigger ones. We also decompose the input as a linear combination of transformed features or parts, whereas they focus on learning "whole-image" transformations between consecutive
video frames.
Therefore applying the blurring approach is not straight-forward and might not be computationally efficient. One of our next steps in this line of work is to fine-tune the features
individually for each data-point (preliminary results indicate that this significantly improves the reconstruction error).
Then our model would resemble a coarse-to-fine optimization approach, where
the selected sparse coding feature corresponds to a "global" step in the parameter space and fine-tuning corresponds to local optimization in that region. The set of sparse coding features
forms a "grid" that spans the parameter space, learned from data.
-sec 3.2 -- I believe by degrees of freedom you mean the number of model parameters, not the number of latent coefficients that must be inferred? Should make this more clear. Is it more appropriate to compare reconstruction error while matching number of model parameters, or number of latent variables?
In the degrees of freedom we include all common across data-points variables that have to be learned. These are the pixel values of each feature plus the transformation parameters for each leaf. Since the norm
of the features is constrained to be equal to 1, we subtract one degree of freedom for each feature (if we know all but one, the last pixel value is determined by the constraint). In the degrees of freedom
of the model we do not include the weights, since these are individual to each data point. The degrees of freedom can be interpreted as the size of the code.
-I wonder if a convolutional version of this algorithm would be practical / would make it more suited as a generative model of whole images.
This is definitely the subject of ongoing work.

Meta-review:
role: PC
data: comment: This paper learns affine transformations from images jointly with object features. The motivation is interesting and sound, but the experiments fail to deliver and demonstrate the validity of the claims advanced -- they are restricted to toy settings. What is presented as logical next steps for this work (extending to higher scale multilayer convolutional frameworks, beyond toy settings) seems necessary for the paper to hold its own and deliver the promised insights.

Decision: Reject