Title: Warped Convolutions: Efficient Invariance to Spatial Transformations
Authors: Joao F. Henriques
Andrea Vedaldi
Venue: ICLR 2017

Abstract: Convolutional Neural Networks (CNNs) are extremely efficient, since they exploit the inherent translation-invariance of natural images. However, translation is just one of a myriad of useful spatial transformations. Can the same efficiency be attained when considering other spatial invariances? Such generalized convolutions have been considered in the past, but at a high computational cost. We present a construction that is simple and exact, yet has the same computational complexity that standard convolutions enjoy. It consists of a constant image warp followed by a simple convolution, which are standard blocks in deep learning toolboxes. With a carefully crafted warp, the resulting architecture can be made equivariant to a wide range of 2-parameters spatial transformations. We show encouraging results in realistic scenarios, including the estimation of vehicle poses in the Google Earth dataset (rotation and scale), and face poses in Annotated Facial Landmarks in the Wild (3D rotations under perspective).


Review discussion:
How does the method deal with local transformations? It seems like the method would only be able to deal with a single global transformation.
For example, the log-polar warp would turn rotations around x_0 into translations, but not rotations about arbitrary points. Is that correct?
Regarding computational complexity: to implement the generalized convolution we could do the following. For each of the n^2 output positions, we can take s^2 bilinear samples from the image (cost O(s^2)) and dot them with the filter (again, cost O(s^2)) for a total of O(n^2 s^2), just like normal convolutions. Is there any reason such an implementation is not preferred? (If so, that should be given as the reason for preferring the presented algorithm over previous work, instead of computational complexity.)
Yes, the reviewer is correct that our theoretical result only deals with a single global transformation;
however in practice it can also be used for different local transformations. In the suggested setting, one can turn rotations around a set of points into translations, by warping the neighborhoods with log-polar grids around different pivots.
Another way to combine heterogeneous transformations (e.g. translation, scale, and rotation) by means of a cascade of warps was described in the answer above (item #4).
It is true that one can perform generalized convolution by accessing 4*n^2*s^2 elements with bilinear sampling, since 4 samples are interpolated. In the case of general sampling (e.g. with Lanczos filters), many more samples may be taken. The proposed warped convolutions access 4*n^2 elements (for the warp) + n^2*s^2 elements (for the standard convolution).
So, although the asymptotic complexity is the same as naive generalized convolution, the trailing constant factor is lower (by a factor of 4 for bilinear sampling, more for general samplers). We agree with the reviewer that the current description is not entirely correct and will rephrase accordingly.
Another argument we make in the paper, and which may be more compelling, is that implementing generalized convolutions naively requires space-varying filters, performing a warp of the filter at every step. This is harder to implement, and much harder with high efficiency, when compared to standard convolutions, which have filters with constant support and predictable memory access. Our method allows having the best of both worlds (simplicity and efficiency).
From my understanding of the paper, you reparametrized an image via a 2 dimensional Lie group. I have few questions.
First I do not understand the difference between equivariance and invariance: in the introduction, you say you will introduce an invariant operator, and finally the networks are equivariant. Could you clarify?
You claim: "Projecting a signal onto such a basis still has an asymptotic complexity higher than that of a simple convolution, and the basis itself requires more storage than a convolutional filter."
I do not understand what it exactly means, for instance, in Cohen & Welling, it is tried "to keep the number of parameters approximatively fixed". Could you clarify?
Also, in their case, they do not explicitly encode the p4/p4m group in the first layer, and they don't compute either the orbits of each images. They first apply 2D-convolutions(covariant with p4/p4m), and then, the convolution becomes 3 dimensional.
So I do not understand why you write "projecting a signal", since they seem to project the filters, not the input.
Is additivity a restriction for the 2 dimensional parametrizations via Lie groups? I understand the condition of fixed point as the fact that the action of the group is transitive, am I correct? If a pivot point exists, then isn't any point a pivot(in the additive case)? Can you give examples of parametrisation where there is no pivot point?
Besides, you used only 2D structures, but could you use scales, pitches, .. all together? In your experiments, you removed the translation variabilities by choosing centered images. Do you have any ideas how to solve this problem if this was not the case? I guess the fact to stay in the 2D domain might be a limitation to keep the Theorem 2 true. Do you have any suggestions to overpass this property?
In section 5.3, you write that in order to maintain additivity, you did not consider the radius of a sphere. Do you think a mapping of the radius like exponential mapping could work or is there a limitation?
In the Google experiment, why did you downscale the images? Can you give more details about the training of the deep network: how did you avoid overfitting (maybe there wasn't?)? Do you have any intuitions on the nature of the learned features? Is the "object scale", the parameter "s" mentioned in the previous sections?
In the faces experiment, can you explain why you had to increase the support of the filters, from 5 to 9?
Thanks!
We will address the questions in order:
1. We started with the term "invariant" since we thought it will be more familiar than "equivariant" to many readers. To be absolutely precise, when it is said informally that a CNN is invariant to translations, it is meant that the architecture is the *same* (does not vary) over translations.
At the same time, its output is equivariant w.r.t. translations of the input. We avoided making this distinction for brevity, but will include a footnote to be unambiguous.
2. It is indeed the case that Cohen & Welling project the filters; "signal" was meant to convey that either the filters or images can be projected. Our argument applies equally to both cases. Perhaps spelling it as "filters or images" will avoid any ambiguity.
3. Yes, additivity is required. The group action is indeed transitive, which is an alternative way to express the condition on the pivot. We thank the reviewer for bringing up this connection; we will point it out in section 4.2.
About the pivot points, the reason why they are not arbitrary is that they exist in the input domain Omega, not the parameter space R^2. The reviewer is correct that additivity implies any parameter in R^2 is valid.
However, we cannot transport this property back to the input domain. As a specific counter-example, placing the pivot at the origin (x = 0) in the scale/rotation equation (section 5.2) yields t_u(0) = 0 for all u, breaking the bijectivity condition of t_u. Intuitively, this creates a degenerate warp grid (all its points are at 0). If Omega is restricted to R^2\{0}, however, the condition is satisfied, and a non-degenerate warp grid is created.
4. Handling multiple transformations simultaneously (>2D) is of great interest. We found it hard to do this without losing the most attractive features of the method, which are its simple implementation (bilinear sampling + 2D convolution) and speed (since it is a 2D -> 2D mapping). It is trivial to generalize Thm. 2 directly to a ND -> ND mapping, but less useful since the input dimensionality is the same as the number of transformation parameters.
For the more general cases (2D input with ND transformation), the work of Cohen & Welling seems to provide a good answer. We believe that it may be possible to embed subspaces of commutative transformations (like the ones we deal with) into larger spaces of non-commutative transformations (as they do), and exploit the same trick to gain efficiency, but doing so will require substantially more research.
From a practical point of view, without modifying Thm. 2, one can simply use a cascade of alternating subnetworks, to handle >2D parameters. Each subnetwork can use Thm. 2 to gain equivariance w.r.t. 2 parameters, and the network as a whole progressively refines a pose registration. Our inclusion of an STN was meant to achieve the same effect.
5. Yes, one could consider scaling of the sphere in the way that you describe. It is worth keeping in mind though that this adds one parameter (so the warp will be 3D -> 3D, instead of 2D).
However, if one wants to deal with 3D data (e.g. voxels), then it is a great suggestion! That would enable equivariance to yaw, pitch and scale simultaneously, in 3D.
6. We avoided overfitting by keeping the networks modest in size (3-4 layers, 20-50 filters). The downscaling on Google Earth and large filters on AFLW both increase the effective receptive fields, which compensates for the lack of depth + subsampled layers. The object scale on Google Earth is the diagonal length of the bounding box, and unrelated to the scale parameter mentioned earlier.
As for the learned features, they can be interpreted as radial filters, that will be sensitive for example to edges radiating outwards from the center, or to edges orthogonal to them (which would respond highly around the perimeter of circles of different sizes). Intuitively, such features must be equivariant to rotation and scale.
Thank you for your answers. I will use them for my final review, however I still have comments for some of them:
1. Ok. I simply noticed it for the sake of clarity. A foot note might not improve the clarity..
2.
But if they keep the number of parameters approximatively fixed, then, your claim that the representation has quite more parameters is wrong, isn't it?
3.
In fact, additivity is not required since in the case of lie group, the exponential maps always give an additive morphism. You're explicitly working with 2d Lie groups that have transitive actions.
In the case you mentioned, it comes from the fact that there are two orbits to this action: {0} dans R^2\{0}. I was simply curious if there was non trivial example that might be used for CV, but I don't see any neither
4. The argument of simplicity of computation is totally fine.I do not think there is any issue with non commutative group, you can still perform convolution along the orbits.(like rototranslation) As soon as it is compact, a haar measure does exist. And if you have a measure, you have the possibility to do convolutions.
6. I'm really curious, do you think it's possible to see the features of the first layer? I'm expecting a lot of averaging, but I might be wrong.
At the end, you claim that directions along which applying convolutions are an important element of deep networks.
1. The idea would be to use the precise term "equivariant" instead of "invariant", addressing your initial comment about "invariant operator" being confusing, and add a footnote to explain how equivariance relates to invariance for those who are not familiar. If you have a specific suggestion we will consider it of course, however we feel that such basic terminology should be defined briefly.
2. The comment is not about the number of parameters (which as you note is the same), but about the intermediate storage. If our understanding of Cohen & Welling is correct, it unpacks the learned parameters into a filter bank, by applying different transformations to it. The amount of storage is not insurmountable in most cases, so this should not be perceived as criticism; merely as a difference w.r.t. our technique. We will make it clear that the claim is about intermediate storage, on which ours provides savings if that is a concern.
3 and 4. Thank you for the suggestion! We will investigate this further and use it to supplement the section about Lie groups. Please note that our result is obtained using nothing but simple calculus, and we think that is an attractive property of our exposition, though of course it is a matter of taste. We tried to provide connections to Lie group theory for the readers that prefer that style and are more knowledgeable in that area.
6.
Of course. I uploaded a quick visualization here:
http://www.robots.ox.ac.uk/~joao/filters.png
Each of the 20 columns is one filter, while each row is an input channel (RGB). They do not differ too much from typical CNN features, though you may note the edges are slightly curvier. They are noisy due to the small size of the datasets.
This paper deals with convolution along Lie groups.
Pros:
- Good numerical results
- Interesting applications linked to convolutions on Lie groups
Cons:
- Sometimes, the writing is not clear/poor.
- The theorems 1 and 2 are in fact direct applications of Lie group theory, but this is not made explicit.
I do not detail more my review, as I believe my questions/answers complete it fairly.
However, if the authors require it, I can be more specific. I think the writing should be improved a bit.
The last item of the list requires supporting evidence. Please cite the exact result from Lie group theory that removes the novelty of our result.
Furthermore, merely showing that an isomorphism exists (as Lie group theory does) does not advance a useful construction. Knowing that the solution to a problem exists is not quite the same as finding it -- which we do.
Please also note that this is not primarily a paper about Lie groups; we only noted the connection for the interested reader.
In the case of a transitive action of a group G acting on R^2 via g.x, let be a x0, set x=g'.x0, u=g.x0 then H(g,I)=\int I(g.(g'^-1.x0)) f(g'.x0)dg'=(\hat I*\hat f)(g)  (the jacobian you used and the dx are hidden in the haar measure dg), I prefer to write your filtering as a convolution (eg with a "minus").
Theorem 1 means that f*I commutes with translation, and theorems 2 says that if one has an exponential map(e.g. an additive morphism), then if g=exp(u),g'=exp(v) u,v being in R^2, then the convolution can be written, since g'^-1=exp(-v), as an integration over R^2.
I'm surprised by your comment, since actually it gives enlightment on your work! Am I writing something wrong?
Apologies, it's simply that the expression "direct applications of Lie group theory" can be misinterpreted as meaning that the theorem is well-known or trivial, with no explanation as to why, and that prompted my request.
We agree that re-deriving the result using these definitions is indeed enlightening, by allowing the interpretation of the warps as exponential maps. We will update the paper with this information and thank you for the suggestion.
We would like to thank the reviewer for the numerous points raised in the pre-review phase, and the review itself. We attempted to incorporate all of these points in the updated version. Identifying the least clear aspects of the exposition was extremely helpful, and we believe that it resulted in a better paper.
Following the suggestion of relating the method to the Haar measure and exponential maps, we can indeed derive the abstract formalism starting from general principles.
However, it is still difficult to generalize Section 4 in the sense of showing that such more general instances can also be reduced to standard spatial convolution, beyond the cases already comprised in the paper. Note again that this is important in order to be able to use off-the-shelf convolution routines.
In any case, the relationship is made explicit in Section 4.2, and we point readers to this fact in the Introduction.
The paper suggests method to make the convolution invariant to other type of spatial transformations besides translation. It is interesting that the convolution can be invariant to different variation types by simple warping of the input image, which is related to arbitrary data perturbation in image recognition.
Experiment section needs more description and technique implementation details.
The relationship of theorems to Lie group is not defined properly.
It need more detailed comparison of method with available methods in the literature.
Thank you for your review! We have updated the paper to incorporate this feedback, we hope the changes are satisfactory.
The paper shows how group convolutions (for two dimensional commutative groups) can be performed by standard CNNs if the input is warped using a fixed warp. The idea is practical and seems to work well. The paper is well written.
I agree with reviewer 1 that the ‘theorems’ do not deserve to be labelled as such. Theorem 1 is equivalent to the second equation from this section of the wikipedia page on convolution: https://en.wikipedia.org/wiki/Convolution#Convolutions_on_groups. The existence of Haar measure (from which commutation of Lg and convolution follows immediately) is a well known and elementary theorem in harmonic analysis, which would be treated in the first few pages of any textbook on the subject. It also immediately clear that any commutative group has an additive parameterization (theorem 2). The fact that the paper does not present new deep mathematical results is not a significant weakness in my opinion, but the derivations should not be camouflaged as such.
The claim that previous methods that use group convolutions are slow and that the presented approach has better computational complexity is not supported by empirical evidence, and the theoretical analysis is still a bit misleading.
For example, the authors write “Unfortunately these approaches do not possess the same memory and speed benefits that CNNs enjoy. The reason is that, ultimately, they have to enumerate all possible transformations”. The presented method also has to enumerate all transformations (in a limited range, on a discretized grid), and this is feasible only because the group is only 2 dimensional (and indeed this is also true for standard CNNs which enumerate translations).
As noted in my pre-review question, I believe the computational complexity analysis is not entirely correct, and assume the authors will correct this.
Equation 4 is presented as a new invention, but this has been used in previous works and is well known in mathematics, so a citation should be added.
The main advantage of the presented method over several earlier methods is that it is very simple to implement, and can re-use highly optimized convolution routines. As I understand it, Dieleman et al. and Cohen & Welling also use standard convolutions (after a fixed filter / feature map warp), but these papers only consider discrete groups.
So it seems like this paper occupies a unique place in the space of equivariant convolutional networks: non-commutative (-), low-dimensional (-), continuous (+) groups, simple (+) and efficient (+) algorithm. Given the proximity though, a more thorough and balanced appraisal of the merits and demerits, as well as the novelty, relative to each of the previous works would be useful.
Provided that these issues are cleared up, I would recommend the paper for publication.
We have updated the paper to make it clear that Section 2 does not present new material, and Section 4 does. We do not claim that e.g. generalized convolution or its equivariance are new concepts; they are well-known, and in fact we used them in our previous works. We added the proper references to make this clear.
Likewise, that any commutative group has additive parameterization is a known fact. Theorem 2 does not prove this: an additive parameterization is one of its conditions (so it applies to commutative groups). What Theorem 2 does, in fact, is to present a case of the general theory (see answer to R1) where calculations can be carried in a simple form which can be reduced to standard convolution operators and hence can be readily implemented numerically.
Finally, we thank the reviewer for helping us place our algorithm w.r.t. the closest state-of-the-art proposals, and clarifying the complexity analysis. We maintain that there are two computational advantages of this approach. One is very practical, as reducing generalized convolution to a standard one allows to directly use highly optimized routine. The second one is a little deeper as this reduction allows to also use methods such as the Fast Fourier Transform, that have indeed a lower order of complexity and that would be much more challenging to implement in full generality. We have updated the discussion to reflect that.
Hi,
I do not see the updated version of the paper in the revision's history. I wish I could read the revision, so that I could change my review. I hope that you made explicit the connexion with group theory, and as said by reviewer 1, that nothing is "camouflaged".
"where calculations can be carried in a simple form" -> Do you have an example where this is not trivial?
Yes, sorry about that -- I used "Edit" instead of "Add Revision", which apparently only changes the main page's PDF. It is visible in both places now.
We tried to make the group theory view as explicit as possible, while also keeping the simple derivation for a broad audience. The expanded section 4.2 is dedicated to this. We also tried to make it very explicit in the introduction, as follows:
"This makes generalized convolution easily implementable in neural networks, including using fast convolution algorithms on GPU hardware, such as Winograd (Lavin, 2015) or the Fast Fourier Transform (Lyons, 2010). We present these notions in the simplest possible way (sections 2 to 4), but we note that they can be derived in broader generality from well know concepts of group theory (section 4.2)."

Meta-review:
role: PC
data: comment: The reviewers found the idea interesting and practical but had concerns about the novelty of the approach and the claims and theory presented in the paper. In particular, it seems that the reviewers feel that the authors' claim to present novel theory is unjustified (i.e. the theorems presented are not novel). Also claims regarding the advantages of this approach relative to related literature (in particular Dieleman et al. and Cohen & Welling) seem to be unsubstantiated. This work would likely be more well received just through rewriting the manuscript with, as stated by AnonReviewer3, "a more thorough and balanced appraisal of the merits and demerits, as well as the novelty, relative to each of the previous works" and weakening the claims of novel theory.

Decision: Reject