Title: WARPED CONVOLUTIONS: EFFICIENT INVARIANCE TO SPATIAL TRANSFORMATIONS

Abstract: Convolutional Neural Networks (CNNs) are extremely efficient, since they exploit the inherent translation-invariance of natural images. However, translation is just one of a myriad of useful spatial transformations. Can the same efficiency be attained when considering other spatial invariances? Such generalized convolutions have been considered in the past, but at a high computational cost. We present a construction that is simple and exact, yet has the same computational complexity that standard convolutions enjoy. It consists of a constant image warp followed by a simple convolution, which are standard blocks in deep learning toolboxes. With a carefully crafted warp, the resulting architecture can be made equivariant to a wide range of 2-parameters spatial transformations. We show encouraging results in realistic scenarios, including the estimation of vehicle poses in the Google Earth dataset (rotation and scale), and face poses in Annotated Facial Landmarks in the Wild (3D rotations under perspective).

1. INTRODUCTION:

A crucial aspect of current deep learning architectures is the encoding of invariances. This fact is epitomized in the success of convolutional neural networks (CNN) , where equivariance to image translation is key: translating the input results in a translated output. When invariances are present in the data, encoding them explicitly in an architecture provides an important source of regularization, which allows to reduce the amount of training data required for learning. Invariances may also be used to improve the efficiency of implementations; for instance, a convolutional layer requires orders of magnitude less memory and also less computation compared to an equivalent fully-connected layer.

The success of CNNs indicates that translation invariance is an important property of images. However, this does not explain why translation equivariant operators work well for image understanding. The common interpretation is that such operators are matched to the statistics of natural images, which are well known to be translation invariant (Hyvärinen et al., 2009) . However, natural image statistics are also (largely) invariant to other transformations such as isotropic scaling and rotation, which suggests that alternative neural network designs may also work well with images. Furthermore, in specific applications, invariances other than translation may be more appropriate.

Therefore, it is natural to consider generalizing convolutional architectures to other image transformations, and this has been the subject of extensive study (Kanazawa et al., 2014; Bruna et al., 2013; Cohen & Welling, 2016) . Unfortunately these approaches do not possess the same memory and speed benefits that CNNs enjoy. The reason is that, ultimately, they have to transform (warp) an image or filter several times (Kanazawa et al., 2014; Marcos et al., 2016; Dieleman et al., 2015) , incurring a high computational burden. Another approach is to consider a basis of filters (analogous to eigen-images) encoding the desired invariance (Cohen & Welling, 2014; Bruna et al., 2013; Cohen & Welling, 2016) , which requires more storage than a convolutional filter.

Although they are able to handle transformations with many pose parameters, in practice most recent proposals are limited to very coarsely discretized transformations, such as horizontal/vertical flips and 90 • rotations (Dieleman et al., 2015; Cohen & Welling, 2014) .

In this work we propose a generalization of CNNs that overcomes these disadvantages. Our main result shows that a linear layer with equivariance w.r.t. a large class of 2-parameters transformations can always be implemented efficiently, using a standard convolution in a warped image space. The image warp can be implemented using bilinear resampling, a simple and fast operation that has been popularized by spatial transformer networks (Jaderberg et al., 2015) , and is part of most deep learning toolboxes. Unlike previous proposals, the proposed warped convolutions can handle continuous transformations, such as fine rotation and scaling. This makes generalized convolution easily implementable in neural networks, including using fast convolution algorithms on GPU hardware, such as Winograd (Lavin, 2015) or the Fast Fourier Transform (Lyons, 2010) . We present these notions in the simplest possible way (sections 2 to 4), but we note that they can be derived in broader generality from well know concepts of group theory (section 4.2).

2. GENERALIZING CONVOLUTION:



2.1. CONVOLUTIONS OF CONTINUOUS IMAGES:

We start by looking at the basic building block of CNNs, i.e. the convolution operator. This operator computes the inner product of an image I ∈ R m×n with a translated version of the filter F ∈ R r×s , producing a new image as output:

EQUATION

where k, j ∈ Z 2 are two-dimensional vectors of indexes, and the summation ranges inside the extents of both arrays. 1 To handle continuous deformations of the input, it is more natural to express eq. 1 as an integral over continuous rather than discrete inputs:

EQUATION

where I(x) and F (x) are continuous functions over a bounded 2D region Ω ⊂ R 2 , that is: I, F : Ω → R. The real-valued 2D vectors x ∈ Ω now play the role of the indexes k ∈ Z 2 . Equation 2 reduces to the discrete case of eq. 1 if we define I(x) and F (x) as the sum of delta functions on grids. Intermediate values can be obtained by interpolation, such as bilinear (which amounts to convolution of the delta functions with a triangle filter (Jaderberg et al., 2015) ). Importantly, such continuous images can be deformed by very rich continuous transformations of the input coordinates, whereas strictly discrete operations would be more limiting.

Over the next sections it will be more convenient to translate the image I instead of the filter F . This alternative form of eq. 2 is obtained by replacing x + u → x:

H(u; I) = ˆI(x -u) F (x) dx.

(3)

2.2. BEYOND IMAGE TRANSLATIONS:

The standard convolution operator of eq. 3 can be interpreted as applying the filter to translated versions of the image. Translations can be replaced by other transformations as follows (Henriques et al., 2014) :

H(t; I) = ˆI(t(x)) F (x) dx, t ∈ G (4)

where G is a set of transformation functions t : Ω → Ω (assumed to be invertible). Intuitively, this generalized convolution performs an exhaustive search for a pattern, at many different poses (Henriques et al., 2014; Kanazawa et al., 2014) . The interest in this definition lies in the fact that it makes convolution equivariant (Lenc & Vedaldi, 2015) :

Lemma 1 (Equivariance). Consider the generalized convolution operator H(t; I) of eq. 4. Generalized convolution "commutes" with any transformation q ∈ G of the image:

H(t; I • q) = H(q • t; I).

Proof. One has immediately H(t; I • q) = ´I(q(t(x))) F (x) dx = H(q • t; I).

A notable case is when transformations have an additive parametrization t : Ω × R 2 → Ω, with (x, u) → t u (x) and t u • t v = t u+v . In this case, the equivariance relation can be written as

EQUATION

In particular, standard convolution is obtained when t u (x) = x -u is the translation operator. In this case, the lemma above simply states that any translation of the input of the convolution results in a corresponding translation of the output.

In section 5, we will look in more detail at a few concrete examples of transformations other than translations. Although we will not do so explicitly, in this construction it is also possible to let one or more dimensions of the parameter space R 2 be given modulus a period Q, in the sense of replacing R with R/Z(Q); the latter is required to parameterise transformations such as rotation.

3. COMPUTATIONAL EFFICIENCY:

Unfortunately, what eq. 4 gains us in generality, it loses in both performance and ease of implementation. Most works in computer vision that looked at filtering under generalized transformations (e.g. scale pyramids (Kanazawa et al., 2014) or rotated filter banks (Marcos et al., 2016; Cohen & Welling, 2014; 2016; Henriques et al., 2014) ) compute eq. 4 directly by evaluating a large number of transformations t ∈ G. This entails warping (transforming) either the image or the filter once per transformation t, which can be expensive.

Opting to transform the filter instead of the image can be advantageous, since it is smaller in size. On the other hand, the filter and its domain then become spatially-varying, which foregoes the benefit of the regular, predictable, and local pattern of computations in standard convolution. It precludes the use of fast convolution routines such as Winograd's algorithm (Lavin, 2015) , or the Fast Fourier Transform (Lyons, 2010), which has lower computational complexity than exhaustive search (eq. 3).

In practice, most recent works focus on very coarse transformations that do not change the filter support and can be implemented strictly via permutations, like horizontal/vertical flips and 90 • rotations (Dieleman et al., 2015; Cohen & Welling, 2014) . Such difficulties explain why generalized convolutions are not as widespread as CNNs.

In section 4 we will show that, for an important class of transformations, including the ones considered in previous works (such as Kanazawa et al. (2014) ; Cohen & Welling (2014) ; Marcos et al. (2016) ) it is possible to perform generalized convolution by composing a single warp with a standard convolution, instead of several warps. Thus, we are able to take full advantage of modern convolution implementations (Lavin, 2015; Lyons, 2010) , including those with lower computational complexity.

4. MAIN RESULT:

Our main contribution is to show that the generalized convolution operator of eq. 4 can be implemented efficiently by a standard convolution, by pre-warping the input image and filter appropriately.

The warp is the same for any image, depending solely on the nature of the relevant transformations, and can be written in closed form. This result, given in theorem 1, allows us to implement very efficient generalized convolutions using simple computational blocks, as shown in section 4.1. We name this method warped convolution.

The strongest assumption is that transformations must have an additive parametrization. By this, we mean that there exists a bijection t u : Ω → Ω such that, for any u, v ∈ R 2 , parameters compose additively t u • t v = t u+v . The second assumption is that there exists a pivot point x 0 ∈ Ω such that u → t u (x 0 ) defines a bijection R 2 → Ω from the parameter space to the real plane. The latter requirements means that any point x ∈ Ω can be "reached" by transforming x 0 under a suitable t u .

We then have that: Theorem 1. Consider the generalized convolution of eq. 4. Assume that the transformation is additive (t u • t v = t u+v ). Assume also that, for a fixed pivot point x 0 , the function u → t u (x 0 ) is bijective. Then we can rewrite generalized convolution (eq. 4) as the standard convolution

EQUATION

where Î and F are the warped image and filter given by:

EQUATION

Proof. We use the variable substitution x = t v (x 0 ) in eq. 4. Then:

H(u; I) = ˆI(t u (x)) F (x) dx = ˆI(t u (t v (x 0 ))) F (t v (x 0 )) dt v (x 0 ) dv dv = ˆI(t u+v (x 0 )) Î(u+v) F (t v (x 0 )) dt v (x 0 ) dv F (v) dv = ˆÎ(u + v) F (v) dv.

The warp that is applied to both inputs in eq. 7 can be interpreted as follows. We start with an arbitrary pivot point x 0 in the image and them sample other points by repeatedly applying the transformation t u (x 0 ) to the pivot (by varying u). When discretized, this sampling is performed over a 2D grid of parameters u. Finally, sampling the input at these points (for example, by bilinear interpolation) yields the warped input.

An illustration is given in fig. 1 , for various transformations (each one is discussed in more detail in section 5). The red dot shows the pivot point x 0 , and the two arrows pointing away from it show the two directions of increasing u values (recall that transformation parameters are two-dimensional). The grids were generated by sampling u at regular intervals. Note that the warp grids are independent of the image contents -they can be computed once offline and then applied to any image.

The last factor in eq. 7 is the determinant of the Jacobian of the image transformation t. It rescales the image values to account for the stretching and shrinking of space due to non-linear warps. It can also be computed offline, and its application amounts to an element-wise product by a constant array. A generalization using group theory is discussed in section 4.2.

4.1. PRACTICAL CONSIDERATIONS:

There are a few interesting aspects that simplify the use of theorem 1 in practice.

First, since in most applications the filter F is learned, we are free to ignore the constant warp and Jacobian in eq. 7 (which amounts to a simple reparametrization), and learn F directly. In practice, this means that we warp only the input image I to obtain Î, and then perform a standard convolution with a filter F . The learned warped filter F has a one-to-one correspondence to an image-space filter F by means of eq. 7, although there is no real need to build the latter explicitly.

Second, we can choose either one or two spatial transformations for the generalized convolution (e.g. scale and rotation, simultaneously). The reason is that the input image is 2D, so the parameterspace after warping is also 2D. The choice is not arbitrary though: the two transformations must commute, in order to respect additivity. This will be the case of the pairs we study in section 5.

Algorithm 1 Warped convolution.

Grid generation (offline):

• Apply the spatial transformation t repeatedly to a pivot point x 0 , using a 2D grid of parameters u = {(u 1 + iδ 1 , u 2 + jδ 2 ) : i = 0, . . . , m, j = 0, . . . , n}, obtaining the 2D warp grid t u (x 0 ).

Warped convolution:

1. Resample input image I using the warp grid t u (x 0 ), by bilinear interpolation.

2. Convolve the warped image Î with filter F .

By theorem 1, these steps are equivalent to a generalized convolution, which performs an exhaustive search across the pose-space of transformation t, but at a much lower computational cost.

4.2. RELATIONSHIP TO GROUP THEORY:

This section relates our results, which have been presented using a simple formalism and in a restricted setting, to a more general approach based on group theory (Folland, 1995) .

To this end, let G be a group of transformations. Under very mild conditions (the group has to be locally compact and Hausdorff), there exists a unique measure on the group, the Haar measure, which is invariant to the group action, in the sense that, given a measurable function Ĩ : G → R, then ´Ĩ(g g) dg = ´Ĩ(g) dg.

Using this measure, one can define generalized convolution as

( Ĩ * F )(t) = ´G Ĩ(tg) F (g -1 )

dg. This resembles our definition (4), although image and filter are defined on the group G instead of the spatial domain R 2 . Lemma 1 translates immediately to this case (Folland, 1995) .

In order to extend Theorem 1, we need to make this general but abstract construction concrete. Here one assumes that the group acts transitively on a subset X ⊂ R 2 (which means that any point x ∈ X can be written as x = gx 0 , for a fixed point x 0 ∈ X and a suitable transformation g ∈ G). Then one can define the image as Ĩ(g) = I(g(x 0 )), where I is a function of the spatial domain X instead of the group G, and likewise for the filter. Next, it is necessary to explicitly calculate the integral over G. If the group is an Abelian (commutative) Lie group, then one can show that there exists a map exp : V → G, the exponential map, defined on a vector space V . Under commutativity, this map is also additive, in the sense that exp(u) exp(v) = exp(u + v). The structure of V depends on the specific group, but under such restrictive conditions, it is a torus, which allows the calculation of ´Ĩ(g) dg as ´Ĩ(exp(u)x 0 ) du.

Finally, in order to swap integration over the group parameters with integration over space, one assumes that x = exp(u)x 0 defines a smooth bijection V → X, so that it is possible to use the change of variable u → u(x) where exp(u(x))x 0 = x. This allows writing the integral as ´Ĩ(exp(u)x 0 ) du = ´I(x) |du/dx| dx. Note that this Jacobian is the inverse of the one found in

(1) due to the fact that we started by defining our convolution using I instead of Ĩ.

5. EXAMPLES OF SPATIAL TRANSFORMATIONS:

We now give some concrete examples of pairs of spatial transformations that obey the conditions of theorem 1, and can be useful in practice.

5.1. SCALE AND ASPECT RATIO:

Detection tasks require predicting the extent of an object as a bounding box. While the location can be found accurately by a standard CNN, which is equivariant to translation, the size prediction could similarly benefit from equivariance to horizontal and vertical scale (equivalently, scale and aspect ratio).

Such a spatial transformation, from which a warp can be constructed, is given by: 

EQUATION

The s constant controls the total degree of scaling applied. Notice that the output must be exponential in the scale parameters u; this ensures the additive structure required by theorem 1: t u (t v (x)) = t u+v (x). The resulting warp grid can be visualized in fig. 1-b . In this case, the domain of the image must be Ω ∈ R 2 + , since a pivot x 0 in one quadrant cannot reach another quadrant by any amount of (positive) scaling.

5.2. SCALE AND ROTATION (LOG-POLAR WARP):

Planar scale and rotation are perhaps the most obvious spatial transformations in images, and are a natural test case for works on spatial transformations (Kanazawa et al., 2014; Marcos et al., 2016) . Rotating a point x by u 1 radians and scaling it by u 2 , around the origin, can be performed with

EQUATION

)

where atan 2 is the standard 4-quadrant inverse tangent function (atan2). The domain in this case must exclude the origin (Ω ∈ R 2 \ {0}), since a pivot x 0 = 0 cannot reach any other points in the image by rotation or scaling. ). With an appropriate warp and a standard CNN, the shaded block becomes equivalent to a generalized CNN (by theorem 1), which performs exhaustive searches across pose-space instead of image-space.

The resulting warp grid can be visualized in fig. 1-c . It is interesting to observe that it corresponds exactly to the log-polar domain, which is used in the signal processing literature to perform correlation across scale and rotation (Tzimiropoulos et al., 2010; Reddy & Chatterji, 1996) . In fact, it was the source of inspiration for this work, which can be seen as a generalization of the log-polar domain to other spatial transformations.

5.3. 3D SPHERE ROTATION UNDER PERSPECTIVE:

We will now tackle a more difficult spatial transformation, in an attempt to demonstrate the generality of theorem 1. The transformations we will consider are yaw and pitch rotations in 3D space, as seen by a perspective camera. In the experiments (section 6) we will show how to apply it to face pose estimation.

In order to maintain additivity, the rotated 3D points must remain on the surface of a sphere. We consider a simplified camera and world model, whose only hyperparameters are a focal length f , the radius of a sphere r, and its distance from the camera center d. The equations for the spatial transformation corresponding to yaw and pitch rotation under this model are in appendix A.

The corresponding warp grid can be seen in fig. 1-d . It can be observed that the grid corresponds to what we would expect of a 3D rendering of a sphere with a discrete mesh. An intuitive picture of the effect of the warp grid in such cases is that it wraps the 2D image around the surface of the 3D object, so that translation in the warped space corresponds to moving between vertexes of the 3D geometry.

6. EXPERIMENTS:

6.1 ARCHITECTURE As mentioned in section 2.2, generalized convolution performs an exhaustive search for patterns across spatial transformations, by varying pose parameters. For tasks where invariance to that transformation is important, it is usual to pool the detection responses across all poses (Marcos et al., 2016; Kanazawa et al., 2014) .

In the experiments, however, we will test the framework in pose prediction tasks. As such, we do not want to pool the detection responses (e.g. with a max operation) but rather find the pose with the strongest response (i.e., an argmax operation). To perform this operation in a differentiable manner, we implement a soft argmax operation, defined as follows:

EQUATION

where σ(a) ∈ R m×n is the softmax over all spatial locations, and σ ij (a) indexes the element at (i, j). The outputs are the two spatial coordinates of the maximum value, s(a) ∈ R 2 .

Our base architecture then consists of the following blocks, outlined in fig. 2 . First, the input image is warped with a pre-generated grid, according to section 4. The warped image is then processed by a standard CNN, which is now equivariant to the spatial transformation that was used to generate the warp grid. A soft argmax (eq. 10) then finds the maximum over pose-space. To ensure the pose prediction is well registered to the reference coordinate system, a learnable scale and bias are applied to the outputs. Training proceeds by minimizing the L 1 loss between the predicted pose and ground truth pose.

6.2. GOOGLE EARTH:

For the first task in our experiments, we will consider aerial photos of vehicles, which have been used in several works that deal with rotation invariance (Liu et al., 2014; Schmidt & Roth, 2012; Henriques et al., 2014) .

Dataset. The Google Earth dataset (Heitz & Koller, 2008) contains bounding box annotations, supplemented with angle annotations from (Henriques et al., 2014) , for 697 vehicles in 15 large images. We use the first 10 for training and the rest for validation. Going beyond these previous works, we focus on the estimation of both rotation and scale parameters. The object scale is taken to be the diagonal length of the bounding box.

Implementation. A 48×48 image around each vehicle is cropped and downscaled by 50%, and then fed to a network for pose prediction. The proposed method, Warped CNN, follows the architecture of section 6.1 (visualized in fig. 2 ). The CNN block contains 3 convolutional layers with 5 × 5 filters, with 20, 50 and 1 output channels respectively. Recall that the output of the CNN block is a single-channel response map over 2D pose-space, which in this case consists of rotation and scale. Between the convolutional layers there are 3 × 3 max-pooling operators, with a stride of 2, and a ReLU before the last layer. All networks are trained for 20 epochs with SGD, using hyperparameters chosen by cross-validation.

Baselines and results. The results of the experiments are presented in table 1, which shows angular and scale error in the validation set. Qualitative results are shown in fig. 3 . To verify whether the proposed warped convolution is indeed responsible for a boost in performance, rather than other architectural details, we compare it against a number of baselines with different components removed. The first baseline, CNN+softargmax, consists of the same architecture but without the warp (section 5.2). This is a standard CNN, with the soft argmax at the end. Since CNNs are equivariant to translation, rather than scale and rotation, we observe a drop in performance. For the second baseline, CNN+FC, we replace the soft argmax with a fully-connected layer, to allow a prediction that is not equivariant with translation. The FC layer improves the angular error, but not the scale error. The proposed Warped CNN has a similar (slightly lower) capacity to the CNN+FC baseline, but we see it achieve better performance, since its architectural equivariance seems to be better matched to the data distribution. 

6.3. FACES:

We now turn to face pose estimation in unconstrained photos, which requires handling more complex 3D rotations under perspective.

Dataset. For this task we use the Annotated Facial Landmarks in the Wild (AFLW) dataset (Koestinger et al., 2011) . It contains about 25K faces found in Flickr photos, and includes yaw (left-right) and pitch (up-down) annotations. We removed 933 faces with yaw larger than 90 degrees (i.e., facing away from the camera), resulting in a set of 24,384 samples. 20% of the faces were set aside for validation.

Implementation. The region in each face's bounding box is resized to a 64 × 64 image, which is then processed by the network. Recall that our simplified 3D model of yaw and pitch rotation (section 5.3) assumes a spherical geometry. Although a person's head roughly follows a spherical shape, the sample images are centered around the face, not the head. As such, we use an affine Spatial Transformer Network (STN) (Jaderberg et al., 2015) as a first step, to center the image correctly. Similarly, because the optimal camera parameters (f , r and d) are difficult to set by hand, we let the network learn them, by computing their derivatives numerically (which has a low overhead, since they are scalars). The rest of the network follows the same diagram as before (fig. 2 ).

The main CNN has 4 convolutional layers, the first two with 5 × 5 filters, the others being 9 × 9. The numbers of output channels are 20, 50, 20 and 1, respectively. A 3 × 3 max-pooling with a stride of 2 is performed after the first layer, and there are ReLU non-linearities between the others. As for the STN, it has 3 convolutional layers (5 × 5), with 20, 50 and 6 output channels respectively, and 3 × 3 max-pooling (stride 2) between them.

Baselines and results. The angular error of the proposed equivariant pose estimation, Warped CNN, is shown in table 2, along with a number of baselines. Qualitative results are shown in fig. 4 . The goal of these experiments is to demonstrate that it is possible to achieve equivariance to complex 3D rotations. We also wish to disentangle the performance benefits of the warped convolution from the other architectural aspects. The first baseline, STN+softargmax, is the same as the proposed method, but without the warp. The large performance drop indicates that the spherical model incorporates important domain knowledge, which is ignored by a translation-equivariant STN. To allow nonequivariant models, we also test two other baselines where the softargmax is replaced with a fullyconnected (FC) layer. The STN+FC includes an affine Spatial Transformer, while the CNN+FC does not, corresponding to a standard CNN of equivalent capacity. We observe that neither the FC or the STN components can account up for the performance of the warped convolution, which better exploits the natural 3D rotation equivariance of the data.

7. CONCLUSIONS:

In this work we show that it is possible to reuse highly optimized convolutional blocks, which are equivariant to image translation, and coax them to exhibit equivariance to other operators, including 3D transformations. This is achieved by a simple warp of the input image, implemented with off-the-shelf components of deep networks, and can be used for image recognition tasks involving a large range of image transformations. Compared to other works, warped convolutions are simpler, relying on highly optimized convolution routines, and can flexibly handle many types of continuous transformations. Studying generalizations that support more than two parameters seems like a fruitful direction for future work. In addition to the practical aspects, our analysis offers some insights into the fundamental relationships between arbitrary image transformations and convolutional architectures. 

:

Note that eq. 1 defines cross-correlation in the signal processing literature, but here we follow the convention used in machine learning and call it convolution. We also ignore the possibility that the input image has more than one channel, and that convolution layers in CNN involve banks of filters instead of single ones. All such details are immaterial to our discussion.

Figures and Tables:

Figure 1: First row: Sampling grids that define the warps associated with different spatial transformations. Second row: An example image (a) after warping with each grid (b-d). Third row: A small translation is applied to each warped image, which is then mapped back to the original space (by an inverse warp). Translation in one axis of the appropriate warped space is equivalent to (b) horizontal scaling; (c) planar rotation; (d) 3D rotation around the vertical axis.

Figure2: Equivariant pose estimation strategy used in the experiments (section 6). With an appropriate warp and a standard CNN, the shaded block becomes equivalent to a generalized CNN (by theorem 1), which performs exhaustive searches across pose-space instead of image-space.

Figure 3: Example pose estimates (rotation and scale) on the Google Earth dataset (Section 6.2).

Figure 4: Example pose estimates (yaw and pitch) on the AFLW dataset (Section 6.3).

Results of scale and rotation pose estimation of vehicles in the Google Earth dataset.

Table content: <table><tr><td/><td colspan="3">CNN+FC CNN+softargmax Warped CNN</td></tr><tr><td>Rotation error (degrees)</td><td>28.87</td><td>30.6</td><td>26.44</td></tr><tr><td>Scale error (px)</td><td>17.51</td><td>5.783</td><td>5.4</td></tr></table>

Results of yaw and pitch pose estimation of faces on the AFLW dataset.

Table content: <table><tr><td/><td colspan="4">CNN+FC STN+FC STN+softargmax Warped CNN</td></tr><tr><td>Yaw err. (deg.)</td><td>13.87</td><td>16.92</td><td>15.01</td><td>10.65</td></tr><tr><td>Pitch err. (deg.)</td><td>7.23</td><td>10.17</td><td>6.88</td><td>6.351</td></tr></table>

Peer review (iclr2017):
The paper shows how group convolutions (for two dimensional commutative groups) can be performed by standard CNNs if the input is warped using a fixed warp. The idea is practical and seems to work well. The paper is well written. I agree with reviewer 1 that the ‘theorems’ do not deserve to be labelled as such. Theorem 1 is equivalent to the second equation from this section of the wikipedia page on convolution: https://en.wikipedia.org/wiki/Convolution#Convolutions_on_groups. The existence of Haar measure (from which commutation of Lg and convolution follows immediately) is a well known and elementary theorem in harmonic analysis, which would be treated in the first few pages of any textbook on the subject. It also immediately clear that any commutative group has an additive parameterization (theorem 2). The fact that the paper does not present new deep mathematical results is not a significant weakness in my opinion, but the derivations should not be camouflaged as such. The claim that previous methods that use group convolutions are slow and that the presented approach has better computational complexity is not supported by empirical evidence, and the theoretical analysis is still a bit misleading. For example, the authors write “Unfortunately these approaches do not possess the same memory and speed benefits that CNNs enjoy. The reason is that, ultimately, they have to enumerate all possible transformations”. The presented method also has to enumerate all transformations (in a limited range, on a discretized grid), and this is feasible only because the group is only 2 dimensional (and indeed this is also true for standard CNNs which enumerate translations). As noted in my pre-review question, I believe the computational complexity analysis is not entirely correct, and assume the authors will correct this. Equation 4 is presented as a new invention, but this has been used in previous works and is well known in mathematics, so a citation should be added. The main advantage of the presented method over several earlier methods is that it is very simple to implement, and can re-use highly optimized convolution routines. As I understand it, Dieleman et al. and Cohen & Welling also use standard convolutions (after a fixed filter / feature map warp), but these papers only consider discrete groups. So it seems like this paper occupies a unique place in the space of equivariant convolutional networks: non-commutative (-), low-dimensional (-), continuous (+) groups, simple (+) and efficient (+) algorithm. Given the proximity though, a more thorough and balanced appraisal of the merits and demerits, as well as the novelty, relative to each of the previous works would be useful. Provided that these issues are cleared up, I would recommend the paper for publication.