Title: ROTDCF: DECOMPOSITION OF CONVOLUTIONAL FIL-TERS FOR ROTATION-EQUIVARIANT DEEP NETWORKS

Abstract: Explicit encoding of group actions in deep features makes it possible for convolutional neural networks (CNNs) to handle global deformations of images, which is critical to success in many vision tasks. This paper proposes to decompose the convolutional filters over joint steerable bases across the space and the group geometry simultaneously, namely a rotation-equivariant CNN with decomposed convolutional filters (RotDCF). This decomposition facilitates computing the joint convolution, which is proved to be necessary for the group equivariance. It significantly reduces the model size and computational complexity while preserving performance, and truncation of the bases expansion serves implicitly to regularize the filters. On datasets involving in-plane and out-of-plane object rotations, Rot-DCF deep features demonstrate greater robustness and interpretability than regular CNNs. The stability of the equivariant representation to input variations is also proved theoretically. The RotDCF framework can be extended to groups other than rotations, providing a general approach which achieves both group equivariance and representation stability at a reduced model size.

1. INTRODUCTION:

While deep convolutional neural networks (CNN) have been widely used in computer vision and image processing applications, they are not designed to handle large group actions like rotations, which degrade the performance of CNN in many tasks (Cheng et al., 2016; Hallman & Fowlkes, 2015; Jaderberg et al., 2015b; Laptev et al., 2016; Maninis et al., 2016) . The regular convolutional layer is equivariant to input translations, but not other group actions. An indirect way to encode group information into the deep representation is to conduct generalized convolutions across the group as well, as in Cohen & Welling (2016a) . In theory, this approach can guarantee the group equivariance of the learned representations, which provides better interpretability and regularity as well as the capability of estimating the group action in localization, boundary detection, etc. For the important case of 2D rotations, group-equivariant CNNs have been constructed in several recent works, e.g., (Weiler et al., 2017 ), Harmonic Net (Worrall et al., 2017) and Oriented Response Net (Zhou et al., 2017) . In such networks, the layer-wise output has an extra index representing the group element (c.f. Table 1 ), and consequently, the convolution must be across the space and the group jointly (proved in Section 3.1). This typically incurs a significant increase in the number of parameters and computational load, even with the adoption of steerable filters (Freeman et al., 1991; Weiler et al., 2017; Worrall et al., 2017) . In parallel, low-rank factorized filters have been proposed for sparse coding as well as the compression and regularization of deep networks. In particular, Qiu et al. (2018) showed that decomposing filters under non-adaptive bases can be an effective way to reduce the model size of CNNs without sacrificing performance. However, these approaches do not directly apply to be group-equivariant. We review these connections in more detail in Section 1.1. This paper proposes a truncated bases decomposition of the filters in group-equivariant CNNs, which we call the rotation-equivariant CNN with decomposed convolutional filters (RotDCF). Since we need a joint convolution over R 2 and SO(2), the bases are also joint across the two geometrical domains, c.f. Figure 1 . The benefits of bases decomposition are three-fold: (1) Reduction of the number of parameters and computational complexity of rotation-equivariant CNNs, c.f. Section 2.3; Figure 1 : Decomposition of the convolutional filter across the 2D space (variable u) and the SO(2) rotation group geometry (variable α) simultaneously. The filter is represented as a truncated expansion under the prefixed bases ψ k (u)ϕm(α) with adaptive coefficients a k,m learned from data. ψ k are Fourier-Bessel bases, ϕm are Fourier bases, the first 3 of each are shown. The filter has N θ group-indexed channels (indexed by α) and only one input and output unstructured channel (indexed by λ and λ respectively) for simplicity. c.f. Table 1 .

(2) Implicit regularization of the convolutional filters, leading to improved robustness of the learned deep representation shown experimentally in Section 4; (3) Theoretical guarantees on stability of the equivariant representation to input deformations, which follow from a generic condition on the filters in the decomposed form, c.f. Section 3.2.

1.1. RELATED WORK:

Learning with factorized filters. In the sparse coding literature, Rubinstein et al. (2010) proposed the factorization of learned dictionaries under another prefixed dictionary. Separable filters were used in Rigamonti et al. (2013) to learn the coding of images. Papyan et al. (2017) interpreted CNN as an iterated convolutional sparse coding machine, and in this view, the factorized filters should correspond to a "dictionary of the dictionary" as in Rubinstein et al. (2010) . In the deep learning literature, low-rank factorization of convolutional filters has been previously used to remove redundancy in trained CNNs (Denton et al., 2014; Jaderberg et al., 2014) . The compression of deep networks has also been studied in Chen et al. (2015) ; Han et al. (2016; 2015) , SqueezeNet (Iandola et al., 2016) , etc., where the low-rank factorization of filters can be utilized. MobileNets (Howard et al., 2017) used depth-wise separable convolutions to obtain significant compression. Tensor decomposition of convolutional layers was used in Lebedev et al. (2014) for CPU speedup. Tai et al. (2015) proposed low-rank-regularized filters and obtained improved classification accuracy with reduced computation. Qiu et al. (2018) studied decomposed-filter CNN with prefixed bases and trainable expansion coefficients, showing that the truncated bases decomposition incurs almost no decrease in classification accuracy while significantly reducing the model size and improving the robustness of the deep features. None of the above networks are group equivariant.

Group-equivariant deep networks. The encoding of group information into network representations has been studied extensively. Among earlier works, transforming auto-encoders (Hinton et al., 2011) used a non-convolutional network to learn group-invariant features and compared with handcrafted ones. Rotation-invariant descriptors were studied in Schmidt & Roth (2012b) with product models, and in Jaderberg et al. (2015a) ; Kivinen & Williams (2011); Schmidt & Roth (2012a) by estimating the specific image transformation. Gonzalez et al. (2016) ; Wu et al. (2015) proposed rotating conventional filters to perform rotation-invariant texture and image classification. The joint convolution across space and rotation has been studied in the scattering transform (Oyallon & Mallat, 2015; Sifre & Mallat, 2013) . Group-equivariant CNN was considered by Cohen & Welling (2016a) , which handled several finite small-order discrete groups on the input image. Rotation-equivariant CNN was later developed in Weiler et al. (2017); Worrall et al. (2017) ; Zhou et al. (2017) and elsewhere. In particular, steerable filters were used in Cohen & Welling (2016b) ; Weiler et al. (2017); Worrall et al. (2017) . SO(3)-equivariant CNN for signals on spheres was studied in Cohen et al. (2018) in a different setting. Overall, the efficiency of equivariant CNNs remains to be improved since the model is typically several times larger than that of a regular CNN. The current paper adopts bases-decomposed filters previously studied in the non-equivariant setting (Qiu et al., 2018) , however, the joint convolution scheme in equivariant CNNs is over the two geometries of space and orientation simultaneously and neither the approach nor the analysis there can be directly applied.

2. ROTATION-EQUIVARIANT DCF NET:



2.1. ROTATION-EQUIVARIANT CNN:

A rotation-equivariant CNN indexes the channels by the SO(2) group (Weiler et al., 2017; Zhou et al., 2017) : The l-th layer output is written as x (l) (u, α, λ), the position u ∈ R 2 , the rotation α ∈ S 1 , and λ ∈ [M l ], M l being the number of unstructured channel indices. Throughout the paper, [m] stands for the set {1, • • • , m}. We denote the group SO(2) also by the circle S 1 since the former is parametrized by the rotation angle. The convolutional filter at the l-th layer is represented as

W (l) λ ,λ (v, α), λ ∈ [M l-1 ], λ ∈ [M l ], v ∈ R 2 , α ∈ S 1

, except for the 1st layer where there is no indexing of α. In practice, S 1 is discretized into N θ points on (0, 2π). We denote the summation over u and α by continuous integration, and the notation

S 1 (• • • )dα means 1 2π 2π 0 (• • • )dα.

Let the 2D rotation by angle t be denoted by Θ t , in the 1st layer of the group-invariant CNN,

x (1) (u, α, λ) = σ M0 λ =1 S 1 R 2 x (0) (u + v , λ )W (1) λ ,λ (Θ α v )dv + b (1) (λ) .

(1)

Note that the 1st layer output has N θ orientations indexed by α, forming a "channel geometry" (Table 1 ). For l > 1, the convolution is jointly over R 2 and SO(2), which takes the form as

x (l) (u, α, λ) = σ   M l-1 λ =1 S 1 R 2 x (l-1) (u + v , α , λ )W (l) λ ,λ (Θ α v , α -α)dv dα + b (l) (λ)   .

(2) The joint convolution over R 2 and SO(2) is both sufficient and necessary to guarantee groupequivariance (Theorem 3.1). While group equivariance is a desirable property, the model size and computation can be increased significantly due to the extra index α ∈ [N θ ].

2.2. DECOMPOSED FILTERS UNDER STEERABLE BASES:

We decompose the filters with respect to u and α simultaneously: Let {ψ k } k be a set of bases on the unit 2D disk, and {ϕ m } m be bases on S 1 . At the l-th layer, let j l be the scale of the filter in u, and ψ j,k = 2 -2j ψ k (2 -j u) (the filter is supported on the disk of radius 2 j l ). Since we use continuous convolutions, the down-sampling by "pooling" is modeled by the rescaling of the filters in space. The decomposed filters are of the form

EQUATION

which is illustrated in Figure 1 (for l > 1). We use Fourier-Bessel (FB) bases for {ψ k } k and Fourier bases for {ϕ m } m . Specifically, ϕ m (α) = e imα , and FB basis has the expression

ψ k (r, θ) = c m,q J m (R m,q r)e imθ , 0 ≤ r ≤ 1, 0 ≤ θ ≤ 2π, k = (m, q) (4)

where m is the angular frequency, q the radial frequency, J m the Bessel function (R m,q the q-th root of J m ) and c m,q a normalizing constant (Abramowitz & Stegun, 1964) . Both bases are "steerable", i.e. the operation of rotation is a diagonalized linear transform under both bases. In the complexvalued version,

ψ k (Θ t v) = e -im(k)t ψ k (v), ϕ m (α -t) = e -imt ϕ m (α)

. This means that after the convolutions on R 2 × S 1 with the bases ψ k (v)ϕ l (α) are computed for all k and l, both up to certain truncation, the joint convolution (1), ( 2) with all rotated filters can be calculated by the algebraic manipulation of the expansion coefficients a fully-connected layer regular convolutional layer CNN with group-indexed channels Apart from the saving of parameters and computation, the bases truncation also regularizes the convolutional filters by discarding the high frequency components. As a result, DCF Net reduces response to those components in the input at all layers, which improves the robustness of the learned feature without affecting recognition performance. The visibly smoother trained filters in RotDCF are shown in Figure A.1, demonstrating the same regularization effects as in Qiu et al. (2018) , the latter being without the rotation-equivariant setting. The theoretical properties of RotDCF Net, particularly the representation stability, will be analyzed in Section 3.

x (l-1) (λ ) → x (l) (λ) x (l-1) (u , λ ) → x (l) (u, λ) x (l-1) (u , α , λ ) → x (l) (u, α, λ) λ → λ: dense u → u: spatial convolution u → u, α → α: joint convolution λ → λ: dense λ → λ: dense

2.3. NUMBERS OF PARAMETERS AND COMPUTATION FLOPS:

Number of trainable parameters: In a regular CNN, a convolutional layer of size L×L×M 0 ×M 0 (L × L being the patch size) has L 2 M 0 M 0 parameters. In an equivariant CNN, a joint convolutional filter is of size L×L×N θ ×M ×M , so that the number of parameters is L 2 N θ M M . In a RotDCF Net, K bases are used in space and K α bases across the angle α, so that the number of parameters is KK α M M . This gives a reduction of K L 2 • Kα N θ compared to non-bases equivariant CNN. In practice, after switching from a regular CNN to a RotDCF Net, typically M ≤ 1 2 M 0 or more due to the adoption of filters in all orientations. The factor K L 2 is usually between 1 8 and 1 3 depending on the network and the problem (Qiu et al., 2018) . In all the experiments in Section 4, K α is typically 5, and N θ = 8 or 16. This means that RotDCF Net achieves a significant parameter reduction from the non-bases equivariant CNN (the factor KKα L 2 N θ is 1 8 or more), and even reduces parameters from a regular CNN by a factor of 1 2 or more. Computation in a forward pass: When the input and output are both W × W in space, the regular CNN layer needs 2M 0 M 0 W 2 L 2 many flops, and a non-bases equivariant convolutional layer needs about 2M M W 2 L 2 N 2 θ . In contrast, the computation in a RotDCF layer is dominated by a term of 2M M W 2 K α KN θ , which is reduced from the non-bases equivariant network by a factor of 

K L 2 • Kα N θ . Detailed calculations in Appendix A. In summary, RotDCF Net achieves a reduction of K L 2 • Kα N θ from non-bases equivariant

3. THEORETICAL ANALYSIS OF DEEP FEATURES:

This section presents two analytical results: (1) Joint convolution (1), ( 2) is sufficient and actually necessary to obtain rotation equivariance; (2) Stability of the equivariant representation with respect to input variations is proved under generic conditions, which is important in practice since rotations are never perfect.

3.1. GROUP-EQUIVARIANT PROPERTY:

We consider the change of the l-th layer output when the input image undergoes some arbitrary rotation. Let rotation around point u 0 by angle t be denoted by ρ = ρ u0,t , i.e. ρ u0,t u = u 0 + Θ t (u -u 0 ), for any u ∈ R 2 , and the transformed image by D ρ x (0) (u, λ) = x (0) (ρ u0,t u, λ), for any λ ∈ [M 0 ]. We also define the action T ρ on the l-th layer output x (l) , l > 0, as

T ρ x (l) (u, α, λ) = x (l) (ρ u0,t u, α -t, λ), ∀λ ∈ [M l ].

(5)

The following theorem, proved in Appendix B, shows that the joint convolution scheme (1), ( 2) not only produces group-equivariant features at all layers in the sense of

EQUATION

but is necessary for a CNN with SO(2)-indexed channels to achieve (6). The sufficiency part is previously shown in Weiler et al. (2017) . The necessity of the joint convolution motivates the design and the efforts of reducing the complexity of such models. Note that RotDCF is a type of the channel-indexed CNNs considered in the theorem, so it follows that RotDCF is group-equivariant.

Theorem 3.1. In a CNN with SO(2)-indexed channels, let x (l) [x (0) ] be the output at the l-th layer from input x (0) (u, λ). The relation (6) holds for all l if and only if the convolutional layers are given by (1), (2).

3.2. REPRESENTATION STABILITY UNDER INPUT VARIATIONS:

Assumptions on the RotDCF layers. Following Qiu et al. (2018) , we make the following generic assumptions on the convolutional layers: First, (A1) Non-expansive sigmoid: σ : R → R is non-expansive.

Second, we also need a boundedness assumption on the convolutional filters W (l) for all l:

(A2) Boundedness of filters: In all layers, A l ≤ 1, where A l is defined as

EQUATION

EQUATION

µ k being the Dirichlet Laplacian eigenvalues of the unit disk in R 2 . Note that (A2) bounds the expansion coefficients, which implies a sequence of boundedness conditions on the convolutional filters in all layers (Proposition B.1), based upon which the stability results below are derived. Since µ k typically increases in order, (A2) suggests truncating the series to only include low-frequency k and m's, which is implemented in Section 4. The validity of the boundedness assumption can be qualitatively fulfilled by normalization layers which is standard in practice.

Non-expansiveness of the network mapping. Let the L 2 norm of x (l) be defined as

x (l) 2 = 1 M l M l λ=1 1 |Ω| R 2 S 1 x (l) (u, α, λ) 2 dudα, l ≥ 1 and x (0) 2 = 1 M0 λ 1 |Ω| R 2 x (0) (u, λ) 2 du. Ω is the domain on which x (0) is supported, usually Ω = [-1, 1] × [-1, 1] ⊂ R 2 .

The following result is proved in Appendix B: Proposition 3.2. In a RotDCF Net, under (A1), (A2), for all l, (a) The mapping of the l-th convolutional layer (including σ), denoted as

x (l) [x (l-1) ], is non- expansive, i.e., x (l) [x 1 ] -x (l) [x 2 ] ≤ x 1 -x 2 for arbitrary x 1 and x 2 . (b) x (l) c ≤ x (l-1) c for all l, where x (l) c (u, α, λ) = x (l) (u, α, λ) -x (l) 0 (λ) (without index α when l=1) is the centered version of x (l) by removing x (l)

0 , defined to be the output at the l-th layer from a zero bottom-layer input. As a result, x

(l) c ≤ x (0) c = x (0) .

Insensitivity to input deformation. We consider the deformation of the input "module" to a global rotation. Specifically, let the deformed input be of the form D ρ • D τ x (0) , where D ρ is as in Section 3.1, ρ = ρ u0,t being a rigid 2D rotation, and D τ is a small deformation in space defined by

EQUATION

with τ : R 2 → R 2 is C 2 . Following Qiu et al. (2018) , we assume the small distortion condition:

(A3) Small distortion: |∇τ | ∞ = sup u ∇τ (u) < 1 5 , with • being the operator norm.

The mapping u → u -τ (u) is locally invertible, and the constant 1 5 is chosen for convenience. With T ρ as defined in (5), the stability result is summarized as Theorem 3.3. Let ρ = ρ u0,t be an arbitrary rotation in R 2 , around u 0 by angle t, and let D τ be a small deformation. In a RotDCF Net, under (A1), (A2), (A3), c 1 = 4, c 2 = 2, for any L, To prove Theorem 3.3, we firstly establish an approximate equivariant relation for all layers l (Proposition B.2), which can be of independent interest for estimating the image transformations. All the proofs are left to Appendix. Unlike previous stability results for regular CNNs, the above result allows an arbitrary global rotation ρ with respect to which the RotDCF representation is equivariant, apart from a small "residual" distortion τ whose influence can be bounded. This is also an important result in practice, because most often in recognition tasks the image rotation is not a rigid in-plane one, but is induced by the rotation of the object in 3D space. Thus the actual transformation of the image may be close to a 2D rotation but is not exact. The above result guarantees that in such cases the RotDCF representation undergoes approximately an equivariant action of T ρ , which implies consistency of the learned deep features up to a rotation. The improved stability of RotDCF Net over regular CNNs in this situation is observed experimentally in Section 4.

x (L) [D ρ • D τ x (0) ] -T ρ x (L) [x (0) ] ≤ (2c 1 L|∇τ | ∞ + c 2 2 -j L |τ | ∞ ) x (0) .

4. EXPERIMENTAL RESULTS:

In this section, we experimentally test the performance of RotDCF Nets on object classification and face recognition tasks. The advantage of RotDCF Net is demonstrated via improved recognition accuracy and robustness to rotations of the object, not only with in-plain rotations but with 3D rotations as well. To illustrate the rotation equivariance of the RotDCF deep features, we show that a trained auto-encoder with RotDCF encoder layers is able to reconstruct rotated digit images from "circulated" codes. All codes will be publicly available.

4.1. OBJECT CLASSIFICATION:

Non-transfer learning setting. The rotMNIST dataset contains 28 × 28 grayscale images of digits from 0 to 9, randomly rotated by an angle uniformly distributed from 0 to 2π (Cohen & Welling, 2016a) . We use 12,000 and 6,000 training samples, and 50,000 testing samples. The baseline net is (Qiu et al., 2018) and SFCNN (Weiler et al., 2017) . "# param." is number of parameters in all convolutional layers, and "Ratio" indicates the proportion to the # param. of the regular CNN.

Notice that the reduction from non-bases rotation-equivariant CNNs (the fair comparison case) can be even smaller, which is the factor of KKα L 2 N θ , c.f. Section 2.3. The case of maximum K and Kα (K=14, Kα=8 in rotMNIST) is mathematically equivalent to using "full" filters without bases truncation. a CNN with 6 convolutional layers (Table A .2), and we also compare with DCF (Qiu et al., 2018) , which is non-rotation-equivariant but uses bases truncation, and SFCNN (Weiler et al., 2017) , which is rotation-equivariant but does not use bases decompostion. The RotDCF net is made by replacing the regular convolutional layers with RotDCF layers, with N θ many rotation-indexed channels and a reduced number of (unstructured) channels M . K many ψ k bases and K α many ϕ m are used. The classification accuracy is shown in Table 3 . We see that RotDCF net obtains improved classification accuracy from CNN with significantly reduced number of parameters, e.g., with 12K training, Rot-DCF Net with M =8, K=3, K α =5 improves the test accuracy from 96.54 to 98.40 with less than 1 20 many parameters of the CNN model and 1

3 of the DCF model. The performance is also comparable to SFCNN which has significantly larger model complexity. The results are similar with reduced training size (5K) and on a shallower net with 3 conv layers, c.f. Table A 

.3.

The CIFAR10 dataset consists of 32 × 32 colored images from 10 object classes (Krizhevsky, 2009) , and we use 10,000 training and 50,000 testing samples. The network architecture is modified from VGG-16 net (Simonyan & Zisserman, 2014) (Table A.4) . As shown in Table 3 , RotDCF Net obtains better testing accuracy with reduced model size from the regular CNN baseline model. Throughout these experiments, the higher accuracy is due to the group equivariance and the lower model complexity is due to the bases decomposition.

Transfer learning setting. We train a regular CNN and a RotDCF Net on 10,000 up-right MNIST data samples, and directly test on 50,000 randomly rotated MNIST samples where the maximum rotation angle MaxRot=30 or 60 degrees (the "no-retrain" case). We also test after retraining the last two non-convolutional layers (the " fc-retrain" case). To visualize the importance of image regions which contribute to the classification accuracy, we adopt Class Activation Maps (CAM) (Zhou et al., 2016) , and the network is modified accordingly by removing the last pooling layer in the net in Table A .1. and inserting a "gap" global averaging layer. The test accuracy are listed in Table 2 , where the superiority of RotDCF Net is clearly shown in both the "no-retrain" and " fc-retrain" cases . The improved robustness of RotDCF Net is furtherly revealed by the CAM maps (Figure 2 ): the redcolored region is more stable for RotDCF Net even in the case with retraining.

4.2. IMAGE RECONSTRUCTION:

To illustrate the explicit encoding of group actions in the RotDCF Net features, we train a convolutional auto-encoder on the rotated MNIST dataset, where encoder consists of stacked RotDCF layers, and the decoder consists of stacked transposed-convolutional layers (Table A .5). The encoder maps a 28×28 image into an array of 16 × 32, where the first dimension is the discretization of the rotation angles in [0, 2π], and the second dimension is the unstructured channels. Due to the rotation equivariant relation, the "circulation" of the rows of the code array should correspond to the rotation of the image. This is verified in Figure 3 

4.3. FACE RECOGNITION:

As a real-world example, we test RotDCF on the Facescrub dataset (Ng & Winkler, 2014) containing over 100,000 face images of 530 people. A CNN and a RotDCF Net (Table A.6 ) are trained respectively using the gallery images from the 500 known subjects, which are preprocessed to be near-frontal and upright-only by aligning facial landmarks (Kazemi & Sullivan, 2014) . See Appendix C for data preparation and training details. For the trained deep networks, we remove the last softmax layer, and then use the network outputs as deep features for faces, which is the typical way of using deep models for face verification and recognition to support both seen and unseen subjects (Parkhi et al., 2015) . Using deep features generated by the trained networks, a probe image is then compared with the gallery faces whose identities are known and classified as that of the top match.

Under this gallery-probe face recognition setup, we obtain 94.10% and 96.92% accuracy for known and unknown subjects respectively using the CNN model; using RotDCF, the accuracies are 93.42% and 96.92%. Testing on unknown subjects are critical for validating the model representation power over unseen identities, and the reason for higher accuracy is simply due to the smaller number of classes. For both cases, RotDCF reports comparable performance as CNN, while the number of parameters in the RotDCF model is about one-fourth of the CNN model (see Appendix).

In-plane rotation. This experiment demonstrates the rotation-equivariance of the RotDCF features. We apply in-plane rotations at intervals of π 4 to the probe images (Figure 4 ), and let the original probe set be the new gallery, the rotated copies be the new probe set. In this setting, using the RotDCF model we obtain 97.04% and 97.58% recognition accuracy for known and unknown subjects respectively, after aligning the deep features by circular shifts (using the largest-magnitude α channel as reference). Notice that the model only sees upright faces. This is due to the rotation-equivariant property of the RotDCF Net, which means that the face representation is consistent regardless of its orientations after the group alignment. Lacking such properties, CNN obtains 0.54% and 5.05% recognition accuracies, which is close to random guess. We further compare CNN and RotDCF models via the CAM maps, which indicate the image regions relied by the CNN to make a classification prediction. In Figure 4 , with regular CNN face regions used for prediction varies dramatically from image to image, and with RotDCF a significantly more consistent region mostly covering eyes and nose are selected across images, which explains its superior accuracy.

Out-of-plane rotation. To validate our theoretical result on representation stability under input deformations, we introduce out-of-plane rotations to the probe. Each probe image is fitted to a 3D face mesh, and rotated copies are rendered at the 10 o intervals with -40 o to 40 o yaw, and -20 o to 20 o pitch, generating 45 synthesized faces in total (Figure 5 ). The synthesis faces at two poses (highlighted in red) are used as the new gallery, and all remaining synthesis faces form the new probe. The out-of-plane rotations here can be viewed as mild in-plane rotations plus additional variations, a situation frequently encountered in the real world. With this gallery-probe setup, the RotDCF model obtains 89.66% and 97.01% recognition accuracy for known and unknown subjects, and the accuracies are 80.79% and 89.97% with CNN. The CAM plots in Figure 6 also indicate that RotDCF Net chooses more consistent regions over CNN in describing a subject across different poses. Since the out-of-plane rotations as in Figure 5 can be considered as in-plane rotations with additional variations, the superior performance of RotDCF is consistent with the theory in Section 3.

5. CONCLUSION AND DISCUSSION:

This work introduces a decomposition of the filters in rotation-equivariant CNNs under joint steerable bases over space and rotations simultaneously, obtaining equivariant deep representations with significantly reduced model size and an implicit filter regularization. The group equivariant property and representation stability are proved theoretically. In experiments, RotDCF demonstrates improved recognition accuracy and better feature interpretability and stability on synthetic and realworld datasets involving object rotations, particularly in the transfer learning setting. To extend the work, implementation issues like parallelism efficiency and memory usage should be considered before the computational savings can be fully achieved. The framework should also extend to other groups and joint geometrical domains.

SUPPLEMENTARY MATERIAL A FLOPS CALCULATION IN SECTION 2 AND CHOICE OF HYPERPARAMETERS:

When the input and output are both W × W in space, the forward pass in a regular convolutional layer needs M 0 M 0 W 2 (1 + 2L 2 ) ∼ 2L 2 M 0 M 0 W 2 flops. (Each convolution with a L × L filter takes 2L 2 W 2 , and there are M 0 M 0 convolution operations, plus that the summation over λ takes W 2 M 0 M 0 flops.) In a rotation equivariant CNN without using bases, an convolutional layer would take ∼ 2M M W 2 L 2 N 2 θ flops. In a RotDCF layer, the computation consists of three parts: (1) The inner-product with ϕ m bases takes W 2 M • 2N θ K α flops. (2) The spatial convolution with the ψ k bases takes

K α M K • 2L 2 W 2 flops.

(3) The multiplication with a λ ,λ (k, m)e -im(k)α-imα and summation over λ , k, m takes M N θ (4KK α M + 2W 2 KK α M ) flops (real-valued version). Putting together, the total is

2M W 2 K α (N θ + L 2 K + M N θ K)

, and when M is large, the third term dominates and it gives 2M M W 2 K α KN θ . Thus the reduction by using bases-decomposed filters is again a factor of K L 2 • Kα N θ , and the relative ratio with a regular CNN is about

M M M 0 M0 • KKαN θ L 2

.

The choice of hyperparameters K and K α can be interpreted as the "frequency" truncation both in R 2 and in S 1 , and it trades-off between filter regularization and filter expressiveness: The more severe the truncation, the less many trainable parameters, the smoother the filter (Figure A.1) and less overfitting, while the potential risk of underfitting is larger. This is revealed in Table 3 : as K and K α are reduced, the accuracy decreases monotonically in the rotMNIST experiment. When K α is reduced to 3 then underfitting becomes noticeable and accuracy drops even more (not reported). The parameter N θ corresponds to spatial discretization along S 1 and, since we adopt a spectral approach, it does not affect the number of trainable parameters, which is determined by K α . In other words, the choice of N θ is only constrained by memory storage.

B PROOFS IN SECTION 3:

B.1 THEOREM 3.1

Proof of Theorem 3.1. Note: The bases expansion under ψ j l ,k and ϕ m does not affect the form of convolutional layers, but only impose regularity of the filters, thus the group-equivariant property of RotDCF follows from the theorem.

Observe that the equivariant relation ( 6) is equivalent to that

T ρ x (l) [x (l-1) ] = x (l) [T ρ x (l-1) ] (A.1)

for all l, where T ρ x (0) means D ρ x (0) .

The sufficiency part: When l = 1, by (1),

T ρ x (1) [x (0) ](u, α, λ) = σ( λ R 2 x (0) (ρ u0,t u + v, λ )W (1) λ ,λ (Θ α-t v)dv + b (1) (λ)), x (1) [D ρ x (0) ](u, α, λ) = σ( λ R 2 x (0) (ρ u0,t (u + v), λ )W (1) λ ,λ (Θ α v)dv + b (1) (λ)). Since ρ u0,t (u + v) = ρ u0,t + Θ t v, we have that T ρ x (1) [x (0) ] = x (1) [D ρ x (0) ] by a change of variable of Θ t v → v.

When l > 1, by (2),

T ρ x (l) [x (l-1) ](u, α, λ) = σ( λ R 2 S 1 x (l-1) (ρ u0,t u + v, α , λ )W (l) λ ,λ (Θ α-t v, α -α + t)dvdα + b (l) (λ)), x (l) [T ρ x (l-1) ](u, α, λ) = σ( λ R 2 S 1 x (l-1) (ρ u0,t (u + v), α -t, λ )W (l) λ ,λ (Θ α v, α -α)dvdα + b (l) (λ)).

Again, inserting ρ u0,t (u + v) = ρ u0,t + Θ t v, the claim follows by changing variables Θ t v → v and α -t → α . The necessity part: When l = 1, denote the general convolutional filter as w (1) (v; λ , λ, α), and

x (1) (u, α, λ) = σ( λ R 2 x (0) (u + v, λ )w (1) (v; λ , λ, α)dv + b (1) (λ)).

Recall that

T ρ x (1) [x (0) ](u, α, λ) = σ( λ R 2 x (0) (ρ u0,t u + v, λ )w (1) (v; λ , λ, α -t)dv + b (1) (λ)), x (1) [D ρ x (0) ](u, α, λ) = σ( λ R 2 x (0) (ρ u0,t (u + v), λ )w (1) (v; λ , λ, α)dv + b (1) (λ))

and then (A.1) with l = 1 holding for any

x (0) implies that λ R 2 x (0) (ρ u0,t u+v, λ )w (1) (v; λ , λ, α-t)dv = λ R 2

x (0) (ρ u0,t (u+v), λ )w (1) (v; λ , λ, α)dv.

By that ρ u0,t (u + v) = ρ u0,t + Θ t v, the above equality gives that w (1) (v; λ , λ, α -t) = w (1) (Θ -1 t v; λ , λ, α), ∀α, t ∈ S 1 . Let t = α, and F λ ,λ (v) = w (1) (v; λ , λ, 0), we have that

EQUATION

, and this gives that w (1) (v; λ , λ, α) = F λ ,λ (Θ α v). This proves that (1) is necessary.

When l > 1, consider the general convolutional filter as w (l) (v; λ , λ, α , α). Using a similar argument, (A.1) implies that w (l) (v; λ , λ, α , α -t) = w (l) (Θ -1 t v; λ , λ, α + t, α), ∀α, t ∈ S 1 . Let t = α, and F λ ,λ (v, α ) = w (l) (v; λ , λ, α , 0), then

F λ ,λ (v, α ) = w (l) (Θ -1 α v; λ , λ, α + α, α), which gives that w (l) (v; λ , λ, α , α) = F λ ,λ (Θ α v, α -α), which proves that (2) is necessary. B.2 PROPOSITION 3.2 Proposition B.1. For all l, B (l) λ ,λ , C (l) λ ,λ , 2 j l D (l) λ ,λ ≤ π a (l) λ ,λ FB , where B (l) λ ,λ := R 2 S 1 |W (l) λ ,λ (v, β)|dvdβ, l > 1, B (1) λ ,λ := R 2 |W (1) λ ,λ (v)|dv C (l) λ ,λ := R 2 S 1 |v||∇ v W (l) λ ,λ (v, β)|dvdβ, l > 1, C (1) λ ,λ := R 2 |v||∇ v W (1) λ ,λ (v)|dv D (l) λ ,λ := R 2 S 1 |∇ v W (l) λ ,λ (v, β)|dvdβ, l > 1, D (1) λ ,λ := R 2 |∇ v W (1) λ ,λ (v)|dv (A.2) As a result, B l , C l , 2 j l D l ≤ A l ,

where

B l := max{sup λ M l-1 λ =1 B (l) λ ,λ , sup λ M l-1 M l M l λ=1 B (l) λ ,λ }, C l := max{sup λ M l-1 λ =1 C (l) λ ,λ , sup λ M l-1 M l M l λ=1 C (l) λ ,λ }, D l := max{sup λ M l-1 λ =1 D (l) λ ,λ , sup λ M l-1 M l M l λ=1 D (l) λ ,λ }, (A.3)

and thus (A2) implies that B l , C l , 2 j l D l ≤ 1 for all l.

Proof of Proposition B.1. The proof for the case of l = 1 is the same as Lemma 3.5 and Proposition 3.6 of Qiu et al. (2018) . We reproduce it for completeness. When l = 1, it suffices to show that for

F (v) = k a k ψ k (v), |F (v)|dv, |v||∇F (v)|dv, |∇F (v)|dv ≤ π( k µ k a 2 k ) 1/2 . (A.4)

Rescaling to ψ j l ,k in v leads to the desired inequality with the factor of 2 j l for D (l) λ ,λ . To prove (A.4), observe that F is supported on the unit disk, and then

F 1 , |v||∇F (v)|dv ≤ ∇F 1 ≤ √ π ∇F 2 , where ∇F 2 2 = π k µ k a 2

k due to the orthogonality of ψ k . For l > 1, similarly, we only consider the rescaled filters supported on the unit disk in

v. Let F (v, β) = k,m a k,m ψ k (v)ϕ m (β), β ∈ S 1 , similarly as above, we have that |F (v, β)|dvdβ, |v||∇ v F (v, β)|dvdβ ≤ |∇ v F (v, β)|dvdβ ≤ (π |∇ v F (v, β)| 2 dvdβ) 1/2

recalling that dβ on S 1 has the normalization of 1 2π . Again,

|∇ v F (v, β)| 2 dvdβ = π k,m µ k a 2

k,m due to the orthogonality of ψ k and ϕ m . This proves that

|∇ v F (v, β)|dvdβ ≤ π( k,m µ k a 2 k,m ) 1/2 ,

which leads to the claim after a rescaling of v.

Remark 1 (Remark to Proposition 3.2). The proposition only needs B l , defined in (A.3), to be less than 1 for all l, in a rotation-equivariant CNN, which is implied by (A2) by Proposition B.1.

Proof of Proposition 3.2. The proof is similar to that of Proposition 3.1(a) of Qiu et al. (2018) . Specifically, in (a), the argument is the same for l = 1, making use of the fact that

|w(Θ α v)|dv = |w(v)|dv, ∀α ∈ S 1

and S 1 dα = 1 due to the normalization of 1 2π . For l > 1, the same technique proceeds with the new definition of B (l) λ ,λ as in (A.2) which involves the integration of S 1 (• • • )dβ. The detail is omitted.

To prove (b), we firstly verify that x (l) 0 only depends on λ. When l = 1, x

(1) 0 (u, α, λ) = σ(b (1) (λ)). Suppose that it holds for (l -1), consider l > 1,

x (l) 0 (u, α, λ) = σ( λ S 1 R 2 x (l-1) 0 (u + v, α + β, λ )W (l) λ ,λ (Θ α v, β)dvdβ + b (l) (λ)) = σ( λ x (l-1) 0 (λ ) S 1 R 2 W (l) λ ,λ (Θ α v, β)dvdβ + b (l) (λ)) = σ( λ x (l-1) 0 (λ ) • S 1 R 2 W (l) λ ,λ (v , β)dv dβ + b (l) (λ)) = x (l) 0 (λ). Thus x (l) 0 (u, α, λ) = x (l)

0 (λ) for all l (without index α for l = 1). The rest of the argument follows from that x

(l) c = x (l) -x (l) 0 = x (l) [x (l-1) ] -x (l) [x (l-1) 0 ] ≤ x (l-1) -x (l-1) 0 = x (l-1) c

, where the inequality is by (a).

B.3 THEOREM 3.3:

Proposition B.2. In a RotDCF Net, under (A1), (A2), (A3), c 1 = 4, for any l,

x (l) [D ρ • D τ x (0) ] -T ρ • D τ x (l) [x (0) ] ≤ 2c 1 l|∇τ | ∞ x (0) ,

where D τ only acts on the space variable u of x (l) similar to (9).

Proof of Proposition B.2. We firstly establish that for all l,

x (l) [T ρ • D τ x (l-1) ] -T ρ • D τ x (l) [x (l-1) ] ≤ 2c 1 |∇τ | ∞ x (l-1) c , (A.5)

where T ρ is replaced by D ρ if applies to x (0) which does not have index α. This is because that

x (l) [T ρ • D τ x (l-1) ] = T ρ x (l) [D τ x (l-1)

] by Theorem 3.1, and that -1) ] by the definition of T ρ (a rigid rotation in u, and a translation in α). This term can be upper bounded by c The rest of the proof is similar to that of Proposition 3.3 of Qiu et al. (2018) -1) ] due to Proposition 3.2 (a), which, by applying (A.5) together with Proposition 3.2 (b), can be bounded by 2c 1 |∇τ | ∞ x (0) . Summing over j gives the claim.

T ρ x (l) [D τ x (l-1) ] -T ρ • D τ x (l) [x (l-1) ] = x (l) [D τ x (l-1) ] -D τ x (l) [x (l

1 (B l + C l )|∇τ | ∞ x (l-1) c (Lemma B.

: Write x (l) [D ρ • D τ x (0) ] -T ρ • D τ x (l) [x (0) ] as the sum of the differences x (l) [x (j) [D ρ • D τ x (j-1) ]] -x (l) [T ρ • D τ x (j) [x (j-1) ]] for j = 1, • • • , l. The norm of the j-th term is bounded by x (j) [D ρ • D τ x (j-1) ] - T ρ • D τ x (j) [x (j

Proof of Theorem 3.3. The proof is similar to that of Theorem 3.8 of Qiu et al. (2018) . With the bound in Proposition B.2, it suffices to show that

T ρ • D τ x (L) [x (0) ] -T ρ x (L) [x (0) ] ≤ c 2 2 -j L |τ | ∞ x (0) .

By the definition of T ρ , the l.h.s. equals D τ x (L) [x (0) ] -x (L) [x (0) ] , which can be shown to be less than c 2 |τ | ∞ D L x (l-1) c by extending the proof of Proposition 3.4 of Qiu et al. (2018) , similar to the argument in proving Lemma B.3. The desired bound then follows by that D L ≤ 2 -j L A L ≤ 2 -j L (Proposition B.1 and (A2)) and that x Proof of Lemma B.3. The proof is similar to that of Lemma 3.2 of Qiu et al. (2018) . Specifically, when l = 1, the argument is the same, making use of the fact that |w(Θ α v)|dv = |w(v)|dv, ∀α ∈ S 1 and S 1 dα = 1 due to the normalization of 1 2π . When l > 1, the same technique applies by considering the joint integration of R 2 S 1 (• • • )dvdβ instead of just dv. The only difference is in using the new definitions of B In the experiments on rotMNIST dataset, the network architecture for Conv-3 and Conv-6 nets are shown in Table A .1 and Table A.2 . Stochastic gradient descent (SGD) with momentum is used to train 100 epochs with decreasing learning rate from 10 -2 to 10 -4 .

In the experiments on CIFAR10 dataset, the VGG16-like network architecture is shown in Table A .4. SGD with momentum is used to train 100 epochs with decreasing learning rate from 10 -2 to 10 -4 . 

:



Figures and Tables:

(k, m), and without any re-computation of the spatial-rotation joint convolution. Standard real-valued versions of the bases ψ k and ϕ m in sin's and cos's are used in practice. During training, only the expansion coefficients a's are updated, and the bases are fixed.

CNNs, in terms of both model size and computation. With typical network architectures, RotDCF Net may be of a smaller model size than regular CNNs. Numbers for specific networks are shown in Section 4.

Figure 2: Representative class activation maps (CAM) on testing images in the rotMNIST transfer learning experiment. The heatmap indicates the importance of image regions used in recognizing a digit class. The CNN and RotDCF networks are trained on up-right samples, with no retrain (left) and retraining the fully connected layers respectively (right) before testing. Testing samples are randomly rotated up to 60 degrees. (c.f. Table2).

Figure 3: Codes and reconstructions of rotMNIST digits. (Top) A test image is encoded into a 16 × 32 array in the red box (the intermediate representation), and the code generates 16 copies by circulating the rows. (Bottom) Images reconstructed from the row-circulated codes above by the decoder.

Figure 4: Example CAM maps for recognizing faces with in-plane rotations. The heatmap indicates the importance of different image regions used by respective models in defining a face, and a good CNN model is expected to select consistent face regions across the same-person images to determine the identity. Across different in-plane rotated copies, RotDCF chooses significantly more consistent discriminative regions than CNN, indicating more stable representations.In this experiment, we obtain 0.54% recognition accuracy using CNN (nearly random guess), and 97.04% accuracy using RotDCF with feature alignment, on known subjects.

Figure 5: Synthesized faces from a testing image with -40 o to 40 o yaw, and -20 o to 20 o pitch, at a 10 o interval.

3), which leads to the desired bound under (A2) by Proposition B.1.

In a rotation-equivariant CNN, B l , C l defined as in (A.3), under (A1), (A3), for all l > 0, with c 1 = 4, x (l) c as in Proposition 3.2,x (l) [D τ x (l-1) ] -D τ x (l) [x (l-1) ] ≤ c 1 (B l + C l )|∇τ | ∞ x (l-1) softmax-loss fc64 ReLu fc10 softmax-loss

for l > 1 as in (A.2), both of which involve the integration of S 1 (• • • )dβ. The detail is omitted. C EXPERIMENTAL DETAILS IN SECTION 4 C.1 OBJECT RECOGNITION WITH ROTMNIST AND CIFAR10

Comparison of a fully-connected layer, a regular convolutional layer, and a rotation-equivariant convolutional layer with group-indexed channels.

Table content: <table/>

.

Table content: <table><tr><td>MNIST to rotMNIST MaxRot=30 Degrees no-retrain fc-retrain CNN 92.61 94.71 RotDCF 96.90 98.48</td><td>MNIST to rotMNIST MaxRot=60 Degrees no-retrain fc-retrain CNN 69.61 85.90 RotDCF 82.36 97.68</td></tr></table>

Test accuracy in the rotMNIST transfer learning experiment. The network is trained on 10K up-right MNIST samples and tested on 50K randomly rotated samples up to the MaxRot degrees.

Table content: <table/>

Classification accuracy using non-equivariant CNN and rotation-equivariant ones on rotMNIST and CIFAR10. Baseline DCF

Table content: <table/>

1: Conv-3 network architectures used in rotMNIST, M = 32, 16 or 8. cLxLxM xM stands for a convolutional layer of patch size LxL and input (output) channel M (M ). apLxL stands for LxL averagepooling. In the RotDCF Net, rcLxLxN θ xM xM stands for a rotation-indexed convolutional layer, which includes N θ -times many number of filters except for the 1st rc layer (see Section 2). Batch-normalization layers (not shown) are used during training.

Table content: <table><tr><td>Conv-6 CNN-M c5x5x1xM ReLu c5x5xM x1.5M ReLu ap2x2 rc5x5xN θ xM x1.5M ReLu ap2x2 Conv-6 RotDCF-M rc5x5x1xM ReLu c5x5x1.5M x2M ReLu rc5x5xN θ x1.5M x2M ReLu c5x5x2M x2M ReLu ap2x2 rc5x5xN θ x2M x2M ReLu ap2x2 c5x5x2M x3M ReLu rc5x5xN θ x2M x3M ReLu c5x5x3M x4M ReLu ap2x2 rc5x5xN θ x3M x4M ReLu ap2x2 fc64 ReLu fc10 softmax-loss fc64 ReLu fc10 softmax-loss</td></tr></table>

2: Conv-6 network architectures used in rotMNIST. Similar to Table A.1 with 3 more layers.

Table content: <table/>

3: Classification accuracy using non-equivariant CNN and rotation-equivariant CNNs on rotMNIST based on Conv-3, c.f. TableA.1. Similar results to the rotmnist experiments in Table3with a net of fewer (three) layers.

Table content: <table><tr><td colspan="3">rotMNIST Conv-3, Ntr = 10K Test Acc. # param.</td><td>Ratio</td><td/></tr><tr><td>CNN M =32</td><td>95.67</td><td>2.570×10 5</td><td>1.00</td><td/></tr><tr><td>DCF M =32, K=5 DCF M =32, K=3 RotDCF N θ = 8 M =16, K=14, Kα=8 M =16, K=5, Kα=8 M =16, K=3, Kα=8 M =16, K=5, Kα=5</td><td>95.58 95.69 97.86 97.81 97.77 97.96</td><td>5.158×10 4 3.104×10 4 2.871×10 5 1.026×10 5 6.160×10 4 6.419×10 4</td><td>0.20 0.12 1.12 0.40 0.24 0.25</td><td>rotMNIST Conv-3, Ntr = 5K Test Acc. # param. 94.04 DCF M =32, K=3 CNN M =32 94.08 RotDCF N θ =8 M =16, K=3, Kα=5 96.79 (same as left) Ratio M =8, K=3, Kα=5 96.53</td></tr><tr><td>M =16, K=3, Kα=5</td><td>97.95</td><td>3.856×10 4</td><td>0.15</td><td/></tr><tr><td>M =8, K=5, Kα=5</td><td>97.81</td><td>1.610×10 4</td><td>0.06</td><td/></tr><tr><td>M =8, K=3, Kα=5</td><td>97.59</td><td>9.680×10 3</td><td>0.04</td><td/></tr></table>

Peer review (iclr2019):
 Summary: This paper combines the benefits of using joint steerable filters (using the SO(2) group) for designing rotation-equivariant CNNs with those of decomposing the filters (using Fourier-Bessel bases) for reducing the computational complexity. In addition, this leads to a compressed model and filter regularization. The authors give theoretical guarantees on the rotation equivariance and representation stability with respect to in and out of plane rotation. Empirical results show that the model attains better accuracy compared to CNNs and non-rotation-equivariant deep networks while using fewer parameters and also performs similarly to a rotation-equivariant model with much bigger capacity. Pros: - Theoretical guarantees, elegant approach - Good empirical results compared to other models - Desirable properties: rotation-equivariance, lower computational complexity, fewer parameters, robustness and guaranteed stability to deformations Cons: - Somewhat incremental technical novelty: combination of two previously published methods (Qiu et al. 2018 & Weiler et al. 2017) Comments: 1. I believe the related work section can be improved by explaining more clearly the connection between your work and the cited ones and emphasizing the advantages and limitations of RotDCF compared to other methods In particular, a reader should be able to precisely understand what is the novelty of this work is and what were the technical challenges in combining previously published ideas (such as DCF and SFCNN) 2. How do you determine the truncation in practice? How robust is the method to this choice? What are the trade-offs between using a value that is too low or too high? It would be interesting to show how performance and complexity vary with this parameter 3. It would also be helpful to have a discussion on choosing the parameters K_{alpha} and N_{theta} and how this affects the performance, computational complexity and number of parameters. This would provide more intuition on the limits of this method and the types of data it can be used for 4. In section 2.3, it would be helpful to specify an estimated range for the parameter reduction from the non-bases rotation-equivariant CNN to RotDCF (similar to the ½ factor from RotDCF to regular CNN) 5. Eq. (4) seems to be missing the definition of R_{m,q} 6. The notation for the supplementary material was confusing at times. I would suggest using the more standard notation for the appendix which can also be a more specific reference (e.g. A.1, A.2, etc.)