Year: 2019

Abstract:
Explicit encoding of group actions in deep features makes it possible for convolutional neural networks (CNNs) to handle global deformations of images, which is critical to success in many vision tasks. This paper proposes to decompose the convolutional filters over joint steerable bases across the space and the group geometry simultaneously, namely a rotation-equivariant CNN with decomposed convolutional filters (RotDCF). This decomposition facilitates computing the joint convolution, which is proved to be necessary for the group equivariance. It significantly reduces the model size and computational complexity while preserving performance, and truncation of the bases expansion serves implicitly to regularize the filters. On datasets involving in-plane and out-of-plane object rotations, RotDCF deep features demonstrate greater robustness and interpretability than regular CNNs. The stability of the equivariant representation to input variations is also proved theoretically. The RotDCF framework can be extended to groups other than rotations, providing a general approach which achieves both group equivariance and representation stability at a reduced model size.

heading: 1 INTRODUCTION
text: While deep convolutional neural networks (CNN) have been widely used in computer vision and image processing applications, they are not designed to handle large group actions like rotations, which degrade the performance of CNN in many tasks (Cheng et al., 2016; Hallman & Fowlkes, 2015; Jaderberg et al., 2015b; Laptev et al., 2016; Maninis et al., 2016). The regular convolutional layer is equivariant to input translations, but not other group actions. An indirect way to encode group information into the deep representation is to conduct generalized convolutions across the group as well, as in Cohen & Welling (2016a). In theory, this approach can guarantee the group equivariance of the learned representations, which provides better interpretability and regularity as well as the capability of estimating the group action in localization, boundary detection, etc. For the important case of 2D rotations, group-equivariant CNNs have been constructed in several recent works, e.g., (Weiler et al., 2017), Harmonic Net (Worrall et al., 2017) and Oriented Response Net (Zhou et al., 2017). In such networks, the layer-wise output has an extra index representing the group element (c.f. Table 1), and consequently, the convolution must be across the space and the group jointly (proved in Section 3.1). This typically incurs a significant increase in the number of parameters and computational load, even with the adoption of steerable filters (Freeman et al., 1991; Weiler et al., 2017; Worrall et al., 2017). In parallel, low-rank factorized filters have been proposed for sparse coding as well as the compression and regularization of deep networks. In particular, Qiu et al. (2018) showed that decomposing filters under non-adaptive bases can be an effective way to reduce the model size of CNNs without sacrificing performance. However, these approaches do not directly apply to be group-equivariant. We review these connections in more detail in Section 1.1.
This paper proposes a truncated bases decomposition of the filters in group-equivariant CNNs, which we call the rotation-equivariant CNN with decomposed convolutional filters (RotDCF). Since we need a joint convolution over R2 and SO(2), the bases are also joint across the two geometrical domains, c.f. Figure 1. The benefits of bases decomposition are three-fold: (1) Reduction of the number of parameters and computational complexity of rotation-equivariant CNNs, c.f. Section 2.3;
(2) Implicit regularization of the convolutional filters, leading to improved robustness of the learned deep representation shown experimentally in Section 4; (3) Theoretical guarantees on stability of the equivariant representation to input deformations, which follow from a generic condition on the filters in the decomposed form, c.f. Section 3.2.

heading: 1.1 RELATED WORK
text: Learning with factorized filters. In the sparse coding literature, Rubinstein et al. (2010) proposed the factorization of learned dictionaries under another prefixed dictionary. Separable filters were used in Rigamonti et al. (2013) to learn the coding of images. Papyan et al. (2017) interpreted CNN as an iterated convolutional sparse coding machine, and in this view, the factorized filters should correspond to a “dictionary of the dictionary” as in Rubinstein et al. (2010). In the deep learning literature, low-rank factorization of convolutional filters has been previously used to remove redundancy in trained CNNs (Denton et al., 2014; Jaderberg et al., 2014). The compression of deep networks has also been studied in Chen et al. (2015); Han et al. (2016; 2015), SqueezeNet (Iandola et al., 2016), etc., where the low-rank factorization of filters can be utilized. MobileNets (Howard et al., 2017) used depth-wise separable convolutions to obtain significant compression. Tensor decomposition of convolutional layers was used in Lebedev et al. (2014) for CPU speedup. Tai et al. (2015) proposed low-rank-regularized filters and obtained improved classification accuracy with reduced computation. Qiu et al. (2018) studied decomposed-filter CNN with prefixed bases and trainable expansion coefficients, showing that the truncated bases decomposition incurs almost no decrease in classification accuracy while significantly reducing the model size and improving the robustness of the deep features. None of the above networks are group equivariant. Group-equivariant deep networks. The encoding of group information into network representations has been studied extensively. Among earlier works, transforming auto-encoders (Hinton et al., 2011) used a non-convolutional network to learn group-invariant features and compared with handcrafted ones. Rotation-invariant descriptors were studied in Schmidt & Roth (2012b) with product models, and in Jaderberg et al. (2015a); Kivinen & Williams (2011); Schmidt & Roth (2012a) by estimating the specific image transformation. Gonzalez et al. (2016); Wu et al. (2015) proposed rotating conventional filters to perform rotation-invariant texture and image classification. The joint convolution across space and rotation has been studied in the scattering transform (Oyallon & Mallat, 2015; Sifre & Mallat, 2013). Group-equivariant CNN was considered by Cohen & Welling (2016a), which handled several finite small-order discrete groups on the input image. Rotation-equivariant CNN was later developed in Weiler et al. (2017); Worrall et al. (2017); Zhou et al. (2017) and elsewhere. In particular, steerable filters were used in Cohen & Welling (2016b); Weiler et al. (2017); Worrall et al. (2017). SO(3)-equivariant CNN for signals on spheres was studied in Cohen et al. (2018) in a different setting. Overall, the efficiency of equivariant CNNs remains to be improved since the model is typically several times larger than that of a regular CNN. The current paper adopts bases-decomposed filters previously studied in the non-equivariant setting (Qiu et al., 2018), however, the joint convolution scheme in equivariant CNNs is over the two geometries of space and orientation simultaneously and neither the approach nor the analysis there can be directly applied.

heading: 2 ROTATION-EQUIVARIANT DCF NET

heading: 2.1 ROTATION-EQUIVARIANT CNN
text: A rotation-equivariant CNN indexes the channels by the SO(2) group (Weiler et al., 2017; Zhou et al., 2017): The l-th layer output is written as x(l)(u, α, λ), the position u ∈ R2, the rotation α ∈ S1, and λ ∈ [Ml], Ml being the number of unstructured channel indices. Throughout the paper, [m] stands for the set {1, · · · ,m}. We denote the group SO(2) also by the circle S1 since the former is parametrized by the rotation angle. The convolutional filter at the l-th layer is represented as W (l)λ′,λ(v, α), λ
′ ∈ [Ml−1], λ ∈ [Ml], v ∈ R2, α ∈ S1, except for the 1st layer where there is no indexing of α. In practice, S1 is discretized into Nθ points on (0, 2π). We denote the summation over u and α by continuous integration, and the notation ∫ S1 (· · · )dα means 12π ∫ 2π 0 (· · · )dα.
Let the 2D rotation by angle t be denoted by Θt, in the 1st layer of the group-invariant CNN,
x(1)(u, α, λ) = σ ( M0∑ λ′=1 ∫ S1 ∫ R2 x(0)(u+ v′, λ′)W (1) λ′,λ(Θαv ′)dv′ + b(1)(λ) ) . (1)
Note that the 1st layer output has Nθ orientations indexed by α, forming a “channel geometry” (Table 1). For l > 1, the convolution is jointly over R2 and SO(2), which takes the form as
x(l)(u, α, λ) = σ Ml−1∑ λ′=1 ∫ S1 ∫ R2 x(l−1)(u+ v′, α′, λ′)W (l) λ′,λ(Θαv ′, α′ − α)dv′dα′ + b(l)(λ)  . (2) The joint convolution over R2 and SO(2) is both sufficient and necessary to guarantee groupequivariance (Theorem 3.1). While group equivariance is a desirable property, the model size and computation can be increased significantly due to the extra index α ∈ [Nθ].

heading: 2.2 DECOMPOSED FILTERS UNDER STEERABLE BASES
text: We decompose the filters with respect to u and α simultaneously: Let {ψk}k be a set of bases on the unit 2D disk, and {ϕm}m be bases on S1. At the l-th layer, let jl be the scale of the filter in u, and ψj,k = 2 −2jψk(2 −ju) (the filter is supported on the disk of radius 2jl ). Since we use continuous convolutions, the down-sampling by “pooling” is modeled by the rescaling of the filters in space. The decomposed filters are of the form
W (1) λ′,λ(v) = ∑ k a (1) λ′,λ(k)ψj1,k(v), W (l) λ′,λ(v, β) = ∑ k ∑ m a (l) λ′,λ(k,m)ψjl,k(v)ϕm(β), l > 1, (3)
which is illustrated in Figure 1 (for l > 1). We use Fourier-Bessel (FB) bases for {ψk}k and Fourier bases for {ϕm}m. Specifically, ϕm(α) = eimα, and FB basis has the expression
ψk(r, θ) = cm,qJm(Rm,qr)e imθ, 0 ≤ r ≤ 1, 0 ≤ θ ≤ 2π, k = (m, q) (4)
wherem is the angular frequency, q the radial frequency, Jm the Bessel function (Rm,q the q-th root of Jm) and cm,q a normalizing constant (Abramowitz & Stegun, 1964). Both bases are “steerable”, i.e. the operation of rotation is a diagonalized linear transform under both bases. In the complexvalued version, ψk(Θtv) = e−im(k)tψk(v), ϕm(α − t) = e−imtϕm(α). This means that after the convolutions on R2 × S1 with the bases ψk(v)ϕl(α) are computed for all k and l, both up to certain truncation, the joint convolution (1), (2) with all rotated filters can be calculated by the algebraic manipulation of the expansion coefficients a(l)λ′,λ(k,m), and without any re-computation of the spatial-rotation joint convolution. Standard real-valued versions of the bases ψk and ϕm in sin’s and cos’s are used in practice. During training, only the expansion coefficients a’s are updated, and the bases are fixed.
Apart from the saving of parameters and computation, the bases truncation also regularizes the convolutional filters by discarding the high frequency components. As a result, DCF Net reduces response to those components in the input at all layers, which improves the robustness of the learned feature without affecting recognition performance. The visibly smoother trained filters in RotDCF are shown in Figure A.1, demonstrating the same regularization effects as in Qiu et al. (2018), the latter being without the rotation-equivariant setting. The theoretical properties of RotDCF Net, particularly the representation stability, will be analyzed in Section 3.

heading: 2.3 NUMBERS OF PARAMETERS AND COMPUTATION FLOPS
text: Number of trainable parameters: In a regular CNN, a convolutional layer of sizeL×L×M ′0×M0 (L×L being the patch size) has L2M ′0M0 parameters. In an equivariant CNN, a joint convolutional filter is of size L×L×Nθ×M ′×M , so that the number of parameters is L2NθM ′M . In a RotDCF Net, K bases are used in space and Kα bases across the angle α, so that the number of parameters is KKαM
′M . This gives a reduction of KL2 · Kα Nθ compared to non-bases equivariant CNN. In practice, after switching from a regular CNN to a RotDCF Net, typically M ≤ 12M0 or more due to the adoption of filters in all orientations. The factor KL2 is usually between 1 8 and 1 3 depending on the network and the problem (Qiu et al., 2018). In all the experiments in Section 4, Kα is typically 5, and Nθ = 8 or 16. This means that RotDCF Net achieves a significant parameter reduction from the non-bases equivariant CNN (the factor KKαL2Nθ is 1 8 or more), and even reduces parameters from a regular CNN by a factor of 12 or more. Computation in a forward pass: When the input and output are both W ×W in space, the regular CNN layer needs 2M ′0M0W
2L2 many flops, and a non-bases equivariant convolutional layer needs about 2M ′MW 2L2N2θ . In contrast, the computation in a RotDCF layer is dominated by a term of 2M ′MW 2KαKNθ, which is reduced from the non-bases equivariant network by a factor of K L2 · Kα Nθ
. Detailed calculations in Appendix A. In summary, RotDCF Net achieves a reduction of KL2 · Kα Nθ
from non-bases equivariant CNNs, in terms of both model size and computation. With typical network architectures, RotDCF Net may be of a smaller model size than regular CNNs. Numbers for specific networks are shown in Section 4.

heading: 3 THEORETICAL ANALYSIS OF DEEP FEATURES
text: This section presents two analytical results: (1) Joint convolution (1), (2) is sufficient and actually necessary to obtain rotation equivariance; (2) Stability of the equivariant representation with respect to input variations is proved under generic conditions, which is important in practice since rotations are never perfect.

heading: 3.1 GROUP-EQUIVARIANT PROPERTY
text: We consider the change of the l-th layer output when the input image undergoes some arbitrary rotation. Let rotation around point u0 by angle t be denoted by ρ = ρu0,t, i.e. ρu0,tu = u0 + Θt(u− u0), for any u ∈ R2, and the transformed image by Dρx(0)(u, λ) = x(0)(ρu0,tu, λ), for any λ ∈ [M0]. We also define the action Tρ on the l-th layer output x(l), l > 0, as
Tρx (l)(u, α, λ) = x(l)(ρu0,tu, α− t, λ), ∀λ ∈ [Ml]. (5)
The following theorem, proved in Appendix B, shows that the joint convolution scheme (1), (2) not only produces group-equivariant features at all layers in the sense of
x(l)[Dρx (0)] = Tρx (l)[x(0)], (6) but is necessary for a CNN with SO(2)-indexed channels to achieve (6). The sufficiency part is previously shown in Weiler et al. (2017). The necessity of the joint convolution motivates the design and the efforts of reducing the complexity of such models. Note that RotDCF is a type of the channel-indexed CNNs considered in the theorem, so it follows that RotDCF is group-equivariant. Theorem 3.1. In a CNN with SO(2)-indexed channels, let x(l)[x(0)] be the output at the l-th layer from input x(0)(u, λ). The relation (6) holds for all l if and only if the convolutional layers are given by (1), (2).

heading: 3.2 REPRESENTATION STABILITY UNDER INPUT VARIATIONS
text: Assumptions on the RotDCF layers. Following Qiu et al. (2018), we make the following generic assumptions on the convolutional layers: First,
(A1) Non-expansive sigmoid: σ : R→ R is non-expansive.
Second, we also need a boundedness assumption on the convolutional filters W (l) for all l:
(A2) Boundedness of filters: In all layers, Al ≤ 1,
where Al is defined as
Al := πmax{sup λ Ml−1∑ λ′=1 ‖a(l)λ′,λ‖FB, sup λ′ Ml−1 Ml Ml∑ λ=1 ‖a(l)λ′,λ‖FB}, (7)
‖a(1)λ′,λ‖ 2 FB = ∑ k µk(a (1) λ′,λ(k)) 2, ‖a(l)λ′,λ‖ 2 FB = ∑ k ∑ m µk(a (l) λ′,λ(k,m)) 2, l > 1, (8)
µk being the Dirichlet Laplacian eigenvalues of the unit disk in R2. Note that (A2) bounds the expansion coefficients, which implies a sequence of boundedness conditions on the convolutional filters in all layers (Proposition B.1), based upon which the stability results below are derived. Since µk typically increases in order, (A2) suggests truncating the series to only include low-frequency k and m’s, which is implemented in Section 4. The validity of the boundedness assumption can be qualitatively fulfilled by normalization layers which is standard in practice.
Non-expansiveness of the network mapping. Let the L2 norm of x(l) be defined as
‖x(l)‖2 = 1 Ml Ml∑ λ=1 1 |Ω| ∫ R2 ∫ S1 x(l)(u, α, λ)2dudα, l ≥ 1
and ‖x(0)‖2 = 1M0 ∑ λ 1 |Ω| ∫ R2 x
(0)(u, λ)2du. Ω is the domain on which x(0) is supported, usually Ω = [−1, 1]× [−1, 1] ⊂ R2. The following result is proved in Appendix B: Proposition 3.2. In a RotDCF Net, under (A1), (A2), for all l,
(a) The mapping of the l-th convolutional layer (including σ), denoted as x(l)[x(l−1)], is nonexpansive, i.e., ‖x(l)[x1]− x(l)[x2]‖ ≤ ‖x1 − x2‖ for arbitrary x1 and x2.
(b) ‖x(l)c ‖ ≤ ‖x(l−1)c ‖ for all l, where x(l)c (u, α, λ) = x(l)(u, α, λ)− x(l)0 (λ) (without index α when l=1) is the centered version of x(l) by removing x(l)0 , defined to be the output at the l-th layer from a zero bottom-layer input. As a result, ‖x(l)c ‖ ≤ ‖x(0)c ‖ = ‖x(0)‖.
Insensitivity to input deformation. We consider the deformation of the input “module” to a global rotation. Specifically, let the deformed input be of the form Dρ ◦Dτx(0), where Dρ is as in Section 3.1, ρ = ρu0,t being a rigid 2D rotation, and Dτ is a small deformation in space defined by
Dτx (0)(u, λ) = x(0)(u− τ(u), λ), ∀u ∈ R2, λ ∈ [M0], (9)
with τ : R2 → R2 is C2. Following Qiu et al. (2018), we assume the small distortion condition:
(A3) Small distortion: |∇τ |∞ = supu ‖∇τ(u)‖ < 15 , with ‖ · ‖ being the operator norm.
The mapping u 7→ u−τ(u) is locally invertible, and the constant 15 is chosen for convenience. With Tρ as defined in (5), the stability result is summarized as
Theorem 3.3. Let ρ = ρu0,t be an arbitrary rotation in R2, around u0 by angle t, and let Dτ be a small deformation. In a RotDCF Net, under (A1), (A2), (A3), c1 = 4, c2 = 2, for any L,
‖x(L)[Dρ ◦Dτx(0)]− Tρx(L)[x(0)]‖ ≤ (2c1L|∇τ |∞ + c22−jL |τ |∞)‖x(0)‖.
To prove Theorem 3.3, we firstly establish an approximate equivariant relation for all layers l (Proposition B.2), which can be of independent interest for estimating the image transformations. All the proofs are left to Appendix. Unlike previous stability results for regular CNNs, the above result allows an arbitrary global rotation ρ with respect to which the RotDCF representation is equivariant, apart from a small “residual” distortion τ whose influence can be bounded. This is also an important result in practice, because most often in recognition tasks the image rotation is not a rigid in-plane one, but is induced by the rotation of the object in 3D space. Thus the actual transformation of the image may be close to a 2D rotation but is not exact. The above result guarantees that in such cases the RotDCF representation undergoes approximately an equivariant action of Tρ, which implies consistency of the learned deep features up to a rotation. The improved stability of RotDCF Net over regular CNNs in this situation is observed experimentally in Section 4.

heading: 4 EXPERIMENTAL RESULTS
text: In this section, we experimentally test the performance of RotDCF Nets on object classification and face recognition tasks. The advantage of RotDCF Net is demonstrated via improved recognition accuracy and robustness to rotations of the object, not only with in-plain rotations but with 3D rotations as well. To illustrate the rotation equivariance of the RotDCF deep features, we show that a trained auto-encoder with RotDCF encoder layers is able to reconstruct rotated digit images from “circulated” codes. All codes will be publicly available.

heading: 4.1 OBJECT CLASSIFICATION
text: Non-transfer learning setting. The rotMNIST dataset contains 28× 28 grayscale images of digits from 0 to 9, randomly rotated by an angle uniformly distributed from 0 to 2π (Cohen & Welling, 2016a). We use 12,000 and 6,000 training samples, and 50,000 testing samples. The baseline net is
L2Nθ , c.f. Section 2.3. The case of maximum K and Kα (K=14, Kα=8 in
rotMNIST) is mathematically equivalent to using “full” filters without bases truncation.
a CNN with 6 convolutional layers (Table A.2), and we also compare with DCF (Qiu et al., 2018), which is non-rotation-equivariant but uses bases truncation, and SFCNN (Weiler et al., 2017), which is rotation-equivariant but does not use bases decompostion. The RotDCF net is made by replacing the regular convolutional layers with RotDCF layers, with Nθ many rotation-indexed channels and a reduced number of (unstructured) channelsM . K many ψk bases andKα many ϕm are used. The classification accuracy is shown in Table 3. We see that RotDCF net obtains improved classification accuracy from CNN with significantly reduced number of parameters, e.g., with 12K training, RotDCF Net with M=8, K=3, Kα=5 improves the test accuracy from 96.54 to 98.40 with less than 120 many parameters of the CNN model and 13 of the DCF model. The performance is also comparable to SFCNN which has significantly larger model complexity. The results are similar with reduced training size (5K) and on a shallower net with 3 conv layers, c.f. Table A.3.
The CIFAR10 dataset consists of 32 × 32 colored images from 10 object classes (Krizhevsky, 2009), and we use 10,000 training and 50,000 testing samples. The network architecture is modified from VGG-16 net (Simonyan & Zisserman, 2014) (Table A.4). As shown in Table 3, RotDCF Net obtains better testing accuracy with reduced model size from the regular CNN baseline model. Throughout these experiments, the higher accuracy is due to the group equivariance and the lower model complexity is due to the bases decomposition. Transfer learning setting. We train a regular CNN and a RotDCF Net on 10,000 up-right MNIST data samples, and directly test on 50,000 randomly rotated MNIST samples where the maximum rotation angle MaxRot=30 or 60 degrees (the “no-retrain” case). We also test after retraining the last two non-convolutional layers (the “ fc-retrain” case). To visualize the importance of image regions which contribute to the classification accuracy, we adopt Class Activation Maps (CAM) (Zhou et al., 2016), and the network is modified accordingly by removing the last pooling layer in the net in Table A.1. and inserting a “gap” global averaging layer. The test accuracy are listed in Table 2, where the superiority of RotDCF Net is clearly shown in both the “no-retrain” and “ fc-retrain” cases . The improved robustness of RotDCF Net is furtherly revealed by the CAM maps (Figure 2): the redcolored region is more stable for RotDCF Net even in the case with retraining.

heading: 4.2 IMAGE RECONSTRUCTION
text: To illustrate the explicit encoding of group actions in the RotDCF Net features, we train a convolutional auto-encoder on the rotated MNIST dataset, where encoder consists of stacked RotDCF layers, and the decoder consists of stacked transposed-convolutional layers (Table A.5). The encoder maps a 28×28 image into an array of 16× 32, where the first dimension is the discretization of the rotation angles in [0, 2π], and the second dimension is the unstructured channels. Due to the rotation equivariant relation, the “circulation” of the rows of the code array should correspond to the rotation of the image. This is verified in Figure 3: The top panel shows the code array produced
from a testing image, and the 16 row-circulated copies of it. The bottom panel shows the output of the decoder fed with the codes in the top panel.

heading: 4.3 FACE RECOGNITION
text: As a real-world example, we test RotDCF on the Facescrub dataset (Ng & Winkler, 2014) containing over 100,000 face images of 530 people. A CNN and a RotDCF Net (Table A.6) are trained respectively using the gallery images from the 500 known subjects, which are preprocessed to be near-frontal and upright-only by aligning facial landmarks (Kazemi & Sullivan, 2014). See Appendix C for data preparation and training details. For the trained deep networks, we remove the last softmax layer, and then use the network outputs as deep features for faces, which is the typical way of using deep models for face verification and recognition to support both seen and unseen subjects (Parkhi et al., 2015). Using deep features generated by the trained networks, a probe image is then compared with the gallery faces whose identities are known and classified as that of the top match.
Under this gallery-probe face recognition setup, we obtain 94.10% and 96.92% accuracy for known and unknown subjects respectively using the CNN model; using RotDCF, the accuracies are 93.42% and 96.92%. Testing on unknown subjects are critical for validating the model representation power over unseen identities, and the reason for higher accuracy is simply due to the smaller number of classes. For both cases, RotDCF reports comparable performance as CNN, while the number of parameters in the RotDCF model is about one-fourth of the CNN model (see Appendix).
In-plane rotation. This experiment demonstrates the rotation-equivariance of the RotDCF features. We apply in-plane rotations at intervals of π4 to the probe images (Figure 4), and let the original probe set be the new gallery, the rotated copies be the new probe set. In this setting, using the RotDCF model we obtain 97.04% and 97.58% recognition accuracy for known and unknown subjects respectively, after aligning the deep features by circular shifts (using the largest-magnitude α channel as reference). Notice that the model only sees upright faces. This is due to the rotation-equivariant property of the RotDCF Net, which means that the face representation is consistent regardless of its orientations after the group alignment. Lacking such properties, CNN obtains 0.54% and 5.05% recognition accuracies, which is close to random guess. We further compare CNN and RotDCF models via the CAM maps, which indicate the image regions relied by the CNN to make a classification prediction. In Figure 4, with regular CNN face regions used for prediction varies dramatically from image to image, and with RotDCF a significantly more consistent region mostly covering eyes and nose are selected across images, which explains its superior accuracy.
Out-of-plane rotation. To validate our theoretical result on representation stability under input deformations, we introduce out-of-plane rotations to the probe. Each probe image is fitted to a 3D face mesh, and rotated copies are rendered at the 10o intervals with -40o to 40o yaw, and -20o to 20o pitch, generating 45 synthesized faces in total (Figure 5). The synthesis faces at two poses (highlighted in red) are used as the new gallery, and all remaining synthesis faces form the new probe. The out-of-plane rotations here can be viewed as mild in-plane rotations plus additional variations, a situation frequently encountered in the real world. With this gallery-probe setup, the RotDCF model obtains 89.66% and 97.01% recognition accuracy for known and unknown subjects, and the accuracies are 80.79% and 89.97% with CNN. The CAM plots in Figure 6 also indicate that RotDCF Net chooses more consistent regions over CNN in describing a subject across different
poses. Since the out-of-plane rotations as in Figure 5 can be considered as in-plane rotations with additional variations, the superior performance of RotDCF is consistent with the theory in Section 3.

heading: 5 CONCLUSION AND DISCUSSION
text: This work introduces a decomposition of the filters in rotation-equivariant CNNs under joint steerable bases over space and rotations simultaneously, obtaining equivariant deep representations with significantly reduced model size and an implicit filter regularization. The group equivariant property and representation stability are proved theoretically. In experiments, RotDCF demonstrates improved recognition accuracy and better feature interpretability and stability on synthetic and realworld datasets involving object rotations, particularly in the transfer learning setting. To extend the work, implementation issues like parallelism efficiency and memory usage should be considered before the computational savings can be fully achieved. The framework should also extend to other groups and joint geometrical domains.

heading: ACKNOWLEDGEMENTS
text: Work partially supported by NSF, DoD, NIH and AFOSR.

heading: A FLOPS CALCULATION IN SECTION 2 AND CHOICE OF HYPERPARAMETERS
text: When the input and output are both W ×W in space, the forward pass in a regular convolutional layer needs M ′0M0W
2(1 + 2L2) ∼ 2L2M ′0M0W 2 flops. (Each convolution with a L × L filter takes 2L2W 2, and there are M ′0M0 convolution operations, plus that the summation over λ
′ takes W 2M ′0M0 flops.) In a rotation equivariant CNN without using bases, an convolutional layer would take ∼ 2M ′MW 2L2N2θ flops. In a RotDCF layer, the computation consists of three parts: (1) The inner-product with ϕm bases takes W 2M ′ · 2NθKα flops. (2) The spatial convolution with the ψk bases takes KαM ′K · 2L2W 2 flops. (3) The multiplication with aλ′,λ(k,m)e−im(k)α−imα and summation over λ′, k,m takes MNθ(4KKαM ′ + 2W 2KKαM ′) flops (real-valued version). Putting together, the total is 2M ′W 2Kα(Nθ + L 2K + MNθK), and when M is large, the third term dominates and it gives 2M ′MW 2KαKNθ. Thus the reduction by using bases-decomposed filters is again a factor of K L2 · Kα Nθ , and the relative ratio with a regular CNN is about M ′M M ′0M0 · KKαNθL2 .
The choice of hyperparameters K and Kα can be interpreted as the “frequency” truncation both in R2 and in S1, and it trades-off between filter regularization and filter expressiveness: The more severe the truncation, the less many trainable parameters, the smoother the filter (Figure A.1) and less overfitting, while the potential risk of underfitting is larger. This is revealed in Table 3: asK and Kα are reduced, the accuracy decreases monotonically in the rotMNIST experiment. When Kα is reduced to 3 then underfitting becomes noticeable and accuracy drops even more (not reported). The parameterNθ corresponds to spatial discretization along S1 and, since we adopt a spectral approach, it does not affect the number of trainable parameters, which is determined by Kα. In other words, the choice of Nθ is only constrained by memory storage.

heading: B PROOFS IN SECTION 3
text: B.1 THEOREM 3.1
Proof of Theorem 3.1. Note: The bases expansion under ψjl,k and ϕm does not affect the form of convolutional layers, but only impose regularity of the filters, thus the group-equivariant property of RotDCF follows from the theorem.
Observe that the equivariant relation (6) is equivalent to that
Tρx (l)[x(l−1)] = x(l)[Tρx (l−1)] (A.1)
for all l, where Tρx(0) means Dρx(0).
The sufficiency part: When l = 1, by (1),
Tρx (1)[x(0)](u, α, λ) = σ( ∑ λ′ ∫ R2 x(0)(ρu0,tu+ v, λ ′)W (1) λ′,λ(Θα−tv)dv + b (1)(λ)),
x(1)[Dρx (0)](u, α, λ) = σ( ∑ λ′ ∫ R2 x(0)(ρu0,t(u+ v), λ ′)W (1) λ′,λ(Θαv)dv + b (1)(λ)).
Since ρu0,t(u+v) = ρu0,t+Θtv, we have that Tρx (1)[x(0)] = x(1)[Dρx (0)] by a change of variable of Θtv 7→ v. When l > 1, by (2),
Tρx (l)[x(l−1)](u, α, λ) = σ( ∑ λ′ ∫ R2 ∫ S1 x(l−1)(ρu0,tu+ v, α ′, λ′)W (l) λ′,λ(Θα−tv, α ′ − α+ t)dvdα′ + b(l)(λ)),
x(l)[Tρx (l−1)](u, α, λ) = σ( ∑ λ′ ∫ R2 ∫ S1 x(l−1)(ρu0,t(u+ v), α ′ − t, λ′)W (l)λ′,λ(Θαv, α ′ − α)dvdα′ + b(l)(λ)).
Again, inserting ρu0,t(u+ v) = ρu0,t + Θtv, the claim follows by changing variables Θtv 7→ v and α′ − t 7→ α′. The necessity part: When l = 1, denote the general convolutional filter as w(1)(v;λ′, λ, α), and
x(1)(u, α, λ) = σ( ∑ λ′ ∫ R2 x(0)(u+ v, λ′)w(1)(v;λ′, λ, α)dv + b(1)(λ)).
Recall that
Tρx (1)[x(0)](u, α, λ) = σ( ∑ λ′ ∫ R2 x(0)(ρu0,tu+ v, λ ′)w(1)(v;λ′, λ, α− t)dv + b(1)(λ)),
x(1)[Dρx (0)](u, α, λ) = σ( ∑ λ′ ∫ R2 x(0)(ρu0,t(u+ v), λ ′)w(1)(v;λ′, λ, α)dv + b(1)(λ))
and then (A.1) with l = 1 holding for any x(0) implies that∑ λ′ ∫ R2 x(0)(ρu0,tu+v, λ ′)w(1)(v;λ′, λ, α−t)dv = ∑ λ′ ∫ R2 x(0)(ρu0,t(u+v), λ ′)w(1)(v;λ′, λ, α)dv.
By that ρu0,t(u+ v) = ρu0,t + Θtv, the above equality gives that
w(1)(v;λ′, λ, α− t) = w(1)(Θ−1t v;λ′, λ, α), ∀α, t ∈ S1.
Let t = α, and Fλ′,λ(v) = w(1)(v;λ′, λ, 0), we have that
w(1)(Θ−1α v;λ ′, λ, α) = Fλ′,λ(v),
and this gives that w(1)(v;λ′, λ, α) = Fλ′,λ(Θαv). This proves that (1) is necessary.
When l > 1, consider the general convolutional filter as w(l)(v;λ′, λ, α′, α). Using a similar argument, (A.1) implies that
w(l)(v;λ′, λ, α′, α− t) = w(l)(Θ−1t v;λ′, λ, α′ + t, α), ∀α, t ∈ S1.
Let t = α, and Fλ′,λ(v, α′) = w(l)(v;λ′, λ, α′, 0), then
Fλ′,λ(v, α ′) = w(l)(Θ−1α v;λ ′, λ, α′ + α, α),
which gives that w(l)(v;λ′, λ, α′, α) = Fλ′,λ(Θαv, α′ − α), which proves that (2) is necessary.
B.2 PROPOSITION 3.2
Proposition B.1. For all l,
B (l) λ′,λ, C (l) λ′,λ, 2 jlD (l) λ′,λ ≤ π‖a (l) λ′,λ‖FB,
where
B (l) λ′,λ := ∫ R2 ∫ S1 |W (l)λ′,λ(v, β)|dvdβ, l > 1, B (1) λ′,λ := ∫ R2 |W (1)λ′,λ(v)|dv
C (l) λ′,λ := ∫ R2 ∫ S1 |v||∇vW (l)λ′,λ(v, β)|dvdβ, l > 1, C (1) λ′,λ := ∫ R2 |v||∇vW (1)λ′,λ(v)|dv
D (l) λ′,λ := ∫ R2 ∫ S1 |∇vW (l)λ′,λ(v, β)|dvdβ, l > 1, D (1) λ′,λ := ∫ R2 |∇vW (1)λ′,λ(v)|dv
(A.2)
As a result, Bl, Cl, 2 jlDl ≤ Al,
where
Bl := max{sup λ Ml−1∑ λ′=1 B (l) λ′,λ, sup λ′ Ml−1 Ml Ml∑ λ=1 B (l) λ′,λ},
Cl := max{sup λ Ml−1∑ λ′=1 C (l) λ′,λ, sup λ′ Ml−1 Ml Ml∑ λ=1 C (l) λ′,λ},
Dl := max{sup λ Ml−1∑ λ′=1 D (l) λ′,λ, sup λ′ Ml−1 Ml Ml∑ λ=1 D (l) λ′,λ},
(A.3)
and thus (A2) implies that Bl, Cl, 2jlDl ≤ 1 for all l.
Proof of Proposition B.1. The proof for the case of l = 1 is the same as Lemma 3.5 and Proposition 3.6 of Qiu et al. (2018). We reproduce it for completeness. When l = 1, it suffices to show that for F (v) = ∑ k akψk(v),∫
|F (v)|dv, ∫ |v||∇F (v)|dv, ∫ |∇F (v)|dv ≤ π( ∑ k µka 2 k) 1/2. (A.4)
Rescaling to ψjl,k in v leads to the desired inequality with the factor of 2 jl for D(l)λ′,λ. To prove (A.4), observe that F is supported on the unit disk, and then ‖F‖1, ∫ |v||∇F (v)|dv ≤ ‖∇F‖1 ≤√
π‖∇F‖2, where ‖∇F‖22 = π ∑ k µka 2 k due to the orthogonality of ψk.
For l > 1, similarly, we only consider the rescaled filters supported on the unit disk in v. Let F (v, β) = ∑ k,m ak,mψk(v)ϕm(β), β ∈ S1, similarly as above, we have that∫ ∫ |F (v, β)|dvdβ, ∫ ∫ |v||∇vF (v, β)|dvdβ ≤ ∫ ∫ |∇vF (v, β)|dvdβ
≤ (π ∫ ∫ |∇vF (v, β)|2dvdβ)1/2
recalling that ∫ dβ on S1 has the normalization of 12π . Again, ∫ ∫ |∇vF (v, β)|2dvdβ =
π ∑ k,m µka
2 k,m due to the orthogonality of ψk and ϕm. This proves that∫ ∫
|∇vF (v, β)|dvdβ ≤ π( ∑ k,m µka 2 k,m) 1/2,
which leads to the claim after a rescaling of v.
Remark 1 (Remark to Proposition 3.2). The proposition only needs Bl, defined in (A.3), to be less than 1 for all l, in a rotation-equivariant CNN, which is implied by (A2) by Proposition B.1.
Proof of Proposition 3.2. The proof is similar to that of Proposition 3.1(a) of Qiu et al. (2018). Specifically, in (a), the argument is the same for l = 1, making use of the fact that∫
|w(Θαv)|dv = ∫ |w(v)|dv, ∀α ∈ S1
and ∫ S1 dα = 1 due to the normalization of 12π . For l > 1, the same technique proceeds with
the new definition of B(l)λ′,λ as in (A.2) which involves the integration of ∫ S1
(· · · )dβ. The detail is omitted.
To prove (b), we firstly verify that x(l)0 only depends on λ. When l = 1, x (1) 0 (u, α, λ) = σ(b (1)(λ)). Suppose that it holds for (l − 1), consider l > 1,
x (l) 0 (u, α, λ) = σ( ∑ λ′ ∫ S1 ∫ R2 x (l−1) 0 (u+ v, α+ β, λ ′)W (l) λ′,λ(Θαv, β)dvdβ + b (l)(λ))
= σ( ∑ λ′ x (l−1) 0 (λ ′) ∫ S1 ∫ R2 W (l) λ′,λ(Θαv, β)dvdβ + b (l)(λ))
= σ( ∑ λ′ x (l−1) 0 (λ ′) · ∫ S1 ∫ R2 W (l) λ′,λ(v ′, β)dv′dβ + b(l)(λ))
= x (l) 0 (λ).
Thus x(l)0 (u, α, λ) = x (l) 0 (λ) for all l (without index α for l = 1). The rest of the argument follows from that ‖x(l)c ‖ = ‖x(l) − x(l)0 ‖ = ‖x(l)[x(l−1)]− x(l)[x (l−1) 0 ]‖ ≤ ‖x(l−1) − x (l−1) 0 ‖ = ‖x (l−1) c ‖, where the inequality is by (a).
B.3 THEOREM 3.3
Proposition B.2. In a RotDCF Net, under (A1), (A2), (A3), c1 = 4, for any l,
‖x(l)[Dρ ◦Dτx(0)]− Tρ ◦Dτx(l)[x(0)]‖ ≤ 2c1l|∇τ |∞‖x(0)‖,
where Dτ only acts on the space variable u of x(l) similar to (9).
Proof of Proposition B.2. We firstly establish that for all l,
‖x(l)[Tρ ◦Dτx(l−1)]− Tρ ◦Dτx(l)[x(l−1)]‖ ≤ 2c1|∇τ |∞‖x(l−1)c ‖, (A.5)
where Tρ is replaced by Dρ if applies to x(0) which does not have index α. This is because that
x(l)[Tρ ◦Dτx(l−1)] = Tρx(l)[Dτx(l−1)] by Theorem 3.1, and that
‖Tρx(l)[Dτx(l−1)]− Tρ ◦Dτx(l)[x(l−1)]‖ = ‖x(l)[Dτx(l−1)]−Dτx(l)[x(l−1)]‖ by the definition of Tρ (a rigid rotation in u, and a translation in α). This term can be upper bounded by c1(Bl + Cl)|∇τ |∞‖x(l−1)c ‖ (Lemma B.3), which leads to the desired bound under (A2) by Proposition B.1.
The rest of the proof is similar to that of Proposition 3.3 of Qiu et al. (2018): Write x(l)[Dρ ◦ Dτx
(0)] − Tρ ◦ Dτx(l)[x(0)] as the sum of the differences x(l)[x(j)[Dρ ◦ Dτx(j−1)]] − x(l)[Tρ ◦ Dτx
(j)[x(j−1)]] for j = 1, · · · , l. The norm of the j-th term is bounded by ‖x(j)[Dρ ◦Dτx(j−1)]− Tρ ◦Dτx(j)[x(j−1)]‖ due to Proposition 3.2 (a), which, by applying (A.5) together with Proposition 3.2 (b), can be bounded by 2c1|∇τ |∞‖x(0)‖. Summing over j gives the claim.
Proof of Theorem 3.3. The proof is similar to that of Theorem 3.8 of Qiu et al. (2018). With the bound in Proposition B.2, it suffices to show that
‖Tρ ◦Dτx(L)[x(0)]− Tρx(L)[x(0)]‖ ≤ c22−jL |τ |∞‖x(0)‖.
By the definition of Tρ, the l.h.s. equals ‖Dτx(L)[x(0)]−x(L)[x(0)]‖, which can be shown to be less than c2|τ |∞DL‖x(l−1)c ‖ by extending the proof of Proposition 3.4 of Qiu et al. (2018), similar to the argument in proving Lemma B.3. The desired bound then follows by that DL ≤ 2−jLAL ≤ 2−jL (Proposition B.1 and (A2)) and that ‖x(l−1)c ‖ ≤ ‖x(0)‖ (Proposition 3.2 (b)).
Lemma B.3. In a rotation-equivariant CNN, Bl, Cl defined as in (A.3), under (A1), (A3), for all l > 0, with c1 = 4, x (l) c as in Proposition 3.2,
‖x(l)[Dτx(l−1)]−Dτx(l)[x(l−1)]‖ ≤ c1(Bl + Cl)|∇τ |∞‖x(l−1)c ‖.
Proof of Lemma B.3. The proof is similar to that of Lemma 3.2 of Qiu et al. (2018). Specifically, when l = 1, the argument is the same, making use of the fact that ∫ |w(Θαv)|dv =∫
|w(v)|dv, ∀α ∈ S1 and ∫ S1 dα = 1 due to the normalization of 12π . When l > 1, the same
technique applies by considering the joint integration of ∫ R2 ∫ S1
(· · · )dvdβ instead of just dv. The only difference is in using the new definitions ofB(l)λ′,λ and C (l) λ′,λ for l > 1 as in (A.2), both of which
involve the integration of ∫ S1 (· · · )dβ. The detail is omitted.

heading: C EXPERIMENTAL DETAILS IN SECTION 4
text: C.1 OBJECT RECOGNITION WITH ROTMNIST AND CIFAR10
In the experiments on rotMNIST dataset, the network architecture for Conv-3 and Conv-6 nets are shown in Table A.1 and Table A.2. Stochastic gradient descent (SGD) with momentum is used to train 100 epochs with decreasing learning rate from 10−2 to 10−4.
In the experiments on CIFAR10 dataset, the VGG16-like network architecture is shown in Table A.4. SGD with momentum is used to train 100 epochs with decreasing learning rate from 10−2 to 10−4.
C.2 CONVOLUTIONAL AUTO-ENCODER FOR IMAGE RECONSTRUCTION
The network architecture is shown in Table A.5. The network is trained on 50,000 training samples, the training set is augmented by rotating each sample at 8 random angles, producing 400k training set. The network is trained for 10 epochs, where the learning rate decreases from 10−3 to 10−6.
C.3 FACE RECOGNITION ON FACESCRUB
To facilitate the evaluation on both known and unknown subjects, we select the first 500 of the 530 identities as our training subjects. The remaining 30 subjects are used for validating out of sample performance, namely the unknown subjects. The experiment on unknown subjects is critical for face models to generate over unseen people. For both known and unknown subjects, we hold 10 images from each person as the probe images, and the remaining as the gallery images. The images are preprocessed by aligning facial landmarks using Kazemi & Sullivan (2014). and crop the aligned face images to 112 × 112 with color. Thus, both our CNN and RotDCF models are trained with near-frontal and upright-only face images.
The network architecture is shown in Table A.6. According to the formula in Section 2, the number of trainable parameters in the RotDCF Net is about ( 12 ) 2 · KL2 ·Kα = 1 4 of that of the CNN.
C.4 EFFECTS OF REGULARIZATION BY FILTER DECOMPOSITION
Sample trained filters in the reconstruction experiment are shown in Figure A.1, which shows that RotDCF obtains smoother trained filters as a result of the regularization applied by the bases truncation. In the reconstruction experiment, increasing the number of bases degrades the synthesis (Right
in Figure A.1). Such improved stability due to the truncated decomposition supports the analysis in Section 3.

Review summaries:
This paper combines the benefits of using joint steerable filters (using the SO(2) group) for designing rotation-equivariant CNNs with those of decomposing the filters (using Fourier-Bessel bases) for reducing the computational complexity. In addition, this leads to a compressed model and filter regularization. The authors give theoretical guarantees on the rotation equivariance and representation stability with respect to in and out of plane rotation. Empirical results show that the model attains better accuracy compared to CNNs and non-rotation-equivariant deep networks while using fewer parameters and also performs similarly to a rotation-equivariant model with much bigger capacity.

Group-equivariant deep networks are used as a solution for rotation-equivariance in CNNs. However, they are computationally expensive as the number of filters increases by a factor proportional to the number of groups. Inspired by ideas of filter decomposition used in CNN model compression, the authors of this work instead propose to use steerable filters across space and rotation, as basis filters for achieving rotation-equivariance, which leads to computational efficiency.