FILTRA: Rethinking Steerable CNN by Filter Transform
Authors: Bo Li
Qili Wang
Gim Hee Lee
Venue:  (2021)

Abstract: Steerable CNN imposes the prior knowledge of transformation invariance or equivariance in the network architecture to enhance the the network robustness on geometry transformation of data and reduce overfitting. Filter transform has been an intuitive and widely used technique to construct steerable CNN in the past decades. Recently, group representation theory is used to analyze steerable CNN and reveals the function space structure of a steerable kernel function. However, it is not yet clear on how this theory is related to the filter transform technique. In this paper, we show that kernel constructed by filter transform can also be interpreted in the group representation theory. Meanwhile, we show that filter transformed kernels can be used to convolve input/output features in different group representation. This interpretation help complete the puzzle of steerable CNN theory and provides a novel and simple approach to implement steerable convolution operators. Experiments are executed on multiple datasets to verify the feasibilty of the proposed approach.


Under review as a conference paper at ICLR 2021
FILTRA: RETHINKING STEERABLE CNN BY FILTER TRANSFORM
Anonymous authors Paper under double-blind review
ABSTRACT
Steerable CNN imposes the prior knowledge of transformation invariance equivariance in the network architecture to enhance the the network robustness on geometry transformation of data and reduce overfitting. Filter transform has been an intuitive and widely used technique to construct steerable CNN in the past decades. Recently, group representation theory is used to analyze steerable CNN and reveals the function space structure of a steerable kernel function. However, it is not yet clear on how this theory is related the filter transform technique. In this paper, we show that kernel constructed by filter transform can also be interpreted in the group representation theory. Meanwhile, we show that filter transformed kernels can be used to convolve input/output features in different group represen- tation. This interpretation help complete the puzzle of steerable CNN theory and provides novel and simple approach to implement steerable convolution opera- tors. Experiments are executed on multiple datasets to verify the feasibilty of the proposed approach.
INTRODUCTION
Beyond the well-known property of equivariance under translation, there has been substantial recent interest in CNN architectures that are equivariant with respect other transformation groups, e-g- reflection and rotation. Applications of such architectures range over scenarios where object orien- tation might variate, including OCR, aerial image processing, 3D point cloud processing, medical image processing, texture analysis and etc.
Previous works on constructing equivariant CNN can coarsely categorized as two aspects. The first aspect designs special steerable filters so that the convolutional output is hard-baked to trans- form accordingly when the input reflects or rotates. A plenty of works develop this idea by filter rotation, including hand-crafted filters (Oyallon & Mallat, 2015) and learned filters (Laptev et al., 2016; Zhou et al., 2017; Cheng et al., 2018: Marcos et al., 2017). TI-Pooling (Laptev et al., 2016) produce invariant output input rotates. ORN (Zhou et al., 2017) and RotDCF (Cheng et al., 2018) produces output which circularly shifted as input rotates. Since each dimension of such permutable output corresponds uniformly discrete rotation angle, RotEqNet (Marcos et al., 2017) propose to extract rotation angle from the permutable features. Another approach to construct steerable fil- ters is linearly combine set of steerable bases. These basis can be solved in discrete function space (Cohen & Welling, 2014; 2016) or continuous function space (Worrall et al., 2017; Weiler & Cesa, 2019). Weiler & Cesa (2019) comprehensively summarize works on steerable bases using polar Fourier basis.
The second aspect exploits specific transforms act on input. Spatial Transformer Network (STN) is a well-known representative, which predicts an affine matrix to transform its input to the canonical form. Tai al. (2019) inherits this idea to design equivariant network. Another choice of transform is to the polar coordinate system (Henriques & Vedaldi, 2017; Esteves et al., 2018). Since 2D rotation in Cartesian coordinate system corresponds to 2D translation in polar coordinate system, rotation equivariance be achieved by conventional translation equivariant CNN.
The approach proposed in this paper falls into the first category. Weiler & Cesa (2019) proves that all steerable convolutional operator could be denoted as the combination of a specific set of polar Fourier bases. However, it is not clear yet how this interpretation is related with the widely used filter transform scheme. In this paper, aim establish the missing connection between the

---

Under review as a conference paper at ICLR 2021
group representation based analysis for steerable filters and filter transform scheme. To this end, propose a new approach (FILTRA) to use filter transform to establish steerability between features in different group representation in cyclic group Cv and dihedral group Dv. We verify the feasibility of FILTRA for the classification and regression tasks on different datasets.
PRELIMINARIES
We make use several NumPy or SciPy functions in equations including roll!, flipud and ci circulant'. We omit the variable in bracket sometimes by writing K = (g) and K; = (0))
2.1 STEERABLE CNN
We recapitulate the basic concepts steerable CNN which will be frequently used in this pa- per. For detailed introduction, readers can refer to Weiler & Cesa (2019) for comprehensive information. We mainly consider the 2D im- age case and denote IK² as a pixel coordi- nate. We use vector field f(z) E RC to denote general multi-channel image, where C is the number of channels. Typical examples of [(x) include RGB image f(x) E R& and gradient image (x) E R22 Consider group G of trans- formations and an element E G. Examples of G include rotation, translation and flip. A vector field f(x) follows the below rules when undergoing the act (g)) of group element g:
f(x)
T(g) f f=
g
p=1
p= 4/2
Figure 1: Examples of images (feature maps) with different group representation p. Both images un- dergo 90deg rotation. The upper row an RGB image whose 3-channel colors remain the same when the image is rotated. The lower row is a gra- dient image whose two channel value should be rotated the same way when the gradient image rotated.
(g)) f= plg)s(g
(1)
where p(g) is group representation related to vector field f. Fig. shows an example of dif- ferent types of p for RGB images and gradient images under a rotation transform element g. The group representation of RGB is p(g) = 1 while for gradient image p(g) a 2D rotation matrix which also rotates vector f(x) by
In the scenario of convolutional neural network, convolution operator ++ K f is considered as steerable if it satisfies
M (g) カo (g)((<: f/,
ie. the output vector field transforms equivariantly under when the input is transformed by g.
2.2 REFLECTION GROUP, CYCLIC GROUP AND DIHEDRAL GROUP
We consider steerable filters reflection group ((=1), *), cyclic group Cx and dihedral group Dy ({#1},+) x Cy. To unify the notations in derivation, we interpret Cx = (f1), )) Cx and ([=1), *) (f=1),* *) x C₁ D2 so that element in these three groups can always be denoted as pair g (40, d1), whose range is Z, x E, for reflection group, 2, x En for cyclic group and Zg x n for dihedral group. Each element in Cy corresponds to rotation angle 0, = 2i
N.
2.3 GROUP REPRESENTATION
linear representation p of a group on vector space R. a group homomorphism from G to the general linear group Gn., denoted as
p: G - GT.(m)
Aog') p(9) Vg.g' G.
(3)
http:://wwwyyooooooea.e/eeeeeeer html
1i ci reulantnnt... ant..hm)
2

---

Under review as a conference paper at ICLR 2021
We consider three types of linear representation in this paper, ie. trivial representation, regular representation and irreducible representation (irrep). Readers can refer to Serre (1977) for further background for these concepts.
The trivial representation of a group element is always (mn(g) 1. The regular representation of finite group acts on a vector space [ by permuting its axis. Therefore, for a rotation element g (0,81) Cn DN, get
Preg
Cx (g) P(ii), Pieg
DN
(g)
P(h1) P(ii)
O
0
where P(ii) rol (Iv,i..0)
(4)
For reflected element = (1, E DN, we get
Preg
DN (g) = B(i1)
0
B(i1) where B(h) = fl (lipud(((-á) 1))-
(5)
By selecting suitable change of basis of the vector space, representation be converted equivalent representation, which is the direct sum of several independent representations on the orthogonal subspaces. A representation is called irreducible representation no non-trivial decom- position exists. This conversion is denoted as
p(g) = Q + (1or)))e [g) Q-
where 7 is an index set specifying the irreducible repsentations vA and Q is the change of basis.
2.4 DECOMPOSING REGULAR REPRESENTATION
We decompose the regular representation into a set of irreps. Define the following base irrep
(( ( (-1)a ((-19)4
k N N is even
k = 2
(1.. (1)
(7)
(oos(k0,)) sin(kO,,) )
((-1)))
otherwise
sin(kO,,) cos(k0,,)
(-1)%)
where j, k referred the reflection and rotation frequency of the irrep. Concretely, if the action reflects/rotates an object once, (9), will reflects/rotates vector space j/k times. We also define the following discrete cosine transform basis
[30
31
91
1N
[ko)(k(p)
cos(k0,)
cos(kO0 N 2
is even
(8)
where
Bg- =
Coos( (k0o) cos(k))) cos(k0
otherwise
sin(k(o)
sin(k(1)
sin(k0x-1)
The following decomposition for Prog 21) holds
Preg (g) VVDVVT DCN
Vo (0, i1).
(9)
0
Dv (io, i1) holds in a bit more complicated form, i.e. The decomposition for Pag
(io. i1) WDD*WT
Preg
where
W V
V
V
DDx
iio
(10)
1
and each column of W is refered by 3j decomposition.
(-138T)
See Fig. 2 for visualization of this
We also mention property of 3& that is easy to verify and will be useful in our derivation.
Voo (0, TT) = BT P(t); (0, = 3 P(iz), (1, (1) S. B(i), (1. = = B(i), where (1o. i1) rotates column vectors of BJ if they circularly shifted.
(11a)
(11b)
3

---

Under review as a conference paper at ICLR 2021
y Preg
(g)
W
DDx
W
D Preg
(g)
W
DD*
W
Figure 2: Illustrations for (10) for g (0, 1) left and g (1, 1) at right. Red, light yellow and green denotes negative, 0 and positive values, respectively.
2.5 HARMONIC FILTERS
Weiler et al. (2018) proposes the condition of filter kernel K. to be equivariant under the action g E G.
Lemma 1. The map f r> R. f is equivariant under G if and only if for all g E G.
<(gx) Poour (Puu(g)) (oul))))())))
(12)
Weiler & Cesa (2019) proves that such filters can be denoted by series of harmonic bases b(Ø), i.e.
<(r, Φ)
Σ (r(r)b(o).
(13)
where (r) is the per radial weights and K. is a set of harmonic hases dervied in the appendix of Weiler & Cesa (2019). For example, consider Pin = fi and Poot = V) in DN
K = (0) = ((O))(()) m sn, {+1}}.
(14)
3
MAIN RESULTS
(12) and (13) provide a general approach to verify and construct steerable CNN with different rep- resentations. In this section, relate these theories with filter transform and show how to use filter transform to construct steerable filters with input/output of different representations.
For readers who not interested in group theory and mathematical derivation of the theory nection, we highlight the key equations to construct steerable filters in boxes. It should not be difficult implement steerable filters directly from these equations using any modern deep learning framework. Fig. 3 shows illustration for these equations.
In our derivation, we mainly consider the angular coordinate of polar coordinate functions x(r, Φ) and write them k()). We will also frequently make use of the following property:
x( 0o) = x( + 0o), k( 0:) x(Ø + ON-
(15)
3.1 FROM TRIVIAL REPRESENTATION TO REGULAR REPRESENTATION
Rotation Group Cn Consider the the rotating filter K and its reflected version K which are com- monly used in previous works, e.g. TI-Pooling, ORN, RotEqNet and RotDCF:
K(Ø) [x
K
K (Ø) = k( - 6n),
(16)
((Ø) [
R1
R RN-1
K (Ø) = k(,, - 6).
The output of convolution with the above kernels naturally permutes as the input rotates in Cy- This intuitively corresponds property of steerable filter transforming from trivial representation regular representation. In this paper, we use K and K the basic filters construct different types of steerable filters in CN and Dy. We verify the observation of the above steerability by substituting into the Ihs of Lemma with g (0, 1) and write:
K(Φ + 01)
[x( + 01) 0
N-2 = Nx 60
N-2
(17a)
Prog (0, 1)Kpui (0. 1)
Cv
(17b)

---

Under review as a conference paper at ICLR 2021
O--reg K
K
N Ko-rog
KCy Hre
diag(K))g
diag (K)33
DD rek
diag(K).8)
diag K)
Figure 3: Visualization of FILTRA filter examples. Based on weight kernel K, generate
0-rrg- K-rreg K and K ky j.k-rre- In this example j 1, N, 8. The two-columns of matrix 3x is splitted as 3 and 3) for visualization. Red, light yellow and green denotes negative, 0 and positive values, respectively. Please view this figure in color.
filters K KCv KDx
The above equation can be similarly verified for other g (0, and also on K. Thus WLOG select the steerable filter which transforms trivial representation to regular representation on Cy as
KCN
K.
(18)
Dihedral Group D The steerable filter that transforms trivial representation to regular represen- tation on Dx be constructed
K K k
0 ())
(19)
which corresponds enumerating each D element and act the kernel For (0, (1), Ko- can be verified to follow (12) in the same way as (17a), i.e. KD Ko (x+0) = pDay (g)k D.v
O-rrggPt
For reflected action, when (1, 1), we write:
K( (- 01) = [r[[- + 01) n(-6 - 6o) k(-6-11 -1)
<(-o - 0N-2)]
(20)
= F
neD N-1
F2
B(1)K.
Similarly, show for (1,4));
K(-Ø+ + 0.,) B(iiK, ((-o+0,,) B
(21)
Thus we verify (12) for the reflected actions = (1, i1) by summarizing the above as K Dv
0 (-Ø
+
0,, = PD (eg)
3.2
FROM IRREP TO REGULAR REPRESENTATION
Rotation Group Cx Consider a Cy irrep (g) with frequency (0, k). We show that the fol- lowing kernel
kCv k-rrog di)
(22)
transforms from (g) to regular representation for the action = (0, 21) The derivation of correctness can be found in the appendix.
5

---

Under review as a conference paper at ICLR 2021
Dihedral Group D Consider D irrep (ss((to with frequency (f. We show that the following kernel:
T
kCN T reg
(-1)²
(23)
k-ree
transforms from (io to regular representation for the action (10,21) DN.
3.3 FROM REGULAR REPRESENTATION TO REGULAR REPRESENTATION
Regular representation possesses nice property that it can be averaged, pooled or activated channel- wise without violating stecrability (Weiler & Cesa, 2019). Thus it is convenient to used regular representation for the intermediate features of a steerable CNN. We show in this subsection that the following kernels can be use to construct a steerable kernel whose input and output features are both in regular representation.
Rotation Group Cn
KCN reg
CCN O
KC -tng
V-1
(24)
Dihedral Group DN
KDY reg-
KDb Ko,0-rro
KDN O,
KDx K.0-ro
KDN 1.14 nug
W-1
(25)
The above two kernels can be verified to transform regular representation to regular representation in similar way and we show the derivation for the Cn case (25) as an example the appendix.
3.4 REVERSED TRANSFORM OF REPRESENTATIONS
It is obvious to find that for (12), if Pin. Pout orthogonal matrices, i.c. Pin - the transpose of (12) naturally proves the equivariance of k under reversed representation trans- form direction, i.e. from Pour to Pin- Thus we can easily obtain equivariance kernel from regular representation to trivial/inreducible representation by simply transposing (18), (19), (22) and (23).
Pin Pout Poun
3.5 CONVENTIONAL ROTATING FILTERS
We comprehensively study the approach to use filter rotation to form steerable convolutional kernels with regular representation features as input or output. Conventional filter rotation based networks exploit some basic forms introduced in this section. Tl-Pooling (Laptev et al., 2016) exploits kemel KCx to transform trivial to regular representation, executes orientation pooling to convert regular to trivial representation and loses orientation information. RotDCF and ORN exploits kernel of form
CCv circulant(K)..
(26)
It is easy to verify that KNN also follows Lemma 1 to be a steerable filter. However, compared to
KCN Keeg- KCy consumes same filter storage but has less weight capacity (N v.s. N )). RotEqNet constructs 2D vector field which could rotate as its input rotates but regards the 2D vector field independent trivial representation in convolution. As shown in this paper, it preserves better steerability to regards the vector field as irrep representation with frequency 1.
3.6 NUMERICAL ACCURACY FOR DISCRETE KERNELS
Note that when implementing discrete convolution, the equality of (17a) does not perfectly hold. For example, consider x² (Ø) x(ő 0.), (0,) x(0) holds for continuous k. However, for discrete K, (Ø) is a rotated interpolation k(Φ) and this equality does not precisely hold in general. There exist
exceptions where the equality
One be achieved for discrete
example is when x² (Ø) is a 90" rotation к and it can be precisely constructed from K. Another
example is when n is 45 rotation interpolated by nearest pixel from of size 3 x 3.

---

Under review as a conference paper at ICLR 2021
layer
output
of (FIL.)
88 (R2)
layer
k
output
conv+relu
5
128 (reg)
0.12
0.17
GroupPool
24 (reg)
conv+relu
5
1
192 (reg)
0.13
0.13
fc+relu
16 (reg)
pool
3
256 (reg)
fc
10 (tri)
conv+relu
5
256 (reg)
0.13
0.13
conv+relu
5
384 (reg)
0.23
0.23
(b) The classification head network struc-
ture used in our experiments uses a Group- ing Pooling (Weiler & Cesa, 2019) to gener-
pool
3
2
384 (reg)
convtrelu
5
1
512 (reg)
0.32
0.48
ate transform invariant features.
convtrelu
5
768 (reg)
0.62
0.91
pool
768 (reg)
layer
$
output
PAMaxPool conv+relu
24 (reg)
(a) The backbone network structure used in our experiments is composed by convolution, ReLU and pooling layers. The convolution layers realized by FILTRA, R2Conv and con- ventional convolution respectively while the rest layers re- main the same. Three realizations have the same number of output channels in each layer but organize the channels to be follow regular representation for FILTRA and R2Conv. k: kernel size. stride. öt: filter generation time
16 (reg)
conv
2 (irrep)
(c) The regression head network struc- ture used in our experiments uses
PointwiseAdaapiieeMaaPoo Pool) (Weiler Cesa, 2019) to summarize feature in regular representation.
(PAMax-
Table 1: Network structure in experiments
3.7 STEERABLE CNN WITH MULTIPLE LAYERS
conventional CNN usually composed convolution, pooling, nonlinearity and fully-connected layers. To achieve equivariance for the overall network, it is desired that all the component layers are steerable. As analyzed the appendix of Weiler & Cesa (2019), channel-wise nonlinearity and channel-wise pooling preserves the steerability on feature maps with regular representation. fully- connected layers is a special case of convolution with x 1 kernels and thus can be easily realized by steerable convolution.
EXPERIMENTS
The proposed equivariant convolution, refered as FILTRA, can interpreted as an alternative for- mulation for the harmonic based (Weiler & Cesa, 2019) implementation of steerable convolution. In this section we show the pros and cons of each implementation by experiments. We make use of the framework E2CNN (Weiler & Cesa, 2019) for our experiments as it provides the general interface and operations for steerable CNN network. Experiments are executed on the MNIST, KM- NIST (Clanuwat et al., 2018), FashionMNIST (Xiao et al., 2017), EMNIST (Cohen et al., 2017) and CIFAR 10 datasets.
We compare FILTRA against convolution operations, i.e. the representative harmonic based convolution R2Conv (Weiler& & Cesa, 2019) from E2CNN and the conventional vanilla convolution. All MNIST-like datasets are experimented on a same feature extraction backbone as described in Table la, with convolution operator realized by the three experimented approaches. CIFAR 10 is experimented with WideResNet (Zagoruyko & Komodakis, 2016) in the setting similar Weiler & Cesa (2019). We found that on CIFAR10, C4 steerable network performs better than Cg for both approaches. For all experiments, we randomly rotate or reflect according to the experiment settings. The settings and evaluation results are listed in Table 2. Different from Weiler et al. (2018), we force the three convolution kernels to output same number of channels. For example, compared to vanilla convolution, the number of free weights for Cs FILTRA is reduced to 1/8 and for a Dg is reduced to 1/16. The filters for all the approaches will thus have exactly same shape at the deploy stage.
Experiments are executed on GTX 2070. The training procedure of FILTRA and R2Conv can both be implemented as a vanilla convolution plus a filter generation step. For C3 case the runtime of both generator is similar and for D& case FILTRA is slightly faster. We show runtime of Dg case in Table la at training stage. R2Conv additionally requires a initialization of about min. Both of the approaches consume the same inference time of vanilla convolution.
7

---

Under review as a conference paper at ICLR 2021
Classification (aec)
Regression (angle deg)
Tasks
mnist
kmmist
fmnist
emnist
cifarl0
mnist
Kmnist
fmnist
emnist
Aug
S
0
S
0
S
0
S
0 wrn
wrn
S
0
S
0
S
0
S
0
eqiv
Cs
D3
Ca
D>s
Cs
Dx
Cx
Ds
C4
D4
Os
Ds
C8
D3
Cs
D3g
Ca
Ds
FILTRA
98.9
98.
97.1 97.0
90.5
90.8
77.1 80.5
93.4
92.8
3.3
3.2
3.6
2.8
29.8
24,9
R2Conv
98.8
98.1
97.3
96.8
90.5
90.8
76.7
80.1
93.6
92.7
4.8
8.9
3.4
4.5
2.9
3.7
34.5
29.2
Conv
98.5 98.0
96.4 95.2
89.3 88.3
72.6
80.1
93.2
6.6
10.6
4.8
6.4
3.1 3.6
37.4 25.5
Table 2: Performance on MNIST and CIFARIO. S: randomly augmented over SO(2). O: randomly
augmented over ((2). wrn: WideResNet. Zagoruyko & Komodakis (2016).
4.1 CLASSIFICATION TASK
The most typical experiment used in previous works on conventional steerable CNN is the clas- sification task. We follow this convention and compare the classification performance of the ex- perimented three approaches in Table 2. FILTRA show comparable performance to R2Conv and slightly improves accuracy for OCR- (*MNIST) tasks where high frequency texture limited. On CIFAR10, 10, the performance of FILTRA is minorly disadvantageous. The explanation comes in the interpolation artifacts mentioned in Subsect. 3.6. As the interpolation of high frequency compo- nents deviates more, this harms the performance on CIFARIO with high frequency texture.
4.2 REGRESSION TASK
Besides the typical classification task, we find that the property of steerability is naturally advan- tageous for many regression tasks whose input might rotate or reflect. In this paper, we evaluate the regression performance with example task to predict the character direction. Similar tasks are commonly used in OCR techniques. When the character rotates, the predicted direction should rotate with the same rotating frequency. This means the predicted 2D direction vector is following irrep 1 for Cx. We reuse the backbone in Table Ia to extract features and use a regression head in Table 1c predict unit 2D vector denoting the direction. The network is trained with MSE loss.
Note that the images should be masked by a disk to avoid the network to overfit the direction from rotated black boundary. Different approaches are evaluated by the mean included angle between the predicted and groundtruth directions as shown in Table 2. FILTRA with Cg steerability performs best when trained on data augmented over S2).). We owe this to the fact that FILTRA weight is naturally organized by the discrete grid layout. Each element of discrete weight matrix contribute to one more DoF of the filters. In contrast, R2Conv uses filters parameterized with a polar coordinate. The DoF of the filters is slightly reduced due to the discretization.
CONCLUSIONS
In this paper, establish the connection between the recent steerable CNN structure based group representation theory and the conventional transformed filters. To this end, we propose an approach construct steerable convolution filters, which transform between features in trival, irreducible and regular representations. We verify the feasibility of FILTRA for the classification and regression tasks on several datasets.
REFERENCES
Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. Rotdef: Decomposition of convolutional filters for rotation-equivaarint deep networks. arXiv preprint arXiv: 18088880-066 2018.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint /11200/8/ 2018.
Gregory Cohen. Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pp- 2921-2926. IEEF, 2017.
8

---

Under review as a conference paper at ICLR 2021
Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. In International Conference on Machine Learning, pp. 1755-1763, 2014.
Taco S Cohen and Max Welling, Steerable cnns. arXiv preprint
2016.
Carlos Esteves, Christine Allen-Blancheee Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. In International Conference on Learning Representations, 2018.
Joao F Henriques and Andrea Vedaldi. Warped convolutions: Efficient invariance to spatial trans- formations. In International Conference on Machine Learning. pp. 1461-1469. PMLR. 2017.
Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Marc Pollefeys. Ti-pooling: transformation-invarian pooling for feature learning in convolutional neural networks. In Pro- ceedings of the IEEE conference on computer vision and paltern recognifion, pp. 289-297, 2016.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5048- 5057, 2017.
Edouard Oyallon and Stéphane Mallat. Deep roto-translation scattering for object classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2865- 2873, 2015.
Jean-Pierre Serre. Linear representations of finite groups, volume 42. Springer, 1977.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. In International Conference on Machine Learning (ICML), 2019.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. In Advances Neural Information Processing Systems, pp. 14334-14345, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Infor- mation Processing Systems, pp. 10381-10392, 2018.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5028-5037, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Yanzhao Zhou, Qixiang Ye. Qiang Qiu, and Jianbin Jiao. Oriented response networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 519-528, 2017.


Reviews:
[{"review_id": "vSikmE43uxO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Equivariant Steerable CNNs for 2D/3D rotation+reflection+translation groups have generally been implemented as a filter transform/expansion step followed by a standard convolution. The filter expansion step involves taking a linear combination of steerable basis filters. These basis filters are pre-computed before network training by solving a linear system or by sampling the continuous analytical solution (this can take a few minutes). Depending on the chosen group representation wrt which the network layer is equivariant, a different filter basis will emerge, but in general one can see that the basis filters come out as rotated and flipped copies of some basis filters, with the occasional sign flip (this has been stated in some earlier works). The precise way in which a basis filter is to be rotated and flipped to obtain the steerable filter basis for the 2D case had not been worked out before, to my knowledge, and this is one of the contributions of this paper. The analysis is done for each (input, output) representation type chosen from {trivial, irreducible, regular} representations. Having worked this out, the paper proposes to use this as a way of implementing the filter expansion step, starting from a basis filter and rotating/flipping it to obtain an expanded filter bank. \n\nThe proposed method (FILTRA) does not require a precomputation step if I understand correctly, which is a significant practical advantage. Experiments further show that the method is similar or faster at filter expansion. Finally, the method is validated by training networks on benchmark tasks and shown to perform similarly to or better than the steerable CNN implementation of Weiler & Cesa, which is the best existing implementation.\n\nThe paper briefly mentions that filters cannot be rotated exactly on a discrete grid, but I didn't figure out how the authors propose to deal with this issue. How exactly are the filters rotated?\n\nI think the method proposed in the paper is useful, as it is both faster and better than existing steerable CNN implementations. The paper itself is fairly well written and technically correct as far as I can tell, but may be challenging to read for those who are not yet knowledgeable about steerable CNNs. Those readers however are unlikely to be interested in learning about the implementation details of steerable CNNs anyway, so perhaps this is fine. The reason I am not giving a higher rating is that I think that although this is a useful contribution to the literature on steerable CNNs, which are being used in an increasing number of applications, the paper does not represent a major breakthrough and although the calculations are non-trivial, does not contain highly unexpected or deep theoretical results.\n\nTypos:\nCadestrian -> Cartesian\nirreduciable -> irreducible\nequity -> equality\n\nEdit:\nHaving read the reviews, rebuttal and updated paper, I have decided to maintain my score of 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fjCq8AI6XAb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "***Summary***\n\nThe paper outlines the construction of filters in a group equivariant convolutional network equivariant to the groups CN (cyclic group of order N (N rotations)) and DN (dihedral group of order N (N rotations and flipping)). The authors achieve this by taking linear combinations of (harmonic) basis filters, as was shown in Weiler et al. (2018). They then proceed to show explicitly how one can build filters between particular representations of each group, namely, the trivial, irreducible, and regular representations. For experiments they demonstrate that the filters perform en par with those of Weiler and Cesa (2019), who had performed a near-exhaustive comparison among representations of SE(2, R).\n\n\n\n***Pros***\n\nTechnically I believe the work to be sound. I did not see any mistakes jump out at me. I also think that it is laudable that the authors included a high amount of detail in their exposition, which lays bare the exact mechanisms by which one would effectively go about building an equivariant layer.\n\nExperimentally it appears that the authors chose a sensible baseline and that they compared on dataset, which are relevant to the topic and commonly used in the equivariance literature. The authors also provide timing information on how fast each filter can be generated, which is something I have not seen before in the equivariance literature and which I appreciate seeing.\n\n\n\n\n***Cons and constructive feedback***\n\nPerhaps my largest criticism is that it is not obvious exactly what the contribution of the paper is meant to be. Perhaps this originates from my not knowing what the authors mean by “filter transform” and “steerable CNN”. My understanding is that the authors believe it is currently unknown how to construct a steerable CNN for the cyclic and dihedral groups. They provide explicit constructions in section 3, but to my knowledge these are already provided in the long appendix of Weiler and Cesa (2019), furthermore they can be found in Cohen and Welling (2016), Cohen and Welling (2017) and Bekkers et al. (2018) and most notably Weiler et al. (2018), who were to first to show how to construct equivariant filter layers some linear combinations of harmonic basis filters. As a result I am unsure of how to gauge the novelty of this contribution.\n\nIn the experiments of Table 2 (classification and regression) the authors compare their framework with an R2conv (representative harmonic based convolution) and a regular translationally equivariant network. I would like to know what exactly is a representative harmonic based convolution? Is that meant to refer to the work of Weiler and Cesa (2019)? \n\nIn the results tables there are no error bars. If possible I would have liked to have seen them. Since they are not there is it very hard to judge the efficacy of the results, which differ from the baselines by very small amounts.\n\nGiven the proximity of the work to Weiler and Cesa (2019), I would like to know exactly what differentiates the two works.\n\nPlease include exactly what the functions roll, flipped, and circulant actually do mathematically. I found these difficult to parse.\n\nThe mathematical level of the paper is pretty heavy for the uninitiated. It may be advisable for the authors to include a glossary of terms, if not short descriptions, in an appendix. If this is too much, at least please point to other papers, which have easily readable background sections for those not already well-read in group theory.\n\nThe authors may wish to have the submission proofread for spelling and grammar.\n\n***Post rebuttal review***\n\nHaving read through the rebuttal, the updated submission, and the reviews of the other reviewers I have upgraded my review from a reject to marginally below acceptance. This is for two main reasons. 1) the authors have vastly improved the presentation of the submission, which now looks a lot easier to read, 2) the authors have clarified for me, at least, what the main contribution of the work is. \n\nThat said I am not entirely sure what this contribution adds to the equivariance community, hence why my recommendation still leans towards reject. As far as I am aware, solving the equivariance equations is not the large bottleneck to progress in our community. They are linear equations, and there is work back into the 80's solving them (check out people like Pietro Perona, Patrick Teo). I think more importantly we need to focus on pushing the boundaries in areas such as extension to non-Euclidean manifolds, convolution over non-compact groups, learning symmetries, etc. While this work is clearly mathematically sound and the authors have demonstrated deep knowledge of the area, it feels a little like retracing prior works. That said, if the other reviewers disagree then I don't mind this paper being accepted. Perhaps since I have worked in this area, what appears as obvious to me is not generally acknowledged and this paper may serve as a useful clarification for those wishing to dive into the literature.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y9XsddlTB2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the connection between steerable CNN and filter transformation. The authors show theoretically that filter transformation can be used to realize steerable filters over different group representations. The authors also empirically show that the filter transformation based steerable CNN performs on par with the implementation based on harmonic bases.\n\nOverall, I think this paper is hard to follow. The presentation and notation depends on that of prior works and is hard to understand without being familiar with Steerable CNN. Compared with prior works, this paper focuses more on the mathematical derivation but falls short of explaining their implication. The result is therefore hard to understand for those that are not familiar with the theoretical basis. Also, the authors do not clearly show why the contribution of this work is significant. The theoretical connection doesn’t seem to provide a significant advantage nor solve the problems in existing methods. Similarly, the experiment doesn’t provide much information and only shows that the proposed implementation works on simple datasets. The authors should try to highlight how this work may benefit other research or applications and verify that the proposed method works on more complicated and realistic data.\n\nAnother aspect that can be improved is to discuss how the theory generalizes when multiple convolution layers are applied. In general, the theoretical properties of steerable convolution does not automatically generalize to multiple convolution layers. This is particularly important in real world data, because the transformation may happen at different scales and may need to be accounted for at different layers in the network. Therefore, the results derived from a single convolution operation may not be sufficient.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "THR6tif-el7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds the connection between various steerable CNN structures based on group representation theory and filter transformations. Using the discrete rotation and reflection group as an example, the paper establishes ways to construct steerable convolutional filters that transform features in trivial, regular, and irreducible representations. In particular, recent works on steerable CNNs such as ORN, RotDCF, TI-pooling, and RotEqNet can all be explained under such framework.\n\nThe paper is generally well written and well organized. However, I reckon the material would be quite dense for readers interested in equivariant CNNs but not well-versed in group representation theory, so a bit more explanation either in text or in the appendix would be helpful. The main theoretical results in the paper on the considered discrete group transformation are interesting, and can potentially lead to other development in the community. The numerical experiments seem to be limited (though this phenomenon seems to be the issue for most of the papers in this area...)\n\n** Pros\n1. Well written and technically correct.\n2. Interesting theoretical results that might inspire other works in the field.\n\n** Cons (to be explained in more details in questions)\n1. Some notation and abbreviation are used without proper explanation, which might be confusing.\n2. Limited experimental results\n\n** Questions and comment\n1. Some of the notations and abbreviations are used without proper explanation. For example, \"K\" in equation (12), OCR on page 1, and missing reference on R2Conv. I would encourage the authors to make a greater effort to clarify their ideas and results.\n\n2. The theory seems to be built only on discrete group. Does it generalize to continuous group transformation (such as SO(2) before discretization)?\n\n3. Also, does the theory generalize to non-compact groups such as  scaling and shearing?\n\n4. The experiments seem to be limited, even though this seems to be a common issue in papers in this field. However, I do recommend the authors to present the \"equivariant loss\" when using their proposed FILTRA, considering that discretization and interpolation might cause a problem in their setting unlike other means of steerable CNNs such as RotDCF and Harmonic Net", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach based on filter transform for designing networks equivariant to different transformation groups.
Especially, the authors rely on the haramonic analysis view of steerable CNNs given in Weiler & Cesa (2019) to design an equivariant filter bank by computing simple transforms over base filters. 

The reviewers finds the paper technically solid but difficult to read and with a limited contribution. 
The AC carefully reads the paper and discussions. Although the connection between steerable CNNs and filter transform are interesting, the AC considers that the main contributions of the paper should be consolidated, especially the positioning with respect to Weiler & Cesa (2019). \
Therefore, the AC recommends rejection.
Decision: Reject