Scale-Equivariant Neural Networks with Decomposed Convolutional Filters
Authors: Wei Zhu
Qiang Qiu
Robert Calderbank
Guillermo Sapiro
Xiuyuan Cheng
Venue:  (2020)
Keywords: scale-equivariant
convolutional neural network
deformation robustness

Abstract: Encoding the input scale information explicitly into the representation learned by a convolutional neural network (CNN) is beneficial for many vision tasks especially when dealing with multiscale input signals. We study, in this paper, a scale-equivariant CNN architecture with joint convolutions across the space and the scaling group, which is shown to be both sufficient and necessary to achieve scale-equivariant representations. To reduce the model complexity and computational burden, we decompose the convolutional filters under two pre-fixed separable bases and truncate the expansion to low-frequency components. A further benefit of the truncated filter expansion is the improved deformation robustness of the equivariant representation. Numerical experiments demonstrate that the proposed scale-equivariant neural network with decomposed convolutional filters (ScDCFNet) achieves significantly improved performance in multiscale image classification and better interpretability than regular CNNs at a reduced model size.

Under review as a conference paper at ICLR 2020
SCALING-RRANSLATION--OUVVVAAA NETWORKS WITH DECOMPOSED CONVOLUTIONAL FILTERS
Anonymous authors Paper under double-blind review
ABSTRACT
Encoding the scale information explicitly into the representation learned by a convolutional neural network (CNN) is beneficial for many vision tasks especially when dealing with multiscale inputs. We study, in this paper, a scaling-translation equivariant (ST -equivaria))) CNN with joint convolutions across the space and the scaling group, which is shown to be both sufficient and necessary to achieve ST- equivariant representations. To reduce the model complexity and computational burden, we decompose the convolutional filters under two pre-fixed separable bases and truncate the expansion to low- low-frequency components. A further benefit of the truncated filter expansion is the improved deformation robustness of the equivariant representation. Numerical experiments demonstrate that the proposed traastaaiannneeuuiteee
networks with decomposed convolutional filters (ScDCFNet) achieves significantly improved performance in multiscale image classification and better interpretability than regular CNNs at reduced model size.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved great success in machine learning problems such as image classification (Krizhevsky et al. (012)) object detection (Ren et a 2015). and semantic segmentation (Long et al.] 2013) Ronneberger al. (2015). Compared to fully-connected networks, CNNs through spatial weight sharing have the benefit of being canslation-equivarian i.e., translating the input leads to a translated version of the output. This property is crucial for many vision tasks such as image recognition and segmentation. However, regular CNNs are not equivariant to other important group transformations such rescaling and rotation, and it is beneficial in some applications to also encode such group information explicitly into the network representation.
Several network architectures have been designed to achieve (2D) roto-translation-een eeuiiarriiiiviiieceee -Se(())-quivarriace) (Cheng et al. 2019 Marcos et al., 2017 Weiler et al. 201 8b] Worrall et al. 2017, Zhou al., 2)17), I.e., roughly speaking, it the input spatially rotated and translated, the output is transformed accordingly. The feature maps of such networks typically include an extra index for the rotation group SŌ(2). Building on the idea of group convolutions proposed by Cohen & Welling (2016) for discrete symmetry groups. Cheng et al. (2019) and Weiler et al. (20T8b) constructed SE([)--qqivvarin CNNs by conducting group convolutions jointly across the space and S2( using steerable filters (Freeman Adelson 1991).
(ST -equivvaari) CNNs, on the other hand, have typically been studied in a less general setting in the existing literature (Kanazawa et al. 2014 Marcos et al. 2018 Xu et al. 20144; Ghosh & Gupta 2019). In particular, to the best of our knowledge, à joint convolution across the space and the scaling group has yet been proposed to achieve equivariance in the most general form. This is possibly because of two difficulties one encounters when dealing with the scaling group: First, unlike SO(2). it is an acyclic and unbounded group; second, extra index in S incurs significant increase in model parameters and computational burden. Moreover, since the scaling transformation is rarely perfect in practice (due to changing view angle or numerical discretization), one needs to quantify and promote the deformation robustness of the equivariant representation (i.e., is the model still "approximately" equivariant if the scaling transformation is "contaminated" by a nuisance input deformation), which, to the best of our knowledge, has yet been studied in prior works.
The purpose of this paper is to address the aforementioned theoretical and practical issues in the construction of -qquivaaria CNN models. Specifically, our contribution is three-fold:
1. We propose general ST-equivariant CNN architecture with a joint convolution over R S, which proved in Section 4 to be both sufficient and necessary achieve ST -qquivariancc,
2. A truncated decomposition of the convolutional filters under a pre-fixed separable basis on the two geometric domains (R22 2 and S) is used to reduce the model size and computational cost.

---

Under review as a conference paper at ICLR 2020
3. We prove the representation stability of the proposed architecture up to equivariant scaling action of the input signal.
Our contribution the family of group-equivariant CNNs is non-trivial; in particular, the scaling group unlike S0(2) is acyclic and non-compact. This poses challenges both in theory and in practice, so that many previous works on group-equivariant CNNs cannot be directly extended. We introduce new algorithm design and mathematical techniques to obtain the first general ST -eqquivari CNN in literature with both computational efficiency and proved representation stability.
RELATED WORK
Mixed-scale and ST -eqiivariant CNNs. Incorporating multiscale information into CNN repre-
sentation has been studied in many existing works. The Inception net (Szegedy et al.
2015) and
its generalizations (Szegedy et al. 2017 2016; Li et al., 2019) stack filters of different sizes in a single layer to address the multiscale e salient features. Dilated convolutions (Pelt & Sethian 2018
Wang et al. 2018, Yu & Koltun 2016 Yu et al., 2017). pyramid architectures (Ke et al. 2017 Lin et al.) 2017). D. and multiscale dense networks (Huang et al. 2017) have also been proposed to take into account the multiscale feature information. Although the effectiveness of such models have been empirically demonstrated in various vision tasks, there is still a lack of interpretability of their ability to encode the input scale information. Group-equivariant CNNs, on the other hand, explicitly encode the group information into the network representation. Cohen & Welling (2016) proposed CNNs with group convolutions that are equivariant to several finite discrete symmetry groups This idea is later generalized in Cohen et al. (2018) and applied mainly to the rotation groups SO(2) and SO(3) (Cheng al., 2019) Weiler et al. 20188a b). Although ST-equivariant CNNs have also been proposed in the literature (Kanazawa al. 2014; Marcos et al., 2018 Xu et al., 2014 Ghosh & Gupta 2019). they are typically studied in a less general setting. In particular, none of these previous works proposed to conduct joint convolutions over I2² x S as necessary and sufficient condition to impose equivariance, for which reason they are thus variants of a special case of our proposed architecture where the convolutional filters in S are Dirac delta functions (c.f. Remark I The scale-space group correlation proposed in the concurrent work (Worrall & Welling] 2019) bears pree the most resemblance to our proposed model, however, their approach is only limited to discrete semigroups, whereas our model does not have such restriction.
Representation stability to input deformations. Input deformations typically induce noticeable variabilities within object classes, some of which are uninformative for the vision tasks. Models that are stable to input deformations are thus favorable in many applications. The scattering transform (Bruna & Mallat] 2013) Mallat 2010 2012) computes translation-invant invariant representations that are Lipschitz continuous to deformations by cascading predefined wavelet transforms and modulus poolings. A joint convolution over TR2 (2) is later adopted in Sifre & Mallat (2033) to build roto-translation scattering with stable aaaaiionraasslaiionnnnaairr
representations. These models, however, use pre-fixed wavelet transforms in the networks, and are thus nonadaptive to the data. DCFNet (Qiu al. 2018) combines pre-fixed filter basis and learnable expansion coefficients in a CNN architecture, achieving both data adaptivity and representation stability inherited from the filter regularity. This idea is later extended by Cheng et al. (219)) to produce (E((2) -eeu)eqquivarian representations that are Lipschitz continuous in T.² norm to input deformations modulo global rotation, i.e., the model stays approximately equivariant even if the input rotation is imperfect. To the best of our knowledge, a theoretical analysis of the deformation robustness of ST--ST--e -qquivaaaia CNN has yet been studied, and direct generalization of the result in Cheng al. (20T9) is futile because the feature maps of a ST STeequivariant CNN is typically not in 1.2 (c.f. Remark 2
3 ST-EQUIVARIANT CNN AND FILTER DECOMPOSITION
Grrop--quivvriance is the property of a mapping f X + Y to commute with the group actions on the domain X and codomain Y. More specifically, let G be group, and Dg. Tg. respectively, be group actions on X and Y. A function f X - y is said to be G-equivariant if
f(DØ=) = (1,x)=T,(((()), (x))()),
NgEG,IEM
(I)
G-invariance is thus a special case of G-equivariance where T, = Idy. For learning tasks where the feature y E Y is known priori to change equivariantly to a group action و E G on the input I E X, e.g. image segmentation should be equivariant translation, would be beneficial to reduce the hypothesis space to include only Geeuuivarintt models. In this paper, we consider mainly the scaling- group ST z S x R22 = R x R2. Given g (B,u) E ST and an input image
2

---

Under review as a conference paper at ICLR 2020
(
(u.
(s.
2
2
2 (a.oo.))
WEAR "
(u.
(y.
:
: (b) The general case of ST eTquiivaar CNN with joint convolutions (Theorem) !
:
(a) A special case of ST--S-T--iaaaeaa equivariant CNN with (multiscale) spatial convolutions.
Figure I: (a) A special case ST -qquivarai CNN with only (multiscale) spatial convolutions. The previous works on ST ST-equivariant CNNs (Kanazawa et al. 20144 Marcos al.] (018) Xu et al.] [ [Ghosh & Gupta 2019) are variants of this architecture. (b) The general case of S7 -cquivamant CNN with joint convolutions [Theorem) where information transfers among different scales. Remark [ for more explanation.
( (u, A) (u E R is the spatial position, and À is the unstructured channel index, e.g. RGB channels of color image), the cealing-translatee group action D2 D3,.v on (())) defined as
D3.0r(0) (u, A) = (0) (2 2--0(v) -/tu-v,,. v),
Constructing ST -equivvaria CNNs thus amounts to finding architecture A such that each trained network f E commutes with the group action D3,, on the input and a similarly defined group action TB, (to be explained in Section 3.1 on the output.
3.1 ST --EUIIAAAL CNNS
Inspired by Cheng al. 2019) and Weiler et al. (2018b), we consider ST ST-eqiivariant CNNs with an extra index a S for the the scaling group S - for each 2 1, the 1-th layer output denoted as 21) (u, a, X), where u TR22 is the spatial position, a E E [M]] = (1,..., Mr) corresponds to the unstructured channels. We use the continuous model for formal derivation, i.e., the images and feature maps have continuous spatial and scale indices. In practice, the images are discretized on Cartesian grid, and the scales computed only discretized finite interval. Similar Cheng al. (019), the group action T3. on the I-th layer output is defined as a scaling- translaatanllliinnn in space as well as a shift in the scale channel:
the scale index, and
na (u, a, A) (1)))
(2 8 (u v), - -BA); 1.
feedforward neural network is said to be
i.e., equivariant to ST,
D (0) Te, (z:
1121,
where we slightly abuse the notation x (z: to denote the I-th layer output given the input x: (0) The following Theorem shows that ST -Teequivariance is achieved if and only if joint convolutions are conducted over S x 2 as in (5) and (6)
Theorem 1. A feedforward neural network with extra index E S for layerwise output is ST -quuivarar if and only if the layerwise operations are defined as and
2 2-2a 2a 2a2 (0) (u + u., X'W (2 du' + 800))) (5)
x (1))) a, a) =
Σ
JR2
Σ
/
2a x 11
=
R22
(D N.A 12 a²) de' + +b(())), <> 1,
(6)
where R R. is pointwise nonlinear function.
We defer the proof of Theorem as well as those of other theorems, to the appendix. We note that the contocoonvllttinn in Theorem is a generalization of the group convolution proposed by Cohen & Welling (0066) to non-compact group ST in the continuous setting.
3

---

Under review as a conference paper at ICLR 2020
Remark 1. When the (joint) convolutional filter W
()) XxX (u, oi) takes the special form W xxx(u,,0)
v1 (u))(a)), the joint convolution
over IR2 x S reduces to only a (multiscale) spatial convolution
N.A x. ([ltt--)]( = < x (u + uu,, , a, (,,,)) X)V (2 (2-xu) 2 20 dul + +b))(2)
(7)
R2
iee. the feature maps at different scales do not transfer information among each other (see Figure Iap. The previous works (Kanazawa et al.) 2014 Marcos et al.) 2018 Xu et al. 2014 Ghosh & Gupta 2019) on ST -equivariant CNNs are all based on this special case of Theorem
Although the joint convolutions (6) on 2R2 S provide the most general way of imposing ST- cquivariance, they unfortunately also incur a significant increase in the model size and computational burden. Following the idea of Cheng et al. (2019) and Qiu et al. (2188), we address this issue by taking truncated decomposition of the convolutional filters under pre-fixed separable basis, which will be discussed in detail in the next section.
3.2 SEPARABLE BASIS DECOMPOSITION
We consider decomposing the convolutional filters W (u, a) under the product of two func- tion bases, {v)x(u)) and {Pm (a)) m. which are the cigenfunctions of the Dirichlet Laplacian on. respectively, the unit disk 1) C IR² and 1,,,1],ie..
{ A
= - the in D,
4hh = -mmmpm in [-1,
and
(8)
2kx = on aD,
fmm(-1) fm 0.
4m
In particular, the spatial basis {whhhs satisfying (8) is the Fourier-Bessel (FB) basis (brrmooitz & Stegun| 1965). In the continuous formulation, the spatial "pooling" operation is equivalent to rescaling the convolutional filters in space. We thus assume, without loss of generality, that the convolutional filters are compactly supported as follows
w (2'h D); and ( Cc(22 D [-1, AI> 1.
Let (5),((u) = 2- 2/ (2 (2- (2---)) u),, then w-(() A can be decomposed under ( and {pm 3m as (1) x (u), w (2, = ΣΣ (k
(1()) M (1) = Σ
>> (10)
11) xx (ks, m)) are the expansion coefficients of the filters. During training, the where ax.x (A) and a
basis functions are fixed, and only the expansion coefficients are updated. In practice, we truncate
the expansion to only low-frequency components (i.e., ax (ks, m) are non-zero only for k E [K], m E [Ka]). which are kept as the trainable parameters. Similar idea has also been considered in the prior works (Qiu et al. 2018:; Cheng et al. 2019 Jacobsen et al. 2016). This directly leads to reduction of network parameters and computational burden. More specthcally, let compare the I-th convolutional layer (6) of a ST -equivvari CNN with and without truncated basis decomposition:
Number of trainable parameters: Suppose the filters W ((()) X. (u, ar) are discretized on Cartesian grid of size x 7. x T.... The number of trainable parameters at the 1-th layer of a ST --t.-vvviaaiiantttt CNN L 1 Mr. On the other hand, in an ScDCFNet with truncated
without basis decomposition basis expansion to K leading coefficients for and K, coefficients for a, the number of parameters instead K K2 M Mj. Hence a reduction to a factor of K K,/LLL, L. in trainable parameters is achieved for ScDCFNet via truncated basis decomposition. In particular, if L = 5, La = 5, K = 8, and K 3, then the number of parameters is reduced (8 x 3)/(5? x 5) = 19.2%.
Computational cost: Suppose the size of the input (1-1)) u, a, A) and output x0 (u, a, A) at the 1-th layer are, respectively, W x W x N& x M:-1 and W x W x Na x Mr. where W x W is the spatial dimension, Na is the number of scale channels, and Mi-((M) is the number of the unstructured input (output) channels. Let the filters ((( X (u, a) be discretized on Cartesian grid of size L L x L.- The following theorem shows that, compared the computational cost in forward pass of ScDCFNet is reduced again to factor of K Ko. L..
regular ST -qquivarria CNN,
Theorem 2. Assume Mj > L2 Law i.e. the number of the output channels much larger than the size of the convolutional filters in u and a, then the computational cost of an ScDCFNet is reduced to factor of K Ka/ 1.2 Tag when compared to ST -qquivrrraa CNN without basis decomposition.
4

---

Under review as a conference paper at ICLR 2020
4 REPRESENTATION STABILITY OF SCDCFNET TO INPUT DEFORMATION
Apart from reducing the model size and computational burden, we demonstrate in this section that truncating the filter decomposition has the further benefit of improving the deformation robustness of the equivariant representation, i.e., the equivaraince relation (4) still approximately holds true even the spatial scaling of the input Dz.,z((0) is contaminated by a local deformation. The analysis is motivated by the fact that scaling transformations are rarely perfect in practice they are typically subject local distortions such as changing view angle or numerical discretization. To quantify the distance between different feature maps at each layer, we define the norm of x(() as
|| (0) 1|2
Mu Σ
Mdo
/
Σ/ x (0) (u. () du, ||=000 = sup M1
1
Mr
10000 //100
(u, (u, a, sof/da, du, 12 (11)
Remark 2. The definition of || is different from that of RotDCFNet (Cheng et al. 2019. where an L² norm is taken for the index as well. The reason why we adopt the L norm for a in is that a0 is typically not T.2 in o, since the scaling group S, unlike (2),), has infinite Haar measure.
We next quantify the representation stability of ScDCFNet under three mild assumptions on the convolutional layers and input deformations. First,
(A1) The pointwise nonlinear activation a R R is non-expansive.
Next, we need a bound on the convolutional filters under certain norms. For each / 2 1, define A, as
Mo
Mo
Mr
A == max
sup
1
sup Mi
|
1F
X=1
A=1
(12)
M
Mi (.
2.M
A = J max
sup
Σ
|| (
l|BBB
FB:
sup A=1
laxa a
Mi
X=1
m
where the Fourier-Bessel (FB) norm ||a]|a of a sequence (a(k))k2 is weighted 12 norm defined
as |||li (k)) where μig is the k-th eigenvalue of the Dirichlet Laplacian on the unit disk defined in We next assume that each A is bounded:
(A2) For all 2 1, A₁ 1.
The boundedness of A, is facilitated by truncating the basis decomposition to only low-frequenc" components (small (t)), which one of the key idea of ScDCFNet explained in Section 3.2. After proper initialization of the trainable coefficients, (A2) can generally be satisfied. The assumption (A2) implies several bounds on the convolutional filters at each layer (c.f. Lemma in the appendix), which, combined with (A1), guarantees that ScDCFNet is layerwise non-expansive:
Proposition 1. Under the assumption (Al) and (A2), an ScDCFNet satisfies the following.
(a) For any 1 2 1, the mapping of the 1-th layer, x! [=] defined in (5) and (6), is non-expansive, i.e.,
(1)) [x2]]] < | ø2ll. 1x1,02.
(13)
(1) (>) depends only on (b) Let xo be the 1-th layer output given a zero bottom-layer input, then 20
(!)
(c) Let de be the centered version of x( after removing x (1) i.e., we (u, >) = x:(0) A)
(1)
zo (0) (A) ( A), and xe (u, a. A) = (,)) a, X) xo (A), Vi 2 1, then || (1) 5
(1)
|| 1- VI 2 1. As a result, ||2 < |
||
Finally, we make assumption the input deformation modulo global scale change. Given a C function R² + R², the spatial deformation D, on the feature maps ((()) is defined as (7x (u, A) = x (0) (p(u), A), and D (u, ar A) = (0) (p(u), A), 12 1,
(14)
where p(u) = u T(u). We assume a small local deformation on the input:
is the operator norm. The following theorem demonstrates the representation stability of an ScDCFNet to input deformation modulo global scale change.
(A3) Vo == sup, [v(ll) < 1/5, where ||
5

---

Under review as a conference paper at ICLR 2020
First-layer features (CNN) [0
First-layer features (ScDCFNct) wa
lece features (CNN) (2
Sccené-layer features (S:DCFNe)
Da (x
Ta [=(00]
Da (200
Ta 1
Da
DD
Dan D
z [D.
D
Da. T
Figure 2: Verification of ST -eequivaaiannce in Section 5.1 Given the original input x0 and its rescaled version D2.,x(")), the four figures in each dashed rectangle are: ="[[z!!"] (I-th layer feature of the original input), a [D).="") (I-th layer feature of the rescaled input), Ta -z "0 [u:10] (rescaled (-th layer feature the original imput), and the difference (a [Da. v. 1 Tz (x) displayed in (signal intensity) scale relative to the maximum value of x " (D It is clear that after numerical discretization, ST -equivaare still approximately holds for ScDCFNet, i.c., zz [Dp,u
T2. ( [x(00] x O, but not for a regular CNN.
Theorem 3. Let D, be small spatial deformation defined in (14) and let D3.0s T3.v be the group 8 arbitrary scaling 2 E R+ centered E. RR defined in 2 and (3)
actions corresponding In ScDCFNet satisfying (Al), (A2), and (A3), we have, for any L,
[D B, v
[0 TB.
[x: [07
22+1 (4T/VTloe + +2-14/7/o ||x
(15)
Theorem 3 gauges how approximately equivariant is ScDCFNet if the input undergoes not only scale change D but also nonlinear spatial deformation D,, which is important both in theory and in practice because the scaling of an object is rarely perfect in reality.
5 NUMERICAL EXPERIMENTS
In this section, we conduct several numerical experiments for the following three purposes.
1. To verify that ScDCFNet indeed achieves -TTequivariance (4)
2. To illustrate that ScDCFNet significantly outperforms regular CNNs at much reduced model size in multiscale image classification.
3. To show that trained ScDCFNet auto-encoder is able to reconstruct rescaled versions of the input by simply applying group actions the image codes, demonstrating that ScDCFNet indeed explicitly encodes the input scale information into the representation.
The experiments tested on the Scaled MNIST (SMNIST) and Scaled Fashion-MNIST (SFashion) datasets, which are built by rescaling the original MNIST and Mashion-MNIST (Xiao et al. 2017) images by factor randomly sampled from uniform distribution on [0.3, 1]. A zero-padding to size of 28 x 28 is conducted after the rescaling. If mentioned explicitly. for some experiments, the images are resized to 64 64 for better visualization. The implementation details of ScDCFNet are explained in Appendix I. In particular, we discuss how to discretize the integral (6) and truncate the scale channel to tinite interval for practical implementation. We also explain how to mitigate the boundary "leakage" effect incurred by the truncation. Moreover, modifications to the spatial pooling and batch normaliaooaiiiiionnn modules of ScDCFNet to maintain ST -equivariance are also explained.
5.1 VERIFICATION OF ST ST-EQUIVARIANCE
We first verify that ScDCFNet indeed achieves ST -equivarria the feature maps of a two-layer ScDCFNet with randomly generated truncated filter expansion coefficients and those of regular CNN. The exact architectures are detailed in Appendix B.2 Figure diguraydssplays the first- and second-layer feature maps of an original image z0) and its rescaled version (0)) using the two comparing architectures. Feature maps at different layers are rescaled to the same spatial dimension for visualization. The four images enclosed in each of the dashed rectangle correspond to: x²() [z:(0)] (I-th layer feature of the original input), (²()DD,,a) [D,,,.z(0)] (I-th layer feature of the rescaled input), Tp. !!) (z:(0) (rescaled i-th layer feature of the original input, where Ta, is understood D3.e for regular CNN due to the lack of scale index a),, and the difference
Specifically, we compare

---

It is clear from Figure that that the boundary "leakage" effect is unavoidable the network becomes deeper. However, the error can be alleviated by either choosing joint convolutional filters with smaller support in scale (i.e., the filter size, or the number of "taps", after discretization in scale is much smaller compared to the number of scale channels in the feature map), or using a replicate- Under review as a conference paper at ICLR 2020
No muriccl eiror equivariance
Zero padding. lap
No currrntal equivartance
Replicate padding, tap 9
9.
Zero padding. lap 7
24
Replicate pedding.
Zero padding, tap 5
Replicate padding. tap
padding,
0.03
Replicate pedding, tap
603
602
Depth
Depth
(a) Zero-padding for the scale channel.
(b) Replicate-padding for the scale channel.
Figure 3: The numerical error in equivariance (i.c., the boundary "leakage" effect incurred by scale channel truncation) function of network depth. Either (a) zero-padding (b) replicate-padding is used for the convolution in scale. The error is unavoidable as depth becomes larger, but it can be mitigated by (1) using joint convolutional filters with smaller support in scale (i.e., a smaller number of "taps" after discretization), and (2) using seplicate-padding instead of zero-pu sero-padding. See Appendix B.1 for detailed explanation.
x00 v [00] To, (0 (z (0)) clear that even with numerical discretization, which can be modeled as form of input deformation, ScDCFNet is still approximately ST "suuqiivariant, i.c. (0) (D [3,100000] = =9,. (,,(0), whereas regular CNN does not have such property.
We also examin how the numerical error in equivariance (incurred by the the boundary "leakage" effect after the scale channel truncation) evolves as the network gets deeper. The error in equivariance measured in relative L sense at particular scale a, i.e.,
Error =
(x a))
(16)
padding in the scale channel for the joint convolution. See Appendix B.1 for detailed explanation.
5.2 MULTISCALE IMAGE CLASSIFICATION
We next demonstrate the improved performance of ScDCFNet in multiscale image classification. The experiments are conducted on SMNIST and SFashion, and regular CNN is used performance benchmark. Both networks are comprised of three convolutional layers with the exact architectures (Table 2 detailed in Appendix B.3, Since the scaling group S = R is unbounded, we compute only the feature maps (1, a. A) with the a index restricted to the truncated scale interval [-1.6. (2 (2-1.6 x 0.3), which discretized uniformly into Na 9 channels (again, see Appendix B. for implementation details.) The performance of the comparing architectures with and without batch-normalizate is shown in Table [] It is clear that, by limiting the hypothesis space to ST- equivaraint models and taking truncated basis decomposition to reduce the model size, ScDCFNet achieves significant improvement in classification accuracy with reduced number of trainable parameters. The advantage of ScDCFNet is more pronounced when the number of training samples is small (N,, 2000), suggesting that, by hardwiring the input scale information directly into its representation, ScDCFNet is less susceptible to overfitting the limited multiscale training data.
We also observe that even when a regular CNN is trained with data augmentation (random cropping and rescaling). its performance still inferior to that of an ScDCFNet without manipulation of the training data. In particular, although the accuracies of the regular CNNs trained on 2000 SMNIST and SFashion images after data augmentation they still underperform the SeDCFNets without data augmentation (93.91% and 79.94%) using only a fraction of trainable parameters. Moreover, if ScDCFNet is trained with data augmentation, the accuracies can be further improved to 94.30% and 80.62% respectively. This suggests that ScDCFNet can be combined with data augmentation for optimal performance in multiscale image classification.
improved
respectively, 93.85% and 79.41%.
7

---

Under review as a conference paper at ICLR 2020
Without batch normalization
SMNIST lest accuracy (%)
STashion lest accuracy (%)
Architectures
Ratio
Nr 2000
No 5000
Nir
10000
N₂. = 2000
Ni 5000
CNN, M 32
1.00
92.60 = 0.17
91.86 + 0.25
96. 13 + 0.18
77.74 0.28
82.57 # 0.38
ScDCFNet, M
10. Ko
0.84
93.75
+ 0.02
95.70
0.00
96.89 0.10
78.95
0.31
83.51
+ 0.71
Ka
93.91 + 0.30
95.71 = 0.10
96.81 0.12
79.22
0.50
$3.06
0.32
K = 5, Ka
0.42
93.52
= 0.29
95.19 0.13
96.77
+ 0.12
79.74
+ 0.44
83.46
0.69
10,
95.54
0.61
8, Ka
0.45
93.67
1: 0.19
95.51 1: 0.20
96.85 0.06
79.15 0.39
83.44
0.37
K.
0.28
93.51
0.30
95.35 + 0.21
96.49
78.57
0.53
82.95
0.46
ScDCFNet, M
8
0.12
0.00000 0.68
K = Ka
0.11
93.39
+ 0.25
95.25
+ 0.17
96.73
78.43
0.76
82.53
0.58
5. K
0.09
93.21
0.20
94.99
0.12
96.35 0.12
77.97
0.37
82.21
0.67
10. Ka
0.52
95.22
96.52
0.21
78.71
83.05
K 8, Ko
0.06
93.61
0.11
95.03
1 0.31
96.61
1 0.18
78.69
0.72
82.96
0.27
0.04
93.23
0.13
94.84
0.38
96.31
0.17
78.38
0.81
82.
0.25
With match-normaiizttinn
SMNIST test necuracy (%)
SFashion test nccuracy (%)
Architectures
Ratio
N₂
2000
Ng 5000
N2
10000
N2 2000
N, 5000
CNN, M 32
1.00
94.78 1. 0.17
96.58 1. 0.17
97.41 1. 0.21
79.79
0.40
84.38 1 0.17
ScDCHNet, M = 16
=3
0.15
0.23
97.71 0.11
81.12
$1.97
0.16
Ko
0.67
95.72
+ 0.29
97.15 - 0.24
97.72
0.07
81.41
1:11::0000 0.35
85.11 + 0.26
5, K. 3 0.42
95.31
0.21
96.75 + 0.13
97.38 0.07
81.73
0.14
81.70
0.19
Ka
0.16
0.07
8, Ka
0.45
95.76
1 0.17
96.74 + 0.22
97.68 + 0.09
80.84
0.28
84.70
0.18
5, Ka
0.28
95.43
1 0.09
96.54 1 0.20
97.43
0.13
81.23
0.23
84.65
0.36
SE-CNN VE-SENN
1,02
97.25 + 0.09
97.56
Table Classification accuracy on the SMNIST and SFashion dataset with and without batch-normallzzaton. The architectures are detailed in Table In particular. M stands for the number the first-layer (unstructured) output channels, which is doubled after each layer, and K/K., the number basis function u/ a used for filter decomposition. The networks are tested with different training data size, N.. 2000, 5000, and 10000, column "ratio" stands for the ratio between the number of trainable parameters of the current architecture and that of the baseline CNN. The results of the Locally Scale-Invariant CNN (SI-CNN) (Kanazawa al.) (2014) and Vector Field Scale-Eqqivariant CNN (VF-SECNN) (Maroos al. bottom of the table for comparison.
the means and standard deviations after three independent trials are reported.
2018 reported in (Marcos al. 2018) displayed at the
CNN auto-encoder
ScDCFNet auto-encoder
Input
Decoderr(C)
De
Decoder(C)
Deodee(T;..))
a
a
a
a
a
5
5
5
5
5 5
5
Figure 4: Reconstructing rescaled versions of the original test image by manipulating its image code C
according to the group action (3). The first two images on the left are the original inputs: Decoder (C) denotes the reconstruction using the (unchanged) image code C: Decoder(Da,..C reconstructions using the "rescaled" image codes Dp..C'' and T3..C' respectively according to 2 and 3 Unlike the regular CNN auto eoooeedoder the ScDCFNet auto-encoder manages to generate rescaled versions of the original input, suggesting that successfully encodes the scale information directly into the representation.
and Decoder(Ta,.
denote the
5.3 IMAGE RECONSTRUCTION
In the last experiment, we illustrate the ability of ScDCFNet to explicitly encode the input scale information into the representation. To achieve this, we train an ScDCFNet auto-encoder on the SMNIST dataset with images resized to 64 x 64 for better visualization. The encoder stacks two ST- equivaraint convolutional blocks with 2 x 2 average-pooling, and the decoder contains succession upsampling. regular CNN auto-encoder also
of two transposed convolutional blocks with trained for comparison (see Table i Appendix B.4 for the detailed architecture.)
Our goal is to demonstrate that the image code produced by the ScDCFNet auto-encoder contains the scale information of the input, i.e., by applying the group action T3.0 e to the code C of test
8

---

Under review as a conference paper at ICLR 2020
image before feeding it to the decoder, reconstruct rescaled versions of original input. This property can be visually verified in Figure In contrast, a regular CNN auto-encoder fails to do so.
6 CONCLUSION
We propose, in this paper, a ST-equivaraii CNN with joint convolutions across the space I² and the scaling group S, which we show to be both sufficient and necessary to impose -eqquivaari network representation. To reduce the computational cost and model complexity incurred by the joint convolutions, the convolutional filters supported on R S are decomposed under separable basis across the two domains and truncated only low-frequency components. Moreover, the truncated filter expansion leads also to improved deformation robustness of the equivaraint representation, i.e,, the model is still approximately equivariant even if the scaling transformation is imperfect. Experimental results suggest that ScDCFNet achieves improved performance in multiscale image classification with greater interpretability and reduced model size compared regular CNN models.
For future work, we will study the application of ScDCFNet other more complicated vision tasks, and pose estimation, where is beneficial to directly encode
such as object poctettin/loccllzatiii the input scale information into the deep representation. Moreover, the memory usage of our current implementation of ScDCFNet scales linearly to the number of the truncated basis functions in order to realize the reduced computational burden explained in Theorem 2 We will explore other efficient implementation of the model, e.g., using filter-bank type of techniques to compute convolutions with multiscale spatial filters, to significantly reduce both the computational cost and memory usage.
REFERENCES
Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions: with formulas, graphs, and mathematical tables, volume 55. Courier Corporation, 1965.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 15(8):1872-1888) 2013.
Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. RotDCF: Decomposition
of convolutional filters for rotation- euuivvarinti deep networks. In International Conference Learning Representations, 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 2016.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. arXiv preprint arXiv: 1.02017, 2018.
William Freeman and Edward H Adelson. The design and use of steerable filters. IEEE Transactions on Pattern Analysis & Machine Intelligence, (9)8891990) 1991.
Rohan Ghosh and Anupam K Gupta. Scale steerable filters for locally male--nvvrriant convolutional neural networks. arXiv preprint arXiv: 038611 2019.
Gao Huang, Danlu Chen, Tianhong Li. Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint 17339:1703.09844, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR.
Jom--enrik Jacobsen, Jan van Gemert, Zhongyu Lou, and Arnold WM Smeulders. Structured receptive fields in cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2610-2619, 2016.
Angjoo Kanazawa, Abhishek Sharma, and David Jacobs. Locally scale-invariant convolutional neural networks. arXiv preprint 1412.v 2014.
Tsung- Wei Ke, Michael Maire, and Stella X Yu. Multigrid neural architectures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6665-6673, 2017.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances neural information processing systems, pp. 1097-1105, 2012.
Xiang Li. Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. 2019.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR),
June 2015.
Stéphane Mallat. Recursive interferometric representation. In Proc. of EUSICO conference, Dane- mark, 2010.
Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics, 65 (10):1331-1398, 2012.
Diego Marcos, Michele Volpi. Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 5048-5057, 2017.
Diego Marcos, Benjamin Kellenberger, Sylvain Lobry, and Devis Tuia. Scale equivariance in cnns with vector fields. arXiv preprint arXiv:1807.11788 2018.
Daniël M Pelt and James A Sethian. A mixed-scale dense convolutional neural network for image analysis. Proceedings of the National Academy of Sciences, 115(2):254 259,, 2018.
Qiang Qiu. Xiuyuan Cheng, Robert Calderbank, and Guillermo Sapiro. DCFNet: Deep neural network with decomposed convolutional filters. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1198-4207, StockholmsmAd ssan, Stockholm Sweden, 10-15 Jul 2018 PMLR.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 91-99. Curran Associates, Inc., 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Cosputer-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24574-44
Laurent Sifre and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1233-1240, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 18-2826. 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Thirty-First AÅAI Conference on Artificial Intelligence, 2017.
Panqu Wang. Pengfei Chen, Ye Yuan. Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell. Understanding convolution for semantic segmentation. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1451-1460. IEEE, 2018
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems, pp. 10381-10392. 2018a.
10

---

Under review as a conference paper at ICLR 2020
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858, 2018b.
Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXiv preprint arXiv:1905.11691 2019.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
Yichong Xu, Tianjun Xiao, Jiaxing Zhang. Kuiyuan Yang, and Zheng Zhang. Scale-invariant convolutional neural networks. arXiv preprint Frxxiv:/411.63000 2014.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In Interna- tional Conference on Learning Representations, 2016.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks.
The IEEE
Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
Richard Zhang. Making convolutional networks shift-invariant again. arXiv preprint
arXiv: /904. 1486, 2019.
Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Oriented response networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 519-528, 2017.
A PROOFS
A.1 PROOF OF THROREM
Proof of Theorem 7
We note first that holds true if and only if the following being valid for all
1 121. 1,
T (1 [1- = x [ 11
(17)
where (0) is understood as ()) We also note that the layer-wise operations of a general feedforward neural network with extra index S can be written
e (1t
Σ N
( (u u' (v N,u, a, A)du' + +b)())+)+(()))...
(18)
and, for 1 1,
2!ll (1
a, =0
E/
Σ
2(1-1) (ut
WW a' X, a, A) du' + (1)).
(19)
To prove the sufficient part: when 1 = 1, 2 and lead to
Tp. T [zx (zx az, A) (110 (2-8 (u w), a 3,
(0) (2 8 (u v) + w/, X w (2- 2 3-2(0-3) 3) du' + +b0)(a))
Σ
(0) (2-8( (u v #),, will (2-na) 2 20 20d +00(2))
(20)
and
x! (0,1 A)
11

---

Under review as a conference paper at ICLR 2020
Σ
D3 (u+ N.A 1 (2-0w") 2 20 20 + ((0
JRa
Σ
/
x (00 (2- (u+ X) W x/) (2 2 20 dul + +b(1(2))
(21)
E2
Hence Tp (x(00] = == (
When > 1, we have
T' T2 () [1x((111)][ (1,, A) /00///0 (2- (u- v), 3)
Σ Σ 200-11 2 (n + a 3 x
2- Wo.d 2 dul do' + 60) (1))
Σ / x((-1) (2- 1 + ū) + a x) W ((10) (2- a²) 2 2-20 duda' + 600) (2)
=0
(22)
and
x: (1)
[) a. 2)
Σ To Nut, (2-an', 20 da' + +b00(2)
=
Σll
B/ - (2'"u' 2-2 do' + (A))
(23)
Therefore T (2) (xol111)] = (1) [T (T 4l > 1.
To prove the necessary part: when = 1, we have
T3, (1) (zx [0 at A) = (2)) (2 (u- a) B,,)
(2 [u + w(1) (u',X22- (u 20
a du' + du''d 640(1))
x (0) (2 3 (u + x)
w (2 p/ 20x22-000 (u- - a 2.2)2-23 23 29 + +1))
(24)
and
- DA { (u+ ( (t
x. D. (DD =o
u' u' X) w()) (u. =0 Σ f (0) (2-
(25)
Hence for (7 to hold when = 1, we need
w (w/ X, u. a) (11) (2-8u' X 2- a- 2) 23 u, a, X, N',,, (26)
Keeping u. a,a, X, n', X, B fixed while changing v in (26), obtain that ('') X, N,u,e a, X) does
not depend on the third variable u. Thus ((1) (w', X u, a, A) ((1) (u', X 0, a, A), Vu. Define
( A (ul) as
= w() (,,,).
(27)
12

---

Under review as a conference paper at ICLR 2020
Then, for any given v', X, u, a, A, setting in (26) leads to
w (u', X u, a, A) w (2- X, 2u (1) 2
=W u'
W (2 (2-x') 2mx
(28)
Hence 18 can be written as (
For l 1, similar argument leads
W (u', a', u, a, A) w P2 a' X, 2 - -3 Vu, a, ,', ,' .,, B.
(29)
Again, keeping u, a, x, u', a', X, B fixed while changing in 29 leads to the conclusion that (w', a', X, u,u,a a, X) does not depend on the fourth variable u. Define
wh x (u' a') w(O) (u', a' 0,0,0,
(30)
After setting f = in 29 for any given u', a', X, u. u, A, we have
W (u','', u, = w (2- u', a' 2 (u--), 0)) 2-20
=W o2 1) W x (2-xu') 2-2a.
(31)
This concludes the proof of the Theorem.
A.2 PROOF OF THEOREM
Proof of Theorem
2 In a regular ST -equivaarian CNN, the /-th convolutional layer (6 is computed
as follows:
f x (1 (w
vlu,a, a a' A,
+ tf
(2-"n',a') 2 du',
(32)
Jx2
a(,,,,,,') / 2'',
(33)
Mi
x (u, = o
A,
(34)
The spatial convolutions in take 2W2,,,, L. M, flops (there are N,, L., M, M,- convo-
lutions in each taking 2 flops.) The summation over a' in (33 requires N,, W M;,
flops. The summation over X', adding the bias, and applying the nonlinear activation in (4 requires
an additional , (2 + M1-1) flops. Thus the total number of floating point computations in forward pass through the (-th layer of regular ST equivarrian CNN is
wa N. (2L. L,, Mi-1 + L,, Mi + M + 2).
(35)
On the other hand, in an ScDCFNet with separable basis truncation up to K KKa leading coefficients, (6) can be computed via the following steps:
y(u, a, X, =
I au,a+ aa d', (a')da'
(36)
JR
[
2(1, 0, X, k,m)
J
y(u + ul, N. "2')2 2a dud
(37)
Mr ΣΣΣ
x (u, (u,
z(u, oz X, 2 (k, m)
(00(
(38)
The convolutions in a (36) require
flops (there are 2 1K2 convolutions in a, each taking NL flops.) The spatial convolutions in (37) take 2 MI flops MN convolutions u. each taking 2 L flops.) The last step (38) requires an additional N, + KK,MM-1) flops. Hence the total number of floating point computation for an ScDCFNet is
2W Na (KK, M, L M, M(-1).
(39)
In particular, when Mr > I Las the dominating terms in (35) and 39 are, respectively, 2W²N, N, MMMM 1 L., and N K.. Thus the computational cost in ScDCFNet has been reduced factor of KKa
L.
13

---

Under review as a conference paper at ICLR 2020
A.3 PROOF OF PROPOSITION
Before proving Proposition | we need the following two lemmas.
Lemma 1. Suppose that {owtE are the FB bases, and F(u)
En a(k))saa)))
=
Ex alk)2 (2 'u) is smooth function 2 ( 1), then
1/2
/ [F()] du, f |u|] IVF(i du, 2
/ [VF(u)) du S < [alaaaBBB γ
Σ Pk a(hi)2 k
(40)
This is Lemma 3.5 and Proposition 3.6 Qiu (2018) after rescaling u. Lemma Teasily leads to the following lemma.
(1) Lemma 2. Let a (k, m) be the coefficients of the filter W W X (u, a) under the joint bases {vitR and [pmm Im m defined 10). and define W
,,m (u) as
Wx Wx... (2) = Σ 2
m) (k,
(41)
We have
B A Ox,, 2n 2D) S &
bm (0) A. 2 (1)
d.dm S 5=1lax, 221a2226mm)llrn v> 1, (42)
where
B = / / (w) du.
Blll ax
/ (u) du, 1> 1,
XX = f [u| V., (( x'x (1)((u) du,
N.d.m / |u| V., w(() 2' 2'2m((u) (w) du, 1> 1, (43)
D X,A = / V /
x (u) du,
D
/ V 2. du) 1>1 1.
=
We thus have
BG, Cr,C,,2 Dr - An
(44)
where
Mo
Mi
B1 == max
sup
((1)
Mo
sup
Σ
(1) X
MI
>==1
x=1
Ma
(1))
Mo
M:
((I X
C = max
sup A
X
M1
sup
(45)
X=1
X
A=1
Me
Mu
Mi
D1
max
sup
D (1)
sup
(1)
M1
and, for 1 1,
Mi
Mr
B = max
sup
Σ
Σ
BDD Amm
2M
Σ
Bl.
Bl == sup
B(())
Mi
1
m
m
Am1
C max
sup
M
Σ
( X.mm
2M-1
Mi
ECam
Ct
CL === sup
Mi
A.m
(46)
==1
A=1
Dj = max
sup
M
Σ
DD
X.A.
Am
1)
2M
Mi
Diim
Di.
sup Dim
X
Mi
D !))
In particular, (A2) implies that Bi, Ci, 2 Dr $ 1, VL.
14

---

Under review as a conference paper at ICLR 2020
Proof of Proposition 7 To simplify the notation, we omit (1) in W W
X. V.A.m and b0) and let
M Mr. M' M The proof of (a) for the case = 1 is similar to Proposition (a) of Qiu
ct all 2018) after noticing the fact that
[e [w(2- 2 2u J (u)l du,
(47)
JR2
and we include it here for completeness. From the definition of B1 in (45) we have
sup Σ
(1) N S B1. and sup Σ XIA
S B
M
(48)
M
Thus, given two arbitrary functions D1 and x2, we have
[m)] - (x2) (u, (u. A)
2- Σ 21 + N' (2-m/ 2 20 dul + +((2)
Σ au(u+al (2- 2 dul + +b(11)
D (u 2d)2 2d''' du' - -E f
u'(u+'''))Wax (2- 2 2-2 du!
x²
Σ / (0) (2) (() (2-))²) 2 2a du'
Σl / |601 - 22)(u Wxx,(2-<u')) 2 do''d'
Σ / Wxx. (2))2 dul
Σ f ((01 -m))(+ Wx.x(2- "n')1 (001|2220) 20 du'
Σ BlT
2- <B Σ / 22) (2 (ū (ū-w)) 2 20 di
(49)
X'
Therefore, for any a,
Σ/ (100)zi) (u, (u, x) du
$ Σ / Bi Σ / 1002 x2)(U, x21 Wwrx(2-0 (ū (ū- w))l 2 20 dūdu
B, Σ / 100 - xx) XI1
Σ WN.N(2 -0 - (m==))) 2 20 dū
B1 Σ/ / |(0 =2)
Σ B/A du
11 M
M Σ / |(2) - =z) 221²2 dā
= B M ||||| - 22112
<M ||] - 2
(50)
where the last inequality makes use of the fact that B₁ S A S 1 under (A2) (Lemma 2 Therefore
1 M
Σ / [110 - {02))) (u, a, 2)) du < || 22112.
[p1] - x! [22]]
sup a
(51)
15

---

Under review as a conference paper at ICLR 2020
This concludes the proof of (a) for the case 1 1. To prove the case for any 1 >> 1, first recall from (46) that
sup ΣΣ
B Am S Bi, and Σ B S B
M
where Bt.m= = sup Σ
B Am (52)
2M
Thus, for two arbitrary functions Iı and 22, have
221) [21] x²0) x0)/za)) (u, a1)
Σ / /
22' 2 2-20 2ad + b(2)
Σ 12(u u' (2 " a') 2 do' du' + b(2)
/ -
m2- a²) 2 2a du'
x R22 J
Σ
Wx (2 (2- pm (a' ddd'ddddd dul / (i) - r2)u a' 20 Σ
JB2 JR
m
Gmlu + a' a, X)2 20 u WN.nm "u")du'
(m
Wx.2. dul
m
W Wx (2 20 du
R22
(m, W (2 u))l 2-20 di
ΣΣ
(1)
xmm
X
R2
x'
1:
(m, a, X3) WWs (2 - w))l 2 2a dii,
(53)
N
m
R2
where
/ man-a)(u, P)
Gm an,
(54)
JR
We claim (to be proved later in Lemma that
M'I = -sup Σ
(m, X') du < 2M' - x21?, Vm.
(55)
x g2
Thus, for any ax,
Σ Σ -x100=a))0 (u, a, 2)) du
Σ
/
Gm a, 211 Wx (2 (2-0 u])l 2 20 düdu
Σ / [W..2mm2 (2 - w)){ 2 20 du dū
B, (G,
B X T7
Σ
(ū, (1)
Σ
Bil) aim
di
16

---

Under review as a conference paper at ICLR 2020
<B ΣΣ
Σ
am(ū,,,)\')'
X
=B:}
Gm
a, 2)) a,
dū Bt.m.
<Br 2M ||| 22l1² Σ Bl.m
M
<B$ 2M - -x2ll
I M =211?-
(56)
2M
Therefore
[2100 - 1up sup
Σ À R22
(2(0)zal- - 222)))) (u, (u,
du S ||| 2211? wsf a)
M
(57)
To prove (b), we use the method of induction. When = 0, xo To (1)
(0) (2, A) = 0 by definition. When = 1, (u, a) (A)). Suppose 2/" (u, a, A) r (1-1) (2) for some 1 1, then
( au, a, A)
Σ
(1-1)
(u + a' X') u2 aa) 2 20 do' do' 800 (2)
50
50
JR
Σ
20 (x)
/ (ith (2 u2' 2 20 do' + (00(2))
J82 182
(x) / / w (ul, a²) do' + add +B00
Σ
X
(1)
(A).
(58)
Part (c) is an easy corollary of part (a). More specifically, for any > 1,
| 0 = |1x00) =101 = | (2) [2 11-1)
=0 1=
1=00--1) 111 S (tx(t_)__ =z0 =011)1 =1|a(--1)
(59)
Lemma 3. Suppose up E (R) with supp(yem) C [-1, 1] and 1|s||12 1, and x is a function of three variables
T:R² x R x M +
(60)
(u, a, A) + a(),a,
(61)
with |00|2 = supa 1/7 Ex SEa (euu,, a, d) Define G(u, at A) as
a(u,a,A) = /
(62)
Then we have
M| sup Σ E/ GG(,, 2) du S 2M
(63)
Proof of Lemma
3 Notice that, for any a, have
GGu, 211 du > RR S
Σ
Ilu, a', Ma(la))da du
$ Σ
/ g2
(
[x a + a', 2)1 do') du
17

---

Under review as a conference paper at ICLR 2020
L Σ / (r(u, a a 21, du) da
M||x||2dd'= 2M|1
(64)
Thus
sup a
2)1, du S 2M|1a.11
(65)
82
A.4 PROOF OF THEOREM 3
To prove Theorem 3 we need the following two Propositions.
Proposition 2. In an ScDCFNet satisfying (Al) and (A3), we have 1. For any / 2 1,
(D - (x < 4(B CoVV
(66)
2. For any 1 2 1, we have
11 11 = 20 ||
(67)
and
1 (Tp.0 D1-0 TB. D Dr [z][-1)] S 20+2 Val (11-1)
(68)
where the first T3 in (68) is replaced by DB.r when = 1.
3. (A2) also holds true, then
[oc0(DA,, D,x: To, Dr (0,0000)000) S
|1:
14 > 1.
(69)
Proposition 3. In an ScDCFNet satisfying (Al) and (A3), we have, for any 2 1,
[Ta.. D T 1 < 23+1 T/x Di 12!!! - < 2 Dj 10 (0)
(70)
If (A2) also holds true, then
TannDDat T <
(71)
Proof of Theorem
Putting together 69 and
71
we have
Da, (0) T
(0))
(()) Dr. B.u (D) Ts. D
Ta (c.00]]]
+
<2 (0) + 23+1-10 |T 1 00
-20+1 (AL + 2- [z)
(72)
This concludes the proof of Theorem
Finally, need to prove Proposition 2 and Proposition 3 where the following lemma from Qiu et al. (2018) is useful. Lemma (Lemma A.I of Qiu et al. (2018) Suppose that /V < 1(,,,((u) = u - T(u), then at every point R.2
|| 1 S (2 VT(x),, where Jp is the Jacobian of p. and [Jp] is the Jacobian determinant. As a result,
(73)
|| 1), 11,1 1| 1 S T
(74)
and,
jJp]. [Dp 2 2.
(75)
18

---

Under review as a conference paper at ICLR 2020
Proof of Proposition 2 Just like Proposition Ta), the proof of Proposition 2 (a) for the case I is similar Lemma 3.2 of Qiu et al. (2018) after the change of variable 47 We thus focus only on the proof for the case / > To simplify the notation, denote x!4) [x 1 as y|ær], and replace
de - W 80) M1 and M,, respectively, by Zes W, b, M', and M. By the definition of the deformation D, (1). we have
(, a Σ [/a x(p(() + +u. (2""t',a')) 2-20 da' + b(X)
(76)
u'), ( od, ( 2-20 du' + b(A)
=|Drz((u,,A)==0 x(
(2"
(77)
Thus
1(D,yla] v D2 o
Σ X R2 JR
( "2""u',a') 2 20 da' du' + b(()
- / (2 2 20 do' du' + b(2)
Σ / (
x(((u) -00000 Wx,, (2 a²) 2 20 do'di'
Σ J Le (e, (
(2, 2 do' dul
ΣΣ / (xc(p(u) + a -dd JR
Wx.Am (2 uu) Pm 2ma) du'??
(78)
where the second equality results from the fact that 2 a. Iu(u,a a, A) =A( depends only on (Proposition) 1 ())) Just like the proof of Proposition [)), we take the integral of a' first, and define
Gm(u, ar X') = [ xolu, a + (),,N))mmm ())m)am
(79)
Thus
(D-y(=) y|D+ 2)1²
(Gm( + a (mlu u'), X\) Wx (2 (2-mu) 2 du'
X
G'lu,a,,')Wxxan (2-W)xxmm--00 (v- p(u))) 2 2-20 20
N
R2
ΣΣ
(2- Gmle,,, (2 w)) 2 2-20 1-
= =I (u, a, A) + 22
(80)
where
2) =E
Wx.A. (2 - p(u))) Wx.. (' (2-"(p-\(u) -w)) (81)
X m
B2
19

---

Under review as a conference paper at ICLR 2020
2 20 a,
(82)
a) = ΣE
-v) Wx WArd. (1) (1- |
(83)
X m
R2
2 2a Gm a,
(84)
Therefore
sup Σ M [ - y1
/ [(Dryla) - DDD,u)) a, 2)12 du
X
sup Σ
| [Bi A) + B2(u, a, 2) du
J22
=M E1 + Ezl1
(85)
Hence
[D+y[a] y [ || $ E1 + E2ll-
(86)
We thus seek to estimate ||Ea]] || and Eg individually.
To bound |E we let
k Nmm (,, = Wxa. (z))
- [1 1-)--())]]] (vv)]2-2 20
(87)
Then
a, = ΣΣ
Gm a,, A.mm (v. u, a),
(88)
X m
R2
and, for any given and
fec | (0) du du=
/
Wx.m (2 (p - -1)) 1- 1Jp (v)ll 2 20du
JR2
- IJp (w)]l /
(.x.mm()) (ū) dū
R2
<4 00 (1)
(89)
A.m
where the last inequality comes from (4). Moreover, for any given and a,
J ( (v. u, a)) du
Wxidm (2- u))l 1Jp 2o
JRa
(0-2
||Jp(2"0) - 1|dö dō
B2
<4 00 (1)
(90)
where the last inequality is again because of 77.. Thus, for any given a,
Σ //e EElu, (,,, du Σ/E ΣΣ
2 (v. u a)do du
R22
(U, 2)
k(3) k (u)) du)
ΣΣ
xx22) A (1, (1, (1, do du
/ a)) (o)) (ΣΣ /
Σ
(G,
k(2)
B(y) A....mm du
g2
N
E (2)
T
B
am)
1 (v, a) du du
m
X
R2
20

---

Under review as a conference paper at ICLR 2020
<4 B1
ΣΣ
/
Σ 4/VT B.g X.dmm du
X
V1 Bi
a
Bt.m
/16 B Σ M' 2
(91)
Since ( < 2|1x (by Lemma 3). and Em B S
M B₁ by definition 46 we thus have
2M
Σ/
X g2 [2(2, (,(,,, du < 16/V7 Bi
M
Bi 2M 21= = (e I)) Va. (92)
2M
Taking sup, on both sides gives us
F2 S 4/ B/Ilzell
(93)
Similarly, to bound || |l. we introduce
- Wxam (2- - -u]]] m uv, = [Wx.2.m (2- (v-
2-a. 20
(94)
Then
a)) m
Gm am( (.mm (U, u, a)de,
(95)
and, for any given v and o, we have
J/ 10 (1) A (v, u, a, du, I
(1)) ( u, a, do V/ A.m.
(96)
The proof of (96) is exactly the same as that of Lemma 3.2 in Qiu et al. (018) after change of variable, and thus omit the detail. Similar the procedure we take to bound Fg || (6 leads to
| & 4 C||
(97)
Putting together (8)) 993 and 97 thus have
| y D, -I]| S E| + Eall S | S 4(Bi + C
(98)
This concludes the proof of (a).
To prove (b), given any B E IR, and v E IR? we have
( 11 ssup sup Mi
1
Σ
Tp Ton (u, (///)//// du
1
Σ
(2() (2- (u=v), -B, BB,1 du
sup M R2
sup M Σ
/
10(0 (i. a - -B,N\/ 2200 dū
229 0011 112
(99)
Thus 67) holds holds true. As for 68 we have
(1) (T a Dr (1-1) T3, v. D. x (2) (cn(0-1)]
To (1, T/s 4 [111111]]
2 |000) (D - D2 (1) [
2 Ci) V (=
(100)
21

---

Under review as a conference paper at ICLR 2020
where the first equality holds valid because of Theorem 1 and the second equality comes from 667 To prove (c), for any j S l, define yj
y x x1 Ta o D,wli) o.... oz())
(101)
We thus have
( D D, 0,z0000000
To, To (0)) |z(00] =11 vol - 5=1
|y 9/ 9--1
(1() .... ... Te, D2x((0) .....r. o()) xx00 .... x(1) To, Dxx(i-1) ....orr o....0)........)))
T3p o Dー 1)200) - (T's D-0(t-111]
2 20+2 (B, +
j=1
2042 2 VT |0 V 1
(102)
where the second inequality is because of Proposition I a), the third inequality is due to 68 and the
last incquality holds true because Bi, C S A < under (A2) (Lemma 2) This concludes the proof of Proposition
Proof of Proposition 3 the first inequality in
The second inequality in 770 is due to Proposition [c). Because of (67). 70 is equivalent to
Dr - x10
S D+lx=
(103)
Just like Proposition 2aa), the proof of (103) for the case = 1 is similar to Proposition 3.4 of Qiu
et al. (2088) after the change of variable (47). A similar strategy that of Proposition 2 can
be used to extend the proof to the case > More specifically, denote ((1-1), xz respectively, as x, Fe: W, and to simplify the notation. We have
(!-1) W (1) (9)
D.H (0) [x] (x)) (u, 2)
2
Σ / /
(2-an', 2 du' + b(A)
R² JR
Σ
/(uuuuaaat (2 2-22 20 du'da' + b(2)
R2 RR
=ll a') 2 20 da'
R2 JR
Σ //u (2- a²) 2 du da'
(e Σ Wx. (2-mn') &m (a') 2 20 du' da' 2-2220200
Σ Wx. (2 pm (a') 2 20 da'
Gm,u a, (m, addu
(104)
X 70
22

---

Under review as a conference paper at ICLR 2020
where
Gm(u,a,)) Ic(u, + a', N))m ())mm(aa)da (a')da',
(105)
1x Ay.,m(u, a) 2-2 W. (2- - p(u))) 3-=(v-p(u))) Wxam (2
(106)
Similar to (96), we have the following bound
J/e 1xx.mm (v, u, a)) du, [ [kx.Am( u, u) de S V (9 A.A.m 1a2
(107)
Again, the proof of [107] the same as that of Proposition 3.4 in Qiu al. (2018) after a change of variable. The rest of the proof follows from a similar argument as in (92) and 99
B EXPERIMENTAL DETAILS IN SECTION 5
B.1 IMPLEMENTATION DETAILS
Discretization and scale channel truncation. In practice, the scale channel S ~ R needs to be truncated to finite interval I C S such that the layerwise feature map x:(0 (u, or, A) is only computed for a E I. This scale channel truncation unavoidably destroys the global scaling symmetry (similar to the fact that truncating an image to a finite spatial support destroys translation symmettry), which leads to the boundary "leakage" effect after (joint) convolution in scale. This boundary "leakage" effect be alleviated through the following two ways: (1)
choose joint convolutional filter
that is compactly supported on a much smaller scale interval I, compared to the truncated scale interval I, ie.. |al < [/] (a similar idea has been explored in (Worrall & Welling 2019):: (2) instead of using a zero- paro-pg (which is typically chosen by default as a spatial padding method before "replicate"-paddiing in the scale channel (i.e., extending the truncated scale channel according to Neumann boundary condition) should be implemented before the convolution in scale. After scale channel truncation, the layerwise feature maps 2²()) (u, a, A) and the separable basis functions (ux(u)) (4 )ax) I are discretized both in space and scale on a uniform grid. To avoid the spatial aliasing effect, the number K of the spatial basis functions (v)x()) is not allowed to be too large (typically we set K < 10.)
spatial convolution),
ST -qquivaaria average pooling. Given feature x00) (u, a, A), the traditional average-por average-pooling in u with the same spatial kernel size across destroys ST -TTequivariance (4) To remedy this issue, similar to (Zhang 2019). we first convolve _!()) with a scale-specific low-pass filter before downsampling the convolved signal on a coarser spatial grid. Specifically, we have z:0 a, A) = Jaz x0) (Nu + u', a, A2n(2-a u2)22222 22 du', where a()) is the feature after pooling, is a low-pass filter, e.g., a Gaussian kerne,, and N E+ the pooling factor. We refer to this as -eeqqivvariant average-pooling.
ST-qquivariant natch-normallzztion Batch-normalization loffe & Szegedy 2015 accelerates network training by reducing layerwise covariate shift, and it has become an integral part in various CNN architectures. With an extra scale index in the feature map z!(1) (u, a, A) of an SDCFNet, we need to include in the normalization in order not to destroy ST ST-equivariance, i.e., batch of features {xv (u, a, } should be normalized as it were a collection of 3D data (two dimensions for u, and one dimension for a.)
B.2 VERIFICATION OF SOUQUIAVAR
The ScDCFNet used in this experiment has two convolutional layers, each of which is composed of a ST -equiirrri convolution (5) (6), batch noneaaoommaiiztion, and 2 x 2 ST ST-equivariant (1) average-pooling. The expansion coefficients a ax (k) and ax (k, m) are sampled from a Gaussian distribution and truncated to K and K2 3 leading coefficients for and respectively. Similarly, a regular CNN with two convolutional layers and randomly generated 5 x convolutional kernels is used as baseline for comparison.
B.3 MULTISCALE IMAGE CLASSIFICATION
In the experiments multiscale image classification on the SMNIST and SFashion dataaset, the network architectures for the ScDCFNet and the regular CNN shown in Table 2 Stochastic
23

---

Under review as a conference paper at ICLR 2020
gradient descent (SGD) with decreasing learning rate from 10 2 to 10 1000(10-3) is used to train all networks without (with) gaich-normalization for 160 epochs.
Layer
(Regular) CNN
ScDCENet
c3x3xIxM ReLU ap2x2
sc(9)9XXXIxM ReLU sap2x2
c3x3xMx2M ReLU ap2x2
X199999XL,xMX2M ReLU sap2x2
3
c3x3x2Mx4M ReLU ap2x2
XM9) ReLU sap2x2
fc64 ReLU fe10 softmax-loss
fc64 ReLU fe10 softmax-loss
Table 2: Network architectures used
experiments in Section 5.2
cLxLxM'xM: regular convolutional
layer with input channels, M output channels, and LxL spatial kernels. sc(.Va LxLxM XMxxM: first-layer convolution operation (5) in ScDCFNet, where Na the number the uniform grid points discretize the scale interval -1.6, 0]. and LxL is the spatial kernel size the largest scale 0. se(N, XxL XM'MM: the I-th layer > convolution operation (6) in ScDCFNet, where the symbol L. stands for the filter size in (spxL(s)) the regular (ST equivariaant LxL cverage-poolingg fcM: fully connected layer with M output channels. natch-normalization layers added each convolutional layer if adopted during training.
B.4 IMAGE RECONSTRUCTION
The network architectures for the SDCFNet and regular CNN auto-encoders are shown in Table The filter expansion in the SDCFNet auto-encoder is truncated to and Ka 3. SGD with decreasing learning rate from 10-2 2 to 10 is used train both networks for 20 epochs.
Layer Regular auto-encoder
ScDCF auto-encoder
c7x7x1x8 ReLU ap2x2
sc(15)13x13x1x4 ReLU sap2x2
2
c7x7x8x16 ReLU ap2x2
5x(15)13x13x3x488 ReLU sap2x2
3
fc128 ReLU fc4096 ReLU
fe128
ReLU fc4096 ReLU
ct7x7x16x8 ReLU us2x2
ct7x7x16x8 ReLU us2x2
5
ct7x7x8x1 ReLU us2x2 et7x7x8x1 ReLU us2x2
Table 3: Architectures of the auto-encoders used for the experiment Section 5.3 The encoded representation the output of the second layer. x transposed-cos conpooodooooooootioollllll layers with M' input channels, M output channels, and LxL spatial kernels. us2x2: 2x2 spatial upsampling. See the caption of Table (2) for the definitions of other symbols. Batch-normalization (not shown in the table) is used after each convolutional layer.
24


Reviews:
[{"review_id": "S1eoXs9itr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Paper summary*\n\nThe authors propose a CNN architecture, that is theoretically equivariant to isotropic scalings and translations. For this they add an extra scale-dimension to activation tensors, along with the existing two spatial dimensions. In practice they implement this with scale-steerable filters, which are discretised and truncated in both the spatial and scale-dimensions. They also provide a deformation robustness analysis.\n\n*Paper decision*\n\nThank you for writing a very interesting paper indeed. I have to admit I am somewhat on the fence about this paper. I think it contains many nice ideas, but the experimental section is somewhat lacking in terms of comparisons or insights which I can gain, and the theory has some missing elements too (which I shall discuss below). For that reason, I am recommending a weak reject, but would very easily upgrade this if the authors provide strong rebuttal to my comments below.\n\n*Supporting arguments*\n\n-Experiments: The experiments are quite light, although I must admit that many other works in the area of equivariance are also light on experimentation and if there is enough theory, that is not such a great issue. The main issues I have are 1) the choice of experiments, 2) the comparisons against an insufficient number of baselines, and 3) the ablation studies.\n1) The choice of experiments: I think looking at scaled-MNIST is not particularly useful as an experiment nowadays, unless used as a toy experiment. A larger dataset with real-life scale variations would have been better. Furthermore, I’m not sure what the image reconstruction task is supposed to tell the reader, that the ScDCFNet is able to generalise to new scales?\n2) There is a lot of concurrent work on multiscale architecture. To name a few:\nMultigrid Neural Architectures, Ke et al., 2017\nFeature Pyramid Networks for Object Detection, Lin et al., 2017\nMulti-Scale Dense Networks for Resource Efficient Image Classification, Huang et al., 2018\n\nDeep Scale-spaces, Worrall and Welling, 2019\n\nI believe these works should at least be cited, but ideally compared against.\n\n3) I would have liked to have seen some numerical results for the verification of scale equivariance. In the first layer the equivariance improbably going to be close to perfect because there is no truncation of the scale-dimension in the network, but after this layer I predict the equivariance error increases due to truncations effects. This would be a similar effect to other works, such as “Deep Scale-spaces” (Worrall and Welling, 2019).\n\n-Theory: I think the theory is very interesting and a meaningful contribution in its own right. The authors treat scale-translation in continuous space as a group action on signals. This motivates the convolution presented in Theorem 1. This is a group convolution as per Cohen and Welling, (2016) modified to continuous space for a non-compact group. (Actually, this should be mentioned in the text as a matter of good scholarship). This is nice, since the group convolution has not been used in for a regular representation of a non-compact group (other than translation) as far as I can tell. What is perhaps not clear for me is how the theory breaks down in practice, since the implementation requires discretisation in space AND scale, which is not discussed much and furthermore, filters are restricted in spatial AND scale dimensions, leading to truncation errors in the equivariance. This last perspective was not discussed, and I feel it rather should be.\nThe theory goes further into deformation stability, which is a fresh perspective in the equivariance literature, so I am happy for its inclusion. Perhaps more motivation for why you think this is necessary would be warmly welcomed. \n\n*Smaller questions/notes for the authors*\n\n- Technically this is scale-translation equivariance, you even write this in the method section of your paper, why is it not in the title? The reason I mention this Is because there are scale equivariant networks, which are not translation equivariant in the literature, see “Warped Convolutions: Efficient Invariance to Spatial Transformations” (Henriques and Vedaldi, 2019).\n- Please make the link between Theorem 1 and the group convolution of Cohen and Welling (2016)\n- Last paragraph of page 1: A steerable-in-scale filter does exist, see “Deformable kernels for early vision” (Perona, 1991)\n- Please use numbering for all display-mode equations.\n- This scheme works perfectly in the continuous-image setting, but how about for discretized images? In that case it cannot be scale-translation equivariant because the scaling-action is no longer part of a group.\n- In equations 6 and 7, what is the specific motivation for using the laplacian eigen-decompositions as a basis? Is it for steerability with respect to the scale-translation action? Otherwise, surely any basis will do?\n- Remark 2: If you are considering a truncation of the scale-axis, surely you can still use an L2 norm when quantifying the robustness of your representation?\n- Bandlimiting of the filters: I would consider citing “Structured Receptive Fields in CNNs”, (Jacobsen et al., 2016)\n- Pooling: A useful citation here would be “Making Convolutional Networks Shift-Invariant Again” (Zhang, 2019). They precise low pass filter before pooling.\n- Experiments: \n—I’m not clear on the reason to include reasons with and without batch normalization. This is quite unconventional\n— Did you ever use scale augmentation? What is the effect of training on one scale and then testing on another? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HygbiOKaFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is introducing a scale-equivariant CNN architecture with joint convolutions over spatial and scale space. Moreover, the authors used decomposable convolutional filters to reduce the number of parameters. Based on Therom 1, it is shown that scale-equivariance is achieved if and only if joint convolutions are conducted over spatial and scale space. \n\nOverall, the contribution seems meaningful but incremental that the method is almost similar to the 'RotDCF: Decomposition of convolutional filters for rotation-equivariant deep network'.\n\n- to verify scale equivariance, visualizing features with tsne would be interesting\n- in table 1, what is the reason for doing experiments with and without batchnorm?\n- what if the baseline's size is close to the proposed method, how it would be improved?\n- it would be better to show how the decomposed filters reduce the total number of parameters.\n- also comparing flops would be informative.\n- the author's name of 'Locally scale-invariant convolutional neural networks' is wrongly written.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "H1lhZLO3qS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a scale-equivariant Neural Networks model to solve the multi-scale image classification task. The main contributions are the proposed joint convolution across the spatial and scale space, and the decomposed filters to reduce computation cost and improve robustness.\n\nThe paper is generally well-written and well placed in the literature. Different from existing work on the scale-equivariance CNNs, they build a more general model that allows different scale information to transfer through different layers. The experiments also suggest the model achieving scale-equivariance and the superior performance of the proposed method compared with several baselines. However, for the novelty part of the proposed method (the separable basis decomposition), it is not convincing enough because the differences between this paper and Cheng et al. (2019) and Weiler et al. (2018b) are not clear.\n\nQuestions:\n1. The authors claim that the joint convolutions across the space and the scaling group are both \"sufficient\" and \"necessary\". Although the experiments show that the proposed method achieves better performance, which might indicate the \"sufficiency\" of the architecture, however, how should the \"necessity\" be proved?\n2. How should \"K\" and \"K_\\alpha\" be selected (Table 1)? When data is enough (e.g., 5000), it seems that only some specific values can improve performance.\n3. How much can the model size be reduced compared with other multiscale image classification models? And could you provide some concrete comparisons about the computation cost?\n\nTypos:\n- 1st line in 2nd paragraph of Introduction: rotation-equivarianc(e)", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a CNN architecture equivariant to scaling and translation which is realized by the proposed joint convolution across the space and scaling groups. All reviewers find the theorical side of the paper is sound and interesting. Through the discussion based on authors’ rebuttal, one reviewer decided to update the score to Weak Accept, putting this paper on the borderline. However, some concerns still remain. Some reviewers are still not convinced regarding the novelty of the paper, particularly in terms of the difference from (Chen+,2019). Also, they agree that experiments are still very weak and not convincing enough. Overall, as there was no opinion to champion this paper, I’d like to recommend rejection this time. 
I encourage authors to polish the experimentations taking in the reviewers’ suggestions. 

Decision: Reject