Neural ePDOs: Spatially Adaptive Equivariant Partial Differential Operator Based  Networks
Authors: Lingshen He
Yuxuan Chen
Zhengyang Shen
Yibo Yang
Zhouchen Lin
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Equivariance
Partial differential operators

Abstract: Endowing deep learning models with symmetry priors can lead to a considerable performance improvement. As an interesting bridge between physics and deep learning, the equivariant partial differential operators (PDOs) have drawn much researchers' attention recently. However, to ensure the PDOs translation equivariance, previous works have to require coefficient matrices to be constant and spatially shared for their linearity, which could lead to the sub-optimal feature learning at each position. In this work, we propose a novel nonlinear PDOs scheme that is both spatially adaptive and translation equivariant. The coefficient matrices are obtained by local features through a generator rather than spatially shared. Besides, we establish a new theory on incorporating more equivariance like rotations for such PDOs. Based on our theoretical results, we efficiently implement the generator with an equivariant multilayer perceptron (EMLP). As such equivariant PDOs are generated by neural networks, we call them Neural ePDOs. In experiments, we show that our method can significantly improve previous works with smaller model size in various datasets. Especially, we achieve the state-of-the-art performance on the MNIST-rot dataset with only half parameters of the previous best model.

Published as a conference paper at ICLR 2023
NEURAL EPDOS: SPATIALLY ADAPTIVE EQUIVARI- ANT PARTIAL DIFFERENTIAL OPERATOR BASED NET- WORKS
Lingshen He', Yuxuan Zhengaann Shen', Yibo Yang", Zhouchen Lin
National Key Lab of General AI, School of Intelligence Science and Technology, Peking University
2 JD Explore Academy, Beijing, China
3 Department of Computer Vision Technology (VIS), Baidu Inc.
Institute for Artificial Intelligence, Peking University
5 Peng Cheng Laboratory
ABSTRACT
Endowing deep learning models with symmetry priors can lead to a considerable performance improvement. As an interesting bridge between physics and deep learning, the equivariant partial differential operators (PDOs) have drawn much researchers' attention recently. However, to ensure the PDOs translation equivari- ance, previous works have require coefficient matrices to be constant and spa- tially shared for their linearity, which could lead to the sub-optimal feature learn- ing each position. In this work, we propose novel nonlinear PDOs scheme that is both spatially adaptive and translation equivariant. The coefficient matri- ces are obtained by local features through a generator rather than spatially shared. Besides, we establish new theory on incorporating more equivariance like rota- tions for such PDOs. Based on our theoretical results, we efficiently implement the generator with an equivariant multilayer perceptron (EMLP). As such equiv- ariant PDOs are generated by neural networks, we call them Neural ePDOs. experiments, show that our method can significantly improve previous works with smaller model size various datasets. Especially, we achieve the state-of- the-art performance the MNIST-rot dataset with only tenth of parameters of the previous best model.
INTRODUCTION
In recent years, convolutional neural networks (CNNs) have achieved superior performance on vari- ous vision tasks (Szegedy et al., 2015 He et al. 2016 Chen et al. 2017). is acknowledged that the success of CNNs is attributed their ability to exploit the intrinsic coanssaatio-ivvriiann symmetry of data to help downstream vision tasks. To incorporate other symmetries like rotation-invariance, various CNNs-based equivariant networks have been studied and carried out to enhance the perfor- mance of vision tasks (Cohen& & Welling 2016a Weiler & Cesal 2019). In another branch, some works (Osher& & Rudin 1990; Perona & Malik| 1990) adopted partial differential operators (PDOs) to process images in the early period. Recently, PDOs with learnable coefficients are adopted by Shen et al. (220)) to design equivariant networks which achieve competitive performance compared previous equivariant networks. Jenner & Weiler (202T) further generalized this work to unified framework on the equivariant linear PDOs on Euclidean spaces of various representation types.
Actually, the coefficient matrices of the current PDOs works spatially shared, e.g. the same PDOs are applied to process features at each position (see ( However, such a coefficient sharing scheme of the PDOs is not the optimal pattern extract features from input images (Wul et al. 2018 Su et al.) 2019 Zhou et al. [022]] He et al., 202 To be specific, the contents of the input images vary according to positions, e.g. some pixels cover the background while some express texture, which would make coefficient-sharii sharing PDOs inefficient to extract features at each position.
"Equal first authorship
1 "ooresponniig author

---

Published as a conference paper at ICLR 2023
In fact, Jenner& & Weiler (2021) have proved that the linear PDOs layer is translation equivariant if and only if its coefficient matrices are spatially shared, so it seems impossible to ensure both the spatial adaptivity and translation equivariance for PDOs.
In this work, to deal with the above issue, we think outside the box of the linear limitation and pro- pose brand new nonlinear PDOs that are both spatially adaptive and translation equivariant. Com- pared with spatially shared PDOs, we construct a coefficient generator that inputs local features and outputs the coefficient matrices. Since different positions produce different coefficient matrices, the PDOs are essentially position-specifie and can extract individual features according to the local con- tent (see Figure [b)). In addition, the coefficient matrices generated by local features guarantee the translation equivariance for such PDOs naturally. However, such a nonlinear PDOs scheme is not intrinsically equivariant to rotations or reflections. incorporate equivariance of these transforma- tions, we establish theory on the equivariant formulation of this nonlinear PDOs scheme under any given symmetry group. Specifically, the theory reveals that this type of PDOs equivariant if and only if the coefficient generators
exactly equivariant maps of particular transformations. In practice, we choose a two-layer EMLP (Finzi et al., 202T) as the coefficient generator to satisfy the equivariance condition and provide an efficient implementation scheme. We name model Neural ePDOs and evaluate its performance on MNIST-rot and ImageNet datasets. Extensive exper- iments show that model significantly improve accuracy with fewer parameters. Especially, we achieve the state-of-the-ar results on MNIST-rot dataset with only a tenth of the parameters compared to previous best models.
I A.
Σ A.
E A.0I
400%
(a)
(b)
Illustration of two different designs for PDOs. Here, use the 2-dimensional vector field to represent the feature map. Fa)For linear PDOs, the coefficient matrices are shared to process features across different positions. (b) For nonlinear PDOs propose in this paper, the coefficient matrices are generated by the local features through neural networks.
Figure 1:
We summarize the main contributions follows:
To our knowledge, we are the first one to propose the nonlinear form of PDOs that are both spatially adaptive and translation equivariant. The coefficient matrices of the novel PDOs are adaptive to local features, which could alleviate the sub-optimal feature learning problem at each position.
We develop theory for such nonlinear PDOs that precisely characterize when it is equiv- ariant under any given symmetry group. The theory reveals that the nonlinear PDOs are equivariant if and only if the coefficient generators are exactly equivariant maps of partic- ular transformations.
We provide an efficient implementation which adopts two-layer EMLP as the coefficient generator and could largely save parameters and computations.
Extensive experiments show that our method significantly improve the results MNIST-rot and ImageNet datasets with significantly fewer parameters. Especially, we achieve state-of-the-art results the MNIST-rot dataset.
2
RELATED WORKS
So far, there are two mainstream approaches to constructing group equivariant networks. One is first developed by Cohen & Welling (2016a) which views the feature maps maps defined
2

---

Published as a conference paper at ICLR 2023
group, and they proposed group convolution operation process these feature maps equivariantly for image recognition. The method is further applied to designing equivariant networks for 3D space
sphere (Cohen et al., 2018), video tracking (Gupaa al. (02TT) and lie groups (Finzi et al. 2020a Bekkers 2019), etc. This approach is further developed to design atten-
(Worrall& & Brostoww 2018).
tive convolution layer (Romero et al. (2020) and self-attention layer (Romero & Cordonnier 2020
Hutchinson et al., 2021 He et | (021b).
The other one follows the approach of steerable CNNs (Cohen Welling 2016b Weiler & Cesa
2019 Weiler et al. 2018 Jenner & Weiler, 2021), which is a generalization of the first approach. Analogous physics, the feature map here viewed as a field, which is transformed according to specified group representation under the act of transformation. In comparison, the feature map in the first approach is simply the field with regular representation. Works (Cohen Welling 2016bb Weiler et al. 2018 Weiler & Cesa 2019) are devoted to finding out all the equivariant convolution operations as a map between any two fields. Later works further generalize the approach to design equivariant transformer (Fuchs et al. 2020) and graph network (Brandstetter et al. 2021). Our work follows these approaches.
Recently, some works focus utilizing PDOs to design equivariant neural networks, as they can build an interesting bridge between physics and deep learning (Jenner & Weiler| [002]. In addi- tion, PDOs are very suitable for processing continuous data (Finzi al. (0220b) and non-Euclidean structure data. The work most closely related to ours is Jenner & Weiler (021 which derives steer- able PDOs as a linear map between any two fields in the language of group representation theory. It is an extension of PDO-eConv (Shen et al. (2020) which employs rotated PDOs to design linear equivariant layers similar to the approach of Cohen Welling (2016a). In our work, to alleviate the spatial- -agios-agnostic problem in linear scheme and develop an equivariant theory that generalizes the Jenner & Weiler (202I). more detailed comparison between our work and steerable PDOs can be found in supplementary material.
equivariant layers, we propose a nonlinear PDO
3 PRELIMINARY
3.1 EQUIVARIANCE
Equivariance measures how the output of a network layer transforms in predictable way with respect to the transformation of the input. In mathematics, map ! is group equivariant if it satisfies: wh E H, V [T(h)]]]] = x' (f)),
(1)
where H is a transformation group, (h) and x' (h) are group actions, and f is the input. In CNNs, f is the feature map which can be seen as vector-valued function f R - R", where IR" is the n--dimeniiona vector space. If we choose to be the translation group, it is easy to prove that the convolution layer satisfies this requirement. In the following, we mainly consider feature maps defined on IR??, and the conclusions can be readily extended to feature maps defined on any dimension.
Following the standard practice of equivariant deep learning, the feature map is modeled as a vector field composed of fiber f(x) located at every point x E RR2. For transformation group H, (R²,+) )) x G, for some G < GL, R). Here, H is constructed by the semi-direct product between translation group (R² +) and linear invertible transformation group G performed on R.2, e.g. rotations and mirrorings. The group action x(h) acts on field f as:
mainly consider affine group of the form H = =(R",
Vr E R., (h))((z) = p(o)(()))))(((( (x)))),
(2)
where / E R² is translation, g G' is linear transformation, h (1, E H. and p(g) is group representation of g. Formally, a group representation p of the group G is a group homomorphism: p(g1) ((g2). It describes how each fiber transforms
G + R= i.e., M91.92 E G, p(g192) under the group action. When p is given, its corresponding feature map is called p-field. See supplementary material for more introduction group representations.
3.2 PARTIAL DIFFERENTIAL OPERATORS
Partial differential operators (PDOs) are commonly used in physics areas, such as gradient, curl or Laplacian. They can be seen as kind of maps between smooth functions. Given a smooth Gia-
3

---

Published as a conference paper at ICLR 2023
dimensional feature map f (f1: fe..) ) the PDO 0₂ acts f as 0,f
)
In general, the PDOs can be formalized as the linear combination of various orders elemen- tary PDO which is denoted as 8 := 0m 023
(2),02) E Ng. Here, adopt the multi-index notation on elementary PDOs as Jenner & Weiler (202T). For simplicity, we utilize No, S i
Iv = {(i,,))
N) index elementary PDOs with their order less than N as we have to set a truncation order to implement PDOs in the computer. For example, the PDOs from C (I22, g ) C (IR2, Cout)) ) with truncation order N =3 can be formalized
D : (o,ooff f + W W1.ooo
f + W ((..))
10.110 f + W (2.,0) f+ Wa.1joll11
W 20.2)000.2) W (3.00) W (e.n W W(22)o01.21 w (.33))s.s..
(3)
where W (1)) R R Cou is the coefficient matrix corresponding 0615),
To study the equivariance of PDOs, we give a description of the transformation property of elemen- tary PDOs. Here, we assume the input f to be scalar field and take @(2.0) as an example. When input of the PDO go through a affine transformation (g, t), resulting in f(x) == f(g- (x t)). According to the chain rule, we get:
[al2.r] (zx ( 2 (] + 29 921
+ (g r(
(4)
where Ii and 921 are matrix elements of g and t == g g-1(2 (x t). In general, for each elementary PDO:
Ya E TNE,EE G, 0-f(x) = E (o)o
(5)
BET'
where (g) denotes the transformation coefficient in front of elementary PDO 8 on the right side of the above equation of a given a. All these transformation coefficients of given group element constitute matrix p(g). We have the following result:
Lemma p(g) defined in Eq. is a group representation of on R|Tx.
Proof of the lemma can be found in supplementary material. We also give a procedure in the sup- plementary material to automatically compute p(g) for any given N.
4 THE NEURAL EPDOS FRAMEWORK
In this section, we first propose the new scheme of PDOs, in which the coefficient matrices are generated by features. Then, we propose the general theory that gives necessary and sufficient condition to ensure equivariance for the operator for any given symmetry. As coefficient generators will induce heavy parameters and computation costs, we propose require coefficient matrices to be diagonal and characterize the equivariant space for it.
4.1 NONLINEAR PDOS SCHEME
As introduced in Section 3.2 PDOs maps from O CRR2 RCn) C (RO RRat)) ) are formulated as
Vz e R²,
[f((()) Σ (=
TCT
where f E CRRRR, W. R22 + Rout XCin, Jenner & Weiler has proved that the necessary and sufficient condition for (6 to be translation equivariant is to require the coefficients W. to be spatially shared, that is, Va,, x." E IR2, E N, W (x) W, (a'). However, it is not efficient for the spatially shared PDOs to learn diverse patterns in the feature map, which may lead to the redundancy of learnable parameters.
To alleviate this problem, we propose a nonlinear PDOs scheme that adjusts PDOs according to fea- tures at different positions. Furthermore, our newly proposed module still keeps translation equiv- ariance as the spatially shared PDOs. Specifically, we adopt W a the coefficient generators to generate coefficient matrices from local input features, which can be formulated as:
Vz E 12,
[[f](z)=
Σ (F((( x
4

---

Published as a conference paper at ICLR 2023
where Wa a R Xax a E N' coefficient generators with local features input. It easy to check the translation equivariance of E7. We could adopt MLP as the structure of the coefficient generators they are the universal approximator of any continuous function. Then, the neural network and local input features decide the specific PDOs applied at each position.
4.2 EQUIVARIANCE THEORY
Although the nonlinear PDOs formulated in Eq. 7 equivariant translation, they not in- trinsically equivariant to common transformations such as rotation or reflection. We now derive complete characterization of their equivariant space of such symmetry.
The equivariant requirement on the operators 77. in the sense defined by Eq.(D, can be reduced to the requirement on the coefficient generators Wa. Supposing the input and output of the operator are any p-field and p' p'-field, respectively, we have,
Proposition 1 The nonlinear PDOs in Eq. 7 are equivariant to affine transformation H if and only if the coefficient generators satisfy the following constraint:
Ya E THEIN.YO E G. G. R
E BETN
(8)
The proof of this proposition be found in supplementary material. The proof makes of the fact that elementary PDOs are independent of each other. It remarkable that the above equation constraints for coefficient generators are imposed for each a E I'v. To uncover the intrinsic struc- ture of War we further concatenate all the coefficient generators side by side as a whole W such Then, the constraint Eq reduces to the
that Vy R W(y)
following form:
Proposition 2 Eq. equivalent to the following form:
Yg R W(p(g)))(Pg)) & p(g)) p'd)
Here, & is the tensor product of two group representations.
We show the proof supplementary material. Applying the vec-operato on Eq 9...9) can be
rewritten as:
Vg G, E R vec Wcc()((())) = (p'(g) @ TT @ pla Dvec W(y),,
(10)
where veel-| operator flattens the matrix into vector by concatenating the rows of a matrix one by TO) reveals that vec [ an equivariant function with input and output vector transform according to p(g) and p' (g) @ p(g 1) & p(g respectively. It is easy to check that p' (g) & p(g-1) pig ' also a representation of G. As we adopt coefficient generators as MLP, vec W is an EMLP and can be efficiently constructed by Finzi et al. (202)).
one. Eq. [0)
coefficient matrices in such operators are generated via MLP, brings much extra computational burden compared to steerable PDOs. To alleviate the problem, we propose novel structure for coefficient generators and give its equivariant characterization in the following.
4.3 EFFICIENT COFFFICIENT GENERATORS
In practice, regular representation and quotient representation (see supplementary material) mostly adopted for cquivariant networks (Weiler Cesal 2019) due to their superior performance. Therefore, attempt to propose efficient coefficient generator for these representation types in this subsection. In Eq. directly generating the whole coefficient matrix W would make suf- fer heavy parameter and computation costs. To alleviate this issue, we assume that the coefficient matrices in Eq.(7) ( are diagonal matrices. Then we can formulate the operator 7 as:
Vx E IR?
([f((x)= = Σ
(f(z)) (f(())))))
(1I)
N
For matrices X, have vec(AXB) (A \vec(X).
5

---

Published as a conference paper at ICLR 2023
where Wa IR + R and is used to denote element wiie product between two vectors. Here, we have assumed output and input to be of the same field type and, if necessary, we can follow it with linear projection to transform it into another field. Such design greatly reduces the computational burden for generating coefficients and also works well in the experiment in Section 7
According to Proposition] 1
the above operator 11 is equivariant if and only if:
Va E I',, Vg E Gy S
Σ BEI
ò3.a (p.alo)ddigglwo(oll)))))e)
p(g) (y)]- (12)
We use diag[=] to denote converting an n--dimensional vector to an n-dimensional diagonal matrix with the vector as diagonal. Because of the diagonal operation, directly using results in Proposition P cannot uncover the structure of wa satisfying the above constraint. By utilizing the special structure of regular representation and quotient representation, we have the following result.
Proposition 3 Suppose the input and output of operator regular or quotient representation of G, the constraint in Eq. 12 is equivalent to:
IT) are both p(g)-fieel field. If the p
Vg E G. E w(p(g)y) (p(g) @ p(g ()
(13)
(y) ())))) is large vector concatenated from all generated vectors (see supplementary material for proof). Similar to the above general case, is an equivariant function with input and output vector transform according to p(g) and (p(g) @ p(g-t) respectively. So far, we have fully characterized the structure of the coefficient generators which ensures operator (TT) is equivariant. As the coefficient generator based on an equivariant neural network, we name our model Neural ePDOs. In the next section, we will show detailed implementation that is both parameters efficient and computationally efficient.
where Vy E IR w(y)= vee oe
5 IMPLEMENTATION OF NEURAL EPDOS
5.1 DESIGN OF COEFFICIENT GENERATOR
As is shown in Proposition 3 coefficient generator w can be viewed as an equívariant vector-valued function. In practice, we implement it as EMLP which can be constructed as Finzi et al. (2021). In this paper, we adopt a two-layer EMLP as w. To reduce parameter and computation costs, we choose bottleneck design here, relatively small size of the hidden layer. Specifically, w(x)
(PRRLu(W'zz) where W E R W E R (ean. Xfmidd Cmidd Sin. where is the reduction ratio, and ReLu(-) is a element-wise ReLu activation function (Nair & Hinton 2010)
Here, we assume p PPo, in other words, representation P can be decomposed into multiple identical representations Po. which is very common in practice. Then the output representation of
((x) WeReLu(W''s can be decomposed in similar way, i.e., (p @ (p ( p(po @ (p (T))))), We reduce both computations and parameters by requiring the outputs of the partitions
be the same.
5.2 DISCRETIZATION OF PDOs
Our theory for Neural ePDOs is developed in the continuous space. In practice, order to process digital images which are defined on two-dimensional grids, we need to discretize our PDOs. In our paper, we mainly consider the finite difference (FD) method and the Gaussian derivatives method (GA) (Jenner & Weiler 2021). In principle, we only need to consider the discretization of the ele- mentary PDO because the discretization of the whole PDOs can be obtained by a linear combination of them.
FD: Finite difference method is widely used in numerical analysis to approximate PDO by linear combination of function values on finite grids. For example, 0.f (f(r + 1) f(x 1))/2. On the regular grids, a PDO can be approximated by a convolution operation (Shen et all 2020 i.e.
[ R Ua F.
(14)
where u; is convolution filter. Corresponding filters for elementary PDOs in P are provided in the supplementary material.

---

Published as a conference paper at ICLR 2023
GA: PDO can also be estimated by taking derivatives of Gaussian function (Jenner & Weiler 2021 i.e, given grid points In E IR, Ya E I',,
(15)
where (( σ)) is Gaussian kernel with standard deviation around O.
6 COMPLEXITY ANALYSIS
In this section, give complexity analysis for both stecrable PDOs and Neural ePDOs. For simplicity, we assume the feature field to be a regular field (Results for other feature fields are similar). We especially consider the complexity for Neural ePDOs with full coefficient matrices (denoted as full) and diagonal coefficient matrices (denoted as diag), respectively. Here, we assume the EMLP of Neural ePDOs (full) have the same design as Neural ePDOs (diag). Suppose the representations type of the input feature map and output feature map to be cpreg The width and height of the feature map are h and w, respectively. n is the number of elements in the group G and k is the discretization kernel size, Both parameters and flops complexity are listed in Table (
We first make comparison of parameters. For both the Neural ePDOs (full) and Neural ePDOs (diag), the first term is the number of parameters in the first layer of EMLP (coefficient generator) and the second term for the second layer. It is obvious that Neural ePDOs (full) have signifi- cantly more parameters than Neural ePDOs (diag) (O(c") vs O(c²). For comparison of parameters between steerable PDOs and Neural ePDOs (diag). both the first term and second term in Neural ePDOs (diag) are much less than the parameters of steerable PDOs (In our paper, we set N 4), hence Neural ePDOs (diag) is much parameter efficient than steerable PDOs.
For flops, the first two terms in the Neural ePDOs (both full and diag) used for coefficient generation and the last term is used for the action of PDOs. As the last term in Neural ePDOs (full) equal to steerable PDOs, the latter one surely requires less computational burden. It is also easy to check all three terms in the Neural ePDOs (diag) are much less than flops of steerable PDOs, hence Neural ePDOs (diag) much more computationally efficient than steerable PDOs.
From the comparison above, both the diagonal restriction and EMLP design (bottleneck structure (r) and partition (p) operation) help to make our Neural ePDOs more efficient than steerable PDOs.
Table I: Complexity Analysis of each PDO layer.
Method
Parameters
Flops
Stecrable PDO
(N + 1)(N + 2)n/2
e n k2 hw n /rp) + e k
Neural ePDOs (full)
en/rr + (N - 1) + +2)n/2rp
(e
Neural ePDOs (diag)
en/r+ + 2(N + I)(N + 22m//rp
(cn2 /r Zn² k22 /rp) + onk hew
EXPERIMENTS
7.1 MNIST-ROT
We first test our model on MNIST-rot dataset (Larochelle et al., 2007). which is a standard bench- mark to test the equivariant models. The dataset contains 62k 28 x 28 randomly rotated gray-seale handwritten digits. Images in the dataset are split into 12k for training and 50k for testing.
As the images in the MNIST-rot dataset are prientatiin-unkkooe model. Following the architecture in the Jenner & Weiler (202I) that consists of 6 steerable PDOs layers followed by two fully connected layers, we construct our model by replacing the last 5 steerable PDOs layers with our Neural ePDOs layers. More details about the model, training and hyperparameters analysis can be found in the supplementary material.
we choose the group as Cie for
Our results be found in Table Some of our models use the regular field intermediate feature fields and others use quotient representations (which are denoted by quotient in Table 22.. Under the setting of using both finite difference and Gaussian derivatives, our model achieves significant

---

Published as a conference paper at ICLR 2023
Table 2: Results in MNIST-rot. The test error with standard deviations are averaged over 5 runs.
Method
Discretization
Test error (%)
Params
Vanilla CNN
1.9611006
1.IM
Stecrable PDOs
FD
1.54=0.32
941K
Neural ePDOs
FD
0.77=10.05
234K
E2CNN(quotient)
0.70=00.33
2.75M
Steerable PDOs (quotient)
Gauss Gauss
0.74
951K
Neural ePDOs (quotient)
0.00
229K
E2CNN
0.72.10.03
2.69M
Steerable PDOs
Gauss
0.75+0.02
941K
Neural ePDOs
Gauss
0.65
234K
E2CNN (D 1616
0.68+0.02
5.36M
Steerable PDOs (16)
Gauss Gauss
0.78
947K
Neural ePDOs (D (1e)
0.59=0.03
525K
Table 3: Results for ImageNet100. DA is short for data augmentation. The test error with standard
deviations are averaged over runs.
Method
Discretization
DA
Test emrrr (%)
Params
ResNet26
23.50= 0.46
13.12M
Res26 Steerable PDOs
FD
X
19,92 0.24
14.4M
Res26 Steerable PDOs
Gauss
18.64 0.29
14.4M
Res26 Neural ePDOs
FD
X
17.76= 0.26
8.24M
Res26 Neural ePDOs
Gauss
17.44 0.25
8.24M
ResNet26
15.11 0.25
13.12M
Res26 Steerable PDOs
FD
13.94 0.15
14.4M
Res26 Steerable PDOs
Gauss
12.54 0.18
14.4M
Res26 Neural ePDOs
FD
12.08 0.13
8.24M
Res26 Neural ePDOs
Gauss
11.84 0.14
8.24M
improvement over the steerable PDOs based model with only 1/4 parameters. It is noteworthy that the Gaussian derivatives method tends to achieve a better performance than the finite difference method for both steerable PDOs and model, which is consistent to experiment results in Jenner & Weiler (202T). However, the performance of our quotient field model is inferior to the regular field, which different from the results of steerable PDOs in Jenner & Weiler (202T). ). As is shown in Weiler & Cesa (2019), quotient fields can help to reduce redundancy in the regular fields in some cases. our models, the redundancy already less than steerable PDOs, hence, applying quotient fields in our model may hurt performance. To further demonstrate the potential capacity of our model, we enlarge the model size and employ D16 regular field the first five layers and restrict to C1e regular field in the last layer (results are denoted as D1G Cie in Table Our model improves the current SOTA model (Weiler Cesal 2019) while consuming only 9.8% of parameters.
7.2 EVALUATION ON NATURAL IMAGES
In general, the objects in real-world images are always uniform orientation. So believe that the models with rotation symmetry can generalize better on real-world images. ImageNet (Deng al. 2009) a large-scale dataset that consists of 1000 classes with roughly 1000 images per class, which Is a common benchmark for image recognition. It contains 1.2 million training images and 50k validation images. Following Hou et al. (2019), we consider two experimental settings which correspond to different data scales. In the first setting, we conduct the experiments a subset of ImageNet which randomly select 100 classes, and denoted it as ImageNet100. In the other one, we evaluate our model the whole 1000 classes.
We choose ResNet-26 as the baseline model and construct equivariant PDO-based networks that are cquivariant to Cs group by taking place the convolution layer with Css equivariant module. We adopt regular representation for all the equivariant modules throughout the model. To make classification results votation-invariant to the inputs, add orientation pooling (Cohen & Welling 2016a)
8

---

Published as a conference paper at ICLR 2023
Table 4: Results for ImageNetlk. The test emor with standard deviations averaged over runs.
Method
Discretization
Test error (%)
Params
ResNet26
26.4 0.52
13.7M
Res26 Steerable PDOs
FD Gauss
25.39 0.35
15.02M
Res26 Steerable PDOs
23.55+ 0.31
15.02M
Res26 Neural ePDOs
FD Gauss
24.02:10.27
8.82M
Res26 Neural ePDOs
23.15 0.33
8.82M
before global average pooling. Notice that the convolutions in ResNet26 are simply replaced with equivariant linear projection layers rather than the PDOs layer. Other convolution layers are replaced by steerable PDOs Neural ePDOs which we denote Res26 Steerable PDOs or Res26 Neural ePDOs. The output dimensions of each layer for the two models are scaled with the same factor make its learnable parameters in Res26 Steerable PDOs comparable with the baseline. More detailed training settings can be found in the supplementary material.
Results on ImageNet100 and ImageNetIk are shown in the Table and Table respectively. In all the settings, our models significantly improve Steerable PDOs based models with fewer parame- ters (8.2M vs 14.4M) and computational costs (flops=24.1G with/without data augmentation is adopted. It is observed that equivariance can help improve the performance of the model and data augmentation can further enhance the performance the equiv- ariant model, which may be attributed to the approximate equivariance of the equivariant model incurred by discretization. In addition, we observe that performance improvement of Neural ePDOs over Stecrable PDOs under data augmentation setting is more significant. We conjecture our Neural ePDOs can help our model casier adapt to unseen patterns and hence tend to be more data efficient. It is also noticeable that the Gaussian discretization method still outperforms the finite difference method on the natural images.
56.6G). For ImageNet100, training
8 CONCLUSION AND FUTURE WORKS
In this work, we propose a new nonlinear PDOs scheme that is both spatially adaptive and translation equivariance. A new equivariant theory is developed for nonlinear PDOs scheme which gives a general equivariant formulation of it under any given symmetry group. The theory systematically characterizes the space of coefficient generators of our equivariant nonlinear PDOs for any given equivariance. Based on this theory, we efficiently implement it by adopting two-layer EMLP as the coefficient generators and, hence, experiments demonstrate our Neural ePDOs can significantly improve performance on MNIST-rot and ImageNet datasets. Particularly, we achieve new SOTA performance on MNIST-rot dataset with only 9.8% parameters compared to the previous best model.
our model Neural ePDOs. Extensive
To reduce the parameters and computational costs of generating coefficients, we have proposed efficient coefficient generators which are designed for features of regular field and quotient field. Efficient coefficient generators for other representation fields, e,g, irreducible representation field, could be further explored.
worth emphasizing that the nonlinearity introduced in our adaptive PDOs has greatly improved their performance compared to linear PDOs even with smaller model sizes. There is still large space to explore how to introduce nonlinearity into PDOs more efficiently.
Although mainly focus on the two-dimensional plane, our framework to other homogeneous spaces such as spheres and 3D spaces. As our Neural ePDOs can achieve significant improvement the two-dimensional image tasks, it is worth believing applications of Neural ePDOs in these domains are promising.
be readily extended
ACKNOWLEDGMENT
Z. Lin supported by National Key RD Program of China (2022ZD0160302), the major key project of PCL, China (No. PCL2021A12), the NSF China (No. 62276004), Qualcomm, and Project 2020BD006 supported by PKU-Baidu Fund.

---

Published as a conference paper at ICLR 2023
Reproducibility Statement
The complete proof of the theorems is provided in supplementary material, and all the experimental details are provided in Section 7 and supplementary material.
Ethics Statement
The research in this paper does NOT involve any human subject, and our dataset is not related to any issue of privacy and can be used publicly. All authors of this paper follow the ICLR Code of Ethics https//licc.cc/uuliiiccddddEddhs
REFERENCES
Erik J Bekkers. B-spline CNNs lie groups. arXiv preprini /90G::/999./20000 2019.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling, Ge- ometric and physical quantities improve E (3) equivariant message passing. In International Conference on Learning Representations, 2021.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834 834,848, 2017.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning. pp. 2990-2999) PMLR, 2016a.
Taco S Cohen and Max Welling. Steerable CNNs. arXiv preprint arXiv:1612.08499. 2016b.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. arXiv preprint arXiv:1801.101 1801.10130, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hi- erarchical image database. In 2009 IFFE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolu- tional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference Machine Learning, pp. 3165-3176. PMLR, 2020a.
Mare Finzi, Max Welling, and Andrew Gordon Wilson. practical method for constructing equiv- ariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning, pp. 3318-3322 PMLR, 2021.
Mare Anton Finzi, Roberto Bondesan, and Max Welling. Probabilistic numeric convolutional neural networks. In International Conference Learning Representations, 2020b.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE (3)-transformers: 3D roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
Deepak K Gupta, Devanshu Arya, and Efstratios Gavves. Rotation equivariant siamese networks for tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 12362-12371, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Lingshen He, Yuxuan Chen, Yiming Dong, Yisen Wang, Zhouchen Lin, et al. Efficient equivariant network. Advances in Neural Information Processing Systems, 34:5290-5302, 5302, 2021a.
Lingshen He, Yiming Dong, Yisen Wang, Dacheng Tao, and Zhouchen Lin. Gauge equivariant transformer. Advances in Neural Information Processing Systems, 34:27331-27343, 2021b.
10

---

Published as a conference paper at ICLR 2023
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups. In International Confer- ence on Machine Learning. pp. 1533-4543 PMLR, 2021.
Erik Jenner and Maurice Weiler. Steerable partial differential operators for equivariant neural net- works. arXiv preprint orXiv:2106.10163, 2021.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empir- ical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference Machine learning, pp. 473 480, 2007.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, 2010.
Stanley Osher and Leonid Rudin. Feature-oriented image enhancement using shock filters. SIAM Journal on numerical analysis, 27(4):919-940, 1990.
Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffusion. JEEE Transactions pattern analysis and machine intelligence, 12(7):629-639) 1990.
David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoorn. Attentive group equivari- ant convolutional networks, In International Conference on Machine Learning, pp. 8188-8199, PMLR, 2020.
David W Romero and Jean-Baptiste Cordonnier. Group equivariant stand-alone self-attention for
vision. In International Conference on Learning Representations, 2020.
Zhengyang Shen, Lingshen He, Zhouchen Lin, and Jinwen Ma. PDO-eConvs: Partial differential operator based equivariant convolutions. In International Conference on Machine Learning, pp- 8697-8706. PMLR, 2020.
Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller and Jan Kautz. Pixel- adaptive convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 11 11166-11175, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Maurice Weiler and Gabriele Cesa. General E (2)-equivariant steerable CNNs. Advances in Neural Information Processing Systems, 32, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. Advances in Neural Infor- mation Processing Systems, 31, 2018.
Daniel Worrall and Gabriel Brostow. Cubenet: Equivariance to 3d rotation and translation. In
Proceedings of the European Conference on Computer Vision (ECCV), pp. 567-584, 2018.
Jialin Wu, Dai Li, Yu Yang, Chandrajit Bajaj, and Xiangyang Ji. Dynamic filtering with large sampling field for convnets. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 185-200, 2018.
Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Ming-Hsuse Yang. Decoupled dynamic filter networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6647-6656, 2021.
11


Reviews:
[{"review_id": "rIArsN9S5V", "reviewer": "Reviewer_rHY6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a spatially adaptive equivariant partial differential operator (ePDO) based networks. Motivated by the prior work on ePDO by Jenner & Weiler (2021), where a linear PDO is restricted to have constant coefficient matrix shared spatially for translation equivariance, the authors propose a nonlinear PDO scheme that is both spatially adaptive and translation equivariant. The spatially-adaptive coefficients in this work are instead learned based on the feature vector using a generator. When the symmetry group is an affine group acting on $\\mathbb{R^2}$, constraints on the coefficients have been derived to ensure equivariance. The coefficient generation (subject to the derived equivariant constraint) is implemented using an equivariant multilayer perceptron. Numerical experiments are conducted to demonstrate the improved performance on MNIST-rot and Imagenet classification.", "summary_of_the_review": "I think the paper is well-motivated and it has high quality in terms of both novelty and clarity. However, the experimental sections of the paper can be improved. I am willing to improve my rating based on the authors' response.\n\n**Post author feedback**\nThe authors answered my concerns in detail. I am willing to increase the rating.", "strength_and_weaknesses": "**Strength**\n1. The paper is well-written and organized. It is pleasant to read.\n2. The motivation of the proposed work is very clear: linear PDOs with translation equivariance is restricted to have shared coefficients. To achieve spatial adaptivity and maintain equivariance, nonlinear models are necessary.\n3. Even though I have not read the details of the proofs, the theoretical results look natural and convincing.\n4. In terms of implimentation, the authors reduce the model size by learning a diagonal coefficient matrix using an equivariant multilayer perceptron, which is novel.\n\n**Weakness**\nMy concern of this paper mainly stems from the experiments\n1. Compared to the prior work steerable PDOs, the proposed neural ePDOs, because of the additional eMLP for coefficient learning, will surely have a larger model size even when learning only a diagonal coefficient matrix. Can the authors clarify why there are less params for neural ePDOs compared to steerable PDOs in the tables?\n2. Since the dataset use MNIST-rot (without flipping), what is the benefit of $D_{16}$ as the symmetry group?\n3. This is related to the second question: can the authors report the result of E2CNN($C_16$) instead in table 1?\n4. Can authors report the comparison of the actual training/testing *time* instead of the computational flops in the experiments?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GXnkr_xOMVm", "reviewer": "Reviewer_V42w", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes an equivariant implementation of non-linear partial differential operator (PDO) based networks. A key contribution of the paper is in the non-linear adaption of equivariant PDOs (ePDOs), which makes them more parameter efficient and expressive. The idea is intuitive and sensible, based on the idea that the operators should be locally adapted to local patterns. This is achieved by letting a local feature vector determine the PDO coefficients locally. This coefficient predictor then has to be equivariant as well, and this is achieved by equivariant MLPs. The paper then provides a clear recipe/theory for the conditions under which the PDO is indeed equivariant, which guides the construction of an efficient implementation. The proposed method is extensively validated on rotated MNIST as well as on ImageNet, and confirm the added benefit of working with non-linear ePDOs as opposed to linear ePDOs.", "summary_of_the_review": "The paper is a great submission to ICLR; it is novel and sound, but could still benefit from additional details and intuitive explanations. I judge these improvement to be possible within the rebuttal period/before the cam-ready as they can mostly be fixed with textual improvements. I therefore recommend accept.", "strength_and_weaknesses": "**Strengths**\n\n1. The paper implements an intuitive and appealing idea: making the main layers of NN non-linear and adaptive to the present features locally. In some sense this also happens in transformers via attention, or via deformable convolutions [Dai] and possibly other works, however, what makes this approach interesting is that is efficient in that the operations (PDOs) are strictly local. Their non-linear adaption boilos down to a point-wise (equivariant) MLP.\n2. The proposed work is timely, and builds up recent advances in PDO-based deep learning.\n3. The experiments are well done and clearly underpin the benefit of non-linear ePDOs vs linear ePDOs and normal resnets.\n4. The paper is theoreticall sound.\n5. The paper is overall well written and precise, but could occasionally benefit form added details and intuition.\n\nDai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017). Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 764-773).\n\n**Weaknesses**\n\n1. Upon first reading, it is hard to distill the differences between the ePDO of Jenner and Weiler, from the contributions in this paper. It is clear though that it is the modification of the PDO coefficient being dependent on the input (which is an important contribution in itself!), however, the kernel constraint is very hard to digest. It would have been nice to put this in perspective relative to the original constraint posed in the paper by Jenner and Weiler.\n2. The paper is a bit untransparent when it comes to the \"steerabble\" aspect of things. Parts are clear, vector that collects derivatives can in principle be steered by some representation of G, like in Eq.5, but it is unclear what the types of the input and output feature fields are, and how to set them. It may help a lot of some intuition is provided on how to pick those reps, also in relation to the kernel constraint. Overall, I think an intuitive breakdown of Eq 8 would be helpful.\n3. Some minor details are missing: \n    1. to which sub-group is the quotient representation defined?\n    2. The method is overall more efficient (in memory and flops), could it be made more precise where this effiency comes from? The operations themselves are more expensive than the original ePDOs, right? Is it mainly due to working with less independent feature channels (e.g. choosing a lower p in the non-linear case compared to the linear ePDO setting)?\n    3. In proposition 3, why is it important to consider regular or quotient representations (which are also regular), and not irreducible representations? I.e., why not formulate it for any representation of G?\n    4. Might this have to do with sec 5.1 where the equivariant MLPs use ReLUs. If working with irreps, not all activation functions may be allowed (though this seems to have nothing to do with proposition 3 otherwise).\n    5. What sigma did you pick for the Gaussian derivatives?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "86CP8S4U4_B", "reviewer": "Reviewer_vWQG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs a novel neural ePDO. PDO has wide applications but needs linearity to ensure the translation equivariance. This paper proposes to use an equivariant MLP as the neural ePDO to realize the translation equivariance. The authors give comprehensive theory and experiments.", "summary_of_the_review": "Overall, I recommend accept.", "strength_and_weaknesses": "I am impressed by the designed method. The authors design a novel equivariant MLP as the neural ePDO to approximate the PDO with translation equivariance. This idea is novel and makes sense.\n\nThe authors theoretically verify the translation equivariance of the proposed neural ePDOs. \n\nThe authors also conducted comprehensive experiments on MNIST and ImageNet. The results show the proposed method significantly outperforms existing methods.\n\nMy major concern is on the literature review and comparison. The authors are encouraged to well position the present work in the existing works.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7jpb835A_A", "reviewer": "Reviewer_j4DB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an equivariant neural network architecture based on partial differential operators (PDOs) in the layers.  These operators are constrained to be translation and rotation equivariant.  Relative to previous work, PDO-eConv (Shen et al, 2020) and Steerable PDO (Jenner & Weiler , 2021), this work adds non-constant coefficients to the PDOs.  This changes the equivariance constraint relative to prior work and the authors work out the new equivariance constraint.  They also provide a version of the constraint under the assumption the operators are diagonal.  Experiments show the architecture is more efficient than Steerable PDO as it attain better accuracy on rotated MNIST and ImageNet using fewer parameters.  ", "summary_of_the_review": "This paper makes a novel and reasonable generalization of previous equivariant PDO work.  The derivation and method seem sound. The experimental results seem quite strong although not as thorough as they could be.  The choice of experimental domains is limited to invariant classification task.  The paper presentation and polish could use work.  ", "strength_and_weaknesses": "### Strengths \n- PDO based networks have been considered by several authors recently including equivariant versions and its reasonable and novel to consider an extension with non-constant coefficients.  In order to maintain shift equivariance, the coefficients must be functions of input function.  So $ f (\\partial_x f)$ is okay but $x (\\partial_x f)$ is not.   \n- The authors do a good job deriving the equivariance constraints for the coefficient networks in both the full and diagonal case.\n- The experimental results are pretty convincing. The most important baseline is Steerable PDO and that is compared to.  Variations such as discretization, different intermediate representation types, and data augmentation are considered.  In both experiments the current method outperforms using fewer parameters than baselines. \n\n### Weaknesses / Questions \n- Some things are missing from the experiments.  Variance should be reported over multiple runs.  How are the hyperparameters tuned?  It is not clear they were.  How does performance of the model and baseline vary as the number of parameters vary?    I'd also like to see a non-equivariant ablation using non-linear PDOs but without the equivariance constraint.\n- The diagonal operators are claimed to be more efficient.  I'd like to see how much more efficient.  That is, I'd like to see a comparison to the largest practical model which can be built using full matrices. \n- The experiments are both image classification.  Given that the motivation for PDOs is in differential equations, it would nice to see an application for modeling differential equations or a non-invariant application.  One is left wondering what specifically about PDOs is important for solving vision tasks?  If I replaced the FD $\\partial^{\\mathbf{i}}$ operator with other fixed operators with similar statistics would I get similar performance?  If so, our understanding for why this network is effective changes.  Perhaps, it is merely that the operators $\\partial^{\\mathbf{i}}$ provide a more efficient basis for the space of spatial kernels than elementary matrices and nothing to do with their meaning as PDOs. \n- In 5.1, it says $\\rho = p \\rho_0$.  If true, this seems very constrained.  It would be good to do a comparison to other choices.  That said, I don't really understand where $p \\rho_0$ is actually used in the architecture since the experiments section refers to using regular and quotient representations. \n- Is it clear the diagonal assumption is not overly restrictive?  The experiments offer good evidence it is not, but it would be nice to see theoretical evidence as well. \n\n### Minor Points\n6.1 Para 2, Incorrect Reference to Weiler & Cesa for PDOs.  \nEqns. 9, 10.  The notations for how the tensor product representations act on the coefficients are a bit unclear.    \nPage 4 \"discription\" -> description\nPage 3 \"semi-product\" -> \"semi-direct product\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%