Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes an equivariant implementation of non-linear partial differential operator (PDO) based networks. A key contribution of the paper is in the non-linear adaption of equivariant PDOs (ePDOs), which makes them more parameter efficient and expressive. The idea is intuitive and sensible, based on the idea that the operators should be locally adapted to local patterns. This is achieved by letting a local feature vector determine the PDO coefficients locally. This coefficient predictor then has to be equivariant as well, and this is achieved by equivariant MLPs. The paper then provides a clear recipe/theory for the conditions under which the PDO is indeed equivariant, which guides the construction of an efficient implementation. The proposed method is extensively validated on rotated MNIST as well as on ImageNet, and confirm the added benefit of working with non-linear ePDOs as opposed to linear ePDOs.
strength_and_weaknesses: **Strengths**

1. The paper implements an intuitive and appealing idea: making the main layers of NN non-linear and adaptive to the present features locally. In some sense this also happens in transformers via attention, or via deformable convolutions [Dai] and possibly other works, however, what makes this approach interesting is that is efficient in that the operations (PDOs) are strictly local. Their non-linear adaption boilos down to a point-wise (equivariant) MLP.
2. The proposed work is timely, and builds up recent advances in PDO-based deep learning.
3. The experiments are well done and clearly underpin the benefit of non-linear ePDOs vs linear ePDOs and normal resnets.
4. The paper is theoreticall sound.
5. The paper is overall well written and precise, but could occasionally benefit form added details and intuition.

Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017). Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 764-773).

**Weaknesses**

1. Upon first reading, it is hard to distill the differences between the ePDO of Jenner and Weiler, from the contributions in this paper. It is clear though that it is the modification of the PDO coefficient being dependent on the input (which is an important contribution in itself!), however, the kernel constraint is very hard to digest. It would have been nice to put this in perspective relative to the original constraint posed in the paper by Jenner and Weiler.
2. The paper is a bit untransparent when it comes to the "steerabble" aspect of things. Parts are clear, vector that collects derivatives can in principle be steered by some representation of G, like in Eq.5, but it is unclear what the types of the input and output feature fields are, and how to set them. It may help a lot of some intuition is provided on how to pick those reps, also in relation to the kernel constraint. Overall, I think an intuitive breakdown of Eq 8 would be helpful.
3. Some minor details are missing: 
    1. to which sub-group is the quotient representation defined?
    2. The method is overall more efficient (in memory and flops), could it be made more precise where this effiency comes from? The operations themselves are more expensive than the original ePDOs, right? Is it mainly due to working with less independent feature channels (e.g. choosing a lower p in the non-linear case compared to the linear ePDO setting)?
    3. In proposition 3, why is it important to consider regular or quotient representations (which are also regular), and not irreducible representations? I.e., why not formulate it for any representation of G?
    4. Might this have to do with sec 5.1 where the equivariant MLPs use ReLUs. If working with irreps, not all activation functions may be allowed (though this seems to have nothing to do with proposition 3 otherwise).
    5. What sigma did you pick for the Gaussian derivatives?
clarity,_quality,_novelty_and_reproducibility: As mentioned above the paper could benefit from additional details and intuitive explanations, though overall the paper is sound.

The work is of high quality and relies on state-of-the-art techniques, the experiments are appropriate.

The work is novel.

In regards to reproducibility, all theoretical details as well as an appendix with details are provided, but releasing code with this submission would greatly improve reproducibility (translating the work to code may be a challenge!)
summary_of_the_review: The paper is a great submission to ICLR; it is novel and sound, but could still benefit from additional details and intuitive explanations. I judge these improvement to be possible within the rebuttal period/before the cam-ready as they can mostly be fixed with textual improvements. I therefore recommend accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces an equivariant neural network architecture based on partial differential operators (PDOs) in the layers.  These operators are constrained to be translation and rotation equivariant.  Relative to previous work, PDO-eConv (Shen et al, 2020) and Steerable PDO (Jenner & Weiler , 2021), this work adds non-constant coefficients to the PDOs.  This changes the equivariance constraint relative to prior work and the authors work out the new equivariance constraint.  They also provide a version of the constraint under the assumption the operators are diagonal.  Experiments show the architecture is more efficient than Steerable PDO as it attain better accuracy on rotated MNIST and ImageNet using fewer parameters.  
strength_and_weaknesses: ### Strengths 
- PDO based networks have been considered by several authors recently including equivariant versions and its reasonable and novel to consider an extension with non-constant coefficients.  In order to maintain shift equivariance, the coefficients must be functions of input function.  So $ f (\partial_x f)$ is okay but $x (\partial_x f)$ is not.   
- The authors do a good job deriving the equivariance constraints for the coefficient networks in both the full and diagonal case.
- The experimental results are pretty convincing. The most important baseline is Steerable PDO and that is compared to.  Variations such as discretization, different intermediate representation types, and data augmentation are considered.  In both experiments the current method outperforms using fewer parameters than baselines. 

### Weaknesses / Questions 
- Some things are missing from the experiments.  Variance should be reported over multiple runs.  How are the hyperparameters tuned?  It is not clear they were.  How does performance of the model and baseline vary as the number of parameters vary?    I'd also like to see a non-equivariant ablation using non-linear PDOs but without the equivariance constraint.
- The diagonal operators are claimed to be more efficient.  I'd like to see how much more efficient.  That is, I'd like to see a comparison to the largest practical model which can be built using full matrices. 
- The experiments are both image classification.  Given that the motivation for PDOs is in differential equations, it would nice to see an application for modeling differential equations or a non-invariant application.  One is left wondering what specifically about PDOs is important for solving vision tasks?  If I replaced the FD $\partial^{\mathbf{i}}$ operator with other fixed operators with similar statistics would I get similar performance?  If so, our understanding for why this network is effective changes.  Perhaps, it is merely that the operators $\partial^{\mathbf{i}}$ provide a more efficient basis for the space of spatial kernels than elementary matrices and nothing to do with their meaning as PDOs. 
- In 5.1, it says $\rho = p \rho_0$.  If true, this seems very constrained.  It would be good to do a comparison to other choices.  That said, I don't really understand where $p \rho_0$ is actually used in the architecture since the experiments section refers to using regular and quotient representations. 
- Is it clear the diagonal assumption is not overly restrictive?  The experiments offer good evidence it is not, but it would be nice to see theoretical evidence as well. 

### Minor Points
6.1 Para 2, Incorrect Reference to Weiler & Cesa for PDOs.  
Eqns. 9, 10.  The notations for how the tensor product representations act on the coefficients are a bit unclear.    
Page 4 "discription" -> description
Page 3 "semi-product" -> "semi-direct product"
clarity,_quality,_novelty_and_reproducibility: The paper could use some editing and proof-reading.  There are some typos and grammatical mistakes.  The introduction is also fairly long and vague.  More seriously, there are some fairly broad and undefended claims in the exposition which should be removed, softened or cited.  
- "our model is much less redundant than steerable PDOs"
- "As in practice, regular representation and quotient representation (see supplementary material) are
mostly adopted for equivariant networks for their superior performance."
- "which can be regarded as a set of functional equations that are not easy to solve."
- "the mappings between smooth functions are usually defined as partial differential
operators (PDOs) such as Laplacian or divergence in the physics area"
- "From the perspective of gradients, the parameters of the PDOs are updated by the globally pooled loss gradients, which results in sub-optimal feature learning at each position."

There does not appear to be code, but the complete proofs, method details, and experimental details are provided and so the results could reasonably be reproduced. 

Though relatively similar to PDO-eConv and Steerable PDO, this method adds non-constant coefficients to the PDOs which requires generalizing the equivariance constraint.  It is novel. 
summary_of_the_review: This paper makes a novel and reasonable generalization of previous equivariant PDO work.  The derivation and method seem sound. The experimental results seem quite strong although not as thorough as they could be.  The choice of experimental domains is limited to invariant classification task.  The paper presentation and polish could use work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper designs a novel neural ePDO. PDO has wide applications but needs linearity to ensure the translation equivariance. This paper proposes to use an equivariant MLP as the neural ePDO to realize the translation equivariance. The authors give comprehensive theory and experiments.
strength_and_weaknesses: I am impressed by the designed method. The authors design a novel equivariant MLP as the neural ePDO to approximate the PDO with translation equivariance. This idea is novel and makes sense.

The authors theoretically verify the translation equivariance of the proposed neural ePDOs. 

The authors also conducted comprehensive experiments on MNIST and ImageNet. The results show the proposed method significantly outperforms existing methods.

My major concern is on the literature review and comparison. The authors are encouraged to well position the present work in the existing works.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: Overall, I recommend accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a spatially adaptive equivariant partial differential operator (ePDO) based networks. Motivated by the prior work on ePDO by Jenner & Weiler (2021), where a linear PDO is restricted to have constant coefficient matrix shared spatially for translation equivariance, the authors propose a nonlinear PDO scheme that is both spatially adaptive and translation equivariant. The spatially-adaptive coefficients in this work are instead learned based on the feature vector using a generator. When the symmetry group is an affine group acting on $\mathbb{R^2}$, constraints on the coefficients have been derived to ensure equivariance. The coefficient generation (subject to the derived equivariant constraint) is implemented using an equivariant multilayer perceptron. Numerical experiments are conducted to demonstrate the improved performance on MNIST-rot and Imagenet classification.
strength_and_weaknesses: **Strength**
1. The paper is well-written and organized. It is pleasant to read.
2. The motivation of the proposed work is very clear: linear PDOs with translation equivariance is restricted to have shared coefficients. To achieve spatial adaptivity and maintain equivariance, nonlinear models are necessary.
3. Even though I have not read the details of the proofs, the theoretical results look natural and convincing.
4. In terms of implimentation, the authors reduce the model size by learning a diagonal coefficient matrix using an equivariant multilayer perceptron, which is novel.

**Weakness**
My concern of this paper mainly stems from the experiments
1. Compared to the prior work steerable PDOs, the proposed neural ePDOs, because of the additional eMLP for coefficient learning, will surely have a larger model size even when learning only a diagonal coefficient matrix. Can the authors clarify why there are less params for neural ePDOs compared to steerable PDOs in the tables?
2. Since the dataset use MNIST-rot (without flipping), what is the benefit of $D_{16}$ as the symmetry group?
3. This is related to the second question: can the authors report the result of E2CNN($C_16$) instead in table 1?
4. Can authors report the comparison of the actual training/testing *time* instead of the computational flops in the experiments?
clarity,_quality,_novelty_and_reproducibility: The paper is very well-written, and the message is clear. The idea builds on the prior works on ePDOs, but making it spatially adaptive is novel.
summary_of_the_review: I think the paper is well-motivated and it has high quality in terms of both novelty and clarity. However, the experimental sections of the paper can be improved. I am willing to improve my rating based on the authors' response.

**Post author feedback**
The authors answered my concerns in detail. I am willing to increase the rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces equivariant neural network-based partial differential operators (PDOs). The prior works in this space assume that weight matrixes in each layer are shared across different locations to guarantee translation equivariance.  However, this submission lifts this assumption by introducing spatially adaptive neural PODs and it shows that additional equivariance (e.g., rotation equivariance) can be obtained as long as the coefficient-generating MLPs are itself equivariant. The proposed technique is analyzed theoretically and the empirical results support the efficacy of the introduced layers. All the reviewers acknowledged the novelty of this submission, the extensivity of empirical and theoretical studies, and the clarity of writing. There were minor issues in the presentation that were resolved during the rebuttal. Given this, I am happy to recommend accept. 
justification_for_why_not_higher_score: Although the theoretical extensions are significant, the empirical results show marginal improvements compared to prior works. 
justification_for_why_not_lower_score: The paper presents novel extensions to neural PDO that are recognized by all the reviewers.
Decision: Accept: notable-top-25%