Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a functional abstraction for group-convolutional networks employing first- and second-order features. Based on this functional abstraction, a new functional form for second-order group-convolutional networks is devised, which can be made more parameter-efficient than existing approaches.
Experiments are performed on rotated MNIST and CIFAR, evaluating error, number of parameters and certain ablations.
main_review: While I enjoyed reading about the generalization idea, I feel that there is a certain amount of imprecision hampering it.

Some general comments and questions:

- The statement that the authors "propose a generalized framework of equivariant models" could be construed as having found the most general formulation. This is certainly not the case, given higher-order interactions that cannot be framed this way.

- The "spatial-agnostic problem" is not a problem. It is just an architecture choice.

- Most CNNs, and incidentally the network proposed in this paper, are not actually equivariant to pixel shifts due to the pooling

- the formulation of the kernels with neighborhoods \mathcal N(g) is a bit vague. It should be made clear that the precise indexing of the elements of the neighborhood is important if one does not want to resort to permutation invariant functions such as averaging. It is mentioned briefly when incorporating group \mathcal A in the example. A further question would be: It would seem that these neighborhoods need to exhibit some form of transformability as well. Can this be simply characterized? At some points, MLPs are proposed to handle whole neighborhoods. How would these be made to transform correctly?

- The experiments on CIFAR are a bit contrived -- please compare to fully data-augmented CNNs. If this approach is to be shown to be useful, it should be able to perform at the level of data augmentation using the symmetries encoded in the network. Data augmentation may also help in the proposed architecture because of the lack of shift-invariance below aggregated stride size.

- A demonstration of object recognition on ImageNet would truly show this approach to be efficient. This benchmark is also a form of water-shed - many architectures work on CIFAR that do not work on ImageNet

- the symmetry groups chosen are not very interesting. It would have been nice to see some more non-trivial rotation at least.

- style suggestion: s/Besides, //g
limitations_and_societal_impact: Limitations are not really discussed.
I do not foresee potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a general framework for group equivariant models which generalizes group equivariant CNNs (G-CNNs) and equivariant self-attention layers. For practical use, the paper proposes a particular framework, called E4-Net, which can significant improve previous G-CNNs with smaller model size.
main_review: Two main contributions of the paper are as follows:
- The authors propose a general framework of group equivariant models which generalizes G-CNNs and equivariant self-attention layers. The authors suggest a layer $\psi:\mathbf{f}^l \mapsto \mathbf{f}^{l+1}$ where $\mathbf{f}^{l+1}(g)$ is defined to be a sum over $\tilde{g} \in \mathcal{G}$ of values of a function, say $H$, at $g,\tilde{g},\mathbf{f}^l(g)$ and $\mathbf{f}^l(\tilde{g})$. It is proved in Theorem 1 of the paper that $\psi$ is equivariant if and only if $H$ can be viewed as a function at $g^{-1}\tilde{g},\mathbf{f}^l(g)$ and $\mathbf{f}^l(\tilde{g})$.
- A special case for the function $H$ is used to construct E4-Net, a new group equivariant layers. In E4-Net, the function $H$ is considered to be a point-wise product of a kernel generator $K_{g^{-1}\tilde{g}}(\mathbf{f}^l(g))$ and an encoder $V(\mathbf{f}^l(\tilde{g}))$. Experiments on Rotated MNIST and CIFAR shows that E4-Net outperforms classical G-CNNs with less numer of parameters and time complexity.

The results are novel and interesting. The paper is well-written. Besides, I have some comments as follows:

1. The construction for the kernel generator $K$ is described in detail in page 5. But I do not see any clear construction for the encoder $V$. It seems to me from Figure 1 that the encoder $V$ is chosen to be a linear map. If yes, is there any reason for choosing $V$ to be a linear map rather than an MLP?

2. An explanation for the relation of E4-layer with G-Conv layer and G-SA layer is needed.

3. In experiments, comparisons of the accuracy of E4-Net with G-CNNs are reasonable. Why do not you compare with G-SA neural nets?

Minor comments:
- page 4, line 132: "querys" --> "queries"
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes a generalization of group equivariant neural networks that typically rely on linear operators such as the group convolution, or pseudo linear operators such as attentive group convolution. The authors take a look at the aggregation function of the convolution operator (which is usually linear via a kernel times feature values followed by sum aggregation) and replaces it with a non-linear function that generally could depend not just on relative positions, but also on the central and neighbor feature values. The result is a more non-linear operator than group convolutions, which are a special case of this framework.

The paper is accurate and has an appropriate experimental section with decent ablation studies. The proposed work compares favorably compared to the baselines both in terms of accuracy, parameters and nr of operations (FLOPs).
main_review: I think the paper describes a great idea of modifying the main workhorse of G-CNNs (the group convolution) to something non-linear. Also the paper focusses on efficiency which is a relevant aspect of G-CNNs. The paper performs a decent set of abblation studies, though these are done on MNIST, so it may not generalize to more challenging datasets, but the method is further validated on CIFAR10 and CIFAR100. Since the paper focussed on efficiency it was a pity not to see the method being applied to problems that actually require efficient implementations (such as e.g. imagenet). I think this is one of the main limitations of the paper, but up to some other concerns (see below), I think it is a great paper.

I think he paper could mainly improve by presenting additional intuition behind the proposed method and discuss the main mechanism that motivates why it should work better (see second upcoming concern). This could possibly be achieved by explicitly framing conventional methods as special cases of this framework, which enables to precisely point to differences that could explain why the current method should work better. I find this important particularly because the method is presented as a generalization of other works, and it would be great to explicitly see related works  as special cases. I think this would help interpretation. 

All in all I have the following comments:

[Thm 1 + proof | concern about input dependency on g and \tilde{g}]
I think the construction is not very transparent with respect to the dependency of of H to it’s inputs. Namely in equation 8, the inputs of H_{g,\tilde{g}} are denoted as arbitrary open “slots” (with \cdot) for arbitrary inputs. However, these inputs in practice do depend again on g and \tilde{g}, as given in equation 7. With this in mind I don't think the proof is formally correct and is in my opinion even misleading. Though when explicitly taking the dependency into account the theorem and proof is valid . After 154 I suggest not writing \cdot for the inputs but f(g) and f(\tilde{g}) respectively. (Or please correct me if my concern is ill founded).

Related to this concern: In equation 11 x and y still depend on g or \tilde{g}. This dependency seems important but is currently obscured, is this a problem?

[related work and interpretation of the generalization]
Then, the proof of thm. 1 follows the exact same structure as thm.1 of [Bekkers] and reference [23] for the attentive version. It would be great if these works and other related works could be discussed in more detail when it comes to claims regarding this work generalizing many group equivariant works. Just to be clear, I do not dispute this claim, it is very well founded, it is just that the paper contains sentence such as “this work… delivers a unified understanding on the previous group equivariant models” (line 357). In my opinion, the understanding of this unified view could be better delivered as the equation is quite abstract and I think can be better understood with some specific (mathematically precises) special cases such as g-conv or attentive g-conv.

For example, an additional intuitive explanation of equation 12 would be very helpful. Why is this layer more powerful than standard convs?

ref:
[Bekkers] Erik J Bekkers. B-spline cnns on lie groups. In International Conference on Learning Representations, 2019.

[solution to the spatial agnostic problem?]
Then I have doubts regarding the claim of solving “the spatial-agnostic problem”. I do not understand how this problem is solved. By construction equivariant methods have to be spatial agnostic otherwise they can’t be equivariant and thus operations only depend on relative positions/transformations. The only difference now is that the convolution kernels are conditioned on the feature values at the central or neighbor locations. Maybe it is mentioned somewhere in other words, but do I understand correctly that merely the dependency of the conv kernel (as in deformable convs), or aggregation function on local features solves the spatial agnostic problem? My apologies for possibly misinterpreting parts of the paper, but perhaps the notions of equivariance vs “conditioning” can be decoupled and explicitly discussed in terms of the spatial agnostic problem. For me this part was slightly confusing.

[Practical question]
How is the aggregation function/convolution kernel parametrized. In continuous kernel convolutions such as e.g. PointConvs [Wu et al.] or LieConvs [Finzi et al.], the kernel is parametrized by an MLP (though it does not depend on neighborhood feature values). Then for every possible g^{-1}\tilde{g} the kernel is defined. Is this also the case in this paper, or is the kernel (e.g. in eq 12) indexed for a finite set of possible values for g^{-1}\tilde{g}? (made possible because only a discrete group is considered)

refs
[Wu et al.] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621–9630, 2019.
[Finzi et al.] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3165–3176. PMLR, 2020.

[Minor comments. ]
When equation 10 is presented I didn’t understand why this property was important. But then later on this is used for efficient implementations where the sub-group part is handled in one step by aggregating neighborhood information. Perhaps the section about eq 10 could be better introduced with such a motivation.

Line 217. What is “s”. I think this symbol is not defined.
Lin 292. Same here, I don’t understand what a “slice numer s” is.


Regarding implementation of eq 12. The output of V represents transformed feature values of some neighboorhood, and it should follow the same ordering as the kernel values outputted by K. Is this automatically obtained or does one have to practically code this?

Line 320, why is only the second group conv layerin a res-block replaced and not all?

Section 5.2. Since the focus is on efficiency I was a bit dissapointed not to see any application to large datasets that actually require efficiency. Where there still practical limitations?

The FLOP analysis is great, still I would like to get a sense of computation times. For example, I can imagine that all the stacking of features and what not leads to quite some overhead which, despite less FLOPs, doesn’t lead to a reduction in computation time. 




limitations_and_societal_impact: The ablation study is done on MNIST and conclusion may not generalize to more challenging datasets. Otherwise I see no concerns.
review_title: null

Meta-review: The authors have written a very convincing rebuttal that leans me toward acceptance of this paper.
Decision: Accept (Poster)