Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents the Bispectral Neural Network (BNN), a network primitive which learns to be invariant to transformations present in the data. The model is defined in terms of two main components. Firstly, the model treats the irreducible representations of a group of transformations as weights in the network, thus essentially learning a Fourier transform on the group, which is equivariant to the group action. The Fourier representation of the input is then mapped to the bispectrum, which is invariant to the transformation but retains the information needed to uniquely restore the image (up to the transformation), by the network. Secondly, a novel loss function encourages the bispectrum for inputs of the same class to be identical, effectively making the trained network invariant to the transformation. A crucial point here is that the group of transformations need not be defined a priori; the network discovers the transformations automatically.

The authors test the model in four different experiments, demonstrating that it can construct the transformation group from data alone, that the learnt irreducible representations are transferable to other datasets, and that the model is robust to adversarial attacks.
strength_and_weaknesses: ### Strengths
Overall, I find this paper very strong. While the bispectrum is extensively used in other fields, it does not seem to be the case in machine learning (as far as my searching abilities go). Simply introducing this incredibly useful concept to the machine learning community has value in itself, and then further parametrising it in terms of a neural network, which, given a novel loss function, learns the group of transformations directly from data, makes this paper very strong. The simplicity and elegance of the proposed method are intriguing, and I could see the ideas introduced here forming the basis of many future papers on invariant and equivariant models.

In summary, the paper
- presents an incredibly useful concept (the bispectrum),
- introduces a novel and interesting loss function, and
- demonstrates how to learn irreducible representations of the transformation group using a neural network.


### Weaknesses
The paper appears to be mostly a proof-of-concept. The datasets used in the experiments are quite simple, which the authors acknowledge themselves. It is also unclear to me if the method is usable in practice in terms of the computations that are required to compute the bispectrum. Since the complex weight matrix needs to be of size $n \times n$, $n$ being the number of input dimensions, does the method scale to inputs of even moderately large dimensionality? A study of this, e.g., in terms of the training time and memory cost for increasing image sizes, would have been interesting. The authors say that they are currently working on a localised version of the model, such that it can be used in a convolutional neural network, which would address this issue. Still (and without knowing exactly how much work this entails), one might have expected such an extension to be included in the current paper.

I see these weaknesses as minor, though, given the significance of the model and the ideas and concepts the paper introduces.

In summary, I see the main weaknesses as being that
- the model is only tested on simple datasets, and that
- it is unclear if the model scale to inputs of moderately large dimensionality.
clarity,_quality,_novelty_and_reproducibility: The quality of the paper is exceptionally high. As someone relatively new to the field of equivariances in machine learning and entirely new to group representation theory, I found the paper extraordinarily pedagogical. Appendix A is very helpful and section 2 is well-written and easy to understand despite the abstract topic. The method itself appears to be novel (I wasn't able to find anything similar) and the ideas and concepts could have a significant impact on the fields of invariances and equivariances in machine learning. The authors provide the code to reproduce all experiments and figures, though I did not try to run this. In general, the paper should be of significant interest to the ICLR community. 


**Questions for the authors**

1. For the loss, did you experiment with other norms? For images, in particular, the $L_2$ norm is not very informative, but I cannot intuitively see if this could cause problems.
2. As mentioned under "weaknesses", it is unclear to me how well the method scales. I don't expect any new experiments, but do you have a feeling for how large inputs the method can handle?
summary_of_the_review: The paper presents a novel method for learning representations of data that are invariant to transformations of the data. Not only is the method novel, the ideas and concepts presented in the paper are intriguing and should be of significant interest to the ICLR community. The experiments are somewhat weak, though, and the paper seems to be mostly a proof-of-concept, yet the strengths far outweigh these weaknesses. I view the paper as a very significant and impactful contribution to the fields of invariance and equivariance in machine learning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a neural network architecture, Bispectral Neural Networks(BNNs), which aims to learn the group-invariant representations of the actions of compact commutative groups. This simple architecture is composed of two layers: a single learnable linear layer, followed by a fixed collection of triple products computed from the output of the previous layer. Then, some comparison experiments are conducted to show the effectiveness of the proposed method. 
strength_and_weaknesses: strengths:
1. This paper has comprehensive and rigorous mathematics proof. 
2. The network is simple yet effective, and the competitive results show the effectiveness. 
3. This paper has demonstrated that the completeness property endows these networks with strong adversarial robustness. 
clarity,_quality,_novelty_and_reproducibility: This paper presents a novel approach for learning groups from data in an efficient, interpretable, and fully feed-forward model that requires no prior knowledge of the group, computation of the exponential map, or bayesian inference. 
summary_of_the_review: From the manuscript, the authors propose BNNs, which could simultaneously learn groups, their irreducible representations, and corresponding complete invariant maps purely from symmetries implicit in data. I think it is good.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The idea is to learn inverariance to commutative groups. The groups are learned from data through the proposed block, which relies on the bispectrum idea. The proposed block (depicted in Fig. 2) resorts to a third-order polynomial that can be implemented on standard frameworks, as it requires an affine transformation of the input and a Hadamard product. In addition to this, an orbit separation loss is proposed. 
strength_and_weaknesses:  * [+] Including symmetries in neural networks is an interesting research direction. 

 * [+] The writing is mostly clear, while I like the summary of the group notation in the appendix. 

 * [+] The proposed block for the bispectrum is simple. 

 * [-] The experiments are weak at the moment. To be specific, I do not find much the application of this in real-world data in the experimental section. 

 * [-] The claim on the adversarial robustness seems exaggerated (see below). 

clarity,_quality,_novelty_and_reproducibility: In table 1, what are the FLOPs of each method? It would help to compare them in addition to the comparison of accuracy. 

The paper mentions that it is not outperforming the compared methods but it has the benefit of learning the group from the data. However, it is not clear to me what is the real-world application of this; definitely I do not notice any related experiment. 

In sec. 4.3 there is the claim that: “The goal of many adversarial attacks is to discover input points which are similar to a target sample in the model’s representation space, yet perceptually dissimilar to the true sample in the input space.”. Are there some citations for this? 

Many of the most popular adversarial attacks are FGSM, PGD, C&W; in all of these cases the idea is the input image to be perceptually similar to the adversarial image. Are there any results on those benchmarks to demonstrate how the proposed network is more robust to those common attacks? That is, since the manuscript mentions that the method is “strongly robust to adversarial attacks”. 

It’s not clear to me what is the role of the proposed bispectral block or the orbit separation loss in practice. Could the authors elaborate on their relationship and the sensitivity to the hyper-paremeter $\gamma$? 

To continue on the previous line: are there any numerics on adversarial robustness when compared with other methods? At the moment only table 1 has numeric results in the main papers and most of the methods score above > 98%, which does not make it easy to appreciate the contributions of the proposed method. 

Minor: Instead of “the context of vision” it should be the context of image/scene recognition in the introduction, since not all the vision tasks require the same symmetries. 

summary_of_the_review: The proposed bispectral block is new to me, and the idea of including invariances to groups in the network is also an interesting research direction. I would say one of the weaknesses in my mind is the experimental validation that is currently weak, given the empirical nature of this work. 

_____________
After the rebuttal: See the discussion below, but the core idea is that some of the concerns have been addressed below.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new neural network called Bispectral Neural Networks (BNNs) that is designed to learn group-invariant representations of the actions of compact commutative groups. The architecture of BNNs consists of two layers: a single learnable linear layer followed by a fixed collection of triple products based on the output of the previous layer. The authors conduct experiments to demonstrate the effectiveness of BNNs.

The reviewers are positive about the paper. Though minor concerns remain, they believe the author further improve the paper in the camera ready version. 
justification_for_why_not_higher_score: The result in this paper is not so significant.
justification_for_why_not_lower_score: NA
Decision: Accept: poster