Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors explore technique towards leveraging equivariances to subgroups rather than the entire group. First they propose equivariance relaxation morphism $-$ which partially removes weight sharing constraints on the entire group (without losing out on functionality) and the mixed layer allows to learn partial equivariance as a weighted sum. The authors then propose two equivariance aware NAS algorithms - one evolutionary and the other differentiable - and show good experimental results in both cases.
strength_and_weaknesses: **Strengths:**
1. The formulation of partial/ approximate equivariances/ invariances $-$ as a mixture over subgroups is interesting and an important problem to tackle.
2. Strong experimental results for both their learning paradigms.
3. For a person with sufficient group theory background - the paper is easy to understand.

**Weaknesses and correspnding questions/ suggestions**:
1. Lack of comparison of the evolutionary paradigm with a similar approach [1] which searches over the subspace lattice  to identify te groups to be equivariant/ invariant to.
2. It is hard to see how this extends to non-finite groups which are not compact. Would suggest that the authors comment on this.
3. While the paper is understandable for someone who has the necessary group theory background - it is almost impossible for general audience - would recommend the authors add necessary background and preliminaries to the appendix.
4. Studies in out of distribution settings are missing (for e.g. train only contains images rotated by 0-15 degrees, but test has those between 60-180 degrees) - this is will give us more insight into the weights learned.

**Minor:**
1. Please make sure the title on openreview matches the title on the paper

**References**
1. Mouli, S. Chandra, and Bruno Ribeiro. "Neural Networks for Learning Counterfactual G-Invariances from Single Environments." ICLR 2021
clarity,_quality,_novelty_and_reproducibility: Quality and Novelty - The paper tackles an important problem and has a reasonable amount of novelty as described in previous section.

Clarity - While the paper is understandable for someone who has the necessary group theory background - it is almost impossible for general audience - would recommend the authors add necessary background and preliminaries to the appendix.

Reproducibility - Currently code not available
summary_of_the_review: Currently the strengths of the paper out weigh the weaknesses. I would be happy to increase my scores if the authors can answer the raised questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to search for neural architectures whose layers have equivariance that can exploit symmetries in the data. They propose two methods—network morphisms combined with evolutionary search and mixture relaxations combined with differentiable NAS—to search the space of symmetries. The methods are evaluated on some image classification tasks.
strength_and_weaknesses: ### Strengths:
1. The problem of finding the correct equivariance that can exploit symmetries in the data is important for the development of general-purpose NAS methods.
2. The proposed approaches are simple and natural procedures.
3. The writing is fairly clear and the experimental results are interesting.
4. Code will be released upon acceptance.

### Weaknesses:
1. The experimental evaluation is somewhat limited. Given the fairly direct extension of prior work (morphisms and weight-sharing) to actually developing the search algorithms, I would expect more emphasis to be placed on experimental depth. This could either come in the form of evaluation on interesting benchmarks where we might expect finding symmetries to outperform regular CNNs (e.g. Tu et al. (2022)) and/or more detailed analysis on more involved search spaces of whether the proposed methods are recovering the “right” symmetry for the dataset, even if synthetic.
2. The authors do not consider the random search baseline for their method. More generally, it would be useful to follow the NAS reproducibility checklist here (Lindauer & Hutter, 2020).

### References:
- Lindauer, Hutter. Best practices for scientific research on neural architecture search. JMLR 2020.
- Tu, Roberts, Khodak, Shen, Sala, Talwalkar. NAS-Bench-360: Benchmarking neural architecture search on diverse tasks. NeurIPS 2022.
clarity,_quality,_novelty_and_reproducibility: Clarity: the writing is quite clear.  
Quality: the empirical evaluation is somewhat limited in scope.  
Novelty: the idea of searching over equivariant groups is new and interesting.  
Reproducibility: code is promised upon publication.  
summary_of_the_review: I am currently leaning against acceptance to the rather limited empirical evaluation.

# Post-rebuttal

Given the additional evaluations, with new datasets and comparisons with baselines, I think the paper has sufficient experimental backing / interesting methodology to be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two approaches to optimizing the weights and architecture of networks with approximate equivariance.  The first approach EquiNAS_E uses an evolutionary algorithm to progessively weaken the symmetry constraints in the layers of an equivariant NN as it trains.  The second EquiNAS_D uses differentiable weights, optimized alternatingly with the normal network weights, which combine different kernel with different levels of symmetry constraints.  Both methods are tested on image classification tasks.
strength_and_weaknesses: ## Strengths 
- Equivariant networks have more hyperparameters to tune due to the choice of symmetry group.  For example, in E(2)-CNN (https://arxiv.org/abs/1911.08251), the authors experiment with symmetry groups $C_n$ and $D_n$ for many values of n before finding n=12 to work best.  This paper proposes a method for automatically determining the correct level of symmetry.
- Moreover, on many problems end-to-end strict symmetry is not desirable if the symmetry in the domain is only approximate or if symmetry constraints interfere with optimization.  The proposed method adds to the relatively few methods which learn relaxed symmetry constraints and adapt to the level of symmetry in the data.
- I found the equivariance relaxation morphism to be a simple and effective idea for relaxing symmetry constraints in networks on a per-layer basis.
- The [G]-mixed equivariant layer also seems like a reasonable idea for selecting the degree of equivariance by differentiably choosing weights over kernels with different levels of equivariance.  In theory this allows the model to learn a specific equivariance or an approximate equivariance which is in-between.  It also allows the network to reduce the symmetry constraint in later layers, which is what is done effectively by CNNs in practice by downsampling with stride.  
- Experimental results on the learned subgroup weights (Fig. 2) reveal interesting trends showing that the networks do select for higher equivariance in early layers and lower equivariance in later layers which matches what has found to be effective in practice for equivariant networks. 
 

## Weaknesses / Questions 
- I am not sure of the framing of the method.  I am not an expert on NAS and am open to correction, but it seems to me both proposed methods find a trained network, not an architecture, since they are training the network weights and selecting an architecture at the same time.  The final architecture may not have great performance if trained from scratch.  In this sense, it seems the method is more a special training procedure than NAS method.   EquiNAS_D, in particular, is simply an architecture in which some weights are optimized alternatingly. 
- Given the evolutionary algorithm starts with a single network, how do we know to what extent its evolution is guided by architecture versus the specific weights in the mutated networks?  In experiments, I think both population level variance and initialization variance should be accounted for.  
- One of the biggest issues is the fact that the relaxation morphism increases the number of parameters making it difficult to distinguish performance increases are from relaxing constrains or increasing parameter count.  The authors do discuss this issue and suggest their optimization strategy over both parameter count and accuracy helps.  It would be better, however, if the number of parameters could be preserved by the relaxation. 
- The experimental results are not completely convincing of when this method would add significant value over simply doing hyperparameter search over equivariant networks.  In Table 1, an equivariant baseline is best on rotMNIST and ISIC, and a non-rotationally equivariant network with equivariant initialization is best for Galaxy10.  In table 2, EquiNAS_D does do best for rotMNIST, but the score is much lower than the best values in E(2)-CNN after full tuning of the best equivariant method.  In Galaxy10, EquiNAS_D does outperform, but with very high variance.  
- I have some issues with the clarity of the description of 4.2.  The condition that all groups in $[G]$ be subgroups or supergroups of all other groups in $[G]$ does not appear to be met by the expermental example since $D_2$ is not a subgroup or supergroup of $C_4$.  It's also not clear to me why this condition is necessary.  It is a fairly strict condition, although I think it would include useful cases.  I am also a bit unclear on the form of the input signal $f$.  In order to be convolved with $\tilde{\Psi}$ and have $G$-equivariance for each $G$ in $[G]$, $f$ should be defined over a supergroup of all $G \in [G]$.  Is this the case?   I am correct that if $z_G$ is a one-hot vector on the group $G$, then the operation is $G$-equivariant? 

## Questions 
- If the evolution algorithm can only mutate in the direction of symmetry constrain relaxation, doesn't this bias the algorithm towards relaxing all the symmetry constraints? 
- Does this work have a relation to diffstride (https://arxiv.org/abs/2202.01653) in that stride is typically a hyperparameter that corresonds to a choice of the subgroup of the translation group and can be shown to differentiably optimized? 
- After equivariance is broken in a given layer, there isn't much point in principle to persevering it downstream.  Are mutations automatically applied to all following layers or did your method discover this?  If so, I think that is interesting enough to be considered a strength. 

## Minor 
- In Eqn 1, if you reverse the order of $f$ and $\Psi$ and drop the sum over c and indices $c,d$, the equation can be simplified using matrix multiplication. 
- 4.1, para 2, "neutral element" -> "identity element" 
- 4.1 may benefit from writing as a proposition
- 4.1, Para 3, Line 5,  $h \in G'$ -> $h' \in G'$ 
- Sec 5., Line 4, typo "optimizffe"

clarity,_quality,_novelty_and_reproducibility: The paper is fairly clear with some weak spots corresponding to my questions above.  It could use illustrations and examples to help explain the method.  The quality is okay.  The experiments could be more thorough but do provide a basic trial for the proposed method.  The method is tested on 3 different datasets. The method seems novel to me.  There is no code provided but it is promised.  The description seems clear enough to repeat similar experiments.  
summary_of_the_review: This method seems quite interesting to me due to the fact that there are datasets in which strict equivariance may not perform well due to being too strict of an inductive bias or because the constraints interfere with optimization.  Moreover, equivariant networks have a great deal of hyperparameters in the amount of symmetry they exhibit at various levels, so a more sophisticated approach than random searching is worthwhile.  The proposed idea for relaxing symmetry constraints is appealingly simple and reasonable.  However, the experimental results are not that convincing as the proposed method is often outperformed by equivariant networks or by relaxed networks with equivariant initialization.  In the case EquiNAS_D does significantly outperform on Galaxy10, it has very high variance.  Since the relaxation method also adds parameters, it is not clear when improved performance is due to parameter count vs. too strong symmetry constraints.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two methods for equivariance-aware neural architectural search (NAS). The motivation for this work stems from the fact that symmetries present in a data set might often be imperfect or non-explicitly known. The first method is based on the proposed equivariance relaxation morphism, a procedure that equivalently reparameterizes a group convolution layer to operate with equivariance constraints on a subgroup; an evolutionary NAS based on such morphism is later adopted. The other method, named $[G]$-mixed equivariant layer, linearly combines layers equivariant to different groups, whose coefficient is search through a differentiable NAS. Thoughtful experiments on three datasets are conducted to demonstrate the performance.
strength_and_weaknesses: **Strength**
1. The paper is well motivated. Most G-CNN literature assume that the symmetry group of the learning task is explicitly known and perfect. The paper instead uses a equivariance-aware NAS procedure to search for the optimal architecture for (potentially) unknown and imperfect group transformations.
2. The paper is mostly well-written.
3. The experiments are detailed and thorough.

**Weakness**
1. The equivariance relaxation morphism, which relies on reparameterizing the group convolution on a subgroup, seems to be limited to discrete group. Can this be extended to continuous groups?
2. Moreover, the relaxation morphism is also restrictive in the sense that the number of the unstructured channel (c_out) cannot be adjusted after changing to a subgroup. This inevitably increases the model size as group size decreases, which is indeed the case for EquiNAS_D.
3. The clarity of the paper in section 5.1 can benefit from more explanation on the technical terms such as population, pareto selection, pareto front, etc.
4. In MNIST_rot, the review would like to see what happens when $C_8$ and $D_8$ are included in the picture.
5. If I understand correctly, the reported EquiNAS_E result on Galaxy10 in Table 1 might be misleading. Figure 4(a) shows that only one NAS output in the end achieves significantly smaller test error compared to all the remaining results.
6. Are the models involved in EquiNAS_D significantly larger compared to the baseline? After all, a linear combination of layers equivariant to different groups is taken. Does that mean the model is slow to train and (especially) test?
clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written and well-organized. The idea is novel.
summary_of_the_review: I think the paper is interesting. I am willing to adjust my rating based on the authors' response to my questions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: We recommend accepting this paper to ICLR 2023. 
Initially there were 3 positive and 1 negative reviewers. The negative reviewer updated their score to a positive one during the discussions after considering rebuttal and revision. 
The paper's revision convinced the reviewers this work can be published in its current state. However, we do encourage the authors to consider/discuss remaining issues: (1) improvements factoring out number of parameters; and (2) practicality for real datasets. 
justification_for_why_not_higher_score: The method has some merit in terms of idea and some proof of concept. 
The practical benefit is still not fully convincible. 
Both merit and practical benefit do not grant higher score. 
justification_for_why_not_lower_score: All reviewers in concensus for the merit of the partial and mixture equivariance and argue for acceptance.
Decision: Accept: poster