Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses an important symmetry in meta-learning.  Namely, the context data consists of a set of datapoints in arbitrary order.  The model should thus be permutation equivariant to their order.  At the same time, the data itself may have its own symmetries, e.g. rotation, which the network should likewise be equivariant to.  The authors follow a theory-driven approach, proving in Thm 2 that a function with these two types of symmetries may be factored and represented by a composition of functions reflecting each symmetry individually.  They then design a Neural Process (NP) model, EquivCNP, which reflects this result.  Other works have used permutation equivariance and translation equivariance in NPs, but this is the first to incorporate other symmetry groups. 

As a weakness, the method description in Section 4 is often imprecise and unclear as noted in the specific points below.   The method is somewhat novel; it may be described as a combination of two existing methods: permutation-equivariant NPs (Garnelo 18) and LieConv (Finzi 20).  The experiments achieve good results, but could be much more convincing.  They concern fitting only functions on low-dimensional space, report only one metric, and have limited comparison to baselines.  In particular, it does not seem necessary to frame image reconstruction as a meta-learning task.  However, the strength of this paper is the theory-guided model design.  The theorem proved is wholly appropriate to the task and informs the model design very nicely.  In particular, it makes use of the realization that $D_C \to C(X,Y)$ can be $G$-equivariant; that is, the symmetry transformation can be inferred from context and transferred onto the prediction function.  Thus I suggest acceptance (6).   

**Specific Point and Questions** 
- Sec 2.1 “Weiler & Cesa ... is costly” I do not agree.  However cost is defined this method is on par or better than CNNs 
- Sec 2.1 “learning the objective from the data.” I do not follow.  Can you explain further how this applies to EquivCNP?
- Sec 2.1 How do the symmetries in this paper compare to those considered in (Maron et al 2020, On Learning Sets of Symmetric Elements) which also considers permutation and Lie group equivariance together?
- Defn 3, It seems here we are giving all the orbits the same weight? Should there be a measure on orbits?  Considering the $SO(2)$ action on the plane, should not the circular orbits be weighted by circumference?
- Sec 4.2 “LieConv … can handle Lie groups” This is true, but many other architectures such as steerable CNN can as well. 
- Sec 4.2 “we define distance in the Lie group.” Since we have lifted from the space $X$, is it important to consider differences between measuring distance on $X$ vs. $G$?
- Sec 4.2 $\| q_i - q_j \|$, how do you define distance between orbits?
- Sec 4.2 Since compact groups have a bi-invariant metric, why not use that instead of $d(u,v)$?
- Page 5, first Eqn, According to which distribution are $(u_i,q_i)$ sampled? 
- Page 6, para 1, it is not clear exactly what precisely is meant by discrete or continuous data.
- Page 6 “although we want to convolve $E(Z)$...” I do not follow this sentence.  Can you clarify it?
- Page 7 “MNIST already contains diverse group equivariance” If this is the case then the equivariant model should still have good performance, but perhaps with less improvement over the baseline, correct?
- Page 8, Why not test SO(2)-equivariant model on test data which has only been rotated? Does not the success of $R \times SO(2)$ over the other groups simply show that the others have misspecified (or perhaps underspecified) inductive biases? 
- Page 11, second equation, should be $\psi(g^{-1} x, g^{-1} x')$ since it is a left-action.  This will not change the rest of the proof.
- Page 11, proof of (II) implies (I) feels incomplete.  Can you define $\rho, \mathcal{H}$ here? 

**Minor Points**
- Sec 3, first equation, $\pi Z_n := \ldots$, the subscripts should be $\pi^{-1}(i)$, not $\pi(i)$. One can see this in a simple example or from the fact $\pi$ is a left-action. 
- Sec 3, by $mult(Z’):=$ is sup over $Z_m$ but contains $Z_n$.
- Sec 4.1, “lifting into $K$ elements” How do we know the stabilizer is finite?
- Sec 4.2, note that not all Lie groups are matrix groups as seems to be implied here. 
- Sec 4.2, Last Eqn on Page 4, the integral can be written over G since the support of $g_\theta$ limits to the neighborhood of $u$
- Bottom Page 4, footnote, triangle, not trigonometric 
- Page 5, first Eqn, sum should be over $(v_j, q_j’)$.  
- Since the support of $g_\theta$ is limited to a neighborhood of 1, it may not be necessary to assume $exp$ is surjective since $exp$ is always surjective in a small neighborhood of 1.
- Sec 4.3, first Eqn, the first line is a product of density functions, but the second is a product of distributions.  This is unclear.  
- Sec 4.3 EquivDeepSet is not introduced
- Sec 4.3 “a single output per one input” Is this backwards?
- Page 6, line 3, $\phi$ with $\psi$
- Sec 5 “two questions” should “three questions”
- Page 11, Proof switches between $X$ and $S$, make consistent
- Page 11, What do the brackets mean? This notation should be defined. 

**Update Based on Author Feedback**
I am grateful to the authors for their detailed replies to my questions. I understand some of the minor points better and am happy they have revised some unclear parts. Overall, I think this paper has some real strengths: theory-guided design, important problem, novel methods. I do feel (similar to R3) that the experiments and applications could have been far more convincing. Weighing these strengths and weakness, I still tilt slightly towards accept (6).
review_title: Good theory-guided design; okay experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### 1. Summary
This paper presents EquivCNP which extends Conditional Neural Processes (CNP) to incorporate symmetries of the data, e.g. rotation and scaling. The approach utilizes a combination of LieConv (Finzi et al., 2020)  and DeepSet (Zaheer et al., 2017) to achieve the equivariance in the data space and permutation invariance across the samples in a dataset. They provide empirical results on a 1D regression task with synthetic and 2D image completion tasks using digital clock digits dataset which they constructed.

### 2. Decision
I am recommending a weak reject for this paper. First, the experimental section does not sufficiently demonstrate the efficacy of the proposed approach. Second, I find the organization of the paper hard to follow at times.
### 3. Supporting arguments
1. The paper’s motivation is interesting and intuitive. Just as ConvCNP (Gordon et al., 2019) extended CNP with translation equivariance, this work proposes to extend CNP with group equivariance. It is generally difficult to build equivariance properties into deep-nets just in the data space alone, CNP requires the additional permutation invariance. Hence, I think the work is of interest to the community.

2. The main weakness of this paper is the experimental section. For example, ConvCNP (Gordon et al., 2019) from last year’s ICLR evaluated on the following datasets: a 1D synthetic dataset, PLastiCC dataset, MNIST, SVHN, CelebA32, CelebA64, ZSMM. On the other hand, this work conducted experiments on a 1D synthetic time series data, and a “digital clock dataset” where the template is shared across train and test sets.

3. The author explained MNIST is “not suitable for evaluating EquivCNP with group equivariance: translation, scaling, and rotation”. I am not convinced by this argument; I think the paper can be strengthened if there is an experiment demonstrating that on a real dataset EquivCNP is beneficial.

4. Furthermore, the image completion results do not compare to any prior benchmark. I don’t see a reason for not comparing.
I think the organization makes the paper difficult to read at times. For example, on page 5, they mentioned $\rho,  \phi, \psi$, which are defined back on page 2. Maybe re-organize such that the modeled conditional distribution, in Sec 4.3,  is at the top of Sec 4; similar to Gordon et al., (2019)’s paper organization.

### 4. Additional feedbacks
- Adding equation numbers and referring to them in the text may help the readers to understand the paper better.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a member of the neural process family. To empower the Conditional NP with group equivariance ability, the idea is to replace the original encoder part of Conditional NP with a group convolution-based encoder, which is similar to ConvCNP (Gordon et al. 2019). I am not familiar with the group convolution part but the neural process. It looks LieConv is used as the main group convolution tool but such tool was proposed in another paper (Finzi et al, 2020). The authors did not highlight the contribution made based on LieConv. The improvement or change on CNP is not significant, but the new group equivariance ability is definitely a new and useful feature of NP. The experiment results verified the proposed idea. This paper is the first to introduce the group convolution to NP to my best knowledge. One minor suggestion is that Algorithm 1 is all about group convolution without any computation or inference of the NP part. Considering the title of the algorithm, it would be better to include some details of NP or change the title. 
review_title: This paper proposes a member of the neural process family. The improvement or change on CNP is not significant, but the new group equivariance ability is definitely a new and useful feature of NP. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper provides an extension of convolution conditional neural processes CNPs to more general Lie group equivariant CNPs. The development of the theory seems sufficiently clear to someone more familiar with the field. However, for newer readers, it seems important to be familiar with background concepts and prior work. This is not a penalizing point but rather just an observation.


The major concern with the body of work is the fact that the experiments seem lacking in application to more realistic data and scenarios. For instance, group invariance is not studied in the context of image classification which would be a direct application where transformations need to be accounted for. Though the image completion task seems interesting, more complex transformations are not handled which leaves this work somewhat incomplete. Indeed, the authors agree to this point in the discussion para.  How would this method scale to other more complex transformation groups such as illumination, object 3D pose etc? Although it might seem easy to brush off these concerns as future work, given that there has been a lot of work in these areas, research that can address these other more challenging yet real-world practical cases need to be addressed atleast in part in a bosy of work. 
review_title: Extension of conditional NPs to Lie groups, however concerns remain regarding scalability to other transformations

Meta-review: This paper provides a natural combination of conditional neural processes with LieConv models. It is a good step forward for stochastic processes with equivariances. While there is still room to improve the experiments, the authors provided a good response to reviewers, and the paper is a nice contribution.
Decision: Accept (Poster)