Group Equivariant Conditional Neural Processes
Authors: Makoto Kawano
Wataru Kumagai
Akiyoshi Sannai
Yusuke Iwasawa
Yutaka Matsuo
Venue: ICLR 2021 Poster (2021)
Keywords: Neural Processes
Conditional Neural Processes
Stochastic Processes
Regression
Group Equivariance
Symmetry

Abstract: We present the group equivariant conditional neural process (EquivCNP), a meta-learning method with permutation invariance in a data set as in conventional conditional neural processes (CNPs), and it also has transformation equivariance in data space. Incorporating group equivariance, such as rotation and scaling equivariance, provides a way to consider the symmetry of real-world data. We give a decomposition theorem for permutation-invariant and group-equivariant maps, which leads us to construct EquivCNPs with an infinite-dimensional latent space to handle group symmetries. In this paper, we build architecture using Lie group convolutional layers for practical implementation. We show that EquivCNP with translation equivariance achieves comparable performance to conventional CNPs in a 1D regression task. Moreover, we demonstrate that incorporating an appropriate Lie group equivariance, EquivCNP is capable of zero-shot generalization for an image-completion task by selecting an appropriate Lie group equivariance.

Published as a conference paper at ICLR 2021
GROUP EQUIVARIANT CONDITIONAL NEURAL PRO- CESSES
Makoto Kawano The University of Tokyo
Wataru Kumagai
The University of Tokyo, RIKEN AIP Tokyo, Japan
Tokyo, Japan
jp
Akiyoshi Sannai
Yusuke Iwasawa & Yutaka Matsuo
RIKEN AIP
The University of Tokyo
Tokyo. Japan
Tokyo, Japan { wasawa, matsuo) @weblab..bbbb...............b..b wtw.ab..ttttot.aee.......
wwwppsiissanniiii..... sannai frmaiirrken... jp ak akiyoshi.com
ABSTRACT
We present the group equivariant conditional neural process (EquivCNP). meta- learning method with permutation invariance in data set in conventional conditional neural processes (CNPs), and also has transformation equivariance in data space. Incorporating group equivariance, such rotation and scaling equivariance, provides a way to consider the symmetry of real-world data. We give decomposition theorem for permutation-invarian which leads us to construct EquivCNPs with an infinite-dimensionn this paper, we build architecture using Lie group convolutional layers for practical implementation. We show that EquivCNP with translation equivariance achieves comparable performance conventional CNPs a ID regression task. Moreover, we demonstrate that incorporating an appropriate Lie group equivariance, EquivCNP is capable of zero-shot generalization for an image-completion task by selecting an appropriate Lie group equivariance.
and group-equivarian maps,
latent space
handle group symmetries.
INTRODUCTION
Data symmetry has played significant role in the deep neural networks. In particular, a convolutional neural network, which play an important part the recent achievements of deep neural networks, has translation equivariance that preserves the symmetry of the translation group. From the same point of view, many studies have aimed incorporate various group symmetries into neural networks, especially convolutional operation (Cohen et al. 2019 Defferrard et al. 2019 Finzi al.) 2020). As example applications, to solve the dynamics modeling problems, some works have introduced Hamiltonian dynamics (Cireydanus et al. 2019 Toth al. 2019 Zhong et al.) 2019). Similarly, Quessard et al. (2020) estimated the action of the group by assuming the symmetry in the latent space inferred by the neural network. Incorporating the data structure (symmetries) into the models as inductive bias, can reduce the model complexity and improve model generalization.
In terms of inductive bias, meta-learning, or learning learn, provides way select an inductive bias from data. Meta-learning use past experiences to adapt quickly to a new task T ~ p(T) sampled from some task distribution pTT). Especially in supervised meta-learning, a task is described as predicting a set of unlabeled data (target points) given a set of labeled data (context points). Various works have proposed the use of supervised meta-learning from different perspectives (Andrychowicr et al. 2016 Ravi & Larochelle 2016 Finn et al.) 2017 Snell et al.) 2017 Santoro et al. 2016 Rusu ct al. 2018 this study, we are interested in neural processes (NPs) (Garnelo al. 2018a b), which meta-learning models that have encoder-decodee architecture (Xu et al. 2019). The encoder is permutation- invarrint function on the context points that maps the contexts into a latent representation. The decoder is function that produces the conditional predictive distribution of targets given the latent representation. The objective of NPs is to learn the encoder and the decoder, so that the predictive model generalizes well to tasks by observing some points of the tasks. To achieve the

---

Published as a conference paper at ICLR 2021
objective, an NP is required to learn the shared information between the training tasks T, T ~ p(T): the data knowledge Lemke et al. (2013). Each task T is represented by one dataset, and multiple datasets provided for training NPs to tackle meta-task. For example. we consider meta-task that completing the pixels that are missing in a given image. Often, images are taken by the same condition in each dataset, respectively. While the datasets contain identical subjects of images (e.g.. cars or apples), the size and angle of the subjects in the image may be different; the datasets have group symmetry, such as scaling and rotation. Therefore, is expected that pre-constraining NPs to have group equivariance improves the performance of the NPs at those datasets.
In this paper, we investigate the group equivalence of NPs. Specifically, we try to answer the following two questions, (I) can NPs represent equivariant functions? (2) can we explicitly induce the group equivariance into NPs? order to answer the questions, we introduce a new family of NPs, EquivCNP, and show that EquivCNP is a permutation-invarian and group-equivariant function theoretically and empirically. Most relevant to EquivCNP, ConvCNP (Gordon et al.) 2019) shows that using general convolution operation leads to the translation equivariance theoretically and experimentally; however it does not consider incorporation of other groups. First, we introduce the decomposition theorem for permutation invarriant and group-equivariant maps. The theorem suggests that the encoder maps the context points into a latent variable, which is a functional representation, in order to preserve the data symmetry. Thereafter, we construct EquivCNP by following the theorem. In this study, we adopt LieConv (Finzi et al.] 2020) to construct EquiyCNP for practical implementation. We tackle ID synthetic regression task (Garnelo et al., 2018a b Kim al., 2019 Gordon et al. 2019) to show that EquivCNP with translation equivariance is comparable to conventional NPs. Furthermore. we design a 2D image completion task investigate the potential of EquivCNP with several group equivariances. As result, we demonstrate that EquivCNP enables zero-shot generalization by incorporating not translation, but scaling equivariance.
RELATED WORK
2.1 NEURAL NETWORKS WITH GROUP EQUIVARIANCE
Our works build upon the recent advances in group equivariant convolutional operation incorporated into deep neural networks. The first approach is group convolution introduced in (Cohen & Welling 2016), where standard convolutional kernels are used and their transformation or the output transfor- mation is performed with respect the group. This group convolution induces exact equivariance, but only to the action of discrete groups. In contrast, for exact cquivariance to continuous groups, some works employ harmonic analysis so parameterize convolutional kernels in the basis (Weiler & Cesa 2019). Although this approach can be applied to any type of general data (Anderson et al.) 2019 Weiler Cesal 2019). it is limited to local application to compact, unimodular groups To address these issues, LieConv (Finzi et al., 2020) and other works (Huang et al. 2017 [Bekkers| 2019) use Lie groups. Our EquivCNP chooses LieConv to manage group equivariance for simplicity of the implementation.
find the basis of equivariant functions, and then
There are several works that study deep neural networks using data symmetry. In some works, in order to solve machine learning problems such as sequence prediction reinforeement learning, neural networks attempt to learn a data symmetry of physical systems from noisy observations directly (Greydanus al. 2019 Toth al. 2019 Zhong al. 2019 Sanchez-Gonzales et al.) 2019). While both these studies and EquivCNP can handle data symmetries, EquivCNP is not limited to specific domains such as physics.
Furthermore, Quessard et al. (2020) let the latent space into which neural networks map data, have group equivariance, and estimated the parameters of data symmetries. In terms of using group cquivariance in the latent space, EquivCNP is similar to this study but differs from being able to use various group equivariance.
2.2 FAMILY OF NEURAL PROCESSES
NPs (Garnelo et al.; 2018 b) are deep generative models for regression functions that map input I; E Th into an output yi E Rdd, In particular, given an arbitrary number of observed data points (3c, == (( 9z) g NPs model the conditional distribution of the target value yr at some new,
2

---

Published as a conference paper at ICLR 2021
unobserved target data point IT, where (Ir, yr) (az) 3, 3111 Fundamentally. there two NP variants: deterministic and probabilistic. Deterministic NPs (Garnelo et al. 2018a), known as conditional NPs (CNPs), model the conditional distribution
plur (rr, #0, JC) == p(urler,rc),
where r represents function that maps data sets (æc, MC)) into mnite-dimensiona vector space a permutation-invarii way and rc == r(ac, yc E IR is the feature vector. The function r can be implemented by DeepSets (Zaheer et al. 2017). The likelihood p(yr|Er,rC is modeled by Gaussian distribution factorized across the targets (ay, My) with mean and variance of prediction {(z,, Mj 8;)) by passing inputs ra and 3y through the MLP. The CNP is trained by maximizing the likelihood.
Probabilisticc NPs include a latent variable z. The NP infers ((2/rc) given an input re using the reparametrization trick (Kingma & Welling 2013) and models such conditional distribution as:
re, Myrler,re FC, vc) = p(yr| er, rc, rc, =){(=/r))d=
and it is trained by maximizing an ELBO: C(o, 0) = pog
K LLagosla yr) (2o
NPs have various useful properties: i) Scalability: the computational cost of NPs scales as On m) with respect to n contexts and m targets of data, ii) Flexibility: NPs can define a conditional distribution of an arbitrary number of target points, conditioning an arbitrary number of observations, iii) Permutation invariance: the encoder of NPs uses Deepsets (/aheer et al. (017) to make the target prediction permutation invariant. Thanks these properties, Galashov et al. (2019) replace Gaussian processes in Bayesian optimization, contextual multi-armed bandit, and Sim2Real tasks.
While there are many NP variants (Kim al. 2019 Louizos al.. 2019: Xu et al. 2019) to improve the performance of NPs, those do not take group equivariance into account yet. The most similar to EquivCNP, ConvCNP (Gordon al. 2019) incorporated only translation equivariance. In contrast, EquivCNP can incorporate not only translation but also other groups such as rotation and scaling.
3 DECOMPOSITION THEOREM
In this section, we consider group convolution. We first prepare some definition and teminology, Let X' and y C IR be the input space and output space, respectively. We define 2M = (X' x ) as collection of M input-output pairs, Z<M U Z, as the collection of at most M pairs, and 2 U 2m as the collection of finitely many pairs. Let [n] = (1,...,77) n) for n N, and let S, be the permutation group on [n). The action of Sn on 2, defined as
π ZZn = ((0=~111)
111; Vn (1)). (x+ (I+-+(n)) ((n) Vn Mm)+(m)));
where π E Sn and Zn E Zn We define the multiplicity of 2n, = ((z1, V ....., (zn, yn)) E Z, by mult (,,) == sup {lii E [m] a, = +]1 c = 31 ,,,..., .nn} and the multiplicity of Z' C Z by mult( (Z)) := supz,ez mult (z.) Then, a collection Z' C Z is said to have multiplicity K if mult(Z') K.
Mathematically, symmetry is described in terms of group action. The following group equivariant maps represent to preserve the symmetry in data.
Definition (Group Equivariance and Invariance). Suppose that a group G acts on sets and S!. Then, a map & S + S' is called G-equivariant when 4gg 3) g ((s) holds for arbitrary g E G and & E S. In particular, when acts on S' trivially (i.e., g o = s' for E G and s' E S'), the G-equivariant map is said be G-invariant: Ø(g .)) (ss).
Then, we can derive the following theorem, which decompose a permutation-invariant and group equivariant function into two tractable functions. Note that this theorem has been proved by Gordon et al. 2019) when G is a translation group.
Theorem 2 (Decomposition Theorem). Let G be group. Let Z'SM S (X x y)&m be topologically invariant and G-invariant with multiplicity K. For function &b : B AM Cr(K,)) )), the following conditions are equivalent:
closed, sermutation-invariann
3

---

Published as a conference paper at ICLR 2021
Functional Representation
Lifting
Group Conv
Projection
Encoding
Discretization
Input Context (XC, (C)
Output
Figure 1: Overview of EquivCNP.
Algorithm Prediction of Group Equivariant Conditional Neural Process
Input: p =LLiCoonv, RBF kernel th context {æ,, y/ target {
lower, upper <- (a, ) U (z)))
(tx)T=1 uniform lower, up
Encoding the context information into representation h(ie. Encoder)
h + Ex= (x+10)))( t.), h.
(13, 2j) LLeeonnNee(h))@an // Decoder
Output: ((1,, E
(I) & is continuous, permutaaionn-nvvriaan and G-equivariant.
(II) There exist function space H and continuous G-equivariant functio p H Co(X))) and continuous G-invariant interpolating kernel ub X22 - R such that
((2) = Σ xx+t (3:) Vas
where ΦK+1 : R.K. K+1 is defined by ΦK+1 [1, TT
Thanks to the Theorem 2 we can construct the sermutation-invarian and group-equivariant NPs whose form of encoder and decoder determined. In this paper, we call & EquivDeepSet.
4 GROUP EQUIVARIANT CONDITIONAL NEURAL PROCESSES
In this section, we represent EquivCNP that is -nviatatiiooniviiiaan and group-equivariant map. EquivCNP models the same conditional distribution well CNPs:
N
p(YT|XT, Dc) = II p (yn Po (xr.)
n=
II N (mni Mny En) with (pm, Ew) (b(Dc) (xx)) n=
where A denotes the density function of a normal distribution, De = (Xc, Yc) the observed context data and is a Squiv.eepSet. The important components of EquivCNP to be determined are p, d, and vb. The algorithm is represented Algorithm
(Ix, We) | is
To describe in more detail, first, Section 4. introduce the definition of group convolution, and then Section 4.2 explains LicConv (Finzi et al. 2020) used for EquivCNP to implement group convolution. Finally, we describe the architecture of proposed EquivCNP in Section
4.1 GROUP CONVOLUTION
When is a homogenous space of a group G, the lift of E is the element of group G that transfers fixed origin Lift(x) (u E G: 2).. That is, each pair of coordinates and
4

---

Published as a conference paper at ICLR 2021
features is lifted into K elements {(,,, fi) (uugk fi) f When the group action is transitive, the space on which it acts on is a homogenous space. More generally, however, the action is not transitive, and the total space contains an infinite number of orbits. Consider a quotient space Q = X/G, which consists of orbits of G' in X. Then each element E Q is a homogenous space of G. Because many equivariant maps use this information, the total space should be G x X //G not G.
Hence, E X is lifted to the pair (u, q), where G and q E Q.
Group convolution is a generalization of convolution by translation, which is used in images, etc., to other groups.
Definition (Group Convolution (Kondor & Trivedi 2018 Cohen et al. 2019).. g. G Q R be functions, and let ju.)) be Haar measure G. For any E G. the convolution of f by is defined as
gl00olllue h(u, 9) Scno - (. (e..feee)dddpppe dp)(tddd
By the definition, verify that the group convolution is G-equivariant. Moreover, Cohen et al.) (2019) recently showed that a G-equivariant linear map is represented by group convolution when the action of group is transitive.
4.2 LOCAL GROUP CONVOLUTION
In this study, we used LieConv as a group convolution (Finzi et al. 2020). LieCony a convolution that can handle Lie groups in group convolutions. LieConv acts on a pair (zi, fa), of coordinates I; X' and values fi E V in vector space V. First, input data 2; transformed (lifted) into group elements and orbits Next, we define the convolution range based on the invariant (pseudo) distance in the group, and convolve using a kernel parameterized by a neural network.
What is important for inductive bias and computational efficiency in convolution is that the range of convolutions local; that is, if the distance between u, and uj is larger than r. Io (1y Uy) 0. First, we define distance in the Lie group deal with locality in the matrix groug
d(u,, v) ||
where log denotes the matrix logarithm, and F' denotes the Frobenius norm. Because d( wu, wv) d(u, v) holds, this function is left-invariant and is pseudo-distance
log(u log(u-"wr (- ww
To further account for orbit q, we extend the distance to d((u,, qi), (1j, q;))² = d(u;, u²)² ado(q. 9;) where do(9; qj) == inf dx(x,, 22,) and dx is the distance on X. It is not necessarily invariant to the transformation g-
Based on this distance, the neighborhood nbhd(u) = {u, q gld((u;, q:), (vi, (j)) < r). The radius r should be adjusted appropriately from the ratio of the range of convolutions the total input, because the appropriate value is difficult to determine depending on the group treated. Therefore, the Lie group convolution is
h(u,
go
V.
Radius r of the neighborhood corresponds to the inverse of the density channel (0) in Gordon et al. (2019).
Discrete Approximation. Given a lifted input data point {(u,, 93 9 and a function value fy f(; qz) at each point, we need to select target (w, g)) } to convolve so that we can approximate the integral of the equation. Because the convolutional range is limited by nbhd(u). LieConv can approximate the integrals by the Monte Carlo method
hu,, q) = (g+f(u,
1
Σ vy (ubbdd(uu))))
,00 " (,))( d))
n
1 "K is hyperparameter and we randomly pick K clements {us 1 in the orbit corresponding to zi. 2 We assume that we have finite-dimensional representation. TThis is because the triangle incquality is not satisfied.
5

---

Published as a conference paper at ICLR 2021
The classical convolutional filter kernel g(-) is only valid for discrete values and is not available for continuous group elements. Therefore, pointconv/Liecom uses a multilayered neural network 9o as a convolutional kemel. However, because neural networks and input G is not a vector space, we let go be map in the Lie algebra g. Therefore, we use Lie groups and logarithmic maps exist in each element of the group. That is, let go(u) (g exp)o (log u), and parameterize ğe (g a exp)s by MLP. We use go: g + Ro Ketm Therefore, the convolution of the equation is
good at computation in Euclidean space,
h2
Σ
Te (log (uj 'u) 4s. (4)4;) f3.
T4i
jE abhd (4)
Here, the input to the MLP is d.j Concat (log(e, "u), qs. 9;))))).
4.3
IMPLEMENTATION
First, we explain the form of c. Because most real-world data have a single output per one input location, we treat the multiplicity of De as one, K 1, and define p(y) [1 y based on (Zaheer et al. 2017). The first dimension of output Φ; indicates whether the data located at is observed, so that the model can distinguish between the observed data, and the unobserved data whose value is zero (3/- 0).
Then, we describe the form of vb. Following our Theorem 2 is required to be stationary, non- negative, and positive-definite kernel. For EquivCNP, we change / depending on whether the input data is continuous or discrete. With continuous input data (e.g. ID regression), we use RBF kernels for vb An RBF kernel has learnable bandwidth parameter and scale parameter and is optimized with EquivCNP. functional representation E(Z) is made up by multiplying the kernel & with ø. On the other hand, when the inputs discrete (e.g. images), we not an RBF kernel but LieConv.
Finally, we explain the form of p. With our Theorem because p needs to be a continuous group equivariant map between function spaces, use LieConv for p. In this study, under the hypothesis of separability (Kaiser et al. 2017), we implemented separable LieConv in the spatial and channel directions, to improve the efficiency of computational processing. The details are given in the Appendix B. EquivCNP requires to compute the convolution of E(Z). However, since E E(Z) itself is functional representation, it cannot be computed in computers discretize E(Z) over the range of context and target points. We space the lattice points (t, (t)) S X on a uniform grid over a hypercube covering both the context and target points. Because the conventional convolution that is used in ConvCNP requires discrete lattice input space to operate on and produces discrete outputs, we need to back the outputs to continuous functions + V. While ConyCNP regards the outputs as weights for evenly-spaced basis functions (i.e., RBF kernel), LieConv does not require the input location to be lattice and can produce continuous functions output directly. Note that the algorithm of EquivCNP can be the same as ConvCNP: it can also use evenly-spaced basis functions. The obtained functions are used to output the Gaussian predictive mean and the variance at the given target points. We can evaluate EquivCNP by log-likelihood using the mean and variance.
is. To address this issue, we
5 EXPERIMENT
To investigate the potential of EquivCNP, we constructed three questions: 1) Is EquivCNP comparable conventional NPs such as ConyCNP? and Can EquivCNP have group equivariance in addition to translation equivariance and 3) does it preserve the symmetries? To compare fairly with ConvCNP, the architecture of EquivCNP follows that of ConyCNP: details are given the Appendix
5.1 1D SYNTHETIC REGRESSION TASK
To answer the first question, we tackle the ID synthetic regression task has been done in other papers (Garnelo et al. 2018a by Kim et al. 2019). At each iteration, a function f is sampled from given function distribution, then, some of the context De and target Dr points function f. this experiment, we selected the Gaussian process with RBF kernel, Matern 3 and periodic kernel for the function distribution. We chose translation equivariance T(1) incorporate
sampled from

---

Published as a conference paper at ICLR 2021
Table 1: Log-likelihood of synthetic I-dimensional regression
Model
RBF
Matern
Periodic
Oracle GP
3.9335 + 0.5512
3.7676 = 0.3542
1.2194 = 5.6685
CNP Garnelo et al. 2018a
-1.7468 + 1.5415
-1.7808 = 1.3124
-1.0034 = 0.5174
CNn (Gordon et al. 2019)
1.3271 + 1.0324
0.8189 + 0.9366
-0.4787 + 0.5448
EquivCNP (ours)
1.2930 + 1.0113
0.6616 + 0.6728
-0.4037 + 0.4968
Figure 2: Predictive mean and variance of ConyCNP and EquivCNP. The first columns show the prediction of the models trained on the RBF kernel and the last two columns show the prediction of the model trained the Matern kernel. The target function and sampled data points the same between the top row and bottom row except for the context. At the top row, the context is within the vertical dash line that is sampled from the same range during the training (black circle). In the bottom row, the new context located out of the training range (white circle) is appended.
into EquivCNP. We compared EquivCNP with GP (as an oracle), with CNP (Gamelo al. 2018a as a baseline, and with ConvCNP.
Table shows the likeliood means and standard deviations of 1000 tasks. In this task, both contexts and targets are sampled from the range 2- 2]. From Table 1 we can see that EquivCNP with translation equivariance is comparable to ConvCNP throughout all GP curve datasets. That is, EquivCNP has the model capacity to learn the functions as well as ConvCNP.
We also conducted the extrapolation regression proposed in (Gordon et al., 2019) as shown in Figure The first two columns show the models trained on an RBF kernel and the last two columns on a Matern- kernel. The top row shows the predictive distribution when the observation given within the same training region; the bottom row for the observation is not only the training region but also the extrapolation region: [-4, 4]. As a result, EquiveCNP can generalize to the observed data whose range is not included during training. This result expected because Gordon et al. (2019) has mentioned that translation equivariance enables the models to adapt to this setting.
5.2 2D IMAGE-CONE COMPLETION TASK
An image-completion task aims to investigate that EquivCNP can complete the images when is given an appropriate group equivariance. The image-completion task can be regarded regression task that predicts the value of y, at the 2D image coordinates z; given the observed pixels Dc ((En y/ ) n=1 (E R for the colored image input, and E for the grayscale image input). apply not only to the images but also to other real-world applications, such as predicting spatial data (Takeuchi et al. 2018).
The framework of the image completion
To evaluate the effect of EquivCNP with specific group equivariance, we introduce new dataset digital clock digits as shown in Figure 3 Since previous works use the MNIST dataset for image completion, also conduct the image completion task with rotated-MNIST. However, we cannot find a significant difference between the group equivariance models (the result of rotated-MNIS is depicted in Appendix E). We think that this happens because (1) original MNIST contains various
7

---

Published as a conference paper at ICLR 2021
Table 2: Log-likelihood of 2D image-completion task
Group
ligelikel
cn
((2)
1.0998 0.4115
S0(2)
-2.2275 = 6.8856
Rso x (2)))
1.8398 = 0.5368
Figure 3: The example of training data (top) and test data (bottom).
SE(()
1.1655 = 0.5420
x1.0
x0.75
x0.5
x0 x0.25
25%
75%
25%
75%
25%
75% %
%
75%
Context
T(2) (Com:CNP)
SO(2)
RxSO(2)
9
SF(2)
task results. The top row shows the given observation and the other rows show the mean of the conditional distribution predicted by EquivCNP with the specific group equivariance: 7(2). S2(,), Rzo x 50(2). and SE(2). Two of each column shows the same image, and the difference between two columns the percentage of context random sampling: 25% and 75%. When the size of digits is the same as that of the training set (i.e. not scaling but rotation equals (O()) symmetry), T(2) and (2) have good quality, but when the size digits smaller than that of training set, R=0 S2(2) has good performance.
Figure 4:
data symmeries including translation, scaling, and rotation, and (2) we cannot specify them precisely. Thus, we provide digital clock digits dataset anew.
In this experiment, we used four kinds of group equivariance; translation group T(2), the 2D rotation group S0(2), the translation and rotation group SE((), and the rotation-scabe group Rs0 x SO(2). The size of the images is 64 x 64 pixels, and the numbers are in the center with the same vertical length. For the test data, we transform the images by scaling within [0.15, 0.5) and rotating within [-900, +. Image completion with our digits data becomes an extrapolation task in that the test. data is never seen during training, though the number shapes are the same in both sets.
The log ligelihkeihod of image completion by EquivCNP with the group equivariance is reported in Table The mean and standard deviation of the log-likelihood is calculated over 1000 tasks (i.e. evaluating the digit transformed in 100 times respectively). As result, EquivCNP with R3o SO(2) performed better than other group equivarinace. On the other hand, the model with SO(2) had the worst performance. This might happen because the SO(2) not able to generalize EquivCNP to scaling. In fact, the log lig-iieeldood of (2)), which is the group equivariance combining translation T(2) and rotation S0()), not improved than that of (2)..
the qualitative result of image completion by EquivCNP with each group equivariance. We demonstrate that EquivCNP was able to predict digits smaller than the training digite While T(2) completes the images most clearly when the sizes of digits and the number of observations
*When the scaling 11.0. equals SO(2) symmetry.
8

---

Published as a conference paper at ICLR 2021
large, other groups also complete the images. The smaller the size of digits is compared to the
training digits, the worse the quality of T(2) completion becomes, and Ryo x SO(2) completes the digits more clearly. This is because the convolution region of (()) is invariant to the location, while that of Ryo x (2))) is adaptive to the location. As a result, for the images transformed by scaling, that EquivCNP with Ryo SO(2) preserved scaling group equivariance.
DISCUSSION
We presented new neural process, EquivCNP, that uses the group equivariant adopted from LieConv. Given a specific group equivariance, such as translation and rotation as inductive bias, EquivCNP has good performance at regression tasks. This is because the kernel size changes depending on the specific equivariance. Real-world applications, such as robot learning tasks (e.g. using hand-eye camera) will be left future work. We also hope EquivCNPs will help in learning group equivariance (Quessard et al. 2020) by data-driven approaches for future research.
REFERENCES
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in Neural Information Processing Systems, pp. 14510-14519, 2019.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989, 2016.
Erik Bekkers. B-spline enns on lie groups. arXiv preprint 11Xiv:1909.120555 2019.
Francois Chollet. Xception:
Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 2016
Taco S Cohen, Mario Geiger, and Maurice Weiler. general theory of equivariant cnns on ho- mogeneous spaces. In Advances in Neural Information Processing Systems, pp. 9142-9153, 2019.
Michaël Defferrard, Nathanaël Perraudin, Tomasz Kacprzak, and Raphael Sgier. Deepsphere: towards an equivariant graph-based spherical cnn. In ICLR Workshop on Representation Learming on Graphs and Manifolds, 2019. URL
httpp////wxxvv.or///oe/ abtp//////aaaaaaa/.eeeeeeeeee 904 05146
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volluee 70, pp. 1126-1135. JMLR. org, 2017
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint arXiv:2002. /2880, 2020.
Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Garnelo, David Saxton, Pushmeet Kohli, SM Eslami, and Yee Whye Teh. Meta-learning surrogate models for sequential decision making. arXiv preprint arXiv: 1907, 2019.
Marta Garnelo, Dan Rosenbaum, Chris Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo J Rezende, and SM Eslami. Conditional neural processes. arXiv preprint arXiv: 1807.0063 2018a.
Marta Garnelo, Jonathan Schwarz. Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1807 2018b.
Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv:1910.13553 2019.

---

Published as a conference paper at ICLR 2021
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pp. 15353-15363, 2019.
Zhiwu Huang, Chengde Wan, Thomas Probst, and Luc Van Gool. Deep learning on lie groups for skeleton-base based action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6099-61000 2017.
Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. Depthwise separable convolutions for neural machine translation. arXiv preprint arXiv: 17Xv::770600355, 2017.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.05661 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. arXiv:1312.611,,22 2013.
Auto-encoding variational bayes.
arXiv preprini
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks the action of compact groups. arXiv preprint urXiv: feo2.0669.. 2018
Christiane Lemke, Marcin Budka, and Bogdan Gabrys. Metalearning: a survey of trends and technologies. Artificial intelligence review, 44(1):1 2015.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process, In Advances in Neural Information Processing Systems, pp. 8743-8754, 2019.
Robin Quessard, Thomas D Barrett, and William R Clements. Learning group structure and disentan- gled representations of dynamical environments. arXiv preprint arXiv:2002.0699/ 2020.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In Proceedings of the 5th International Conference on Learning Representation, 2016.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprini arXiv: 1807.05960, 2018.
Alvaro Sanchez- Gonzalzz, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint urXiv: 12790, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentted neural networks. In International conference on machine learning, pp. 1842-1850, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In Advances in neural information processing systems, pp. 4077-4087, 2017.
Koh Takeuchi, Hisashi Kashima, and Naonori Ueda. Angle- convolution networks for extracting local spatial features. In Neurips Workshop Modeling and decision-making in the spatiotemporal domain, 2018
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. arXiv preprint arXiv: 1909. J 13789, 2019.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. In Advances Neural Information Processing Systems, pp. 14334-14344 14345, 2019.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointcony: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9621-9630 2019.
Jin Xu, Jean- Francois Ton, Hyunjik Kim, Adam R Kosiorek, and Yee Whye Tch. Metafun: Meta- learning with iterative functional updates. arXiv preprint 1912 2019.
10

---

Published as a conference paper at ICLR 2021
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp- 3391-3401,200
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint (eXXiv:900.12000 2019.

---

Published as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIAL
A. PROOF OF THEOREM
First, prove that (II) implies (I). We define the action of G on the set of univariate maps f X RR by
())) =- f1
11.111100 -x)).
and define the action of G the of bivariate maps // X²2 R by
(g (2r, x²) = (0 az x. x²).
Lemma 4. For map db : X2 TR and sample X, sample dependent function the X Td is defined by
vn vx(x') == (/
Here, ta G-invariant if and only if the map X 3 a ++ than Map(XX, R$) G-equivariant
The above lemma is derived as follows:
Vg-wr (z) (x 9 x') r0 1 x, x²) the (g - x))
(g (xx).
The left hand side represents the action the sample space and the right hand side does the action on the function space.
When denote the set of all G-equivariant maps from S S' by Equiv(S, the above lemma is represented as
Inv(2? RR) = Equiv(X, Map(XX, R*)).
Thus, since v/ x22 + R is invariant from (II), + v is equivariant. Then, for Z E Z'gM> the (x:+1(9))aa (y:), is also equivariant. Since p is equivariant from (II),
correspondence Z + Σ 0k+1 obtain (I) because the composition ofequivariant maps is equivariant.
notations and lemmas in the following. Let di be an interpolating continuous kernel that satisfies & (æ, x') 2 0. Then, for m E and Z (X' ") define
Next, we prove that (I) implies (II). We prepare
m
(Z'n)) =
Σ
Φk+ (Bi) vh (;,x)) x.): (a,, v)) S 2/h C k
where HK+1 x constructed from the RKHS H for which vi is reproducing kernel and endowed with the inner product (f.9) = _i= KK+ (Ji, gi)x where (::) is the inner product of the RKHS H. When the permutation group 5m acts on a set (X x D)mm, the set equivalence classes of this action denoted by (X' x 3S)"/Sm, Then, for element Z E (X x 3) the equivalent class of the action denoted by [2]. Similarly, for subset Z C (X x V) the set of equivalent classes is denoted by [2/m] : {[2] Z 2 Furthermore, we denote
is the (K +
Hilbert space
M
M
[Z = U [2
and HeM = U Hm (Z'))- m=1
m=
Lemma 1 and Lemma 3 in Gordon et al. (2019) provides the following lemma.
Lemma 5. For m E N, let 2/, S (X y)m be a set with multiplicity K and 1p an interpolating contimuous kernel. Then, (Hm (2),, continuous:
M M) are pairwise disjoint and the embedding E is injective and
E: [ - (Z E(ZZ)) Em ([Z)
[2] [Z/m]
where
Em [2/m] + Hmm (2,)) Em (1(01, 11) ,...... (xm, vm)ll = > 0k (y:) 2 (., (.,x,) x.)
12

---

Published as a conference paper at ICLR 2021
Similarly, Lemma 2 and Lemma 4 in Gordon al. 2019 provides the following lemma.
Lemma 6. Suppose that Zyy is topologically closed set in (X x y)M and cngrmutationnimvarian and that i6 satisfies (i) v) 2 0, (ii) vx x) o² > 0 for any æ, and (iii) (1 x') - as || + co. Let 1 o E .HeM + CV) continuous.
22mM] - Cx) be map such that every restriction P|[=(11 is continuous. Then,
When a G-equivariant function f is injective, f Imff is also G-equivariant on the image of f. Denoting @ E-1 by p. rewrite + po E.
B. SEPARABLE LIECONV
In this section, we introduce the separable LieConv that we design and implement for EquivCNP. LieConv (Finzi et al.) 2020) is based on PointConv (Wu et al. (2019), which is proposed for point cloud convolution. That is, the lifted inputs are convolved by PointConv. Therefore, we can adopt techniques that are used for general convolution. One of such techniques is separable convolution(hholles 2017). Separable convolution consists depthwise convolution and pointwise convolution (as known as x 1 convolution). The mathematical formulation of normal convolution, the pointwise convolution, and the depthwise convolution is as follow:
K.L.M
Conv(W
Σ k.l.m
W(k.m) M(itk.++Lm))
M
Dointisseoom(V.uJ// = Σ
Wm V(a.i))
m
K.L
DepthwiseConv(V.yyle Σ
W(R ((++J++)
SepConv (Wp: Wd, y) (f.j) PointwiseConn((.))) (Wps Depthwise Conv (i, (Wd, v))
Thanks to the assumption that convolution operation is separable to the spatial direction and the channel direction, the separable convolution provides the way to operate convolution more efficiently than general convolution. Note that the difference of the efficiency between LieConv and separable LieConv is slight; the difference between the matrix production and element-wise product. Following the equation above, design and implemented separable LieConv. Figure illustrates the processing of (a) normal LieConv and (b) separable LieConv. The memory consumption also different that the output shape of after the convolutional weights (kernel) is calculated in normal LieConv is B x NMG x Cmid while that of separable LieConv is x NMC x Cin.
C. EQUIVCNP ARCHITECTURE
The architecture of EquivCNP is following that of ConvCNP (Gordon et al. 2019). so that we can fairly compare them. It difficult determine radius r of LieConv because the radius varied substantially between the different groups due to the different distance functions. Instead, we parametrized the radius by specifying the average fraction of the total number of convolved elements that would fall into this radius. Therefore, we describe the value of the average fraction instead of kernel size that described other papers as usual. Simultaneously, while the conventional convolutional layer has parameter called stride that determines the target elements (pixels) to be convolved, LieConv has a parameter sampling fraction instead of stride to subsample the group elements; sampling fraction is 1.0.
C. ID SYNTHETIC REGRESSION TASK
For ID regression tasks, 4-layer LieConv architecture with ReLU activations. The average fraction of those LieConv is 5 [16, 32, 16, 8). Functional representation F ((Z) is concatenated with target point ær, followed by
32 and the number MC sampling is 25. The channels of LieConv
13

---

Published as a conference paper at ICLR 2021
LicConv
Lifted Coordinates
MLP
Coondy yr:
Nap (D, D
Noc X Con
ML.P
(C X Gral
1x Cu
Features
Noç x C.
(a) LieConv
LieConv
Coced,
Lifted Coordinates
MLP
Coond,
Nac (D)
MLP
XX C.al
1 CoB
Features
Nuc x C.
(b) Separable LieConv
Figure 5: Separable LieConv. Difference between (a) normal LieConv and (b) separable LieCony is the matrix product and elemente- wise product
lifting. After operating convolution to the lifted inputs, we use softplus activation following the last collecconeeeted layer (FC) as standard deviation. Note that the output of EquivCNP, mean and standard deviation, is sliced to get those of yr. The architecture of EquivCNP for ID regression task illustrated in Figure 6
y
ye
Me
TT
(LieConv ReL.U)
Sigmoid
FC
UT,
x
E(Z)
FC
PC
OT,r
x=
V/
(Tok,))
(T+ix
(Texx)
BxTx
(16 32 16
Figure 6: The architecture of EquivCNP for ID regression task. & represents dot product and + represents concatenation. vb is RBF kernel and [y* 1) ,....y
C.2 2D IMAGE-COMPLETIE TASK
For the 2D image-completion task, LieCony Conv, instead of RBF kernels v. The channels of this LieConv is 128, the average fraction is + and the number of MC sampling is 121. After the LieConv of v/, four residual blocks. Each block is composed by separable LieConv layers
14

---

Published as a conference paper at ICLR 2021
Supurable LieCoev
ReLUU
Sepurable LieConv
ReLU
Figure 7: Residual Block
and residual connections shown in Figure 7 The channel of each residual block is 128, the average fraction is 15. and the number of MC sampling is 81.
We employ the same procedure of ConvCNP (Gordon al. 2019) for imagelco as follows:
Given an input image E RC where C is color channel, // and represents height and width respectively, sample context points features O M. from bemoulli distribution. Mc means the density as same as we define during 1D regression task.
2. After lifting the imputs, apply LieConv to both I @ M, and M. get functional representa- tion: EZZ) Convo([M, @ M.]) E IR (128+ 128) HV
3. Then, functional representation E Z)) is passed through one FC followed by four residual R xxW
blocks: h ResBlocks(FC((((())
4. Finally, we use one FC to get mean and standard deviation channels and split the output
E 2 into those statistics.
D. EXPERIMENT DETAILS
In this section, we describe the experiments in more detail. Code and dataset are available on
D.1 ID SYNTHETIC REGRESSION TASK
The kernels used in Section 5.1 for generating the data via Gaussian Processes are defined as follows:
RBFKemel:
(- (I) - x2)²
k(01, exp
Matern-
k(x1, = (1 V5d +
5 {(P
5 (
exp
2d
with d= ||| x2l12
Periodic
k( x22 exp (-2 sin Iz 12)1)
To train all NPs, the GPs generate the context and target points; the number of context points and target points is random- random-sampled sampled uniformly from [3, 50) respectively. All NPs were trained for 200 epochs by 256 batches per epoch and the size of each batch 16, We used Adam optimizer (Kingma & Ba 2014) with learning rate 10-3, An architecture of CNP was based on the original code We visualize the result of periodic kernel regression at Figure X
We also demonstrate EquivCNP with the algorithm following that of ConvCNP (Gordon et al. 2019): regarding the output EquivCNP as weights for evenly-spaced basis functions (i.e. RBF kernel) in Figure The result of predictive distribution is much smoother than the result of our Algorithm ] though using RBF kernel is redundant.
15

---

Published as a conference paper at ICLR 2021
Figure 8: Predictive mean and variance of ConvCNP and EquivCNP at periodic kernels. First two columns show the result without outlier observation and last two columns show the result with outlier observation.
20
Figure Predictive mean and variance of EquivCNP that using algorithm proposed in (Gordon et al., 2019). Blue line and region represents EquivCNP and green line and region represents Gaussian Process. Each plot shows diffent sampled data. Although the algorithm is redundant compared with proposed Algorithm due to using RBF kernel to map the output of LieConv back continuous function, the result much smoother than Figure and
D.2 2D
TASK
The original image of the digital clock number is shown in Figure 10 We first inverted in colors of black and white of the image. Then, cropped the image so that each cropped image contains one digit and resize them to 64 x 64. Note that the vertical size of each number is set up to 56, while the horizontal size range.
fixed. The values of all pixels devided by 255 rescale them to the [0, I]
As mentioned in Section C.2, the context points sampled from bernoulli distribution. The parameter of bernoulli distribution, probability that the value is 1, is determined at rate of the number uniformly from 24 Buad
100 per Teotal The batch size is 4, epoch 100, and the optimizer
Ruw!
Adam (Kingma Ba 2014) whose learning rate is 5 10-4
E. ADDITIONAL COMPLETION TASK: MNIST
We also conduct the image completion task using rotated MNIST. It is thought that (1) original MNIST contains various data symmetries including translation, scaling, and rotation, and (2) cannot specify them precisely. Figure II shows the actual images from the original MNIST datasets. We can confirm that yet we did not conduct any transformation, the images have been already rotated.
16

---

Published as a conference paper at ICLR 2021
123 7799
6
Figure 10: The original data that is used for 2D image-completion task.
5
5 5
5
5
5
5
5
5
5
Figure 11: Actual images from original MNIST.
Moreover, factors other than symmetry such as personal habit exist. Indeed, the original MNIST is not good for verify the effectiveness of EquivCNP.
During this experiment, the batch size is 16, epoch is 30, and the optimizer is Adam whose leraning rate is 5 x 10 As result, the model misses the completion when the number of context points is quite a few. On the other hand, when the number context points is sufficient, the completion results seem well except the ((())-equivarina) model.
The result is depicted in Figure 12
17

---

Published as a conference paper at ICLR 2021
4
2
2
1
9
9
8
6
p
300
0
0
0
2
22 2
3
4
3
3
4
4
(a) T()
(6) S0
5
5
b 1
b
4
188
8
2
)
9
(c) R x S0)
(d) SE(2)
Figure 12: Image-completion task results using rotated-MNIS, In each image, the 1st and 4th columns show context pixels, the 2nd and 5th columns show ground truth images, and the 3rd and 6th columns show completion results. As a result, the model misses the completion when the number of context points is quite a few. On the other hand, when the number of context points is sufficient, the completion results seem well except the 50())) model.
18


Reviews:
[{"review_id": "n3ImXnzzc5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses an important symmetry in meta-learning.  Namely, the context data consists of a set of datapoints in arbitrary order.  The model should thus be permutation equivariant to their order.  At the same time, the data itself may have its own symmetries, e.g. rotation, which the network should likewise be equivariant to.  The authors follow a theory-driven approach, proving in Thm 2 that a function with these two types of symmetries may be factored and represented by a composition of functions reflecting each symmetry individually.  They then design a Neural Process (NP) model, EquivCNP, which reflects this result.  Other works have used permutation equivariance and translation equivariance in NPs, but this is the first to incorporate other symmetry groups. \n\nAs a weakness, the method description in Section 4 is often imprecise and unclear as noted in the specific points below.   The method is somewhat novel; it may be described as a combination of two existing methods: permutation-equivariant NPs (Garnelo 18) and LieConv (Finzi 20).  The experiments achieve good results, but could be much more convincing.  They concern fitting only functions on low-dimensional space, report only one metric, and have limited comparison to baselines.  In particular, it does not seem necessary to frame image reconstruction as a meta-learning task.  However, the strength of this paper is the theory-guided model design.  The theorem proved is wholly appropriate to the task and informs the model design very nicely.  In particular, it makes use of the realization that $D_C \\to C(X,Y)$ can be $G$-equivariant; that is, the symmetry transformation can be inferred from context and transferred onto the prediction function.  Thus I suggest acceptance (6).   \n\n**Specific Point and Questions** \n- Sec 2.1 “Weiler & Cesa ... is costly” I do not agree.  However cost is defined this method is on par or better than CNNs \n- Sec 2.1 “learning the objective from the data.” I do not follow.  Can you explain further how this applies to EquivCNP?\n- Sec 2.1 How do the symmetries in this paper compare to those considered in (Maron et al 2020, On Learning Sets of Symmetric Elements) which also considers permutation and Lie group equivariance together?\n- Defn 3, It seems here we are giving all the orbits the same weight? Should there be a measure on orbits?  Considering the $SO(2)$ action on the plane, should not the circular orbits be weighted by circumference?\n- Sec 4.2 “LieConv … can handle Lie groups” This is true, but many other architectures such as steerable CNN can as well. \n- Sec 4.2 “we define distance in the Lie group.” Since we have lifted from the space $X$, is it important to consider differences between measuring distance on $X$ vs. $G$?\n- Sec 4.2 $\\| q_i - q_j \\|$, how do you define distance between orbits?\n- Sec 4.2 Since compact groups have a bi-invariant metric, why not use that instead of $d(u,v)$?\n- Page 5, first Eqn, According to which distribution are $(u_i,q_i)$ sampled? \n- Page 6, para 1, it is not clear exactly what precisely is meant by discrete or continuous data.\n- Page 6 “although we want to convolve $E(Z)$...” I do not follow this sentence.  Can you clarify it?\n- Page 7 “MNIST already contains diverse group equivariance” If this is the case then the equivariant model should still have good performance, but perhaps with less improvement over the baseline, correct?\n- Page 8, Why not test SO(2)-equivariant model on test data which has only been rotated? Does not the success of $R \\times SO(2)$ over the other groups simply show that the others have misspecified (or perhaps underspecified) inductive biases? \n- Page 11, second equation, should be $\\psi(g^{-1} x, g^{-1} x')$ since it is a left-action.  This will not change the rest of the proof.\n- Page 11, proof of (II) implies (I) feels incomplete.  Can you define $\\rho, \\mathcal{H}$ here? \n\n**Minor Points**\n- Sec 3, first equation, $\\pi Z_n := \\ldots$, the subscripts should be $\\pi^{-1}(i)$, not $\\pi(i)$. One can see this in a simple example or from the fact $\\pi$ is a left-action. \n- Sec 3, by $mult(Z’):=$ is sup over $Z_m$ but contains $Z_n$.\n- Sec 4.1, “lifting into $K$ elements” How do we know the stabilizer is finite?\n- Sec 4.2, note that not all Lie groups are matrix groups as seems to be implied here. \n- Sec 4.2, Last Eqn on Page 4, the integral can be written over G since the support of $g_\\theta$ limits to the neighborhood of $u$\n- Bottom Page 4, footnote, triangle, not trigonometric \n- Page 5, first Eqn, sum should be over $(v_j, q_j’)$.  \n- Since the support of $g_\\theta$ is limited to a neighborhood of 1, it may not be necessary to assume $exp$ is surjective since $exp$ is always surjective in a small neighborhood of 1.\n- Sec 4.3, first Eqn, the first line is a product of density functions, but the second is a product of distributions.  This is unclear.  \n- Sec 4.3 EquivDeepSet is not introduced\n- Sec 4.3 “a single output per one input” Is this backwards?\n- Page 6, line 3, $\\phi$ with $\\psi$\n- Sec 5 “two questions” should “three questions”\n- Page 11, Proof switches between $X$ and $S$, make consistent\n- Page 11, What do the brackets mean? This notation should be defined. \n\n**Update Based on Author Feedback**\nI am grateful to the authors for their detailed replies to my questions. I understand some of the minor points better and am happy they have revised some unclear parts. Overall, I think this paper has some real strengths: theory-guided design, important problem, novel methods. I do feel (similar to R3) that the experiments and applications could have been far more convincing. Weighing these strengths and weakness, I still tilt slightly towards accept (6).", "extra_scores": {}, "extra_text": {}}, {"review_id": "fEezdmXBQlF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### 1. Summary\nThis paper presents EquivCNP which extends Conditional Neural Processes (CNP) to incorporate symmetries of the data, e.g. rotation and scaling. The approach utilizes a combination of LieConv (Finzi et al., 2020)  and DeepSet (Zaheer et al., 2017) to achieve the equivariance in the data space and permutation invariance across the samples in a dataset. They provide empirical results on a 1D regression task with synthetic and 2D image completion tasks using digital clock digits dataset which they constructed.\n\n### 2. Decision\nI am recommending a weak reject for this paper. First, the experimental section does not sufficiently demonstrate the efficacy of the proposed approach. Second, I find the organization of the paper hard to follow at times.\n### 3. Supporting arguments\n1. The paper’s motivation is interesting and intuitive. Just as ConvCNP (Gordon et al., 2019) extended CNP with translation equivariance, this work proposes to extend CNP with group equivariance. It is generally difficult to build equivariance properties into deep-nets just in the data space alone, CNP requires the additional permutation invariance. Hence, I think the work is of interest to the community.\n\n2. The main weakness of this paper is the experimental section. For example, ConvCNP (Gordon et al., 2019) from last year’s ICLR evaluated on the following datasets: a 1D synthetic dataset, PLastiCC dataset, MNIST, SVHN, CelebA32, CelebA64, ZSMM. On the other hand, this work conducted experiments on a 1D synthetic time series data, and a “digital clock dataset” where the template is shared across train and test sets.\n\n3. The author explained MNIST is “not suitable for evaluating EquivCNP with group equivariance: translation, scaling, and rotation”. I am not convinced by this argument; I think the paper can be strengthened if there is an experiment demonstrating that on a real dataset EquivCNP is beneficial.\n\n4. Furthermore, the image completion results do not compare to any prior benchmark. I don’t see a reason for not comparing.\nI think the organization makes the paper difficult to read at times. For example, on page 5, they mentioned $\\rho,  \\phi, \\psi$, which are defined back on page 2. Maybe re-organize such that the modeled conditional distribution, in Sec 4.3,  is at the top of Sec 4; similar to Gordon et al., (2019)’s paper organization.\n\n### 4. Additional feedbacks\n- Adding equation numbers and referring to them in the text may help the readers to understand the paper better.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dFiSYxRQnYA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a member of the neural process family. To empower the Conditional NP with group equivariance ability, the idea is to replace the original encoder part of Conditional NP with a group convolution-based encoder, which is similar to ConvCNP (Gordon et al. 2019). I am not familiar with the group convolution part but the neural process. It looks LieConv is used as the main group convolution tool but such tool was proposed in another paper (Finzi et al, 2020). The authors did not highlight the contribution made based on LieConv. The improvement or change on CNP is not significant, but the new group equivariance ability is definitely a new and useful feature of NP. The experiment results verified the proposed idea. This paper is the first to introduce the group convolution to NP to my best knowledge. One minor suggestion is that Algorithm 1 is all about group convolution without any computation or inference of the NP part. Considering the title of the algorithm, it would be better to include some details of NP or change the title. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ys7DRctTwkX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides an extension of convolution conditional neural processes CNPs to more general Lie group equivariant CNPs. The development of the theory seems sufficiently clear to someone more familiar with the field. However, for newer readers, it seems important to be familiar with background concepts and prior work. This is not a penalizing point but rather just an observation.\n\n\nThe major concern with the body of work is the fact that the experiments seem lacking in application to more realistic data and scenarios. For instance, group invariance is not studied in the context of image classification which would be a direct application where transformations need to be accounted for. Though the image completion task seems interesting, more complex transformations are not handled which leaves this work somewhat incomplete. Indeed, the authors agree to this point in the discussion para.  How would this method scale to other more complex transformation groups such as illumination, object 3D pose etc? Although it might seem easy to brush off these concerns as future work, given that there has been a lot of work in these areas, research that can address these other more challenging yet real-world practical cases need to be addressed atleast in part in a bosy of work. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a natural combination of conditional neural processes with LieConv models. It is a good step forward for stochastic processes with equivariances. While there is still room to improve the experiments, the authors provided a good response to reviewers, and the paper is a nice contribution.
Decision: Accept (Poster)