$SE(3)$  Equivariant Convolution and Transformer in Ray Space
Authors: Yinshuang Xu
Jiahui Lei
Kostas Daniilidis
Venue: NeurIPS 2023 spotlight (2023)
Keywords: equivariance
light field
equivariant convolution over homogeneous space
TL;DR: We propose the $SE(3)$ equivariant convolution and transformer in $3D$ ray space.

Abstract: 3D reconstruction and novel view rendering can greatly benefit from geometric priors when the input views are not sufficient in terms of coverage and inter-view baselines. Deep learning of geometric priors from 2D images requires each image to be represented in a $2D$ canonical frame and the prior to be learned in a given or learned $3D$ canonical frame. In this paper, given only the relative poses of the cameras, we show how to learn priors from multiple views equivariant to coordinate frame transformations by proposing an $SE(3)$-equivariant convolution and transformer in the space of rays in 3D. We model the ray space as a homogeneous space of $SE(3)$ and introduce the $SE(3)$-equivariant convolution in ray space. Depending on the output domain of the convolution, we present convolution-based $SE(3)$-equivariant maps from ray space to ray space and to $\mathbb{R}^3$. Our mathematical framework allows us to go beyond convolution to $SE(3)$-equivariant attention in the ray space. We showcase how to tailor and adapt the equivariant convolution and transformer in the tasks of equivariant $3D$ reconstruction and equivariant neural rendering from multiple views. We demonstrate $SE(3)$-equivariance by obtaining robust results in roto-translated datasets without performing transformation augmentation.

Equivariant Light Field Convolution and Transformer in Ray Space
Yinshuang Xu University of Pennsylvania xuyindseas upenn. edu
Jiahui Lei University of Pennsylvania lei leijhoseas upenn. edu
Kostas Daniilidis University of Pennsylvania and Archimedes, Athena RC kostasâcis upenn. edu
Abstract
3D reconstruction and novel view rendering can greatly benefit from geometric priors when the input views baselines. Deep learning of geometric priors from 2D images requires each image to be represented in 2D canonical frame and the prior to be learned in a given or learned 3D canonical frame. In this paper, given only the relative poses of the cameras, show how learn priors from multiple views equivariant coordi- nate frame transformations by proposing an S SE(3 (e33)-qquivarian convolution and transformer in the space of rays in 3D. We model the ray space homogeneous space of SE(3) and introduce the (3)33)quivaaiii) convolution in ray space. Depending on the output domain of the convolution, we present convolution-base S E(3 (B(3)--qquvariant maps from ray space to ray space and to RRS Our mathematical attention
sufficient in terms of coverage and inter- view
framework allows to go beyond convolution to S (33)-equivarria) the ray space. We showcase how to tailor and adapt the equivariant convolution and transformer in the tasks of equivariant 3D reconstruction and equivariant neural rendering from multiple views. We demonstrate (E(3)- -equivariance by obtain- ing robust results in roto-translated datasets without performing transformation augmentation.
Introduction
Recent years have seen significant advances in learning- techniques 66 68 60
69
61
73
[1] [53] harnessing the power of deep learning for extraction of geometric priors from multiple images and associated ground-truth shapes. Such approaches extract features from each view and aggregate these features into geometric prior. However, these approaches are not -3euuqqquiaar to transformations of the frame where the priors and images are defined. While view pooling or can be used aggregate features and tackle equivariance, view pooling discards the rich geometric information contained in a multiple view setup.
calculating variance [69 72 46
In this paper, we address the problem of learning geometric priors that (E(3)-equivarri) with respect to transformations of the reference coordinate frame. We argue that all information needed for tasks like novel view rendering or 3D reconstruction contained the light field 6 37 Our input is light field, a function defined on oriented rays in 3D whose values can be the radiance or features extracted from pixel values. We will use the term light field, and we will be specific when is radiance field or feature field. Images are discrete samples of this field: the camera position determines which rays are sampled, while the camera orientation leaves the sample of the light field unchanged up to pixel discretization. We model the light field as field over a homogeneous space of SE(3), the ray space R parameterized by the Plücker coordinates. The ray space R is the space of oriented light rays, for any ray E R, the Plücker coordinate is x (d, m)) where d E @ is the direction of the ray, and
where æ is any point on the ray.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We define convolution in the continuous ray space as an equivariant convolution on homogeneous space [8] Since our features are not limited scalar values, we will draw upon the tools of tensor field networks and representation theory, discussed in detail in the Appendix. In Sec. 3.1 Te study the group action of S (3) on R, the stabilizer group for R. and how SF(3) transforms the feature field over R. In Sec. 3.4 we focus on developing the equivariant convolution in R, providing analytical solutions for the kernels with the derived constraints in convolution from R to R and from R to R", respectively, Meanwhile, we make the kernel locally supported without breaking the equivariance. By varying the output domain of the convolution, we introduce equivariant convolutions from the ray space to the ray space and from the ray space to the 3D Euclidean space.
The constraint of the kernel limits the expressiveness of equivariant convolution when used without a deep structure. In Sec 3.5 we introduce an eguivariant transformer in R. The equivariant transformer generates the equivariant key, query, and value by leveraging the kernel derived in the convolution, resulting, thus, in invariant attention weights and, hence, equivariant outputs. We provide detailed derivation of two cases of cross- aoss-attention:: the equivariant transformer from R to R and the equivariant transformer from R to R In the first case, the features that generate the key and value are attached to source rays, while the feature generating the query is attached to the target ray. In the second case, the feature generating the query is attached to the target point.
We demonstrate the composition of equivariant convolution and transformer modules in the tasks of 3D reconstruction from multi-views and novel view synthesis given the multi view features. The inputs consist of finite sampled radiance fields or finite feature fields, while our proposed equivariant convolution and transformer are designed for continuous light fields. If an object or scene undergoes a rigid transformation and resampled by the same multiple cameras, the FF) group action not transitive in the light field sample. This lack of transitivity can significantly impact the computation of equivariant features, mainly because the views are sparse, unlike densely sampled point clouds. Object motion introduces new content, resulting in previously non-existing rays in the light field sampling. Hence, our equivariance is an exact equivariance with respect to the choice of coordinate frame. In the 3D reconstruction task, we experimentally show that equívariance is effective for small camera motions or arbitrary object rotations and generally provides more expressive representations. In the 3D object reconstruction application, ray space to obtain the equivariant features attached to rays. We then apply equivariant convolution and equivariant transformer from R. to 3R3 to obtain equivariant features attached to the query point, which are used to calculate the signed distance function (SDF) values and ultimately reconstruct the object. In the generalized rendering task, our model queries target ray and obtains neighboring rays from source views. Our composition of equivariant modules is based on IBRNet [61], which consists of view feature aggregation and ray transformer. We replace the view feature aggregation in 61 with the equivariant convolution and transformer over rays and the ray transformer part with the equivariant transformerover the points along the ray to get the density and color of the point, see Sec.
first apply an equivariant convolutional network in
3.3 We summarize here our main contributions:
(1) We model the ray space as homogeneous space with SE(3) the acting group, and we propose the -Equiirauu generalized convolution as the fundamental operation on a light field whose values may be radiance or features. We derive two SE) -equiivrarrr convolutions, both taking input ray features and producing output ray features and point features, respectively.
(2) To enhance the feature expressiveness, we extend the equivariant convolution to equivariant transformer in R, in particular, a transformer from R to R and a transformer from R to Rd.
(3) We adapt and compose the equivariant convolution and transformer module for 3D reconstruc- tion from multiple views and generalized rendering from multi-view features. The experiments demonstrate the equivariance of our models.
2 Related Work
Equivariant Networks Group equivariant networks 15
65
55
63
provide deep learning pipelines that are equivariant by design with respect to group transformations of the input. While inputs like point clouds, 2D and 3D images, and spherical images have been studied extensively, our work is the first, as far as we know, to study equivariant convolution and cross-attention on light fields. The convolutional structure on homogeneous spaces or groups is sufficient and necessary for equivariance with respect to compact group actions as proved in [18 36 Recently, Cesa et al. (8) Xu et al. [70] provided uniform way to design the steerable kernel in

---

an equivariant convolutional neural network on a homogeneous space using Fourier analysis of the stabilizer group and the acting group, respectively, while Finzi et al. [26] proposed numerical algorithm to compute kernel by solving the linear equivariant map constraint. For arbitrary Lie groups, Finzi et al. [25]. MacDonald et al. [39]. Bekkers [4] designed the uniform group convolutional neural network. The fact that any (n) equivariant function can be expressed in terms of collection of scalars is shown in [59]. For general manifolds, Cohen et al. [16], Weiler et al. [64] derived the general steerable kernel from differential geometry perspective, where the group convolution on homogeneous space is a special case. The equivalent derivation for the light field is in the Appendix. Recently, equivariant transformers drew increasing attention, in particular for 3D point cloud analysis 10 7 A general equivariant self-attention mechanism for arbitrary groups 43 while equivariant transformer model for Lie groups was introduced in
and reconstruction [2 48
was proposed in
44
Hutchinson et al. 32 We are the first to propose an equivariant attention model in the 3D ray space.
Light Field and Neural Rendering from Multiple Views The plenoptic function introduced in perception [6] and later in graphics (37 brought a new light into the scene representation problem and was directly applicable to the rendering problem. Instead of reconstructing and then rendering. light fields enabled rendering just by sampling the right rays. Recently, learning-based light field reconstruction
became increasingly popular for novel view synthesis, while proposed non non-equivariant networks in the ray space. Due to the smaller dimension of the ray space, the networks in the ray space are more efficient compared to neural radiance fields 42 which leverages volumetric rendering. Several studies 73 61
3 66 51
50 [54 53
53 50
38
13
3
58 20
concentrate on generalizable rendering. These works are similar to ours in that they obtain the 3D prior from the 2D images, but they are not equivariant since they explicitly use the coordinates of the points or the rays in the network.
The equivariance in the paper [21 not the equivariance we mean in this work: 21 enforces the geometric consistency via a loss function. [47] is not strictly equivariant and it depends on the assumption of upright views and the camera II embeddings, which results in its non-equivariance to camera permutation. It achieves data-driven equivariance by randomly choosing the first camera frame as the canonical frame. While (45 addresses the frame problem through relative pose, is not theoretically equivariant in cases of individual camera rotations around their axes or minor individual rotations accompanied by small content changes. We want to emphasize that our central contribution is to propose an equivariant convolution and transformer on ray space, which can be integrated into wide range of 3D learning models.
The most related equivariant rendering approaches to us are
45
Reconstruction from Multiple Views Dense reconstruction from multiple views is a well- established field of computer vision with advanced results even before the introduction of deep learning (28). Such approaches cannot take advantage of shape priors and need a lot of views pro- vide dense reconstruction. Deep learning enabled semantic reconstruction, i.c., the reconstruction from single or multiple views by providing the ground-truth 3D shape during training [4] 67 68 40).
These approaches decode the object from global code without using absolute or relative camera poses. Regression of absolute or relative poses applied in [35 non-equivariant.
56
69
72
57
46
33 20 is
3 Method
In this section, will first introduce the (feature) field the ray space and 3D Euclidean space. respectively, and the corresponding SE(3) group actions on the values of the fields in Sec. [.1 To offer readers a holistic grasp-five from broad overview down to the intricate specifies and from foundational concepts to advanced techniques- we will present the reconstruction and the generalized rendering with the neural components of their architectures (convolutional and attentional) and their inputs and outputs in Sec. 3.2 and Sec. [3.3] Following that, we expose our central contribution: equivariant convolution and attention in ray space in Sec. 3.4 and Sec. 3.5
3.1 Feature field on Ray Space and 3D Euclidean Space
3.1.1 Ray Space
The ray space is the space of oriented light rays. As introduced in the introduction and App. Ex. we use Plücker coordinates to parameterize the ray space R: for any ray x E R, x can denoted as (d, m), where d e 52 is the direction of the ray, and x d is the moment of the ray with
3

---

scaler festuree vector features type-1 features)
E SE(3)
type-1 features
type-1 feetures
lype-1 fealures
ocaler fealures
scalar features
scalar features
Figure I: Feature attached to rays: we show the scalar feature and type-1 feature. When P2 is the trivial representation, tensor fea- tures can be viewed in the plane orthogo- nal to the ray (the blue plane). When rota- tions act on the feature field, the scalar fea- ture only changes position as attached to the rays: (C,
Figure 2: Features attached to points: we show scalars and vectors (type-I features). The black dot in the figure is the point, and the square and the vectors are the scalar and type-1 features attached to the point. When g SE) acts on the feature field, we will see that the scalars are kept the same while the attached position is rotated, and the vector features change their posi- tion and alter their direction.
f(g "x): while the type-1 feature changes position and is itself rotated: (z) = p(((g xx)- (f)=)) 12), where p(7;1) e
being a point on the ray. Then any g (R,t) E SE(3) acts on the the ray space as: gr = g(d, m) = (Rd, Rm + t x (Rd)).
The ray space R a homogeneous space with transitive group action by SF(3). Given the origin in the homogeneous space as 1 ([0. 0, 1 [0, 0, 0)²) (the line representing z-axis), the stabilizer group H that leaves 7/ unchanged is SO( x IR (the rotation around and translation along the ray). The ray space is, thus, isomorphic to the quotient space R _ (E3))(SO(2) x IR.. We parameterize the stabilizer group as H = {(7, 1) E 20, ( E IR.
We follow the generalized convolution derivation for other homogeneous spaces in 18 which requires the use of principal bundles, section maps, and twists 29 explained in the appendix section A.2 and onwards. S F3) can be viewed as the principal SO(2) x R-bundle, where we have the projection p : SF((3 >> R. for any g E SF3)). p(g) = gry a section map 8 : R -> S F33) can be defined such that p = idg. In App. Example 6. we elaborate on how we define the section map from the ray space to SE(3) our model. Generally, the action of SE(3) induces a twist as gs(a) + s(gz). The twist can be characterized by the twist function h SE(3) x R SO(2) x R. gs(x) s(gr)h(g, r),, we provide the twist function in our model and its visualization in App. E
3.1.2 Light Field
The light field can be modeled function from the ray space to a vector space, f R -> V. We also need to define the (3) group action on the values of that field. Since the group action will be on vector space V, we will the corresponding group representation of the stabilizer group p: SO()) x TR + GV.(V), see details in App. A.3 For example, a light field can be a radiance field that maps the ray space of oriented rays to their observed radiance (RGB) f: R IR3 which is concatenation of three scalar fields over R. The group representation in this case is the identity and g E SE(3) acts on the radiance field f as (f)
f(g 22), shown as the scalar features in Fig. I Given that the stabilizer H = S0(2) x R is a product group. the stabilizer representation can be written as the product p(7,t) (1(7) @ Pz (t), where P1 is the group representation of SO and P2 is the group representation of IR. If the light field is feature field (Fig. [ with P2 being the identity representation and P1 corresponding to a type-I field, p1(7) = c²>, then type-1 features change position and orientation when E SE(3) acts on it. Having explained the examples of scalar (type-0) and type-1 fields, we introduce the action on any feature field f as [8]:
(z,)))x) p(h(g =))=
(2)
where p is the group representation of SO()) x R corresponding to the space V. determined by the field type of S, and is the twist function introduced by SE3) as shown App. Example
3.1.3 Feature Field on 3
R33 is also a homogeneous space of SFF(3) like the ray space R. with the stabilizer group as 50(3).
as stated in App. Example 2 For any g (R, t) E SE(3), it acts on the field f over RR3 also follows

---

point feyture
convolution points
quary
a. convolution from rays rays
(ay)
c. equivariant
cross attention
network
quary
key
point
points and rays
light fleld
light feature field
point feature field
Figure 3: The pipeline equivariant 3D reconstruction: Firstly, we obtain the feature field over the ray space. Secondly, we perform an equivariant convolution from ray space to point space. Thirdly, we apply a (3)(3 equivariant cross-attention module to obtain an equivariant feature for a query.
18
(c,f)(z) =C_) (A
.where is the group representation of SO(3), since the twist function can be independent of the 3D position due to the fact that SE(3) = R$ SO(3) is a semidirect product group as stated App. Example 4 The feature field over R and the corresponding group action is also used in [55] 63 Fig. 2 visualizes the scalar feature (lou = 0) and vector feature (lout = 1) attached to one point, offering an intuitive understanding of the feature field over R.
Given the feature field on the ray space and 3D Euclidean space and the corresponding group actions of (3)... We will show two 3D multi-view applications of the equivariant convolution and transformer: 3D reconstruction and generalized neural rendering. In each application, we start with the specific definition of equivariance and then outline the corresponding pipeline.
3.2 Equivariant 3D Reconstruction
The radiance field serves as the input for the 3D reconstruction, which ultimately generates a signed distance field (SDF) denoted by the function T TR. As aforementioned, the radiance field is the multi-channel scalar field over R, while SDF is the scalar field over RR3 A AD reconstruction $ F E, where F denotes the space of radiance fields and denotes the space of signed distance fields, is equivariant when for any E SE(3), any x E R3, and any f E F, f)) = C (z)) where Lg and Cg are group actions on the light field and the SDF, respectively. Specifically, as ʃ and are scalar fields, (z) = f(g-'x) for any x E R, and (Z)))(z) = e(g-'x) for any E R.3.
In practice, we have finite sampling of the radiance field corresponding the pixels of multiple views V = [s()) E }v where Lv denotes the ray set of multi-views and E F is the radiance field induced by multi views sample from. The 3/) reconstruction O is equivariant when for any E SE)) and any & E R3; 4(g V)( = 4 (VV g (g Lv, Lggf), where g Lv is acting on the rays defined Eq.
we denote V as (Iv,J),
achieve equivariance using three steps as illustrated in Fig. (1) the transition from pixel colors to feature-valued light field (cqui-CNN over rays), (2) the computation of features in 3 from features on the ray space by equivariant convolution from R to R3, and (3) the equivariant transformer with the query generated by the feature the point want compute SDF and key/value generated by features on rays. Note that we need (3) following (2) because the output feature of a single convolution layer is expressive enough due to the constrained kernel.For the detailed practical adaption of the convolution and transformer in 3D reconstruction, please see the App. B where we approximate the intra-view with SE(2) equivariant convolution.
3.3 Generalized Neural Rendering
The light feature field fm R - V serves as the input for neural rendering, which ultimately generates the light field f R + R a multi-channel scalar field over R. A neural rendering
V V I + F, where Z denotes the space of the light feature fields and denotes the space of the light field, equivariant when for any g E SE(3), any x E R, and any J' E I, (Cy (x) T (finn) -g- -1r: where Lg the group operator on the light feature field fin. as shown Eq. depending on the feature type. In the experiment of this paper, the input light feature field
5

---

mult-haad attention
urggu reproserttioon
Convolution from rays to
resentationn
Equitoriant Transformrr over points
Transfomer rays
Scaler Beld
mguaar representation
reguler representation
density
Figure 4: The pipeline of equivariant neural rendering. Firstly, obtain the features of the points along the target ray through convolution over rays. Secondly, we apply the equivariant cross- cross-attention module to obtain features for generating the color of the points. Finally, we use equivariant self- attention over the points along the ray to obtain features for generating the density of points.
scalar, i.e., (z) (inl Similar to reconstruction, in practice, the neural rendering vr equivariant when for any E SF((3 and any x E R: (g ())x) = (g) x), where V (x)]( }v}, and if we denote V as [Lv,fin) then g
Lv. finn))
By restricting the field type of the output field over rays have group representation SO x IR as ((7,)) p1 (7) & (t). where P2 is the regular representation, we can obtain the feature of points along the ray by convolution or transformer from R. to R. See App. Example 9 for more explanation of the regular representation. Alternatively, we can obtain the desired feature by applying convolution or transformer from R to R. with output features attached to the target ray corresponding to different irreducible representations of the stabilizer group. These features can be interpreted as Fourier coefficients of the function of the points along the ray. The Inverse Fourier Transform yields features for the points along the ray. More details in the App.
The feature of the points along the ray can be used to generate density and color for volumetric
rendering [61 73), or fed into attention and pooling for the final ray feature [58]. In this paper, we opt to generate the density and color and utilize volumetric rendering, which can be viewed as specialized equivariant convolution from points to the ray. Method details are in App. []
We achieve the equivariant rendering through three steps as shown in Fig. 4 (1) we apply equivariant convolution from rays to rays to get the equivariant feature for points along the rays, which is specific field type over R; to enhance the feature expressivity, we apply an equivariant transformer from rays to rays to get the color for each point; (3) we apply the equivariant self-attention over the points along the ray reason over the points on the same ray; the output feature of the points will be fed to multiple perceptron layers to get the density of the points.
3.4
Convolution in Ray Space
3.4.1 Convolution from Rays to Rays
The convolution, as stated in App. A.4 and 18 is then defined as
f [(s(()) y) (((()))))((((()(sss())())))))))) -ls(v)))f (y))),
plant (x) =
(3)
where h(g) is the simplified form of the twist h(g, m). Eq is equivariant to SE(3) if and only if the convolution kernel satisfies that x(hx)= Pout (h) (in (h, where Pin and Pout are the group representations of SO(2) x R corresponding to the input feature type 4n and output feature type lout. respectively. We derive the solutions of the kernel in the App. Example
Local kernel support The equivariance stands even we constrain the kernel to be local. When (dz, my)) meets the condition that L(dg, [0, 0, 11²') Bo and d(x, 11) S dg. x(x) # 0, this local support will not violate the constraint that k(hz) Paun (ha(h))(()pp) (h,, (h,z)).

---

Then, convolution in Eq. 3 is accomplished over the neighbors only as visualized in Fig. 5 In Fig. any ray y (d, my (denoted in blue) the neighborhood of a ray x (d,, m, will go through the cylinder with x as the axis and dg as the radius since d(x, v) I dg-
Moreover, for any y, Z (dy, d.) S 9o- Any ray y (x:) is on one tangent plane of a cylinder with I as the axis and d(æ, y) as the radius when d(x, y) > 0.
3.4.2 Convolution from Rays to Points
In applications such as 3D reconstruction, key point detection, and 3D segmentation, we expect the output to be the field over IR' Using a convolution, we will define an equivariant map from light fields (fields on R. to fields on R We denote with Hj and H2 the stabilizer groups for the input and output homogeneous spaces, respectively. i... SO(2) R and SO(3) in this case. As shown in the App. Example we can choose the section map 82 IR3 (E))): $2(x)= = (I, æ) for any x E IR3 and is the identity matrix. Following [8]. the convolution from rays to points becomes:
Bo
Bo
do
X
Figure 5: Neighborhood of a ray x in the convolution.
1 (x) = / x(x2(x)~~y))pmn) (82(x) - (w))) (y)dy.
where h is the twist function corresponding to section 81 R (E(3) defined aforementioned, Pin is the group representation of H, (SO(2) x I)) corresponding to the feature type lin- The subscripts I and 2 denote the homogeneous spaces the features are defined on. The convolution is equivariant if and only if the kernel satisfies that x (12z) Poou Pour(12) (h2) (22)) (h, (h2,z)) for any h2 H2, where Pout is the group representation of H2 (SO(3)) corresponding to the feature type lout-
In 3D reconstruction, flan is the scalar field over R, i.c., Pin 1. The convolution is simplified to How (a)) = Sa/H, (x)2(x)) --)yy))" """ (y)dy and the corresponding constraint becomes k(hgar) Poia (h2) App. Example 10 provides analytical kernel solutions.
3.5 Equivariant Transformer over Rays
We can extend the equivariant convolution to the equivariant transformer model. In general, the equivariant transformer can be formulated as:
(x fe(o,, BEN(x) EyEN(h) (fp((ff(1. ff( fo(z.y. Fout (x)
Σ
where the subscript 1 denotes the homogeneous space M2 S G/H, of the feature field fi" that generates the key and value in the transformer; the subscript denotes the homogeneous space M2 x H/H2 of the feature field få" that generates query in the transformer, which is also the homogeneous space the output feature fout, x and y represent elements in the homogeneous spaces M2 and M,, respectively, where y E (xx)) indicates that the attention model is applied over y. neighbor of based on a defined metric. Jt. Jq. and fo are constructed equivariant keys, queries, and values in the transformer. Sk and fo are constructed by equivariant kernel and K, while f, is constructed through an equivariant linear map, see App. for detailed construction.
When the transformer is a self- self-attention model, homogeneous space M1 and M2 are the same since f fin The above equivariant transformer could be applied the other homogeneous space other than R, 3 and acting group other than (E()). This paper presents the equivariant cross-aa aots-atttnnion model over rays, i.e., M, R. When the transformer is the cross-attention from rays to rays, M, is also R, the equivariant kernel Kę and K. is the convolution kernel we derived in convolution from rays to rays in Sec. 3.4.1 When the transformer is the cross-attention from rays to points, M2 R3, the equivariant kernel and K2, is the convolution kernel we derived in convolution from rays to points in Sec. 3.4.2] With the construction in App.
[ we claim that the transformer from rays to rays is equivariant. The proof is in App.
or from rays points, shown in the equation
7

---

Win
Roin
queeryy
kay
value
Multihedd Antention
Mult Mult-head Attention
Irst
Frem-
query
key
voluuer
In the equivariant transformer (L), positional en- coding is not directly used due to its lack of equivariance. Instead, the relative position within the kernel is utilized. To generate the query fq. we multiply the feature f2" (z) (pre-existing or yielded by convolution) attached to x (in R or IR3, depending on the task) by the designed equivari- ant linear matrix W (see App. FD. The key fk and value fe are constructed using designed equivariant kernels Ką and K.. The transformer is equivariant due to equivariant Su. Sa. and As.As Sk. Jq and fa are equivariant, the entire transformer equivariant. The conventional transformer (R) uses point position encoding for the query feature and obtains the query, key, and value through nonequivariant conventional linear mappings.
Figure 6:
Figure 7: In the equivariant trans- former (U), the query, key, and value are equivariant and can be composed of different types of features; they can be scalars, vee- tors, or higher-order tensors. The inner product, determined by the feature type, should apply to the same type features. In con- trast, the feature in conventional transformer (D) is not equivariant, it does not contain vectors and tensors, and the inner product is conventional.
To better understand the equivariant transformer, we visualize the comparison of the equivariant cross- attention transformer and conventional transformer shown in Fig. Meanwhile, as stated in App. FF) key, query, and value are generally composed of different types of features and are multi-channel. allowing for the multi-head attention mechanism. In Fig. 7 we visualize the comparison of the equivariant multi-head attention module from rays points with conventional multi-head attention module. The attention module from rays to rays follows a similar concept but with variations in the feature types due to the differing group representations of SO(2) x R and SO(3). Experiment
4
4.1
3D Object Reconstruction from Multiple Views
Datasets and Implementation We use the same train/val/test split of the Shapenet Dataset [9] and render ourselves for the equivariance test. To render the views for each camera, we fix eight cameras to one cube's eight corners. The cameras all point in the same direction toward the object's center. We use the following notation to denote the variety of transformations in training and testing: (no transformation), Z (optical axis rotation), R (bounded 3-dof camera rotation), Y (vertical axis object rotation), SO(3) (full object rotation). The details of the five settings are in App. J.1
As described App. B we use (2) equivariant CNNs to approximate the equivariant convolution over the rays. For the fusion from the ray space to the point space model, we use one layer of convolution and three combined blocks of updating ray features and SE(3) transformers. For more details, please see the App. J.2
Results We evaluate 50(3)/5033) The setting A/B indicates training the model on the A setup of the dataset and evaluating it the B setup. Following the previous works, use loU and Chamfer-LI Distance as the evaluation metric. Quantitative results are reported in table T and qualitative results are Fig. g
model in seven experiment settings, I/I, 1/Z, I/R, R/R, Y/SO(3),
We compare with two other approaches [69] which follows classic paradigm that queries 3D positions that are then back-projected to obtain image features for aggregation, and [72), which was state of the art in 3D object reconstruction from multi siittivvees.Nook that both baselines originally estimate the object poses, but we directly provide ground truth poses to them. See App. J.4 for more qualitative results.
In table I. our model outperforms the 72 and (69 by a large margin on 1/2, I/R, and Y/SO(3)
settings. Although theoretically, our model is not equivariant to the arbitrary rotation of the object, Y/SO(3) shows the robustness of our model to the object rotation and the generalization ability to some extent. Our model outperforms other models for the chair and car categories in R/ R and
8

---

Method
Chair
1/1
1/Z
V/R
R/R
Y/Y
Y/SO(3)
SO(3)SO(3)
w/ gt pose 72
0.691/0.099
0,409/0.253
0.398/0.257
0.669/0. 113
0.687/0. 103
0.518/0. 194
0.664/0.1 114
DISN w/ gi pose
0.725/0.094
0.335/0.396
0.322/0.405
2050000.201
0.659/0 120
0,419/0.303
0.549/0. 174
Ours
0.731/0.090
0.631/0.130
0.592/0.137
0.689/0.05
0.698/0.102
0,589/0.142
0,674/0.113
Method
uirplane
I/2
1/R
R/R
Y/Y
Y/SO(3)
SOL3WSOL3)
Fvor gt pose
0.770/0.051
0.334/0.16
0.533/0.174
0.766/0.053
0.379/0.147 0.7622
0.746
DISN w/ gt pose(69
0.752/0.058
0.465/0,
1.462/0.171
1/6.1/0.104 104
0. 0.706/0.069
0.530/0.15 151
0,631/0.103
Ours
0.773/0.050
0.600/0.092
0.579/0.100
0.759/0.051
0.734/0.052
0.597/0.101
0.722/0.056
Method
car
1/2.
I/R
R/R
Y/Y
Y/SO(3)
SOG3YS0(3)
Fvor w/ gt pose 72
0.837/0.090
0,466/0.254
0.484/0.258
0.816/0 107
0.830/0.094
0,496/0.240
0.798/0.111
DISN w/ gt pose(69
18222000000 089
0.610/0.2.33 232
0.567/0.236
0.772/0. 135
0,802/0.098
0.614/0.205
0.769/0 123
Ours
0.844/0.081
0.739/0.142
0,741/0.150
0.836/0.089
0.830/0.089
0.744/0.137
0.813/0.097
Table I: The results for the seven experiments of 8- v-veww 3D reconstruction for the ShapeNet dataset.
The metrics in the cell are IoU't and Chamfer-LI Distancel. We implement 722 and [69] ourselves on
equivariant dataset. For the performance of [69], we follow their work to conduct the multi-view reconstruction by pooling over the feature of every view. The value of Chamfer-I.I Distance x 10.
50(3)/50(3) settings while it is slightly inferior to 72 in the airplane category. Notably, our model only requires relative camera poses, while 72 and 69 utilize camera poses relative the object frame, leveraging explicit positional encoding of the query point in the object frame, which is concatenated the point feature. In addition, our model performs better in several experiments in T/T and Y/Y settings. This superiority can be attributed to the SFF(3) equivariant attention model, which considers scalar features and ray directions. For detailed discussion of the results, please see appendix Sec. J.3
We provide ablation study of the effectiveness of (2) CNNs, equiv- ariant convolution, transformer, and type- 1 feature (vector feature) in our model. Meanwhile. we compare our method with the model that explicitly encodes the direction of rays. Please see the App. J.5 for the details of the ablation study.
FvOR
DISN
Ours
GT
Y/
4.2 Neural Rendering
Datasets and Implementation We the same training and test dataset as in [61], which consists of both syn- thetic and real data. Two experiment settings illustrate our model's equiv- ariance: I/I and I/SO(3). I/I is the canonical setting. where we train and canonical
RIR
50(3)
test the model in the frame defined in the dataset. In the I/SO(3) setting,
US0(3)
test the model trained in the conical frame under arbi- trarily rotated coordinate frames while preserving relative camera poses and the relative poses between the cam- era and the scene, thereby preserv- ing the content of the multiple views. Each individual view itself is not trans- formed. Note that this experiment's SO(3) setup differs from the R and SO(3) setups used in the reconstruc-
Figure 8: Qualitative results for equivariant reconstruction. Left: input views; Right: reconstruction meshes of different models and ground truth meshes show how the model is trained and tested, explained in the text.
tion. Further details and discussions on this difference can be found in App. K.I

---

IBRNet IBRNet (1/S03) Ours (V1) Ours (V503)
GT
IBRNet (1/I) IBRNet (VS033) Ours (I/1 Ours lV/Oss)
Figure 9: Qualitative results for Generalized Rendering. We observe a performance drop for IBRNet from to S3(3), while ours are robust to rotations in the testset.
Dataset
Method
1/1
I/SO(3)
PSNR
SSIM
LPIPS.
PSNRT
SSIM
LPIPS.
Pix- Var.
Realistic Synthetic
IBRNet
26.91
0.928
0.084
26.77
0.923
0.091
66.58
360 [42
Ours
26.90
0.929
0.086
26.90
0.929
0.086
0.00
Real
1BRNe(611
25.13
0.817
0.205
24.60
0.797
0.223
52.66
Forward-Fand Fenwadd-Facing [4]]
Ours
24.93
0.808
0.212
24.93
0.808
0.212
0.00
Diffuse Synthetic
IBRNet [61
37.21
0.989
0.019
37.07
0.988
0.019
34.51
360 44
Ours
37.11
0.987
0.019
37.11
0.987
0.019
0.00
Table 2: The results for the experiments of generalized rendering without per-scene tuning. The metrics in the cell are PSNR1, SSIM1, and Pixel Variance,, (denoted as Pix- Var). The evaluation of IBRNet performed by testing the released model on both canonical and rotated test datasets.
Our model architecture is based on IBRNet[]], with view feature aggregation and ray transformer components modifications. Specificallyy, we replace the view feature aggregation in [61] with the equivariant convolution and transformer over rays and the ray transformer part with the equivariant self-attention over the points along the ray. For more information on the implementation details, please refer to App.
Results We compare with IBRNet on 1/1 and Fond//S00 1/30(3) settings to show that our proposed models can be embedded in the existing rendering framework and achieve equivariance. Following previous works on novel view synthesis, our evaluation metries are PSNR, SSIM, and LPIPS (74] In the 1/)0(3) test period, we randomly rotate each data six times and report the average metrics. Meanwhile, we record the max pixel variance and report the average value. We show a qualitative result in Fig. D where IBRNet presents several blurred transverse lines in the I t (3) setting while ours are robust the rotation. table 2 our model performs comparably with IBRNet [6 I/I setting without performance drop in T 7/50(3) setting. The slight decrease in PSNNR/SSI/LPPIE for IBRNet from T/T to ///tto//$30 (3) can attributed the training process involving multiple datasets with different canonical frames, which includes transformation augmentation and makes the model more robust to coordinate frame changes. Additionally, conventional metrics like PSNR/SSIM may not directly capture image variations. Therefore, introduce additional metric, pixel variance, to illustrate the changes better. observe that IBRNet 66 exhibits pixel variance for different rotations, whereas our approach remains robust to rotation. Our method performs comparably with IBRNet in the I (3) setting in Deep Voxels [49 because the synthetic data consists of Lambertian objects with simple geometry, where the ray directions do not significantly affect the radiance. For more qualitative results, see App. K.3
Conclusion and Broader Impacts
To learn cquivariant geometric priors from multiple views, we modeled the convolution on the light field as a generalized convolution on the homogeneous space of rays with SE(3) the acting group. To obtain expressive point features, we extended convolution to equivariant attention over rays. The main limitation of the approach is the finite sampling of the light field. The sampling of the light field by sparse views cannot account for large object motions with drastic aspect change, leading to breakdown of equivariance. This novel general equivariant representation framework for light fields can inspire further work on 3D vision and graphics tasks. We do not see any direct negative impact of our work, could have negative societal consequences if misused without authorization, for example, when using images violating privacy.
10

---

6 Acknowledgement
The authors gratefully acknowledge support by the support by the following grants: NSF FRR 2220868, NSF IIS-RI 2212433, NSF TRIPODS 1934960, NSF CPS 2038873.
References
[1] Jimmy Aronsson. Homogeneous vector bundles and g-equivariant convolutional neural networks. Sampling Theory, Signal Processing, and Data Analysis, 20(2):1-35, 2022.
[2] Benjamin Attal, Jia-Bin Huang, Michael Zollhöfer, Johannes Kopf, and Changil Kim. Learning neural light fields with ray-space embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[3] Miguel Angel Bautista, Walter Talhott, Shuangfei Zhai, Nitish Srivastava, and Joshua M Susskind. On the generalization of learning- based 3d reconstruction. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2180-2189. 2021.
[4] Erik Bekkers. B-spline cnns on groups. arXiv preprint 15kiv:9909.120000 2019.
[5] Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. X-fields: Implicit neural view-, light-and time-image interpolation. ACM Transactions on Graphics (TOG), 39(6):1115, 2020.
[6] James R Bergen and Edward H Adelson. The plenoptic function and the elements of early vision. Computational models of visual processing, 1:8, 1991.
[7] Johannes Brandstetter, Rob Hesselink, Elise der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:2110.02005, 2021.
[8] Gabriele Cesa, Leon ang, and Maurice Weiler. A program to build e (n)-cquivariant stecrable cnns. In International Conference on Learning Representations, 2021.
[9] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rXiv:151..03012. 2015.
[10] Evangelos Chatzipantazis, Stefanos Pertigkiozoglau, Edgar Dobriban, and Kostas Daniilidis. SFG3)- equivariant attention networks for shape reconstruction in function space. arXiv preprint arXiv:2204.02394 2022.
[11] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124-14133, 2021.
[12] Haiwei Chen. Shichen Liu. Weikai Chen, Hao and Randall Hill. Equivariant point network point cloud analysis. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14514-14523, 2021.
[13] Yuedong Chen, Haofci Xu, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfci Cai. Explicit correspondence matching for generalizable neural radiance fields. arXiv preprint 2xXi:23044112244 2023.
[14] Christopher Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: unified approach for single and multi-view object reconstruction. European conference on computer vision, pages 628-644 228-444. Springer, 2016.
[15] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999, PMLR, 2016.
[16] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling, Gauge equivariant convolutional networks and the icosahedral cnn. In International conference Machine learning, pages 1321-1330. PMLR, 2019.
[17] Taco S Cohen, Mario Geigor, Jonas Köhler, and Max Welling. Spherical cnns. arXiv:1801. 10130 2018.
arXiv preprint
[18] Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
11

---

[19] Congyue Deng, Or Litany, Yueqi Duan. Adrien Poulenard, Andrea Tagliasacchi. and Leonidas Guibas. Vector neurons: A general framework for so (3)-equivariant networks. arXiv preprint arXiv:2104. 12229. 2021.
[20] Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann, Learning to render novel views from wide-baseline stereo pairs. arXiv preprint arXiv:230.008463 2023.
[21] Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference Machine Learning, pages 2761-2770. PMLR, 2020.
[22] Carlos Esteves, Christine Allen Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-68, 2018.
[23] Carlos Esteves, Yinshuang Xu, Christine Blen-Blanchette. and Kostas Danillidis. Equivariant multi- view networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1568-1577, 2019.
[24] Carlos Esteves, Ameesh Makadia, and Kostas Danillidis. Spin-weighted spherical cnns, arXiv preprint arXiv:2006.100 10731, 2020.
[25] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson, Generalizing convolutional neural networks for equivariance to lie groups arbitrary continuous data. In International Conference Machine Learning, pages 3165-3176. PMLR, 2020.
[26] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons arbitrary matrix groups. arXiv preprint arXiv:2104.094 09459, 2021.
[27] Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. Se transffooore:s: 3d roto- translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020.
[28] Yasutaka Furukawa, Carlos Hernández, et al. Multi-view stereo: A tutorial. Foundations and Trends@ in Computer Graphics and Vision, 91-2):1-148, 2015.
[29] Jean Gallier and Jacelyn Quaintance. Differential geometry and Lie groups: a computational perspective, volume 12. Springer Nature, 2020.
[30] Jiaming Han, Jian Ding, Nan Xue, and Gui-Song Xia. Redet: rotation- -quiiarrant detector for acrial object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2786 2795, 2021.
[31] Xin Huang. Qi Zhang, Ying Feng, Xiaoyu Li, Xuan Wang, and Qing Wang. Local implicit ray function for generalizable radiance field representation. arXiv preprint arXiv:2304. 12746. 2023.
[32] Michael Hutchinson, Charline Lan, Sheheryar Zaidi, Emilien Dupont, Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups. In International Conference on Machine PMLR. 2021.
Learning, pages
[33] Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. arXiv preprint 2rX1v:22/2.04449. 2022.
[34] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG), 35(6):1-10, 2016.
[35] Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. Advances in neural information processing systems, 30, 2017.
[36] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference Machine Learning, pages 2747-2755. PMLR, 2018
[37] Mare Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31-42, 1996.
[38] Xiaoxiao Long, Cheng Lin, Peng Wang. Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. arXiv preprint 2rXiv:2206.05737 2022.
12

---

[39] Lachlan E MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbitrary lie groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8183-8192, 2022.
[40] Lars Mescheder, Michael Oechsle. Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4470,, 2019.
[41] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz- Prüiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4): 1-1, 2019.
[42] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99-1066 106, 2021.
[43] David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoom. Attentive group equivariant convolutional networks. In International Conference on Machine Learning, pages 8188 8199. PMLR, 2020.
[44] David W Romero and Jean-Bapiin Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. arXiv preprint arXiv:2010.00977, 2020.
[45] Aleksandr Safin, Daniel Durckworth, and Mehdi SM Sajjadi. Repast: Relative pose attention scene representation transformer. arXiv preprint arXiv:2304.0000 00947, 2023.
[46] Shunsuke Saito, Zeng Huang, Ryota Natsume. Shigeo Morishima, Angjoo Kanazawa, and Hao Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2304-2042000 2414,,314, 2019.
[47] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229-6238, 2022.
[48] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323-9332. PMLR, 2021.
[49] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437-2446 2019.
[50] Vincent Sitzmann, Semon Rezchikov, Bill Freeman. Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems,
2021.
[51] Pratul Srinivasan. Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. Learning to synthesize a 4d rgbd light field from a single image. Proceedings of the IEFE International Conference on Computer Vision, pages 2243-2251, 2017.
[52] Norman Steenrod. The topology of fibre bundles, volume 27. Princeton university press. 1999.
[53] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering, arXiv preprint arXi::2207. 10662, 2022.
[54] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8269-8279, 2022.
[55] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-ant and translation-eguivrrinn neural networks for 3d point clouds. arXiv preprint arviv:
/XXXx:8002.8888 802.00222221 0821,, 2018.
[56] Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Multi-view consistency as supervisory signal for learning shape and pose prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2897-2905, 2018.
[57] Michal J Tyszkiewicz, Povis-Koiitsi Maninis, Stefan Popov, and Vittorio Ferrari. Raytran: 3d pose estimation and shape reconstruction of multiple objects from videos with tractr transformers. arXiv preprint 220i 230.,120.0.,26 2022.
13

---

[58] Mukund Varma, Peihao Wang. Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In The Eleventh International Conference Learning Representations, 2022.
[59] Soledad Villar. David W Hogg. Kate Storey-Fisher, Weichi Yao, and Ben Blum SlitmShmthh Scalars universal: Equivariant machine learning, structured like classical physics. Advances Neural Information Processing Systems, 04128848-28865 2021.
[60] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 52 67, 2018.
[61] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser, Ibmet: Learning multi-view mage-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690 4699, 2021.
[62] Maurice Weiler and Gabriele Cesa. 1..0v:191..0025), 2019.
General e(2) -quiiiarvar steerable cnns.
arXiv preprint
[63] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d stecrable enns: Learning rotationally equivariant features in volumetric data, arXiv preprint arXiv:1807.022 025477 2018.
[64] Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolu- tional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2 06020, 2021.
[65] Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov. and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
[66] Gaochang Wu, Yebin Liu, Lu Fang, and Tianyou Chai, Revisiting light field rendcring with deep anti- aliasing neural network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[67] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang. Pix2vox: Context- aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference computer vision, pages 2690-2698, 2019.
[68] Haozhe Xie, Hongxun Yao, Shengping Zhang, Shangchen Zhou, and Wenxiu Sun. Pix2vax++; Multi-scale context-aware 3d object struction from single and multiple images. International Journal of Computer Vision, 128( 122229299992 2020.
[69] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and UIrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances Neural Information Processing Systems, 32, 2019.
[70] Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniilidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. International Conference on Machine Learning, pages 24596-24614. PMLR, 2022.
[71] Mingyue Yang, Yuxin Wen, Weikai Chen, Yongwei Chen, and Kui Jia. Deep optimized priors for 3d shape modeling and reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages
2021.
[72] Zhenpei Yang, Zhile Ren, Miguel Angel Bautista, Zaiwei Zhang, Qi Shan, and Qixing Huang. Fvor: Robust joint shape and pose optimization for few-view object reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2497-2507. 2022.
[73] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 4578-4587, 2021.
[74] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 586-595, 2018.
14

---

Appendix
The introduction of convolution and attention the space of rays in 3D required additional geometric representations for which there was no space in the main paper to elaborate. We will introduce here all the necessary notations and definitions. We have accompanied this presentation with examples of specific groups to elucidate the abstract concepts needed in the definitions.
A Preliminary
A.1 Group Actions and Homogeneous Spaces
Given the action of the group on a homoge- neous space X, and given zo as the origin of X, the stabilizer group H of zo in is the group that leaves zo intact, i.e., H {h E G G 2o.. The group, G, can be partitioned into the quotient space (the set of left cosets) G/ H and X is isomorphic to G H since all group ele- ments in the same coset transform To to the same element in X, that is, for any element g' E g we have g' 3ro = gzo-
d
x
m
Example 1. SE(3) acting on the ray space
R: Take SE(3) as the acting group and the ray space R as its homogeneous space. We use Plücker coordinates to parameterize the ray space R: any x R. can be denoted (d, m), where d E S is the direction of the ray, and m x d where x is any point the ray, as shown in figure 10 A group element
The visualization of Plücker coordi- nates: A ray z can be denoted as (d, m) where is any point on the ray x, and d is the direction of the ray x. m is defined x d.
Figure 10:
g (R, E SE(3) acts on the the ray space as:
gr g(d, m) (Rd, Rm + t (Rd)).
(5)
We can choose the fixed origin of the homogeneous space to be n (00 0, 1J", 0
the line identical with the z-axis of the coordinate system. Then, the stabilizer group H (the rotation around and translation along the ray) can be parameterized as H {R2())) 0, 1)7) E [0. 2 t E RJ. i.e., H = SO(2) x R. We can simplify H as H = (7,00) E [0, 2T), E R). R is the quotient space (SO x R) up to isomorphism. Example 2. SE(3) acting on the 3D Euclidean space R3: TR& is isomorphic to SE(3)/50(3))
Consider another case when SE(3) acts the homogeneous space R for any g (R, E SE and x E R3, gæ = Rx + t. If the fixed origin is [0, 0, 0J" the stabilizer subgroup is H = SO(3) since any rotation g = (R, 0) leaves [0, 0, 0j unchanged.
Example 3. SO(3) acting the sphere S @2 is isomorphic to (0(3)/50(2). The last example is SO(3) acting on the homogeneous space sphere $2. Given the fixed origin point as [0, 0, 1)² the stabilizer group (2))).
A.2 Principal Bundle
As stated in 29 18,, the partition of the group G into cosets allows us to treat the group G as the principal bundle where the total space is G, the base space is the homogeneous space G/ G/H the canonical fiber the stabilizer group H, the projection map p G + G/ H reads p(g) = gHl G/H + G of p should satisfy that p o = idg/H. where ida/ is the identity map on C.H. Note that non-trivial principal bundles do not have continuous global section, but we can define a continuous section locally on the open set U S G//.. The action of G causes a twist of the fiber, i.e., gs(x) might not be equal to s(zz) though they are in the same coset. We use the twist function G x G/H + H to denote the twist: gs(x) = =(gz))h(g 2). Same as [18), we simplify hgg eff) to be h(g). where is the identity element in G and eHl 20-
geo = J. The section
We use G/H H to denote the homogeneous space since the homogeneous space X can be identified with G/ H up isomorphism, i.e., X ~ G/H.
15

---

p='(x) 50(2) X
(d, m)
H S0(2) x R
Figure 11: We can view SE(3) as an SO(2) x IR-rrincipal bundle, where the projection map p = SE(3) R is (R, (R(0, 0, 1]=, x (R|0, 0, 1 and the inverse of p is p (xx) {(R, (),( = z}. We use the coordinate frames (red axis denotes Z-axis, green axis denotes X-axis, and purple axis denotes Yaxxis) to denote the element
SE) because we can use the position of the coordinate origin to denote the translation t and use X-axis, Yaxxis, and Z-axis to represent the first, second and third columns in rotation R. When we say next "the coordinate frame on the line/ray" we will mean that its origin is on the line/ray. By this convention, the coordinate frames representing the element in H S0(2) x R are the frames whose Z-axis aligns with [0, O, 1)" and whose origin is [0, 0 for any E RR i.e., frames on the yellow line in the left of the figure. For one ray x = (d, m) (illustrated as the chosen blue ray), the coordinate frames on the ray x whose Z-axis aligns with the ray d are in p p-*((r). (ar). As shown in the figure, there exists bijection (gray double arrow line between p (x) and H = 50( x R. p (z) is isomorphic to H SO(2) R.
Example 4. Projection, section map and twist function for RR3 and SE(3): According to Ex. 2 we can consider a bundle with total space as (3), base space as RS, and the fiber as SO(3). For any g = (R, t) E SE()), the projection map p: (3) + R projects g as pl R, t) = t. For any E RR3 we can define the section map R - SE(3) as s(x)
(I. x). The twist function h : SE(3) R + SO(3) is that h(g. x) s(gx)) gs(x)) R for any x R and any g
(R, E SE(3). This twist function is independent of x due to the fact that SE(3) = R x SO(3) a semidirect product group as stated in [18]
Example Projection, section map, and twist function for S2 and SO(3): As shown in Ex. 3 SO(3) can be viewed as principal bundle with the base space S and the fiber S0(2). With
the rotation R E S0(3) parameterized as R = (2(aa) (3) (t)),, the projection p: G + G/H maps R as follows:
p(R) (3)00
(7)
[sin( sin (i)))oss)()sssmmm)sss))))).).))) cos
For any d E S2, the section map 8 32 - 50(3) of p should satisfy that p o 8 = idga as mentioned above, i.e., (( = d. For instance, we could define the section map as:
s (d) = (aa)
where aaa and Ba satisfies that
d = [sin (sin(())
Specifically, when d = 00,0, 1J 0a = and 3a 0: when d = -0,,0, 1]" ad = 0 and Ba m As defined, the twist function h 50(3) 2 + 50(2) is that h(R, d) = s( s(Rd)- -Rs(d). Example Projection, section map, and twist function for R. and (3): The final example is SE(3) with R as the base space and SO(2) x R as the fiber, which is the focus of this work, as shown
16

---

R (B)
d (B) ca)( SB) sin(a), cos
Figure 12: For ray (d, m.) need to choose element (R, t) SE(3) as the representative element s(x) such that s(x) [0, 0, 11" (0, ) = z. This figure shows one example the section map from ray space to (3). This map also serves the section provided in this paper. The axes of the coordinate frame the figure represent R = Sa (d) = Rz (a Ry (9a), where the green axis, purple axis, and red axis represent 1st, 2nd and 3rd column in the rotation matrix R, respectively. The origin of the frame,t = sp(d, (d, m) x m, denotes the translation.
figure
According to the group action defined in Eq. 5 the projection map p: SE(3) - R is: r((R, t)) (R, t)r (R(0, 0, 1)" x (R(0, 0, 1)))))
This represents ray direction d with the 3rd column of rotation matrix and the moment m with the cross product of the translation and the ray direction. We can construct a section 8: G/H G using the Plücker coordinate:
s((d, m)) (sa(d), (o,dm))).
where sa (d) e SS(3) is rotation that (d)(0, 0, 1)"h d. i.c., Sa is section map from S to SO(3) as shown in Ex. 5 and sb sy m) E R2 is a point on the ray (d, m). In this paper, we define the section map (( (d) (Rz(aa) Ry (Ba). d m.) where ad and fa satisfy that d Rz (o:a) Ry (34)(0 O, 1f" which is the same as Ex. Figure displays the visualization of the section map.
Given the section map, for any g == (Ry.ty) E SE(3) and x (d,, mz) E R, we have the twist function SE(() x + SO(()
is h(g, s (g=)gs((1) (he (Ry. d.), hs 2)) where h, (0(() x 8 50(2) is the twist function corresponding 8a, shown in Ex. and hy(g. 2) = (x) + ty sg(gz), Rgdz). With the above section s defined in this paper, the twist function h (3))) R S0(2) i
hig. x) = s (gz) (g)))s(z) = (Rg de), (ty. (Rydz))).
where R2 de) = Ry (BR,d,) ) (ARRd, a. RR Rz (ad, Ry (Ba.).
To understand the twist function clearly, we visualize a twist induced by translation in SE(3) in figure 13
A.3 Associated Vector Bundle
Given the principal bundle G, we can construct the associated vector bundle by replacing the fiber И with the vector space V, where V ~ 1 and H acts on V through group representation
17

---

gx
(t, (t, d) gs(x)
X = (d,
(gx)
9 = (1,t) gs(x) = sgx)) (g,x) h (g,x) = (0, (t, d))
s(x)
Figure 13: When we translate a ray x
(d, m)) with = (1 E SE(3), we will find that gs(()
does not agree with s(gx). As defined in figure 12 we have 86 sx(x) 1 d and st(gr) 1 d. Following the geometry of the figure, we obtain that h, (g, 2) (1, d) 0, O, 11a ie..hhg x) (ggz)=>gs(a) (I, (t, d) 11T) (0, d)).
p: H GL(V). The group representation corresponds the type of geometric quantity in the vector space V, for example, the scalar, the vector, or the higher-order tensor.
The quotient space E G x, H is defined through the right action of H G V: (g, v) (gh, p(h)-1a) for any h E H, E G and v E V. With the defindd projection map p G Xp V > G/ H:: p([g, ul) = gH, where [g. u] = [ghh p(h) - H), the element in G X V, we obtain the fiber bundle of the associated vector bundle, we recommend referring to the following sources: 522 and [8]
G xp V associated the principal bundle G. For more background and details
The feature function U C G/H
can encode the local section of the associated vector bundle Su U C G/H + G xxp V: so(x) = [s(x), f(r)], where s is the section map of the principal bundle defined in Sec. A.2 The group G acting on the field f as shown in [18
(z). p(h(g ->)==>(>( f(g 'x)),
where h : G x G/H H the twist function as defined in Sec. A.2
A.4
Equivariant Convolution Over Homogeneous Space
The generalized equivariant convolution over homogeneous space, as stated (18] that maps a feature field fiom over homogeneous space G/H,, to a feature flans over homogeneous space G/ H2 by convolving with kernel к is defined as:
Jar ((s2(=)~\))pmm (h)2x))) (t)))
flone (xx)
JaIn,
where in and Yout denote the input and output feature types, respectively, Pin is the group representation of Hi corresponding to the feature type lin- $1 is the section map from G/ H2 to G (see Sec. |A.2.. $2 is the section map from G/Hz G (see Sec. A.2), h is the twist function corresponding to $1 (see Sec. A.2
The convolution is equivariant with respect to G, that is
Cout flee Cin plu..
IIn this context, the feature type indicates specific geometric quantity vector spaces Va, and Vout. Vi. corresponds to the stabilizer H; and Va.. corresponds to the stabilizer Hz, Hi and Ha can be distinct; therefore, to differentiate the types of features corresponding to different stabilizers, we utilize I and l' as notations for the feature types.
18

---

if and only if n(h2a) = Pout (()))) (hn (h2, representation of H2 corresponding the feature type l'ut-
for any h2 E H2, where Pout is the group
In the following examples, we will illustrate three instances where the input and output homogeneous identical, meaning that H1 H2. These examples involve convolutions from I to R2, from & to 8", and from R to R. Furthermore, we will show an example where H1 and H2 differ, explicitly focusing
spaces, denoted G/H1 and G/H2, respectively,
the convolution from R to RR. Example 7. (3)3) equivariant convolution from IR3 to R If we use the section map as stated Ex.
4
will find that
I, therefore convolution becomes:
J k(s(z)) ((g(x))')(f0)))) f' (y)dy
flumi (x)
Ra
= J 22
k(y - x) (1)d
and R. should satisfy
R( Rx) = Pout (
(R, Pout (t) Pout (n (R)
for any R E SO(3). When the feature type lin and lout corresponds to the irreducible representation, we have
k: [ Di... Di., (R)-1
where D'ie and D'out are the Wigner-D matrices, i.e. irreducible representations corresponding to the feature types lin and lout, which the same as the analytical result
63
Example 8. SO(3) equivariant spherical convolution from $ to S?: For spherical convolution, when we substitute the section in Eq. 7 with the section we defined in Ex. the convolution integral takes
the following form:
ploue (0,
<<
Je1E)o.20), Pin (n((h (B) (B)) (a', S' da' sin( )d3
where 0, is the fixed original point as stated in Ex. Pin is the group representation of SO(2) corresponding to the feature type lin- When Pin and Pout are the irreducible representations of S0(2). Pin and Pour can be denoted as Pin (0) e -il.n6 and Pout (0) e -ooue
To simplify the notation, we utilize R(0) to represent Rz E SO(2), where E (0,2x). When considering the cases where
0, 1)" h(R(0)z) R(-0); and when E 2 - {10, 0, 1] J0, 11T) h(R(() = R(-0)= T. Therefore, the kernel should satisfy the following conditions: x(R(f)z) - (ooooo0(( for any R(0) E SO(2) and any x E 2 {10,0 0, 1]", [0, 0, 1 s(x) e -00lomt-hhw00 for [0,0, 1)"; and x(x) = e iilou Ho. x(x) for x = - [0, 0, 1.
[0, 0, T) h(R(0)z) R(0); when
Specifically, when the input and output are scalar feature fields over the sphere, convolution reads
fout (0,3)
(a',8"))a' (d)(
R. has such constraint:
x(R())z) =x() k(z)
for any R(0) E SO((),, which is consistent with the isotropic kernel of the convolution in (22]
19

---

Example 9. (3) equivariant convolution from R to R: In our case, the equivariant convolu- tion from ray space to ray space also based on the generalized equivariant convolution over homogeneous space. See Sec. 3.4.1 for the details. We solve the constraint of the kernel here:
<(hxx) Pout (2))) (h (h, 2)),
(8)
for any h E (2( x R.
The irreducible group representation Pin for the corresponding feature type Lin (win w ). where Win E N and w E R, can be written Pin (7; 1) e e-((w,, ? 4) for any (7, SO(2) R; and the irreducible group representation Pout (7; t) = e e ? for the feature type lout = (w w where whut E N and W R, for any h (y, 50(2) x R.
To simplify the notation, we utilize R(7) to represent (z( E SO((), where y E (0,27), For any h = (7, () E SO(2) x R and any = (da, m. E R, we have h(h, a)) s(har-
(Rz(R()) dz), (4 0, 1J", d,))) according to Ex. Since S0(2) x R is product group, we can have x(x)) Ki (1))2(()) where
R1 (o
(9)
(2((7, - Pous (one((0))))))))))p) (10, (40, 1,",,,,,",d,)))
(10)
Now we solve the constraint for the kernel K1:
One can check that for any de 3 { {10, 0, 1|" -0, 0, 1"" R2(R()), dz) = when d2
[0, O, 1)² (2(()()), dz) R(7); and when d., = -,,0, 11 Rx(R(1)) = -((-)).
Therefore, we obtain the constraint that
KI (1(()))))) = twwe (1(r)
(11)
when d2 E 110,0.117. 10.0.
<1((),t)r) e s(wbur -wh 1
(12)
when d 00,0, 1)";;
(1((),t))) e- -iludue twudur Mi
(13)
when d, = -0, 0, 1]:;
The solution for Eq. 11 is that ki (x) [d(n), x), 0, 1J2 dz)) - (oao where alan2 is the 2-argument arctangent function, and ʃ is arbitrary function that maps (d(1) (((n),,,<<(()) ([0, 0, 1)"h, dd.))
) to the complex domain.
The solution for Eq. 12 that when what
win K1 C. where C is any constant
value; when what * Win and x = 77. K1 k1(x) = O; when what * wh. and * 77. Ki (x) = [d(n,, x))e ())eer())ettee i(wbue -wh, )nton2(() (.0)m.. (1.0000mm) m.,, f is an arbitrary function that maps d(x, m)
to the complex domain.
The solution for Eq. 13 is that when what = -w ss (x) C, where C' is any constant value; when what * with and -7, (x) 0; when what * -wi and x * -n. Ki (x) = f(((n, z))ee -(woue (atn)))tt222(0o " (. mz) where f is an arbitrary function that maps d(x,n) n) to the complex domain.
Next, we will solve the constraint for the kernel K2, which that n2((1, e eii(u... -00 "".,
(o)))))
When de [0.0. 1|", and w * w K2(2) = 0; When d. 10. 1TTH and w # -wf.. <2(x) 0; When d, 0, 11TT, and w w K2(2) f(d(z, m)), where f is arbitrary
20

---

function that maps d(z,1) 7) to the complex domain; When dN = - 10, 0. 1", and w = -w N2(x) f(d(x, 7)). where f is an arbitrary function that maps d(x, 11) to the complex domain; when de E 8² - [ 0, 1"", - - 0, 1]").
K2 x (x)x) = f(d(n, x), (0,0, d. d i1 - (10.000,)
(14)
where f is an arbitrary function that maps (d(n, 2). 2(10, d.))) to the complex domain: g(z) [0, 0, (ang [0. 0, 0)"), where æq represents the 3D coordinates of a point Q. This point Q can be defined as the intersection of x and v if x and 1 intersect. Alternatively. if æ and rj do not intersect, Q is determined as the intersection of 1) and the ray y. which is perpendieular
both æ and y, and intersects with both x and v). Refer to Figure 16 for a visual representation. One can easily check that g((7, t)x) + g(z), as shown in figure 16
which makes the solution valid.
x and are intersected, i.c., [0, 0, 1mm O,
11,0, X mz) diz)
g(x) [0. 0, [] x m,
11,0,
when [1,0, 0]d2 # O:
(, x mxx dxr) [0, 1,
g = [0. 0, []( x m,
when [1,0, 0]d- O:
When x and 17 are not intersected,
((x) = [0, 0, [d m, [,, x m [L,)01, + 10,1, x [0) 0, d,). (1,0,0dd,) 1,,,),
Regular Representation Here, we delve into the case where the output field type corresponds to the group representation of S0(2) x RR that p(T,,) t) = (7) @ (2 (f) for any (7, E SO(2) x I, where P2 is the regular representation. The regular representation of a group G is a linear representation that arises from the group action of G on itself by translation, that is when P2 : R - GL(V) is the regular representation, for any v E V. for any t, R. have (p2(t')o) et-e,, in other words, v E V can be viewed as function defined on R or an infinite dimensional vector. Then according to Ex. 6 the group (3) acting on the the field would be:
(c,f)(z) = (o(b(g - f(g ((gg((()))) x))e
(ha d.)) ff)) (000)=))_bss)
-M1(Rz ( - f(g (t,
for any & F. E and g E SE(3).
The points x on the ray x = (dg, mz) can be uniquely expressed as a = sy(x) + tx = d, x m, + tz, therefore for any x E R, any E R. f(x) can be expressed as feature attached to the point sb(r) + td, along the ray x.i..., f(z); = ff (s((x) + tdz, d.) as shown in figure 14 Therefore, we have fz d) = f((d, x x d)) (a- ()) dx one can easily check:
(LgJ') d) (ba 1,) ('(,,, ia + ty 1, Rg d) p: (R 1,
x. R. id) (15)
We should note the difference of the point æ along the ray and the independent point æ. as shown in the above equation, the point æ along the ray x (d, x x d) is denoted as (æ, d) instead of æ. Actually, it can be viewed as a homogeneous space of E(3) larger than RR3 whose elements are in R3 x S?, shown in figure 15
To summarize, the features attached the ray, whose type corresponds the regular representation of translation, can be considered as the features attached to the points along the ray. The action of S3) on features attached to these points can be expressed as shown in Eq. 15
The solution x also can be expressed as
<(x), = (x)))((())
(16)
21

---

f (dxx, m t
f' (x,dx
Figure 14: The feature attached to the ray, which corresponds to the regular representation of translation, can also be treated as the features attached to the points along the ray.
for any & IR, and their constraint is also the same as Eq. and Eq. 61 should be the same. We only need to solve K2:
10 As result, the solution for
<2((7, (((,,())z)) = whf [.....). d,Y (2(x)--
(17)
for any (y,t') E SO(2) x R. When d₂ ES [00,0, -0,0011]
<2(2), f(d(n, m), <(10, 0, 1]" d de))e"t (0000.1 d=)a(m) 5(t - g(x)); where / and g are the same function defined in 14 and 5(t) only when = 0.
(18)
when d E {00, 0, - 0, 1]"",,
ne(z); = 0 for any t R. Example 10. SE(3) equivariant convolution from to R Following the convolution from rays to points becomes:
So (2) - x(>2(x))++)) y) (hr (82(1) (2))) (y)dy.
(19)
where h is the twist function corresponding section 81 R. SF)) defined aforementioned. Pin is the group representation of SO()) x IR, corresponding to the feature type lin. 3 + SE is the section map defined in paper s2 (x: (1,
In this paper, we give analysis and solutions for kernel where the input the scalar field over the ray space, i.e.,ptn = 1, the trivial group representation, which is also the case of our application in reconstruction.
The convolution is equivariant if and only if
x(h2z) Pout (12 )x())
22

---

vector features(type-1) vector (eaturestttt
As shown in the figure, the point along the ray is distinet from the independent point. Moreover, we can observe that the type-1 feature the point along the ray differs from that of the independent point. Specifically, the type-1 feature for the point along the ray can be interpreted as vector on the plane orthogonal to the ray direction. In contrast, the type-1 feature for the independent point can be interpreted as three-dimensional vector.
Figure 15:
for any ha E (SO(3), where Pout is the group representation of SO(3) corresponding the feature type lout.
We can derive x(h2a) = Poue (ha))(a) analytically. For irreducible representation Pous and any I = (dz, mz) E R, if ||m-l|| = 0, x((x) cYloui (dz), where e an arbitrary constant and ylure the spherical harmonics and lout is the order (type) of output tensor corresponding to the representation Pout; With ||m jf O, x(x) becomes Pour (f) [[]m]]|=) where denotes the element in SO(3) and f IR + +1)x
(dz, 1 dz x
Similar to the convolution from rays to rays, we also can have the local support of the kernel. We set k(x) A 0 when |malll || < dg. otherwise <xx) One can easily check that doesn't break the equivariant constraint for the kernel.
Specifically, when we dg O, the neighborhood of the target points in the convolution only includes the rays from all views going through the point. Hence, we can simplify the convolution to flomt (z) faky.=)= lous (d 1y) (y)dy. This equation shows that for every point I, we can treat the ray y going through z with feature fi as point y', where y' x dsg(x)-' as shown in figure 17
B Equivariant 3D Reconstruction
B.1 Approximation of the Equivariant Convolution from Rays to Rays
In practical 3D reconstruction, we have multiple views instead of the whole light field. Although convolution above defined on the continuous ray space, the equivariance still strictly holds when the ray sampling (pixels from camera views) is the same up coordinate change. In this case, we will show how we adjust the equivariant convolution from rays to rays and approximate by an intra-view {E(()-comvolutton.
23

---

t1,t[0,0,1))xx
(I, t1,,(0,),11')x
(0.0, (00.0((0..(00011))
(00.g((..t(0000)))
t
x
y
(0,0,
(0,0,g(x))
Figure 16: Visualization of g(x). The left the case that the ray and the ray 1 are intersected, and the right is the case that the ray and the ray are not intersected. For the left, the point Q is the intersection of and 7) and Q = (0, 0, (x))); for the right, the point Q is the intersection of the line y and the ray 7), where is perpendicular to both y and x, and intersects with both and From the figure, in both cases, we can see that for any & IR, g t)z) + g(z). In general, actually have for any (7, t) E SO(2) x R. (((7, t)x) + g(x).
|| (2x) = 1
y
Y1
X
Y2
x
y
Y3
y
Figure 17: Interpreting rays ys as points y/
B.1.1 From Light Field Intra-view Convolution
Following Fig. I8 neighboring rays are composed two parts: a set of rays from the same view and another set of rays from different views. For one ray in view A. the neighboring rays from view B are in the neighborhood neighborhood in the view B would be very large.
the epipolar line of view B. When the two views are close, the
The kernel solution in Ex. 9 suggests that s(x)) is related to <(dz, 00, 1)7) and d((z, n)), where = (j0, 0, 1)T, [0, O, T as mentioned before. It would be memory- and time-consuming to memorize the two metrics beforehand or to compute the angles and distances on the fly. Practically, the light field is only sampled from few sparse viewpoints, which causes the relative angles of the rays in different views to be large and allows them be excluded from the kernel neighborhood; therefore, in our implementation, the ray neighborhood is composed of only rays in the same view.

---

B
A
do
Bo
X
Figure 18: For simplification, we show a situation of two views. For a ray x from view A, one part of the neighboring rays is from view A (the blue rays in the figure), Ma(x). For any ray y E Na(z), we have d(y. x) = O, and we require L(dy. d.) < So. The other part is from the other view B (the red rays in the figure). As illustrated figure 5 the neighboring rays always cross cylinder around æ; therefore, the neighboring rays from view B are the projection of the cylinder with radius r = dg in view B, that is, Nn is composed of the neighboring pixels of the epipolar line (the black dotted dash) corresponding to æ in view B. For any ray y in the projection of the cylinder, we have d(y, æ²) of do- Since we require that (d.y, dz) S Bo for any ray y E N ( (x)) is part of the projection the cylinder, denoted as the shaded yellow part in view B.
B.1.2 From Intra-view Light Field to Spherical Convolution
After showing that small kernel support in the case of sparse views affects only intra-view rays, can prove that an intra-view light-field convolution is equivalent to a spherical convolution when we constrain the feature field types over R.
We exploit the desired property that a feature defined on a ray is constant along the ray. This means that the translation part of the stabilizer group (translation along the ray) leaves the feature is. In math terms, the irreducible representation for the translation E is the identity, which means that the field function is scalar field for the translation group, with the formula (zr f()-lx)). We prove that, in this case, the intra-view convolution over rays is equivalent to the spherical convolution; please Sec.
B.1.3 From S0(3)- SE())coonoolution
While there is an established framework for spherical convolution using a Fourier transform 17 24 it is not applicable in our case because the boundaries of the constrained field of view cause explosion in the high frequencies of the spherical harmonics. We will make a compromise here and approximate the SO(3) convolution with an SE(2) convolution assumption that the field of view is small. One can see the rationale behind this approximation by keeping only the first order terms in the optical flow equation: the rotational term is only due Ω while the translational term is (-T, Pyy -T, y + 22)) with (P,, Rzy, Ω2) as the angular velocity. We provide justification using the formalism of the previous paragraphs in appendix Sec. E
22
the image plane by making the
B.2
Ray Fusion: Equivariant Convolution and Transformer
To reconstruct a 3D object, we use an implicit function known as the signed distance function (SDF) defined on R², As result, require equivariant model that transform features from rays to
25

---

points to obtain the SDF. This can be achieved using the equivariant convolution in Sec. 3.4.2 and transformer in Sec. 3.2 in the paper[3.5] which allows to transform features from the ray space to points in 3D space while maintaining equivariance.
B.2.1 Equivariant Convolution from Rays to Points
In this paper. we obtain the scalar feature field over rays after the SE(2)-equivariant CNNs. As illustrated in figure 3 we utilize the equivariant convolution (discussed in Sec. (34.2) to compute features for a query point by convolving over neighboring rays. Our experiments have shown that convolving only over rays that go through the point achieves the best results, and the equivariant kernel used for this convolution is provided in Ex Moreover, in the implementation, we can concatenate the input feature fi" with the depth embedding of the query point x. While this theoretically breaks the ideal equivariance for continuous light fields, it does not affect the practical equivariance, is rare for two cameras to share the same ray.
B.2.2 Equivariant Transformer from Rays to Points
For the third step, we introduce an equivariant transformer to alleviate the loss of expressivity due to the constrained kernel K in Eq. 19 Again, the attention key and values are generated from the feature attached to rays, while the query is generated from the feature attached points.
In the implementation, we apply a transformer over the rays going through the query point. We continue to the interpretation that treats any ray y passing through the point such that y' x =2( as shown in figure 17 Since becomes point y'. the ray feature fin
point y'
becomes the feature over IR attached to "points" y We can update the neighboring ray feature by directly concatenating the equivariant feature the point every ray feature before through SO equivariant MLP. The transformer in Eq. 4 would be converted to the transformer in (27] over R See appendix Sec. for details. The composition of the ray updating block and transformer block are shown in figure 22
C Proof of Equivalence of Intra-view Light Field Convolution and Spherical Convolution
The property that a feature defined on ray is constant along the ray means that the translation part of the stabilizer group (translation along the ray) leaves the feature as is. In math terms, the irreducible representation for the translation E is the identity, which means that the field function is a scalar field for the translation group, with the formula (z) = f(t 1 2). The equivariant condition on the kernel can then be simplified as
<((h, f)x) - Pour (h) ())x)(())))))))))) Pm (h, (h, dz)),
where h E (2) and E IR, Pin and Pout are irreducible representations for SO()). and h2 the twist function as shown in Ex 6 that h(g. x) = (ha (Rg. dz), (g i.e. the twist of the fiber introduced by action of S3(3) corresponding to the section map 8, of S0(3) in Ex. [5) and Ex. 6 Now we describe the relationship between the intra-view light-field convolution and the spherical convolution:
Proposition C.1. When the translation group acts feature f R - V as (L,f)z)) f(()xx) for any R, the equivariant intra-view light-field convolution: (u)ddy
(o = JVENUE)
becomes a spherical convolution:
four())
(((s (i) (dg)))
(
(20)
where (( ( Cz dy), Ox denotes the camera center that x goes through.
Sa is the section map of S0(3) as defined in appendix Ex. 5 and K' (sa(d))-)d ((s(dd))- (s(z)-1x.) (s
26

---

Proof. The SE(3) equivariant convolution over rays transforms into intra-view convolution when the neighboring lights are in the same view. Moreover, the simplified kernel constraint derived in the paper is that for any (h, 1) E S0(2) x R. and x = (dz, m.) E :
<((h, t)æ) Paur (2)x(z)) (h (h,
dz)),
where h. SO(3) x S + S2( is the twist function: ha (g. d) = sa(gd) fasgd) for any g E 50(3) and d E S
With the simplified kernel constraint, we can prove that intra-view light field convolution equivalent to spherical convolution:
plout (x)
J Jalyoee) y)p (y)dd
(21)
=
se, (g)dy
(22)
/ J,,E k(sa(da) ( s(x) (8o(dz) -d,))
Pin ( so (dy) (dy, Co dyjdd,
(23)
f K (s((((())) dy Pin hh (do sa (d()))
flhmn (dy)
(24)
In line 21 C2 the camera center that goes through.
The line 21 is equal to the line 22 because we assume that the irreducible representation for the translation R is the identity mentioned in the paper.
From line 22 to line 23 We can replace s(x) -' with
(sa(d))) dly (s(x) mm)) (d.( dd))
due to the facts that sa (d dy = dx(x)-ty and point s(x) -' is on the ray s(x) goes through Co. we can replace with (dy, Co x dy).
Since
From line 23
24 have pd (dy) pm (dy, Cz dy) because ea is fixed for any view.
Additional y, from Tine 23 to 24 we replace
((sa(d.))) dyy (s(x) xx) (sa(dn))'
with n' (sssadz)) d.y). It is because according to
<((h, t)x) = Paut (h)n(z) (h (h, dz)),
we have x(() = x(()) for any E R, where e is the identity element in SO(2); thus when
I ((--s(x)-__r)) 10, 1|TT, we have
x(sa(x)- dg s(x) s(x))((x_)))))) xe (sa(x)-'d))
= ((sa(z) dy (s(x) xe + t(0 0, 11²) x (ea(z))'d.)
(25)
= x((sa(x) dy 0)
(26)
= (1)
Line 25 is equal to 26 because s(x) l is always the axis, and thus s(x)-12. + + 0, 1]7
00
0,00 0
D Spherical Convolution Expressed in Gauge Equivariant Convolution Format
Group convolution is particular case of gauge equivariant convolution 64 where gauge equivariant means the equivariance with respect to the transformation of the section map (transformation of the
27

---

[0y000
hx-r
s [L,00,)
s(x - y)1
- y)2
[x)000
(0x000
Figure 19: Illustration of hu-ry- (1x 0, 0J and s(x) 0, 1, 0J (yellow) attached are tangent vectors We parallel transport (1,) 0, 0j and s0x)]0, 1, 0) along the geodesic (black dashed line) between a and The transported tangent vectors need to undergo a transformation /a-ay in S0(2) to align with the vectors sty 0, 0. and (0y) 1, 0,07 (green) attached to
tangent frame). the following paragraph we give the elaborated definition of gauge equivariance for the sphere.
Suppose S V is the field function corresponding the section choice 8a : 50(3).
we use -,,-,,; acting on f to denote the change of section map from 8a to s':: p(sa(z) (x))-s/ (x)) (x))) where p is the irreducible representation of SO(2) corresponding to the field type of f. The convolution
(1)
gauge equivariant when (CC,,+e,, f) -._ (f())
In this section, we show that the spherical convolution can be expressed in terms of the gauge equivariant convolution [16). which provides the convenience for us to verify the approximation of spherical convolution through the SE(2) convolution:
plout flout()) = / (((s(x)-1))pm) -ha (y)d
vEN(a)
where x' k'(hx) = Poue (h) for any E S0(2).
Since the focus of this section's discussion is spherical convolution, here s(x) to denote sa (x) for any x E S?
For any x, S2, (1,00 s(x) 1, 17 attached to are tangent vectors on we parallel transport (1, 0J and (0)) 1, 10 along the geodesic between æ and y and get two tangent vectors on y, denoted as s(x + y)1 and s(z + y)2 as shown in the figure 19] where the parallel transport along smooth curve a way to translate vector "parallelly" based on the affine connection, that is, for smooth curve 7 [0, y satisfies that V = O, where Im(7) = {r(t)]t [0, 1]} and V is the affine connection.
S the parallel transport X Im(r) T along the curve
s(x + y) and s(x -> y)2 need undergo a transformation in SO(2) to align with s(y) 0, Of and s(y) 1, 00 y shown in the figure 19 We denote the transformation h h
28

---

With the above notation, the spherical convolution can be expressed as:
flout (x) = SSENTO) () (m (v)dy
- x(s(z) pp ty-
Pin (oahhata)- Pmn (h(((()))))))))))))))))))))))))) pim s(y)) (y)
-/yEN(e) ((5(z))~)))) y)pin (
Pin Pin(hg-re) ((gn-)))-(hh((())))) (v)dy
= =SEENt)) (((sx)) y) (
(y)dy,
where - [0, 0, 1] the fixed origin point in S², and n² (x) = (h2))in(h2-m)) for any x E (()).
We can derive the equivariant condition that x' should satisfy:
K (hxx) k(hx))pin (hhnz--n))
Pout (h)r<1))) (b(h, 2))
Pin Pin(hha (hrr-sn)
Pout (h)xsz)) (ha- - Pin (h 2
Pout (h)r' r (2) pm))))
Therefore, the spherical convolution can be expressed as the gauge equivariant convolution format:
plaut (x) = SSE
(v)dy,
where k¹ k((hx) = Pous (())))(()))))))) (h) for any E S0(2).
E Converting Spherical Convolution to SE(2) Equivariant Convolution
As stated in Sec. D spherical convolution is gauge equivariant with respect to the choice of section map Sa. and the spherical convolution can be written gauge equivariant convolution. In this section, we use the gauge equivariant convolution to analyze the SE(2) equivariant convolution's s approximation of spherical convolution.
Since each view performs spherical convolution on its own, we only analyze the convolution for one view for the sake of simplicity. We use V to denote the space of the rays in the view, where V C $2 For any x E V, we can choose the section map 8a such that ha- = e, where E S that aligns with the optical axis as shown in the figure 20 Again, E S in this section.
s(x) to denote (x)) for any
When FOV is small, for any F, y E V. we can have such approximation: Kr-y Then the above gauge equivariant convolution in Sec. D can be approximated as
plaus (x) VEN(()
n' x((s(x)-1y) (y)dy
(=
K k'(t) (s(z))tdtt
(LEN())
where 7 = [0, 0, 1J", the fixed origin in $2, and n ((hhr) = Pont (h) (x) (z) for any E (0(2).
Additionally, as illustrated in figure 21 we have map from V to the projection points the picture plane represented as w V - IR. where w(o) defined as [0, O. When FOV is small, we have such approximation that for any h E SO(2), & E N()). and x E V.
w(s(z)t) x w(x) (()))
29

---

Figure 20: Section choice for every view
is because
(((()(() = =
+ (sso)( sing,
+ r( sinß:
cosB:
cosB3 (3r + Br)
and we have
sinB:
sin
li
cosB, cos3, (3x + Br)
=r(tan3,) + ((6)).
when 83 is small (FOV is small), the approximation stands.
Then ()o)(x) = K² (t) (s(x)t)dt can be approximately conducted the image plane:
Mlon (w(x))
(at))))
d(w(s(o)t)),
(27)
where for any E 32, (w()) f()), and for any E (()),, k" (w((s())) =())).
Since for any h S0( and any t E W())), (((()htt) = hw(s(o)t), we have for any h E SO(2)
and any 1 An))
x" (n((s(((()t))= = x"" = =x
(k)) (f)p (h) (t)) Poue (t)xx (h)
[=)())))))RR k²' (hp)
= (P) (p) (h).
30

---

[uto
Image plane
camera center
Figure 21: Illustration of projection map
Therefore, convolution 27 is exactly S2 equivariant convolution and can be used to approximate the spherical convolution.
In other words, we can intuitively approximate the equivariant convolution over the partial sphere using the S2( equivariant network when the distortion of the sphere and the tangent plane of the optical axis is modest.
Construction of Features in Equivariant Light Field Transformer
Noted that fout, . and f are features that composed of fields of different types, denoted as fgui = Bor SEm = = and f = e [ks Jq. and fu are constructed equivariant key features, query features, and value features, respectively, which composed of fields of different types as well.
We use fa = e.ft"' fg = and fo = @x to denote fur fa and fcr respectively. We con-
1xy Mime Ko By
by dime
struct the features fu.fq and fo through the equivariant kernels Kı, 0g and equivariant matrix Wg = Bg.. whi
din;
fk (x, U. l(m)
Σ
by tome (82(x)) "y)e E' (((=)(()=>><))))))))) (v): Kik
(28)
f fire
be logikine (82(2)) v)p V'm (h1 (s(())) (v):
Σ
Key
(29)
H (x, (2i) = Σ W bhy done plon (x),
(30)
by Mey and Ke Ay: should satisfy that:
where for any i.j. any h2 E SO(3), and any x E R rig
bxy V'n, (h2rr) = P2 (h2) Ly
Ius
Iag H'me (x)p1 Ei, (0T, x));
k
1 Myy (122)) = (h2) K
1 Mim Einy (h2 x)),
(x)p1
"Since here the homogeneous spaces of input and output might be different, so as the stabilizer groups, we and l' to denote the representations of different stabilizer groups
31

---

where hı x1) = $1 (122a) (2)1(2))) (22(h)))"h2s1(
is the twist function, and for any i,j and any h2 E SO(3).
whyychi satisfies that:
P2" (h2) Wi who W tins fimi Pi (It2).
(31)
When the group representation is irreducible representation, due to Schur's Lemma, we have Wg lk Lins = el when lG = Lin,, where e is an arbitrary real number, otherwise Wg ding = 0.
G Proof for Equivariance of Light Field Transformer
The equivariant light field transformer defined in the paper reads:
12
expl 12"), (. EvENTO exp( (f (,
Σ
VEN(()
f0 fi
(32)
general form.
According to [18], one can prove that fgr fo and fo are equivariant, that is, for any E SE(3),
x E 3 and y E R,
f/ (g- Cg (f2n)) = P2 16 (h2(g 199.019.3000 (.g.z)--)11 un (x, f2"):
ph: [g- x. y, 2
(sin)) at
= P2 le (h2(g ...-.,... x) -1) (e,
2. y, - (fin))
P2 Lug (h2 1 x) -1)=" (z.y. fir).
where L and L are group action of SE(3) on f and fin respectively..
The inner product (fg. /b) = C. y dtt is invariant due to the property of unitary representation, which results in the equivariance of the transformer.
H From SE(3) Equivariant Transformer in Ray Space to SE(3) Equivariant Transformer in Euclidean Space
In our implementation for the reconstruction task, the attention model is always only applied over the rays going through the points. We can continue to use the interpretation in the convolution from ray space to R3 in Ex. 10 that treats any ray passing through the point point y such that y' - r d=)(z)=' as shown in the figure 17
After we get the initial feature query points through equivariant convolution from to R3, we update the neighboring ray feature by directly concatenating the query point feature to every ray feature before through a (3()) cquivariant MLP as shown the figure 22 SO(3) equivariant MLP composed of an equivariant nonlinear layer and self-interaction layer as in the tensor field networks
55
Since y becomes point y/, and fi" is the feature over Rd attached to "points" y'. it becomes @ Then transformer 32 would be converted to the transformer in [27] over 3
Since here f is the fields over R2, use instead of l' the denotation
32

---

vi
Equi-MLP
f
X
f2
Equi-MLP
Equi-MLP
Kk Oy' x)
Ky Cy' x)
Wq
q
query
key
value
fq f2(x,f2")
fe ,1n))
ff(x,y', "
Multi-head Attention
fouu
Equi-MLP
The structure of ray updating and SE(3) transformer. We treat any ray going through point x as a point y' E R23 such that s/ - x d2y(z)-1y- The blue block indicates the ray feature update, and the pink block is the equivariant attention model. For the ray feature updating. the point feature (lavender) concatenated to every ray feature (light yellow, light blue, and light red) and goes through equivariant MLP. For the transformer, feature through the designed linear matrix Wg, designed kernels K2 and Ky, then apply multi-head attention to obtain the output point feature, which can subsequently be fed into the next ray feature updating and SE(3) transformer block.
Figure 22:
get the equivariant query, key, and value
pout (x)
E
VEN(#) Ever exp((sg(2, (a(.
fo si")),
(33)
where the subseript denotes the points which the feature is attached, i.e., and y'
The features fv. fo are constructed by the equivariant kernels Ki = Bj be, dins Ky = @j,ttin
luy dons.
th (2.y, fim)
Σ
Iky dins
(g' - (z);
A (z. fir
Σ
bey din; (3) x) (y)-
Re
where for any i, any 2y SO(3), and any TR3 Ki
Jayding and No
loyoong should satisfy that:
Iny time (hez)) = -22
lay (h2 (h2)
Lay ley (x) lim: (hez
Rg
Ly dsn; (1221) = P2 (h2)
lug by dons
(x)) Aon (11g
Kig
as stated 27
The feature f, is constructed in the same way as Equation 30
33

---

YL
Equi MLP
PSYx
X
f2
Equi-MLP
pa(x
PS
MLP
EquI-MLP
PS
MLP
KR Oy'
x)
Kpp Ey'
x)
W
W
query
key
value
query
key
value
Wxf2"((x)
futxy'' "h
f
(xP))
Wo
Multihead Attention
Multihead Attention
Equi-MLP
MLP
Figure 23: The comparison of the equivariant light field transformer and the conventional transformer. The left the equivariant light field transformer, and the right is the conventional transformer. In our light field transformer, the position encoding is not directly concatenated to the features because this is not equivariant. We first obtain the equivariant feature attached to the point by equivariant convolution over the rays. We then construct features St. and fo with derived designed kernels and Ky to keep them equivariant; we construct Jg by the designed equivariant linear layer WG. Since fu. fg. and fu are all equivariant, the inner product of fa and fa is invariant, which results in invariant attention weight. Therefore, the whole transformer is equivariant. In contrast, the conventional transformer concatenates the ray position encoding with the feature attached to the ray, uses the point position encoding for the query feature for the point, and applies multi-head attention using fu. fa- and fu. which are obtained by the Linear layer. We should note that W in the light field transformer designed to be equivariant, satisfying equation 31 which differs from the conventional linear map Wg in the conventional transformer. For the attention blocks after the first block, the query features of the point our model and the conventional model are both the output of the last attention block. The difference is that query feature keeps equivariant while the feature in the conventional transformer not.
Figure 22 shows the structures of ray feature update and SSF() equivariant transformer.
In figure 23 we compare the SE(3) equivariant transformer and the conventional transformer to illustrate how the equivariance is guaranteed in the equivariant transformer. In figure 24 we present the types of futures (E) equivariant attention head and conventional attention head, respectively. It indicates that geometric information is aggregated equivariantly in multi-head attention in the equivariant transformer.
I Equivariant Neural Rendering
Equivariant rendering relates equivariant 3D reconstruction, where focus multiple views instead of the entire light field. The equivariance property is maintained when the ray sampling is invariant up coordinate change.
I.1 Convolution from Rays to Rays
For neural rendering tasks, we query one ray and apply the convolution over the neighboring rays to obtain the feature attached to the target query ray. Similar to the reconstruction, we utilize kernel with local support. However, there a distinction in that for neural rendering, the kernel constrained to be nonzero only when d(æ, m) 0, while there
constraints L(dz, [0, 0, 1)T).
34

---

scalars
scalars
scalars (type-0)
(type-0)
(type-0)
(.)
vectors
vectors (type-1)
vectors (type-1)
(type-1)
tensors (type-l)
tensors (type-i)
tensors (type-i)
query
key
value
scalars (type-0)
(,:))
scalars (type-0)
scalars (type-0)
query
key
value
Figure 24: The comparison of multi-head attention modules in the equivariant light field transformer and in the conventional transformer. The figure above is the multi-head attention module in an equivariant light transformer, and the figure below is the conventional transformer. In the light field transformer. the query, key, and value features are composed of different types of features; they can be scalars, vectors, or higher-order tensors. The inner product should apply to the same type of features, and the type of feature determines the way of applying the inner product. In contrast, the feature in conventional transformer doesn't contain vectors and tensors, and the inner product conventional.
As result, the neighboring rays exclusively encompass the rays on the epipolar line for the target ray each source view, depicted Figure 25
The scalar field over rays serves as the input to the convolution. The output field type corresponds the regular representation of translation. This because this field type serves the input for the cross-attention module later on. If this field type were not utilized, the transformer would reach the entire neighboring set, leading to inferior performance compared applying the transformer individually for each point and then applying it over the points along the ray. A similar observation made [58). which states that the two-stage transformer outperforms the one-stage transformer. Using the field type corresponding to the regular representation of the translation as the input, the transformer from rays to rays is equivalent performing a transformer for each point, respectively, as explained in the following section.
In Eq. 18] already provide the solution of the kernel. We give detailed explanation in this and show that is equivalent to performing convolution from rays to rays with output field types corresponding irreducible representations, followed by applying Inverse Fourier Transform. Given that the input field is a scalar field, we have wh, 0 and w O. When considering an output field type of (wowts reg). where reg represents the regular representation of translation, the convolution can be expressed as follows:
(((ubuerra))
I (EN())
S
(dd( ((dta)""))fddd,..)=>>)
d=(w)=;,)))(t ((((((t-((())))))_()))) ",,"::"""""".."...""""""""""""""
Soleee 1n)=
(s(s)) ()(s())) y) ((dd)))))) ( d.)))))))
From the above equation, we can intuitively find that when the output field corresponds to the regular representation of the translation, the convolution happens at every point along the ray, respectively.
We treat fout ((g) as function over RR and for any w E R. we apply the Fourier Transform ((wbue
35

---

source view 1
target ray
source view 2
Figure 25: For simplification, show two views. For target query ray æ, the neighboring rays (denoted by red rays) are on the epipolar lines (denoted as yellow dotted dashes) for the target ray in each source view. For any ray y (xx), d(x,y) 0.
(leone Sout (t)e iut dt
F(w)
[J ((660)"))]((((,n333"""""," ([0,0, 17", do(s) fin twot g(a(u)--))==
"aldaa"~nnnfddnnn<)>))>0000000 da(m))-,))) (mals(o) 11) (y)
(,, - iwo(s(s) 2v), which is exactly the kernel corresponding to w = w and w = 0 as stated in Eq. 14 Therefore, we know that the field corresponding to the irreducible representation of the translation coefficients the field corresponding to the regular representation. We can first obtain the features of different irreducible representations attached the ray and subsequently apply the Inverse Fourier Transform to get the features for points along the ray,as shown in figure 26
where n/g
be treated as the Fourier
I.2 Cross-attention over Rays
The feature that generates the query in the transformer is the feature attached to the target ray, whose feature type corresponds the regular representation of the translation. The feature that generates the key and value in the transformer is attached to the neighboring rays in the source view, whose
36

---

Figure 26:
The features for points along the ray (the field type corresponds to the regular represen- tation) can be obtained by the Inverse Fourier Transform of features attached to the ray, where the types of feature fields correspond to the irreducible representation of the translation.
feature type corresponds to the scalar field. The output the feature attached to the target ray, whose feature type corresponds to the regular representation. Therefore, the transformer becomes:
exp(((f((a fir, (. (So(1, (( EvEN(e) expl ((S((, (E"(( v, (1)))))
( (x))e
Σ MEN(a)
(34)
where
(fal,,,, fir) = (x)(s2(=))\))),fff (fo(z, =
(fg(z, fit CC(f""
In the equations above, K& and are the kernels derived in Ex. Eq. 16 C' is the equivariant weight matrix satisfying Eq. 31
The expression above indicates that the feature types of both key and value correspond to the regular representation translation, as well as the feature type of the query. Moreover, the transformer operates on each point along the ray independently. It should be noted that the features (fa) )- (J,).. (f=)) and (f²²), r may have multiple channels and may consist of different types of features corresponding to various representations of S0(2). The inner product (,, can only happen in the field type of the same representation of SO(2). This allows for the implementation of multi-head attention module, where each head can attend to specific type of feature and multiple channels.
37

---

1.3 Self-attention over Points Along the Ray
After the cross-attention over rays, we get the features of the points along the ray, i.e., the feature attached to the ray corresponding to the regular representation of translation. SFF(3 acts the feature f' attached to the point along the ray as mentioned in Eg 15
(z) d) = pz ( ,
(1 'x, Rg-
where P1 is the group representation of (2)
We will apply the self-attention model to these points along the same ray. For two points 2t and D2 on the same ray (d, æ x d), one can observe that for the same type of feature, ((C, (x1,
, d), (Z, d)) = (f' Ry-: f' Rg- which makes atten-
tion weight invariant, the transformer could be formulated as:
expl (rpp( x)), farSS ,x, xr,]))
fou =
Σ
fe fmm) (35)
Ey on the same as ezp (fmm
(f x²), fu(fin x, v)
on the same ray as
where
fl y, fth = cx(((xx (()((x))){{((m)))
f. fmm cu(e(z, (f
fe (fm
and æ and the points along the same ray with direction d, denote (xr, d) and y as (y, d), d(x; y) is the signed distance (d, y [y-x) x), CR,Cy are arbitrary functions that take signed distance the input and output complex values and e2 is arbitrary constant complex. It should be noted that the features fE. fg. fu, and fr may have multiple channels and consist of different types of features corresponding to various representations of SO(2). the inner product (:, can only happen in the same type of field. This allows for implementing multi-head attention module, where each head can attend to specific feature type and multiple channels. Here, S/ denotes the type- feature in feature fE. ft. represents the type-/ feature in feature for fy denotes the type- feature in feature fi, and (f represents the type- | feature in feature fan
Note that this transformer architecture also follows the general format of the transformer in Eq. We only simplify the kernel KR, Ky to be trivial equivariant kernels.
To obtain a scalar feature density for each point, the feature output of each point can be fed through an equivariant MLP, which includes equivariant linear layers and gated/norm nonlinear layers. These layers are similar to the ones used in 62 and 63).
J
3D Reconstruction Experiment
J.1 Generation of the Dataset
The dataset is obtained by fixing the orientation of the object well as the eight camera orientations. With the object orientation fixed, we can independently rotate each camera around its optical axis by random angle in a uniform distribution of (- rotate every camera randomly by any rotation in SO(3) while fixing the object. The equivariance stands with the content unchanged. Therefore, in practice, we require that the object projection after the rotation does not have new parts of the object. We satisfy this assumption by forcing the camera to fixate on a new random point inside small neighborhood and subsequently rotate each camera around its optical axis with the uniformly random angle in (-T, t). We generate the Y dataset by rotating the object only with azimuthal rotations while keeping the camera orientations the same. The SO(3) dataset is generated by rotating the object with random rotation in SO(3) with the orientations of cameras unchanged, which will potentially result in new image content. Equivariance is not theoretically guaranteed in this setup, but we still want to test the performance of our method.
obtain the dataset. For the R dataset, we
38

---

DeapSpeed Flops Profiler
Profile Summary step 18:
Notations:
data parellel size (dp, size), model parallel size(mp_size),
parancters (panass),
auttiply-accumiate openstiors
number of floating paint operations (flops), floating-point operations per secend (FLOPS),
latuney (forward propagation latancy), bed latency (backward propagation latency),
step (veights update latency), iter latency (sun of fidd, bid and step latency)
parass gou:
10.82 M
parass model parans per ap
Fiel MMCx dPU:
NiCs
flops per
105.29
flops of model fid flops mp_size:
89
fid latency::
42.56 md
Aggregated Profile per FLOPS. EPU Fad
lateney:
2.47 GFLOPS
Top nodules in terms of parans, MMCs fwd latency at different model depthe;
dopth 8:
parans
('RoNct inpoint_ Nobn': '18.82 H*) Thetet_inpaing Nobnn" MACs
MACa
fail Tatency
"Retet_ Enpwint Nobe' 142.56 86 ns')
depth 1:
Modulellsst' '10.82 H')
NACs
"Mductttist's 1 PACs")
latency
ModuCallst'
depth 2:
parans
Re
HACs
'Sequential'; "0 MACs')
latancy
('ReLLaye":: '39.21 ns")
depth 3:
рагans
"BauicSluck': '9.89 H*)
HACs
R2conn''s 'o MACs
fwd latency
"Bosicalock" : '39. m=')
perais
'R2Conv': : 19.72 N')
"Blockssuusistsammiiii "R2conv'; '34.7 m=')
Wics
fid lntency
dopth S:
раrans
('R2conV": '178.64 k*)
("Singg Saingggeeloddssssssssaas...nn MACx")
fid latency
WKttem" 12.00 mx')
depth 6:
BBBacchoorddd': '632")
NACS
(sSingeelloceessseeeaaaas MACs')
Zatency
"BBucchleradd":
Figure 27: The number parameters and FLOPs of SE(() equivariant CNNs. We set batch size as one to calculate number of FLOOPs.
J.2 Implementation Details
We use SE(2) equivariant CNNs to approximate the equivariant convolution over the rays. We use the same ResNet backbone as implemented in [30] that is equivariant the finite group Cs, which we find achieves the best result compared with other SEE(2) equivariant CNNs. We use similar pyramid structure as [69 that concatenates the output feature of every block. Since every hidden feature is the regular representation, in the final layer we use 1 x (?(())=-quivariann convolutional layers to transfer the hidden representation scalar type.
For the fusion from the ray space to the point space model, we use one layer of convolution and three combined blocks of updating ray features and SE(3) transformers. For the equivariant SE(3) we only use the scalar feature and the vector (type-1) feature in the hidden layer. The kernel matrix includes the spherical harmonics of degrees and 1. We also concatenate every output point feature of every block as in the 2D backbone. Since the output feature of every block includes the vector feature, we transfer it to the scalar feature through one vector neuron layer and the inner vector product. We use the same weighted SDF loss as in 69 during training, which applies both uniform and near-surface sampling. We report the number of parameters and foating-poiin operations (FLOPs) of our 2D backbone and light fusion networks in Fig. 27 and Fig. 28 respectively.
39

---

DeepSpeed Flops Profiler
Profile Summary step 10:
Notations:
data parallel size (dp_size), model parallel (ieemm_sizee)
number parameters (params), number multiply multiply-accumllte accumulate operations
number of floating point operations (flops), floating point operat per second (FLOPS),
fwdd latency (forward propagation latency), bwd latency (backward propagation latency),
step (weights update latency), iter latency (sum of fud, bed and step latency)
params per gpu:
1.53 N
params model params per GPU up_sips si_size:
1.53 N
fied MACS per GPU:
19.57 GMACS
Fwdd flops per GPU:
39.15
Fidd flops of model fwd flops per GPU mp_ sp_size:
39.15
fwel latency:
39.67 ms
fwdd FLOPS per GPU fwd flops per GPU / fwd latency:
986.86 GOPS
Aggregated Profile per GPU
Top modules in terms of parans, MACS or fud latency at different model depths:
depth 8:
params
{"Fusion_ od"; '1.53 M'}
MACS
{'Fusion od"; '19.57 GMACs") }}
fwd latency
("Fusion_ae edd". '39.67 ms")
depth 1:
params
('sequential' : '895.9 k')
MACS
('Hodulet.st':: '15.54 GMACS' '}
fwd latency ('HoduleList': '32.23 ms "
depth 2:
params
"Sequential' : 122918 k')
MACs
'Sequential ial : '15.54 GMACs') )
fwdd latency
('Sequential' : '32.23 ms
depth 3:
params
('Linear':
'752.06 k')
MACS
'8. GMACS")
fwd latency
(" ('VII WwlLinearLeayRReUU
'23.61 ms'!
Figure 28: The number of parameters and FLOPPs of the ray fusion model, which composed of convolution from rays to points and transformer from rays to points. We batch size as one to calculate the number of FLOPs.
J.3 Discussion of Results
There is still performance gap between 1/1 and I/Z. Although SE(2) equivariant networks are theoretically strictly equivariant, the error in practice is introduced by the finite sampling of the image and the pooling layers. Additionally, we use the ResNet that is equivariant to Cg approximation of (2()), which causes this gap but increases the whole pipeline performance in the other tasks. There is no significant difference between I/Z and 1/ R, which shows that approximating the spherical field convolution SE()) equivariant convolution is reasonable in practice.
J.4 Qualitative Results
Figure
29
shows a qualitative result for the chair category. There more qualitative results shown
in Fig. 37 Fig. 38
and Fig. 39
J.5 Ablation Study
First, we replace the SE(2) CNNs backbone with the conventional CNNs to test the effectiveness of S (2) CNNs. Secondly, we remove the equivariant convolution/ransfrrner part and use trivial aggregation (max-pooling) combined with MLP. Finally, we run an equivariant convolution and transformer without using the type-1 (vector) feature while keeping the number of parameters similar to our model.
40

---

FvOR
DISN
Ours
GT
N/
Y/Y
I/Z
R
I/R
R/R
S0(3)
V/SO3)
S0(3)/S0(3)
Figure 29: Qualitative results for equivariant reconstruction. Left: input views; Right: reconstruction meshes of different models and ground truth meshes. The captions below the meshes show how the model is trained and tested, explained in the text.
Table 3 summarizes the result on the chair category, which illustrates that in the T/T and Y/Y trials, SE(2) CNN is less expressive than traditional CNN, but it contributes to the equivariance of our model looking at the results of / 1/2 1 RR, and Y S//SO(3) Equivariant ray convolution and transformer improve both the reconstruction performance and the equivariance outcome. We also compare the ray convolution and transformer with the models operating only on scalar features without vector drop in performance in every setting, proving the value of taking ray
features, and again, directions into account.
We also compare to baseline where the ray difference information is encoded in the feature explicitly. Most models that encode ray directions aim at rendering. like IBRnet. Here, we modified IBRnet (Fig.2 of IBRnet paper) query 3D points only for their SDF value instead of querying all densities along the ray that would be necessary for rendering. We replaced the ray direction differences with the ray directions themselves because
query point and not query ray. We report in table IoU result for Y/Y and Y/SO(3) (where Y augmentation only along the vertical axis) for two models IBRNet with conventional CNNs as 2D backbone and IBRNet with SE(()--quivariant CNNs as 2D backbone. For the SO(3) setting. we rotate the whole 8 cameras with the same rotation, which is equivalent rotating the object with the inverse rotation, and frame to encode the ray information.
4
the object canonical
The baseline is not equivariant: It explicitly uses the ray directions inputs MLPs. Ray directions or their differences change when the coordinate system is transformed, breaking, thus, equivariance. that our model is more resilient object rotations. We enhance equivariance
41

---

Method
w/o SE(2)
w/o conv& trans
w/o type-1
Full model
I/I
0.767/0.079
0.695/0.105
0.722/0.093
0.731/0.090
I/Z
0,430/0.234
0.533/0. 175
0.553/0.
0.631/0.130
I/R
0.417/0.249
0,442/0.241
0.466/0.203
0.592/0.137
R/R
0.672/0. 112
0.658/0. 122
0.682/0.109
0.689/0.
Y/Y
0.731/0.090
0.644/0. 124
0.677/0.111
0.698/0. 102
Y/SO(3)
0,467/0.0.217
0.534/0. 170
0,569/0. 163
0.589/0.14
SO(3)S0(3)
0.655/0.120
0.616/0.142
0.636/0.130
0.674/0.113
Table 3: Ablation: w/o (2) means replacing (2) equivariant network with conventional: w/o
ray conv& trans denotes the model where replace the light field convolution and the light field equivariant transformer with max- max-pooling; w/o type-1 means using only scalar features convolution and transformers.
Method
Y/Y
Y/SO(3)
SO(3)/S0(3)
IBRNet [61] w/o SE(2)
0.689
0.432
0.611
IBRNet 61l w/SE(2)
0.652
0.501
0.619
Ours
0.698
0.598
0.674
Table 4: Comparison of our model and a baseline which encodes the ray information explicitly. IBRNet w/o SE(2) is the modified IBRNet with conventional CNN backbone, IBRNet w/SE(2) is the model where we replace the conventional CNN backbone with the SE(2) equivariant CNN.
by using SE(2) equivariant modeling, and our model outperforms the baseline in the Y/Y setting. We believe that the transformer in our model is responsible for the performance improvement.
K
Neural Rendering Experiment
K.1
Experiment Settings Discussion
Two experiment settings illustrate model's equivariance: T/T and T/SO(3). I/I is the canonical setting, where we train and test the model in the same canonical frame defined in the dataset. I/SO(3) is that test the model trained in the canonical frame under arbitrary rotated coordinate frames, which means that all the camera poses in one scene are transformed by the same rotation without changing their relative camera poses and relative poses between the camera and the scene, which doesn't change the content of the multiple views. The reason we don't 't apply translation to the cameras that there exists depth range for points sampling in the model and the comparing baseline [61] which effectively mitigates the impact of translation.
We should note that the SO(3) setting in this experiment setting differs from R and SO(3) settings in reconstruction. R changes the relative pose the cameras, and each image is transformed due to the rotation of each camera without altering the content, i.e., the sampling of the light field is nearly unchanged. The R setting aims to demonstrate that replacing the conventional method with ray-based convolution can get rid of the canonical frame for each view.
SO(3) reconstruction is to rotate the object pose randomly without changing the pose of the camera, which is equivalent to transforming the cameras by the inverse rotation but fixing the object, resulting in changes in the relative poses between the camera and the object, the content of the image and, therefore, the sampling of the light field. This setting shows that even for non-theoretically cquivariant cases, our model in reconstruction still demonstrates robustness.
In the rendering experiment using the SO(3) setting, each image itself is not transformed, unlike the R setting the reconstruction. The content of the images remains unchanged, including the light field sampling, unlike the SO(3) setting in the reconstruction. Since each image is not transformed, even the conventional 2D convolution applied to the image, the scalar feature attached to ray is not altered, and the light feature field sampling remains the same up to the transform of the coordinate frame. This setting was used demonstrate that our model is S -equiviaiaa when the input is the scalar light feature field.
42

---

DeepSpeed Flops Profiler
Profile Sunnary at step 18:
data parallel (ip_rize), nodel parallel size(mp_size),
number of parameters (perans), number of floating- operations (flops), flosting-point operstions par second (FLOPS),
multiply-accumuleee operations (MACs),
fud latency (forward propagation latency), bud latency (backward prepagation latency),
step (wel ihtt update latency), iter latency (sum of Fai, bud and step latency)
parans gru:
21.96
parans of model parans per GPU mp size:
21.96 k
fud MACS per GPU:
9.1 NMACs
fud flops per GPU:
20.29
fiid flops model fud flops per GPU np_size:
20.29
latancy:
8. ms
per GPU flops per latency:
GFLOPS
Aggregated Profile GPU
1 modules of parans, MACS fud latency different nodel depths:
depth 6:
parans
"'RNNet_l '21.96 k*) {' (BIRNtet_]) 1f" 19.1 MAcs' }
MACS
Fid latency
(' ['IRNNet_FF If";
18.81 ms
depth 1:
paran:
"equential''
'13.75 k"}
MACS
''equuentil"'
18.47 NMACs") "}
Fwd latency ''equuntiaa'' 12.84 ns")
parans
('Linear': '14.33 k'}
('Linear': '8.51 MRACS
Fud lateney
("ELU":
"2.56 ms')
Figure 30: The number of parameters and FLOPs of the model, which takes the scalar feature attached to rays input and predicts the color and density for points along the target ray. The calculation of FLOPs is performed for single-pixel rendering with 10 source views.
Figure 31:
In terms qualitative results for rendering, we compare the performance of IBRNet and model in both the given canonical frame (denoted "IBRNet(I)' and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNet(SO(3)) and "Ours(SO(3))" respectively). Our model performs comparably IBRNet in the canonical setting. However, IBRNet experiences performance drop in the rotated frame, while our model remains robust to the rotation.
K.2 Implementation Details
As described in the paper, we use a similar architecture as 61, where we replace the aggregation
of view features by equivariant convolution and equivariant transformer over rays. equivariant convolution, the input is scalar feature field over rays, which means that wi = 0 and w 0: for the output field, we use regular representation of translation as described in Sec. 3.3 and we use wi 0, 2 ..... 27 for group representation of SO(2), each field type has 4 channels. In
equivariant transformer over rays, we update the key and value before going to the attention module in the experiment; the specific operation is that we concatenate key fx and query fq. concatenate
43

---

Ground Truth
IBRNet())
(BR3) Ours(I) Ours(SO(3))
Figure 32: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and rotated frame (denoted as "IBRNe((SO(3))" and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
Ground Truth
IBRNet())
(BR)) Ours(I) Ours(SO(3))
Figure 33: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNe((SO(3)) and "Ours(SO(3)) respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
fo and query fq. and then we feed the concatenated key and value into two equivariant MLPs (equivariant linear layers and gated/norm nonlinear layers, similar to the ones used in (62) to get the newly updated key and updated value, which will be fed into attention module. In line with [61], our approach does not involve generating features for the color of every point. In our implementation, directly multiply the attention weights obtained from the softmax operator in the transformer with the corresponding colors in each view perform color regression.
We replace the ray transformer with the equivariant transformer over the points along the ray; the input features comprise the feature types corresponding to the group representations Win .... 2 for (2).). Each feature type has 4 channels; the output comprises the same feature type, and each type has 2 channels. We will first convert the feature into scalar feature by equivariant MLP (equivariant linear layers and gated/norm nonlinear layers, similar to the ones used in (62]. and then feed it into conventional MLP get the density. We report in Fig. the number of parameters and floating-pon point operations (FLOPs) the model composed of the convolution and transformers.
K.3
Qualitative Results
Fig. Fig.
Fig. 35 and Fig. 36 show the qualitative results Real-Forward- Facing 41 and Realistic Synthetic 360
32
33 Fig. 34 Fig.
49 data. Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences performance drop in the rotated frame, while our model remains robust rotation.
44

---

Ground Truth
IBRNet() (BRNe((SO(3)) Ours(I) Ours(SO(3))
Figure 34: In terms qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I) and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNe((SO(3)" and "Ours(SO(3))) respectively). Our model performs comparably IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
Figure 35: In terms qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNe((SO(3)" and "Ours(S0(3)) respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
Negue
Figure 36: In terms of qualitative results for rendering, compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and rotated frame (denoted as "IBRNe((SO(3))) and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
45

---

FvOR
DISN
Ours
GT
1/Z
Z
I/R
R
R/R
Y/Y
Y3S0(3)
SO(3)
(0(3)/0((3)
Figure 37: Qualitative Result for the chair. Left: input views; Right: reconstruction meshes of different models. The captions below the meshes show how the model is trained and tested.
46

---

FVOR
DISN
Ours
GT
I/Z
R/R
Y/Y
YISO(3)
S0(3)
50(3)/50(3)
Figure 38: Qualitative Result for the Left: input views; Right: reconstruction meshes of different models. The captions below the meshes show how the model is trained and tested.
47

---

FVOR
DISN
Ours
GT
I/1
1/Z
Z
I/R
R
R/R
Y/Y
Y
YIS0(3)
S0(3)
S0(3)/S0(3)
Figure 39: Qualitative Result for the car. Left: input views; Right: reconstruction meshes different
models. The captions below the meshes show how the model is trained and tested.
48


Reviews:
[{"review_id": "hUFAP95hmy", "reviewer": "Reviewer_MuuL", "summary": "The paper proposes a novel approach for reconstruction from multiple views using equivariant shape priors. The paper proposes the $SE(3)$-equivariant generalized convolution as the fundamental operation on a light field whose values may be radiance or features. Input ray features and producing output ray features and point features using two different $SE(3)$-equivariant convolutions. The paper demonstrates $SE(3)$-equivariance by obtaining robust results in roto-translated datasets without performing transformation augmentation.\n\nTo say it upfront, I do not have background in Equivariant Networks and consider the paper outside my area of expertise. I find it difficult to understand the technical details of the paper. As such, I can only provide some high-level feedback to the paper and hope the AC and other reviewers can provide more detailed evaluations.", "questions": "Please see Weaknesses.", "limitations": "Please see Weaknesses.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. The experiment results show that the provided models can outperform previous methods on various tasks.\n3. This novel general equivariant representation framework for light fields can inspire further work on 3D vision and graphics tasks.\n", "weaknesses": "1. The figures in this paper could be redesigned; some of them (Figure 4, 5) are so small that it is difficult to read the text and symbols they contain. In addition, the effect shown in Figure 9 is not intuitive.\n2. The paper demonstrated good reconstruction results on Shapenet. Can the method proposed in this paper reconstruct surfaces on real datasets such as SparseNeuS.\n3. About neural rendering experiments, the paper only shows results comparing with IBRNet, which doesn't seem to be the latest method. Can the authors provide the results of more experiments? (a. Local Implicit Ray Function for Generalizable Radiance Field Representation  b. Learning to Render Novel Views from Wide-Baseline Stereo Pairs c. Neural Rays for Occlusion-aware Image-based Rendering)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. “The figures in this paper could be redesigned; some of them (Figure 4, 5) are so small that it is difficult to read the text and symbols they contain. In addition, the effect shown in Figure 9 is not intuitive.”**\n\nWe will improve the design of Figures 4 and 5, and make them larger while saving space by relocating theoretical content from the paper to appendices. We hope that this will make the figures legible and increase the understanding of the reader. \n\n In reference to Figure 9, in the top row of images, a noticeable distinction arises between the images generated by IBRNet(I) and IBRNet(SO(3)) as they contain a non-existent black area within the pillar. Contrarily, our results exhibit no such black area. Furthermore, when comparing IBRNet(I) and IBRNet(SO(3)), the latter presents several blurred transverse lines, whereas our method remains robust against rotations. \n\nFor the second row, our performance closely matches that of IBRNet in nonrotated scenarios. However, when the reference frame is rotated, our method maintains its quality, while IBRNet's output becomes increasingly blurred, particularly in the bottom right corner. We are committed to enlarging the figures to enhance their visual clarity and intuitiveness.\n\nBecause the rebuttal pdf is limited to one page the figures are still small there, but they will be shown in legible size in the revised paper.\n\n**2.“The paper demonstrated good reconstruction results on Shapenet. Can the method proposed in this paper reconstruct surfaces on real datasets such as SparseNeuS.”**\n\nWe invested considerable effort in the theoretical formulation of equivariant convolution and attention in a light field, and we were planning to conduct experiments on real datasets in a subsequent paper. We believe that the community will benefit from our exposition of a novel convolution and attention instead of squeezing it into a few paragraphs so that we make the real experiments fit. While we recognize the significance of real-world reconstruction, ShapeNet has been an established dataset for surface reconstruction in hundreds of papers in computer vision.\n\n**3. “Regarding neural rendering experiments, the paper only shows results compared with IBRNet, which doesn't seem to be the latest method. Can the authors provide the results of more experiments? (a. Local Implicit Ray Function for Generalizable Radiance Field Representation b. Learning to Render Novel Views from Wide-Baseline Stereo Pairs c. Neural Rays for Occlusion-aware Image-based Rendering)”**\n\nWe have selected IBRNet as our baseline due to its status as a classic generalized NeRF method. This choice is rooted in the fact that IBRNet employs the conventional ray transformer, a technique also referenced in the pertinent literature. Our architecture builds upon IBRNet's foundation, with modifications limited to the replacement of the aggregation mechanism using equivariant convolution and the conventional transformer substituted by an equivariant transformer. Therefore, utilizing IBRNet as a baseline is justifiable to showcase the resilience of our proposed fundamental operations (convolution, transformer).\n\nWe have added a comparative analysis with NeuRay (Neural Rays for Occlusion-aware Image-based Rendering) on the Real Forward-facing LLFF dataset. Referencing Table 2 and Figure 2 in the provided PDF, our observations reveal that our method demonstrates comparable performance to NeuRay in the I/I setting. In the I/SO(3) configuration, NeuRay experiences a performance decline and inconsistency, whereas our method remains robust against rotations.\n\nUnfortunately, we were unable to locate the publicly available code for the \"Local Implicit Ray Function for Generalizable Radiance Field Representation.\" This absence of code renders a direct comparison unfair. Furthermore, \"Learning to Render Novel Views from Wide-Baseline Stereo Pairs,\" a method centered around rendering from two views, is not optimal for our approach, as our method assumes ample information extraction through light field sampling. Nevertheless, it would be worth considering whether expanding the ray neighborhood in our technique could offset the sparse light field sampling. We will cite these valuable references.\n\n\n\n\n\n"}, {"review_id": "pOrgxM9Ox0", "reviewer": "Reviewer_sFoS", "summary": "This paper proposes to study the geometric priors for 3D reconstruction and neural rendering with a novel perspective of multi-view equivariance. A theoretically-sound definition of the ray neighborhoods with SE(3) is obtained with the theoretical deduction of characterizing ray space with group theory. Then, the SE(3)-equivariant convolution and cross-attention operators are presented. In the experiments, the authors evaluate the proposed mathematical framework on two tasks, multi-view 3D object reconstruction and neural rendering. For the multi-view 3D object reconstruction, the presented SE(3)-equivariant network outperforms Fvor and DISN in most settings for chars, airplanes, and cars on the ShapeNet dataset. For neural rendering, the proposed method obtains better results on the I/SO(3) setting to justify its design. Overall, I think this paper brings us some new messages to understand the ray space by using group theory. ", "questions": "I am confused about the sentence \"given only the relative poses of the cameras, we show how to learn priors...\". Why do authors need to highlight the relative poses in the paper? It is strange for me.", "limitations": "The authors have discussed their limitations and broader impacts.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper presents a theoretical perspective to understand the ray spaces for learning geometric priors for multi-view 3D reconstruction. The thorough theoretical analysis presented in this paper is new to me.\n\n2. Based on the theoretical findings, the authors presented equivariant convolution and transformer in ray space and justified their design on two tasks with positive results obtained. \n\n3. This paper is well written with a detailed appendix to understand their theory. ", "weaknesses": "1. The core of this paper is studying the geometric relationship between rays to define the ray neighborhoods, and then, convolution and attention can be induced in the ray space. If I understand correctly, what's the relationship between the point correspondences and the neighbors in the ray space? For example, are the corresponding rays near or the same in the ray space if we have a pair of keypoint matches $x$ and $x'$ for a pair of input images? \n\n2. As for the generalized neural rendering task, the authors stated that their model queries a target ray and obtains neighboring rays\nfrom source views in the first. To my knowledge, such operations can be done by following the epipolar geometry. Thus I am curious about the difference between the operation used in this paper and the alternative solutions using epipolar geometry.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. “The core of this paper is studying the geometric relationship between rays to define the ray neighborhoods, and then, convolution and attention can be induced in the ray space. If I understand correctly, what's the relationship between the point correspondences and the neighbors in the ray space? For example, are the corresponding rays near or the same in the ray space if we have a pair of keypoint matches x and x’  for a pair of input images?”**\n\nWe really appreciate this question. Ultimately, it can be argued that every IBR or reconstruction method inherently boils down to addressing a correspondence challenge.\n    \nWhen a pair of input images share matching keypoints, denoted as x and x', these keypoints represent neighboring rays. Figure 3 of the paper illustrates that when considering a specific ray x, its neighboring rays traverse within a confined cylinder. This cylinder utilizes x as its central axis and the maximum neighboring distance d as its radius. In the context of a two-view setup, as depicted in Figure 9 in the appendix, for a given ray x within view A, the neighboring rays can be categorized into two distinct groups. The first group encompasses rays originating from view A, where the angle between these rays and x is smaller than the specified threshold for the neighboring maximum angle. The second group consists of rays emanating from view B, located in close proximity to the epipolar line associated with x in view B. This proximity is attributed to the fact that the neighboring region corresponds to the projection of the confined cylinder onto view B.\n\nIn this configuration, keypoint correspondence entails that subsequent to convolution and attention, querying the density of the keypoint would yield a significantly elevated value.\n\n**2. “As for the generalized neural rendering task, the authors stated that their model queries a target ray and obtains neighboring rays from source views in the first. To my knowledge, such operations can be done by following the epipolar geometry. Thus I am curious about the difference between the operation used in this paper and the alternative solutions using epipolar geometry.”**\n\nThis question is also spot on! \nAs outlined in the first question and depicted in Figure 9 of the appendix, the neighboring rays within the source view of the target ray are situated within the vicinity of the epipolar line associated with the target rays in that source view. When we establish the maximum neighboring distance as 0 and set the angle threshold to $-\\pi$, the resulting neighboring rays precisely align with the rays located on the epipolar line.\n\n**3.  “I am confused about the sentence \"given only the relative poses of the cameras, we show how to learn priors...\". Why do authors need to highlight the relative poses in the paper? It is strange for me.”**\n\nCertainly, some clarification is needed to contextualize this concept within classical multi-view geometry. In approaches such as classical space carving or plane-sweep reconstruction, a crucial step involves the selection of a reference frame for voxels or sweeping planes, followed by an optimization process. In those cases, there is no question of equivariance, and many times we choose as reference frame one of the camera frames. \n\nHowever in learning methods that use a prior (encoded in the weights of the network), this prior depends on the choice of the reference frame, and by default generalized reconstruction or rendering methods are not equivariant to the choice of this frame. In calibration cases, where only relative poses are given by an SfM method (instead of a world coordinate system by standard calibration) choosing one of the camera frames as a reference frame will alter the inference of the network if the network is not equivariant. Our method guarantees that this will not happen.\n\n\n\n\n\n"}, {"review_id": "Z7embqnNsr", "reviewer": "Reviewer_gW8w", "summary": "This paper introduces a method for leveraging geometric priors in 3D reconstruction and novel view rendering when the input views are insufficient. The authors propose learning priors from 2D images using a 2D canonical frame and a 3D canonical frame. They achieve coordinate frame equivariance by introducing an SE(3)-equivariant convolution and transformer in the 3D ray space. The paper demonstrates the efficacy of their approach in tasks such as 3D reconstruction and neural rendering using multiple views, showcasing robust results without transformation augmentation.", "questions": "1. I appreciate the soundness of your math, while in Sec 3.1, could you briefly explain, what should the convolution from rays to point/rays to rays handle and why are they necessary? Any ablation on these designs, like using any of the convolution alone in the setting of overfitting/generalized NeRF?\n\n2. This paper spends a large paragraph explaining how to achieve equivariance in the light field space using convolution and transformer. However, I still don't quite see the advantages behind it, .e.g., why in the airplane dataset, w/o the equivariance design, Fvor is still superior compared to the proposed method under R/Y/SO(3) transformations? Also, IBR-Net also surpasses the proposed method over many datasets and metrics.\n\n3. I wonder in Sec 4.2, should the original NeRF & PixelNeRF and other strong baselines that uses canonical pose be included in the comparisons? Besides, I wonder whether this method could benefit the noisy pose reconstruction setting, e.g., comparing with \"SPARF Neural Radiance Fields from Sparse and Noisy Poses\".", "limitations": "Already discussed in the weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method in this paper offers several strengths:\n\n1. Geometric Priors: The approach leverages geometric priors to enhance 3D reconstruction and novel view rendering. By incorporating prior knowledge, it improves the quality of results, particularly when the input views have limited coverage and inter-view baselines.\n\n2. Equivariance to Coordinate Frame Transformations: The method ensures equivariance to coordinate frame transformations, allowing it to handle different orientations and positions of the cameras. This enables more accurate and consistent reconstruction and rendering across various coordinate frames.\n\n3. SE(3)-Equivariant Convolution and Transformer: The introduction of SE(3)-equivariant convolution and transformer in the 3D ray space provides a powerful tool for learning and exploiting geometric priors. It allows for effective feature extraction and representation, leading to improved reconstruction and rendering results.\n\n4. Adaptability to Different Tasks: The method demonstrates adaptability to different tasks, including equivariant 3D reconstruction and equivariant neural rendering. It can be tailored to specific applications, making it versatile and applicable in various scenarios.\n\n5. Robustness without Transformation Augmentation: The approach achieves robust results even in datasets with roto-translated transformations, without the need for additional transformation augmentation techniques. This reduces the complexity and computational requirements while maintaining high-quality outputs.\n\nOverall, these strengths make the proposed method valuable for improving 3D reconstruction and novel view rendering by incorporating geometric priors and addressing challenges related to coordinate frame transformations.", "weaknesses": "1. I appreciate the solid math and notation the author adopted in this paper, which makes the writing theoretically sound. However, this may make the reading very difficult to follow when many mathematical explanations irrelevant to the method itself appears in the paper (like the whole paragraph from line 131-138, which in my understanding could be all moved to the appendix). I believe you could lint your writing and leave more space for the qualitative/quantitative experiments section.\n2. I appreciate the notation explanations in the appendix. However, when I first read the terms like \"type-1 features\", it confuses me and I need to look back to see what I have missed before. A more intuitive name or a brief explanation of these important terminology is required, I think. This is just a recommendation on the writing and won't change my final rating.\n3. More experiments (ablations, baselines) are well needed to demonstrate the soundness of this paper as a new 3D representation learning framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Indeed, lines 131-138 introduce material that is beyond the standard toolbox even for a mathematically avid NeurIPS reader.  Nevertheless, this material is essential for establishing the definition of the generalized convolution as depicted in Equation 3 of the paper. We agree with the reviewer that we can, though, introduce the convolution and still refer to the appendix to explain the terms in the convolution.\n\n**W2:** We really appreciate these comments and we could easily replace type-0 with scalar and type-1 with vectors avoiding, thus, the cryptic representation theory terms.\n\n**W3:** We conducted ablation studies during the initial submission, and the results are presented in Tables 1 and 2 on page 28 of the appendix. We could move them to the paper upon saving space after moving theoretical segments to the appendix. In Table 1 we showed ablations about the importance of convolution and transformer as well as the vector (type-1) features.  \n\nIn addition, we provide an ablation in the rebuttal PDF. We replaced the equivariant transformer for points along the ray with the canonical transformer; Our rationale for not exclusively altering the equivariant convolution and transformer from rays to rays lies in the fact that such a change would eliminate the presence of a feature specifically tailored for equivariance in subsequent modules. Consequently, we would be left with the option of applying a conventional transformer to this feature, resulting in a setup identical to IBRNet, which we have already compared against.\n\nIn order to apply a conventional transformer to the equivariant feature, we first convert the equivariant feature to the invariant (scalar) feature, which makes the model also equivariant, but the performance is inferior to the full model and it shows that the equivariant feature contains more information and the equivariant transformer therefore will be more expressive. \n\n**Q1:** We need ray-to-ray convolution where both the input and output consist of fields distributed across rays. Conversely, when the input constitutes a field distributed across rays, like a radiance (feature) field, but the output takes the form of a field across 3D Euclidean space, such as an occupancy field or Signed Distance Function (SDF), ray-to-point convolution is necessary.\n\nThe selection between ray-to-ray and ray-to-point convolutions is contingent upon the specific task at hand. The pivotal characteristic of the proposed convolution lies in its equivariance to SE(3).  Equivariant ray convolution is essential, providing consistency under SE(3) transformations and overcoming challenges in 3D vision with unknown poses of objects or scenes.\n\nAs stated in W3, we did the ablation by replacing the equivariant transformer over points along the ray, in addition, IBRNet is our baseline by replacing equivariant convolution and transformer from rays to rays with the conventional view aggregation in IBRNet and replacing the equivariant transformer over points along the ray by the ray transformer in IBRNet. \n\nThrough this analysis, we ascertain that equivariant convolution and transformer not only exhibit robustness towards transformations but also highlight the increased potency of the equivariant transformer when maintaining equivariant features within the model. \n\n**Q2:** It is important to highlight that, across all settings, the object pose is provided in Fvor, which significantly aids the 3D reconstruction process by incorporating point positions into the method. In the Y/Y setting, akin to I/I, Fvor outperforms our method due to the provision of more informative data.\n\nRegarding the SO(3)/SO(3) and R/R settings within the airplane category, we attribute the observed trend to the unique characteristics of airplanes. Being slender objects with distinct features, airplanes are relatively straightforward for non-equivariant networks to memorize transformations through training augmentation or data-driven methods. This can explain the comparable performance between our method and Fvor in such scenarios.\n\nFor the neural rendering experiment, IBRNet surpasses our method in the I/I setting, which is understandable, since it uses the absolute positioning encoding while ours uses the relative pose and constrained kernels. \n\nIBRNet is comparable to our method in I/SO(3) settings in the Diffuse Synthetic 360 dataset. We think it is because the synthetic data is less sensitive to the ray direction and has sparser source views, which makes it less dependent on ray directions. The only difference between IBRnet and ours is how to deal with the geometric relation of the rays, when the task is much more dependent on the image feature rather than the geometric information, the priority of the method would be weakened.\n\n**Q3:**   NeRF does not aim to learn a 3D prior by using the radiance field as input, but rather to fit a scene by regressing its radiance field. NeRF's application is primarily geared towards individual scenes, requiring network retraining or optimization reruns whenever the reference frame for densities and colors changes. In this sense, a comparison with the NeRF-based SPARF is not adequate. We will cite it nevertheless.\n\nOur rebuttal PDF includes a robust baseline, NeuRay, as illustrated in Table 2 and Figure 2 of the PDF. NeuRay, like our method, builds upon the IBRNet baseline but goes further by incorporating a depth map or cost volume to estimate point visibility.  We conduct a comparative analysis with NeuRay (Neural Rays for Occlusion-aware Image-based Rendering) on the Real Forward-facing LLFF dataset.\nOur observations reveal that our method demonstrates comparable performance to NeuRay in the I/I setting. In the I/SO(3) configuration, NeuRay experiences a performance decline and inconsistency, whereas our method remains robust against rotations.\n\n\n\n\n\n\n\n\n\n\n     \n"}, {"review_id": "8tsKDRZPcX", "reviewer": "Reviewer_VaKN", "summary": "The aim of this paper is to build equivariance constraints into neural rendering and 3D reconstruction networks. As far as I can tell from the paper, the representation used is a mapping from a ray space to a vector space. The paper describes equivariant convolution and transformer layers that operate over the ray space. Finally, these layers are inserted into existing neural rendering and 3D reconstruction networks to impose equivariance over re-parameterization of the coordinate space in which we represent camera extrinsics.", "questions": "1. How well does the proposed method perform on novel view reconstruction?\n\n2. How does the proposed method compare to Equivariant NeRF or the Scene Representation Transformer?", "limitations": "The limitations of the proposed method are not sufficiently addressed except for a brief mention in the conclusion.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "1. The paper establishes a theory of equivariance in the ray space. This is an important contribution that could guide the design of future equivariant 3D scene representation approaches.\n2. Adding equivariance leads to robustness to rotations and translations in 3D reconstructions. While it does not improve the quality of neural rendering, it makes renders consistent under coordinate frame transformations.", "weaknesses": "Overall, the paper lacks clarity and is not self-contained *at all*. I would suggest a major rewrite that adds a background section, re-structures the methods section and adds needed details to the experiments section.\n\n1. The paper needs a background section. The background should cover at least the definition of rays (using your parameterization), ray space and the IBRNet. A background section is much more important than a 2-page introduction.\n2. The methods section is missing a clear overview at the start about what neural components we need and what equivariance constraints we want to build into these components. The methods section directly jumps into dense exposition with frequent references to the 34-page appendix.\n3. The proposed model that is actually used in the experiments section is delegated to the appendix.\n4. It is not clear why IBRNet is the only baseline for novel view reconstruction. What about NeRF [1], Equivariant NeRF [2] or SRT [3]. The proposed method should be compared with other equivariant novel view reconstruction methods.\n5. It is unclear if Section 4.2 involves novel view reconstruction or only rendering the same images in different coordinate frames. If the latter is the case, novel view reconstruction seems like a vital experiment.\n6. There are very few qualitative examples of the rendered frames, both in the main paper and in the appendix. There should be a much more extensive qualitative comparison of image rendering across several SOTA methods.\n\n**References:**\n\n[1] https://arxiv.org/abs/2003.08934\n\n[2] https://arxiv.org/abs/2006.07630\n\n[3] https://arxiv.org/abs/2111.13152", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** We agree with the reviewer, and we will present the ray space definition and the context of IBRnet in the introduction. The mathematical definition and background of rays were initially placed in the appendix, a decision we acknowledge as not ideal.\n\n**W2:** We will rearrange and rewrite the methods section in the following way: Present first the generalized rendering and reconstruction from multiple views with their neural components of their architectures (convolutional and attentional) and their inputs and outputs. Then, move on with exposing the convolution and attention formula and then explain why we need all the math tools from representation theory necessary in the definition of convolution in homogeneous spaces which is the master-tool of our approach.\n\n All this material exists, In the introduction lines 26-27, we give the clear purpose of the paper, that is we address the problem of learning geometric priors that are SE(3)-equivariant with respect to transformations of the reference coordinate frame. Further in the introduction, between lines 28 and 30, we clarify that our input entails a light field—a function characterized by its orientation within 3D space—whose values encompass radiance or a range of features derived from pixel data. Transitioning to lines 60-70 of the introduction, we provide a detailed exposition of the neural components or features that play a pivotal role in each stage of the 3D reconstruction and neural rendering tasks. We acknowledge the reviewer's input regarding the order of presentation, recognizing its potential to enhance the paper's overall comprehensibility.\n\n**W3:** The main contribution of the paper is the novel introduction of the equivariant convolution and transformer in a light field. We believe that NeurIPS is a conference appreciating novel methodological contributions and this is why we emphasized the description of them instead of the models of the task we applied them on. This is also the case in all novel equivariance papers published at NeurIPS. We still presented the architectures of the rendering and reconstruction models in Figures 6 and 7. We would prefer to devote any available space to explaining the background as requested in your previous point. Of course, if we have space we will elaborate more on the models in the main paper.\n\n**W4 (Q2):** We chose IBRNet as a baseline because it is the classic generalized Nerf method, which makes use of a conventional transformer of rays, similar to approaches in the literature mentioned in your question. Our architecture is based on the IBRNet, and we only replace the aggregation method with equivariant convolution and replace the conventional transformer with the equivariant transformer. Therefore it is fair to take IBRNet as the baseline to show the robustness of our proposed basic operation (convolution, transformer)).\n\nNeRF is not relevant to our approach because it performs a single scene optimization without using any prior. Any time we change the reference frame we have to rerun NeRF because NeRF is not equivariant to the choice of coordinate system where we compute density and color. \n\nThe equivariance in the paper Equivariant Neural Rendering is not the equivariance we mean in our paper: Equivariant Neural Rendering enforces only approximate equivariance via a loss function rather than by design as we do. We will definitely add it to the citations.\n\nIn the rebuttal PDF, we added a comparison to SRT (Scene Representation Transformer), as outlined in Table 3. We present the quantitative outcomes obtained from the MultiShapeNet dataset. Notably, both IBRnet and our model's performances lag behind that of SRT.\n\nWe want to emphasize that our central contribution is to propose an equivariant convolution and transformer on ray space, which can be generally embedded into broad models in 3D learning. SRT is not strictly equivariant, it has the assumption that the views are upright, is inconsistent with respect to the permutation of the cameras, and depends on the camera ID embedding. By randomly choosing the first camera frame as the canonical frame, it approximately achieves data-driven equivariance. \n\nOur framework is sufficiently generic to allow us to take architectures like the SRT and convert them to be equivariant as long as they consist of transformer (or convolutional) modules.\n\n**W5 (Q1):**  This is a misunderstanding. We perform novel view rendering (=reconstruction) from completely different novel 3D poses (section 3.4).\n\n**W6:** We will augment the paper with additional qualitative results to enhance the overall presentation, and we have included extra visualizations in the rebuttal PDF. Although the page constraints of the rebuttal PDF limited the number of visualizations we could include, rest assured that we will incorporate an expanded set of results in the forthcoming revised version.\n\n**Limitation:**  The primary constraint of this approach stems from the finite sampling of the light field. Sparse view-based sampling inadequately addresses substantial object displacements accompanied by significant changes in perspective, resulting in the breakdown of equivariance, which also explains its suboptimal performance on the MultiShapeNet dataset due to noncomprehensive light field sampling. We will expand upon this aspect in a more detailed section within the forthcoming revised paper.\n\n\n\n\n\n\n\n\n\n\n"}, {"review_id": "jbaG1smIZu", "reviewer": "Reviewer_mek5", "summary": "This paper presents a series of methods on SE(3) equivariant convolution and transformer, which can operate on ray space. The experimental results show that the proposed method can help establish SE(3) reconstruction of signed distance functions and neural radiance fields. I'm not an expert in equivariant networks, I feel sorry but I may only be able to provide a general review.\n", "questions": "- What are the model sizes and training times compared to DISN (for SDF reconstruction) and IBRNet (for novel view synthesis)?", "limitations": "It looks the limitations are properly discussed.\n", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper looks solid in theory.\n- Adequate transformation settings in the experiments demonstrate its equivariance property of the proposed network modules.", "weaknesses": "- The entire method description section is obscure to me.\n- It is a bit hard to capture the overall idea when reading the sentences from Line 44 to Line 70.\n- The qualitative results (visual comparisons) are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. “The entire method description section is obscure to me.”**\n\nAs mentioned in the response to Reviewer VaKN, we will change the order of presentation in the method section. We will first define the two problems of generalized rendering and reconstruction from multiple views and then elaborate on how to make their main ingredients (convolution and transformer) equivariant. The definition of equivariance convolution requires several concepts from representation theory. We will introduce first the definition and then elaborate on the concepts so that even if the concepts themselves are quite complex, the motivation for using them is clear. We employed visualization techniques to vividly illustrate the key concepts outlined in the paper, with additional visual aids provided in the appendix. We will try to make them more comprehensible. \n\n**2. “It is a bit hard to capture the overall idea when reading the sentences from Line 44 to Line 70.”**\n\nWe recognize that lines 53-61 describing the definition of equivariance might be confusing and we will rewrite them drawing parallels from other applications of equivariance, as follows. \nEverybody understands what equivariance is in 2D image operations. The output should be transformed the same way as the input image. Exact equivariance holds only when rotations are in multiples of 90 degrees because of the orthogonal sampling grid of the image. \nThings become a bit more complicated in 3D tasks when the input is a point cloud. There, equivariance means that if all input points are transformed, then the output (segmentation, reconstruction) should be transformed as well. Exact equivariance in this case means that all points will be transformed rather than a new sampling of the environment from a rotated sensor configuration. In our case, the input consists of features on rays in 3D. Exact equivariance, here, means that when all rays undergo the same transformation because of a change of reference frame the output features on rays or points will be transformed the same way. This differs from rotating the object because the rays (and content) captured after the rotation will be different from the content before the rotation.\n\nIn lines 44 to 51, we extend the equivariant convolution in the ray space to the equivariant transformer in the ray space, which leverages the kernel in equivariant convolution.  We provide two cases of cross-attention: one is from rays to points, where the query feature is attached to points;  and key, value features are attached to the rays; another one is from rays to rays, where the query feature is attached to the target ray and the key, value features are attached to the source rays. \n\nProceeding from lines 52 to 70, we delve into the practical applications of our model in 3D reconstruction and novel view synthesis, particularly within the realm of multiple views. We initially elucidate the specifics of equivariance within the context of multi-view settings. Subsequently, we detail how we seamlessly integrate the equivariant convolution and transformation modules to cater to the distinct requirements of these two tasks.\n\n**3. “The qualitative results (visual comparisons) are limited.”**\n\nWe have introduced an additional visual comparison in Figure 3, which is illustrated in the rebuttal PDF. Due to space constraints within the PDF, the number of visualizations presented is limited. Nevertheless, we are dedicated to enriching the visual aspect of the paper by including more comparative visualizations to enhance clarity and understanding in the revised paper.\n\n**4. “What are the model sizes and training times compared to DISN (for SDF reconstruction) and IBRNet (for novel view synthesis)?”**\n\nFor a fair comparison, our model size (num of parameters) is similar to that of DISN ( about 16.2M parameters).  Training duration extends to around 1.2 times that of DISN. For novel view synthesis, our model size (num of parameters) is similar to that of IBRnet (about 9.04M parameters). Training time experiences an approximately 1.8-fold increase compared to IBRnet due to the heightened complexity of calculations within each layer.\n\n\n\n\n\n\n\n"}]

Author rebuttal:
We extend our gratitude for the invaluable feedback and suggestions provided by the reviewers. We are pleased to note that all reviewers appreciated the contribution (2 excellent, 3 good) and the soundness (1 excellent, 4 good). All reviewers listed multiple strengths of the approach and none of them doubted its novelty. 

The most negative reviewer (VaKN) says: “The paper establishes a theory of equivariance in the ray space. This is an important contribution that could guide the design of future equivariant 3D scene representation approaches.” Reviewer MuuL writes: “The paper proposes a novel approach for reconstruction from multiple views using equivariant shape priors….The paper is well written and easy to follow.”
Reviewer sFoS says: “This paper presents a theoretical perspective to understand the ray spaces for learning geometric priors for multi-view 3D reconstruction. The thorough theoretical analysis presented in this paper is new to me…This paper is well written with a detailed appendix to understand their theory.” Reviewer gW8w lists 5 strengths and says: “Overall, these strengths make the proposed method valuable for improving 3D reconstruction and novel view rendering by incorporating geometric priors and addressing challenges related to coordinate frame transformations.” 

Two are the main concerns raised: the lack of clarity in the presentation and the lack of comparisons to approaches beyond IBRNet.

We take your feedback about the presentation into serious consideration, we are committed to addressing this concern in the revised version as described in the individual responses to the reviewers. We will convert the introduction section into a background section including necessary definitions that we had in the appendix. We will clarify the definition of equivariance in light fields.
We will change the order of the presentation so that the example tasks are described first in order to motivate the use of equivariant modules like convolution and transformer. Last, we will add more qualitative results to show the performance of our approach.

The second primary concern raised pertains to the rationale behind comparing our approach with IBRNet, along with suggestions for incorporating comparisons with other state-of-the-art methods and conducting further ablation studies.

We have selected IBRNet as our baseline due to its status as a classic and widely recognized generalized NeRF method. IBRNet employs the conventional ray transformer, a technique also prevalent in the literature referred to in the question. Our architecture builds upon the foundations of IBRNet, with a key modification involving the replacement of the combining method with equivariant convolution and the substitution of the conventional transformer with an equivariant counterpart. This reasoned approach justifies IBRNet's selection as the baseline, effectively showcasing the robustness of our fundamental operations, namely convolution, and transformer.

We have added a comparative analysis with NeuRay (Neural Rays for Occlusion-aware Image-based Rendering) on the Real Forward-facing LLFF dataset and with SRT (Scene Representation Transformer) on the MultiShapeNet. We further added more ablations, all included in the rebuttal PDF. 

We are committed to acknowledging and citing all the valuable references they have contributed to the paper.



Meta-review: Existing methods extract features from multi-view images and aggregate them as geometric priors, which are not SE(3)-equivariant concerning the transformation of the reference frame. This paper introduces the SE(3)-equivariant generalized convolution in ray space, resulting in consistent features that could facilitate 3D reconstruction and novel view synthesis. However, reviewers have highlighted significant concerns, noting that the paper is mainly theoretical and lacks clarity. Additionally, its effectiveness in scenarios with sparse views has not been assessed. The AC acknowledges the potential impact of SE(3)-equivariant feature extraction and supports this paper for a spotlight presentation.  Additionally, the AC encourages the author(s) to refine the presentation's clarity and address the unexplored scenarios.
Decision: Accept (spotlight)