Title: $SE(3)$  Equivariant Convolution and Transformer in Ray Space
Venue: NeurIPS 2023
Accepted: yes
Keywords: equivariance

light field

equivariant convolution over homogeneous space
Abstract: 3D reconstruction and novel view rendering can greatly benefit from geometric priors when the input views are not sufficient in terms of coverage and inter-view baselines. Deep learning of geometric priors from 2D images requires each image to be represented in a $2D$ canonical frame and the prior to be learned in a given or learned $3D$ canonical frame. In this paper, given only the relative poses of the cameras, we show how to learn priors from multiple views equivariant to coordinate frame transformations by proposing an $SE(3)$-equivariant convolution and transformer in the space of rays in 3D. We model the ray space as a homogeneous space of $SE(3)$ and introduce the $SE(3)$-equivariant convolution in ray space. Depending on the output domain of the convolution, we present convolution-based $SE(3)$-equivariant maps from ray space to ray space and to $\mathbb{R}^3$. Our mathematical framework allows us to go beyond convolution to $SE(3)$-equivariant attention in the ray space. We showcase how to tailor and adapt the equivariant convolution and transformer in the tasks of equivariant $3D$ reconstruction and equivariant neural rendering from multiple views. We demonstrate $SE(3)$-equivariance by obtaining robust results in roto-translated datasets without performing transformation augmentation.

Equivariant Light Field Convolution and Transformer in Ray Space

Abstract

3D reconstruction and novel view rendering can greatly benefit from geometric priors when the input views are not sufficient in terms of coverage and inter-view baselines. Deep learning of geometric priors from 2D images requires each image to be represented in a 2 D canonical frame and the prior to be learned in a given or learned 3 D canonical frame. In this paper, given only the relative poses of the cameras, we show how to learn priors from multiple views equivariant to coordinate frame transformations by proposing an SE (3) -equivariant convolution and transformer in the space of rays in 3D. We model the ray space as a homogeneous space of SE (3) and introduce the SE (3) -equivariant convolution in ray space. Depending on the output domain of the convolution, we present convolution-based SE (3) -equivariant maps from ray space to ray space and to R 3 . Our mathematical framework allows us to go beyond convolution to SE (3) -equivariant attention in the ray space. We showcase how to tailor and adapt the equivariant convolution and transformer in the tasks of equivariant 3 D reconstruction and equivariant neural rendering from multiple views. We demonstrate SE (3) -equivariance by obtaining robust results in roto-translated datasets without performing transformation augmentation.

1 Introduction

Recent years have seen significant advances in learning-based techniques [67, 68, 60, 69, 61, 73, 11, 53] harnessing the power of deep learning for extraction of geometric priors from multiple images and associated ground-truth shapes. Such approaches extract features from each view and aggregate these features into a geometric prior. However, these approaches are not SE (3) -equivariant to transformations of the frame where the priors and images are defined. While view pooling or calculating variance [69, 72, 46, 73, 11] can be used to aggregate features and tackle equivariance, view pooling discards the rich geometric information contained in a multiple view setup.

In this paper, we address the problem of learning geometric priors that are SE (3) -equivariant with respect to transformations of the reference coordinate frame. We argue that all information needed for tasks like novel view rendering or 3D reconstruction is contained in the light field [6, 37]. Our input is a light field, a function defined on oriented rays in 3D whose values can be the radiance or features extracted from pixel values. We will use the term light field, and we will be specific when it is a radiance field or a feature field. Images are discrete samples of this field: the camera position determines which rays are sampled, while the camera orientation leaves the sample of the light field unchanged up to pixel discretization. We model the light field as a field over a homogeneous space of SE (3) , the ray space R parameterized by the Plücker coordinates. The ray space R is the space of oriented light rays, for any ray x ∈ R , the Plücker coordinate is x = ( d , m ) , where d ∈ S 2 is the direction of the ray, and m = x × d where x is any point on the ray.

Kostas Daniilidis

We define a convolution in the continuous ray space as an equivariant convolution on a homogeneous space [18]. Since our features are not limited to scalar values, we will draw upon the tools of tensor field networks and representation theory, discussed in detail in the Appendix. In Sec. 3.1 we study the group action of SE (3) on R , the stabilizer group for R , and how SE (3) transforms the feature field over R . In Sec. 3.4, we focus on developing the equivariant convolution in R , providing analytical solutions for the kernels with the derived constraints in convolution from R to R and from R to R 3 , respectively. Meanwhile, we make the kernel locally supported without breaking the equivariance. By varying the output domain of the convolution, we introduce equivariant convolutions from the ray space to the ray space and from the ray space to the 3 D Euclidean space.

The constraint of the kernel limits the expressiveness of equivariant convolution when used without a deep structure. In Sec 3.5, we introduce an equivariant transformer in R . The equivariant transformer generates the equivariant key, query, and value by leveraging the kernel derived in the convolution, resulting, thus, in invariant attention weights and, hence, equivariant outputs. We provide a detailed derivation of two cases of cross-attention: the equivariant transformer from R to R and the equivariant transformer from R to R 3 . In the first case, the features that generate the key and value are attached to source rays, while the feature generating the query is attached to the target ray. In the second case, the feature generating the query is attached to the target point.

We demonstrate the composition of equivariant convolution and transformer modules in the tasks of 3 D reconstruction from multi-views and novel view synthesis given the multi-view features. The inputs consist of finite sampled radiance fields or finite feature fields, while our proposed equivariant convolution and transformer are designed for continuous light fields. If an object or a scene undergoes a rigid transformation and is resampled by the same multiple cameras, the SE (3) group action is not transitive in the light field sample. This lack of transitivity can significantly impact the computation of equivariant features, mainly because the views are sparse, unlike densely sampled point clouds. Object motion introduces new content, resulting in previously non-existing rays in the light field sampling. Hence, our equivariance is an exact equivariance with respect to the choice of coordinate frame. In the 3D reconstruction task, we experimentally show that equivariance is effective for small camera motions or arbitrary object rotations and generally provides more expressive representations. In the 3 D object reconstruction application, we first apply an equivariant convolutional network in ray space to obtain the equivariant features attached to rays. We then apply equivariant convolution and equivariant transformer from R to R 3 to obtain equivariant features attached to the query point, which are used to calculate the signed distance function (SDF) values and ultimately reconstruct the object. In the generalized rendering task, our model queries a target ray and obtains neighboring rays from source views. Our composition of equivariant modules is based on IBRNet [61], which consists of view feature aggregation and ray transformer. We replace the view feature aggregation in [61] with the equivariant convolution and transformer over rays and the ray transformer part with the equivariant transformerover the points along the ray to get the density and color of the point, see Sec. 3.3. We summarize here our main contributions:

(1) We model the ray space as a homogeneous space with SE (3) as the acting group, and we propose the SE (3) -equivariant generalized convolution as the fundamental operation on a light field whose values may be radiance or features. We derive two SE (3) -equivariant convolutions, both taking input ray features and producing output ray features and point features, respectively.

(2) To enhance the feature expressiveness, we extend the equivariant convolution to an equivariant transformer in R , in particular, a transformer from R to R and a transformer from R to R 3 .

(3) We adapt and compose the equivariant convolution and transformer module for 3 D reconstruction from multiple views and generalized rendering from multi-view features. The experiments demonstrate the equivariance of our models.

2 Related Work

Equivariant Networks Group equivariant networks [15, 65, 62, 55, 63, 12, 19, 17, 22, 24, 23] provide deep learning pipelines that are equivariant by design with respect to group transformations of the input. While inputs like point clouds, 2D and 3D images, and spherical images have been studied extensively, our work is the first, as far as we know, to study equivariant convolution and cross-attention on light fields. The convolutional structure on homogeneous spaces or groups is sufficient and necessary for equivariance with respect to compact group actions as proved in [18, 1, 36]. Recently, Cesa et al. [8], Xu et al. [70] provided a uniform way to design the steerable kernel in

an equivariant convolutional neural network on a homogeneous space using Fourier analysis of the stabilizer group and the acting group, respectively, while Finzi et al. [26] proposed a numerical algorithm to compute a kernel by solving the linear equivariant map constraint. For arbitrary Lie groups, Finzi et al. [25], MacDonald et al. [39], Bekkers [4] designed the uniform group convolutional neural network. The fact that any O ( n ) equivariant function can be expressed in terms of a collection of scalars is shown in [59]. For general manifolds, Cohen et al. [16], Weiler et al. [64] derived the general steerable kernel from a differential geometry perspective, where the group convolution on homogeneous space is a special case. The equivalent derivation for the light field is in the Appendix. Recently, equivariant transformers drew increasing attention, in particular for 3D point cloud analysis and reconstruction [27, 48, 10, 7]. A general equivariant self-attention mechanism for arbitrary groups was proposed in [44, 43], while an equivariant transformer model for Lie groups was introduced in Hutchinson et al. [32]. We are the first to propose an equivariant attention model in the 3 D ray space.

Light Field and Neural Rendering from Multiple Views The plenoptic function introduced in perception [6] and later in graphics [37] brought a new light into the scene representation problem and was directly applicable to the rendering problem. Instead of reconstructing and then rendering, light fields enabled rendering just by sampling the right rays. Recently, learning-based light field reconstruction [41, 34, 5, 66, 51, 2] became increasingly popular for novel view synthesis, while [50, 54, 53] proposed non-equivariant networks in the ray space. Due to the smaller dimension of the ray space, the networks in the ray space are more efficient compared to neural radiance fields [42], which leverages volumetric rendering. Several studies [73, 61, 53, 50, 11, 38, 13, 31, 58, 20] concentrate on generalizable rendering. These works are similar to ours in that they obtain the 3 D prior from the 2 D images, but they are not equivariant since they explicitly use the coordinates of the points or the rays in the network.

The most related equivariant rendering approaches to us are [21, 47, 45]. The equivariance in the paper [21] is not the equivariance we mean in this work: [21] enforces the geometric consistency via a loss function. [47] is not strictly equivariant and it depends on the assumption of upright views and the camera ID embeddings, which results in its non-equivariance to camera permutation. It achieves data-driven equivariance by randomly choosing the first camera frame as the canonical frame. While [45] addresses the frame problem through relative pose, it is not theoretically equivariant in cases of individual camera rotations around their axes or minor individual rotations accompanied by small content changes. We want to emphasize that our central contribution is to propose an equivariant convolution and transformer on ray space, which can be integrated into a wide range of 3D learning models.

Reconstruction from Multiple Views Dense reconstruction from multiple views is a wellestablished field of computer vision with advanced results even before the introduction of deep learning [28]. Such approaches cannot take advantage of shape priors and need a lot of views to provide a dense reconstruction. Deep learning enabled semantic reconstruction, i.e., the reconstruction from single or multiple views by providing the ground-truth 3D shape during training [14, 67, 68, 40]. These approaches decode the object from a global code without using absolute or relative camera poses. Regression of absolute or relative poses applied in [35, 71, 56, 69, 72, 57, 3, 46, 33, 20] is non-equivariant.

3 Method

In this section, we will first introduce the (feature) field on the ray space and 3 D Euclidean space, respectively, and the corresponding SE (3) group actions on the values of the fields in Sec. 3.1. To offer readers a holistic grasp-from a broad overview down to the intricate specifics and from foundational concepts to advanced techniques- we will present the reconstruction and the generalized rendering with the neural components of their architectures (convolutional and attentional) and their inputs and outputs in Sec. 3.2 and Sec. 3.3. Following that, we expose our central contribution: equivariant convolution and attention in ray space in Sec. 3.4 and Sec. 3.5.

3.1 Feature field on Ray Space and 3 D Euclidean Space

3.1.1 Ray Space

The ray space is the space of oriented light rays. As introduced in the introduction and App. Ex. 1, we use Plücker coordinates to parameterize the ray space R : for any ray x ∈ R , x can be denoted as ( d , m ) , where d ∈ S 2 is the direction of the ray, and m = x × d is the moment of the ray with x

type-1features

type-1features

type-1features

scalarfeatures

scalarfeatures

scalarfeatures

scalarfeatures

vectorfeatures(type-1features)

g E SE(3)

being a point on the ray. Then any g = ( R, t ) ∈ SE (3) acts on the the ray space as:

The ray space R is a homogeneous space with a transitive group action by SE (3) . Given the origin in the homogeneous space as η = ([0 , 0 , 1] T , [0 , 0 , 0] T ) (the line representing z -axis), the stabilizer group H that leaves η unchanged is SO (2) × R (the rotation around and translation along the ray). The ray space is, thus, isomorphic to the quotient space R ∼ = SE (3) / ( SO (2) × R ) . We parameterize the stabilizer group H as H = { ( γ, t ) | γ ∈ [0 , 2 π ) , t ∈ R } .

̸

We follow the generalized convolution derivation for other homogeneous spaces in [18], which requires the use of principal bundles, section maps, and twists [29] explained in the appendix section A.2 and onwards. SE (3) can be viewed as the principal SO (2) × R -bundle, where we have the projection p : SE (3) → R , for any g ∈ SE (3) , p ( g ) = gη ; a section map s : R → SE (3) can be defined such that p ◦ s = id R . In App. Example 6, we elaborate on how we define the section map from the ray space to SE (3) in our model. Generally, the action of SE (3) induces a twist as gs ( x ) = s ( gx ) . The twist can be characterized by the twist function h : SE (3) ×R→ SO (2) × R , gs ( x ) = s ( gx ) h ( g, x ) , we provide the twist function in our model and its visualization in App. Example 6.

3.1.2 Light Field

The light field can be modeled as a function from the ray space to a vector space, f : R → V . We also need to define the SE (3) group action on the values of that field. Since the group action will be on a vector space V , we will use the corresponding group representation of the stabilizer group ρ : SO (2) × R → GL ( V ) , see details in App. A.3. For example, a light field can be a radiance field f that maps the ray space of oriented rays to their observed radiance (RGB) f : R→ R 3 which is a concatenation of three scalar fields over R . The group representation ρ in this case is the identity and g ∈ SE (3) acts on the radiance field f as ( L g f )( x ) = f ( g -1 x ) , shown as the scalar features in Fig. 1. Given that the stabilizer H = SO (2) × R is a product group, the stabilizer representation can be written as the product ρ ( γ, t ) = ρ 1 ( γ ) ⊗ ρ 2 ( t ) , where ρ 1 is the group representation of SO (2) and ρ 2 is the group representation of R . If the light field is a feature field (Fig. 1) with ρ 2 being the identity representation and ρ 1 corresponding to a type-1 field, ρ 1 ( γ ) = e iγ , then type-1 features change position and orientation when g ∈ SE (3) acts on it. Having explained the examples of scalar (type-0) and type-1 fields, we introduce the action on any feature field f as [18]:

where ρ is the group representation of SO (2) × R corresponding to the space V , determined by the field type of f , and h is the twist function introduced by SE (3) as shown in App. Example 6.

3.1.3 Feature Field on R 3

R 3 is also a homogeneous space of SE (3) like the ray space R , with the stabilizer group as SO (3) , as stated in App. Example 2. For any g = ( R, t ) ∈ SE (3) , it acts on the field f over R 3 also follows

pointfeature

b.convolution

B

fromraysto

query

points

a.convolution

ray feature (key)

c.equivariant

SDF

from rays to

key

cross-attention

network

rays

query

point

key

points andrays

between

light field

lightfeaturefield

pointfeaturefield

[18]:

,where ρ is the group representation of SO (3) , since the twist function can be independent of the 3 D position due to the fact that SE (3) = R 3 ⋊ SO (3) is a semidirect product group as stated in App. Example 4. The feature field over R 3 and the corresponding group action is also used in [55, 63]. Fig. 2 visualizes the scalar feature ( l out = 0 ) and vector feature ( l out = 1 ) attached to one point, offering an intuitive understanding of the feature field over R 3 .

Given the feature field on the ray space and 3 D Euclidean space and the corresponding group actions of SE (3) . We will show two 3D multi-view applications of the equivariant convolution and transformer: 3 D reconstruction and generalized neural rendering. In each application, we start with the specific definition of equivariance and then outline the corresponding pipeline.

3.2 Equivariant 3D Reconstruction

The radiance field serves as the input for the 3 D reconstruction, which ultimately generates a signed distance field (SDF) denoted by the function e : R 3 → R . As aforementioned, the radiance field is the multi-channel scalar field over R , while SDF is the scalar field over R 3 . A 3 D reconstruction Φ : F → E , where F denotes the space of radiance fields and E denotes the space of signed distance fields, is equivariant when for any g ∈ SE (3) , any x ∈ R 3 , and any f ∈ F , Φ( L g f )( x ) = L ′ g (Φ( f ))( x ) , where L g and L ′ g are group actions on the light field and the SDF, respectively. Specifically, as f and e are scalar fields, ( L g f )( x ) = f ( g -1 x ) for any x ∈ R , and ( L ′ g e )( x ) = e ( g -1 x ) for any x ∈ R 3 .

In practice, we have a finite sampling of the radiance field corresponding to the pixels of multiple views V = { f ( x ) | x ∈ L V } , where L V denotes the ray set of multi-views and f ∈ F is the radiance field induced by multi views sample from. The 3 D reconstruction Φ is equivariant when for any g ∈ SE (3) and any x ∈ R 3 : Φ( g · V )( x ) = Φ( V )( g -1 x ) . If we denote V as ( L V , f ) , g · V = ( g · L V , L g f ) , where g · L V is g acting on the rays defined Eq. 1.

We achieve equivariance using three steps as illustrated in Fig. 3: (1) the transition from pixel colors to a feature-valued light field (equi-CNN over rays), (2) the computation of features in R 3 from features on the ray space by equivariant convolution from R to R 3 , and (3) the equivariant transformer with the query generated by the feature on the point we want to compute SDF and key/value generated by features on rays. Note that we need (3) following (2) because the output feature of a single convolution layer is not expressive enough due to the constrained kernel.For the detailed practical adaption of the convolution and transformer in 3 D reconstruction, please see the App. B, where we approximate the intra-view with SE (2) equivariant convolution.

3.3 Generalized Neural Rendering

The light feature field f in : R → V serves as the input for neural rendering, which ultimately generates the light field f : R → R 3 , a multi-channel scalar field over R . A neural rendering Ψ : I → F , where I denotes the space of the light feature fields and E denotes the space of the light field, is equivariant when for any g ∈ SE (3) , any x ∈ R , and any f in ∈ I , Ψ( L g f in )( x ) = Ψ( f in )( g -1 x ) , where L g is the group operator on the light feature field f in , as shown in Eq. 2 depending on the feature type. In the experiment of this paper, the input light feature field is

key

(regular'representation)

multi-head

Value

attention

(regularrepresentation)

Convolution

(regularrepresentation)

Query

Equivariant

fromraysto

Transformer

rays

Transformerfromrays to rays

overpoints

scalarfield

regular

regular

representation

representation

color

density

scalar, i.e., L g f in ( x ) = f in ( g -1 x ) . Similar to reconstruction, in practice, the neural rendering Ψ is equivariant when for any g ∈ SE (3) and any x ∈ R : Ψ( g · V )( x ) = Ψ( V )( g -1 x ) , where V = { f in ( x ) | x ∈ L V } , and if we denote V as ( L V , f in ) , then g · V = ( g · L V , L g f in ) ,

By restricting the field type of the output field over rays to have a group representation of SO (2) × R as ρ ( γ, t ) = ρ 1 ( γ ) ⊗ ρ 2 ( t ) , where ρ 2 is the regular representation, we can obtain the feature of points along the ray by convolution or transformer from R to R . See App. Example 9 for more explanation of the regular representation. Alternatively, we can obtain the desired feature by applying convolution or transformer from R to R , with output features attached to the target ray corresponding to different irreducible representations of the stabilizer group. These features can be interpreted as Fourier coefficients of the function of the points along the ray. The Inverse Fourier Transform yields features for the points along the ray. More details are in the App. I.1.

The feature of the points along the ray can be used to generate density and color for volumetric rendering [61, 73], or fed into attention and pooling for the final ray feature [58]. In this paper, we opt to generate the density and color and utilize volumetric rendering, which can be viewed as a specialized equivariant convolution from points to the ray. Method details are in App. I.

We achieve the equivariant rendering through three steps as shown in Fig. 4: (1) we apply equivariant convolution from rays to rays to get the equivariant feature for points along the rays, which is a specific field type over R ; 2) to enhance the feature expressivity, we apply an equivariant transformer from rays to rays to get the color for each point; (3) we apply the equivariant self-attention over the points along the ray to reason over the points on the same ray; the output feature of the points will be fed to multiple perceptron layers to get the density of the points.

3.4 Convolution in Ray Space

3.4.1 Convolution from Rays to Rays

The convolution, as stated in App. A.4 and [18] is then defined as

where h ( g ) is the simplified form of the twist h ( g, η ) . Eq.3 is equivariant to SE (3) if and only if the convolution kernel κ satisfies that κ ( hx ) = ρ out ( h ) κ ( x ) ρ in ( h -1 ( h, x )) , where ρ in and ρ out are the group representations of SO (2) × R corresponding to the input feature type l in and output feature type l out , respectively. We derive the solutions of the kernel in the App. Example 9.

̸

Local kernel support The equivariance stands even if we constrain the kernel to be local. When x = ( d x , m x ) meets the condition that ∠ ( d x , [0 , 0 , 1] T ) ≤ β 0 and d ( x, η ) ≤ d 0 , κ ( x ) = 0 , this local support will not violate the constraint that κ ( hx ) = ρ out ( h ) κ ( x ) ρ in ( h -1 ( h, x )) .

Then, convolution in Eq. 3 is accomplished over the neighbors only as visualized in Fig. 5. In Fig. 5, any ray y = ( d y , m y ) (denoted in blue) in the neighborhood of a ray x = ( d x , m x ) will go through the cylinder with x as the axis and d 0 as the radius since d ( x, y ) ≤ d 0 .

Moreover, for any y , ∠ ( d y , d x ) ≤ β 0 . Any ray y ∈ N ( x ) is on one tangent plane of a cylinder with x as the axis and d ( x, y ) as the radius when d ( x, y ) > 0 .

3.4.2 Convolution from Rays to Points

In applications such as 3 D reconstruction, key point detection, and 3 D segmentation, we expect the output to be the field over R 3 . Using a convolution, we will define an equivariant map from light fields (fields on R ) to fields on R 3 . We denote with H 1 and H 2 the stabilizer groups for the input and output homogeneous spaces, respectively, i.e., SO (2) × R and SO (3) in this case. As shown in the App. Example 4, we can choose the section map s 2 : R 3 → SE (3) : s 2 ( x ) = ( I, x ) for any x ∈ R 3 and I is the identity matrix. Following [18], the convolution from rays to points becomes:

where h 1 is the twist function corresponding to section s 1 : R→ SE (3) defined aforementioned, ρ in is the group representation of H 1 ( SO (2) × R ) corresponding to the feature type l in . The subscripts 1 and 2 denote the homogeneous spaces the features are defined on. The convolution is equivariant if and only if the kernel κ satisfies that κ ( h 2 x ) = ρ out ( h 2 ) κ ( x ) ρ in ( h -1 1 ( h 2 , x )) for any h 2 ∈ H 2 , where ρ out is the group representation of H 2 ( SO (3) ) corresponding to the feature type l out .

In 3D reconstruction, f l in is the scalar field over R , i.e., ρ in = 1 . The convolution is simplified to f l out 2 ( x ) = ∫ G/H 1 κ ( s 2 ( x ) -1 y ) f l in 1 ( y ) dy and the corresponding constraint becomes κ ( h 2 x ) = ρ out ( h 2 ) κ ( x ) . App. Example 10 provides analytical kernel solutions.

3.5 Equivariant Transformer over Rays

We can extend the equivariant convolution to the equivariant transformer model. In general, the equivariant transformer can be formulated as:

where the subscript 1 denotes the homogeneous space M 1 ∼ = G/H 1 of the feature field f in 1 that generates the key and value in the transformer; the subscript 2 denotes the homogeneous space M 2 ∼ = G/H 2 of the feature field f in 2 that generates query in the transformer, which is also the homogeneous space of the output feature f out 2 ; x and y represent elements in the homogeneous spaces M 2 and M 1 , respectively, where y ∈ N ( x ) indicates that the attention model is applied over y , the neighbor of x based on a defined metric. f k , f q , and f v are constructed equivariant keys, queries, and values in the transformer. f k and f v are constructed by equivariant kernel κ k and κ v while f q is constructed through an equivariant linear map, see App. F for detailed construction.

When the transformer is a self-attention model, homogeneous space M 1 and M 2 are the same since f in 2 = f in 1 . The above equivariant transformer could be applied to the other homogeneous space other than R , R 3 , and acting group other than SE (3) . This paper presents the equivariant cross-attention model over rays, i.e., M 1 is R . When the transformer is the cross-attention from rays to rays, M 2 is also R , the equivariant kernel κ k and κ v is the convolution kernel we derived in convolution from rays to rays in Sec. 3.4.1. When the transformer is the cross-attention from rays to points, M 2 is R 3 , the equivariant kernel κ k and κ v is the convolution kernel we derived in convolution from rays to points in Sec. 3.4.2. With the construction in App. F, we claim that the transformer from rays to rays or from rays to points, as shown in the equation 4, is equivariant. The proof is in App. G.

fim(yk)

fi"(yk)

f(x)

fi"(y)

fin(y2)

PE(x)

fi"(y)

fi(y2)

Wq

Kk（Sz（x)-1y)

Kp（S2（x)-y)

Wa

Wk

W

query

key

value

query

key

value

f（x.fm)

fk（x.y.f）f(x.y.fi)

WPE(x)

Wfi"(y)

Multi-head Attention

Multi-head Attention

f(（x)

fout(x)

(type-0)

(type-0)

scalars

(type-0)

Scalars

vectors

vectors.

(type-1)

(type-1)

(type-1)

tensors

(-)

tensors

tensors

(type-i)

(type-i)

(type-i)

query

key

value

(type-0)

calars

()

scalars

(type-0)

(type-0)

sealars

query

key

value

To better understand the equivariant transformer, we visualize the comparison of the equivariant crossattention transformer and conventional transformer shown in Fig. 6. Meanwhile, as stated in App. F, key, query, and value are generally composed of different types of features and are multi-channel, allowing for the multi-head attention mechanism. In Fig. 7, we visualize the comparison of the equivariant multi-head attention module from rays to points with conventional multi-head attention module. The attention module from rays to rays follows a similar concept but with variations in the feature types due to the differing group representations of SO (2) × R and SO (3) .

4 Experiment

4.1 3D Object Reconstruction from Multiple Views

Datasets and Implementation We use the same train/val/test split of the Shapenet Dataset [9] and render ourselves for the equivariance test. To render the views for each camera, we fix eight cameras to one cube's eight corners. The cameras all point in the same direction toward the object's center. We use the following notation to denote the variety of transformations in training and testing: I (no transformation), Z (optical axis rotation), R (bounded 3-dof camera rotation), Y (vertical axis object rotation), SO (3) (full object rotation). The details of the five settings are in App. J.1.

As described in App. B, we use SE (2) equivariant CNNs to approximate the equivariant convolution over the rays. For the fusion from the ray space to the point space model, we use one layer of convolution and three combined blocks of updating ray features and SE (3) transformers. For more details, please see the App. J.2.

Results We evaluate our model in seven experiment settings, I/I , I/Z , I/R , R/R , Y/ SO (3) , SO (3) /SO (3) . The setting A/B indicates training the model on the A setup of the dataset and evaluating it on the B setup. Following the previous works, we use IoU and Chamfer-L1 Distance as the evaluation metric. Quantitative results are reported in table 1, and qualitative results are in Fig. 8.

We compare with two other approaches [69], which follows a classic paradigm that queries 3D positions that are then back-projected to obtain image features for aggregation, and [72], which was state of the art in 3D object reconstruction from multi-views.Note that both baselines originally estimate the object poses, but we directly provide ground truth poses to them. See App. J.4 for more qualitative results.

In table 1, our model outperforms the [72] and [69] by a large margin on I/Z , I/R , and Y/SO (3) settings. Although theoretically, our model is not equivariant to the arbitrary rotation of the object, Y/SO (3) shows the robustness of our model to the object rotation and the generalization ability to some extent. Our model outperforms other models for the chair and car categories in R/R and

SO (3) /SO (3) settings while it is slightly inferior to [72] in the airplane category. Notably, our model only requires relative camera poses, while [72] and [69] utilize camera poses relative to the object frame, leveraging explicit positional encoding of the query point in the object frame, which is concatenated to the point feature. In addition, our model performs better in several experiments in I/I and Y/Y settings. This superiority can be attributed to the SE (3) equivariant attention model, which considers scalar features and ray directions. For a detailed discussion of the results, please see appendix Sec. J.3

We provide an ablation study of the effectiveness of SE (2) CNNs, equivariant convolution, transformer, and type-1 feature (vector feature) in our model. Meanwhile, we compare our method with the model that explicitly encodes the direction of rays. Please see the App. J.5 for the details of the ablation study.

4.2 Neural Rendering

Datasets and Implementation We use the same training and test dataset as in [61], which consists of both synthetic and real data. Two experiment settings illustrate our model's equivariance: I/I and I/SO (3) . I/I is the canonical setting, where we train and test the model in the same canonical frame defined in the dataset. In the I/SO (3) setting, we test the model trained in the conical frame under arbitrarily rotated coordinate frames while preserving relative camera poses and the relative poses between the camera and the scene, thereby preserving the content of the multiple views. Each individual view itself is not transformed. Note that this experiment's SO (3) setup differs from the R and SO (3) setups used in the reconstruc-

FvOR

DISN

Ours

GT

1/1

YY

Z

I/Z

R

I/R

R/R

S0(3)

1/SO(3)

tion. Further details and discussions on this difference can be found in App. K.1.

GT

IBRNet (I/I)IBRNet (1/SO3) Ours (1/I)Ours (1/SO3)

GT

IBRNet(I/I1)IBRNet (1/SO3)Ours (I/I)Ours(/SO3)

Our model architecture is based on IBRNet[61], with view feature aggregation and ray transformer components modifications. Specifically, we replace the view feature aggregation in [61] with the equivariant convolution and transformer over rays and the ray transformer part with the equivariant self-attention over the points along the ray. For more information on the implementation details, please refer to App. K.2.

Results We compare with IBRNet on I/I and I/SO (3) settings to show that our proposed models can be embedded in the existing rendering framework and achieve equivariance. Following previous works on novel view synthesis, our evaluation metrics are PSNR, SSIM, and LPIPS [74]. In the I/SO (3) test period, we randomly rotate each data six times and report the average metrics. Meanwhile, we record the max pixel variance and report the average value. We show a qualitative result in Fig. 9, where IBRNet presents several blurred transverse lines in the I/SO (3) setting while ours are robust to the rotation. In table 2, our model performs comparably with IBRNet [61] in I/I setting without performance drop in I/SO (3) setting. The slight decrease in PSNR/SSIM/LPIPS for IBRNet from I/I to I/SO (3) can be attributed to the training process involving multiple datasets with different canonical frames, which includes transformation augmentation and makes the model more robust to coordinate frame changes. Additionally, conventional metrics like PSNR/SSIM may not directly capture image variations. Therefore, we introduce an additional metric, pixel variance, to illustrate the changes better. We observe that IBRNet [61] exhibits pixel variance for different rotations, whereas our approach remains robust to rotation. Our method performs comparably with IBRNet in the I/SO (3) setting in DeepVoxels [49] because the synthetic data consists of Lambertian objects with simple geometry, where the ray directions do not significantly affect the radiance. For more qualitative results, see App. K.3.

5 Conclusion and Broader Impacts

To learn equivariant geometric priors from multiple views, we modeled the convolution on the light field as a generalized convolution on the homogeneous space of rays with SE (3) as the acting group. To obtain expressive point features, we extended convolution to equivariant attention over rays. The main limitation of the approach is the finite sampling of the light field. The sampling of the light field by sparse views cannot account for large object motions with drastic aspect change, leading to a breakdown of equivariance. This novel general equivariant representation framework for light fields can inspire further work on 3D vision and graphics tasks. We do not see any direct negative impact of our work, but it could have negative societal consequences if misused without authorization, for example, when using images violating privacy.

References

Jimmy Aronsson. Homogeneous vector bundles and g-equivariant convolutional neural networks. Sampling Theory, Signal Processing, and Data Analysis , 20(2):1-35, 2022.

Benjamin Attal, Jia-Bin Huang, Michael Zollhöfer, Johannes Kopf, and Changil Kim. Learning neural light fields with ray-space embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19819-19829, 2022.

Miguel Angel Bautista, Walter Talbott, Shuangfei Zhai, Nitish Srivastava, and Joshua M Susskind. On the generalization of learning-based 3d reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2180-2189, 2021.

Erik J Bekkers. B-spline cnns on lie groups. arXiv preprint arXiv:1909.12057 , 2019.

Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. X-fields: Implicit neural view-, light-and time-image interpolation. ACM Transactions on Graphics (TOG) , 39(6):1-15, 2020.

James R Bergen and Edward H Adelson. The plenoptic function and the elements of early vision. Computational models of visual processing , 1:8, 1991.

Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:2110.02905 , 2021.

Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build e (n)-equivariant steerable cnns. In International Conference on Learning Representations , 2021.

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

Evangelos Chatzipantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Daniilidis. SE(3)equivariant attention networks for shape reconstruction in function space. arXiv preprint arXiv:2204.02394 , 2022.

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14124-14133, 2021.

Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14514-14523, 2021.

Yuedong Chen, Haofei Xu, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Explicit correspondence matching for generalizable neural radiance fields. arXiv preprint arXiv:2304.12294 , 2023.

Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision , pages 628-644. Springer, 2016.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016.

Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In International conference on Machine learning , pages 1321-1330. PMLR, 2019.

Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10130 , 2018.

Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems , 32, 2019.

Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for so (3)-equivariant networks. arXiv preprint arXiv:2104.12229 , 2021.

Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. arXiv preprint arXiv:2304.08463 , 2023.

Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning , pages 2761-2770. PMLR, 2020.

Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 52-68, 2018.

Carlos Esteves, Yinshuang Xu, Christine Allen-Blanchette, and Kostas Daniilidis. Equivariant multiview networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1568-1577, 2019.

Carlos Esteves, Ameesh Makadia, and Kostas Daniilidis. Spin-weighted spherical cnns. arXiv preprint arXiv:2006.10731 , 2020.

Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning , pages 3165-3176. PMLR, 2020.

Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. arXiv preprint arXiv:2104.09459 , 2021.

Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d rototranslation equivariant attention networks. arXiv preprint arXiv:2006.10503 , 2020.

Yasutaka Furukawa, Carlos Hernández, et al. Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision , 9(1-2):1-148, 2015.

Jean Gallier and Jocelyn Quaintance. Differential geometry and Lie groups: a computational perspective , volume 12. Springer Nature, 2020.

Jiaming Han, Jian Ding, Nan Xue, and Gui-Song Xia. Redet: A rotation-equivariant detector for aerial object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2786-2795, 2021.

Xin Huang, Qi Zhang, Ying Feng, Xiaoyu Li, Xuan Wang, and Qing Wang. Local implicit ray function for generalizable radiance field representation. arXiv preprint arXiv:2304.12746 , 2023.

Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups. In International Conference on Machine Learning , pages 4533-4543. PMLR, 2021.

Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. arXiv preprint arXiv:2212.04492 , 2022.

Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG) , 35(6):1-10, 2016.

Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. Advances in neural information processing systems , 30, 2017.

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning , pages 2747-2755. PMLR, 2018.

Marc Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques , pages 31-42, 1996.

Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. arXiv preprint arXiv:2206.05737 , 2022.

Lachlan E MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbitrary lie groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8183-8192, 2022.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4460-4470, 2019.

Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) , 38(4):1-14, 2019.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65 (1):99-106, 2021.

David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In International Conference on Machine Learning , pages 8188-8199. PMLR, 2020.

David W Romero and Jean-Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. arXiv preprint arXiv:2010.00977 , 2020.

Aleksandr Safin, Daniel Durckworth, and Mehdi SM Sajjadi. Repast: Relative pose attention scene representation transformer. arXiv preprint arXiv:2304.00947 , 2023.

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2304-2314, 2019.

Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Luˇ ci´ c, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometryfree novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6229-6238, 2022.

Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning , pages 9323-9332. PMLR, 2021.

Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2437-2446, 2019.

Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems , 34:19313-19325, 2021.

Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. Learning to synthesize a 4d rgbd light field from a single image. In Proceedings of the IEEE International Conference on Computer Vision , pages 2243-2251, 2017.

Norman Steenrod. The topology of fibre bundles , volume 27. Princeton university press, 1999.

Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. arXiv preprint arXiv:2207.10662 , 2022.

Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8269-8279, 2022.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 , 2018.

Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Multi-view consistency as supervisory signal for learning shape and pose prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2897-2905, 2018.

Michał J Tyszkiewicz, Kevis-Kokitsi Maninis, Stefan Popov, and Vittorio Ferrari. Raytran: 3d pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers. arXiv preprint arXiv:2203.13296 , 2022.

Mukund Varma, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In The Eleventh International Conference on Learning Representations , 2022.

Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems , 34:28848-28863, 2021.

Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV) , pages 52-67, 2018.

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4690-4699, 2021.

Maurice Weiler and Gabriele Cesa. General e (2) -equivariant steerable cnns. arXiv preprint arXiv:1911.08251 , 2019.

Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. arXiv preprint arXiv:1807.02547 , 2018.

Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.06020 , 2021.

Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5028-5037, 2017.

Gaochang Wu, Yebin Liu, Lu Fang, and Tianyou Chai. Revisiting light field rendering with deep antialiasing neural network. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang. Pix2vox: Contextaware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2690-2698, 2019.

Haozhe Xie, Hongxun Yao, Shengping Zhang, Shangchen Zhou, and Wenxiu Sun. Pix2vox++: Multi-scale context-aware 3d object reconstruction from single and multiple images. International Journal of Computer Vision , 128(12):2919-2935, 2020.

Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in Neural Information Processing Systems , 32, 2019.

Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniilidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In International Conference on Machine Learning , pages 24596-24614. PMLR, 2022.

Mingyue Yang, Yuxin Wen, Weikai Chen, Yongwei Chen, and Kui Jia. Deep optimized priors for 3d shape modeling and reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3269-3278, 2021.

Zhenpei Yang, Zhile Ren, Miguel Angel Bautista, Zaiwei Zhang, Qi Shan, and Qixing Huang. Fvor: Robust joint shape and pose optimization for few-view object reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2497-2507, 2022.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Appendix

The introduction of convolution and attention to the space of rays in 3D required additional geometric representations for which there was no space in the main paper to elaborate. We will introduce here all the necessary notations and definitions. We have accompanied this presentation with examples of specific groups to elucidate the abstract concepts needed in the definitions.

A Preliminary

A.1 Group Actions and Homogeneous Spaces

g = ( R, t ) ∈ SE (3) acts on the the ray space as:

We can choose the fixed origin of the homogeneous space to be η = ([0 , 0 , 1] T , [0 , 0 , 0] T ) , the line identical with the z -axis of the coordinate system. Then, the stabilizer group H (the rotation around and translation along the ray) can be parameterized as H = { ( R Z ( γ ) , t [0 , 0 , 1] T ) | γ ∈ [0 , 2 π ) , t ∈ R } , i.e., H ≃ SO (2) × R . We can simplify H as H = { ( γ, t ) | γ ∈ [0 , 2 π ) , t ∈ R } . R is the quotient space SE (3) / ( SO (2) × R ) up to isomorphism.

Example 2 . SE (3) acting on the 3 D Euclidean space R 3 : R 3 is isomorphic to SE (3) /SO (3) . Consider another case when SE (3) acts on the homogeneous space R 3 ; for any g = ( R, t ) ∈ SE (3) and x ∈ R 3 , g x = R x + t . If the fixed origin is [0 , 0 , 0] T , the stabilizer subgroup is H = SO (3) since any rotation g = ( R, 0 ) leaves [0 , 0 , 0] T unchanged.

Example 3 . SO (3) acting on the sphere S 2 : S 2 is isomorphic to SO (3) /SO (2) . The last example is SO (3) acting on the homogeneous space sphere S 2 . Given the fixed origin point as [0 , 0 , 1] T , the stabilizer group is SO (2) .

A.2 Principal Bundle

As stated in [29, 18], the partition of the group G into cosets allows us to treat the group G as the principal bundle where the total space is G , the base space is the homogeneous space G/H 1 , the canonical fiber is the stabilizer group H , the projection map p : G → G/H reads p ( g ) = gH = gx 0 = x . The section s : G/H → G of p should satisfy that p ◦ s = id G/H , where id G/H is the identity map on G/H . Note that non-trivial principal bundles do not have a continuous global section, but we can define a continuous section locally on the open set U ⊆ G/H . The action of G causes a twist of the fiber, i.e., gs ( x ) might not be equal to s ( gx ) though they are in the same coset. We use the twist function h : G × G/H → H to denote the twist: gs ( x ) = s ( gx ) h ( g, x ) . Same as [18], we simplify h ( g, eH ) to be h ( g ) , where e is the identity element in G and eH = x 0 .

Given the action of the group G on a homogeneous space X , and given x 0 as the origin of X , the stabilizer group H of x 0 in G is the group that leaves x 0 intact, i.e., H = { h ∈ G | hx 0 = x 0 } . The group, G , can be partitioned into the quotient space (the set of left cosets) G/H and X is isomorphic to G/H since all group elements in the same coset transform x 0 to the same element in X , that is, for any element g ′ ∈ gH we have g ′ x 0 = gx 0 .

Example 1 . SE (3) acting on the ray space R : Take SE (3) as the acting group and the ray space R as its homogeneous space. We use Plücker coordinates to parameterize the ray space R : any x ∈ R can be denoted as ( d , m ) , where d ∈ S 2 is the direction of the ray, and m = x × d where x is any point on the ray, as shown in figure 10 . A group element

p-1(x) = S0(2) × R

x = (d,m)

H =S0(2)×R

Example 4 . Projection, section map and twist function for R 3 and SE (3) : According to Ex. 2, we can consider a bundle with total space as SE (3) , base space as R 3 , and the fiber as SO (3) . For any g = ( R, t ) ∈ SE (3) , the projection map p : SE (3) → R 3 projects g as p ( R, t ) = t . For any x ∈ R 3 , we can define the section map s : R 3 → SE (3) as s ( x ) = ( I, x ) . The twist function h : SE (3) × R 3 → SO (3) is that h( g, x ) = s ( g x ) -1 gs ( x ) = R for any x ∈ R 3 and any g = ( R, t ) ∈ SE (3) . This twist function is independent of x due to the fact that SE (3) = R 3 ⋊ SO (3) is a semidirect product group as stated in [18].

Example 5 . Projection, section map, and twist function for S 2 and SO (3) : As shown in Ex. 3, SO (3) can be viewed as a principal bundle with the base space as S 2 and the fiber as SO (2) . With the rotation R ∈ SO (3) parameterized as R = R Z ( α ) R Y ( β ) R Z ( γ ) , the projection p : G → G/H maps R as follows:

For any d ∈ S 2 , the section map s : S 2 → SO (3) of p should satisfy that p ◦ s = id S 2 as mentioned above, i.e., s ( d )[0 , 0 , 1] T = d . For instance, we could define the section map s as:

where α d and β d satisfies that

Specifically, when d = [0 , 0 , 1] T , α d = 0 and β d = 0 ; when d = -[0 , 0 , 1] T , α d = 0 and β d = π .

As defined, the twist function h : SO (3) × S 2 → SO (2) is that h( R, d ) = s ( R d ) -1 Rs ( d ) .

Example 6 . Projection, section map, and twist function for R and SE (3) : The final example is SE (3) with R as the base space and SO (2) × R as the fiber, which is the focus of this work, as shown

H

R = Rz(α)Ry(β)

in figure 11. According to the group action defined in Eq. 5, the projection map p : SE (3) →R is:

This represents a ray direction d with the 3rd column of a rotation matrix and the moment m with the cross product of the translation and the ray direction. We can construct a section s : G/H → G using the Plücker coordinate:

where s a ( d ) ∈ SO (3) is a rotation that s a ( d )[0 , 0 , 1] T = d , i.e., s a is a section map from S 2 to SO (3) as shown in Ex. 5; and s b ( d , m ) ∈ R 3 is a point on the ray ( d , m ) . In this paper, we define the section map as s (( d , m )) = ( R Z ( α d ) R Y ( β d ) , d × m ) , where α d and β d satisfy that d = R Z ( α d ) R Y ( β d )[0 , 0 , 1] T , which is the same as Ex. 5. Figure 12 displays the visualization of the section map.

Given the section map, for any g = ( R g , t g ) ∈ SE (3) and x = ( d x , m x ) ∈ R , we have the twist function h : SE (3) × R → SO (2) × R is h( g, x ) = s -1 ( gx ) gs ( x ) = (h a ( R g , d x ) , h b ( g, x )) , where h a : SO (3) × S 2 → SO (2) is the twist function corresponding to s a , as shown in Ex. 5, and h b ( g, x ) = ⟨ R g s b ( x ) + t g -s b ( gx ) , R g d x ⟩ . With the above section s defined in this paper, the twist function h : SE (3) ×R→ SO (2) × R is

where R Z ( R g , d x ) = R -1 Y ( β R g d x ) R -1 Z ( α R g d x ) R g R Z ( α d x ) R Y ( β d x ) .

To understand the twist function clearly, we visualize a twist induced by a translation in SE (3) in figure 13,

A.3 Associated Vector Bundle

Given the principal bundle G , we can construct the associated vector bundle by replacing the fiber H with the vector space V , where V ≃ R n and H acts on V through a group representation

gx

gs(x)

LP

t，

x=

(d,m)

s(gx)

t

g = (I,t)

gs(x) = s(gx)h (g,x)

s(x)

h (g,x) = (0,(t,d)

ρ : H → GL ( V ) . The group representation corresponds to the type of geometric quantity in the vector space V , for example, the scalar, the vector, or the higher-order tensor.

The quotient space E = G × ρ V/H is defined through the right action of H on G × V : ( g, v ) h = ( gh, ρ ( h ) -1 v ) for any h ∈ H , g ∈ G and v ∈ V . With the defined projection map p : G × ρ V → G/H : p ([ g, v ]) = gH , where [ g, v ] = { ( gh, ρ ( h ) -1 v ) | h ∈ H } , the element in G × ρ V , we obtain the fiber bundle E = G × ρ V associated to the principal bundle G . For more background and details of the associated vector bundle, we recommend referring to the following sources: [52] and [18].

The feature function f : U ⊆ G/H → V can encode the local section of the associated vector bundle s v : U ⊆ G/H → G × ρ V : s v ( x ) = [ s ( x ) , f ( x )] , where s is the section map of the principal bundle as defined in Sec. A.2. The group G acting on the field f as shown in [18]:

where h : G × G/H → H is the twist function as defined in Sec. A.2.

A.4 Equivariant Convolution Over Homogeneous Space

The generalized equivariant convolution over homogeneous space, as stated in [18], that maps a feature field f l in over homogeneous space G/H 1 to a feature f l ′ out over homogeneous space G/H 2 by convolving with a kernel κ is defined as:

where l in and l ′ out 2 denote the input and output feature types, respectively. ρ in is the group representation of H 1 corresponding to the feature type l in , s 1 is the section map from G/H 1 to G (see Sec. A.2), s 2 is the section map from G/H 2 to G (see Sec. A.2), h 1 is the twist function corresponding to s 1 (see Sec. A.2).

The convolution is equivariant with respect to G , that is

if and only if κ ( h 2 x ) = ρ out ( h 2 ) κ ( x ) ρ in ( h -1 1 ( h 2 , x )) for any h 2 ∈ H 2 , where ρ out is the group representation of H 2 corresponding to the feature type l ′ out .

In the following examples, we will illustrate three instances where the input and output homogeneous spaces, denoted as G/H 1 and G/H 2 , respectively, are identical, meaning that H 1 = H 2 . These examples involve convolutions from R 3 to R 3 , from S 2 to S 2 , and from R to R . Furthermore, we will show an example where H 1 and H 2 differ, explicitly focusing on the convolution from R to R 3 . Example 7 . SE (3) equivariant convolution from R 3 to R 3 : If we use the section map as stated in Ex. 4, we will find that h( s ( x ) -1 s ( y )) = I , therefore convolution 7 becomes:

for any R ∈ SO (3) . When the feature type l in and l out corresponds to the irreducible representation, we have

where D l in and D l out are the Wigner-D matrices, i.e. irreducible representations corresponding to the feature types l in and l out , which is the same as the analytical result in [63].

Example 8 . SO (3) equivariant spherical convolution from S 2 to S 2 : For spherical convolution, when we substitute the section in Eq. 7 with the section we defined in Ex. 5, the convolution integral takes the following form:

where [0 , 0 , 1] T is the fixed original point as stated in Ex. 3, ρ in is the group representation of SO (2) corresponding to the feature type l in . When ρ in and ρ out are the irreducible representations of SO (2) , ρ in and ρ out can be denoted as ρ in ( θ ) = e -il in θ and ρ out ( θ ) = e -il out θ .

To simplify the notation, we utilize R ( θ ) to represent R Z ( θ ) ∈ SO (2) , where θ ∈ [0 , 2 π ) . When considering the cases where x = [0 , 0 , 1] T , h ( R ( θ ) x ) = R ( θ ) ; when x = -[0 , 0 , 1] T , h ( R ( θ ) x ) = R ( -θ ) ; and when x ∈ S 2 -{ [0 , 0 , 1] T , -[0 , 0 , 1] T } , h ( R ( θ ) x ) = R ( -θ ) = I . Therefore, the kernel κ should satisfy the following conditions: κ ( R ( θ ) x ) = e -il out θ κ ( x ) for any R ( θ ) ∈ SO (2) and any x ∈ S 2 -{ [0 , 0 , 1] T , -[0 , 0 , 1] T } ; κ ( x ) = e -i ( l out -l in ) θ κ ( x ) for x = [0 , 0 , 1] T ; and κ ( x ) = e -i ( l out + l in ) θ κ ( x ) for x = -[0 , 0 , 1] T .

Specifically, when the input and output are scalar feature fields over the sphere, convolution reads

κ has such constraint:

and κ should satisfy

for any R ( θ ) ∈ SO (2) , which is consistent with the isotropic kernel of the convolution in [22].

Example 9 . SE (3) equivariant convolution from R to R : In our case, the equivariant convolution from ray space to ray space is also based on the generalized equivariant convolution over a homogeneous space. See Sec. 3.4.1 for the details. We solve the constraint of the kernel here:

for any h ∈ SO (2) × R .

The irreducible group representation ρ in for the corresponding feature type l in = ( ω 1 in , ω 2 in ) , where ω 1 in ∈ N and ω 2 in ∈ R , can be written as ρ in ( γ, t ) = e -i ( ω 1 in γ + ω 2 in t ) for any h = ( γ, t ) ∈ SO (2) × R ; and the irreducible group representation ρ out ( γ, t ) = e -i ( ω 1 out γ + ω 2 out t ) for the feature type l out = ( ω 1 out , ω 2 out ) , where ω 1 out ∈ N and ω 2 out ∈ R , for any h = ( γ, t ) ∈ SO (2) × R .

To simplify the notation, we utilize R ( γ ) to represent R Z ( γ ) ∈ SO (2) , where γ ∈ [0 , 2 π ) . For any h = ( γ, t ) ∈ SO (2) × R and any x = ( d x , m x ) ∈ R , we have h ( h, x ) = s ( hx ) -1 hs ( x ) = ( R Z ( R ( γ ) , d x ) , ⟨ t [0 , 0 , 1] T , d x ⟩ ) according to Ex. 6. Since SO (2) × R is a product group, we can have κ ( x ) = κ 1 ( x ) κ 2 ( x ) , where

Now we solve the constraint for the kernel κ 1 :

One can check that for any d x ∈ S 2 -{ [0 , 0 , 1] T , -[0 , 0 , 1] T } , R Z ( R ( γ ) , d x ) = I ; when d x = [0 , 0 , 1] T , R Z ( R ( γ ) , d x ) = R ( γ ) ; and when d x = -[0 , 0 , 1] T , R Z ( R ( γ ) , d x ) = R ( -γ ) .

Therefore, we obtain the constraint that

when d x ∈ S 2 -{ [0 , 0 , 1] T , -[0 , 0 , 1] T } ;

when d x = [0 , 0 , 1] T ;

when d x = -[0 , 0 , 1] T ;

The solution for Eq. 11 is that κ 1 ( x ) = f ( d ( η, x ) , ∠ ([0 , 0 , 1] T , d x )) e -iω 1 out atan 2([0 , 1 , 0] d x , [1 , 0 , 0] d x ) , where atan 2 is the 2-argument arctangent function, and f is an arbitrary function that maps ( d ( η, x ) , ∠ ([0 , 0 , 1] T , d x )) to the complex domain.

̸

The solution for Eq. 12 is that when ω 1 out = ω 1 in , κ 1 ( x ) = C , where C is any constant value; when ω 1 out = ω 1 in and x = η , κ 1 ( x ) = 0 ; when ω 1 out = ω 1 in and x = η , κ 1 ( x ) = f ( d ( η, x )) e -i ( ω 1 out -ω 1 in ) atan 2([0 , 1 , 0] m x , [1 , 0 , 0] m x ) ,where f is an arbitrary function that maps d ( x, η ) to the complex domain.

̸

̸

̸

The solution for Eq. 13 is that when ω 1 out = -ω 1 in , κ 1 ( x ) = C , where C is any constant value; when ω 1 out = -ω 1 in and x = -η , κ 1 ( x ) = 0 ; when ω 1 out = -ω 1 in and x = -η , κ 1 ( x ) = f ( d ( η, x )) e -i ( ω 1 out + ω 1 in ) atan 2([0 , 1 , 0] T m x , [1 , 0 , 0] T m x ) ,where f is an arbitrary function that maps d ( x, η ) to the complex domain.

̸

̸

Next, we will solve the constraint for the kernel κ 2 , which is that κ 2 (( γ, t ) x ) = e -i ( ω 2 out -ω 2 in ⟨ [0 , 0 , 1] T , d x ⟩ ) t κ 2 ( x ) .

̸

̸

When d x = [0 , 0 , 1] T , and ω 2 out = ω 2 in , κ 2 ( x ) = 0 ; When d x = -[0 , 0 , 1] T and ω 2 out = -ω 2 in , κ 2 ( x ) = 0 ; When d x = [0 , 0 , 1] T , and ω 2 out = ω 2 in , κ 2 ( x ) = f ( d ( x, η )) , where f is an arbitrary

function that maps d ( x, η ) to the complex domain; When d x = -[0 , 0 , 1] T , and ω 2 out = -ω 2 in , κ 2 ( x ) = f ( d ( x, η )) , where f is an arbitrary function that maps d ( x, η ) to the complex domain; when d x ∈ S 2 -{ [0 , 0 , 1] T , -[0 , 0 , 1] T } ,

where f is an arbitrary function that maps ( d ( η, x ) , ∠ ([0 , 0 , 1] T , d x )) to the complex domain; g ( x ) = [0 , 0 , 1]( x Q -[0 , 0 , 0] T ) , where x Q represents the 3D coordinates of a point Q . This point Q can be defined as the intersection of x and η if x and η intersect. Alternatively, if x and η do not intersect, Q is determined as the intersection of η and the ray y , which is perpendicular to both x and η , and intersects with both x and η . Refer to Figure 16 for a visual representation. One can easily check that g (( γ, t ) x ) = t + g ( x ) , as shown in figure 16, which makes the solution valid.

If x and η are intersected, i.e., [0 , 0 , 1] m x = 0 ,

when [1 , 0 , 0] d x = 0 ;

When x and η are not intersected,

Regular Representation Here, we delve into the case where the output field type corresponds to the group representation of SO (2) × R that ρ ( γ, t ) = ρ 1 ( γ ) ⊗ ρ 2 ( t ) for any ( γ, t ) ∈ SO (2) × R , where ρ 2 is the regular representation. The regular representation of a group G is a linear representation that arises from the group action of G on itself by translation, that is when ρ 2 : R → GL ( V ) is the regular representation, for any v ∈ V , for any t, t ′ ∈ R , we have ( ρ 2 ( t ′ ) v ) t = v t -t ′ , in other words, v ∈ V can be viewed as a function defined on R or an infinite dimensional vector. Then according to Ex. 6, the group SE (3) acting on the the field f would be:

for any t ∈ R , x ∈ R and g ∈ SE (3) .

The points x on the ray x = ( d x , m x ) can be uniquely expressed as x = s b ( x ) + t x d x = d x × m x + t x d x , therefore for any x ∈ R , any t ∈ R , f ( x ) t can be expressed as a feature attached to the point s b ( x ) + t d x along the ray x ,i.e., f ( x ) t = f ′ ( s b ( x ) + t d x , d x ) as shown in figure 14.

Therefore, we have f ′ ( x , d ) = f (( d , x × d )) ⟨ x -d × ( x × d ) , d ⟩ , one can easily check:

We should note the difference of the point x along the ray and the independent point x , as shown in the above equation, the point x along the ray x = ( d , x × d ) is denoted as ( x , d ) instead of x . Actually, it can be viewed as a homogeneous space of SE (3) larger than R 3 , whose elements are in R 3 × S 2 , as shown in figure 15.

To summarize, the features attached to the ray, whose type corresponds to the regular representation of translation, can be considered as the features attached to the points along the ray. The action of SE (3) on features attached to these points can be expressed as shown in Eq. 15.

The solution κ also can be expressed as

̸

when [1 , 0 , 0] d x = 0 ;

dx,mx

f

t

f'(x,dx)

for any t ∈ R , and their constraint is also the same as Eq. 9 and Eq. 10. As a result, the solution for κ 1 should be the same. We only need to solve κ 2 :

for any ( γ, t ′ ) ∈ SO (2) × R .

When d x ∈ S 2 -{ [0 , 0 , 1] T , -[0 , 0 , 1] T } ,

where f and g are the same function as defined in 14, and δ ( t ) = 1 only when t = 0 .

when d x ∈ { [0 , 0 , 1] T , -[0 , 0 , 1] T } , κ 2 ( x ) t = 0 for any t ∈ R .

Example 10 . SE (3) equivariant convolution from R to R 3 : Following [18], the convolution from rays to points becomes:

where h 1 is the twist function corresponding to section s 1 : R→ SE (3) defined aforementioned, ρ in is the group representation of SO (2) × R , corresponding to the feature type l in , s 2 : R 3 → SE (3) is the section map defined in paper as s 2 ( x ) = ( I, x ) .

In this paper, we give the analysis and solutions for the kernel where the input is the scalar field over the ray space, i.e., ρ in = 1 , the trivial group representation, which is also the case of our application in reconstruction.

The convolution is equivariant if and only if

vector features(type-1)vector features(type-1)

for any h 2 ∈ SO (3) , where ρ out is the group representation of SO (3) corresponding to the feature type l out .

We can derive κ ( h 2 x ) = ρ out ( h 2 ) κ ( x ) analytically. For irreducible representation ρ out and any x = ( d x , m x ) ∈ R , if ∥ m x ∥ = 0 , κ ( x ) = cY l out ( d x ) , where c is an arbitrary constant and Y l out is the spherical harmonics and l out is the order (type) of output tensor corresponding to the representation ρ out ; With ∥ m x ∥ ̸ = 0 , κ ( x ) becomes ρ out (ˆ x ) f ( ∥ m ∥ x ) , where ˆ x denotes the element ( d x , m x ∥ m x ∥ , d x × m x ∥ m x ∥ ) in SO (3) and f : R → R (2 l out +1) × 1 .

̸

Similar to the convolution from rays to rays, we also can have the local support of the kernel. We set κ ( x ) = 0 when ∥ m x ∥ ≤ d 0 , otherwise κ ( x ) = 0 . One can easily check that it doesn't break the equivariant constraint for the kernel.

Specifically, when we set d 0 = 0 , the neighborhood of the target points in the convolution only includes the rays from all views going through the point. Hence, we can simplify the convolution to f l out 2 ( x ) = ∫ d ( y,x )=0 Y l out ( d s 2 ( x ) -1 y ) f in 1 ( y ) dy . This equation shows that for every point x , we can treat the ray y going through x with feature f in 1 as a point y ′ , where y ′ -x = d s 2 ( x ) -1 y , as shown in figure 17.

B Equivariant 3D Reconstruction

B.1 Approximation of the Equivariant Convolution from Rays to Rays

In practical 3 D reconstruction, we have multiple views instead of the whole light field. Although the convolution above is defined on the continuous ray space, the equivariance still strictly holds when the ray sampling (pixels from camera views) is the same up to coordinate change. In this case, we will show how we adjust the equivariant convolution from rays to rays and approximate it by an intra-view SE (2) -convolution.

(L,t[0,0,1]T)x

n

n

(L,t[0,0,1]T)x

(0,0, g((I,t[0,0,1]T)x)

(0,0, g((I,t[0,0,1]T)x)

x

t

(0,0, g(x))

(0,0, g(x))

y'

y1

1

Ids2(x)

x

y2

x

y

2

y3

y

3

B.1.1 From Light Field to Intra-view Convolution

Following Fig. 18, neighboring rays are composed of two parts: a set of rays from the same view and another set of rays from different views. For one ray x in view A , the neighboring rays from view B are in the neighborhood of the epipolar line of x in view B . When the two views are close, the neighborhood in the view B would be very large.

The kernel solution in Ex. 9 suggests that κ ( x ) is related to ∠ ( d x , [0 , 0 , 1] T ) and d (( x, η ) , where η = ([0 , 0 , 1] T , [0 , 0 , 0] T ) as mentioned before. It would be memory- and time-consuming to memorize the two metrics beforehand or to compute the angles and distances on the fly. Practically, the light field is only sampled from a few sparse viewpoints, which causes the relative angles of the rays in different views to be large and allows them to be excluded from the kernel neighborhood; therefore, in our implementation, the ray neighborhood is composed of only rays in the same view.

B

A

do

βo

x

̸

B.1.2 From Intra-view Light Field to Spherical Convolution

After showing that a small kernel support in the case of sparse views affects only intra-view rays, we can prove that an intra-view light-field convolution is equivalent to a spherical convolution when we constrain the feature field types over R .

We exploit the desired property that a feature defined on a ray is constant along the ray. This means that the translation part of the stabilizer group (translation along the ray) leaves the feature as is. In math terms, the irreducible representation for the translation R is the identity, which means that the field function is a scalar field for the translation group, with the formula ( L t f )( x ) = f ( t -1 x ) . We prove that, in this case, the intra-view convolution over rays is equivalent to the spherical convolution; please see Sec. C.

B.1.3 From SO(3)- to SE(2)-convolution

While there is an established framework for spherical convolution using a Fourier transform [17, 22, 24] it is not applicable in our case because the boundaries of the constrained field of view cause an explosion in the high frequencies of the spherical harmonics. We will make a compromise here and approximate the SO(3) convolution with an SE(2) convolution on the image plane by making the assumption that the field of view is small. One can see the rationale behind this approximation by keeping only the first order terms in the optical flow equation: the rotational term is only due to Ω z while the translational term is ( -T x -Ω y , -T y +Ω x ) with (Ω x , Ω y , Ω z ) as the angular velocity. We provide a justification using the formalism of the previous paragraphs in appendix Sec. E.

B.2 Ray Fusion: Equivariant Convolution and Transformer

To reconstruct a 3D object, we use an implicit function known as the signed distance function (SDF) defined on R 3 . As a result, we require an equivariant model that can transform features from rays to

points to obtain the SDF. This can be achieved using the equivariant convolution in Sec. 3.4.2 and transformer in Sec. 3.2 in the paper 3.5, which allows us to transform features from the ray space to points in 3D space while maintaining equivariance.

B.2.1 Equivariant Convolution from Rays to Points

In this paper, we obtain the scalar feature field over rays after the SE(2)-equivariant CNNs. As illustrated in figure 3 , we utilize the equivariant convolution (discussed in Sec. 3.4.2 ) to compute features for a query point by convolving over neighboring rays. Our experiments have shown that convolving only over rays that go through the point achieves the best results, and the equivariant kernel used for this convolution is provided in Ex.10. Moreover, in the implementation, we can concatenate the input feature f in 1 with the depth embedding of the query point x . While this theoretically breaks the ideal equivariance for continuous light fields, it does not affect the practical equivariance, as it is rare for two cameras to share the same ray.

B.2.2 Equivariant Transformer from Rays to Points

For the third step, we introduce an equivariant transformer to alleviate the loss of expressivity due to the constrained kernel κ in Eq. 19. Again, the attention key and values are generated from the feature attached to rays, while the query is generated from the feature attached to points.

In the implementation, we apply a transformer over the rays going through the query point. We can continue to use the interpretation that treats any ray y passing through the point x as a point y ′ such that y ′ -x = d s 2 ( x ) -1 y , as shown in figure 17. Since y becomes point y ′ , the ray feature f in 1 becomes the feature over R 3 attached to 'points" y ′ . We can update the neighboring ray feature by directly concatenating the equivariant feature of the point to every ray feature before through a SO (3) equivariant MLP. The transformer in Eq. 4 would be converted to the transformer in [27] over R 3 . See appendix Sec. H for details. The composition of the ray updating block and transformer block are shown in figure 22.

C Proof of Equivalence of Intra-view Light Field Convolution and Spherical Convolution

The property that a feature defined on a ray is constant along the ray means that the translation part of the stabilizer group (translation along the ray) leaves the feature as is. In math terms, the irreducible representation for the translation R is the identity, which means that the field function is a scalar field for the translation group, with the formula ( L t f )( x ) = f ( t -1 x ) . The equivariant condition on the kernel can then be simplified as

where h ∈ SO (2) and t ∈ R , ρ in and ρ out are irreducible representations for SO (2) , and h a is the twist function as shown in Ex. 6 that h ( g, x ) = ( h a ( R g , d x ) , h b ( g, x )) ,i.e., the twist of the fiber introduced by action of SO (3) corresponding to the section map s a of SO (3) in Ex. 5 and Ex. 6. Now we describe the relationship between the intra-view light-field convolution and the spherical convolution:

Proposition C.1. When the translation group acts on feature f : R → V as ( L t f )( x ) = f ( t -1 x ) for any x ∈ R , the equivariant intra-view light-field convolution:

becomes a spherical convolution:

where f ′ l in ( d y ) = f l in ( d y , c x × d y ) , c x denotes the camera center that x goes through, s a is the section map of SO (3) as defined in appendix Ex. 5, and κ ′ ( s a ( d x ) -1 d y ) = κ ( s a ( d x ) -1 d y , ( s ( x ) -1 x c ) × ( s a ( d x ) -1 d y )) .

Proof. The SE (3) equivariant convolution over rays transforms into intra-view convolution when the neighboring lights are in the same view. Moreover, the simplified kernel constraint derived in the paper is that for any ( h, t ) ∈ SO (2) × R and x = ( d x , m x ) ∈ R :

where h a : SO (3) × S 2 → SO (2) is the twist function: h a ( g, d ) = s a ( g d ) -1 gs a ( d ) for any g ∈ SO (3) and d ∈ S 2 .

With the simplified kernel constraint, we can prove that intra-view light field convolution is equivalent to spherical convolution:

In line 21, c x is the camera center that x goes through.

The line 21 is equal to the line 22 because we assume that the irreducible representation for the translation R is the identity as mentioned in the paper.

From line 22 to line 23, We can replace s ( x ) -1 y with

due to the facts that s a ( d x ) -1 d y = d s ( x ) -1 y and point s ( x ) -1 x c is on the ray s ( x ) -1 y . Since y goes through c x , we can replace y with ( d y , c x × d y ) .

From line 23 to 24, we have f ′ l in ( d y ) = f l in ( d y , c x × d y ) because c x is fixed for any view. Additionally, from line 23 to 24 we replace

with κ ′ ( s a ( d x ) -1 d y ) . It is because according to

we have κ (( e, t ) x ) = κ ( x ) for any t ∈ R , where e is the identity element in SO (2) ; thus when t = (( -s ( x ) -1 x c )) T [0 , 0 , 1] T , we have

Line 25 is equal to 26 because s ( x ) -1 x c is always on the z axis, and thus s ( x ) -1 x c + t [0 , 0 , 1] T = [0 , 0 , 0] T .

D Spherical Convolution Expressed in Gauge Equivariant Convolution Format

Group convolution is a particular case of gauge equivariant convolution [64], where gauge equivariant means the equivariance with respect to the transformation of the section map (transformation of the

s(y)[0,0,1]T

hx→y

s(x →y)1

s(y)[1,0,0]7

s(x →y)2

s(x)[0,0,1]T

s(x)[0,1,0]T

tangent frame). In the following paragraph we give the elaborated definition of gauge equivariance for the sphere.

Suppose f : S 2 → V is the field function corresponding to the section choice s a : S 2 → SO (3) , we use L s a → s ′ a acting on f to denote the change of section map from s a to s ′ a : ( L s a → s ′ a f )( x ) = ρ ( s a ( x ) -1 s ′ a ( x )) -1 f ( x ) , where ρ is the irreducible representation of SO (2) corresponding to the field type of f . The convolution Φ is gauge equivariant when Φ( L s a → s ′ a f ) = L s a → s ′ a (Φ( f )) .

In this section, we show that the spherical convolution can be expressed in terms of the gauge equivariant convolution [16], which provides the convenience for us to verify the approximation of spherical convolution through the SE (2) convolution:

where κ ′ ( hx ) = ρ out ( h ) κ ′ ( x ) ρ -1 in ( h ) for any h ∈ SO (2) .

Since the focus of this section's discussion is spherical convolution, here we use s ( x ) to denote s a ( x ) for any x ∈ S 2 .

For any x, y ∈ S 2 , s ( x )[1 , 0 , 0] T , s ( x )[0 , 1 , 0] T attached to x are tangent vectors on x , we parallel transport s ( x )[1 , 0 , 0] T and s ( x )[0 , 1 , 0] T along the geodesic between x and y and get two tangent vectors on y , denoted as s ( x → y ) 1 and s ( x → y ) 2 as shown in the figure 19, where the parallel transport along a smooth curve is a way to translate a vector 'parallelly" based on the affine connection, that is, for a smooth curve γ : [0 , 1] → S 2 , the parallel transport X : Im ( γ ) →T S 2 along the curve γ satisfies that ∇ ˙ γ ( t ) X = 0 , where Im ( γ ) = { γ ( t ) | t ∈ [0 , 1] } and ∇ is the affine connection.

s ( x → y ) 1 and s ( x → y ) 2 need to undergo a transformation in SO (2) to align with s ( y )[1 , 0 , 0] T and s ( y )[0 , 1 , 0] T on y as shown in the figure 19. We denote the transformation as h x → y .

With the above notation, the spherical convolution can be expressed as:

where η = [0 , 0 , 1] T , the fixed origin point in S 2 , and κ ′ ( x ) = κ ( x ) ρ in ( h x → η ) -1 for any x ∈ N ( η ) . We can derive the equivariant condition that κ ′ should satisfy:

Therefore, the spherical convolution can be expressed as the gauge equivariant convolution format:

where κ ′ ( hx ) = ρ out ( h ) κ ′ ( x ) ρ -1 in ( h ) for any h ∈ SO (2) .

E Converting Spherical Convolution to SE (2) Equivariant Convolution

As stated in Sec. D, spherical convolution is gauge equivariant with respect to the choice of section map s a , and the spherical convolution can be written as gauge equivariant convolution. In this section, we use the gauge equivariant convolution to analyze the SE (2) equivariant convolution's approximation of spherical convolution.

Since each view performs spherical convolution on its own, we only analyze the convolution for one view for the sake of simplicity. We use V to denote the space of the rays in the same view, where V ⊂ S 2 . For any x ∈ V , we can choose the section map s a such that h x → o = e , where o ∈ S 2 that o aligns with the optical axis as shown in the figure 20. Again, we use s ( x ) to denote s a ( x ) for any x ∈ S 2 in this section.

When FOV is small, for any x, y ∈ V , we can have such approximation: h x → y = e . Then the above gauge equivariant convolution in Sec. D can be approximated as

where η = [0 , 0 , 1] T , the fixed origin in S 2 , and κ ′ ( hx ) = ρ out ( h ) κ ′ ( x ) ρ -1 in ( h ) for any h ∈ SO (2) .

Additionally, as illustrated in figure 21, we have a map from V to the projection points on the picture plane represented as ω : V → R 2 , where ω ( o ) is defined as [0 , 0] T . When FOV is small, we have such approximation that for any h ∈ SO (2) , t ∈ N ( η ) , and x ∈ V ,

It is because and we have

when β x is small (FOV is small), the approximation stands.

Then f l out ( x ) = κ ′ ( t ) f l in ( s ( x ) t ) dt can be approximately conducted in the image plane:

where for any x ∈ S 2 , f ′ ( ω ( x )) = f ( x ) , and for any t ∈ N ( η ) , κ ′′ ( ω ( s ( o ) t )) = κ ′ ( t ) .

Since for any h ∈ SO (2) and any t ∈ N ( η ) , ω ( s ( o ) ht ) = hω ( s ( o ) t ) , we have for any h ∈ SO (2) and any t ∈ N ( η ) ,

@(x)

[(0)∞]0

x

Imageplane

camera

center

Therefore, convolution 27 is exactly SE (2) equivariant convolution and it can be used to approximate the spherical convolution.

In other words, we can intuitively approximate the equivariant convolution over the partial sphere using the SE (2) equivariant network when the distortion of the sphere and the tangent plane of the optical axis is modest.

F Construction of Features in Equivariant Light Field Transformer

Noted that f out 2 , f in 2 and f in 1 are features that are composed of fields of different types, denoted as f out 2 = ⊕ i f l out i 2 , f in 2 = ⊕ i f l in i 2 , and f in 1 = ⊕ i f l ′ in i 1 3 . f k , f q , and f v are constructed equivariant key features, query features, and value features, respectively, which are composed of fields of different types as well.

We use f k = ⊕ i f l k i k , f q = ⊕ i f l k i q , and f v = ⊕ i f l v i v to denote f k , f q and f v , respectively. We construct the features f k , f q and f v through the equivariant kernels κ k = ⊕ j,i κ l k j ,l ′ in i k , κ v = ⊕ j,i κ l v j ,l ′ in i v and equivariant matrix W q = ⊕ j,i W l k j ,l in i q :

where for any i, j , any h 2 ∈ SO (3) , and any x ∈ R κ l k j ,l ′ in i k and κ l v j ,l ′ in i v should satisfy that:

where h 1 ( h 2 , x ) = s 1 ( h 2 x ) -1 h 2 s 1 ( x ) is the twist function, and for any i, j and any h 2 ∈ SO (3) , W l k j ,l in i q satisfies that:

When the group representation is irreducible representation, due to Schur's Lemma, we have W l k j ,l in i q = cI when l k j = l in i , where c is an arbitrary real number, otherwise W l k j ,l in i q = 0 .

G Proof for Equivariance of Light Field Transformer

The equivariant light field transformer defined in the paper reads:

is in a general form.

According to [18], one can prove that f q , f k and f v are equivariant, that is, for any g ∈ SE (3) , x ∈ R 3 and y ∈ R ,

where L in and L ′ in are group action of SE (3) on f in 2 and f in 1 , respectively.

The inner product ⟨ f q , f k ⟩ = ∑ i ( f l k i q ) T f l k i k is invariant due to the property of unitary representation, which results in the equivariance of the transformer.

H From SE (3) Equivariant Transformer in Ray Space to SE (3) Equivariant Transformer in Euclidean Space

In our implementation for the reconstruction task, the attention model is always only applied over the rays going through the points. We can continue to use the interpretation in the convolution from ray space to R 3 in Ex. 10 that treats any ray y passing through the point x as a point y ′ such that y ′ -x = d s 2 ( x ) -1 y as shown in the figure 17.

After we get the initial feature of query points through equivariant convolution from R to R 3 , we update the neighboring ray feature by directly concatenating the query point feature to every ray feature before through a SO (3) equivariant MLP as shown in the figure 22. SO (3) equivariant MLP is composed of an equivariant nonlinear layer and self-interaction layer as in the tensor field networks [55].

Since y becomes point y ′ , and f in 1 is the feature over R 3 attached to 'points" y ′ , it becomes ⊕ i f l in i 1 4 . Then transformer 32 would be converted to the transformer in [27] over R 3 :

as stated in [27].

The feature f q is constructed in the same way as Equation 30.

fin

Equi-MLP

x

fin

fin

Equi-MLP

Equi-MLP

fin

Wa

Kk(y'

一

Kv(y' -x)

query

key

value

fq(x,fin)

fk(x,y',fin)fo(x,y',fin)

Multi-head Attention

fout

Equi-MLP

/2

where the subscript denotes the points to which the feature is attached, i.e., x and y ′ .

The features f k , f v are constructed by the equivariant kernels κ k = ⊕ j,i κ l k j ,l in i k , κ v = ⊕ j,i κ l v j ,l in i v :

where for any i, j , any h 2 ∈ SO (3) , and any x ∈ R 3 κ l k j ,l in i k and κ l v j ,l in i v should satisfy that:

Y

fin

Equi-MLP

fin

PS(y1)

MLP

fin

x

fin

Equi-MLP

PS(x)

PS(y2)

MLP

fin

Equi-MLP

(PS(y3)

MLP

fin

Wa

X

K(y'

X

Wa

Wk

W,

query

key

value

query

key

value

Wqfin(x)

fk(x,y',fin)f(x,y',fin)

WqPS(x)

Wkfin(y)

Wufin(y)

Multihead Attention

Multihead Attention

f2

fout

Equi-MLP

MLP

Figure 22 shows the structures of ray feature update and SE (3) equivariant transformer.

In figure 23, we compare the SE (3) equivariant transformer and the conventional transformer to illustrate how the equivariance is guaranteed in the equivariant transformer. In figure 24, we present the types of futures in SE (3) equivariant attention head and conventional attention head, respectively. It indicates that geometric information is aggregated equivariantly in multi-head attention in the equivariant transformer.

I Equivariant Neural Rendering

Equivariant rendering relates to equivariant 3 D reconstruction, where we focus on multiple views instead of the entire light field. The equivariance property is maintained when the ray sampling is invariant up to a coordinate change.

I.1 Convolution from Rays to Rays

For neural rendering tasks, we query one ray and apply the convolution over the neighboring rays to obtain the feature attached to the target query ray. Similar to the reconstruction, we utilize a kernel with local support. However, there is a distinction in that for neural rendering, the kernel κ is constrained to be nonzero only when d ( x, η ) = 0 , while there are no constraints on ∠ ( d x , [0 , 0 , 1] T ) .

scalars

scalars

scalars

(type-0)

(type-0)

(type-0)

vectors

vectors

vectors

(type-1)

(type-1)

(type-1)

(,)

tensors

tensors

tensors

(type-i)

(type-i)

(type-i)

query

key

value

scalars

(·,)

scalars

scalars

(type-0)

(type-0)

(type-0)

query

key

value

As a result, the neighboring rays exclusively encompass the rays on the epipolar line for the target ray in each source view, as depicted in Figure 25.

The scalar field over rays serves as the input to the convolution. The output field type corresponds to the regular representation of translation. This is because this field type serves as the input for the cross-attention module later on. If this field type were not utilized, the transformer would reach the entire neighboring set, leading to inferior performance compared to applying the transformer individually for each point and then applying it over the points along the ray. A similar observation is made in [58], which states that the two-stage transformer outperforms the one-stage transformer. Using the field type corresponding to the regular representation of the translation as the input, the transformer from rays to rays is equivalent to performing a transformer for each point, respectively, as explained in the following section.

In Eq. 18, we already provide the solution of the kernel. We give a detailed explanation in this case and show that it is equivalent to performing convolution from rays to rays with output field types corresponding to irreducible representations, followed by applying Inverse Fourier Transform. Given that the input field is a scalar field, we have ω 1 in = 0 and ω 2 in = 0 . When considering an output field type of ( ω 1 out , reg ) , where reg represents the regular representation of translation, the convolution can be expressed as follows:

From the above equation, we can intuitively find that when the output field corresponds to the regular representation of the translation, the convolution happens at every point along the ray, respectively.

We can treat f ( ω 1 out ,reg ) out as a function over R , and for any ω ∈ R we apply the Fourier Transform to f ( ω 1 out ,reg ) out :

source view 1

target ray

source view 2

where κ ′ 2 = f ( d ( η, s ( x ) -1 y ) , ∠ ([0 , 0 , 1] T , d s ( x ) -1 y )) e -iωg ( s ( x ) -1 y ) , which is exactly the kernel corresponding to ω 2 out = ω and ω 2 in = 0 as stated in Eq. 14. Therefore, we know that the field corresponding to the irreducible representation of the translation can be treated as the Fourier coefficients of the field corresponding to the regular representation. We can first obtain the features of different irreducible representations attached to the ray and subsequently apply the Inverse Fourier Transform to get the features for points along the ray,as shown in figure 26.

I.2 Cross-attention over Rays

The feature that generates the query in the transformer is the feature attached to the target ray, whose feature type corresponds to the regular representation of the translation. The feature that generates the key and value in the transformer is attached to the neighboring rays in the source view, whose

F-1

feature type corresponds to the scalar field. The output is the feature attached to the target ray, whose feature type corresponds to the regular representation. Therefore, the transformer becomes:

where

In the equations above, κ k and κ v are the kernels derived in Ex. 9 Eq. 16, C is the equivariant weight matrix satisfying Eq. 31.

The expression above indicates that the feature types of both key and value correspond to the regular representation of translation, as well as the feature type of the query. Moreover, the transformer operates on each point along the ray independently. It should be noted that the features ( f k ) t , ( f q ) t , ( f v ) t and ( f in 2 ) t may have multiple channels and may consist of different types of features corresponding to various representations of SO (2) . The inner product ⟨· , ·⟩ can only happen in the field type of the same representation of SO (2) . This allows for the implementation of a multi-head attention module, where each head can attend to a specific type of feature and multiple channels.

I.3 Self-attention over Points Along the Ray

After the cross-attention over rays, we get the features of the points along the ray, i.e., the feature attached to the ray corresponding to the regular representation of translation. SE (3) acts on the feature f ′ attached to the point along the ray as mentioned in Eq.15 :

where ρ 1 is the group representation of SO (2) .

We will apply the self-attention model to these points along the same ray. For two points x 1 and x 2 on the same ray ( d , x 1 × d ) , one can observe that for the same type of feature, ⟨ ( L g f ′ )( x 1 , d ) , ( L g f ′ )( x 1 , d ) ⟩ = ⟨ f ′ ( g -1 x 1 , R g -1 d ) , f ′ ( g -1 x 1 , R g -1 d ) ⟩ , which makes attention weight invariant, the transformer could be formulated as:

where

and x and y are the points along the same ray with direction d , we can denote x as ( x , d ) and y as ( y , d ) , d ( x, y ) is the signed distance ⟨ d , y -x ⟩ , c k , c v are arbitrary functions that take signed distance as the input and output complex values and c q is an arbitrary constant complex. It should be noted that the features f k , f q , f v , and f in may have multiple channels and consist of different types of features corresponding to various representations of SO (2) , the inner product ⟨· , ·⟩ can only happen in the same type of field. This allows for implementing a multi-head attention module, where each head can attend to a specific feature type and multiple channels. Here, f l k denotes the type -l feature in feature f k , f l v represents the type -l feature in feature f v , f l q denotes the type -l feature in feature f l , and ( f in ) l represents the type -l feature in feature f in .

Note that this transformer architecture also follows the general format of the transformer in Eq. 4. We only simplify the kernel κ k , κ v to be trivial equivariant kernels.

To obtain a scalar feature density for each point, the feature output of each point can be fed through an equivariant MLP, which includes equivariant linear layers and gated/norm nonlinear layers. These layers are similar to the ones used in [62] and [63].

J 3 D Reconstruction Experiment

J.1 Generation of the Dataset

The I dataset is obtained by fixing the orientation of the object as well as the eight camera orientations. With the object orientation fixed, we can independently rotate each camera around its optical axis by a random angle in a uniform distribution of ( -π, π ] to obtain the Z dataset. For the R dataset, we rotate every camera randomly by any rotation in SO (3) while fixing the object. The equivariance stands with the content unchanged. Therefore, in practice, we require that the object projection after the rotation does not have new parts of the object. We satisfy this assumption by forcing the camera to fixate on a new random point inside a small neighborhood and subsequently rotate each camera around its optical axis with the uniformly random angle in ( -π, π ] . We generate the Y dataset by rotating the object only with azimuthal rotations while keeping the camera orientations the same. The SO (3) dataset is generated by rotating the object with random rotation in SO (3) with the orientations of cameras unchanged, which will potentially result in new image content. Equivariance is not theoretically guaranteed in this setup, but we still want to test the performance of our method.

J.2 Implementation Details

We use SE (2) equivariant CNNs to approximate the equivariant convolution over the rays. We use the same ResNet backbone as implemented in [30] that is equivariant to the finite group C 8 , which we find achieves the best result compared with other SE (2) equivariant CNNs. We use a similar pyramid structure as [69] that concatenates the output feature of every block. Since every hidden feature is the regular representation, in the final layer we use 1 × 1 SE (2) -equivariant convolutional layers to transfer the hidden representation to scalar type.

For the fusion from the ray space to the point space model, we use one layer of convolution and three combined blocks of updating ray features and SE (3) transformers. For the equivariant SE (3) multi-head-attention, we only use the scalar feature and the vector (type-1) feature in the hidden layer. The kernel matrix includes the spherical harmonics of degrees 0 and 1. We also concatenate every output point feature of every block as in the 2 D backbone. Since the output feature of every block includes the vector feature, we transfer it to the scalar feature through one vector neuron layer and the inner vector product. We use the same weighted SDF loss as in [69] during training, which applies both uniform and near-surface sampling. We report the number of parameters and floating-point operations (FLOPs) of our 2 D backbone and light fusion networks in Fig. 27 and Fig. 28 respectively.

J.3 Discussion of Results

There is still a performance gap between I/I and I/Z . Although SE (2) equivariant networks are theoretically strictly equivariant, the error in practice is introduced by the finite sampling of the image and the pooling layers. Additionally, we use the ResNet that is equivariant to C 8 approximation of SO (2) , which causes this gap but increases the whole pipeline performance in the other tasks. There is no significant difference between I/Z and I/R , which shows that approximating the spherical field convolution by SE (2) equivariant convolution is reasonable in practice.

J.4 Qualitative Results

Figure 29 shows a qualitative result for the chair category. There are more qualitative results shown in Fig. 37, Fig. 38, and Fig. 39.

J.5 Ablation Study

First, we replace the SE (2) CNNs backbone with the conventional CNNs to test the effectiveness of SE (2) CNNs. Secondly, we remove the equivariant convolution/transformer part and use trivial aggregation (max-pooling) combined with MLP. Finally, we run an equivariant convolution and transformer without using the type-1 (vector) feature while keeping the number of parameters similar to our model.

FvOR

DISN

Ours

GT

I/1

Y/Y

Z

I/Z

R

I/R

R/R

S0(3)

I/S0(3)

SO(3)/S0(3)

Table 3 summarizes the result on the chair category, which illustrates that in the I/I and Y/Y trials, SE (2) CNN is less expressive than traditional CNN, but it contributes to the equivariance of our model looking at the results of I/Z , I/R , and Y/SO (3) . Equivariant ray convolution and transformer improve both the reconstruction performance and the equivariance outcome. We also compare the ray convolution and transformer with the models operating only on scalar features without vector features, and again, we see a drop in performance in every setting, proving the value of taking ray directions into account.

We also compare to a baseline where the ray difference information is encoded in the feature explicitly. Most models that encode ray directions aim at rendering, like IBRnet. Here, we modified IBRnet (Fig.2 of IBRnet paper) to query 3D points only for their SDF value instead of querying all densities along the ray that would be necessary for rendering. We replaced the ray direction differences with the ray directions themselves because we use a query point and not a query ray. We report in table 4 IoU result for Y/Y and Y/SO(3) (where Y is augmentation only along the vertical axis) for two models - IBRNet with conventional CNNs as 2D backbone and IBRNet with SE(2)-equivariant CNNs as 2D backbone. For the SO (3) setting, we rotate the whole 8 cameras with the same rotation, which is equivalent to rotating the object with the inverse rotation, and we use the object canonical frame to encode the ray information.

The baseline is not equivariant: It explicitly uses the ray directions as inputs to MLPs. Ray directions or their differences change when the coordinate system is transformed, breaking, thus, equivariance. Table 4 demonstrates that our model is more resilient to object rotations. We can enhance equivariance

by using SE(2) equivariant modeling, and our model outperforms the baseline in the Y/Y setting. We believe that the transformer in our model is responsible for the performance improvement.

K Neural Rendering Experiment

K.1 Experiment Settings Discussion

Two experiment settings illustrate our model's equivariance: I/I and I/SO (3) . I/I is the canonical setting, where we train and test the model in the same canonical frame defined in the dataset. I/SO (3) is that we test the model trained in the canonical frame under arbitrary rotated coordinate frames, which means that all the camera poses in one scene are transformed by the same rotation without changing their relative camera poses and relative poses between the camera and the scene, which doesn't change the content of the multiple views. The reason we don't apply translation to the cameras is that there exists a depth range for points sampling in the model and the comparing baseline [61], which effectively mitigates the impact of translation.

We should note that the SO (3) setting in this experiment setting differs from R and SO (3) settings in reconstruction. R changes the relative pose of the cameras, and each image is transformed due to the rotation of each camera without altering the content, i.e., the sampling of the light field is nearly unchanged. The R setting aims to demonstrate that replacing the conventional method with ray-based convolution can get rid of the canonical frame for each view.

SO (3) in reconstruction is to rotate the object pose randomly without changing the pose of the camera, which is equivalent to transforming the cameras by the inverse rotation but fixing the object, resulting in changes in the relative poses between the camera and the object, the content of the image and, therefore, the sampling of the light field. This setting shows that even for non-theoretically equivariant cases, our model in reconstruction still demonstrates robustness.

In the rendering experiment using the SO (3) setting, each image itself is not transformed, unlike the R setting in the reconstruction. The content of the images remains unchanged, including the light field sampling, unlike the SO (3) setting in the reconstruction. Since each image is not transformed, even if the conventional 2 D convolution is applied to the image, the scalar feature attached to the ray is not altered, and the light feature field sampling remains the same up to the transform of the coordinate frame. This setting was used to demonstrate that our model is SE (3) -equivariant when the input is the scalar light feature field.

K.2 Implementation Details

As described in the paper, we use a similar architecture as [61], where we replace the aggregation of view features by equivariant convolution and equivariant transformer over rays. In equivariant convolution, the input is scalar feature field over rays, which means that ω 1 in = 0 and ω 2 in = 0 ; for the output field, we use regular representation of translation as described in Sec. 3.3 , and we use ω 1 out = 0 , 2 1 , · · · , 2 7 for group representation of SO (2) , each field type has 4 channels. In equivariant transformer over rays, we update the key and value before going to the attention module in the experiment; the specific operation is that we concatenate key f k and query f q , we concatenate

Ground Truth

IBRNet(I)

IBRNet(SO(3)) Ours(l)

Ours(SO(3))

Ground Truth

IBRNet(I)

IBRNet(SO(3)）Ours(l)

Ours(SO(3))

f v and query f q , and then we feed the concatenated key and value into two equivariant MLPs (equivariant linear layers and gated/norm nonlinear layers, similar to the ones used in [62]) to get the newly updated key and updated value, which will be fed into attention module. In line with [61], our approach does not involve generating features for the color of every point. In our implementation, we directly multiply the attention weights obtained from the softmax operator in the transformer with the corresponding colors in each view to perform color regression.

Wereplace the ray transformer with the equivariant transformer over the points along the ray; the input features comprise the feature types corresponding to the group representations ω in = 0 , 2 1 , · · · , 2 7 for SO (2) . Each feature type has 4 channels; the output comprises the same feature type, and each type has 2 channels. We will first convert the feature into a scalar feature by an equivariant MLP (equivariant linear layers and gated/norm nonlinear layers, similar to the ones used in [62].) and then feed it into a conventional MLP to get the density. We report in Fig. 30 the number of parameters and floating-point operations (FLOPs) of the model composed of the convolution and transformers.

K.3 Qualitative Results

Fig. 32, Fig. 33, Fig. 34, Fig. 31, Fig. 35 and Fig. 36 show the qualitative results on Real-ForwardFacing [41] and Realistic Synthetic 360 ◦ [49] data. Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.

Ground Truth

IBRNet(l)

IBRNet(SO(3))

Ours(SO(3))

EEEE

FvOR

DISN

Ours

GT

I/I

I/Z

I/R

R

R/R

Y/SO(3)

SO(3)

FvOR

DISN

Ours

GT

I/I

I/Z

Z

R

R/R

Y/Y

Y

YISO(3)

S0(3)

FvOR

DISN

Ours

GT

1/1

I/Z

Z

I/R

R

R/R

Y/Y

Y

X

YISO(3)

X

SO(3)

Peer reviews:
comments: SUMMARY:
This paper presents a series of methods on SE(3) equivariant convolution and transformer, which can operate on ray space. The experimental results show that the proposed method can help establish SE(3) reconstruction of signed distance functions and neural radiance fields. I'm not an expert in equivariant networks, I feel sorry but I may only be able to provide a general review.

STRENGTHS:
- The paper looks solid in theory.
- Adequate transformation settings in the experiments demonstrate its equivariance property of the proposed network modules.

WEAKNESSES:
- The entire method description section is obscure to me.
- It is a bit hard to capture the overall idea when reading the sentences from Line 44 to Line 70.
- The qualitative results (visual comparisons) are limited.

QUESTIONS:
- What are the model sizes and training times compared to DISN (for SDF reconstruction) and IBRNet (for novel view synthesis)?

LIMITATIONS:
It looks the limitations are properly discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe aim of this paper is to build equivariance constraints into neural rendering and 3D reconstruction networks. As far as I can tell from the paper, the representation used is a mapping from a ray space to a vector space. The paper describes equivariant convolution and transformer layers that operate over the ray space. Finally, these layers are inserted into existing neural rendering and 3D reconstruction networks to impose equivariance over re-parameterization of the coordinate space in which we represent camera extrinsics.\n\nSTRENGTHS:\n1. The paper establishes a theory of equivariance in the ray space. This is an important contribution that could guide the design of future equivariant 3D scene representation approaches.\n2. Adding equivariance leads to robustness to rotations and translations in 3D reconstructions. While it does not improve the quality of neural rendering, it makes renders consistent under coordinate frame transformations.\n\nWEAKNESSES:\nOverall, the paper lacks clarity and is not self-contained *at all*. I would suggest a major rewrite that adds a background section, re-structures the methods section and adds needed details to the experiments section.\n\n1. The paper needs a background section. The background should cover at least the definition of rays (using your parameterization), ray space and the IBRNet. A background section is much more important than a 2-page introduction.\n2. The methods section is missing a clear overview at the start about what neural components we need and what equivariance constraints we want to build into these components. The methods section directly jumps into dense exposition with frequent references to the 34-page appendix.\n3. The proposed model that is actually used in the experiments section is delegated to the appendix.\n4. It is not clear why IBRNet is the only baseline for novel view reconstruction. What about NeRF [1], Equivariant NeRF [2] or SRT [3]. The proposed method should be compared with other equivariant novel view reconstruction methods.\n5. It is unclear if Section 4.2 involves novel view reconstruction or only rendering the same images in different coordinate frames. If the latter is the case, novel view reconstruction seems like a vital experiment.\n6. There are very few qualitative examples of the rendered frames, both in the main paper and in the appendix. There should be a much more extensive qualitative comparison of image rendering across several SOTA methods.\n\n**References:**\n\n[1] https://arxiv.org/abs/2003.08934\n\n[2] https://arxiv.org/abs/2006.07630\n\n[3] https://arxiv.org/abs/2111.13152\n\nQUESTIONS:\n1. How well does the proposed method perform on novel view reconstruction?\n\n2. How does the proposed method compare to Equivariant NeRF or the Scene Representation Transformer?\n\nLIMITATIONS:\nThe limitations of the proposed method are not sufficiently addressed except for a brief mention in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for leveraging geometric priors in 3D reconstruction and novel view rendering when the input views are insufficient. The authors propose learning priors from 2D images using a 2D canonical frame and a 3D canonical frame. They achieve coordinate frame equivariance by introducing an SE(3)-equivariant convolution and transformer in the 3D ray space. The paper demonstrates the efficacy of their approach in tasks such as 3D reconstruction and neural rendering using multiple views, showcasing robust results without transformation augmentation.\n\nSTRENGTHS:\nThe proposed method in this paper offers several strengths:\n\n1. Geometric Priors: The approach leverages geometric priors to enhance 3D reconstruction and novel view rendering. By incorporating prior knowledge, it improves the quality of results, particularly when the input views have limited coverage and inter-view baselines.\n\n2. Equivariance to Coordinate Frame Transformations: The method ensures equivariance to coordinate frame transformations, allowing it to handle different orientations and positions of the cameras. This enables more accurate and consistent reconstruction and rendering across various coordinate frames.\n\n3. SE(3)-Equivariant Convolution and Transformer: The introduction of SE(3)-equivariant convolution and transformer in the 3D ray space provides a powerful tool for learning and exploiting geometric priors. It allows for effective feature extraction and representation, leading to improved reconstruction and rendering results.\n\n4. Adaptability to Different Tasks: The method demonstrates adaptability to different tasks, including equivariant 3D reconstruction and equivariant neural rendering. It can be tailored to specific applications, making it versatile and applicable in various scenarios.\n\n5. Robustness without Transformation Augmentation: The approach achieves robust results even in datasets with roto-translated transformations, without the need for additional transformation augmentation techniques. This reduces the complexity and computational requirements while maintaining high-quality outputs.\n\nOverall, these strengths make the proposed method valuable for improving 3D reconstruction and novel view rendering by incorporating geometric priors and addressing challenges related to coordinate frame transformations.\n\nWEAKNESSES:\n1. I appreciate the solid math and notation the author adopted in this paper, which makes the writing theoretically sound. However, this may make the reading very difficult to follow when many mathematical explanations irrelevant to the method itself appears in the paper (like the whole paragraph from line 131-138, which in my understanding could be all moved to the appendix). I believe you could lint your writing and leave more space for the qualitative/quantitative experiments section.\n2. I appreciate the notation explanations in the appendix. However, when I first read the terms like \"type-1 features\", it confuses me and I need to look back to see what I have missed before. A more intuitive name or a brief explanation of these important terminology is required, I think. This is just a recommendation on the writing and won't change my final rating.\n3. More experiments (ablations, baselines) are well needed to demonstrate the soundness of this paper as a new 3D representation learning framework.\n\nQUESTIONS:\n1. I appreciate the soundness of your math, while in Sec 3.1, could you briefly explain, what should the convolution from rays to point/rays to rays handle and why are they necessary? Any ablation on these designs, like using any of the convolution alone in the setting of overfitting/generalized NeRF?\n\n2. This paper spends a large paragraph explaining how to achieve equivariance in the light field space using convolution and transformer. However, I still don't quite see the advantages behind it, .e.g., why in the airplane dataset, w/o the equivariance design, Fvor is still superior compared to the proposed method under R/Y/SO(3) transformations? Also, IBR-Net also surpasses the proposed method over many datasets and metrics.\n\n3. I wonder in Sec 4.2, should the original NeRF & PixelNeRF and other strong baselines that uses canonical pose be included in the comparisons? Besides, I wonder whether this method could benefit the noisy pose reconstruction setting, e.g., comparing with \"SPARF Neural Radiance Fields from Sparse and Noisy Poses\".\n\nLIMITATIONS:\nAlready discussed in the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to study the geometric priors for 3D reconstruction and neural rendering with a novel perspective of multi-view equivariance. A theoretically-sound definition of the ray neighborhoods with SE(3) is obtained with the theoretical deduction of characterizing ray space with group theory. Then, the SE(3)-equivariant convolution and cross-attention operators are presented. In the experiments, the authors evaluate the proposed mathematical framework on two tasks, multi-view 3D object reconstruction and neural rendering. For the multi-view 3D object reconstruction, the presented SE(3)-equivariant network outperforms Fvor and DISN in most settings for chars, airplanes, and cars on the ShapeNet dataset. For neural rendering, the proposed method obtains better results on the I/SO(3) setting to justify its design. Overall, I think this paper brings us some new messages to understand the ray space by using group theory.\n\nSTRENGTHS:\n1. This paper presents a theoretical perspective to understand the ray spaces for learning geometric priors for multi-view 3D reconstruction. The thorough theoretical analysis presented in this paper is new to me.\n\n2. Based on the theoretical findings, the authors presented equivariant convolution and transformer in ray space and justified their design on two tasks with positive results obtained. \n\n3. This paper is well written with a detailed appendix to understand their theory.\n\nWEAKNESSES:\n1. The core of this paper is studying the geometric relationship between rays to define the ray neighborhoods, and then, convolution and attention can be induced in the ray space. If I understand correctly, what's the relationship between the point correspondences and the neighbors in the ray space? For example, are the corresponding rays near or the same in the ray space if we have a pair of keypoint matches $x$ and $x'$ for a pair of input images? \n\n2. As for the generalized neural rendering task, the authors stated that their model queries a target ray and obtains neighboring rays\nfrom source views in the first. To my knowledge, such operations can be done by following the epipolar geometry. Thus I am curious about the difference between the operation used in this paper and the alternative solutions using epipolar geometry.\n\nQUESTIONS:\nI am confused about the sentence \"given only the relative poses of the cameras, we show how to learn priors...\". Why do authors need to highlight the relative poses in the paper? It is strange for me.\n\nLIMITATIONS:\nThe authors have discussed their limitations and broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach for reconstruction from multiple views using equivariant shape priors. The paper proposes the $SE(3)$-equivariant generalized convolution as the fundamental operation on a light field whose values may be radiance or features. Input ray features and producing output ray features and point features using two different $SE(3)$-equivariant convolutions. The paper demonstrates $SE(3)$-equivariance by obtaining robust results in roto-translated datasets without performing transformation augmentation.\n\nTo say it upfront, I do not have background in Equivariant Networks and consider the paper outside my area of expertise. I find it difficult to understand the technical details of the paper. As such, I can only provide some high-level feedback to the paper and hope the AC and other reviewers can provide more detailed evaluations.\n\nSTRENGTHS:\n1. The paper is well written and easy to follow.\n2. The experiment results show that the provided models can outperform previous methods on various tasks.\n3. This novel general equivariant representation framework for light fields can inspire further work on 3D vision and graphics tasks.\n\nWEAKNESSES:\n1. The figures in this paper could be redesigned; some of them (Figure 4, 5) are so small that it is difficult to read the text and symbols they contain. In addition, the effect shown in Figure 9 is not intuitive.\n2. The paper demonstrated good reconstruction results on Shapenet. Can the method proposed in this paper reconstruct surfaces on real datasets such as SparseNeuS.\n3. About neural rendering experiments, the paper only shows results comparing with IBRNet, which doesn't seem to be the latest method. Can the authors provide the results of more experiments? (a. Local Implicit Ray Function for Generalizable Radiance Field Representation  b. Learning to Render Novel Views from Wide-Baseline Stereo Pairs c. Neural Rays for Occlusion-aware Image-based Rendering)\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nPlease see Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-30"}