SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Equivariant Light Field Convolution and Transformer in Ray Space
Yinshuang Xu
University of Pennsylvania xuyin@seas.upenn.edu
Jiahui Lei
University of Pennsylvania leijh@seas.upenn.edu
Kostas Daniilidis
University of Pennsylvania and Archimedes, Athena RC kostas@cis.upenn.edu
3D reconstruction and novel view rendering can greatly benefit from geometric priors when the input views are not sufficient in terms of coverage and inter-view baselines. Deep learning of geometric priors from 2D images requires each image to be represented in a  $2D$  canonical frame and the prior to be learned in a given or learned  $3D$  canonical frame. In this paper, given only the relative poses of the cameras, we show how to learn priors from multiple views equivariant to coordinate frame transformations by proposing an  $SE(3)$ -equivariant convolution and transformer in the space of rays in 3D. We model the ray space as a homogeneous space of  $SE(3)$  and introduce the  $SE(3)$ -equivariant convolution in ray space. Depending on the output domain of the convolution, we present convolution-based  $SE(3)$ -equivariant maps from ray space to ray space and to  $\mathbb{R}^3$ . Our mathematical framework allows us to go beyond convolution to  $SE(3)$ -equivariant attention in the ray space. We showcase how to tailor and adapt the equivariant convolution and transformer in the tasks of equivariant  $3D$  reconstruction and equivariant neural rendering from multiple views. We demonstrate  $SE(3)$ -equivariance by obtaining robust results in roto-translated datasets without performing transformation augmentation.
Recent years have seen significant advances in learning-based techniques [67, 68, 60, 69, 61, 73, 11, 53] harnessing the power of deep learning for extraction of geometric priors from multiple images and associated ground-truth shapes. Such approaches extract features from each view and aggregate these features into a geometric prior. However, these approaches are not  $SE(3)$ -equivariant to transformations of the frame where the priors and images are defined. While view pooling or calculating variance [69, 72, 46, 73, 11] can be used to aggregate features and tackle equivariance, view pooling discards the rich geometric information contained in a multiple view setup.
In this paper, we address the problem of learning geometric priors that are  $SE(3)$ -equivariant with respect to transformations of the reference coordinate frame. We argue that all information needed for tasks like novel view rendering or 3D reconstruction is contained in the light field [6, 37]. Our input is a light field, a function defined on oriented rays in 3D whose values can be the radiance or features extracted from pixel values. We will use the term light field, and we will be specific when it is a radiance field or a feature field. Images are discrete samples of this field: the camera position determines which rays are sampled, while the camera orientation leaves the sample of the light field unchanged up to pixel discretization. We model the light field as a field over a homogeneous space of  $SE(3)$ , the ray space  $\mathcal{R}$  parameterized by the Plücker coordinates. The ray space  $\mathcal{R}$  is the space of oriented light rays, for any ray  $x \in \mathcal{R}$ , the Plücker coordinate is  $x = (d, m)$ , where  $d \in \mathbb{S}^2$  is the direction of the ray, and  $m = x \times d$  where  $x$  is any point on the ray.
We define a convolution in the continuous ray space as an equivariant convolution on a homogeneous space [18]. Since our features are not limited to scalar values, we will draw upon the tools of tensor field networks and representation theory, discussed in detail in the Appendix. In Sec. 3.1 we study the group action of  $SE(3)$  on  $\mathcal{R}$ , the stabilizer group for  $\mathcal{R}$ , and how  $SE(3)$  transforms the feature field over  $\mathcal{R}$ . In Sec. 3.4, we focus on developing the equivariant convolution in  $\mathcal{R}$ , providing analytical solutions for the kernels with the derived constraints in convolution from  $\mathcal{R}$  to  $\mathcal{R}$  and from  $\mathcal{R}$  to  $\mathbb{R}^3$ , respectively. Meanwhile, we make the kernel locally supported without breaking the equivariance. By varying the output domain of the convolution, we introduce equivariant convolutions from the ray space to the ray space and from the ray space to the  $3D$  Euclidean space.
The constraint of the kernel limits the expressiveness of equivariant convolution when used without a deep structure. In Sec 3.5, we introduce an equivariant transformer in  $\mathcal{R}$ . The equivariant transformer generates the equivariant key, query, and value by leveraging the kernel derived in the convolution, resulting, thus, in invariant attention weights and, hence, equivariant outputs. We provide a detailed derivation of two cases of cross-attention: the equivariant transformer from  $\mathcal{R}$  to  $\mathcal{R}$  and the equivariant transformer from  $\mathcal{R}$  to  $\mathbb{R}^3$ . In the first case, the features that generate the key and value are attached to source rays, while the feature generating the query is attached to the target ray. In the second case, the feature generating the query is attached to the target point.
We demonstrate the composition of equivariant convolution and transformer modules in the tasks of  $3D$  reconstruction from multi-views and novel view synthesis given the multi-view features. The inputs consist of finite sampled radiance fields or finite feature fields, while our proposed equivariant convolution and transformer are designed for continuous light fields. If an object or a scene undergoes a rigid transformation and is resampled by the same multiple cameras, the  $SE(3)$  group action is not transitive in the light field sample. This lack of transitivity can significantly impact the computation of equivariant features, mainly because the views are sparse, unlike densely sampled point clouds. Object motion introduces new content, resulting in previously non-existing rays in the light field sampling. Hence, our equivariance is an exact equivariance with respect to the choice of coordinate frame. In the 3D reconstruction task, we experimentally show that equivariance is effective for small camera motions or arbitrary object rotations and generally provides more expressive representations. In the  $3D$  object reconstruction application, we first apply an equivariant convolutional network in ray space to obtain the equivariant features attached to rays. We then apply equivariant convolution and equivariant transformer from  $\mathcal{R}$  to  $\mathbb{R}^3$  to obtain equivariant features attached to the query point, which are used to calculate the signed distance function (SDF) values and ultimately reconstruct the object. In the generalized rendering task, our model queries a target ray and obtains neighboring rays from source views. Our composition of equivariant modules is based on IBRNet [61], which consists of view feature aggregation and ray transformer. We replace the view feature aggregation in [61] with the equivariant convolution and transformer over rays and the ray transformer part with the equivariant transformerover the points along the ray to get the density and color of the point, see Sec. 3.3. We summarize here our main contributions:
(1) We model the ray space as a homogeneous space with  $SE(3)$  as the acting group, and we propose the  $SE(3)$ -equivariant generalized convolution as the fundamental operation on a light field whose values may be radiance or features. We derive two  $SE(3)$ -equivariant convolutions, both taking input ray features and producing output ray features and point features, respectively.
(2) To enhance the feature expressiveness, we extend the equivariant convolution to an equivariant transformer in  $\mathcal{R}$ , in particular, a transformer from  $\mathcal{R}$  to  $\mathcal{R}$  and a transformer from  $\mathcal{R}$  to  $\mathbb{R}^3$ .
(3) We adapt and compose the equivariant convolution and transformer module for  $3D$  reconstruction from multiple views and generalized rendering from multi-view features. The experiments demonstrate the equivariance of our models.
Equivariant Networks Group equivariant networks [15, 65, 62, 55, 63, 12, 19, 17, 22, 24, 23] provide deep learning pipelines that are equivariant by design with respect to group transformations of the input. While inputs like point clouds, 2D and 3D images, and spherical images have been studied extensively, our work is the first, as far as we know, to study equivariant convolution and cross-attention on light fields. The convolutional structure on homogeneous spaces or groups is sufficient and necessary for equivariance with respect to compact group actions as proved in [18, 1, 36]. Recently, Cesa et al. [8], Xu et al. [70] provided a uniform way to design the steerable kernel in
an equivariant convolutional neural network on a homogeneous space using Fourier analysis of the stabilizer group and the acting group, respectively, while Finzi et al. [26] proposed a numerical algorithm to compute a kernel by solving the linear equivariant map constraint. For arbitrary Lie groups, Finzi et al. [25], MacDonald et al. [39], Bekkers [4] designed the uniform group convolutional neural network. The fact that any  $O(n)$  equivariant function can be expressed in terms of a collection of scalars is shown in [59]. For general manifolds, Cohen et al. [16], Weiler et al. [64] derived the general steerable kernel from a differential geometry perspective, where the group convolution on homogeneous space is a special case. The equivalent derivation for the light field is in the Appendix. Recently, equivariant transformers drew increasing attention, in particular for 3D point cloud analysis and reconstruction [27, 48, 10, 7]. A general equivariant self-attention mechanism for arbitrary groups was proposed in [44, 43], while an equivariant transformer model for Lie groups was introduced in Hutchinson et al. [32]. We are the first to propose an equivariant attention model in the  $3D$  ray space.
Light Field and Neural Rendering from Multiple Views The plenoptic function introduced in perception [6] and later in graphics [37] brought a new light into the scene representation problem and was directly applicable to the rendering problem. Instead of reconstructing and then rendering, light fields enabled rendering just by sampling the right rays. Recently, learning-based light field reconstruction [41, 34, 5, 66, 51, 2] became increasingly popular for novel view synthesis, while [50, 54, 53] proposed non-equivariant networks in the ray space. Due to the smaller dimension of the ray space, the networks in the ray space are more efficient compared to neural radiance fields [42], which leverages volumetric rendering. Several studies [73, 61, 53, 50, 11, 38, 13, 31, 58, 20] concentrate on generalizable rendering. These works are similar to ours in that they obtain the  $3D$  prior from the  $2D$  images, but they are not equivariant since they explicitly use the coordinates of the points or the rays in the network.
The most related equivariant rendering approaches to us are [21, 47, 45]. The equivariance in the paper [21] is not the equivariance we mean in this work: [21] enforces the geometric consistency via a loss function. [47] is not strictly equivariant and it depends on the assumption of upright views and the camera ID embeddings, which results in its non-equivariance to camera permutation. It achieves data-driven equivariance by randomly choosing the first camera frame as the canonical frame. While [45] addresses the frame problem through relative pose, it is not theoretically equivariant in cases of individual camera rotations around their axes or minor individual rotations accompanied by small content changes. We want to emphasize that our central contribution is to propose an equivariant convolution and transformer on ray space, which can be integrated into a wide range of 3D learning models.
Reconstruction from Multiple Views Dense reconstruction from multiple views is a well-established field of computer vision with advanced results even before the introduction of deep learning [28]. Such approaches cannot take advantage of shape priors and need a lot of views to provide a dense reconstruction. Deep learning enabled semantic reconstruction, i.e., the reconstruction from single or multiple views by providing the ground-truth 3D shape during training [14, 67, 68, 40]. These approaches decode the object from a global code without using absolute or relative camera poses. Regression of absolute or relative poses applied in [35, 71, 56, 69, 72, 57, 3, 46, 33, 20] is non-equivariant.
In this section, we will first introduce the (feature) field on the ray space and  $3D$  Euclidean space, respectively, and the corresponding  $SE(3)$  group actions on the values of the fields in Sec. 3.1. To offer readers a holistic grasp—from a broad overview down to the intricate specifics and from foundational concepts to advanced techniques—we will present the reconstruction and the generalized rendering with the neural components of their architectures (convolutional and attentional) and their inputs and outputs in Sec. 3.2 and Sec. 3.3. Following that, we expose our central contribution: equivariant convolution and attention in ray space in Sec. 3.4 and Sec. 3.5.
3.1 Feature field on Ray Space and  $3D$  Euclidean Space
3.1.1 Ray Space
The ray space is the space of oriented light rays. As introduced in the introduction and App. Ex. 1, we use Plücker coordinates to parameterize the ray space  $\mathcal{R}$ : for any ray  $x \in \mathcal{R}$ ,  $x$  can be denoted as  $(d, m)$ , where  $d \in \mathbb{S}^2$  is the direction of the ray, and  $m = x \times d$  is the moment of the ray with  $x$
Figure 1: Feature attached to rays: we show the scalar feature and type-1 feature. When  $\rho_{2}$  is the trivial representation, tensor features can be viewed in the plane orthogonal to the ray (the blue plane). When rotations act on the feature field, the scalar feature only changes position as attached to the rays:  $(\mathcal{L}_g f)(x) = f(g^{-1}x)$ ; while the type-1 feature changes position and is itself rotated:  $(\mathcal{L}_g f)(x) = \rho (\mathrm{h}(g^{-1},x)^{-1})f(g^{-1}x)$ , where  $\rho (\gamma ,t) = e^{i\gamma}$ .
Figure 2: Features attached to points: we show scalars and vectors (type-1 features). The black dot in the figure is the point, and the square and the vectors are the scalar and type-1 features attached to the point. When  $g \in SE(3)$  acts on the feature field, we will see that the scalars are kept the same while the attached position is rotated, and the vector features change their position and alter their direction.
being a point on the ray. Then any  $g = (R, t) \in SE(3)$  acts on the ray space as:
$$
g x = g (\boldsymbol {d}, \boldsymbol {m}) = (R \boldsymbol {d}, R \boldsymbol {m} + \boldsymbol {t} \times (R \boldsymbol {d})). \tag {1}
$$
The ray space  $\mathcal{R}$  is a homogeneous space with a transitive group action by  $SE(3)$ . Given the origin in the homogeneous space as  $\eta = ([0,0,1]^T,[0,0,0]^T)$  (the line representing  $z$ -axis), the stabilizer group  $H$  that leaves  $\eta$  unchanged is  $SO(2)\times \mathbb{R}$  (the rotation around and translation along the ray). The ray space is, thus, isomorphic to the quotient space  $\mathcal{R}\cong SE(3) / (SO(2)\times \mathbb{R})$ . We parameterize the stabilizer group  $H$  as  $H = \{(\gamma ,t)|\gamma \in [0,2\pi),t\in \mathbb{R}\}$ .
We follow the generalized convolution derivation for other homogeneous spaces in [18], which requires the use of principal bundles, section maps, and twists [29] explained in the appendix section A.2 and onwards.  $SE(3)$  can be viewed as the principal  $SO(2)\times \mathbb{R}$ -bundle, where we have the projection  $p:SE(3)\to \mathcal{R}$ , for any  $g\in SE(3)$ ,  $p(g) = g\eta$ ; a section map  $s:\mathcal{R}\rightarrow SE(3)$  can be defined such that  $p\circ s = id_{\mathcal{R}}$ . In App. Example 6, we elaborate on how we define the section map from the ray space to  $SE(3)$  in our model. Generally, the action of  $SE(3)$  induces a twist as  $gs(x)\neq s(gx)$ . The twist can be characterized by the twist function  $\mathrm{h}:SE(3)\times \mathcal{R}\to SO(2)\times \mathbb{R}$ ,  $gs(x) = s(gx)\mathrm{h}(g,x)$ , we provide the twist function in our model and its visualization in App. Example 6.
3.1.2 Light Field
The light field can be modeled as a function from the ray space to a vector space,  $f: \mathcal{R} \to V$ . We also need to define the  $SE(3)$  group action on the values of that field. Since the group action will be on a vector space  $V$ , we will use the corresponding group representation of the stabilizer group  $\rho: SO(2) \times \mathbb{R} \to GL(V)$ , see details in App. A.3. For example, a light field can be a radiance field  $f$  that maps the ray space of oriented rays to their observed radiance (RGB)  $f: \mathcal{R} \to \mathbb{R}^3$  which is a concatenation of three scalar fields over  $\mathcal{R}$ . The group representation  $\rho$  in this case is the identity and  $g \in SE(3)$  acts on the radiance field  $f$  as  $(\mathcal{L}_g f)(x) = f(g^{-1}x)$ , shown as the scalar features in Fig. 1. Given that the stabilizer  $H = SO(2) \times \mathbb{R}$  is a product group, the stabilizer representation can be written as the product  $\rho(\gamma, t) = \rho_1(\gamma) \otimes \rho_2(t)$ , where  $\rho_1$  is the group representation of  $SO(2)$  and  $\rho_2$  is the group representation of  $\mathbb{R}$ . If the light field is a feature field (Fig. 1) with  $\rho_2$  being the identity representation and  $\rho_1$  corresponding to a type-1 field,  $\rho_1(\gamma) = e^{i\gamma}$ , then type-1 features change position and orientation when  $g \in SE(3)$  acts on it. Having explained the examples of scalar (type-0) and type-1 fields, we introduce the action on any feature field  $f$  as [18]:
$$
\left(\mathcal {L} _ {g} f\right) (x) = \rho \left(\mathrm {h} \left(g ^ {- 1}, x\right) ^ {- 1}\right) f \left(g ^ {- 1} x\right), \tag {2}
$$
where  $\rho$  is the group representation of  $SO(2) \times \mathbb{R}$  corresponding to the space  $V$ , determined by the field type of  $f$ , and  $\mathrm{h}$  is the twist function introduced by  $SE(3)$  as shown in App. Example 6.
3.1.3 Feature Field on  $\mathbb{R}^3$
$\mathbb{R}^3$  is also a homogeneous space of  $SE(3)$  like the ray space  $\mathcal{R}$ , with the stabilizer group as  $SO(3)$ , as stated in App. Example 2. For any  $g = (R, t) \in SE(3)$ , it acts on the field  $f$  over  $\mathbb{R}^3$  also follows
Figure 3: The pipeline of equivariant  $3D$  reconstruction: Firstly, we obtain the feature field over the ray space. Secondly, we perform an equivariant convolution from ray space to point space. Thirdly, we apply a  $SE(3)$  equivariant cross-attention module to obtain an equivariant feature for a query.
[18]:
$$
\left(\mathcal {L} _ {g} f\right) (x) = \rho (R) f \left(R ^ {- 1} \left(x - t\right)\right)
$$
where  $\rho$  is the group representation of  $SO(3)$ , since the twist function can be independent of the  $3D$  position due to the fact that  $SE(3) = \mathbb{R}^3 \rtimes SO(3)$  is a semidirect product group as stated in App. Example 4. The feature field over  $\mathbb{R}^3$  and the corresponding group action is also used in [55, 63]. Fig. 2 visualizes the scalar feature ( $l_{out} = 0$ ) and vector feature ( $l_{out} = 1$ ) attached to one point, offering an intuitive understanding of the feature field over  $\mathbb{R}^3$ .
Given the feature field on the ray space and  $3D$  Euclidean space and the corresponding group actions of  $SE(3)$ . We will show two 3D multi-view applications of the equivariant convolution and transformer:  $3D$  reconstruction and generalized neural rendering. In each application, we start with the specific definition of equivariance and then outline the corresponding pipeline.
3.2 Equivariant 3D Reconstruction
The radiance field serves as the input for the  $3D$  reconstruction, which ultimately generates a signed distance field (SDF) denoted by the function  $e:\mathbb{R}^3\to \mathbb{R}$ . As aforementioned, the radiance field is the multi-channel scalar field over  $\mathcal{R}$ , while SDF is the scalar field over  $\mathbb{R}^3$ . A  $3D$  reconstruction  $\Phi :\mathcal{F}\rightarrow \mathcal{E}$ , where  $\mathcal{F}$  denotes the space of radiance fields and  $\mathcal{E}$  denotes the space of signed distance fields, is equivariant when for any  $g\in SE(3)$ , any  $x\in \mathbb{R}^3$ , and any  $f\in \mathcal{F}$ ,  $\Phi (\mathcal{L}_g f)(x) = \mathcal{L}_g^\prime (\Phi (f))(x)$ , where  $\mathcal{L}_g$  and  $\mathcal{L}_g^\prime$  are group actions on the light field and the SDF, respectively. Specifically, as  $f$  and  $e$  are scalar fields,  $(\mathcal{L}_g f)(x) = f(g^{-1}x)$  for any  $x\in \mathcal{R}$ , and  $(\mathcal{L}_g^\prime e)(x) = e(g^{-1}x)$  for any  $x\in \mathbb{R}^3$ .
In practice, we have a finite sampling of the radiance field corresponding to the pixels of multiple views  $V = \{f(x)|x\in L_V\}$ , where  $L_{V}$  denotes the ray set of multi-views and  $f\in \mathcal{F}$  is the radiance field induced by multi views sample from. The 3D reconstruction  $\Phi$  is equivariant when for any  $g\in SE(3)$  and any  $x\in \mathbb{R}^3$ :  $\Phi (\pmb {g}\cdot \pmb {V})(\pmb {x}) = \Phi (\pmb {V})(\pmb{g}^{-1}\pmb {x})$ . If we denote  $V$  as  $(L_V,f)$ ,  $g\cdot V = (g\cdot L_V,\mathcal{L}_gf)$ , where  $g\cdot L_V$  is  $g$  acting on the rays defined Eq. 1.
We achieve equivariance using three steps as illustrated in Fig. 3: (1) the transition from pixel colors to a feature-valued light field (equi-CNN over rays), (2) the computation of features in  $\mathbb{R}^3$  from features on the ray space by equivariant convolution from  $\mathcal{R}$  to  $\mathbb{R}^3$ , and (3) the equivariant transformer with the query generated by the feature on the point we want to compute SDF and key/value generated by features on rays. Note that we need (3) following (2) because the output feature of a single convolution layer is not expressive enough due to the constrained kernel. For the detailed practical adaption of the convolution and transformer in  $3D$  reconstruction, please see the App. B, where we approximate the intra-view with  $SE(2)$  equivariant convolution.
3.3 Generalized Neural Rendering
The light feature field  $f_{in}:\mathcal{R}\to V$  serves as the input for neural rendering, which ultimately generates the light field  $f:\mathcal{R}\rightarrow \mathbb{R}^3$ , a multi-channel scalar field over  $\mathcal{R}$ . A neural rendering  $\Psi :\mathcal{I}\rightarrow \mathcal{F}$ , where  $\mathcal{I}$  denotes the space of the light feature fields and  $\mathcal{E}$  denotes the space of the light field, is equivariant when for any  $g\in SE(3)$ , any  $x\in \mathcal{R}$ , and any  $f_{in}\in \mathcal{I}$ ,  $\Psi (\mathcal{L}_gf_{in})(x) = \Psi (f_{in})(g^{-1}x)$ , where  $\mathcal{L}_g$  is the group operator on the light feature field  $f_{in}$ , as shown in Eq. 2 depending on the feature type. In the experiment of this paper, the input light feature field is
Figure 4: The pipeline of equivariant neural rendering. Firstly, we obtain the features of the points along the target ray through convolution over rays. Secondly, we apply the equivariant cross-attention module to obtain features for generating the color of the points. Finally, we use equivariant self-attention over the points along the ray to obtain features for generating the density of points.
scalar, i.e.,  $\mathcal{L}_g f_{in}(x) = f_{in}(g^{-1}x)$ . Similar to reconstruction, in practice, the neural rendering  $\Psi$  is equivariant when for any  $g \in SE(3)$  and any  $x \in \mathcal{R}$ :  $\Psi(g \cdot V)(x) = \Psi(V)(g^{-1}x)$ , where  $V = \{f_{in}(x) | x \in L_V\}$ , and if we denote  $V$  as  $(L_V, f_{in})$ , then  $g \cdot V = (g \cdot L_V, \mathcal{L}_g f_{in})$ ,