$SE(3)$ Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation
Authors: Yinshuang Xu
Dian Chen
Katherine Liu
Sergey Zakharov
Rares Andrei Ambrus
Kostas Daniilidis
Vitor Campagnolo Guizilini
Venue: NeurIPS 2024 poster (2024)
Keywords: $SE(3)$ Equivariance
Stereo Depth Estimation
Primary area: machine_vision
TL;DR: We propose an $SE(3)$ equivariant model with spherical harmonics ray embeddings and demonstrate its effectiveness in the task of generalized stereo depth estimation.

Abstract: Incorporating inductive bias by embedding geometric entities (such as rays) as input has proven successful in multi-view learning. However, the methods adopting this technique typically lack equivariance, which is crucial for effective 3D learning. Equivariance serves as a valuable inductive prior, aiding in the generation of robust multi-view features for 3D scene understanding. In this paper, we explore the application of equivariant multi-view learning to depth estimation, not only recognizing its significance for computer vision and robotics but also addressing the limitations of previous research. Most prior studies have either overlooked equivariance in this setting or achieved only approximate equivariance through data augmentation, which often leads to inconsistencies across different reference frames. To address this issue, we propose to embed $SE(3)$ equivariance into the Perceiver IO architecture. We employ Spherical Harmonics for positional encoding to ensure 3D rotation equivariance, and develop a specialized equivariant encoder and decoder within the Perceiver IO architecture. To validate our model, we applied it to the task of stereo depth estimation, achieving state of the art results on real-world datasets without explicit geometric constraints or extensive data augmentation.

SE(3) Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation
Yinshuang Xu University of Pennsylvania xuyin0seas. upenn. edu
Dian Chen Toyota Research Institute dian. wwwwehhennnneeeee chenetri.global global
Katherine Liu Toyota Research Institute katherine. liuctri, global
Sergey Zakharov Toyota Research Institute sergey zakharov@tri.ccoommo lareooovrriiggbbaall global
Rares Ambrus Toyota Research Institute rares wwwrusmtri.ggoooooo
Kostas Danilidiis
Vitor Guizilini Toyota Research Institute
University of Pennsylvania kostas@cis upenn. edu
vitor cttorrguizzliii@tt.. global
Abstract
Incorporating inductive bias by embedding geometric entities (such as rays) as input has proven successful in multi-view learning. However, the methods adopting this technique typically lack equivariance, which crucial for effective 3D Iearning. Equivariance serves robust multi-view features for 3D scene understanding. In this paper, we explore the application of equivariant multi-view learning to depth estimation, not only recognizing its significance for computer vision and robotics but also addressing the limitations of previous research. Most prior studies have either overlooked equivariance in this setting or achieved only approximate equivariance through data augmentation, which often leads to inconsistencies across different reference frames. To address this issue, we propose to embed SE(3) equivariance into the Perceiver IO architecture. We employ Spherical Harmonies for positional encoding to ensure 3D rotation equivariance, and develop specialized equivariant encoder and decoder within the Perceiver IO architecture. To validate our model, we applied it to the task of stereo depth estimation, achieving state of the art results on real-world datasets without explicit geometric constraints extensive data augmentation.
valuable inductive prior, aiding in the generation of
Introduction
Equivariance valuable property in computer vision, leveraging various symmetries reduce sample and model complexity while boosting generalization. It has seen broad application in fields such as 3D shape analysis [8 52 13), panoramic image prediction [10] [4 19), and robotics
46
25 42 2 In particular, there is an increasing interest in equivariant scene representation from
multiple viewpoints equivariant representations are desirable for their robustness and efficiency.
43 (55, as the multi-view setting fundamental challenge in the field and
Meanwhile, multi-view depth estimation has always been topic in computer vision. Previous works [1 26 leverage the explicit geometric constraint to construct the feature cost-volume for depth prediction. Recently, the paradigm of combining implicit representations with generalist
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

architectures has been widely adopted and gaining success. Inserting inductive bias via the embedding of geometric entities (rays) in the multi-view setting [58
47 44 has become popular. Notably, in
multi-view depth estimation, Yifan et al. [57] effectively combined geometric epipolar embeddings with image features for stereo depth estimation, outperforming traditional methods that depend on explicit geometric constraints. State-of-the-art work by [23] integrated multi-view geometric embeddings with image features for video depth estimation. These methods show that the implicit multi-view geometry learned by the Perceiver IO architecture, which is more efficient general architecture compared to the vision transformer [4), can improve upon approaches that rely on traditional explicit geometric constraints, such cost volumes, bundle adjustment, and projective geometry. However, the implicit view geometry promoted by the Perceiver 10 architecture lacks equivariance. This limitation becomes apparent when transforming the coordinate frame representing input geometry, such as camera poses, ray directions, or 31) coordinates. These transformations change the input in such way that non-equivariant architectures are unable achieve the same results, as shown in Figure
Although 23 have tried to approximate equiv- ariance through extensive data augmentation, achieving true equivariance at architectural level remains an ongoing challenge. In this pa- per, we propose to embed equivariance with re- spect to SE(3) transformation of the global co- ordinate frame, i.e., gauge equivariance, to the Perceiver 10 model. We substitute traditional Fourier positional encodings for the ray embed- ding with Spherical Harmonics, which are more suitable to represent 3D rotations. We custom- develop a SE(3) equivariant attention module seamlessly interact with different types of equivariant features. This is achieved using combination of invariant attention weights and equivariant fundamental layers. During the de- coding stage. this equivariant latent space is disentangled into the equivariant frame and in- variant global features. Our approach not only simplifies the integration of existing modules without requiring a specialized design, but also allows the network to focus on effective scene analy- sis via an invariant latent space, reducing the effects of global transformations. The equivariant frame used to "standardize" the query ray, serving as an invariant input for the decoder. This method ensures that both sets of inputs for the decoder of the decoder used. Consequently, we can employ the conventional Perceiver 10 decoder in our equivariant framework. In summary, our key contributions are follows:
Prediction
SBIS)
Equivariont
Equisariant Percsiver IO
Figure I: Given a sparse set of posed images (red), the task is to estimate depth for a novel view- point (bluc). The Perceiver I0 struggles to ac- curately predict depth when the reference frame (gray) changes, equivalent to an inverse transforma- tion applied the object and cameras. In contrast, our model delivers the consistent result due to its equivariant design.
invariant, leading to invariant output regardless
We integrate SE(3) equivariance into multi-view depth estimation model by design, using spherical harmonics positional encodings for ray embeddings. as well as specialized equivariant encoder.
By leveraging the equivariant learned latent space, we introduce a novel scene representation scheme for multi-view settings, featuring disentangled equivariance frame and an invariant scene representation.
We assess our model's ability to learn 3D structures through wide-baseline stereo depth estimation. Our model delivers state-of-the-arr results this task, significantly surpassing the non-equivariant baseline.
2 Related Work
Equivariant Networks Equivariant Networks are garnering interest vision for their efficiency
and powerful inductive bias. These networks can be categorized by the data structures over which
they operate, spanning 2D images [15 (39, graphs 45 38) 3D point clouds (0 5) manifolds [7] and spherical images
[1 77. From an architectural perspective, methods can also be classified by the

---

image embedddnng per pic
Global inverient Intert Global equlvariant latent
Ray embedding Foureer Prxition embedding (Fouiee)
Fay emoodding (SPH)
Repressniziinn
Position entwdding (SPH)
overy
camera
(c)
Equivariant
EquiMLL
Equivariant frams
Perceiver 10
Encoder
K
Decoder
leross-attention
Input matrix
(8)
(b) Global invariant latent
(d)
Figure 2: Our proposed Equivariant Perceiver IO (EPIO) architecture. (a) We take as input the concatenation of per-pixel image, ray, and camera embeddings, the latter two calculated using spherical harmonics. (b) The output of our equivariant encoder is global latent code, including both global invariant and equivariant components. From those, extract equivariant reference frame through an equivariant MLP, while simultancously obtaining invariant latents through inner product. (c) When query camera is positioned in this equivariant reference frame, its pose becomes invariant, which enables the use of conventional Fourier basis to encode it. (d) Given an invariant latent and invariant pose, we use conventional Perceiver 10 decoder generate predictions for each query ray.
steerable convolution on homogeneous In the context of this paper, we highlight significant (E) equivariant transformer works. Fuchs et al. [22] first introduced an SE(3) equivariant transformer for point clouds, using steerable kernels for transformers and focusing on local features in point clouds. Liao and Smidt [33]. Liao et al. (34] adopted a mnessage-passinn architecture for 3D equivariant transformers in point clouds. Xu et al. [55] applied similar techniques for ray space, Our approach differs by using direct input-level positional encodings, rather than modifying the kernel with relative poses. We learn global, oon-hicarce representation. Safin et pairwise relative poses in self-attention with conventional attention module, requiring quadratic computation and lacking compact scene representation, unlike our method. Closely related to our work, Assaad et al. 1 proposed vector neuron transformers embedded in the Perceiver 10 encoder for point clouds. However, they replace the original latent array with a learnable transformation, did not use spherical harmonics for equivariant positional encoding, design an equivariant decoder within the Perceiver IO framework. We treat the original latent array as invariant, and learn disentangled representation for the decoder with versatile queries. Esteves et al. [18] uses spherical harmonics for positional encoding, primarily to enhance spherical function learning, not for equivariance.
tools they rely on, such as group convolution 8 16 36
20
spaces [53 5 2 54], and recently transformers 40
33
24
inserts
Implicit Multi-View Geometry Even in the age of deep learning, traditional multi-view stereo methods like COLMAP (21) are still widely used for woweture-frommmotiee accurate but slow due to complex post-processing steps. To speed things up while maintaining accuracy, learning-based methods adapt traditional cost volume-based techniques for depth estimation
These methods are
29 3 26 27. Recently, transformers [50] have become prevalent approaches, replacing CNNS in terms of popularity and performance. The Stereo Transformer [32) replaces cost volumes with attention- based matching procedure inspired by sequence modeling. IIB [57] leverages Perceiver IO
28 for generalized stereo estimation by incorporating the epipolar geometric bias into the model.
Liu et al. [35]. Chen et al. [ inject 3D geometry into the transformer akin to IIB for object detection, while Chen al. 4 learns equivariance in data-driven way. A closely related study
is DeFiNe [23). in which camera information is incorporated into Perceiver I0 and used to decode predictions from arbitrary viewpoints. However, their approach relies on data augmentation to approximate equivariance in the Perceiver IO, whereas our design inherently ensures equivariance at an architectural level.
Method
In this section we start with some preliminaries about Perceiver I0 and our baseline, Depth Field Networks (DeFiNe) (23), sfate-of-the-ie art method integrating camera geometries into Perceiver IO for multi-view depth estimation. We then outline the concept of equivariance in multi-view scenarios in Section 3.2 Given these preliminaries, in Section 3.3 we delve into the details of our proposed equivariant positional encoding for rays, Section 3.4 we elaborate on the attention mechanisms used in our model, and in Section 3.5 we describe choice of encoder parameterizattin. Finally,
3

---

in Section 3.6 we describe our decoder procedure, focusing on the task of depth estimation. The pipeline of our proposed (3) equivariant model in multi-view context is shown in Figure 2
3.1 Preliminaries: Input-level Inductive Biases to Perceiver I0
The Perceiver 10 (28) is a generalist transformer architecture that encodes input data I E RNxc into latent space R E IR RaxxCC by cross-attending I with R. Further refinement of this latent space R is achieved using self-attention layers, followed by cross-attention to decode predictions O E R N.xC. using queries Q &N,xC Many works exploit its generic nature by introducing inductive biases at an input level, namely, providing prior knowledge about the data for implicit reasoning. Specifically, DeFiNe [3 uses camera geometries to construct 3D positional encodings for the multi-view problem. Given N images { [7)) from a set of cameras with poses (7:)
uv=(1.1) from each camera center to each pixel (u, v) on image 4, and obtains positional encodings PE uvs (4) with a mapping PE((). These positional encodings are combined with image embeddings extractor to be encoded by R such that:
and intrinsics { DeFiNe caleulates 3D rays {ra ) (H-W))
{fue} from visual feature
R1 cross- vross-attm(Ro, (f @ PE(ri 4,))
Rx = (Rtf-atttnR+-1) == 2...... K
To obtain predictions for set of M novel viewpoints, rays from poses {T and intrinsics {K}}}=1 and map them to query positional encodings Q (PE(rue t,)), which will be used to decode the latent space R via cross attention: O cross-attn( Q, R In this way, prior knowledge, i.e., 3D camera geometries, is directly fed into the model as additional input features for the implicit learning of multi-view geometry.
similarly calculate 3D query
3.2 Equivariance Definition in Multiview Context
After introducing the input-level inductive bias framework, it is worth noting that the poses of the encoding cameras, as well as the query viewpoints, are defined in a global reference frame To. However, this choice of global reference frame is subject to change, and the property of cquivariance ensures that predictions remain identical under these changes. Assuming the global reference frame undergoes transform T-1 (SE(3) to T/ = T- the ray representations t). The equivariant model & should satisfy uv: Rt, + t)}, (PE(RrL Rt, + +)})
become ( (lf) Rlu + () when representing 7 = 7(R,,)
(( @ PE = ((f (D PF(r u. (,),, (PE uu: 4))).
For further details on the definition of the equivariance, please see Appendix A.2
3.3 Equivariant Positional Encoding
To ensure the positional encoding process is equivariant w.r.t transformation group G, we would like to enforce that PEpp 2)) = Φ(,, P PE(x)) for any g G, where px is the group representation
coordinate space, and x" is the group representation space. The traditional Fourier basis is translationally equivariant, as detailed in Appendix BB] Kitaev al. [30] used this to attain translational equivariance, employing a conjugate product for invariant attention. However, this approach lacks rotational equivariance.
the positional encoding
This raises key question: Are there any basis functions equivariant both 3D translations and rotations? Unfortunately, none exist. However, a common method in equivariant works for transla- tional equivariance is to subtract the center point, technique we apply in our context as illustrated in Figure 3 This enables translational invariance, leaving the model to focus solely on achieving rotational equivariance. To address the 3D rotational equivariance, we to spherical harmonics (SPH), known for their inherent rotation-equivariant properties. They offer a way to accommodate 3D rotational changes, thereby achieving FI3) equívariance.
3.3.1 Spherical Harmonics
We provide detailed introduction to Spherical Harmonies in Appendix A.4 where we also discuss how their application in previous equivariant transformers differs from their use in our model. Below,

---

embadding
enbeading PaT
Equitt
1/2
@
(b)
(a)
Invariant
('(ww) (4
Phin-tt
amrbacting
Pun
Pheppk
entedingg
fxx
embedcing
Perple my (P
ambodding 1g
hrnsiationn (E
Eqge-ic
Commenton Ft (rounee)
Figure 3: Comparison between an equivariant input embedding in our model (left) and the conven- tional input embedding in DeFiNe (right). (a) Pipeline used to generate input embeddings for the encoder, resulting in cross-attention keys and values. (b) To generate geometric information, we calculate embeddings for each ray r and camera relative position t, f; (c) The final composed embedding format includes both image embeddings, which are invariant, and geometric embeddings, which equivariant. In contrast, the conventional approach by Perceiver IO, highlighted in parts (a) and (c), integrates Fourier positional encodings with image embeddings to form the input embeddings. Furthermore, as indicated in (b), Perceiver IO utilizes each ray rav and the absolute translation tÜ for positional encoding purposes.
we present a brief overview of Spherical Harmonics for clarity. Similar to the varying frequencies of sines and cosines in Fourier series, spherical harmonies characterized by different degrees (orders), denoted as / E N. An order-I spherical harmonics, denoted as Y RR3 + 22++1 follows the transformation rule: ('Rrr) D'(R)Y'(r), Y'(r) [|r])'Y'(A, where R E SO(3), is the unit vector, D' (3) TR. (21+)) is called the Wigner-D matrix, serving the irreducible representation of SO(3) corresponding to the order 1. The Wigner-D matrix is an orthogonal matrix, that is D' (R)) = These important properties allow us to achieve equivariance in the Perceiver 10 transformer architecture.
3.3.2 Equivariant Hidden features
In our model, we embed both camera centers and viewing rays using spherical harmonies. The embedding is given by: PE 4.) Dter. (rk..) ) @ Y (4: 0)), where each part corresponds
1 Σn' t, to the same order of spherical harmonics (Figure 3). Here, L = {1, (1.2...., kmax) } and i = N
highlighting the extraction of the cameras' central position for translational invariance. Due to the properties of spherical harmonics, the positional encoding of transformed input, PE( Rr wos Rt; + t), equal to D (D((
@ YI(I, - D)) = R PE(r 44), for any rotation E SO(3) and translation E R². In other words, it guarantees that these embeddings are both rotationally equivariant and translationally invariant. The transformation of these embeddings operates by multiplying each block with its respective Wigner-D matrix.
The image remains unchanged when the reference frame is transformed, as its contents are unaffected. Mathematically, this property akin to multiplying by a 0-order Wigner Matrix, equivalent to an identity. Thus, combining image features with positional encodings (Figure 3 transform as:
(fu.. PE aoy Rl, +() = (D) R. PE(r uv:
= -R- PE t2))
In our model, the equivariant hidden features mirror the structure of our embeddings, composed as DieL Hi with subscripts indicating the feature type and L {0. 1. .lmaz): The size for each feature type H, follows (2l + 1, Ct), where 27 + 1 is the intrinsic dimension and C: is the number of channels. For more intuitive understanding, we visualize these embeddings Appendix Appendix[(Figurr (Figure 10 Similar to the input embeddings, any rotation R in 80(3) rotates the hidden features
5

---

EQU-MMUP
Output
EquHLayerNoom
Equi-PE (R))
Add
Attention Output
Inveriatt
Attention
MetMul
Matrix
Equi-PE (R)
Inveriant Inner Product
Averaged PE
Average
Equí-Linear
Equ-Linnear
Equi-Lineer
Equi-iinerr
Latent Array Re
Input
Input
Global Geometric Latent @ G
(a)
(b)
Figure 4: Left: Our equivariant module is distinct from traditional implementations (50) in its fundamental layers and the key-query product, that are crafted to be respectively equivariant and invariant. Right: Equivariant latent array used as additional input to the encoder. We apply equivariant positional encoding to each camera rotation, which is then averaged. We leverage an equivariant linear layer get global geometric latent @xGr, which is concatenated with the conventional latent array Ro to compose our proposed equivariant latent array Ry-
R
D.
H.
θ D'(H) LEL
where D' are the Wigner-D matrices. We disregard any translation action since the input and queries become translation-invariant after center subtraction.
3.4 Basic Attention Modules
This section highlights the equivariant attention module, fundamental ensure encoder equivariance as depicted in Figure 4a consists of basic equivariant layers and an invariant multi-head inner product. Our architecture, unlike typical equivariant transformers 22 33, does not enforce geometric constraints in the equivariant kernel. Instead, it incorporates all geometric features at an input-level. Our module, utilizing the Perceiver IO structure, learns global latent representations, in contrast to other methods that emphasize the hierarchical learning of local features.
Equivariant F Foundamental Layers For the fundamental layers, we use the equivariant linear layer
and layer normalization commonly used in previous works [48] 22 33 34], and provide additional details in Appendix A.3 For equivariant nonlinear layers, there have been multiple proposed methods for features with the same format as ours: Norm-based Nonlinearity, Gate Nonlinearity. and Fourier- based Nonlincarity. Here, we take inspiration from the nonlinearity of Vector Neuron [13] and adapt a similar vector operation to higher-order features. Please see Appendix E for details of equivariant nonlicarity. To better understand the differences between the basic layers in equivariant attention module and those in conventional one, we have visualize them in Appendix A.3 and Appendix
Multi-Head Attention Inner Product As done in previous equivariant transformer works 22 33
we can obtain the invariant attention matrix through inner product of the same types of features. These transformers that emphasize the hierarchical learning of local features suggest using tensor products of edge feature and node feature to mix different feature types, which computationally demanding. We discard the tensor product and only calculate attention weights using various feature types and then multiply these weights with multi-type features to efficiently integrate different types of feature. Please see Appendix for more details. Alternatively, we can mix feature types by treating them as Fourier coefficients for spheres, apply transformers on the sphere, and use Fourier Transform to obtain new coefficients. Please refer to Appendix H for details.

---

3.5 Equivariant Encoder
3.5.1 Equivariant Cross-attention
the cross-attention input is in the format DIEL Hi with C = 2 for 2 and Co being the channel number of fur- To facilitate a clearer understanding, comparison of this input embedding with the used in DeFiNE is also depicted in Figure 3 The latent array Rg = ((Ro)1, (Ro),.... (Ro)NR) E R RNxxCR can be treated as scalar (0-order) feature, remaining constant during transformations in the reference frame. To make the latent array also learn the geometric information, we apply a technique similar to [[]. learning equivariant features from the input's averaged geometric information. Specifically, we apply the positional encoding (PE) for each camera rotation, with each order being the concatenation of embeddings of the rotation matrix's 's three columns. The PE then averaged over cameras. For the specific formulation please see Appendix
As shown in the left part in Figure B and Section 3.3.2
We obtain global geometric latent 9 using an equivariant linear layer, where the size of the weight matrix Wi for each type is (3, N,,N with C' being the channel count for type-l feature in each latent. We then append this equivariant feature to the latent Ro, forming new latent array Ry = ((R),, (R)),, (R)) where (R)); = (Ro); @ Diez(Gi), with L f1,2. -marr) and (Gi), E (22++1)xx' 4 ibllusttrts the construction of this equivariant latent array. With both an equivariant input embedding and latent array, we apply equivariant cross-and coss-attention to get the equivariant latent output.
3.5.2 Equivariant Self-Attention
We apply self-attention equivariant attention mechanism to the equivariant output of cross-attention, producing a conditioned equivariant latent code. For visualization purposes (Figure 5), we can treat the equivariant latent code as the Fourier coefficients of spherical functions. Note that we do not map the features to a 2D color image. Since we have features with type-O, type-1, and type-2, etc, but randomly select each channel from different types of feature and apply the inverse Fourier transform to get a spherical function and visualize it on a 3D sphere. For proof of this result (i.c., the visualized sphere is also rotated when the latent code is rotated), please see Appendix D
3.6 Decoder
In Figure 2 we show that, before inputting the equiv- ariant latent space and geo- metric query the decoder, they are converted into an invariant form by establish- ing an equivariant frame. Specifically, the equivari- ant latent space RR is rep- resented as Dier, (Rx),, )t- From its type-1 feature (Rx)1, employ an equivariant MLP and the Gram-Schmidt orthogonal- ization [59] to derive an equivariant frame, repre- sented by rotation matrix R. As depicted in Figure 5) the equivariant frame's s rotation aligns with that of both the equivariant latent and the 3D scene. Applying the inverse of to RR results in a notation-invarian latent code Dier (R)) (Rx). i. obtaining invariant representation. See Appendix for proof.
Predicted
Frame
Object
Latent Code
Figure 5: Equivariant latent code and predicted frame. For simplicity, object rotation to denote the inverse rotation of the reference frame. When the object rotated, our latent code and predicted canon- ical frame also rotated.
7

---

Input images
Ground-truth depth
DeFiNe (Non-Equi)
Ours (Equi)
Figure 6: Stereo depth estimation results ScanNet, using proposed EPIO architecture.
For the embedding of camera center and viewing rays, we apply RD to r and tj f to obtain invariant coordinates (see Appendix D for proof), denoted as R² R and Rº (t, t). We then use traditional sine and cosine positional encoding for these invariant coordinates, which allows us to leverage higher frequency information beyond the dimensional constraints of SPH. Since both the latent hidden state and the query are invariant the transformation, this enables us to apply conventional cross cross-attention mechanisms to obtain invariant outputs and predictions, capturing higher frequency details and improving expressiveness.
Experimental Results
4.1 Datasets and Implementation
use ScanNet [1] and DeMoN [49 to validate our model on the task of stereo depth estimation. For ScanNet, we use the same setting as [3], which downsamples scenes by factor of 20 and splits them to obtain 94212 training and 7517 test pairs. The DeMoN dataset includes the SUN3D, (GBD-SLAY and Scenesl datasets, where SUN3D and RGBD-SLAN,
real world datasets a synthetic dataset. There are a total of 166285 training image pairs from 50420 scenes, and we use the same test split as 31 (80 pairs in SUN3D, 80 pairs in RGBD and 168 pairs in Scenesl1) 1). We include details on the network architecture and implementation in Appendix
and Scenes
4.2 Stereo Depth Estimation
We compare our equivariant model with other state-of-the-art methods on stereo depth estimation, and report quantitative results in Table T we can see, significantly outperforms competing methods all real-world datasets and shows comparable results to the state-of-the-an a synthetic dataset. This superior performance on real-world datasets is evidence of the benefits of using equivariance in multi-view scene representation. Synthetic datasets, unaffected by real-world lighting conditions, camera miscalibration and view-dependent artifacts, might benefit approaches such DPSNet (77 and NAS 31 that cost volume achieve view consistency. It's also noteworthy that NAS 31 uses additional ground truth surface normals as supervision.
Scenes 1,
We denote our model with "Equi" and our baseline, De- FiNe, with "Nonequi" to highlight that both use the same architecture, Perceiver 10, with the key difference being the presence of equiv- ariance in our model.
Method
Abs.Rel.
RMSE |
< 1.25
DeFiNe (wl VCA)
0.117
0.291
0.870
Ours (w/o VCA)
0.104
0.247
0.893
DeFiNe (w/o jitter)
0.099
0.261
0.891
DeFiNe
0.093
0.246
0.911
Ours (w/o jitter)
0.086
0.229
0.923
Additionally, to assess the advantages incorporat- ing equivariance into our model, we conducted com- parative analysis of our model against nonequivariant DeFiNe bascline 23), both with and without
Table 2: Comparison of EPIO model and DeFiNe ScanNet regarding the use of data augmentation. VCA stands for virtual camera augmentation, and jitter stands for canonical camera jittering.

---

Dataset
Method
Abs.Rel.
RMSE L 1.25 Dataset
Method
Abs.Rel. RMSE 1 < 1.25 1
DPSNet
0.126
0.315
DeMoN
0.157
1.780
0.801
NAS
0.107
0.281
DeepMVS
0.294
0.868
0.549
IIB
0.116
0.281
0.908
DPSNet
0.151
0.695
0.804
ScanNet
DeFiNe
0.093
0.246
0.911
RGBD-SLAM
NAS
0.131
0.619
0.857
Ours
0.086
0.229
0.923
IIB
0.095
0.550
0.907
Ours
0.080
0.433
0.912
DeMoN
0.214
2.421
0.733
DeMoN
0,536
2.603
0.496
DeepMVS
0.282
0.944
0.562
DeepMVS
0.210
0.891
0.688
DPSNet
0.147
0.449
0.781
DPSNet
0.050
0.466
0.961
SUN3D
NAS
0.127
0.378
0.829
Scenesl1
NAS
0.038
0.371
0.975
IBB
0.099
0.293
0.902
IIB
0.055
0.523
0.963
Ours
0.090
0.260
0.912
Ours
0.069
0,617
0.965
Table I: Stereo depth estimation results compared with the state-of-the-art: DPSNet [27 NAS 31
IIB [57.
DeFine 23
DeMoN 49
DeepMMS (26].
(Non-Equi)
Ours (Equi)
(a)
(b)
Figure 7: The figure (a) shows the equivariance of changing reference frame (Red: reference frame): For the same input and varying camera frames as reference, the Perceiver IO's predictions change, but our model's 's predictions stay consistent and the predicted frame is equivariant to the reference frame transformation. The figure (b) shows the approximate equivariance for different camera sets.
data augmentation. We explore two kinds of data augmentation: virtual camera augmentation (VCA), in which novel viewpoints are generated via pointcloud reprojection; and canonical camera jittering (CCJ), in which the reference frame is perturbed with random rotation and translation, reported in 22
To further showcase equivariant properties, visualize the predicted canonical frame and reconstructed 3D point clouds from depth maps. In Figure 7a we see that, for the same scene, when we switch the reference frame (in red) between cameras, the output point clouds change when using the standard Perceiver IO architecture, while ours remain constant, since the predicted depth when
is equivariant to transformations. Furthermore, same scene, which theoretically cannot be guaranteed equivariant due to changes in image content, our model still predicts near-consistent canonical frames and point clouds, as illustrated in Figure In the meanwile, we compare our model with current state-of-art depth estimation model Depth anything [56), and we provide the results in Appendix 0.1
different image pairs within the
4.3 Ablation Study
We performed ablation study on the geometric positional encodings, spherical harmonics encoding. equivariant attention, and the decoder architecture, and report the quantitative results Table 3 As expected, when positional encoding is not used, results are significantly degraded due to missing geometric information. Our results demonstrate that the model leverages geometric information to learn implicit multi-view geometry. Although using Fourier positional encodings with our method breaks the equivariant properties, we conducted ablations by replacing spherical harmonies with Fourier encodings to assess the specific contribution of spherical harmonics.

---

As shown in the table, Fourier encodings, which are not equivariant, are incompatible with an equivariant architecture, resulting in significantly worse performance. Additionally, we replaced the equivariant attention module with a conventional one in another ablation study. Removing the equivariant attention layers also disrupts the equivariance of our architecture, leading to a substantial drop in performance since the model loses its theoretical equivariance. We also ex- plore the impact of the maximum order of spherical harmonics in the positional encodings, in- dicated by lman- For network architecture, we evaluated the impact of not learning the canon- ical frame, that is, we use the equivariant attention module the decoder followed by trans- ferring the equivariant output to invariant output via inner product, see Appendix KK for details. We noticed that higher order of spherical harmonics improve depth estimation, since high fre- quency promotes fine-ge une-gnaieed learning and differentiate positions in higher-dimensiona. space.
Unlike Fourier basis, the dimension of the spherical harmonics grows two times linearly with increasing der, which is a limitation of our method, and there- fore we keep the highest SPH order as in our final model. This is also a reason why learning equivariant canonical frame for invari- ant decoding with Fourier basis and conventional de- coder is a better approach than directly using equivariant decoder. Another factor is that learning a canonical frame enforces all inputs to the decoder be invariant, which should facilitate 3D reasoning. Moreover, we per- formed additional small-scale experiments to study the impact of the number of available views, see Appendix 0.2
Variation
Abs.Rel.
RMSEL
0 < 1.25
w/o camera information
0,229
0.473
0.661
w/ Fourier
0.131
0.318
0.843
w/o equi-attention
0.127
0.314
0,851
Type lmaz = 1
0.134
0.310
0,869
Type lmaz = 2
0.125
0.302
0,875
Type lmaz = 4
0.116
0.283
0.898
EquiDecoder
0.128
0.317
0.857
Full Model
0.086
0.229
0,923
Table 3:
Ablation study on the choice of positional encoding frequency and decoder architecture.
Conclusion and Discussion
We introduce an SE(3) equivariant model designed to learn the equivariant 3D scene prior across multiple views, utilizing spherical harmonics for positional encoding and specialized equivariant attention mechanisms within the Perceiver IO architecture. Additionally, our design exploits its cquivariant latent space to disentangle equivariant frames and invariant scene details, enabling the seamless integration of various existing decoders in conjunction with our specialized encoder. our model's ' capability in 3D structure comprehension is showcased through its superior performance in stereo depth estimation, significantly exceeding that of non-equivariant models. Our architecture can be modified to accommodate a wider range of vision tasks, which we leave to future work (for more detailed discussion please see Appendix M
Limitation As discussed in Section 4.3 unlike the Fourier basis, the dimension of spherical harmonics increases linearly at twice the rate with each order. This limits the number of spherical harmonics and the maximum frequency utilized, resulting in an inability to preserve detailed features in cameras and images. Additionally, the presence of different types of features, each with its own linear and nonlinear layers, slightly slows down the forward process compared to traditional methods. Moreover, we observe instability in training the equivariant network, which may be due to the magnitude explosion of high-order spherical harmonics.
10

---

Acknowledgment
This research was supported by Toyota Research Institute, whose funding and resources were invaluable in advancing this work.
References
Serge Assaad, Carlton Downey, Rami Al-Rfou, Nigamaa Nayakanti, and Ben Sapp. Vn-transformer: Rotation-cquivaariat attention for vector neurons. arXiv preprint (rXiv:2206004777 2022.
[2] Johann Brehmer, Jocy Bose, Pim De Haan, and Taco Cohen. Edgi: Equivariant diffusion for planning with embodied agents. arXiv preprint arXiv:2303. 12000, 2023.
[3] Joao Carreira, Skanda Koppula. Daniel Zoran, Adria Recasens, Catalin Ionescu, Olivier Henaff, Evan Shelhamer, Relja Arandjelovie, Matt Botvinick, Oriol Vinyals, et al. Hip: Hierarchical perceiver. arXiv preprint Ar0: 10890, 2022.
[4] Dian Chen, Jie Li, Vitor Guizilini, Rares Andrei Ambrus, and Adrien Gaidon. Viewpoint equivariance for multi-view 3d object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9213-9222, 2023.
[5] Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 14514-14523, 2021.
[6] Yunlu Chen, Basura Fernando, Hakan Bilen, Matthias NieBner, and Efstratios Gavves. 3d equivariant graph implicit functions. In European Conference on Computer Vision, pages 485 502. Springer, 2022.
[7] Oliver J Cobb, Christopher GR Wallis, Augustine N Mavor-Parker, Augustin Marignier, Matthew Price, Mayeul d' Avezac, and Jason D McEwen. Efficient generalized spherical cmns. arXiv preprint arXiv:2010. 1.200.116660 1661, 2020.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMLR, 2016.
[9] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In International conference on Machine learning, pages 1321-1330. PMLR, 2019.
[10] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cans. 1801.10130, 2018
arXiv preprint
[11] Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
[12] Angela Dai, Angel Chang, Manolis Savva, Macicj Halber, Thomas Funkhouser, and Matthias Nicßner. Scannet: annota-ad 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
2017.
[13] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for (3)-cquivariant networks. In Proceedings the IEFF/CVF International Conference on Computer Vision, pages 12200-12209 2021.
[14] Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19999, 2020.
[15] Carlos Esteves, Christine Alanh-Blanceette, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. arXiv preprint 170.02770..01889, 2017.
[16] Carlos Esteves, Yinshuang Xu, Christine Allen-Bie Blanchette, and Kostas Daniilidis, Equivariant multi-view networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages (568-1577 2019.
[17] Carlos Esteves, Ameesh Makadia, and Kostas Daniilidis. Spin weighted spherical cnns. Advances in Neural Information Processing Systems, 33:8614 8625, 2020.

---

[18] Carlos Esteves, Tianjian Lu, Mohammed Suhail, Yi-fan Chen, and Ameesh Makadia. Generalized fourier features for noordinate-based learning of functions on manifolds. 2021.
[19] Carlos Esteves, Jean-Jacques Slotine, and Ameesh Makadia. Scaling spherical cnns. arXiv preprint arXiv:2306.05442 2023.
[20] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3165-3176, PMLR, 2020.
[21] Alex Fisher, Ricardo Cannizzaro, Madeleine Cochrane, Chatura Nagahawatte, and Jennifer I. Palmer. Colmap: A memory-efficient occupancy grid mapping framework. Robotics and Autonomous Systems, 142:103755, 2021.
[22] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se transforrrr:::: 3d anto-translation equivariant attention networks. Advances neural information processing systems, 33:1970-199 2020.
[23] Vitor Guizilini, Igor Vasiljevie, Jiading Fang, Rare Ambru, Greg Shakhnarovich, Matthew R Walter, and Adrien Gaidon. Depth field networks for generalizable multi-view scene representation. In European Conference on Computer Vision, pages 245-262. Springer. 2022.
[24] Lingshen He, Yiming Dong, Yisen Wang. Dacheng Tao, and Zhouchen Lin. Gauge equivariant transformer. Advances in Neural Information Processing Systems, 34:27331-27343, 2021.
[25] Haojie Huang, Dian Wang, Xupeng Zhu, Robin Walters, and Robert Platt. Edge grasp network: graph- based se (3)-invariant approach to grasp detection. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3882-3888. IECE, 2023.
[26] Po-Han Huang, Kevin Matzen, Johannes Kopf. Narendra Ahuja, and Jia-Bin Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEP Conference on Computer Vision and Pattern Recognition, pages 2821-2830. 2018.
[27] Sunghoon Im, Hae-Gon Jeon, Stephen Lin, and In So Kweon. Dpsnet: End-to-end deep plane sweep stereo, arXiv preprint arXiv:1905.00533) 2019.
[28] Andrew Jaegle, Sebastian Borgeaud, -apttttt Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2 14795, 2021.
[29] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End- end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE international conference on computer vision, pages 66 75, 2017.
[30] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint SKKv:2001.0045:1 2020.
[31] Uday Kusupati, Shuo Cheng, Rui Chen, and Hao Su. Normal assisted stereo depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2189-2199, 2020.
[32] Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X Creighton, Russell HI Taylor, and Mathias Unberath. Revisiting stereo depth estimation from a sequence-oo-sequence perspective with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6197 6006- 2021.
[33] Yi-Lum Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs arXiv preprint arXiv:2206. / /1990, 2022.
[34] Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt. Equiformerv2: Improved equivariant transformer for scaling to higrer-degree representations. arXiv preprint arXiv:2306. 12059, 2023.
[35] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for milti-view 3d object detection. In European Conference on Computer Vision, pages 531-548, Springer, 2022.
[36] Lachlan E MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbitrary lie groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
12

---

[37] DE Pim, Maurice Weiler, Taco Sebastiaan Cohen, and Max Welling, Gauge equivariant geometric graph convolutional neural network, August 12 2021. US Patent App. 17/169,338.
[38] Omri Puny, Derek Lim, Bobak Kiani, Haggai Maron, and Yaron Lipman. Equivariant polynomials for graph neural networks. In International Conference Machine Learning, pages 28191-28222 PMLR, 2023.
[39] Md Ashiqur Rahman and Raymond A Ych, Truly scale-cquivariant deep nets with fourier layers. arXiv preprint 2rXiv:2311.02222 2023.
[40] David Romero, Erik Bekkers, Jakub Tomezak, and Mark Hoogendoom. Attentive group equivariant convolutional networks. In International Conference on Machine Learning, pages 3188-8199. PMLR, 2020.
[41] David W Romero and Jean- Can-BBatiste Cordonnier. Group equivariant stand-alone self-attention for vision. arXiv preprint arXiv:2010.00977 2020.
[42] Hyunwoo Ryu, Hong-in in Lec, Jeong- Lee, and Jongeun Choi. Equivariant descriptor fields: Se (quuivarai energy- based models for end nn--to-end visual robotic manipulation learning. arXiv preprint arXiv:2206.0 08321, 2022.
[43] Alcksandr Safin, Danicl Durckworth, and Mchdi SM Sajjadi. Repast: Relative pose attention scene representation transformer. arXiv preprint wrXiv:2304.00941 2023.
[44] Mehdi SM Sajaadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229-6238, 2022.
[45] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning. pages 9323 9332. PMLR, 2021.
[46] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6394-6400.
IEEE 2022.
[47] Mohammed Suhail. Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision, pages 156-174. Springer. 2022.
[48] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint 180.:1800.0822 2018.
[49] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy llg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5038-5047, 2017.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention all you need. Advances neural information processing systems, 30, 2017.
[51] Maurice Weiler and Gabriele Cesa. General e (2)equivariant steerable enns. enns. Advances neural information processing systems, 32, 2019.
[52] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31, 2018.
[53] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5028-50377 2017.
[54] Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniliidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In International Conference on Machine Learning, pages
PMLR, 2022.
[55] Yinshuang Xu, Jiahui Lei, and Kostas Daniilidis. se(3) equivariant convolution and transformer in ray space. In Thirty-seventt Conference on Neural Information Processing Systems, 2023.
13

---

[56] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint wrXi:2441110891, 2024.
[57] Wang Yifan, Carl Doersch, Relja Arandjelović, Joao Carreira, and Andrew Zisserman. Input-level inductive biases for 3d reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 6176-6186, 2022.
[58] Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang. Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. arXiv preprint arXiv:2402.14817 2024.
[59] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang. and Hao Li. On the continuity of rotation representa- tions neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745-5753, 2019.
[60] Minghan Zhu, Maani Ghaffari, William Clark, and Huei Peng. E2pn: Efficient se equiiiarai point network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1223-1232, 2023.
14

---

Appendix
A Preliminaries
A.1 Perceiver 10
The Perceiver I0 (28) efficiently encodes multi multi-modalliyy inputs by utilizing cross-attention between the inputs themselves and learnable, fixed-dimension latent code. This latent code is then refined through a series of self-attention layers. the decoding phase, the model employs cross-attention between given query and the refined latent code to generate predictions.
In DeFiNe [23], this framework used address scenarios involving multiple cameras with predetermined relative poses, denoted as (T:) and their respective images (7:) Within this context, the system queries arbitrary camera poses, represented as {TH -=1 Importantly, this queried pose T3 can either be one of the already established camera positions (as is common in stereo depth estimation), position outside the range of the input cameras (which is typical in video depth estimation). Based on input data and queried pose, the network generates the corresponding predicted output Dy for the query.
Like other vision tasks that utilize the Perceiver framework, DeFiNe uses as input composite of geometric information and corresponding image features, while the query utilizes only geometric information. Specifically, the input is formulated as { + PE(r"v) @ PE(t.)}, where füv repre- sents image features associated with each pixel (u, v) in camera i. PE(r(e) denotes the positional encoding with Fourier cosine and sine series of the ray direction r'y calculated relative to camera T; (F((4) refers to the positional encoding of the camera's 's translation & in T; using Fourier cosine and sine series. The query in this model is represented as {P @ PE((t)) and the network is designed to output depth estimation Di. for each query pixel (u, v) of query camera
A.2 Equivariance Definition
Concretely, assuming the global reference frame undergoes transform 7 E SE(3) to Ta = T-TT, the set of input and query poses would become {TT;} and {TT}} with respect Ta (note that the corresponding input images {4J) == remain unchanged). Mathematically, we use (TT,} denote the SF(33 actions the
Ar(([T)) (T:))) ((TT)), (T+)) and ((T))) input and query cameras. An equivariant network satifies
(AT.), (4H), (r(T)) ((([T))) (TJ)).
Readers may recognize the above equation as describing the invariance of the network to the transformation of both input and query. In fact, is also equivalent to the statement that the network is equivariant when viewed that it learns an implicit field. To demonstrate this equivalence, let F(.) (([T)). (T) and define the SE(3) operator on F: NgF(((T))) = F(A7"((T))) We then derive
+ (AT((T:), (4))) =f (T (1))),
i.e., that the network t is equivariant. A similar statement can be found in 6
With input level inductive bias and representing T =(R,)) f), the equivariant model & should satisfy
((fur @ PE(R uv: Rt, + +t)}, (PE(Rr) Rt, + t)})
= *(ffi @ PE(ri t)), (PE(
A.3
Fundamental Layers
A.3.1 Equivariant Linear Layer
With the transformation acting on the features, we can define the equivariant linear layer
C(R (())) R ((H())) = R H(++1),
15

---

C( (k) _1111 (k)
[k)
(11
Wo. 111 (1)
W1. (1)
(k) W Wmaa)
Wi: R R x (1+))
(
(141))
c(k+1)
Lasca
C((H))) (k) ,
H( Vec(H))) _ H/ wwlk) (Hb(tt)), H x(++1)
H H(tt1)
WWA
x cat 1 (2
+1)
Of x
O x (2lmer 1)
((++1)
(21 +
Figure 8: Comparison between Equivariant and Non-Equivariant Linear Layers: The figure above depicts the equivariant linear layer, wherein each type of feature is linearly combined using specific matrix W,, treating the vector or tensor as a cohesive geometric entity. Conversely, the figure below illustrates the traditional linear approach, where all channels within an intrinsic feature, as well as different types features, are linearly intermixed, since vectorizes and concatenates all features and applies unified weight matrix
where the superscript denotes the index of layer (R) DIEr. H (H( ( ....
H. (k) and
the same for (k++1), To achieve equivariance, we use the same linear layer L. as stated in 488
33
(((H))) (K) (* ..... .H(A) Imox
= (( Wo. H((h) W1. (() W Wlmaa).
The weights W; have the format (O(*) c(+t1)) where c is the number of channels in (k) representing the corresponding input channels, and c(t++1) the number of channels in (1 representing the corresponding output channels. The difference of equivariant linear layers and conventional linear layers are depicted in Figure g
A.3.2 Equivariant LayerNormalization
We use the commonly used equivariant layer normalization in equivariant works that apply the layer normalization to the norm of the features, and then multiply those with unit tensor features. The normalization layer LN is defined as
16

---

CN((H)) HH , ,1. .. ))=
(In(H6 (u(uorm((()")
An (x
normal (1
norm H
CN((11)) _I((1)
-H )) InVee(((H _I(())
(1) max
(0
c
(k) Jmme
(2)
(K)
3
m20
Figure 9: Comparison Between Equivariant and Non-Equivarriat Layer Normalization: The figure above illustrates the equivariant nonlinear layer. where layer normalization is applied the norm of each feature type, followed by multiplication with unit for each feature. The figure below depicts the traditional nonlinear layer, employing plement-wise layer normalization across features, thereby disrupting equivariance by treating each feature type as a concatenation of individual channels rather than unified geometric entity.
CN((H)) (H))).....
(
H
= =(n(H)()) In([1
(m((I(((())) (+(
(
H70)) Imax
|
Imax
where In is the conventional layer normalization. The difference of equivariant layenoormalization and conventional layernormalization are depicted in Figure
A.4 Spherical Harmonics
The spherical harmonics constitute complete set of orthogonal functions, making them orthonor- mal basis. Any spherical function f L (S²) can be expressed as linear combination these spherical harmonics. In simpler terms, they serve the Fourier basis for functions defined sphere.
Similar to the varying frequencies of sines and cosines in Fourier series, spherical harmonics are characterized by different degrees (orders), denoted as 1 N. Each degree of spherical harmonics corresponds to specific pattern shape on the surface of a sphere. Higher orders (degrees) indicate higher frequencies, resulting more intricate and complex patterns on the sphere's surface. To apply spherical harmonics in cire-dimensional space ( ( ), incorporate radial component r'
17

---

into the original spherical harmonics of the corresponding degree (order)-I. This method scales the spherical harmonics for biree-dimensiona upplications, extending their utility beyond the sphere S? This adaptation scales the spherical harmonics for applications beyond the two-dimensional sphere surface, effectively extending their utility to three-dimensional analyses and applications. Despite this adaptation, these functions retain their fundamental characteristics and are still referred to as spherical harmonics. In this work, we utilize spherical harmonics, incorporating 3D Cartesian coordinates, as positional embeddings in transformer models.
The fascinating properties of spherical harmonies make them crucial in 50(3) group representations, allowing us to harness their power to achieve equivariance in transformers. A order-/ spherical harmonics, denoted as Y R 22+ is vector function with 2/ dimension, following the transformation rule:
('(r) = (T)))) r'(r) = (r](''(f).
where is an arbitrary rotation, f = - and D' :5((3) (2141))) is called the Wigner-D matrix, serving the irreducible representation of SO(3) corresponding to the order 1. The Wigner-D matrix are orthogonal matrix, i.e., (RR) = I. To mitigate the impact of the large scaling factor ||r|[|,, we explored alternative approaches: one involved substituting the scaling factor ||| with Gaussian radial basis functions, expressed as e -(1r-11-1) Another approach entailed employing Fourier sine and cosine series represent ||r]], creating an invariant embedding. This embedding was then combined with the positional encoding of the unit vector i using the spherical harmonics for sphere. However, these alternatives did not demonstrate any significant benefits over the use of spherical harmonics adapted for stree-dimensiona space (R.), leading to continue with our initial methodology.
A.4.1 The use of Spherical Harmonics in Equivariant Transformer
We acknowledge that most equivariant transformer works Fuchs et al. [22). Liao and Smidt [33]. Liao al. [34] also uses spherical harmonics in the transformer layers. However, the use of the spherical harmonics is to derive the equivariant kernel basis based on the relitive position for specific geometire entities. For example, Fuchs et [22] first proposes the SE(3) equivariant transformer for point clouds, where spherical harmonics is served as the equivariant kernel as in steerable 3D convolutions. Equiformer 33 34 is graph- architecture that leverages spherical harmonics for the edges, using depth-wise tensor product to embed it into the node. For this graph embedding (atom + edge-degree) and graph attention, the use of spherical harmonics could interpreted more as an equivariant kernel basis. The difference here is that Equiformer uses spherical harmonics to integrate edge information into the equivariant kernel, while ours uses spherical harmonics directly in each token.
B Equivariance of Conventional Positional Encoding
The conventional positional encoding leveraging Fourier sine and cosine functions is translational equivariant. When the input is translated by a translation i, the output will be transformed in a specific way, multiplied by the representation of i.c., PE(x complex format or
elwt e for specific frequency w in
cos (wos -sin(ut)]
coos (war) sin (war)
- -sin (w:x)
PE +
sin (wi) cos(wt)
cos(war)
(P)
matrix format.
C Equivariant Hidden Feature Format
The equivariant latent code is structured as H = DIEL H, = (Ho, H,........ Hi), with each Hi having dimensions (27 1, Ci). The action of rotation on this latent code is depicted in Figure
10
18

---

Chas
C1
Co
Ho
H1
Hi mmux
Ho
Dimaz
Ht)
Figure 10: Latent code transformation.
D
Visualization of the Equivariant Latent
With the spherical harmonics, we can introduce the Fourier Transform for the sphere. The Fourier coefficient F of function on the sphere, f S IR, corresponding the order is obtained by
F / (z)dr,
Jg2
and the inverse Fourier Transform without normalization follows the equation:
f(x) = (F))
When we rotate the function I with any rotation R SO(3), i.e., we get new function f" J(R-1x). The Fourier coefficients corresponding to the order are:
F F" [(I (0 / [(o) - -
D'())
This indicates that when spherical function is rotated by any rotation R, its Fourier coefficients will be multiplied by the corresponding Wigner-D matrix. Inversely, we have that when all Fourier coefficients F are multiplied by Wigner-D matrices D' DR)),, the obtained spherical function is rotated by R.
With such preliminary, features are the /-th order coefficients, enabling us to derive the spherical function. As a result, when features are transformed with each type being multiplied its respective Wigner-D matrix the spherical function undergoes a corresponding rotation.
treat our latent code DzEL Hi the Fourier coefficients, where type-/
E Equivariant Nonlinear Layer
In our proposed nonlinear layer, we first generate intermediate hidden features DIEL H (H((*, .... .... (/)))) of the same size as the input via an equivariant linear layer. Subsequent to this, employ the specified nonlinearity:
(((H)())
(k) (K))
(b)
H((k)
= (a((H (1) (M((H(() H((h) (H ((()))))
+ H
norm (orm(H1()) (K))
H (k)
Imnc (1(h)) (..,((((H
(1)
H (
H Lmne
norm ((()) Imax
19

---

A((Hb) H .... .H ,.. =a((Ho)) (a((HT .H! (115 . HH
H
H!
norm(H?
HP norm H
(a((1)) (o 11 (11 11 ))
H (()
(80) H H
activation
projection-activatee
A((H))) (K)
( .... (mma )) = a(Vec((H)))
H
(k)
Imex
((A) Imsn
(2 +1)
activation
(R)
CiA 3
Imax
(21mas + 1)
Figure 11: Comparison of Equivariant and -onn-aquvar Nonlinear Layers: The figure above illustrates the equivariant nonlinear layer, similar to those in vector neuron models. This layer estab- lishes equivariant directions using an equivariant linear layer, applies nonlinearity to the projection of the original feature in these directions, and then adds this to the orthogonal component relative to the direction in the input. On the other hand, the figure below shows the conventional nonlinear layer, which applies element wine nonlinearity to the vectorized feature, thus failing to preserve equivariance.
Here, ( denotes the per-channel inner product, meaning the size of (H(")) H((()) is in the format (1, (.))). The function represents conventional activation operation, such RcLU, Sigmoid, or LeakyReLU. The symbol indicates broadcast multiplication. This approach bears resemblance gated normalization (52). However, in model, the scalar for the "gate" derived from the inner product of two outputs of the equivariant linear layer, rather than being a scalar present in the hidden features veemselves. The difference of equivariant nonlinear layers and conventional nonlinear layers are depicted in Figure I]
F Multi-head Attention Inner Product
Given the input of the attention module formulated as + iel Hi, we generate query Q, key K, and value V by equivariant linear layers. As Q, K, and V are equivariant features, they are expressed as Qi = D (Q:)s, K, Diez (Ki) and V, = Dier where and j are indices of the latents. The inner product is calculated between the equivariant key K and the equivariant query Q, and here we describe how we use in multi-head attention. With N,, multi-heads, we split the features K, Q, and V into N₁, heads along the channel dimension. Taking as instance for clarity,
20

---

and denoting Ct as the number of channels for type-l feature (Qi); in Q, Q: gets divided into various heads (Q,) A with as the head index. (Q,)* maintains the equivariant feature format, represented as (Q:)* Dier. Q1) with the channel count for type-/ feature (Q1), being C. This division also applies K and The inner product of (Qt)* and (Ky)* is defined Ns."
(QQ)),(G)))
ΣΣ ((O))))
(1)
With the defined inner product, we have the output:
ezp((((.)) (K))")) (0,) ΣE ((p((((())) (K)) (V))*
(2)
The final output of the attention mechanism, composed in the channel dimension, can denoted as DieL Og. For proof of equivariance, refer to Sec. G
G
Proof of Equivariance for Multi-Head Attention
The inner product is formulated as follows:
CL
NR.
(K,)),(K)))) = ΣΣ ((Q0))))) IEL c=1
Owing to the equivariance of the Linear Layer, when the input undergoes a rotation R, the components Q, K, V are correspondingly transformed, denoted as R. Q, R- K, and Rt V. Under these conditions, the corresponding inner product becomes:
((R 1.)*,, (R ((R.K,)*> Σ E ((D((((Q)))))))))(()))))
JEL
N
(())))))
IEL
Σ M
JEL
((Q.)) (Ky)))
which proves the invariance the defined inner product. Thereby, the output becomes:
(Q)), (R vih
Σ E (xp((((.)), (K,)),
exp(((())) ((.) (K,))
Σ E, exp ((Q.)) (K))a) LET.
D'(R)(()))
cxp(((Q.)) (K,)),()))))))
θ D((R))
Σ, (0pp(((2)))))))))))))(()))()))) (K,)*)) (V)))
LEL
! (O())(O)))
leT.
R Oz.
which proves that the whole attention mechanism is equivariant.
21

---

H
Alternative Equivariant Attention
When comes attention mechanisms involving latents with different types of equivariant features. a direct method is to use tensor product to entangle these different types, which is complicated and computationally expensive. However, there is an alternative approach that treats the different types of features as Fourier coefficients to obtain spherical features. By applying the conventional transformer to these spherical features, followed by the Fourier Transform, we can retrieve different types of equivariant features. This method offers a more efficient way to entangle and handle various types of features within the attention mechanism. For latent features Dier. Hi, we apply the Inverse Fourier Transform so that:
Si = Y'))=;"())"H,
which implies that S has size (NR. Ct Ns), where NR is the number of latents and Ns is the number of samplings for the sphere. From the preliminary in appendix [D] we know that when the input features are rotated by R, the output becomes S xx) which means these spheres are rotated as well. By concatenating the {St] on the channel dimension, we get the features after an inverse Fourier Transform with size (Nn. n. E, Ct, Ns), i.e., we have Nn spheres with Σ, Ct channels and Ns number of sampling. We can directly apply the self-attention mechanism to these spheres without breaking equivariance, resulting in spherical features F with dimensions (NR. >, Ct, Ns) after the self-attention. Finally, we apply the Fourier Transform as follows:
Hi = E
(z,))
where (x;,) F(; Ind, i). with Ind representing the index of channels for spheres that correspond to type-l features H, and i denoting the index of the sample on the sphere. By composing different types of features, of inverse Fourier Transform, transformer on the sphere and the Fourier Transform is equivariant, as confirmed by the preliminary properties of the Fourier Transform in appendix D In practice, the computational load is increased due to the number of samples and the complexity of spherical convolution. To address this, we can utilize icosahedron sampling and apply equivariant correlation on the icosahedron for the linear layer in attention. Additionally, we can use standard nonlinear layers and typical layer normalization in the self-attention mechanism.
obtain outputs in the format Die L Hi. It is evident that the composition
I Averaged Global Geometric Embedding
The formula for the averaged global geometric embedding is as follows:
Y' Y (R)) @ (R:).
N
(2(R)) + Y (R²) + Y (
N
1
Σ ylmas (R.) + Y! R mo ))
N
where the superscript denotes the index of the column in the matrix.
J Proof of Invariant Latent and Query
Given that the predicted frame R is equivariant and the latent code Dier (Rx),, is also equivariant, when the input undergoes transformation by rotation (Ru, to) (3),,, the frame is modified to Ro Rt and the latent code transforms to Ro RR = DieL D'(RR) (Rx) Applying the inverse of the equivariant frame to the latent code yields:
D'(RoR) D'Da =
("a
= + (Rx)) LEL
LEL
LEL
demonstrating that the transformed latent code is invariant. Furthermore, when the input is subjected to transformation by rotation (Ro,to) to) SE(3), the decoded camera j's pose shifts to (RoR,, Rot,
22

---

Equi-Linear
(Vo, Vi,..., V mazz)
(Rx)), (Rx),, ... (Rx) monz)
Multihead
Equi-Linear
(K1, Kimasi
Attention
Equi-Linear
(Q1,..., Qlmax)
(PF,, PF2, ... )FAmoz)
(Oo, O
Otmas )
Inv
O
Figure 12: Equivariant Decoder.
to). After subtracting the center, the pose becomes RRo Ro(ts - D)). Applying the inverse of the equivariant frame to the query camera pose, arrive at:
((Ro (RoR)) (Rolt, -))) (R" R,, R (t, (t,-t)) -)), -)),
which confirms that the transformed query camera pose remains invariant.
K Alternative Equivariant Decoder
The pipeline of the equivariant decoder show in Figure 12 The cross-attention mechanism with equivariance processes two inputs: firstly, the resulting hidden features from self-attention, denoted as 1Der ) with L, {0, 1.... Imae) }. and secondly, the positional encoding of query rays and cameras, represented as (PE (PE(ri ty f)) with spherical harmonics. In this context, ri signifies the (u, th ray in the j-th query camera, ty refers the translation of the j-th query camera, and è is the pre-caleulated center of the encoded cameras. The positional encoding is structured as ). Note that this encoding does not include the O-type (invariant)
OieL PF2 with I, (1..... [maa)). image features, in contrast to the encoded input. Hence, when generating the Q, K, V features in the attention module, Q features do not include 0-th type features. To achieve invariant attention weights, K features should also exclude 0-th type features, which is accomplished by setting Wo in the equivariant linear layer. The multi-head attention mechanism adheres the equations in Sec.
ΣV ((O))))))(()))))) TET.
eapl ((Q,)* (Kz)*t)
(0,)* Σ E, crp(((Qi)) (K,))) (K))
(Vz)*,
where L= (1,, ... ,ma }. and (Vr)* conforms to the D (0,10, Mmax) (Vi) format. The output O is derived Die )maz) O
Since the final prediction value is unchanged when the reference frame is transformed, it is character- ized as an invariant type-0 (scalar) feature. To extract the invariant features for prediction, we utilize an "invariant layer", as depicted in Figure 13 In this process, we initially generate two intermediate features, H' and H" of the same size. Subsequently, for each type-l, we perform an inner product operation between Hi and H,", which results in invariant features /t. each with channel count of C)- By concatenating these invariant features (7t), we formulate final invariant features O = I with combined channel count of Σj Cj which is then utilized for the final prediction.
23

---

Equi-Linear
(A
(
(1
(H(ht))
Inner-product
Equi-Linear
(11g)) H
..... Hi
(B|+ (((++))
B
(H)*))
Figure 13: Invariant Layer
Task
Input
Transformation Pos. Encoding Feature Embedding
Query
Prediction
Novel View Synthesis
Image Camera
SE(3)
SPH
Image embedding
camera pose
RGB
Neural Volume Rendering
Image- Camera
SE(3)
SPH
Image embedding
point+ray dir
o, RGB
Pose Estimation
Image Camera
SE(3)
SPH
Image embedding
image (Inv)
R,
Implicit Field for PC
Point
SE(3)
SPH
Point embedding/-
point
Field value
2D Dense Prediction
Image + Pixel
SE(2)
Trig
SO(2)-equi feature
pixel
Field value
Table 4: Different tasks and their corresponding geometric information in Equivariant Perceiver IO
L Network Architecture and Implementation Details
Regarding architecture, we use a ResNet18 as the visual backbone, resulting in 960-dimensional features. The order of spherical harmonics [1,2,4,8], resulting in (3+5+9+17)* 17)+9++172+>+>> = 68-dimensional features. For encoding, visual and geometric features are concatenated to produce 960 + 68 1028-dimensional embeddings. For decoding, we use the same Fourier encoding as the standard Perceiver 1O, resulting in 186-dimensional embeddings. Our original latent representation R is of dimensionality 1024 x 512. We set the number of channels for each type of the equivariant hidden feature as [512, 64,32, 8J. In the Perceiver IO implementation, we have block of cross-attention with head, 8 self-attention layers with 8 heads, and cross-attention with head for a decoder.
Our DeFiNe baseline has 73M parameters, and our EPIO implementation has 147M parameters. This increase is due to: additional parameters for the global geometric latent code as shown in Figure 5; inference for frame prediction as shown in Figure 2; and additional parameters for type-2, type-3, and type-4 features, where we set the channel numbers as 64, 32, and 8 respectively. Regarding runtime, we observed an increase of roughly 2x in training iteration times and 1.5x in per-pixel queries during inference. However, we would like note that our approach converges in roughly 15%. Training and evaluation was conducted using distributed training (DDP) on 8 A100 GPUs, with 80 GB each.
Regarding experiments, we used Pytorch to implement our Equivariant Perceiver IO and will open- source our code and pre-trained weights upon acceptance. We used a batch size of 192, the AdamW W optimizer with B = 0.9, and B3 = 0.999, weight decay of 10-4, and an initial learning rate Ir at 2 x 10-4 4. For ScanNet, the training duration was 200 epochs, with the learning rate being reduced by half every 80 epochs; For DeMon datasets, the training duration was 200 epochs, with the learning rate being reduced by half every 80 epochs. We used the same losses as DeFiNe, i.e., the L.I-log loss, with weight of 1.0 for real views and 0.2 for virtual views. Following standard practice, we used images of size 128x192 for ScanNet, and images of size 240x320 for DeMoN, using two images as input, with corresponding intrinsics and extrinsics, and ground-truth depth maps supervision (see Section I). This the standard training and evaluation protocol and was used by our baselines as well, ensuring a fair comparison.
M Extended Discussion for General Tasks of Equivariant Periceiver IO
Our model, designed as a general architecture, is adaptable to various tasks. This paper focuses on demonstrating the ad advantages of integrating equivariance into Perceiver IO for scene representation,

---

Figure 14: Qualitative Results
primarily evaluated through depth estimation, core problem in vision. While implementing our model for other tasks is beyond this paper's scope, we provide brief overview of its potential extensions to different applications, as shown in table
N
More Qualitative Results for Depth Estimation
Please see Figure 14 for more qualitative results.
More Experiments
0.1
Comparison with Current Prevalent Depth Estimation Model
We have evaluated DepthAnything the same ScanNet stereo benchmark we report results as shown Table5] we can confidently say that our method outperforms DepthAnything on this benchmark. However, would like to emphasize that these are not meaningful comparisons. DepthAnything
25

---

Models
Abs.Rel.4
RMSEL
8 < 1.25
Depth anything
0.099
0.226
0,903
Ours
0.076
0.217
0.934
Table 5: Comparison of our model and Depth Anything
views
views
views
DeFiNe
0.324 0.215
0.315
0.307
Ours
0.209
0.198
Table 6: Novel View Depth Estimation across a varying number views.
monocular depth estimation network that outputs affine-invariant predictions, while ours is multi-view depth estimation network that outputs metric predictions. Hence, to achieve the reported DeythAnything numbers, we had artificially shift and scale predictions using ground-truth depth maps (the same thing is done in their paper). We also could not use the second image as input to DepthAnything since it is monocular network, while method can leverage multiple images as input by design (and even benefits from that during training via the virtual camera augmentation procedure).
0.2 Varying views
We performed additional small-scale experiments to study the impact of the number of available views. In this setting, we have 500 views of one scene, and randomly choose N encoding views and different decoding camera viewpoint for novel depth estimation.
For DeFiNe, we train with jittering augmentation on the reference frames and test with augmentation as well. For our model, we train without jittering augmentation and also test with augmentation. We explored 2, 3, and 4 views, and the Abs. Rel. depth estimation results are reported in Tab. 6
As see, method consistently surpasses DeFiNe across varying number of views, without employing augmentation during training, with the performance gap remaining similar across different view counts.
P Impact Statements
This work aims to advance 3/ effective learning, with an application in 3D reconstruction. While the direct outcome our research may not have immediate societal implications, the broader application of 3D reconstruction technologies could have some impact. A primary concern is the potential for privacy violations, particularly in scenarios where 3D reconstruction is used to create detailed representations of real-world environments individuals without their consent. Such applications could lead to unauthorized surveillance or data collection, posing ethical and privacy challenges that need to be addressed as this technology advances and becomes more accessible.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification: We don't include propositions or theorems in the paper, but we provide the proofs the appendix for the statement the paper for soundness. We provide proof of multi-head attention, invariant latent and query in Appendix, G and Appendix.
Guidelines:
The answer NA means that the paper does our architecture is not limited to specific geometric entities. Safin et al.not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide network architecture and implementation details in Appendix L
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc.
hublli//guidass/ddeeelbmm.ssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccppp..cc/pulli///gg.. guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide the training and test details in Section. 4.1 and Appendix,
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: It was not possible to train independent models enough times to produce enough samples for a statistical analysis. We provide ablations showing the improvements generated by our contributions (all starting from the same random seed), and comparisons to state-of-the-an baselines (which also do not report error bars).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the detials of compute resources in the Appendix. L
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpss//neuripps http://hwwrrppssccppppeaeeeeeeee httpp/ubbl..hhiccciieeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited the original papers that produce the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines: The paper does not release new assets.
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The paper does involve crowdsourcing nor research with human subjects.
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "TkuBNfRs6c", "reviewer": "Reviewer_KQ34", "summary": "This paper presents a SE(3) rotational and translational equivariant variation of Perceive IO for multi-view depth estimation with known camera poses. The authors first encode both the pixel-wise ray direction and the camera translation using spherical harmonics as the position encoding, and then to maintain equivariance under global transformations through the network's forward pass, the authors modify several components, including the linear projection, the latent array construction, and the output decoding. To demonstrate the effectiveness of the proposed method, the authors conducted experiments on several RGBD datasets, including ScanNet, SUN3D, TUM-RGBD, and Scene11, and achieved better performance than existing implicit multi-view depth estimations, such as DeFiNe, and multi-view stereo (MVS) models, such as DPSNet.", "questions": "Please refer to the weakness.", "limitations": "The limitations have been sufficiently discussed in the Appendix. Q.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors introduce the problem well, explaining the importance of equivariance to the task of multi-view depth estimation effectively. They also provide a brief yet sufficient review of existing works, clearly positioning this work within the field.\n- The authors have carefully designed several novel equivariant components:\n  - A SE(3) equivariant positional encoding, where besides rotation, the authors smartly encode camera translation also using spherical harmonics.\n  - An equivariant linear projection layer where the linear projection is applied to each group of features that corresponds to position embedding derived from the spherical harmonics of a specific order.\n  - Equivariant latent array construction and the reversal of the rotation from the latent array before being cross-attended to the output queries.\n  \n   These designs, along with the adoption of existing equivariant components through the Perceive IO pipeline, ensure good performance and can be inspiring for other tasks that require equivariance.\n- The experiments are sufficient and demonstrate the equivariance of the output and the overall accuracy.", "weaknesses": "The major weakness of this paper lies in its presentation and organization, which makes the paper difficult to read:\n\n- Many important details from Sections 3.4 to 3.6 are placed in the appendix, making the main paper not self-contained. For instance, details in Appendices A.3 and E would be better suited in the main paper.\n\n- Sections 3.4 to 3.6 are organized into fragmented components, where the holistic process of the Perceiver IO is missing. Specifically, the authors should introduce each modification in the order of the Perceiver IO pipeline. \n\n- The description of individual components are also confusing:\n   - It is better to only briefly discuss components that are equivariant itself, such as attention, and discuss only how they made the input to the attention equivariant, such as the latent array in Section 3.5.1. Otherwise, it might be misleading to suggest that there are new equivariant attention modules themselves.\n   - Why is only rotation sampled and encoded when constructing the latent array in Section 3.5.2 and Figure 4, while the inputs have the encoded camera translation?\n   - Similarly, in Section 3.6, only reverse rotation is applied to the latents after several self-attention transformation blocks, while the translation is omitted.\n   - Line 261-262: \"which allows us to leverage higher frequency information beyond the dimensional constraints of SPH.\" The authors indicate that the Fourier encoding is not equivariant but use it for the output query, therefore, the authors should elaborate more on the insight behind this choice and provide sufficient proof to support this design.\n  -  Many illustrations (Figures 8-11) in the appendix are confusing and do not help to clarify the equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*W1. Many important details from Sections 3.4 to 3.6 are placed in the appendix, making the main paper not self-contained.*\t\n\nWe appreciate and thank the reviewer for the valuable feedback. We decided to organize the paper this way not only due to limited space but also because we want readers to have a clear understanding of our main proposed components, especially those without a prior equivariance background.  Thus, we leave standard components (e.g., equivariant linear and normalization layers in appendix A.3) in the appendix, where readers unfamiliar with the field can get a better understanding through visualization and more detailed analysis and proofs. \n\nHaving said that, we agree that a more balanced structure might be better, and will move more technical details and visualization from the appendix E to the main paper for the camera-ready version. \n\n *W2. Sections 3.4 to 3.6 are organized into fragmented components, where the holistic process of the Perceiver IO is missing.*\n\nThank you for the suggestion. We decided to introduce fundamental attention layers in Section 3.4 before discussing the encoder in Section 3.5 and the decoder in Section 3.6 because attention mechanisms are essential components of both modules (encoder and decoder). By first explaining these fundamental operations, we aim to help readers build an understanding going from the basics to the overall more complex structures. \n\nIf this approach is causing confusion, we propose to instead first briefly give a whole overview of our holistic method with the illustration of Figure 2, then present the input design of the encoder first, followed by an introduction of the encoder, the “canonicalization” of the encoder output to serve as input to the decoder, the design of the decoder, and finally the prediction process. We believe this new order would create a better flow that approximates more closely the holistic nature of the Perceiver IO architecture, from input to output.\n\n*W3. It is better to only briefly discuss components that are equivariant itself, such as attention, and discuss only how they made the input to the attention equivariant.*\n\nThank you for the suggestion. The reason we provide an overview of the attention mechanism and compare our module structure with previous work is to lay emphasis on the equivariant operation of proposed module  and help readers without a relevant background to have a better understanding of why the conventional operation is not equivariant and ours is.\n\nHowever, to avoid any potential confusion, we will move more details from Section 3.4 (especially the descriptions of previous works) to the appendix for reference, so the focus is on our contributions. We will also move Appendix I to Section 3.5.1 to provide more information about the input construction for the encoder.\n\n\n*W4. Why is only rotation sampled and encoded when constructing the latent array in Section 3.5.2 and Figure 4, while the inputs have the encoded camera translation?*\n\nThe translations of the two cameras are opposites because we subtract the mean of the translation to achieve translational invariance. As discussed in Section 3.5.2, we need to average the PE (positional encodings) of the two cameras to obtain the latent array. Since averaging the PE of two opposite translations might cause some zero values, we discard the PE for the translation of the latent array construction. Note that we keep the spherical harmonics PE for translation in inputs, as a way to preserve translational information.\n\n*W5. In Section 3.6, only reverse rotation is applied to the latents after several self-attention transformation blocks, while the translation is omitted.*\n\nAs stated in Section 3.3. 2, after we subtract the cameras’ central position, our model becomes translationally invariant. Because the hidden features are now rotationally equivariant and translationally invariant, we apply the reverse rotation to the latent array to make it rotationally invariant. Note that, when we provide query cameras for the decoder, their position is also extracted from the center, which makes the whole model translationally invariant.\n\n*W6. The authors indicate that the Fourier encoding is not equivariant but use it for the output query.*\n\nAs stated in L334-339, a limitation in the use of spherical harmonics (SH)  is that dimensionality grows linearly (2x) with increasing orders, which constrains the highest frequency that can be used in the positional encodings. Therefore, we learn instead an **equivariant**  frame of reference designed to make the input to the decoder **invariant**, which enables us to use traditional decoders without SH, which can then reach higher frequencies. It is important to note that, even though we use traditional Fourier positional encodings for the decoder, the model is still **equivariant**, since the input to the decoder is **invariant** by design. The theoretical proof that guarantees an invariant input for the decoder is provided in Appendix J. There is an ablation study in Table 3, where “EquiDecoder” indicates the use of an equivariant decoder with equivariant positional encoding for the query, our design is better than the equivariant decoder, which supports the effectiveness of our design.\n\n*W7. Many illustrations (Figures 8-11) in the appendix are confusing.*\n\nThank you for pointing that out. We have enhanced these figures and included the updated versions in the rebuttal PDF (Figure 1,2,3,and 4). Any additional feedback would be highly appreciated, so we can further improve the quality of our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2NCIWK4dQy", "reviewer": "Reviewer_QCgu", "summary": "This paper introduces a ray embedding representation with rotational and translational equivariance, integrating the existing Perceiver IO architecture to achieve robust multi-view implicit depth estimation. The paper first utilizes the mean shift and spherical harmonics to achieve translational equivariance, and then builds upon this to use spherical harmonics to achieve a rotationally equivariant representation, ultimately combining to obtain a three-dimensional transformation embedding with equivariance. By further designing equivariant encoders and decoders, the paper realizes robust estimation of depth from new perspectives. Experiments on the ScanNet and DeMoN datasets demonstrate the effectiveness of the proposed method.", "questions": "-The task setting of implicit depth estimation seems to be very compatible with the existing sparse view NeRF/GS methods. Although the focus of the two is different, with NeRF/GS focusing more on rendering images, while DeFiNe and EPIO mainly focus on geometry, there is a possibility of mutual exchange between the two. Can you report the comparative results with such methods? For example, ENeRF.\n\nLin, et al. \"Efficient Neural Radiance Fields for Interactive Free-viewpoint Video\", SIGGRAPH-ASIA 2022.\n\n-DeFiNe can synthesize novel view images, can EPIO do the same? What are the results like?", "limitations": "see questions", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-The motivation is clear, the algorithm design makes sense, and the experimental results are complete.", "weaknesses": "-Ablation study: Since the equivariance consists of two parts, namely translation and rotation, what would be the qualitative and quantitative impact of removing these two parts respectively?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*W1. Since the equivariance consists of two parts, namely translation and rotation, what would be the quantitative impact of removing these two parts respectively?*\n\nThank you for the valuable suggestion. We conducted an ablation study where we individually integrated only rotation equivariance and translation equivariance into the model. These were conducted using a short training schedule of 100K training steps due to the time limit.  The results, displayed in Table 1 of the rebuttal PDF, show that models without translation or rotation equivariance perform worse than our complete model.\n\n*Q1/2. The task setting of implicit depth estimation seems to be very compatible with the existing sparse view NeRF/GS methods. DeFiNe can synthesize novel view images, can EPIO do the same?* \n\nThank you for pointing that out, this is a great direction that we would like to explore in follow-up works. EPIO and DeFiNe were designed for “single-query” predictions, without relying on volumetric rendering (like NeRF) or explicit 3D structures (like 3DGS). Single-query novel view synthesis is challenging, especially in the generalizable setting, and DeFiNe itself does not report results in this task, but rather shows that by jointly learning novel view synthesis and depth estimation it can marginally improve depth estimation. Our proposed EPIO architecture could be extended to novel view synthesis, especially because it accepts traditional non-equivariant decoders (i.e., from DeFiNe), and we indeed show results on a toy novel depth synthesis experiment in Appendix O.2. Unfortunately, extending EPIO for an additional task and retraining the model is impractical in the rebuttal period, but we will aim to provide initial results (similar to the novel depth synthesis experiments) in the camera-ready version.\n\nHaving said that, there is a follow-up work to DeFiNe (DeLiRa) [1] which explores the use of Perceiver IO for volumetric rendering, focusing on novel view synthesis with depth estimation guidance. We believe EPIO could be used in this setting as well, and that would be a very interesting extension. \n\n[1] Guizilini et al. “DeLiRa: Self-Supervised Depth, Light, and Radiance Fields”, ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vxjny9S0tW", "reviewer": "Reviewer_xRYu", "summary": "This paper introduces an SE(3)-equivariant multi-view depth estimation model based on the Perceiver IO framework. Specifically, each feature ray is treated as a token, and the feature vector of each ray is concatenated with an equivariant positional embedding. To achieve equivariance, the authors propose using spherical harmonics to encode the ray poses. Ray features are treated as type-0 (rotation-invariant) irreps. These equivariant ray encodings are processed through several equivariant self-attention layers and aggregated into global features and a canonical reference frame. The camera pose encoding is first inverse-transformed into this inferred canonical frame, resulting in an SE(3)-invariant query. A series of cross-attention layers between the encoded global features and the query features is then used to predict pixel colors. The authors demonstrate the effectiveness of the proposed approach on the ScanNet and DeMoN datasets.", "questions": "1. According to the equations in Appendix F, different types of irreps do not mix in the self-attention layer. They also do not mix in the proposed nonlinearities in Appendix F. It seems like in the proposed method, each of the irreps (except for type-0) can only indirectly modulate other irreps of different types via attention. Am I correct?\n\n2. Subtracting the mean of the center is not stable under the addition or removal of camera points. Is it possible to use relative positional encoding, similar to rotary embedding, to achieve translational equivariance without relying on centroid subtraction?", "limitations": "1. Spherical harmonics can only encode the orientation of $t_i-\\bar{t}$ and not the length $|t_i-\\bar{t}|$. Therefore, typical SE(3)-equivariant networks address this by incorporating additional length encoding. However, in this paper, the distance information $|t_i-\\bar{t}|$ is discarded.\n\n2. Using irreps features inevitably introduces a band-limit. Increasing this band-limit is difficult because the feature dimension increases quadratically, which is also discussed by the authors.\n\n3. The authors also mentioned that higher-degree spherical harmonics caused instability in training. However, this might be due to the choice of equivariant nonlinearities. Liao et al. [3] reported that certain nonlinearities cause instability in training.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. To the best of the reviewers' knowledge, this is the first paper to address SE(3)-equivariant positional embedding for the transformer/PerceiverIO framework for multiview applications. While Fuchs et al [1]. and Liao et al [2,3]. have addressed SE(3)-equivariant attention for GNNs, their methods are more complex and computationally inefficient compared to the proposed approach.\n2. The proposed method shows competitive benchmark results compared to state-of-the-art methods across multiple datasets. The ablation study convincingly demonstrates the significance of the equivariant embedding.\n3. In the appendix, the authors put significant effort to make the concepts accessible to beginners,  including detailed visualizations for how the computations are done. This contrasts with typical papers on SE(3)-equivariance, which often include difficult equations that can be a barrier to entry for newcomers.\n\n\n\n\n[1] Fuchs et al., “SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks,” NeurIPS`20\n\n[2] Liao et al., “Equiformer: Equivariant graph attention transformer for 3d atomistic graphs,” ICLR’23\n\n[3] Liao et al., “Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations,” ICLR’24", "weaknesses": "1. The authors introduced a new equivariant nonlinearity inspired by [4], but the motivation and benefits are not clearly demonstrated. What is the distinctive advantage of this new nonlinearity, compared to existing SE(3)-equivariant nonlinearities?\n\n2. The number of parameters was not fixed during the ablation experiments regarding the maximum spherical harmonics degrees. A recent study [5] claimed that the reported increase in performance due to incorporating higher-type irreps in various works could actually be due to the increased number of parameters. It is essential to control the number of parameters to be similar between the ablated models and the proposed model.\n\n[4] Deng et al., \"Vector Neurons: A General Framework for SO(3)-Equivariant Networks,” ICCV’21\n\n[5] Wang et al., “Rethinking the Benefits of Steerable Features in 3D Equivariant Graph Neural Networks,” ICLR’24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*W1. The authors introduced a new equivariant nonlinearity inspired by [4], but the motivation and benefits are not clearly demonstrated. What is the distinctive advantage of this new nonlinearity, compared to existing SE(3)-equivariant nonlinearities?*\n\nOur nonlinearity, unlike norm and Gate nonlinearities, can change the direction of the feature, by reducing/keeping the projection of the tensor to an equivariance space. In the meanwhile, compared with the nonlinearity method that can change the direction of the feature by applying Fourier and Inverse Fourier transforms [1, 2], ours is more computationally efficient.\n\n[1]  Maurice, et al. \"General E(2)-equivariant Steerable CNNs.\"  NeurIPS 2019\n\n[2] Adrien, et al. \"A Functional Approach to Rotation Equivariant Non-linearities for Tensor Field Networks.\" CVPR 2021\n\n*W2. The number of parameters was not fixed during the ablation experiments regarding the maximum spherical harmonics degrees. A recent study [5] claimed that the reported increase in performance due to incorporating higher-type irreps in various works could actually be due to the increased number of parameters. It is essential to control the number of parameters to be similar between the ablated models and the proposed model.*\n\nThank you for the great suggestion. We agree that this is an essential variable to be controlled in these experiments. Thus, we include here additional ablation experiments on the ScanNet benchmark with increasing the numbers of parameters, by using the highest orders of spherical harmonic embeddings to match the parameter count of our model. These were conducted using a short training schedule of 100K training steps,  and the results are shown in Table 1 in the rebuttal PDF.  We observe a similar trend as reported in our submission (Table 3), showing that these improvements do not come from the increased model complexity. We will include these additional results in the camera-ready version of the paper. \n\n*Q1. According to the equations in Appendix F, different types of irreps do not mix in the self-attention layer. They also do not mix in the proposed nonlinearities in Appendix F. It seems like, in the proposed method, each of the irreps (except for type-0) can only indirectly modulate other irreps of different types via attention. Am I correct?*\n\nYou are correct. When generating key, query, and value features, we do not mix different feature types. While calculating the attention matrix, as shown in Appendix F, we use all types of features, which makes them indirectly modulate other irreps of different types.\n\n*Q2. Subtracting the mean of the center is not stable under the addition or removal of camera points. Is it possible to use relative positional encoding, similar to rotary embedding, to achieve translational equivariance without relying on centroid subtraction?*\n\nWe thank the reviewer for pointing that out. When there are two cameras (which is the primary setting explored in our experiments), the relative position is exactly twice our current translation (the translation after subtracting the centroid). We use spherical harmonics for our current translational positional embedding, which is equivariant to $SO(3)$ while rotary embedding is not equivariant to $SO(3)$.  For multiple cameras, introducing a relative positional encoding would be a good idea, however, this approach introduces the problem of permutation equivariance of the camera order in the Perceiver IO architecture (the index of the camera impacts the index of relative translation). We believe there is a more elegant solution to the translation equivariance problem, and hope to look into it in future work. \n\n\n*L1. Spherical harmonics can only encode the orientation of and not the length. Therefore, typical SE(3)-equivariant networks address this by incorporating additional length encoding. However, in this paper, the distance information is discarded.*\n\nAs stated in Section 3.3.1, we incorporate an additional radial component $||r||^l$  into the original “order-l” spherical harmonics of the corresponding degree. Despite this adaptation, these functions retain their fundamental characteristics and are still referred to as spherical harmonics. We provide an introduction and discussion of spherical harmonics in Appendix A.4, as well as the design of incorporating the invariant length. \n\n*L2. Using irreps features inevitably introduces a band-limit. Increasing this band-limit is difficult because the feature dimension increases quadratically, which is also discussed by the authors.*\n\nThanks for the deep comment. Yes, the dimension of the spherical harmonics is a limitation in our approach. That's why we address it in the decoder by predicting an equivariant frame followed by a conventional decoder, which enables higher frequency embedding for the query.\n\n*L3. The authors also mentioned that higher-degree spherical harmonics caused instability in training. However, this might be due to the choice of equivariant nonlinearities. Liao et al. [3] reported that certain nonlinearities cause instability in training.*\n\nThank you for the insightful remark. We have indeed observed the same behavior stated in [3] about the instability of $S^2$ activation, which treats the features as the Fourier coefficients for $S^2$, processing scalars, and high-order features together. They addressed this behavior by separating the scalars and higher-order features. On the other hand, our approach separates different orders of features and processes them independently (as mentioned in Q1). Replacing our nonlinearity with norm linearity did not stabilize training, suggesting that this instability is not solely due to the choice of nonlinearity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for the valuable comments and positive feedback regarding our submission. 

As mentioned by reviewer **xRYu**, we are the “first to address SE(3)-equivariance in the transformer-based Perceiver IO architecture for multi-view applications.” They also praise the significant effort put into our appendix, in an attempt to make the complex topic of neural network equivariance accessible to beginners. Reviewer **KQ34** states that our proposed method “can be inspiring for other tasks that require equivariance”, and that we “introduce the problem well, effectively explaining the importance of equivariance to the task of multi-view depth estimation”. Reviewer **QCgu** mentions that our general multi-view architecture can be extended beyond depth estimation to also improve novel view synthesis, which is an exciting direction for future work that we will explore. All reviewers equally praise the motivation behind our work, our algorithmic and design contributions, which include several novel equivariance components, and our state-of-the-art results in multi-view depth estimation, evaluated in multiple benchmarks, as well as convincing ablation studies. 

We address each point raised by the reviewers in their respective replies and will include all proposed modifications and additional experiments in the revised version of our manuscript. In particular, reviewer **KQ34** mentioned that our appendix contains important information that should be included in the main paper to improve reader experience. We are committed to balancing the main paper and appendix in order to give readers a fluent understanding of our motivation and contributions, as well as improving figures to help those not familiar with fundamental equivariance concepts. We also would like to emphasize that reviewer **QCgu** awarded us with an Excellent score for presentation.

Meta-review: This paper introduces a ray embedding representation with rotational and translational equivariance, integrating the existing Perceiver IO architecture to achieve robust multi-view implicit depth estimation. 
Concerns are about motivation and the ablation studies.
The authors well resolve the problems.
AC recommends acceptance for this paper.
Decision: Accept (poster)