Title: $SE(3)$ Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: $SE(3)$ Equivariance

Stereo Depth Estimation
Abstract: Incorporating inductive bias by embedding geometric entities (such as rays) as input has proven successful in multi-view learning. However, the methods adopting this technique typically lack equivariance, which is crucial for effective 3D learning. Equivariance serves as a valuable inductive prior, aiding in the generation of robust multi-view features for 3D scene understanding. In this paper, we explore the application of equivariant multi-view learning to depth estimation, not only recognizing its significance for computer vision and robotics but also addressing the limitations of previous research. Most prior studies have either overlooked equivariance in this setting or achieved only approximate equivariance through data augmentation, which often leads to inconsistencies across different reference frames. To address this issue, we propose to embed $SE(3)$ equivariance into the Perceiver IO architecture. We employ Spherical Harmonics for positional encoding to ensure 3D rotation equivariance, and develop a specialized equivariant encoder and decoder within the Perceiver IO architecture. To validate our model, we applied it to the task of stereo depth estimation, achieving state of the art results on real-world datasets without explicit geometric constraints or extensive data augmentation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an SE(3)-equivariant multi-view depth estimation model based on the Perceiver IO framework. Specifically, each feature ray is treated as a token, and the feature vector of each ray is concatenated with an equivariant positional embedding. To achieve equivariance, the authors propose using spherical harmonics to encode the ray poses. Ray features are treated as type-0 (rotation-invariant) irreps. These equivariant ray encodings are processed through several equivariant self-attention layers and aggregated into global features and a canonical reference frame. The camera pose encoding is first inverse-transformed into this inferred canonical frame, resulting in an SE(3)-invariant query. A series of cross-attention layers between the encoded global features and the query features is then used to predict pixel colors. The authors demonstrate the effectiveness of the proposed approach on the ScanNet and DeMoN datasets.\n\nSTRENGTHS:\n1. To the best of the reviewers' knowledge, this is the first paper to address SE(3)-equivariant positional embedding for the transformer/PerceiverIO framework for multiview applications. While Fuchs et al [1]. and Liao et al [2,3]. have addressed SE(3)-equivariant attention for GNNs, their methods are more complex and computationally inefficient compared to the proposed approach.\n2. The proposed method shows competitive benchmark results compared to state-of-the-art methods across multiple datasets. The ablation study convincingly demonstrates the significance of the equivariant embedding.\n3. In the appendix, the authors put significant effort to make the concepts accessible to beginners,  including detailed visualizations for how the computations are done. This contrasts with typical papers on SE(3)-equivariance, which often include difficult equations that can be a barrier to entry for newcomers.\n\n\n\n\n[1] Fuchs et al., “SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks,” NeurIPS`20\n\n[2] Liao et al., “Equiformer: Equivariant graph attention transformer for 3d atomistic graphs,” ICLR’23\n\n[3] Liao et al., “Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations,” ICLR’24\n\nWEAKNESSES:\n1. The authors introduced a new equivariant nonlinearity inspired by [4], but the motivation and benefits are not clearly demonstrated. What is the distinctive advantage of this new nonlinearity, compared to existing SE(3)-equivariant nonlinearities?\n\n2. The number of parameters was not fixed during the ablation experiments regarding the maximum spherical harmonics degrees. A recent study [5] claimed that the reported increase in performance due to incorporating higher-type irreps in various works could actually be due to the increased number of parameters. It is essential to control the number of parameters to be similar between the ablated models and the proposed model.\n\n[4] Deng et al., \"Vector Neurons: A General Framework for SO(3)-Equivariant Networks,” ICCV’21\n\n[5] Wang et al., “Rethinking the Benefits of Steerable Features in 3D Equivariant Graph Neural Networks,” ICLR’24\n\nQUESTIONS:\n1. According to the equations in Appendix F, different types of irreps do not mix in the self-attention layer. They also do not mix in the proposed nonlinearities in Appendix F. It seems like in the proposed method, each of the irreps (except for type-0) can only indirectly modulate other irreps of different types via attention. Am I correct?\n\n2. Subtracting the mean of the center is not stable under the addition or removal of camera points. Is it possible to use relative positional encoding, similar to rotary embedding, to achieve translational equivariance without relying on centroid subtraction?\n\nLIMITATIONS:\n1. Spherical harmonics can only encode the orientation of $t_i-\\bar{t}$ and not the length $|t_i-\\bar{t}|$. Therefore, typical SE(3)-equivariant networks address this by incorporating additional length encoding. However, in this paper, the distance information $|t_i-\\bar{t}|$ is discarded.\n\n2. Using irreps features inevitably introduces a band-limit. Increasing this band-limit is difficult because the feature dimension increases quadratically, which is also discussed by the authors.\n\n3. The authors also mentioned that higher-degree spherical harmonics caused instability in training. However, this might be due to the choice of equivariant nonlinearities. Liao et al. [3] reported that certain nonlinearities cause instability in training.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper introduces a ray embedding representation with rotational and translational equivariance, integrating the existing Perceiver IO architecture to achieve robust multi-view implicit depth estimation. The paper first utilizes the mean shift and spherical harmonics to achieve translational equivariance, and then builds upon this to use spherical harmonics to achieve a rotationally equivariant representation, ultimately combining to obtain a three-dimensional transformation embedding with equivariance. By further designing equivariant encoders and decoders, the paper realizes robust estimation of depth from new perspectives. Experiments on the ScanNet and DeMoN datasets demonstrate the effectiveness of the proposed method.

STRENGTHS:
-The motivation is clear, the algorithm design makes sense, and the experimental results are complete.

WEAKNESSES:
-Ablation study: Since the equivariance consists of two parts, namely translation and rotation, what would be the qualitative and quantitative impact of removing these two parts respectively?

QUESTIONS:
-The task setting of implicit depth estimation seems to be very compatible with the existing sparse view NeRF/GS methods. Although the focus of the two is different, with NeRF/GS focusing more on rendering images, while DeFiNe and EPIO mainly focus on geometry, there is a possibility of mutual exchange between the two. Can you report the comparative results with such methods? For example, ENeRF.

Lin, et al. "Efficient Neural Radiance Fields for Interactive Free-viewpoint Video", SIGGRAPH-ASIA 2022.

-DeFiNe can synthesize novel view images, can EPIO do the same? What are the results like?

LIMITATIONS:
see questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a SE(3) rotational and translational equivariant variation of Perceive IO for multi-view depth estimation with known camera poses. The authors first encode both the pixel-wise ray direction and the camera translation using spherical harmonics as the position encoding, and then to maintain equivariance under global transformations through the network's forward pass, the authors modify several components, including the linear projection, the latent array construction, and the output decoding. To demonstrate the effectiveness of the proposed method, the authors conducted experiments on several RGBD datasets, including ScanNet, SUN3D, TUM-RGBD, and Scene11, and achieved better performance than existing implicit multi-view depth estimations, such as DeFiNe, and multi-view stereo (MVS) models, such as DPSNet.\n\nSTRENGTHS:\n- The authors introduce the problem well, explaining the importance of equivariance to the task of multi-view depth estimation effectively. They also provide a brief yet sufficient review of existing works, clearly positioning this work within the field.\n- The authors have carefully designed several novel equivariant components:\n  - A SE(3) equivariant positional encoding, where besides rotation, the authors smartly encode camera translation also using spherical harmonics.\n  - An equivariant linear projection layer where the linear projection is applied to each group of features that corresponds to position embedding derived from the spherical harmonics of a specific order.\n  - Equivariant latent array construction and the reversal of the rotation from the latent array before being cross-attended to the output queries.\n  \n   These designs, along with the adoption of existing equivariant components through the Perceive IO pipeline, ensure good performance and can be inspiring for other tasks that require equivariance.\n- The experiments are sufficient and demonstrate the equivariance of the output and the overall accuracy.\n\nWEAKNESSES:\nThe major weakness of this paper lies in its presentation and organization, which makes the paper difficult to read:\n\n- Many important details from Sections 3.4 to 3.6 are placed in the appendix, making the main paper not self-contained. For instance, details in Appendices A.3 and E would be better suited in the main paper.\n\n- Sections 3.4 to 3.6 are organized into fragmented components, where the holistic process of the Perceiver IO is missing. Specifically, the authors should introduce each modification in the order of the Perceiver IO pipeline. \n\n- The description of individual components are also confusing:\n   - It is better to only briefly discuss components that are equivariant itself, such as attention, and discuss only how they made the input to the attention equivariant, such as the latent array in Section 3.5.1. Otherwise, it might be misleading to suggest that there are new equivariant attention modules themselves.\n   - Why is only rotation sampled and encoded when constructing the latent array in Section 3.5.2 and Figure 4, while the inputs have the encoded camera translation?\n   - Similarly, in Section 3.6, only reverse rotation is applied to the latents after several self-attention transformation blocks, while the translation is omitted.\n   - Line 261-262: \"which allows us to leverage higher frequency information beyond the dimensional constraints of SPH.\" The authors indicate that the Fourier encoding is not equivariant but use it for the output query, therefore, the authors should elaborate more on the insight behind this choice and provide sufficient proof to support this design.\n  -  Many illustrations (Figures 8-11) in the appendix are confusing and do not help to clarify the equations.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nThe limitations have been sufficiently discussed in the Appendix. Q.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}