NU-MCC: Multiview Compressive Coding with Neighborhood Decoder and Repulsive UDF
Authors: Stefan Lionar
Xiangyu Xu
Min Lin
Gim Hee Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: single-view 3d reconstruction
neural fields
3d reconstruction
TL;DR: A single-view 3D reconstruction method leveraging large-scale training data

Abstract: Remarkable progress has been made in 3D reconstruction from single-view RGB-D inputs. MCC is the current state-of-the-art method in this field, which achieves unprecedented success by combining vision Transformers with large-scale training. However, we identified two key limitations of MCC: 1) The Transformer decoder is inefficient in handling large number of query points; 2) The 3D representation struggles to recover high-fidelity details. In this paper, we propose a new approach called NU-MCC that addresses these limitations. NU-MCC includes two key innovations: a Neighborhood decoder and a Repulsive Unsigned Distance Function (Repulsive UDF). First, our Neighborhood decoder introduces center points as an efficient proxy of input visual features, allowing each query point to only attend to a small neighborhood. This design not only results in much faster inference speed but also enables the exploitation of finer-scale visual features for improved recovery of 3D textures. Second, our Repulsive UDF is a novel alternative to the occupancy field used in MCC, significantly improving the quality of 3D object reconstruction. Compared to standard UDFs that suffer from holes in results, our proposed Repulsive UDF can achieve more complete surface reconstruction. Experimental results demonstrate that NU-MCC is able to learn a strong 3D representation, significantly advancing the state of the art in single-view 3D reconstruction. Particularly, it outperforms MCC by 9.7% in terms of the F1-score on the CO3D-v2 dataset with more than 5x faster running speed.

NU-MCC: Multiview Compressive Coding with Neighborhood Decoder and Repulsive UDF
Stefan Lionar
Xiangyu 3e0
Min Lin
Gim Hee Lee²
Sea Al Lab
National University of Singapore
Xi an Jiaotong University
Project page:
io/
Abstract
Remarkable progress has been made 3D reconstruction from single-view RGB-D inputs. MCC is the current state-of- of-the-art method in this field, which achieves unprecedented success by combining vision Transformers with large-scale training. However, we identified two key limitations MCC: 1) The Transformer decoder is inefficient in handling large number of query points; 2) The 3D representation struggles to recover high-fidelity details. In this paper. we propose a new approach called NU-MCC that addresses these limitations. NU-MCC includes two key innovations: a Neighborhood decoder and a Repulsive Unsigned Distance Function (Repulsive UDF). First, Neighborhood decoder introduces center points efficient proxy of imput visual features, allowing each query point to only attend to small neighborhood. This design not only results in much faster inference speed but also enables the exploitation of finer-scale visual features for improved recovery of 3DD textures. Second, our Repulsive UDF is novel alternative the occupancy field used in MCC, significantly improving the quality of 3D object reconstruction. Compared standard UDFs that suffer from holes in results, our proposed Repul- sive UDF can achieve more complete surface reconstruction. Experimental results demonstrate that NU-MCC is able learn a strong 3D representation, significantly advancing the state of the art in single-view 3D reconstruction. Particularly, it outperforms MCC by 9.7% in terms of the Fi-score on the CO3D-v2 dataset with more than 5x faster running speed.
Introduction
3D reconstruction from single-view RGBD input is fundamental problem in computer vision with applications in robotics [1, 2] and VR/AR [3]. The state-of-the-art approach for this task MCC [4], which leverages large-scale multi-view images [5] to develop scalable model for 3D reconstruction from single RGB-D image. MCC utilizes the depths and 3D point clouds for supervision obtained using the COLMAP framework [6, 7J. By combining Vision Transformer [8, 9] with large-scale training, MCC can learn generalizable textured single-view 3D reconstruction model that generalizes diverse zero-shot settings.
However, we have identified two key limitations of MCC that affect its reconstruction quality and model efficiency. First, the Transformer decoder in MCC directly takes in the 3D locations of query points to predict their respective occupancy and color. Due to the quadratic complexity of Transformers, this approach incurs high computational cost when the number of query points is large, as is often the case for detailed 3D reconstruction. Second, MCC uses the occupancy field as the underlying 3D representation, which hinders the recovery of high-fidelity geometry and texture details. This can be observed in Figure where the reconstruction lacks intricate details.
Corresponding Author. Work was partially done Sea Lab.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

IMAGE
SEEN
MCC [4]
OURS
Figure 1: Single-view reconstruction given RGB-D input. Our approach achieves higher texture and geometry fidelity compared to the state-of-the-art method [4].
To address these limitations, we propose Neighborhood decoder, where each query point only attends to a small set features around its neighborhood and thus enhances the efficiency. One key component of the Neighborhood decoder is the anchor predictor that predicts sparse-yet-complete anchor proxies that captures the coarse shape and appearance the imput object. The Neighborhood decoder further aggregates the neighboring proxies of the query point for the final prediction. Impor- tantly, the number of anchors is fixed and therefore the computational cost in the Transformer does not depend the number of query points. This design not only results in much faster inference speed but also enables the exploitation of finer-scale visual features without significantly increasing the computational cost.
Our second contribution lies in the improved design of the output 3D representation. In MCC [4], the occupancy field requires selecting radius around ground truth points determine if query point is occupied or unoccupied during training. However, this radius selection poses challenges, with small radius leading incomplete surface coverage and large radius causing over-thickening reconstruction. In addition, the occupancy- -aased surface reconstruction process is wasteful since the empty query points are discarded. To overcome these limitations, query points to the closest surface using its gradient field. UDF does not require a radius and obtains supervision signals by calculating distances to the nearest ground truth points. While iteratively shifting points can move them to the surface, our UDF analysis as detailed in Section 3.3 shows that the point-shifting process tends to move points toward high-curvature regions like corners edges, which eventually results in hole artifacts. Therefore, we propose Repulsive UDF as an alternative to vanilla UDF that incorporates repulsive forces between nearby points. Our Repulsive UDF mitigates the hole artifacts and produces uniform distribution of points on the surface.
leverage UDF, which shifts
In summary, contributions are follows:
We propose NU-MCC, new single-view 3D reconstruction model with two key innovations: 1) Neighborhood decoder that efficiently handles large amounts of query points and combines both coarse and fine-scale visual features for 3D reconstruction, and 2) Repulsive UDF that achieves significantly higher surface quality than the occupancy field and standard UDF.
We conduct experiments on CO3D-v2 dataset [5] and show that our model outperforms the tite-of-the-rrt [4] by 9.7% with more than 5x faster inference speed. We also show that our method is generalizable to diverse challenging zero-shot single-view reconstruction settings, such as from iPhone RGB-D capture, Al-generated images [10], and ImageNet [11].
2 Related works
Single-view 3D reconstruction. Inferring the complete 3D shape from single image is chal- lenging problem due to the ambiguity caused by occlusions. Previous works have shown impressive single-view 3D reconstruction results using voxels [12-16], point clouds [17, 18], meshes [19-21],
and implicit representations [22-25]. However, the results are commonly demonstrated on simplistic synthetic dataset, such as ShapeNet [26] and Pix3D [27]. The notable exception is MCC [4] which learns compressive 3D appearance and geometry from large-scale multi-view images. While MCC [4] can generalize diverse in-the-wild settings, it operates slowly and produces reconstruction with lacking details due to simplistic Transformer decoder that attends to all visual features. In contrast, we propose an efficient Neighborhood decoder that attends to a small set of relevant local features.
Shape completion. Shape completion methods aim to complete the full geometry given incomplete parts. Relevant to our approach is PoinTr [28] which predicts proxies of the complete shape to

---

Neighoorhood Decoder
Tepur BOG
hedwors
Ourpi
SAP
Pralicter
Unnejette
Saet polne clead
Repaline LED
Figure 2: Overview of the proposed NU-MCC. Given an input single-view RGB-D image, we first unproject the pixels into the 3D world frame, resulting in textured partial point cloud. We then employ a standard VIT to extract visual features from the partial point cloud. Next, we introduce our Neighborhood decoder, which utilizes the extracted visual features to estimate the UDF and RGB values of each query point in 3D space. Unlike existing Transformer decoders where each query point needs attend to all the visual features, the Neighborhood decoder allows each query point (blue star in the figure) to attend to only a small set of features in its neighborhood, significantly improving the efficiency our model. During inference, the predicted UDF used shift the query points to the surface of the 3D object, leading to high-quality 3D reconstruction. The anchor predictor elaborated in Figure 3. More details regarding the point-shifting mechanism with the proposed Repulsive UDF are illustrated in Figure 4.
generate a fixed resolution of point clouds in explicit manner. Other methods employ conditional generation from learned shape priors [29, 30] for better and diverse completions, but suffer from slow generation speed and may not scale well for large-scale training data. In contrast to those methods, our approach generates implicit functions that allow any desired output resolution, and leverages large-scale data for training. Additionally, our method learns the object texture, which not addressed by those previous methods.
Neural fields for 3D reconstruction. Recently, numerous methods using neural fields have been proven effective for 3D reconstruction [31]. These approaches employ various output representations, such as occupancy [32], signed distance function [33], UDF [34], and radiance fields [35]. Among these representations, UDF methods have demonstrated their effectiveness in reconstructing open surfaces [34, 36, 37j. While early works in neural fields consider simple structures of latent represen- tations, more sophisticated structures such planes [38, 39], grids [38, 40-42), gaussians [43, 44], and anchors [45] have been proposed for more efficient reconstruction. We adopt anchor structure for efficiency in the attention mechanism [8] in our Neighborhood decoder and propose Repulsive UDF as an alternative output representation.
3 Method
We introduce NU-MCC, a novel approach for 3D reconstruction from a single-view RGBD image. Inspired by MCC [4], the proposed algorithm employs an encoder-decoder architecture as illustrated in Figure 2. While following MCC design the encoder, we propose new Neighborhood decoder which offers improved efficiency over the baseline decoder without compromising performance. Moreover, unlike the original MCC that uses occupancy fields, we propose Repulsive UDF that significantly enhances the reconstruction results.
3.1 Encoder
We use the same encoder as MCC [4] to process the input RGB-D image. Denoting I E Hxxxx as the RGB image and E HxW as the depth map, we first unproject the depth map D into the world frame and obtain the 3D positions of the pixels P E R xx xx3, Essentially, the image / and 3D position map P describe the textured seen point cloud, which both noisy and incomplete as shown in Figure 2. We then embed the partial point cloud into sequence of feature tokens R E RR xd
3

---

using ViTs [8, 9], where N = 14 x + 1 is the number of tokens (14 X 14 patch tokens plus global token), and is the number of feature channels. We refer to MCC [4] for more details of the encoder.
3.2 Neighborhood decoder
In this section, we discuss the decoder component, which takes the output of the encoder R and Ng point queries in 3D space to predict UDF and colors for each point.
Existing decoders [4] commonly employ a simple concatenation strategy, where the N visual tokens in R and the Ng query tokens are first concatenated and fed into normal Transformer layers. However, this approach incurs a high computational cost of O ((N + N,))), particularly when the number of queries N₂ is large, is often the case for detailed 3D reconstruction.
To address this issue, our key insight is that the inefficiency stems from each query having to interact with all visual features. A better solution should allow each query to efficiently attend to small subset of tokens that are most relevant to it. To this end, we introduce anchors as proxy representation of visual features R, which facilitates the identification of relevant features for each query.
Anchor predictor. Our anchor representation is denoted as F. = (Z. X.) ) E IR Mx which consists of two parts. First, Ze IR. Mxd is set of anchor features, which represents transformed vi- sual features. Second, X E IR denotes the 3D locations of the anchors (green points in Figure 2), which serve as the identifier of each feature vector. M represents the number of anchors, while the dimension of each anchor vector. This design enables each query point q E IR3 to conveniently identify its relevant features by locating the m-nearest anchors within its 3D neighborhood based on the Euclidean distance from q to X
As shown Figure we employ Transformer as the anchor predictor:
Anchor Predictor
F- Z X.) = G(R).
Zo
global feature d-dm
To bootstrap the anchor predictor, we begin by ran- domly initializing set of learnable positional em- beddings for each anchor, denoted as Eg E IR [xxx Subsequently, we incorporate the global token Zo into Eg through broadcasting and addition, resulting in the anchor embeddings E TR Mxd. Next, concatenate the visual feature with the anchor em- beddings E, which then passed through standard Transformer layers, yielding the anchors F. and an updated global feature vector z. The output patch tokens (the grey squares in Figure 3) are not used in subsequent modules. Note that the computational cost of the anchor predictor O ((N + M))). which does not depend on the number of query points, making it manageable for a reasonable number of anchors M1
R:N x
Transformer G
M:Mx4
P.:Mx
Figure 3: Illustration of the anchor predictor.
To ensure the generation meaningful anchor representations, we directly supervise the anchor locations X. during training by encouraging the anchors be close to the ground-truth 3D surface. The anchor features Z, are not directly supervised and are only trained with the final RGB and UDF prediction loss. The predicted anchors can be seen sparse- yeass--yy-- point cloud that captures the coarse shape and apperance of the input object. This approach allows us to efficiently generate informative and context-aware anchor representations for our Neighborhood decoder.
Feature aggregation. With the proposed anchor representation, we generate the feature of each query point q. denoted as zzz E RRd by aggregating the anchor features in its neighborhood. Our aggregation process is derived from [46, 45] and detailed as follows:
W ④ ( Wo),
where Zo. E Rn represents the m-nearest anchors to with Z denoting the i-th row. A linear projection layer W. 1ddd is applied to each anchor feature before aggregation. W e gd denotes

---

the aggregation weight of the i-th nearest anchor Z and @ is the Hadamard product of two vectors. The vector- based weighted sum provides more flexibility than scalar weighted sum by allowing different weights for different channels.
The aggregation weights W RRuxd are estimated based three factors: 1) the displacement between and the m neighboring anchors, denoted by Aq E R=xx,, 2) the features of the neighboring anchors Ze and 3) the global token z obtained from the anchor predictor. The estimation process can be written as:
W = a (tb (zW, + Ze.q Wx 5(Aa)))
where both to and å are two-layer MLPs, and Wa and W₂ denote linear projection layers. a is a softmax function along the first dimension, normalizing the aggregation weights across different anchors. The global token vector is broadcasted before being added to the matrices.
Finally, we can map the aggregated feature z to the UDF f(q) and color c(q) of the query point
f(g), c(q) = Vg 2gg)
where V/ is an MLP as shown in Figure 2.
Incorporating fine features. Considering the large patch size used in the VIT encoder, typically 16 x 16, we hypothesize that the encoded representation R. as well as the anchors features Ze obtained from R. mainly capture coarse information from the input. This limitation hinders existing methods, such as MCC, from recovering fine details in the reconstructed output.
In addition to the high efficiency, another advantage of our proposed Neighborhood decoder is its ability to conveniently incorporate fine-scale information directly from the input, without significantly increasing computational costs. Specifically, as the RGB-D input provides textured partial point cloud as introduced in Section 3. we can transform the input into a format similar to the anchors and construct fine features as F f P} (( Here, Z, E (HW) is obtained by applying a linear projection layer to the RGB values, representing the low-level color information of each pixel. P represents their corresponding locations in 3D space as introduced in Section 3.1.
To integrate the coarse anchor features F. with the fine input features F,, we identify the m nearest features from F, and the nearest features from F for each query point q. These features are concatenated to form a new feature set Zg E R (md+n)xxd accompanied by their respective 3D locations. Subsequently, we can perform the feature aggregation as in Eq. 1 by replacing Ze.q with Zq By aggregating the concatenated feature set Zq we achieve an enhanced representation that incorporates both coarse and fine-scale information, facilitating more detailed and accurate 3D) reconstruction.
Model flexibilityy Since the query points in our Neighborhood decoder attend to features pinpointed in continuous space, our Neighborhood decoder provides some flexibility to control the reconstruction quality. First, a higher resolution of fine input features can be adopted at test time in the feature aggregation process to generate higher-detailed reconstruction. Second, the feature aggregation mechanism in our Neighborhood decoder allows flexible number of features that can differ during training. Aggregating higher number of features result in cleaner and smoother reconstruction that can be helpful in handling input with noisy depth.
3.3 Repulsive UDF
MCC [4] utilizes the occupancy field to represent 3D objects. While achieving state-of-the-art performance, the occupancy representation faces two significant challenges. First, during training, determining whether point is occupied involves checking if it falls within radius of any point in the ground-truth point cloud. This causes the issue of selecting an appropriate radius. A small radius may lead to incomplete surface coverage, resulting in false supervision signals. Conversely, large radius can cause thhr-thin of reconstructed results, as many non-surface points may be classified occupied.
Second, during inference, we need to uniformly query the 3D space for reconstruction. Thus, a significant portion of the query points will be empty and do not directly contribute to the reconstructed surface. This could lead to insufficient density in the 3D reconstruction, as many areas on the surface may be under-queried and lack adequate number of points for accurate reconstruction.
5

---

Attractive force
Surface
Repulsive
force
Surface
(a) Force analysis
(b) Potential field
(c) Mation field
(d) Point shifting result
Figure 4: Comparison between the standard UDF (top) and the Repulsive UDF (bottom). By improving the potential field (b), rectify the motion field (c) for the point-shifting process, resulting in improved surface reconstruction (d). The blue points in (d) are the initial query points randomly sampled in the space, and the orange
are the queries after point shifting.
Recent studies [34] show that UDF addresses the above issues effectively. The UDF ((g) is defined the closest distance from the point q to the surface. During the training of UDF, one directly regress the distance to the closest point in the ground truth, avoiding the radius selection dilemma in the occupancy field.
Meanwhile, as (q) a potential field as shown Figure 4(b)-top, the points in the 3D space can move to the surface by flowing from a high- eigh-potential region to a low-potential region, resembling the water or electricity flow. Thus, during inference, most of the queried points can be iteratively shifted to the surface with the following process:
Vaf(g)
q f(g)
[
where the point q moves along the gradient field fqf(q) in Figure 4(c)-top, which is induced from the potential field. ||-|| is the Euclidean norm.
In this work, we introduce UDF into NU-MCC, which ensures greater number of query points contribute the formation of the 3D surface. However, we find that the standard UDF suffers from drawback that the query points tend to be attracted to regions with large curvature during point shifting, which often leads holes in Figure 4(a)-top, where the two query points qi and q2 are drawn towards the corner area of the surface with high curvature. This phenomenon also evident in the motion field in Figure 4(c)-top, where a large portion of arrows point to the corner region. As result, the reconstructed surface exhibits undesirable holes as shown in Figure 4(d)-top.
regions. We visualize this observation with a toy example
To mitigate this issue, we propose the Repulsive UDF to encourage more uniform distribution of query points on the reconstructed surface. As shown in Figure 4(a)-bottom, we aim develop a new potential field that provides repulsive force between qı and q2, preventing an excessive concentration of points in certain areas. Thus, we design the new potential field for qi as:
(5) where ((x) is monotonically decreasing function that assigns high potential when qı moves close to q2. We empirically find g(x) = (z) yields stable results. The rectified potential field and vector field shown in Figure 4(b)-bottom and (c)-bottom. To shift to the low-potential region of Eq. 5, we alternatively move in the inverse direction V. f(q)) and V 9(|qq) - qall), fostering more balanced transition the 3D surface.
f(qi) + Ag(I q2ll),
Additionally, to incorporate more surrounding points to build the potential function, we select k-nearest points of query q. denoted as Nq. This leads to the gradient field:
Vg Σ g(lg qull) = Va NE
In
q q
(6)
q
q
q
NeN
N

---

To prevent excessively large forces, we apply gradient clamping during implementation. As illustrated in Figure 4(d)-bottom, the proposed Repulsive UDF effectively addresses the problem of UDF and achieves a more comprehensive surface reconstruction.
3.4 Optimization details
We supervise the coarse anchor locations X. using L1 -hhhmeeeer distance loss with respect to the ground truth points sampled sparsely using farthest point sampling (FPS). The UDF predictions are supervised by the UDF between query and ground truth points using L, loss with clamping of maximal distances [34]. Following MCC [4], the colors are supervised by the colors of closest ground truth points within 0. radius from the query points with cross-entropy loss for each color channel. We randomly sample 550 query points from the 3D world space per training batch.
4 Experiments
We conduct extensive experiments to show the representational power and generalization capability of NU-MCC for object-level single-view reconstruction using CO3D-v2 dataset [5]. Additionally, we show the zero-shot generalization for in-the-wil reconstruction of RGB-D iPhone capture, Al generated images [10], and ImageNet [1 [] in Section 4.2.
Dataset. We follow the task established from MCC [4] which uses CO3D-v2 dataset [5] for single- view object reconstruction. CO3D-v2 contains approximately 37,000 short videos of 51 object categories in real-world settings. We use the rraiiin-vvlldattion split from MCC all categories experiment. The reconstruction task focuses the foreground objects identified by the segmentation masks provided in CO3D dataset. CO3D provides the point cloud of the object and depth maps obtained from the videos via COLMAP [6, 7], albeit noisy. The sparsely sampled point clouds (20,000 points) from the COLMAP reconstruction serve as ground truths. The objects are normalized to have zero-mean and variancee The query points are sampled from [-3, 3] along each axis.
Metrics. To evaluate the geometry. we report the Fi-score with distance threshold of 0.1 following MCC [4]. Additionally,
also show the L Ly-chaffer distance (CD) for evaluations omitting the sensitivity of the chosen distance threshold in Fl-score. We also introduce Li-GGB distance metrics to evaluate the colors. The I, RRB distance calculates the L -n-nmm of the RGB difference between the predicted points and their nearest ground truths located within 0. radius. We detail the mathematical formulation of the RGB distance in the supplementary material.
Training details. Our model is trained with an effective batch size of 512 using 4xNVIDIA A100 GPUs for 100 epochs. One epoch takes approximately 2 hours. We follow the optimizer and 3D data augmentation of MCC. Adam optimizer [47] with base learning rate of 10 cosine schedule, and linear warm-up up for the first 5% of iterations are used. 3D data augmentation is performed by random scaling of s E [0.8, 1.2] and rotation B E [-000, 180%] along each axis.
Implementation details. We employ 200 coarse anchor representations. Each query point attends to its 4-nearest coarse anchor features and 4-nearest fine features in the feature aggregation. During training. we set the distance clamping for UDF supervision to 0.5. At test time, we shift points with an initial UDF prediction lower than 0.23 for 10 iterations. To calculate the repulsive force, we process batches of 48,000 points where each point considers k = 16 nearest neighbors. We clamp the shifting gradient to [-0.03, 0.03] in each direction.
4.1 Results on CO3D-v2 Validation Set
The quantitative results on CO3D-v2 [5] validation set summarized in Table 1. We first conduct an ablation study of our decoder design with occupancy as the output representation. The occupancy is supervised following MCC [4] where query point is considered "occupied" if it is located within a radius of 0.1 to a ground truth point, and "unoccupied" otherwise. Utilizing our Neighborhood decoder with only coarse anchor features results in comparable metrics MCC [4], but with 15x inference speedup. The incorporation of fine features in the feature aggregation process produces notably more accurate colors and slightly better geometry.
Replacing the occupancy with UDF potentially enhances the geometry and color. However, as seen in Figure 5, iteratively shifting points towards surface results in holes due to some points clumping in
7

---

ARCHITECTURE
LyCDD
LGRGBB
RUNTIME
MCC [4]
Oce
0.284
76.4
0.376
18.5
fine)
Ours
Oce
0.282
79.0
0.340
1.5
0.264
0.316
RepUDF
0.237
83.8
0.320
3.5
Table I: Quantitative results on CO3D-v2 [5] validation set.
Figure 5: The effect of re- pulsive force. Left: Stan- dard UDF. Right: Repul-
Results are obtained using 216k query points and averaged over three different views.
sive UDF
IMAGE
SEEN
GT
MCC [4]
OURS
Figure 6: Qualitative comparison CO3D-v2 [5] validation set. Our method captures higher details on the seen part and predicts the occlusion more accurately.
corners and edges. The use of the repulsive field mitigates the hole artifacts and creates surface with uniformly distributed points, which overall produces the best geometry and color.
Figure 6 shows the qualitative comparison with MCC [4]. While MCC [4] fails to reconstruct the fine details, such the eyes of the teddybear and the intricate structure of the motoreycle, our method recovers significantly better details on the seen part, while more sensibly reconstructing the occlusion.
4.2 Zero-Shot Generalization In-the-Wild
We show the generalization capability of NU-MCC for in-the-wild reconstructions of RGB-D iPhone capture, AI generated images (Stable Diffusion [10]), and ImageNet [11]. Off-the-shelf segmentation tool [48] and depth estimator [49] are used generate the object masks and depth images of the ImageNet [11] and Al generated [10] images. These experiments are challenging due to the large distribution shifts of depth images, object categories, and scene settings. Figure shows NU-MCC is capable of accomplishing faithful reconstructions in zero-shot settings, including challenging object classes such as lawnmower ImageNet (11) example.
Conclusion
We present NU-MCC, a novel single-view 3D reconstruction model leveraging large-scale training. Our approach includes key innovations: Neighborhood decoder and Repulsive UDF that in combination result in fast inference speed and high-quality textured reconstruction. We show that our model generalizable challenging zero-shot settings with large distribution shifts from training data, such as iPhone capture, Al generated images, and images curated from the web.
We discovered and rectified a bug in MCC [4] Fl-score calculation. Additionally, we use score threshold of 0.4 that results in the best chamfer distance. Thus, reported Fl-score is much higher than MCC paper.

---

IMAGE
SEEN
MCC
OuRS
iPhone
Generative AI
ImageNet
Figure 7: Zero-shot generalization in-the-wild, NU-MCC trained on CO3D-v2 [4] is capable of faithful reconstructions of challenging in in-the-wild images with large distribution shifts of depths, object categories, and scene settings with higher details than MCC [4].
Limitation. While NU-MCC produces remarkable results compared to the state-of-the-an [4], is still inherently challenging to reconstruct the occluded parts with high-fidelity details. Additionally, our results degrade when there is a significant amount of outliers and noise in the input depth, which may result in "spikes" "bumps", although these artifacts be alleviated by selecting higher number of features in the feature aggregation.
Acknowledgments
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-022 and the Tier grant MOE-T2EP20 12E22012000011 from the Singapore Ministry of Education. The authors acknowledge the computational resources provided by the HPC platform of Xi' an Jiaotong University.
References
[1] L.. Schmid, M. N. Cheema, V. Reijgwart, R. Siegwart, F. Tombari, and C. Cadena, "Incremental 3d scene completion for safe and efficient exploration mapping and planning," arXiv preprins arXiv:2208.088 08300, 2022.
[2] R. Menon, T. Zaenker, and M. Bennewitz, "Viewpoint planning based on shape completion for fruit mapping and reconstruction," arXiv preprint arXiv:2209. 15376, 2022.
[3] R. G. Boboc, E. Băutu, Girbacia, N. Popovici, and D.-M. Popovici, "Augmented reality in cultural heritage: An overview of the last decade of applications," Applied Sciences, 2022.
[4] C.-Y. Wu, Johnson, J. Malik, C. Feichtenhofer, and G. Gkioxari, "Multiview compressive coding for 3d reconstruction." in CVPR, 2023.

---

[5] J. Reizenstein, R. Shapovalov. P. Henzler, L. Sbordone, P. Labatut, and D. Novotny, "Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction." in ICCV, 2021.
[6] L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys, "Pixelwise view selection for unstructured multi-view stereo," in ECCV, 2016.
[7] Schonberger and J.-M. Frahm, "Structure-from frommmooinn revisited," CVPR, 2016.
[8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin, "Attention is all you need," in NIPS, 2017.
[9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani,
M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in ICLR, 2021.
[10] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," " in CVPR, 2022.
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in CVPR, 2009.
[12] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese, "3d-r2n2: A unified approach for single and multi-view 3d object reconstruction," in ECCV, 2016.
[13] R. Girdhar, D. F. Foubey, M. Rodriguez, and A. Gupta, "Learning predictable and generative vector representation for objects," in ECCV, 2016.
[14] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik, "Multi-view supervision for single-view reconstruction via differentiable ray consistency," in CVPR, 2017.
[15] J. Wu, Y. Wang, T. Xue, X. Sun, B. Freeman, and J. Tenenbaum, "Marmet: 3d shape recon- struction via 2.5 d sketches," NIPS, 2017.
[16] J. Wu, C. Zhang, X. Zhang, Z. Zhang, W. T. Freeman, and J. B. Tenenbaum, "Learning shape priors for single-vies 3d completion and reconstruction." in ECCV, 2018
[17] H. Fan, H. Su, and L. J. Guibas, "A point set generation network for 3d object reconstruction from a single image," in CVPR, 2017.
[18] P. Mandikal, K. Navancet, M. Agarwal, and R. V. Babu, "3d-Imnet: Latent embedding matching for accurate and diverse 3d point cloud reconstruction from single image," in BMVC, 2018.
[19] N. Wang. Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, "Pixe12mesh: Generating 3d mesh models from single rgb images," in ECCV, 2018
[20] T. Groueix, M. Fisher, G. Kim, B. C. Russell, and M. Aubry, "A papier-måché approach to learning 3d surface generation," CVPR, 2018.
[21] W. Wang, D. Ceylan, R. Mech, and U. Neumann, "3dn: 3d deformation network," in CVPR, 2019.
[22] Q. Xu, W. Wang. D. Ceylan, R. Mech, and U. Neumann, "Disn: Deep implicit surface network for high-quality single-view 3d reconstruction." in NeurlPS, 2019.
[23] R. Wu, Y. Zhuang, K. Xu, H. Zhang, and B. Chen, "Pq-net: generative part seq2seq network for 3d shapes," in CVPR, 2020.
[24] Y. Jiang, D. Ji, Z. Han, and M. Zwicker, "Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization," in CVPR, 2020.
[25] C. Zhang, 7. Cui, Zhang, Zeng, Pollefeys, and S. Liu, "Holistic scene understanding from single image with implicit representation." in CVPR, 2021.
10

---

[26] A. X. Chang. T. Funkhouser, L. Guibas, P. Hanrahan. Q. Huang, Z. Li, S. Savarese, M. Savva, Song, H. Su et al., "Shapenet: An information-rich 3d model repository," arXiv preprint arXiv: 1512.0301, 2015.
[27] X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B. Tenenbaum, and W. T. Freeman, "Pix3d: Dataset and methods for single-image 3d shape modeling," in CVPR, 2018.
[28] X. Yu, Y. Rao, Z. Wang, Z. Liu, J. Lu, and J. Zhou, "Pointr: Diverse point cloud completion with geometry-aware transformers," in ICCV, 2021.
[29] P. Mittal, Y.-C. Cheng, M. Singh, and S. Tulsiani, "Autosdf: Shape priors for 3d completion, reconstruction and generation," in CVPR, 2022.
[30] X. Yan, L. Lin, N. J. Mitra, D. Lischinski, D. Cohen-Or, and H. Huang, "Shapeformer: Transformer-based shape completion via sparse representation," in CVPR, 2022.
[31] Y. Xie. T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, "Neural fields in visual computing and beyond," in Computer Graphics Forum, 2022.
[32] L. Mescheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. "Occupancy networks: Learning 3d reconstruction in function space," in CVPR, 2019.
[33] J. Park, Florence, Straub, R. Newcombe, and Lovegrove, "Deepsdf: Learning continuous signed distance functions for shape representation," in CVPR, 2019.
[34] J. Chibane, G. Pons-Moll Moll al., "Neural unsigned distance fields for implicit function learning," in NeurIPS, 2020.
[35] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, "NeRF: Representing scenes as neural radiance fields for view synthesis," " ECCV, 2020.
[36] X. Long, C. Lin, L. Liu, Y. Liu, P. Wang. C. Theobalt, T. Komura, and W. Wang, "Neuraludf: Learning unsigned distance fields for multi-view reconstruction of surfaces with arbitrary topologies," in CVPR, 2023.
[37] Y.T.. Liu, L. Wang, W. Chen, X. Meng, B. Yang, L. Gao et al., "Neudf: Leaning neural unsigned distance fields with volume rendering," in CVPR, 2023.
[38] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger, "Convolutional occupancy networks," in ECCV, 2020.
[39] S. Lionar, D. Emtsev, D. Svilarkovic, and S. Peng. "Dynamic plane convolutional occupancy networks," in WACV, 2021.
[40] Chibane, T. Alldieck, and G. Pons-Moll, "Implicit functions in feature space for 3d shape reconstruction and completion," in CVPR, 2020.
[41] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, "Tensorf: Tensorial radiance fields," in ECCV, 2022.
[42] C. Reiser, S. Peng, Y. Liao, and A. Geiger, "Kilonerf: Speeding up neural radiance fields with thousands of tiny mips," ICCV, 2021.
[43] K. Genova, F. Cole, D. Vlasic, A. Sama, W. T. Freeman, and T. Funkhouser, "Learning shape templates with structured implicit functions," in ICCV, 2019.
[44] K. Genova, Cole, A. Sud, A. Samna, and T. Funkhouser, "Local deep implicit functions for 3d shape," in CVPR, 2020.
[45] S. Giebenhain and B. Goldlückee "Air-nets: An attention-based framework for locally condi- tioned implicit representations," in 3DV, 2021.
[46] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun, "Point transformer," in ICCV, 2021.
11

---

[47] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in ICLR, 2015.
[48] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitchead, A. C. Berg, W.-Y, Lo et al., "Segment anything."" arXiv preprint arXiv:2304. 02644,,, 2023.
[49] R. Ranftl, A. Bochkovskiy, and Koltun, "Vision transformers for dense prediction," in /CCV, 2021.
12


Reviews:
[{"review_id": "iwVE6NUS54", "reviewer": "Reviewer_yY6d", "summary": "The paper proposes a novel framework for generating 3D object from single view RGB-D input. Based on the previous work MCC, this work implements a neighborhood decoder that is more efficient and a repulsive udf representation that has higher quality than the previous occupancy field.", "questions": "Please refer to weakness.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The major improvements are sound. I appreciate the neighborhood decoder that efficiently combines coarse and fine-scale features for 3D generation. The repulsive UDF is also convincing.", "weaknesses": "- What limits the generation resolution? Is it because of the point cloud quality in Co3D? Does the framework support rendering arbitrary resolution result images at inference time?\n- When inferencing on in-the-wild images, does the model require segmenting out the foreground mask? Fig.2 seems a bit confusing because the image involves a background.\n- How to determine the focal length and depth scale for in-the-wild images?\n- How are the camera poses set for the input at training and inference time? Do you use the ground truth camera poses provided in CO3Dv2 dataset? Are the generated 3d objects placed in their canonical pose, or are they aligned with the input camera (meaning that there's one-to-one mapping between input image and output 3d object?\n-  Does the pipeline generate plausible 3d objects when the input depth is problematic (e.g. distorted / noisy / scaled)? For example, when working with monocular depth.\n- How many neighbors are used for the feature aggregation module? What's the value of $m$ in Eq. (1)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the appreciation of the novelties introduced in our paper. We address your questions below.\n\n**Generation resolution:** Since the proposed NU-MCC learns a continuous neural field (UDF), we can query an arbitrary number of query points at inference. Note that the point generation process is much more efficient than the baseline algorithm MCC: the proposed method has linear complexity w.r.t. number of query points in a small feature aggregation network, while MCC incurs quadratic complexity in its heavy Transformer decoder.\n\n**Mask:** Following MCC, we apply a foreground mask to the input depth map, which is obtained via an off-the-shelf segmentation model.\n\n**Focal length and depth-scale:** For in-the-wild images whose focal length is unknown, we manually tune the focal length to ensure reasonable (not overly stretched/squeezed/distorted) point clouds from the depth map. This tuning process is independent from the subsequent 3D reconstruction approach, and we apply the same focal length to both our method and the baseline approaches. Note that it is possible to automate this procedure by techniques such as LeReS (CVPR 2021), which will be explored in future work. For the depth scale, the unprojected point coordinates are normalized as in MCC.\n\n**Camera pose:** We use the ground truth pose from CO3D-v2 dataset. The 3D generation is aligned with the input camera and thus also with the unprojected seen points.\n\n**Generation with problematic depth:** 1) Noisy. Our model has been trained with depth images obtained from COLMAP that are noisy. Therefore, it will be robust to noisy depth maps up to a certain extent. Under more severe noisy depths, the proposed Neighborhood decoder allows us to aggregate more point features to alleviate the artifacts caused by the noise (as explained in the supplement Sec. 2.2). 2) Distorted. Distorted depths will lead to distorted seen points, which in turn will result in distorted reconstruction. 3) Scaled. The depth scale does not matter since the unprojected points will be normalized. \n\n**Number of neighbors:** As stated in Line 267, each query point attends to its 4-nearest coarse anchor features and 4-nearest fine features in the feature aggregation, i.e., $m=n=4$.\n"}, {"review_id": "SP1a4RwfPA", "reviewer": "Reviewer_97oK", "summary": "This paper embarks on the task of 3D reconstruction using single-view RGB-D inputs. Building on Multiview compressive coding (MCC), it proposes two enhancements: 1) incorporating nearest neighbors in the decoder to minimize query time, and 2) the utilization of a repulsive unsigned distance function (UDF) as the output representation. The former also introduces an intermediate representation of anchor points for efficient NN query, while the latter replaces occupancy, delivering a higher quality of 3D reconstruction.", "questions": "1. Line. 175, the dimension of F_f is (HW)x(dx3), are all points unprojected to the 3D space?\n2. Line. 254, does the number of GT points affect the model performance?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper effectively builds upon the latest research in Multiview Compressive Coding (MCC), addressing two main limitations encountered in the original study: 1) a relatively slow query speed and 2) the lack of high-fidelity details in the 3D reconstruction. In my opinion, this research is original, and I particularly like the idea of the repulsive unsigned distance function (UDF), and I believe it provides contributions to scholarly discourse in this field.", "weaknesses": "The manuscript could benefit from a more in-depth explanation of the anchor predictor's impact when it is used as an intermediate representation, particularly when neighborhood points are drawn from both the anchors and the initial point cloud. Augmenting the work with an additional ablation study could aid in comprehending its effectiveness and provide more depth to the reader's understanding. \n\nIn the supplementary, the quality of the scene-level reconstruction is acceptable, but it does not seem to be remarkable. It would be beneficial if the authors could identify and discuss the major barriers preventing high-fidelity scene reconstruction in their study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the overall positive opinion to the contributions introduced in our paper. We address your concerns below.\n\n**Ablation study on neighborhood points:** We conduct an additional ablation study: aggregating different numbers of anchor points and seen points in the proposed neighborhood decoder. Results are as follows:\n\n| **n anchors** | **n seen points** | **Color (&#8595;)** | **Prec (&#8593;)** | **Recall (&#8593;)** | **F1 (&#8593;)** | **Acc (&#8595;)** | **Comp (&#8595;)** | **CD (&#8595;)** |\n|---------------|-------------------|---------------------|--------------------|----------------------|------------------|---------|----------|--------|\n| 4             | 0                 |               0.449 |               80.6 |                 58.0 |             66.3 |   0.103 |    0.213 |  0.316 |\n| 0             | 4                 |               0.393 |               41.3 |                 21.0 |             24.7 |   0.188 |    0.607 |  0.795 |\n| 4             | 4                 |               0.320 |               84.8 |                 83.7 |             83.8 |   0.103 |    0.134 |  0.237 |\n| 12            | 12                |               0.331 |               88.1 |                 79.7 |             83.0 |   0.089 |    0.160 |  0.249 |\n\nComparing row 1-3, we find that the features from predicted anchors and seen points are complementary to each other, and combining both features leads to high-quality reconstruction results.\nIn addition, increasing the number of aggregation points exhibits a smoothing effect (example shown in Sec. 2.2 of the supplement). For row 4, as smoothing operations typically decrease variance, the recovered point cloud concentrates more on high-confidence regions (high precision) and reduces coverage of the whole surface (low recall).\n\n**Scene-level reconstruction:** When applying our approach to scene reconstruction, we observe that the majority of anchor points are concentrated on prominent structures, such as walls, floors, and ceilings. Conversely, only a minority of anchors are situated on finer structures like chairs, sofas, and lamps. This distribution could potentially limit the fidelity of scene reconstruction. In future work, we intend to investigate methods to encourage anchor points to better distribute on intricate structures to address this limitation.\n\n**Are all points unprojected to the 3D space?**  We only unproject the pixels in the foreground to 3D points and set the points outside the foreground to a very high value so that they are not selected in the feature aggregation.\n\n**Does the number of GT points affect the model performance?** A higher number of GT points will likely result in a finer reconstruction. However, it will incur slower and more memory intensive training since each query point needs to search for its nearest GT point to obtain the GT UDF.\n"}, {"review_id": "0COU4emJtp", "reviewer": "Reviewer_WpNt", "summary": "The paper introduces a novel method, NU-MCC, for reconstructing 3D objects from single-view RGB-D images. The proposed approach builds upon the existing MCC method by incorporating two important advancements. Firstly, it introduces a neighborhood decoder to efficiently manage a large number of query points, which helps to reduce computational complexity and improve processing speed. Secondly, it adopts a Repulsive unsigned distance function (Repulsive UDF) to enhance the quality of surface reconstruction, resulting in more accurate and detailed reconstructions. The authors have conducted extensive experiments on multiple datasets to evaluate the effectiveness of their proposed approach. The results demonstrate that NU-MCC outperforms the existing state-of-the-art methods, achieving higher accuracy and better quality reconstructions.\n", "questions": "- Analysis of using different numbers of anchor points.\n- More results (both quantitative and qualitative) to justify the repulsive UDF representation.\n- Higher image resolution for testing.\n- Video results for the reconstruction.\n- Design of aggregation weight estimation.\nMore details can be found in the Weaknesses part.\n", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper identifies and addresses two major limitations of the state-of-the-art MCC method. The first limitation is related to the Transformer decoder, which struggles to handle a large number of query points efficiently. The second limitation is associated with the inability of the 3D representation to recover high-fidelity details accurately.\n\nTo address these limitations, the authors propose a new method called NU-MCC that significantly improves the quality of 3D object reconstruction. The proposed method employs a set of anchor points to provide global coarse structure prediction and utilizes the Neighborhood decoder to accelerate inference speed while enabling the exploitation of finer-scale visual features for improved recovery of 3D textures. Additionally, the Repulsive Unsigned Distance Function (UDF) is utilized to replace the occupancy field for more accurate surface representation, outperforming the traditional UDF in this problem.\n\nThe effectiveness of the proposed NU-MCC method is demonstrated through experimentation on multiple datasets. Specifically, the method quantitatively outperforms MCC by 9.7% in terms of F1-score on the CO3D-v2 dataset. Moreover, it showcases its superior performance qualitatively in zero-shot settings such as iPhone RGB-D capture, AI-generated images, and ImageNet.\n\nOverall, this paper provides valuable discussion into the challenges associated with 3D object reconstruction and demonstrates the efficacy of the proposed NU-MCC method in overcoming these challenges.\n", "weaknesses": "Overall, I find the method to be promising, but there are several concerns that need to be addressed regarding the evaluation section.\n\nFirstly, I believe the analysis of the effects of different anchor point numbers is missing. Given that the use of anchor points is one of the key designs of the method, it is important to understand their effect on the performance of the model. Therefore, it would be beneficial to conduct experiments analyzing the effects of different anchor point numbers to improve the evaluation section.\n\nSecondly, while the repulsive UDF is another important design of the method, there are no quantitative results provided to ablate its influence. Although a single visual comparison is presented in Fig. 5, more experimental results are needed to justify this design and its impact on the overall performance of the model.\n\nThirdly, there seems to be some inconsistency with regards to the resolution used for training and testing. While the supplementary material states that the model trained with 128128 image resolution can be directly applied to 256256 image inputs, the main paper suggests that the model is trained with a 256256 resolution. It would be better to clarify these discrepancies and consider using a higher resolution such as 512512 for testing.\n\nFurthermore, no video results are shown to check the 3D reconstruction results, which could provide a more comprehensive evaluation of the proposed method.\n\nLastly, there is a need to explain the design philosophy behind the aggregation weight estimation in Equation 2. It is unclear why the sum operation is used when the input to the network is the sum of three features. Therefore, providing more insights into the selection of weights and the reasoning behind it would enhance the clarity of the method.\n\nIn conclusion, while the method shows promise, the issues outlined above need to be addressed to improve the evaluation section and to strengthen the overall contribution of the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the value and promise of our method. We address your concerns as follows.\n\n**Analysis of the effect of different anchor point numbers:** To study this effect, we retrain our model with different numbers of anchor points. Given the constrained timeframe of the rebuttal phase, we conduct training on the “plant” object class. Results on plant validation set are as follows:\n\n| **n anchors**     | **Color (&#8595;)** | **Prec (&#8593;)** | **Recall (&#8593;)** | **F1 (&#8593;)** | **Acc (&#8595;)** | **Comp (&#8595;)** | **CD (&#8595;)** |\n|-------------------|-----------|----------|------------|--------|---------|----------|--------|\n| **10**            |     0.348 |     81.6 |       75.4 |   78.1 |   0.110 |    0.152 |  0.262 |\n| **50**            |     0.333 |     79.8 |       77.8 |   78.5 |   0.114 |    0.144 |  0.258 |\n| **100**           |     0.330 |     82.7 |       76.6 |   79.2 |   0.105 |    0.148 |  0.253 |\n| **200 (default)** |     0.330 |     81.4 |       77.7 |   79.2 |   0.108 |    0.142 |  0.250 |\n\nIt is interesting to see that as few as 10 anchor points are able to well represent the overall structure of the objects. While a higher number of anchor points typically improves the reconstruction quality, the gain becomes smaller with more anchors.\n\n**Quantitative results of Repulsive UDF:** The quantitative comparison between the Repulsive UDF and the standard UDF is provided in the last two rows of Table 1 of our paper. More qualitative comparisons are provided in Figure A of the PDF.\n\n**Input resolution:** To clarify, there are two types of input resolutions in our pipeline. The first one is the input resolutions to be passed to the encoder ViTs. These resolutions are kept fixed during training and inference (224 x 224 for image and 112 x 112 for depth). \n\nThe second type is the resolution for the mapping of fine features (seen points) in the feature aggregation process. We use a resolution of 112 x 112 during training and all of our evaluations (e.g., Table 1). Since this mapping process is not restricted to any network structure, we can map into arbitrary resolution during inference. As shown in the supplement, we can leverage this flexibility to map higher resolution (e.g., 224 x 224) seen points to recover finer details. However, using finer resolution incurs additional computational cost since every query point would then need to search more seen points to find the closest distances. Employing very high resolution (e.g., 448 x 448) therefore leads to computationally heavy inference, requiring the query points to search from 16x more points compared to the default resolution of 112 x 112.\n\n**Video results:** We provide video results at the link.\n\n**Design philosophy behind the aggregation weights:** As introduced on Line 158-161, we predict the aggregation weights based on three types of the information of the query point: 1) The displacement between $q$ and the anchors; 2) The anchor features; 3) The global token providing a global context. To fuse these information, we use the widely-used sum operation for its simplicity, and linear projection layers are used to adjust the features for better fusion. While there are other alternative operations, such as concatenation or multiplication, the specific choice is not a focus of this work, which we leave for future explorations.\n"}, {"review_id": "foN9G29Yi2", "reviewer": "Reviewer_iFNG", "summary": "This paper approaches the task of 3D reconstruction given a single RGBD image. It builds off of recent SOTA approach MCC, adding four main contributions: (1) applying attention on only anchor points instead of each query point and using weighted interpolation, (2) using linear projection for “fine” features at each input point (3) replacing occupancy representation with UDF, (4) adding a “repulsive” component to UDF. Experiments on CO3D-v2 validation set show quantitatively (1) provides significant speedup, (2) improves fidelity, (3+4) improve fidelity further. Qualitative and quantitative results show favorable performance vs. MCC.", "questions": "To further strengthen the contributions of this paper, a more thorough ablation study upon contributions could really help. For instance:\n\n- Does increasing the resolution of fine features help, as claimed in L187?\n- How sensitive are results to the number of anchors or anchor radius? \n- Do details of repulsive UDF sampling matter? \n\nOther questions:\n- I don’t follow the section on bootstrapping the anchor predictor (L131). How is global token z0 used exactly? Where does z0 come from?\n- Are the same weights used for both coarse and fine, given they are concatenated and in practice the same number of coarse and fine weights are used at inference?\n- Figure 3 could use a key, as currently one has to look at the text to understand int\n- Figures 2-3 have text that is very small, making it a bit hard to read\n- L147: “depends” should be “depend”", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Method’s contributions are strong:\n\n- The quadratic nature of attention means carefully constructing queries can yield efficiency. This combined with the insight of extracting fine detail through linear projection is a nice combination.\n- Repulsive UDF is intuitive and positive contribution that is clearly better than occupancy. In the well-studied field of reconstruction, this is a helpful insight.\n- In the quantitative result that is reported, results are compelling. Ablations show (1) anchors provide significant speedup, (2) fine features improve fidelity, (3+4) Rep. UDF improves fidelity further. A qualitative example is convincing the repulsive UDF is meaningfully better than UDF.\n\nWriting is generally excellent. \n\n- For the most part, one could reproduce most of the method and experiments from reading the paper.\n- Technical details are well-motivated through a thorough comparison with MCC. The reader feels like an expert in these methods after reading the paper.\n", "weaknesses": "Edit: after reading the rebuttal and other reviewer comments, the lack of comparisons weakness is remedied. \n\nLack of quantitative comparisons and metrics; existing evaluation is inconsistent with prior work.\n\n- The key contributions (1-4 from summary) are validated in only one quantitative result (and one quantitative result in Supplemental). Results on CO3D are not broken down into seen and unseen categories, which would be helpful.\n\n- Why not include more qualitative results in Supplemental? Two examples on e.g. ImageNet are not convincing, especially when no quantitative comparison is reported.\n\n- The existing quantitative results do not follow the evaluation protocol of prior work (MCC) -- the paper appears not to report the accuracy, completeness, absolute error or MSE metrics of MCC. The numbers reported for MCC following F1-score metric do not match the prior work – presumably this is because numbers are reported on the validation split, while MCC uses the test split? This is a strange choice. Why are F1 metrics on Hypersim are significantly different from MCC? \n\n- No comparison or at the very least reference to PointE is surprising. This relates to why I gave a 3 instead of 4 for presentation: it is strange the paper builds off MCC (unpublished, arXiv Jan 23), while ignoring entirely Point-E (unpublished, arXiv Dec 22). While not comparing to unpublished work is acceptable, it is strange to not use this work which other works (e.g. 3DFuse, Zero 123) have found outperforms MCC.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our proposed method compelling. We address your concerns below.\n\n**F1-score metrics do not match the prior work:** As explained in the footnote in our paper (page 8), there is a bug in MCC codebase F1 calculation. The bug can be traced on their GitHub (as of the writing of this rebuttal) → MCC/engine_mcc.py → Line 44. In the recall calculation, they loop using the number of predicted points (predicted_xyz), but then divide the summation using the number of ground truth points (gt_xyz) in Line 49. This results in a lower recall than it should be when the number of predicted points is smaller than the number of ground truth points. We have rectified this issue in our evaluation by changing predicted_xyz to gt_xyz in Line 44. \nNote that fixing the bug is in favor of the baseline method. The F1-score comparison between MCC and NU-MCC is 56.7 vs 83.8 before fixing the bug, and is 76.4 vs 83.8 after fixing the bug. This is because our method always generates dense points greater than the number of ground truths, whereas MCC with a high occupancy threshold generates fewer points. Thus, the bug penalizes MCC but not our method. \n\n**F1-score difference for Hypersim:** We believe the bug is a major reason that causes the difference. On top of it, MCC performs evaluation using the scene meshes that are not freely available and need to be purchased, while we directly use the ground truth provided by Hypersim.\n\n**CO3D seen/unseen categories:** As suggested, we show the results on CO3D following MCC’s novel categories setup (unseen categories). Due to the constrained timeframe of the rebuttal phase, we train both models for 20 epochs. As shown in the table below, the proposed method performs favorably against MCC on unseen categories.\n\n| **Model** |  **Color (&#8595;)** | **Prec (&#8593;)** | **Recall (&#8593;)** | **F1 (&#8593;)** | **Acc (&#8595;)** | **Comp (&#8595;)** | **CD (&#8595;)** |\n|-----------|-----------|----------|------------|--------|---------|----------|--------|\n| **MCC**   |     0.363 |     56.8 |       89.3 |   68.9 |   0.172 |    0.144 |  0.316 |\n| **Ours**  |     0.307 |     79.2 |       84.0 |   80.9 |   0.121 |    0.146 |  0.266 |\n\n**More metrics:** As suggested, we provide more metrics, including precision, recall, accuracy,  and completeness in Table A in the PDF. \n\n**More qualitative results:** We provide more qualitative results on the video at the link.\n\n**Comparison to Point-E:** As Point-E and our model uses different training data, it is difficult to make a fair comparison on the CO3D dataset on which our model is trained. Moreover, as Point-E is based on a fundamentally different concept (diffusion model) from our work, it is non-trivial to retrain Point-E on CO3D. Thus, we directly compare the two methods in an in-the-wild setting using images captured by iPhone. As shown in Figure C of the PDF, our model can generate denser point clouds with higher fidelity to the input than Point-E. \n\nIn addition to the improved reconstruction quality, our method is also superior in terms of runtime. Note the methods, such as Point-E, 3DFuse, Zero123, involve iterative generation process and/or per-sample optimization, resulting in a much slower running speed. For example, it takes 80 seconds for Point-E to generate 4096 points, 30 minutes for 3DFuse generation, and 15 minutes for Zero123. In contrast, our model is a generalizable one-step feed-forward network, resulting in fast inference that can generate ~200k points in 3.5 seconds or faster.\n\n**Resolution of fine features:** As shown in Sec. 2.1 of the supplement, increasing the resolution of fine features typically facilitates reconstruction of more intricate details. Meanwhile, this will increase the computation cost (see response to Reviewer WpNt - Input resolution).\n\n**Number and radius of anchors:** For the effect of the number of anchors, see the response to Reviewer WpNt - Analysis of the effect of different anchor point numbers. For the radius, increasing the anchor radius (including more points in aggregation) gives a smoothing effect as shown in Sec. 2.2 of the supplement.\n\n**Details of Repulsive UDF sampling:** On Line231-232, we select k-nearest points of a query point to construct the repulsive potential field. While we use a default k=16, the Repulsive UDF is robust to different choices with stable performance on k=4, 8, 32, 64.\n\n**Global token z0:** As introduced on Line108, the global token $z_0$ is obtained from the encoder ViTs. It is the global token of the encoded features $\\mathbf{R}$.\n\n**Are the same weights used for coarse and fine features?** Yes\n\n**Figures and typo:** We will rectify accordingly\n"}, {"review_id": "tKOjGPGWWa", "reviewer": "Reviewer_AYkM", "summary": "The paper studies the problem of shape reconstruction from single-view RGBD images and addresses two key limitations in a major prior work, MCC: 1) inefficient decoder of quadratic time complexities, and 2) suboptimal implicit representation for pointclouds. \n\nTo improve computational efficiency, the authors propose an anchor-based neighborhood aggregation scheme to downsample the seen points. They also propose a way to sample features from the high-resolution seen points directly that improves the modeling of the fine surface details. \n\nOn the other hand, instead of using occupancy representation, UDF is adapted in this paper for higher sample efficiency and more accurate surface modeling. To further resolve the problem of point concentration, the authors add a concentration penalty in the energy function.\n\nThe authors show evaluation of their model on CO3D and HyperSim (supplement) and demonstrate state-of-the-art performance over MCC. An ablation of different designs is also included.\n", "questions": "Please see the weakness for my concerns and questions. My final rating will be highly dependent on authors' response to these questions.", "limitations": "The limitations of the work have been adequately discussed.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The limitation of the prior work (MCC) is well analyzed in this paper.\n2. The design of the repulsive UDF makes sense and looks novel.\n3. The paper is relatively easy to follow.", "weaknesses": "1. Missing quantitative comparison to PointTr. Based on my understanding, PointTr leverages a similar approach of anchor+neighbor aggregation. Given the results in MCC (Table 1(f)), PointTr performs significantly worse than MCC's decoder, while the finding in this paper shows similar idea to PointTr actually leads to better performance. A detailed discussion on this seemingly contradictory finding, as well as an apple-to-apple comparison to a baseline with PointTr's feature aggregation method would be very helpful.\n\n2. It is not clear where the major speedup of the inference actually comes from. In L116, it is stated that MCC's decoder is quadratic in the number of query points. However, if my understanding is correct, this is more of an implementation/engineering issue. In fact, query points do not communicate with each other in both works, so the effective time complexity should be linear on the number of query points if implemented well. My worry here is that if the authors implement MCC in some naive quadratic way while implement their method in the effective linear way, the main speedup can actually come from this implementation difference rather than the claimed novelty. It is very important to clarify on this. \n\n3. It would have been very helpful to break down CD into accuracy and completeness. The RepUDF representation makes sense in terms of completeness improvement. But I think it is very important to see whether it helps accuracy as well. \n\n4. The usage of nearest points to improve fine details makes sense for seen surface, but does not make much sense for unseen surface. Is it possible to evaluate the model's performance on seen and unseen surface separately? If it only improves RGB/surface reconstruction quality on seen surface, I wonder whether a simple baseline that directly overwrite the seen part of the reconstruction with seen points would actually perform similarly to the proposed method. For example, for every queried point in the reconstruction, if there are any seen points that are close enough to this queried point, we can replace the queried point with the seen point.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison to PoinTr:** There are fundamental differences between PoinTr and our NU-MCC. First, PoinTr is based on an explicit 3D representation and can only predict a fixed number of points. In contrast, our method allows an arbitrary number of output points, providing better flexibility and higher-resolution reconstruction. Besides, PoinTr employs FPS and Graph network to extract features while we use ViTs similar to MCC.\n\nSecond, to generate the 3D output, we use neighborhood feature aggregation to predict repulsive UDF, while PoinTr uses FoldingNet to generate pointclouds. Note that the feature aggregation operation in PoinTr is only used to predict the center points instead of the final output pointclouds. In contrast, our anchor prediction process uses a simple Transformer structure, which does not involve any neighborhood feature aggregation.  Moreover, the aggregation strategies are also significantly different (adaptive weighted sum as in Eq. 1-2 of our paper vs max-pooling of PoinTr).\n\nThird, PointTr only predicts the missing parts of the point cloud and concatenates them with the input point cloud to obtain the complete output. Consequently, it cannot deal with noisy input point clouds that are commonly encountered in real applications. In contrast, our method predicts the complete 3D representation from a noisy partial point cloud, which is more suitable for many real-world scenarios.\nIn addition, our method takes RGB-D input, while PoinTr can only handle a partial point cloud without color information. Thus, PoinTr lacks texture recovery capacity.\n\n**Speedup:** We use the official implementation of MCC instead of re-implementing it ourselves. In MCC (and also our method), the number of query points ($N_q$) is quite large, often in the order of 100k or millions. Processing them all at once in MCC’s 8-layer Transformer decoder incurs a computational complexity of O(($N_e + N_q)^2$), which is computationally prohibitive ($N_e$ is the number of encoded features). Note that in MCC, each query point needs to attend to both the input features and the query itself, making a more efficient implementation non-trivial. To circumvent this issue, MCC proposes to run the query points in batches. Suppose the batch size is $B$, MCC needs to run a for-loop with a length of $N_q/B$, processing $B$ query points in each iteration. This incurs a computational complexity of O($(N_q/B) \\cdot (Ne+B)^2$) that is linearized w.r.t. the total number of query points to be processed ($N_q$). \n\nHowever, there is a dilemma here. A small batch size $B$ increases the first term and necessitates running a longer for-loop, while a larger $B$ leads to a quadratically growing computational complexity in the second term. We have used a number $B$ that leads to the most optimal runtime for MCC, as reported in Table 1 in our paper. It is worth mentioning that we also considered another strategy: parallelizing the $N_q/B$ runs instead of a for-loop. However, this operation is also inefficient as the encoded features ($N_e$) need to be copied $N_q/B$ times, leading to burdensome memory management overhead.\n\nIn contrast, our method only has a small fixed computational complexity of O($(N_e + M)^2$) in the 8-layer Transformer decoder ($M$ corresponds to the number of predicted anchors and is a small number). The burden to process $N_q$ query points is then shifted into the feature aggregation process that is far more efficient than 8-layer Transformer and incurs a linear computational complexity of O($N_q \\cdot k$), where $k$ is the number of nearest neighbors.\n\n**Breakdown into accuracy and completeness:** Please find the breakdown in Table A of the PDF. Repulsive UDF leads to marginally lower precision but significantly better recall, culminating in an overall improvement in the F1 score.\n\n**Baseline by replacing to seen points directly:** As the seen point clouds are quite noisy, directly replacing the queried point with the seen point leads to noisy reconstructions. For a better analysis, we compare with a baseline using model trained without incorporating any fine features and replacing the color of a point to the nearest seen point if it is within a small distance. As shown in Table B of the PDF, this baseline does not perform as well as the proposed method where fine features are more effectively incorporated by neighborhood aggregation. Visual examples are shown in Figure B of the PDF."}]

Author rebuttal:
First and foremost, we thank all the reviewers for the time and effort in reviewing our paper. We believe all of the reviewers’ comments and suggestions provide useful feedback to improve the quality of our paper. We hope to be able to clarify some concerns and doubts in this rebuttal and discussion period. To address some of the reviewers’ comments, we include a PDF and video (link is shared to Area Chair).


Meta-review: In this paper, the authors propose NU-MCC for single image 3D reconstruction. The paper improves MCC in two ways: 1) a "neighborhood decoder" for more efficient computation, and 2) a repulsive UDF for improving reconstruction quality.

After rebuttal, all reviewers find this paper making solid contribution to our field and recommend acceptance. The AC agrees with the decision.
Decision: Accept (poster)