Adaptive 3D Reconstruction via Diffusion Priors and Forward Curvature-Matching Likelihood Updates
Authors: Seunghyeok Shin
Dabin Kim
Hongki Lim
Venue: NeurIPS 2025 spotlight ()
Keywords: 3D reconstruction
point cloud generation
diffusion model
Primary area: applications

Abstract: Reconstructing high-quality point clouds from images remains challenging in computer vision. Existing generative models, particularly diffusion models, based approaches that directly learn the posterior may suffer from inflexibility—they require conditioning signals during training, support only a fixed number of input views, and need complete retraining for different measurements. Recent diffusion-based methods have attempted to address this by combining prior models with likelihood updates, but they rely on heuristic fixed step sizes for the likelihood update that lead to slow convergence and suboptimal reconstruction quality. We advance this line of approach by integrating our novel Forward Curvature-Matching (FCM) update method with diffusion sampling. Our method dynamically determines optimal step sizes using only forward automatic differentiation and finite-difference curvature estimates, enabling precise optimization of the likelihood update. This formulation enables high-fidelity reconstruction from both single-view and multi-view inputs, and supports various input modalities through simple operator substitution—all without retraining. Experiments on ShapeNet and CO3D datasets demonstrate that our method achieves superior reconstruction quality at matched or lower NFEs, yielding higher F-score and lower CD and EMD, validating its efficiency and adaptability for practical applications. Code is available at https://github.com/Seunghyeok0715/FCM.

Adaptive 3D Reconstruction via Diffusion Priors and Forward Curvature-Matching Likelihood Updates
Seunghyeok Shin
Dabin Kim Hongki Lim"
Department of Electrical and Computer Engineering, Inha University (ssh8642, 1124db) @inha. edu, hklim8 inna accaaacc.. .r.
Abstract
Reconstructing high-quality point clouds from images remains challenging in com- puter vision. Existing generative models, particularly diffusion models, based ap- proaches that directly learn the posterior may suffer from inflexibility- they require conditioning signals during training, support only a fixed number of input views, and need complete retraining for different measurements. Recent diffusion-basee methods have attempted to address this by combining prior models with likelihood updates, but they rely on heuristic fixed step sizes for the likelihood update that lead slow convergence and suboptimal reconstruction quality. We advance this line of approach by integrating our novel Forward Curvature-Matehhe method with diffusion sampling. Our method dynamically determines optimal step sizes using only forward automatic differentiation and difite-difference curvature estimates, enabling precise optimization of the likelihood update. This formulation enables high-fide figh-fideiity reconstruction from both single-view and multi view inputs, and supports various input modalities through simple operator substitution all without retraining. Experiments ShapeNet and CO3D datasets demonstrate that our method achieves superior reconstruction quality at matched or lower NFEs, yielding higher F-score and lower CD and EMD, validating its efficiency and adaptability for practical applications. Code available at here.
(FCM) update
Introduction
Three-dimensional reconstruction has become increasingly important across diverse applications including robotics, autonomous driving, augmented reality, and virtual environments. Among various 3D representations, point clouds serve fundamental data structure for representing objects and scenes due to their simplicity and flexibility. However, generating high-quality point clouds that accurately capture intricate details remains challenging, particularly when working with limited input information such as single-view images.
Recent advances in deep generative models, particularly diffusion models, have shown remarkable success generating high-fidelity images [4] and 3D data. Diffusion models use an iterative denoising process to progressively transform random noise into structured outputs, making them ef- fective for capturing complex geometric patterns. the domain of point cloud generation, researchers
have begun exploring diffusion- approaches with promising results [19 39
38
24
34).
While diffusion models offer powerful generative capabilities, applying them to 3D reconstruction presents unique challenges due to its nature as an inverse problem. In typical inverse problems (for- mulated as y Ax), iterative optimization methods solving least-squares objectives can determine optimal step sizes analytically using gradients that incorporate A (the adjoint of A). However, 3D object rendering represents a particularly challenging case where the rendering operator complex and non-lineat making the computation of the adjoint operation intractable. Since classical step-size
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

7 256
7=22 221
160
T= T
T=0
Ground Truth
(NFF-256)
PC
HOM (NFE-276)
Ours (NFE-250)
Sindl
Xae
Output
0.310
Figure 1: Left: Visualization of our diffusion process from random noise (T 256) to final recon- structions (T' 0) for various object categories, Right: Comparison of point cloud reconstructions between Ground Truth, previous methods (PC² [20], BDM (34) ) and our approach. Our method achieves higher fidelity reconstructions with better F-scores (0.382) than existing approaches while using fewer function evaluations, particularly excelling at preserving fine structural details.
formulas require adjoint of the forward operator, the absence of tractable renderer adjoint complicates step-size selection. This fundamental challenge impacts how researchers approach diffusion-based point cloud generation, especially when incorporating image-based guidance.
Current image-to-point-clout methods predominantly learn the score of the posterior distribution V log p(x (((X) yy) directly, where X represents the point cloud and represents image measurements. This direct approach incurs significant limitations: necessitates including images as conditioning signals during training [20), restricts models to fixed number of input views without specialized encoders [8) and requires computationally expensive retraining whenever measurement types change (e.g., from RGB images depth maps).
promising alternative approach p(X) and an updatable likelihood p(y[X). X). employing Diffusion Posterior Sampling (DPS) [6] with gradient updates via V 'logp(y X) for Gaussian splatting- 3D reconstruction. While this decomposition is conceptually straightforward and modular, current implementations struggle with a critical limitation: determining appropriate step sizes for the likelihood updates. The non-linear nature of 3D rendering prevents analytical step size determination, leading cxisting methods to rely on heuristic, fixed step sizes (22). This results in slow convergence, suboptimal reconstruction quality, and often necessitates additional 2D diffusion models for refinement, further complicating the pipeline.
decomposes the posterior p(Xly) into trainable prior
To address these limitations, present novel approach that combines a point cloud diffusion model with Forward Curvature-Mateon Matching (FCM) optimization. Our approach, illustrated in Fig. [2] computes an adaptive step size using Barzilai-Borwein rule and refines it with an Armijo backtracking condition, enabling more precise control. Our key insight is that by incorporating FCM's principled, curvature informed step size determination into the diffusion sampling process without any adjoint operations, we can effectively navigate the complex optimization landscape of 3D reconstruction.
Unlike previous DPS-based methods that rely on heuristic step sizes for the likelihood update, our approach employs FCM optimization dynamically determine optimal step sizes. The key step-size
innovation is our reliance solely on the differentiable forward pass for survature-infommt determination, obviating the adjoint. This enhancement enables significantly more efficient and accurate optimization during the diffusion sampling process. The technical contributions of our work include:

---

We integrate the FCM method with the reverse process of diffusion models, enabling high-fidelity point cloud reconstruction that accurately matches imput images.
Our gradient-based updates point cloud reconstruction from either single-view or multi-view images without modifying the base model.
constrained by the number of input images, allowing for
Our method can be applied to various mcasurement modalities (such as RGB image to 3D object or depth map to 3D object) by simply substituting the appropriate operator rather than retraining the entire model, significantly enhancing flexibility and efficiency.
We demonstrate the effectiveness of our approach by reconstructing colored point clouds from both synthetic and real-world datasets. Our method achieves more accurate reconstruction with fewer neural function evaluations (NFEs) compared to existing techniques, validating the efficiency of our FCM-based likelihood optimization. We further demonstrate the adaptability of our approach by applying it to both multi-view reconstruction and depth map to point cloud generation without retraining, highlighting its potential for diverse applications. The remainder of this paper is organized as follows: Section 2 reviews related work, Section presents the proposed method, Section details our experimental results, and Section concludes with discussion of future directions.
2 Related Work
3D Reconstruction from Images. As interest in 3D content creation continues grow, research on reconstructing 3D shapes from 2D observations has advanced significantly. This challenging task requires inferring complete 3D structures, including both visible and occluded regions, from limited viewpoints. The difficulty is compounded by the scarcity of large-scale 3D datasets.
Various 3D representations have been explored for reconstruction, each with distinct advantages: offer compact representation but struggle with topological complex- ity; voxel-based approaches 15] provide a regular structure but face resolution limitations; point cloud methods [20 34] [6] offer flexibility with additional rendering requirements; implicit functions
mesh-based methods [3
32
23 10 12 enable high-quality rendering but computationally intensive; and Gaussian splatting
techniques 31 30 (22 balance quality and efficiency.
Point cloud generative models have evolved from early GAN-based 1 9 28 and VAF-based (36
approaches to more recent diffusion-based methods. Diffusion models offer several advantages: stable training dynamics, high-quality generation capabilities, flexibility in conditioning, and a strong probabilistic foundation. The seminal work by Luo et al. [9] introduced diffusion models for point cloud generation, with subsequent research extending these methods for various applications [38
Building on these advancements in 3D generative modeling, recent diffusion-based approaches have significantly advanced mage-to-poing cloud reconstruction. PC2 (20] performs single-view reconstruction by denoising a point cloud with projection conditioning, which ensures geometric consistency between the reconstruction and input view. However, it directly Iearns the posterior distribution, requiring images during training and limiting adaptability to varying input conditions. Bayesian Diffusion Models (BDM) 34 offer a complementary perspective by factorizing the 3D) reconstruction task into a learned score of the prior V log p(X) trained solely on 3D shapes and a learned score of the posterior V log p( (Xly trained with paired image-shape data. During inference, the prior and posterior models exchange intermediate outputs over multiple denoising steps. While this fuusion-wit--diffusion paradigm is effective, BDM relies on a l trained posterior score function that requires images during training, thus limiting its adaptability to varying input modalities.
proposes framework for solving inverse problems using diffusion models without retraining for each new measurement type. This approach decomposes the posterior p(Xly) into pre-trained prior p(X) and an adaptable likelihood term p(y X). During sampling, the intermediate predictions are adjusted using gradient updates from the likelihood term.
Diffusion Posterior Sampling. Diffusson Posterior Sampling (DPS) 66)
which uses DPS with Gaussian Splatting for view-guided 3D generation. However, these methods rely on heuristic, manually- step sizes for the likelihood update, which often requires careful calibration for each task and can lead to suboptimal convergence reconstruction quality.
Recent works applying DPS to 3D reconstruction include GSD [22],
3

---

va(Kip)
Point
Reference Image {()dlenderingg Openator
Karl
laala
g'x v.I -d9e)
18
DDIM
X
0g 8x
sampling
Point Cloud
Uxftaston Model
(X.)
FCM
Ban
418gy
Sampling
PCM
Phase
Likelthood Update
Figure 2: Overview of our FCM-guided point cloud diffusion framework. The sampling phase (left) shows how the diffusion model progressively transforms random noise X, into structured point clouds through DDIM sampling. The FCM likelihood update (right) illustrates our key innovation dynamically determining optimal step sizes for the likelihood gradient V R(Xo)+)]l2 This principled optimization approach enables high-fidelity reconstruction that accurately matches input images while requiring fewer function evaluations than existing methods.
Adaptive Step Size Methods in Optimization. Our FCM approach has roots in several founda- tional optimization techniques while introducing novel algorithmic elements. In numerical optimiza- tion, determining appropriate step sizes is a well-studied challenge with various classical solutions. Quasi-Newton methods
(25) approximate the Hessian using rank-one or rank-two updates (e.g., BFGS, L-BFGS (18), but require matrix storage and operations. Barzilai-Borwein (BB) methods [4] provide scalar approximations to the secant equation using the differences of consecutive iterates and gradients. Line search techniques with Armijo [2] Wolfe conditions 33 ensure sufficient descent but typically involve multiple function evaluations.
Building on these foundations, FCM introduces several innovations specifically for diffusion- based 3D reconstruction: (1) scale-adaptive curvature probe (6, 6, [|ge|| that automatically calibrates to the geometry of point clouds and gradient magnitudes, (2) a forward-difference directional curvature estimate that requires no adjoint operations of the renderer- ceittccai for complex neural renderers where adjoint computation is intractable, (3) robust BB-inspired step-size computation combined
with principled capping that offers theoretical guarantees, and (4) "once-only" Armijo check.
3 Method
Our goal is to perform high-quality, flexible 3D reconstruction by decomposing the posterior distri- bution p(X y) into learned prior (X) and likelihood update p(y X) that does not require separate training. We train only the score of the prior V log logpo(XX on unlabeled 3D data. Then, inference, incorporate the measurement information (e.g., single-view multi-vie view images, depth maps) through an adaptive Forward SewaatureMMatchigg (FCM) update, which approximates V Vlogp(S X).
Any forward operator R. (e.g., a differentiable renderer for images or a map from 3D to depth measurements) can be plugged in to guide the generation of point clouds via the same trained diffusion prior. This design separates the learned model from the measurement modality, eliminating the need for retraining whenever the measurement operator changes. In this section, we detail our method four parts. First, we describe how we train the diffusion model log po (X). Next, we present differentiable renderer R for the image-based scenario. We then introduce the FCM-based likelihood update, highlighting why FCM is needed in non-linear settings and how step sizes are optimally determined through principled approach. Finally, we extend the method to the multi-view setting.

---

3.1 Diffusion Prior for Point Clouds
We begin by training a diffusion model pe (x( on large dataset of colored point clouds. Following the standard DDPM [[]] framework, we define forward diffusion process that corrupts a clean point cloud X into XT with Gaussian noise over T timesteps. The reverse process is modeled by a neural network that estimates the noise at each timestep. Formally, in the forward process:
((; -1) N (X,; v 3e X,-1, I);
where Be is variance schedule. This process can be written in closed form from Xo:
(X, Xo) .V (X ai Xo, (1 - (--a)) I):
(2)
with 0 = IT (1 3.). The reverse process approximates po(X,-1 X,) via learned Gaussian:
(X X,) (X no (X:: (), En (X);):))
During training, we minimize the simplified loss:
L = Ex, - E0 (x.
(4)
where IO,
Once trained, we use the DDIM sampler 29 for inference, generating point clouds from noise in (t) fewer steps. Let cg (X+) be the noise estimate at step t. Then the DDIM update from X₂ to X is:
X,-1 vor- Xole + V1 ā oe(n) 600 (X.) oz(1)) E
(5)
where
Xox= X, vI a (t) (X,)
0z
and
(1)
(1- (1-8()
V
oc 1
is a variance term controlling the sampling stochasticity. This DDIM sampler, combined with our trained model, provides 3D prior that can generate plausible point clouds.
3.2 Differentiable Renderer as the Measurement Operator
l|2 can be computed. In this section we introduce forward operator RR that projects point cloud X into 2D measurements.
Our method only requires that R differentiable, so both R(X) and its gradient Vx lly R(X))
A point cloud X comprises points { Zi, [.). where (æ;, 2;) are 3D coordinates and f; includes attributes such as color. Each point is projected onto the 2D image plane using known camera parameters. At each pixel v). R collects the K points with the smallest depth values 22 (i.e., the nearest points along the viewing direction) and blends their colors via alpha compositing:
K i-1
Reolor (X) u) = Σ
a.
(()-)))) f.
(6)
i=1
j=1
Here, the opacity a, is computed from the image space footprint
a,
72
where is the radius of the rasterizer and p. is the Euclidean distance between the center of the pixel
and the projected position of the point in the image space. The product term [[;=] (1 a,) ensures that closer points dominate the final color, while partially occluded points contribute less. Repeating this calculation for each pixel (u, v) yields a 2D image matching the resolution of y.
In addition to bosor-based rendering, R can produce depth map by applying inverse-square weighting each point's distance. At each pixel (u, v), the depth is computed from the same set K nearest points:
K
Rdcpth (X) 2)
(8)
Σ
1= 7
5

---

Reforence
Imsge
Ground 1ruth
PC
BDN
OuES
Graud Truth (hovel Vies)
PC (lood vies)
EDM (Noved
Ours (Novet Veew
Figure 3: Qualitative comparison of single-view 3D reconstructions on the ShapeNet dataset. The figure displays point cloud reconstructions from our method, PC?, and BDM for various object categories, highlighting the superior detail and accuracy of our approach.
EMD( x 10)
CDX x10)
F-gcore
Category
PC 20
BDM [4]
Ours
PC 20
BDM X4
Ours
PC 20
BDM [34]
Ours
sirplane
0.587
0.577
0.476
0.399
0.417
0.378
0.498
0,543
0.543
car
0.565
0.723
0.517
0.558
0.664
0.460
0.262
0.289
0.386
chair
0.701
0.643
0.662
0.636
0.613
0.679
0,241
0.271
0.282
table
0.735
0.647
0.691
0.703
0.656
0.727
0,240
0,268
0.319
Average
0.647
0.648
0,587
0.574
0.588
0.561
0.310
0.343
0.382
Table 1: Quantitative evaluation of single-view 3DD reconstruction on the ShapeNet dataset. NFEs were matched equally across our method, PC and reconstruction model of BDM (T = 256). For BDM, additional NFEs were incurred due the prior model (T 20).
so that points closer to the camera have larger influence on the final depth. Repeating this process for each pixel yields a 2D depth map matching the resolution of y.
Because we do not learn a dedicated score function for V log log X), different operators R can be swapped in with minimal effort. If is single-view image, then R. Reooor with single camera. For multi-view input, each view is rendered separately and their pixel or feature errors are averaged, as described in Section 3.4 If y a depth map, then Rdepth from Eq. (8).
3.3 Likelihood Update via Forward Survature-Matching
In standard diffusion posterior sampling (DPS) [6), one iteratively updates the current sample X, with a term proportional to the gradient Vx log p(y X). However, for complex, non-linear forward operators R. determining an appropriate step size is non-trivial. Previous approaches resort to heuristics [6] or empirically tuned factors [22] to balance the data fidelity term with the learned diffusion prior. While this can be effective, it may hamper convergence speed or degrade reconstruction quality if not carefully tuned.
To address these limitations, we propose Forward Curvature-Matching (FCM), a novel algorithm designed specifically for diffusion- based 3D reconstruction. The development of FCM was guided by key requirements: working without adjoint operations (intractable for neural renderers), maintaining predictable computational cost, and using universal parameters across different reconstruction tasks.
Our approach relies on key insight: we can estimate curvature information through a scaled directional probe without requiring full Hessian approximations [25]. For the measurement loss [(x) = ||y - R(x)]l2, given the current estimate x and gradient gx = VL(xx), we compute:
|x
5z 50
x = X2
8x. gk:
(9)
gA
gk gl
g VC(x())
h
(10)
6%

---

Image
Grurd Trush
PC
Duss
Image
Ground
PC
Ours
Rnfermsse View
View
Novel
Vicw
Figure 4: Comparison of rendered images from reconstructed point clouds on the CO3D dataset. The figure shows renderings from our method and PC?, illustrating the higher fidelity and better preservation of details in our reconstructions.
This h& approximates (²(((x)) ge along the gradient direction. The scale-adaptive probe (6&) automatically calibrates the geometry of the point cloud, erucial advantage traditional finiteddifference approaches (7].
We then compute Barzilai- Borrrlii-BBowein-inniieeddd 4 step size, modified for robustness:
ig:
0g min(a 1/L}
(1I)
where: 100 and is the Lipschitz constant of V.C The capping mechanism ensures stability while maintaining theoretical guarantees. Unlike classical line searches that require multiple function
evaluations 25). we incorporate single Armijo check (2: if L(x; agg+) > L(xx) fJсM ag.
lgg 12,
we halve ag once and accept.
This design yields fixed computational cost of exactly two backward and three forward passes per step- signifinnylc more efficient than traditional optimization methods like L-BFGS [8 or Wolfe with unpredictable evaluation counts.
line searches
3
3.3.1 Theoretical Guarantees
Our approach built on the following assumptions, which are typically satisfied in the context of 3D reconstruction:
Assumption 3.1 (Smoothness). L is L-smooth: |VL[(u) - VC(v)] < L u v/l.
Assumption 3.2 (Lower bound). Lint == infx L(x) > -
Assumption 3.3 (Local convexity). C is convex on the set of iterates (which is typically small or "benign" in practice).
This approach provides theoretical guarantees on convergence and optimality, as captured in the following theorem:
Theorem 3.4 (Guaranteed Loss Decrease). Let min 1 }. Our FCM algorithm ensures:
SL
((x++1) S [(xx) - c. [[xx))
(12)
When integrated into the DDIM sampling process, FCM preserves the contraction properties of diffusion models:

---

Dexred (hove VINS
lefreme
Oyrs
Goand
Ki
Figure 5: Reconstruction from depth maps. The figure showcases point cloud reconstructions generated from depth map inputs.
Figure 6: Qualitative results of multi-view re- construction. The figure displays point cloud reconstructions using varying numbers of input views, demonstrating the enhancement in recon- struction quality as more views are incorporated.
Proposition 3.5 (Contraction Preservation). Under Assumptions 3.1 3.3 and og 1/L,, the com- bined DDIM+FCM operator remains contraction in expectation, thus preserving the diffusion contraction property.
Our FCM method fixed constant уiсм 10-1 1 for all tasks, this principled approach leads faster convergence and higher-quality reconstructions compared to methods that rely on heuristic step sizes. Detailed proofs and additional theoretical analysis are provided in the Appendix.
3.4 Multi- View Reconstruction
The same FCM-based likelihood update extends naturally to milti-view reconstruction. Suppose we have N images {y:) J. with known camera parameters. We define
1 N Σ y. (x(
LMv (X)
(13)
where R, the differentiable renderer for the viewpoint. The gradient VxLMv(X) can be used in Algorithm (replacing the single-view line lyy - R(-)]l with the multi-view average). As the number of views grows, reconstruction quality improves, yet the diffusion prior remains the same, illustrating the aogasity-agnostic nature of our approach.
By training only the diffusion prior on unlabeled 3D shapes and introducing an FCM-based likelihood update with an arbitrary forward operator R, we achieve a flexible, adaptive 3D reconstruction pipeline. The FCM approach ensures stable and fast convergence even with non- con-linear rendering operators, outperforning fixed-step DPS approaches.
4 Experiments
We evaluate the reconstructed point clouds using three different metrics: Earth Mover's Distance (EMD), L-1 Chamfer Distance (CD), and F-score at threshold of 0.01. Details of the implementation are provided in the appendix.
ShapeNet. In our method, colors are essential during the rendering process. However, sampling colored point clouds from mesh-based objects is a challenging task. To address this, we train our model using the dataset provided by KeypointNet (37]. The color information in the KeypointNet point cloud does not correspond to the actual mesh color in ShapeNet. Instead, the model assigns colors according to object parts.
We perform our evaluation using the categories (airplane, car, chair, table) from the ShapeNet rendered image dataset 355

---

900
Views
EMD(x10)
CD(x10) 10)
F-score
Qun(NFE:256)
500
DDIM-DPSC
0.587
0.561
0.382
700
DDIM-DPS(--0055)
DDIM DPSK)-0.06)
0.436
0.386
0.512
Difference 1.2 norm
DDPM DFS0=0.09)
0.425
0.361
0.548
600
D.S()
300
DDPM--DSS00000)
Table 2: Impact of the number of input views on reconstruction performance. The table presents scores for reconstructions using 1, and 5 views, showing improved quality with additional views.
400
300
200
Method
EMD(x10)
CD(x10)
F-score
1.0
0.6
Timestop
0.2
0.0
DDPM DPS DDIM DPS
0.674
0.688
0.337
0.716
0.728
0.312
Figure 7: Convergence analysis during sampling. y step size of DPS-update. The plot shows the 1.2 norm difference between the reference image and the rendered image (ly-- - (()o) over diffusion timesteps for method and other sampling approaches. illustrating more stable convergence of our FCM- M-M-baaed method.
Ours
0.587
0.561
0.382
Table 3: Comparison with DPS-based pes-based methods. The table presents reconstruction metrics for our method versus DDPM+DPS and DDIM+DPS, demonstrating our approach's 's superior perfor- mance with fewer NFEs.
CO3D. The CO3D dataset is large-scale collection of real-world multi-view images from common object categories. provides colored point cloud obtained using COLMAP from multi-view images, which is then used for model training and evaluation. We perform our evaluation using the categories hydrant and teddybear from the CO3D dataset.
4.1 Quantitative Results
We evaluate the performance of reconstruction in the ShapeNet dataset. In Tab. our method is compared with PC² [20] and BDM (3]. the original paper. BDM is evaluated using 4,096 points. whereas PC is evaluated using 8, 192 points. In this work, we adopt the evaluation approach of PC for quantitative experiments, For BDM, we adopted the blending method that achieved the best results in their study and used PC² the reconstruction model. Our method achieves the best results in all metries. In this experiment, we ensured that the NFEs for all other models were similarly for a fair comparison. Detailed comparisons with the settings proposed by their studies and quantitative results on CO3D are provided in the appendix.
4.2 Qualitative Results
In Fig. we show the reconstructed point clouds of different models using the ShapeNet dataset. In Fig. we present comparison the rendered results of reconstructed colored point clouds using the CO3D dataset. Other models fail to accurately follow the given image in their rendering results for the reference view, instead focusing on generating a plausible object within the learned category. However, our method achieves the highest level of detail for the reference image.
4.3 Adaptivity Analysis
Our method has the advantage of performing various tasks without requiring retraining of the model. In this section, we demonstrate this capability through multi-view reconstruction and depth map reconstruction. The models used in this section are the same as those used in the previous section for the ShapeNet dataset. Fig. and Tab, ( reconstruction. As the number of views increases, the generated point cloud becomes more refined, demonstrating the improved quality of reconstruction. Fig. presennss the results of applying our method depth maps rendered using Eq. 8 The results show high fidelity to the reference depth map and the ability to generate natural-looking objects.
illluttreee the effectiveness of our method in multi-view
4.4 Ablation Study
To show the effectiveness of our method, compare with other DPS-based methods. Fig. and Tab. compare our method with DPS-based approaches. Fig. presents the plot of the difference

---

in L2 norm between the reference image and the rendered image during the sampling process over timesteps. We observed that both DDPM+DPS and DDIM+DPS methods achieve their best performance at the step size of 0.05. The reason DPS-based methods struggle to follow the reference image is that they update with a fixed step size, leading to suboptimal convergence. demonstrates that our method converges more optimally compared to other approaches. As shown in Tab. [ our method achieves the best point cloud reconstruction performance. Since the DDPM sampling process does not approximate Xo. and the iterative FCM updates from noisy X, using measurement are not ideal, we exclude the DDPM+FCM scheme from our comparison. Qualitative comparisons with uPS-based methods are provided in the appendix.
Conclusion
In this paper, we proposed the novel point cloud diffusion sampling approach for adaptive 3D reconstruction. Our method reconstructs the colored point cloud by updating it using likelihood V ply X) with given images through FCM during the reverse process of the point cloud diffusion model. In our experiments, we qualitatively demonstrate high-fidelity reconstruction of reference images with color, generating high-quality point cloud structures compared to prior works. Moreover, we quantitatively surpass previous works in point cloud reconstruction performance. Our method is applicable to various tasks, demonstrating its versatility, Additionally, it can extended to different domains (e.g., Gaussian Splatting, meshes, etc.), highlighting its adaptability. As future work, we are interested in exploring larger datasets across diverse domains.
Acknowledgement
This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) RRS-2025-24683103 in part by Korea Basic Science Institute (National research Facilities and Equipment Center) grant funded by the Ministry Science and ICT (No. RS-2024-0040188 in part by Institute of Information & communications Technology Planning & Evaluation (TTPP) under the Leading Generative AI Human Resources Development (ITTP-2025- 0S-2022--00302277 grant funded by the Korea government (MSIT), and in part by Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. N0.RS-20022-0115555 55915, Artificial Intelligence Convergence Innovation Human Resources Development (Inha University)).
References
[1] Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas J. Guibas. Learning representations and generative models for 3D point clouds. In International Conference on Machine Learning, 2017.
[2] Larry Armijo. Minimization of functions having Lipschitz continuous first partial derivatives. Pacific Journal of Mathematics, 16:1-3, 1966.
[3] Mohammad Arshad and William J. Beksi. LIST: single-view 3D reconstruction. In International Conference on Computer Vision, 2023.
Learning implicitly from spatial transformers for
[4] Jonathan Barzilai and Jonathan Michael Borwein. Two-point step size gradient methods. Ima Journal of Numerical Analysis, 8:141-148, 1988.
[5] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Haoran Su. Single-stage diffusion NeRF: unified approach 3D generation and reconstruction. In International Conference on Computer Vision, 2023.
[6] Hyungjin Chung, Jeongsol Kim, Michael McCann, Mare Louis Klasky, and J. C. Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference on Learning Representations. 2023.
[7] John E. Dennis and Bobby Schnabel. Numerical methods for unconstrained optimization and nonlinear equations, volume 16 of Classics in Applied Mathematics. SIAM, 1996.
[8] Yu Feng, Xing Shi, Mengli Cheng, and Yun Xiong. DiffPoint: Single and multi-view point cloud reconstruction with VIT based diffusion model. ArXiv, 112/2 2024.
10

---

[9] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. A Papier- Mache approach to learning 3D surface generation. In Conference on Computer Vision and Pattern Recognition, 2018.
[10] Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M. Susskind, Christian Theobalt, Lingjie I. Liu, and Ravi Ramamoorthi. NerfDiff: Single-image view synthesis with NeRF-guided distillation from 3D-aware diffusion, In International Conference on Machine Learning, 2023.
[11] Jonathan Ho. Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[12] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image 3D. In International Conference on Learning Representations, 2024.
[13] Tao Hu, Liwei Wang, Xiaogang Xu, Shu Liu, and Jiaya Jia. Self-supervised 3D mesh reconstruction from single images. In Conference on Computer Vision and Pattern Recognition, 2021.
[14] Tero Karras, Miika Aittala. Timo Aila, and Samuli Laine. Elucidating the design space of deffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.
[15] Vladimir Vladimirovich Kniaz, Vladimir Alexandrovich Knyaz, Fabio Remondino, A.N. Bordodymov, and Petr Moshkantsev. Image-to-voxel model translation for 3D scene reconstruction and segmentation. In European Conference on Computer Vision, 2020.
[16] Jac Joong Lee and Bedrich Benes. RGB2Point: 3D point cloud generation from single RGB images. In Winter Conference Applications of Computer Vision, 2025,
[17] Yukun Li and Li-Ping Liu. Enhancing diffusion-based point cloud generation with smoothness constraint. ArXiv, abs/2404, 0.039,0222662.0 2024.
[18] Dong C. Liu and Jorge Nocedal, On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45:503-528, 1989.
[19] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3D point cloud generation. In Conference on Computer Vision and Pattern Recognition, 2021.
[20] Luke Melas-Kyriare C. Rupprecht, and Andrea Vedaldi. PC²: Projection-condiiione point cloud diffusion for single-image 3D reconstruction. Conference on Computer Vision and Puttern Recognition, 2023.
[21] Shentong Mo. Enze Xie, Ruihang Chu, Lewei Yao, Lanqing Hong, Matthias Nießner, and Zhenguo DIT-3D: Exploring plain diffusion transformers for 3D shape generation. Advances in Neural Information Processing Systems, 2023.
[22] Yuxuan Mu, Xinxin Zuo, Chuan Guo, Yilin Wang, Juwei Lu, Xiaofeng Wu, Songcen Xu, Peng Dai, Youliang Yan, and Cheng. GSD: View-guided Gaussian splatting diffusion for 3D reconstruction. In European Conference on Computer Vision, 2024.
[23] Norman Muller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buld, Peter Kontschieder, and Matthias Nießner. DifTRF: Rendering- guided 3D radiance field diffusion. In Conference on Computer Vision and Pattern Recognition, 2023.
[24] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-E: A system for generating 3D point clouds from complex prompts. ArXiv, abs/2212.08751, 2022.
[25] Jorge Nocedal and Stephen J. Wright. Numerical optimization. Springer Series in Operations Research and Financial Engineering. 2006.
[26] Nikhila Ravi, Jeremy Reizenstein, David Novotný, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D deep learning with PyTorch3D. In SIGGRAPH Asia, 2020.
[27] Yi Rong, Haoran Zhou, Kang Xia, Cheng Mei, Jiahao Wang, and Tong Lu. RepKPU: Point cloud upsampling with kernel point representation and deformation. In Conference on Computer Vision and Pattern Recognition, 2024.
[28] Dong Wook Shu, Sung Woo Park, and Junscok Kwon. 3D point cloud generative adversarial network based tree structured graph convolutions. In International Conference Computer Vision. 2019.
[29] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.

---

[30] Stanislaw Szymanowicz. C. Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3D reconstruction. In Conference Computer Vision and Pattern Recognition, 2024.
[31] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: Large multi-view Gaussian model for high-resolution 3D content creation. In European Conference Computer Vision, 2024.
[32] Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, W. Liu, and Yu-Gang Jiang. Pixel2Mesh: Generating 3D mesh models from single RGB images. In European Conference on Computer Vision, 2018.
[33] Philip Wolfe. Convergence conditions for ascent methods. II: Some corrections. SIAM Review, 13:185-188. 1971.
[34] Haiyang Xu, Yu Lei, Zeyuan Chen, Xiang Zhang, Yue Zhao, Yilin Wang. and Zhuowen Tu. Bayesian diffusion models shape reconstruction. Conference on Computer Vision and Pattern Recognition, 2024.
[35] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mëch, UIrich Neumann. DISN: Deep implicit surface network for high-quality single-view 3D reconstruction. In Advances in Neural Information Processing Systems, 2019.
[36] Guandao Yang, Xun Huang. Zekun Hao, Ming-Yu Yu Liu, Serge J. Belongie, and Bharath Hariharan. Point- Flow: 3D point cloud generation with continuous normalizing flows. In International Conference on Computer Vision, 2019.
[37] Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, Liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. KeypointNet: large-scale 3D keypoint dataset aggregated from numerous human annotations. In Conference on Computer Vision and Pattern Recognition, 2020.
[38] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: Latent point diffusion models for 3D shape generation. In Advances Neural Information Processing Systems, 2022.
[39] Linqi Zhou, Yilun Jiajun Wu. shape generation and completion through point-voxel diffuusion. In International Conference on Computer Vision, 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims regarding the novel approach to 3D reconstruction using diffusion priors and FCM-basee likelihood updates, which are validated by the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section A.3 discusses limitations including sensitivity thin structures and lack of colored point clouds in ShapeNet, as well as trade-offs between CD and shape- preserving metries like F-score.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Appendix A.3 states all assumptions (c.g.. L-smoothness, local convexity/- monotone gradient map, and step-size bounds tied to BB+Armijo and provides complete, self-contained proofs (lemmas -> main theorem) establishing descent/contraction of the FCM update.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper specifies the datasets used (ShapeNet and CO3D), the evalua- tion metrics (EMD, CD, and F-score), and the implementation details. It also provides information on how the baseline models were evaluated for a fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The abstract provides
GitHub
link for the code:
ht https:/ //ggt:/ggttt com./Seunggyyeck0155////
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https:://iig cc/ public wetpc/guids//caee SodeeSuomde ssionPolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//npsscc// httcc/ublicc/ggo / iwg.uuides... /Cads as i Poiioonoiiiyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details are provided in Section A.2 Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
The paper presents quantitative results in tables, but it does not include bars or statistical significance tests. The paper provides quantitative results using EMD, CD, and F-score, but does include error bars or statistical significance tests.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper reports inference time per sample (Table. 6 and Section specifies the use of NVIDIA RTX 6000 Ada Generation GPU and batch size. Although memory explicitly reported, the provided information sufficient for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /tt
Answer: [Yes]
Justification: The paper does not describe any deviations from the NeurIPS Code of Ethics. The research appears to be aligned with ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
16

---

Answer:
[No]
Justification: The paper does not explicitly discuss the broader societal impacts of the work. The paper focuses the technical aspects of the method and its performance.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not discuss the release of data models that would require specific safeguards. The paper introduces novel method for 3D reconstruction, but does not discuss releasing models or datasets that would require safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper cites the original papers for the datasets and models used, and appears to properly credit the original owners of the assets.
Justification:
Guidelines:
17

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppapersw soperrwwiccoeee com/ dat asets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new datasets or models. The paper focuses on novel method for 3D reconstruction.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects. The research focused on 3D reconstruction using existing datasets.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
18

---

Answer: [NA]
Justification: The paper does not involve research with human subjects, so IRB approval is not applicable. The research is focused 3D reconstruction using existing datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper does not utilize LLMs core component of its methodology. The paper focuses on novel method for 3D reconstruction.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer L.M policy (https: /ttp///eerpppp//p// wwt.eo/eeeeeennes////c/ccoccccee 2025/ L.LM) for what should or should not be described.
19

---

A.1 Algorithm
Algorithm I DDIM Sampling with FCM Likelihood Update
Require: Trained noise predictor 6g., Measurement y. Total diffusion steps T, DDIM cumulative
schedule 12" Stochasticity coefficient 7), Differentiable renderer R. FCM haper-parameters
oo, 1FCM, Lipschitz bound L, Numerical stabilizer
1: XT (0,
initialize with noise
2: for = downto 1 do
(a) DDIM prior prediction
3: ég x Eg. tX,,t)
4:
Xute (X, -1- a; e
(b) FCM likelihood refinement
5:
Xa +- Xoje
6:
for k t K 1 do
K= outer refinements
gs f Uxi ||y - R(xx))l
7:
8:
6g + 50 | |xx|1/1|B
9:
X Xx-
òF gi:
10:
S& Vx,
11y
R(xx))
11:
hy (gk
/6k
12:
0 + ||gN 11 ((g,, h&) +E)
13:
Og min
mnin(a) (at kaw 1/T.]
14:
X + X₁ at Bk
15:
if y R(xx) 2 ||y - R(xx)112
7FCM 0g ||g:| then
16:
a₁ /
single Anmijo back-off off
17:
Xx + X ag gk
18:
end if
19:
Xk+1 f- 5
b update iterate
20:
end for
21:
Xoje < XK
(c) DDIM update
σ: " 1-6r-1
22:
l-a:
23:
c~ (0,T
X,-1 f- var-1 Xoje= + /1
of é t
25: end for
Xo f- (X1 VT ai (xx))
27: return X
A.2 Implementation Details
To model the reverse process Ps. we employed a Diffusion Transformer, originally introduced in Point-E's unconditional model (24). as the neural network parameterized by 6, which predicts both Ne and Lig. All images were set to resolution of 224x224. For the ShapeNet dataset, 2.048 points were sampled and then upsampled 8,192 points for comparison [7]. In the case of CO3D, 8.2 points were directly sampled. To sufficiently refine the point cloud, we perform four FCM updates per DDIM sampling step. We the hyperparameters as follows: Песм 10
2/3. For ShapeNet, we set T 256 and őy = 2 x 10 For CO3D, we set T = 512 and & = 6 x 10 We use point cloud rendering processes provided by PyTorch3D 26 For ShapeNet, we set the radius of the point cloud rasterizer to 0.018 for airplane category and 0.027 for the other categories. For CO3D, we set the radius to 0.013. All experiments were performed using an NVIDIA RTX 6000 Ada Generation with batch size of 16.

---

A.3 Theoretical Analysis of FCM
In this appendix. provide detailed proofs for the theoretical guarantees of our Forward Curvature- Matching (FCM) method. We begin by formally establishing the properties of FCM step sizes, followed by proofs of loss decrease and convergence guarantees. Finally, we analyze how FCM integrates with DDIM sampling while preserving its contraction properties.
A.3.1 Bounds on FCM Step Sizes
We first establish that the FCM step size is guaranteed to lie within cell-behaved range, ensuring stable iterations. Our FCM approach relies on a directional curvature estimate:
X
6g = 60
(14)
|g
X X
òR gi:
(15)
g V (xx)),
(16)
gk
g%
hg
(17)
6%
This hę approximates the directional curvature along the gradient direction. Specifically, h2 is an
approximation of the geessan-vvector product V [² )gk
Lemma A.3.1 (Step Size Bounds). Assume L| | in the calculation of nuw Then the FCM
step size ogk (prior to any Armijo halving) satisfies:
1
1
S ak S
(18)
2L
L
Proof. From the finite difference approximation with h3s = (gk- B6)/ we analyze (gt, ha):
gk
Bi
(gk h₂)
gk
(19)
(1gga ||?
(ge, g())
(20)
8%
Under Assumption 3.1
(L (-ssoothnesss), we can establish that:
(gk: gk)
|| 2 Ld ||g ||
(21)
This implies:
(gk h4) 5 -
[1|g 1| (1|gx]1 LPd ||ga 113)
(22)
0x
= =||g
(23)
Therefore:
(gk. hg) + S T.1lga 11 2 +e
(24)
$ 2L gg
||
(25)
where the last inequality holds given our assumption that S
L /gg 12 This implies:
||
1
O
(26)
(gx, ha) +E
2L
Since we cap ag min(a 1/LJ, we ensure On S 1/L while maintaining the lower bound an 2 1/(2L). Remark A.3.2. If I/gga|l Rs 0, the raw step size of could become very large. However, in such cases, the Armijo condition will catch insufficient decrease and halve the step size once, still ensuring stable updates.

---

A.3.2 Firm Non-Expansiveness of the Gradient Step
Next, we establish that gradient step with the FCM step size firmly non-expansive, which crucial for integrating with the diffusion process.
Lemma A.3.3 (Firmly Non-Expansive Gradient Step). Lei T (u)
= u og VL(u) be the gradient
step operator with fixed O4k O. Under Assumptions 3.1 3.3 if0 < on 2/I,, then Ts is firmly
non-expansive:
[T&(u)= T&(V)ll < 1 v112 Okk (2 0kk |IV[(u) [[(2)]
(27)
Hence, Tg in particular non-expansive: [Tx(u)) [v(l & | -- vll-
Proof. Let = and A VC(u) vL(v). Then:
(T) - [vv]] = /| وx 1|
(28)
= 1 20 A A,) + o2ll
(29)
By the Baillon-Haddad theorem (which applies when L convex and L-smooth), VC is 1/L- cocoercive, meaning:
(A, Ag)
1
(30)
Substituting this into expression:
20c
- - (v)) [
A + a
(31)
L
2 T
||||
Onk
|ل 11
(32)
Since 0 < as 11/L our FCM algorithm (as established Lemma A.3. the factor - 0g
Thus, T3 is firmly non-expansive, and consequently, [[ (u) - Ta (v(v)
A.3.3 Guaranteed Loss Decrease
We now prove Theorem 3.4 from the main paper, which guarantees that FCM decreases the measure- ment loss each iteration.
Theorem A.3.4 (Guaranteed Loss Decrease). Let min
Neww 21
1 ]. The FCM algorithm ensures:
((xx+1) < ((xx) - cl [xxx)]]
(33)
Proof. We consider two cases:
Case (Armijo condition satisfied): When the initial step satisfies the Armijo condition, we have:
[(xx+1) S [(xx) //FC v
|
(34)
< [xxx) 2L
7FCM
|g |
(35)
where we used the lower bound ag 2 1. from Lemma A.3.1
27.
Case 2 (Armijo halving required): If the initial step fails the Armijo condition and we halve
then or 4L remains. By the descent lemma for L-smooth functions:
[(x++1) & C(xx) - ax mexl/gl11 +
Ig
(36)
2
= C(xx) 0% (1
Lo
||g ||
(37)
3

---

Since as S 2L
after halving, we have 1 Lac 2 1 Combined with ay: 2 1
this gives:
aL.
((x++1) I [(xx)
O |g:11
(38)
2
- ((xx)
7L g
(39)
Taking the minimum of the guaranteed decrease in both cases, get:
7FCM
((x++1) & [(x+)
min
|B
(40)
2L
8L
Corollary A.3.5 (Gradient Norm Convergence). Under FCM iterations, [| + as h: oo. and every cluster point is stationary.
Proof. By Theorem A.3.4 and Assumption 3.2 (lower bound on L), we have:
V[(x))]] S ((xo) Lint
(41)
Since O, must have k=o v oo, which implies [[V || 0 as & + This means that every cluster point the sequence {x+} a stationary point of L.
A.3.4 FCM Integration with DDIM
Finally, we analyze how FCM integrates with the DDIM sampling process and prove Proposition 3.5 from the main paper.
Proposition A.3.6 (Contraction Preservation). Let D be a DDIM step that is p-contractive (with L) in mean- mean-square sense:
( - -. & p||--v] v|l
(42)
Define ((( = Tx (u)),, where Tu(u)
ag VL()). Under Assumptions 3.1 3.3 and
On S L/L, Vr, is also p-contructive expectation, thus preserving the diffusion contraction property.
Proof. From Lemma A.3.3. we know that Tg is non-expansive: [[Tx(a) Ta (b) < ||a bll Therefore, for any u, V:
(. v.(v) = BI[ (+. - (v))
(43)
| (u) (v)
(44)
و u v
(45)
Thus, V, remains p-contraction mean-square sense.
A.3.5 Robustness to Non-Convexity
While Assumption 3.3 (local convexity) is used in our theoretical analysis, FCM shows empirical robustness even when this assumption is violated.
Remark A.3.7 (Behavior Under Non Non-Convexity), If local convexity fails, the firm non-expansiveness Tf may break. However, Theorem A.3.4 and CorollaryA.33 remain remain valid, guaranteeing that the
FCM step decreases and drives L(xx)) - 0. This makes FCM robust practice even for non-convex C.
A.3.6 Practical Parameter Settings
Over-estimating L in the algorithm only tightens the cap a 1/L and preserves all theoretical guarantees. Under-estimating L triggers the single Armijo halving, which prevents divergence while maintaining efficiency.
This combination of theoretical guarantees and practical robustness makes FCM an ideal choice for likelihood updates in eeffussio-bbsedd 3D reconstruction, enabling high-quality results.

---

Ground Truth
Image (Novel
View
Views
5 Views
(Novel View)
Figure 8: Qualitative results of multi view reconstruction on CO3D dataset.
EMD( 10)
CDX
F-score
Category
PC
200
BDM [34]
Ours
PC (20
BDM 24
Ours
PC 200
PC
BDM
Ours
airplane
0.551
0.552
0.476
0.434
0.409
0.378
0.473
0.457
0.524
0.543
0.524
0.535
0.517
0.487
0.507
0.460
0.359
0.331
0.330
0.386
0.274
table
0.662
0.635
0.691
0.649
0.644
0.727
0,270
0.260
0.284
0.319
Average
0.597
0.594
0.587
0.542
0.544
0.561
0.348
0.332
0.353
0.382
Table 4: Extended comparison of single-view 3D reconstruction on ShapeNet. The table includes results from the original studies with about 1000 NFEs. Scores marked with are reported from the original paper. Our method, however, achieves competitive performance with fewer function evaluations.
A.4 Additional Experiments
Comparison with other methods proposed in their original papers. Tab. 4ssows a comparison of different models evaluated on the ShapeNet dataset, where each model is applied according to the method originally proposed its respective study. PC uses an NFEs of 1000, while BDM uses total of 1080 NFEs- 1000 for its reconstruction model and an additional 80 for its prior model.
11)
CDOX10)
F-scrre
EMD(x10)
CD(x10)
F-score
&g
EMD(x10)
CDXx10)
F-score
100
0.731
0.770
0.307
10-6
0.585
0.563
0,373
5 10
0.644
0.615
0.343
0.585
0.373
10-5
0.579
0.377
0.587
0.561
0.588
0.564
0.376
10-4
0.587
0,561
0.382
10 2
0.594
0.574
0.369
2/3
0.587
0.561
0.382
0.586
0.565
0.370
10
0.665
0.660
0.330
Table 7: Hyperparameter study for FCM-guided sampling. Varying Lipschitz constant T., Armijo factor V/CM and the initial discrepancy radius &n for the scaled curvature probe.
5

---

Avsugge
XMIX 10)
1D)
F-score
PC
220
2.662
3,933
0.244
Method
1M)(x10)
CD(x10)
F-score
tine(e/somple)
1,206
0
PC
20
0.597
0.542
0.332
8.88
Durs(-vvjew)
1,011
1:11
0.388
BDM
34
0.594
0.544
0.353
10.56
Ouns(S-view)
Ours
0.587
0,561
0,382
6.03
Table 5: Quantitative results of CO3D dataset. The results represent the aver- age values for two categories used in the experiments, with ground truth reg- ularized to the range [-0.5, 0.5]. The F-score threshold is set to 0.2, and the CD corresponds to the results for the L1 metric.
Table 6: Time efficiency analysis of different methods. We report average scores of ShapeNet dataset and total sampling time (seconds per sample). The highest scores are marked in bold, and the second accond- scores are underlined. Our method achieves higher reconstruction accuracy with respect to F-score and EMD than prior ap- proaches while maintaining comparable inference speed.
Component
Ours
PC
BDM
single Armijo check strong Wolfe line search
FCM update(1 iteration)
5.339 ms
Local Conditioning
3.486 ms
(same as PC
0.889
6.589
NFEs
256
1000
1080
Table 8: Runtime breakdown. Left: component costs and NFE counts for each method (ours: 256
NFEs; PC 1000; BDM: 1180=1000+80) Right: cost of a single Armijo check (ours) versus a strong Wolfe line search. Our once-only Armijo strategy is substantially cheaper while preserving reliable descent, contributing to the overall speedup.
Despite using fewer NFEs, method still achieves the highest scores in terms of both EMD and F-score.
More experiments CO3D dataset. Tab. 5 presents a comparison with PC on the CO3D dataset and additionally provides scores for the multi-view setting. These results demonstrate the improved performance of our method over existing approaches. Furthermore, Fig. 8 illlutrrates the qualitative results of multi-view reconstruction on CO3D. To demonstrate broader applicability. Fig. 10 presents qualitative comparisons two additional CO3D categories remote and vase against PC
Hyperparameter Experiments. As shown in Tab the sampler behaves robustly once each knob is kept within reasonable range. In particular, as discussed in A.3.6 an overly conservative choice of 1/L (i.c., taking L too large) activates the clamping in Eq. [1| so the effective update becomes overly damped and Armijo progress can stall, leading slow or failed convergence. Nevertheless, the table shows substantial tolerance: the sampler remains reliable for L E [2/3, 10).
Time analysis. As shown in Tab. 6 our method achieves the best performance in both F-score and EMD score, highlighting its effectiveness even with a 32. 1 % reduction runtime. As shown in Tab. [. our once-only Armijo check is dramatically cheaper than conventional strong Wolfe line search. While single step in PC or BDM can be faster than our FCM step, our sampler deliberately opts for far fewer steps: 256 NFEs versus 1000 (PC2) and 1080 (BDM). This NFE gap dominates the end-to-end runtime-reducio the number denoiser/forward evaluations and leads to overall faster and more efficient reconstructions, without sacrificing accuracy.
Qualitative comparison with DPS-based methods. Fig. 9 shows comparison between our method and different methods. Here, the step size for the DPS-based method is set to the optimal value of y = 0.05 as identified in Fig. While the DPS- besed method captures the overall shape reasonably well, it fails to recover accurate color information. This limitation is attributed to the use of a fixed step size, which leads to suboptimal updates. In contrast, our method produces results that are more optimal with respect to the given measurements.
More examples and failure cases. Additional qualitative results are shown in Figs. and failure cases of our method are presented in Figs. 19 21 Most failure cases occur when the object has a complex structure, which can be attributed to the diffusion prior being misled by unfamiliar or uncommon data distributions.

---

Ground Truth
DDPM+DDP
DDIM+DPS
Ours
Reference View
Novel View
Reference View
Novel View
Figure 9: Qualitative comparison of reconstructions for different methods sampling, Both DPS based methods capture the overall shape but fail to preserve the correct colors due to the suboptimality.
A.5 Limitations
While the rendered image may resemble the reference image, the structure of the point cloud appears slightly thinner than the ground truth point cloud due to the radius of the rasterizer. This effect is partieularly noticeable in thin structures, such the legs of chair. Additionally, due to the unavailability of a colored point cloud dataset for ShapeNet, we used dataset generated by KeypointNet. However, since KeypointNet does not assign the actual mesh colors, this may lead to degradation in the quality of the reconstruction.
Direct control of point cloud positions for likelihood updates might limit CD metric per- formance compared reconstructions using only the learned prior(or posterior). However, considering the objective of our task "image(s) to 3D reconstruction", metrics such as F-score and EMD, which measure the shape similarity, are more aligned with the task's purpose than measuring the distance between individual points.

---

Image
Ground Tmath
PC
Ours
Image
Ground Tmihh
PC?
Ours
Reference Vieei
Novel View
CITTA O
TH
Referenc Viary
Novel
Figure 10: Qualitative comparison for single-view reconstruction on additional CO3D categories (remote, vase), against PC

---

Generation Process
Reconstruction
Figure 11: Generation trajectory and final reconstruction. Top: starting from pure noise T = 256 the sampler progressively denoises toward a coherent airplane; we display every 8th diffusion level (AT 8) down to T = 0. Bottom: the resulting T 0 sample rendered from multiple viewpoints.

---

Generation Process
Reconstruction
Figure 12: Generation trajectory and final reconstruction. Top: starting from pure noise T = 256. the sampler progressively denoises toward a coherent chair; we display every 8th diffusion level (AT 8) down to T = 0. Bottom: the resulting T 0 sample rendered from multiple viewpoints.
10

---

Reference Image
Reference View
Novel Views
Figure 13: Additional qualitative results for single-view reconstruction on ShapeNet: Airplane

---

Reference Image
Reference View
Novel Views
Figure 14: Additional qualitative results for single-view reconstruction on ShapeNet.: Car
12

---

Reference Image
Reference View
Novel Views
Figure 15; Additional qualitative results for single-view reconstruction on ShapeNet.: Chair
13

---

Reference Image
Reference View
Novel Views
AMM M
AMM M
APA
Figure 16: Additional qualitative results for single-view reconstruction on ShapeNet.: Table
14

---

Reference Image
Reference View
Novel Views
Figure 17: Additional qualitative results for single-view reconstruction CO3D.: Hydrant
15

---

Reference Image
Reference View
Novel Views
Figure 18: Additional qualitative results for single-view reconstruction on CO3D.: Teddybear
16

---

Reference Image
Reference View
Novel Views
Figure 19: Analysis of failure cases in single-view reconstruction on ShapeNet.: Airplane&Car
17

---

Reference
Reference View
Novel Views
Image
K E
Figure 20: Analysis of failure cases in single-view reconstruction on ShapeNet.: Chair&Table
18

---

Reference Image
Reference View
Novel Views
Figure 21: Analysis of failure cases in single-view reconstruction on CO3D.
19


Reviews:
[{"review_id": "3x8RRU5oAC", "reviewer": "Reviewer_rq8v", "summary": "This work proposes a method for efficiently and accurately reconstructing 3D point clouds from various types of measurements, such as RGB images and depth maps, by combining a pretrained diffusion prior with Forward Curvature-Matching (FCM)-based adaptive likelihood updates. The approach begins by generating an initial 3D point cloud using DDIM sampling. It then uses a differentiable renderer to compute gradients and curvature of the observation loss, and determines a dynamic step size in a Barzilai–Borwein style, updating the likelihood accordingly. This allows for faster and more stable convergence compared to previous methods that rely on fixed step sizes, and the method can be applied to RGB, depth, and multi-view inputs without retraining the prior model. Experimental results on ShapeNet and CO3D demonstrate superior performance over prior methods.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts\n1) The paper is well-written and easy to follow.\n2) The use of FCM along with Barzilai–Borwein-style dynamic step size estimation, combined with a one-time Armijo condition check, is both novel and practical. This overcomes the limitations of fixed step sizes in previous DPS methods while allowing adaptation to various modalities (RGB, depth, multi-view) without retraining the prior model.\n3) The modularity of the method—only requiring the renderer to be changed for different modalities—suggests it could be highly useful in practice.\n\nWeaknesses\n1) While the use of a differentiable renderer reduces the need for retraining the prior model, this also means that the overall reconstruction quality heavily depends on the renderer's performance. It is unclear how well the method performs under varying lighting conditions across views in multi-view input scenarios.\n2) On the real-world CO3D dataset, the evaluation is limited to only two categories: 'hydrant' and 'teddy bear'. A broader evaluation across more categories would strengthen the claims.\n3) The proposed method involves several hyperparameters (e.g., $\\eta_{FCM}$, $L_\\text{est}$, $T$, $\\delta_{0}$), but no detailed analysis is provided. Since different hyperparameters are used for different datasets (e.g., $T=256$, $\\delta_{0}=2×10^{-2}$ for ShapeNet; $T=512$, $\\delta_{0}=6×10^{-3}$ for CO3D), a justification or ablation study is needed to explain how these values were chosen and how sensitive the method is to these choices.", "was_revised": true, "final_justification": "My concerns have been sufficiently addressed. Therefore, I maintain my original positive rating.", "rebuttal": "Thank you for the detailed review and constructive feedback. The following is our response to the specific comments and questions.\n\n> While the use of a differentiable renderer reduces the need for retraining the prior model, this also means that the overall reconstruction quality heavily depends on the renderer's performance. It is unclear how well the method performs under varying lighting conditions across views in multi-view input scenarios.\n\nYou correctly identify that our reconstruction quality depends on the differentiable renderer. This is indeed a deliberate design choice—by decoupling the learned prior from the rendering-based likelihood, we gain modularity and flexibility. As differentiable rendering improves, our method can immediately benefit without retraining. This contrasts with methods like PC$^2$ and BDM that embed rendering assumptions into their learned posteriors.\n\nRegarding lighting variations in multi-view scenarios: Our renderer uses simple alpha compositing (Eq.~6) without complex shading models, making it relatively robust to lighting changes. The point colors we optimize represent diffuse appearance rather than view-dependent effects. Additionally, our multi-view loss (Eq. 12) averages gradients across all views, which naturally balances variations between views.\n\nThat said, we acknowledge we haven't explicitly tested extreme lighting variations. Our CO3D experiments include real-world captures with natural lighting differences between views, and the method performs well in these scenarios. However, systematic evaluation under controlled lighting changes would be valuable future work.\n\nFor applications requiring explicit lighting modeling, our framework could incorporate more sophisticated renderers that separate geometry and appearance—the modularity of our approach makes such extensions straightforward. \n\n> On the real-world CO3D dataset, the evaluation is limited to only two categories: 'hydrant' and 'teddy bear'. A broader evaluation across more categories would strengthen the claims.\n\nWe selected these categories to cover diverse real-world conditions and object characteristics. Hydrant represents an outdoor object with strong geometric symmetry, while teddy bear represents an indoor object with complex, asymmetric geometry. Together, they capture two contrasting settings that we believe are sufficient to demonstrate the robustness and versatility of our method. \n\nThat said, we agree that expanding the evaluation to a broader set of categories would further strengthen our claims. We made a sincere attempt to include additional categories during the rebuttal period. However, the computational cost of additional training is substantial, and unfortunately, it could not be accommodated within the limited time available for the rebuttal period. \n\nWe will address this in the camera-ready version by conducting an expanded evaluation across more CO3D categories, thereby providing a more comprehensive validation of our method. \n\n> The proposed method involves several hyperparameters (e.g., $\\eta_{\\text{FCM}}, L_{\\text{est}}, T, \\delta_0$), but no detailed analysis is provided. Since different hyperparameters are used for different datasets (e.g., $T=256,\\delta_0=2\\times 10^{-2}$ for ShapeNet; $T=512,\\delta_0 = 6\\times 10^{-3}$), a justification or ablation study is needed to explain how these values were chosen and how sensitive the method is to these choices.\n\n|$L_{\\text{est}}$ | EMD($\\times10$) | CD($\\times10$) | F-score | $\\\\eta_{\\text{FCM}}$ | EMD($\\times10$) | CD($\\times10$) | F-score |\n|:---|---|---|---|:---|---|---|---|\n|$L_{\\text{est}}=100$ | 0.731 | 0.770 | 0.307 | $\\\\eta_{\\text{FCM}} = 10^{-6}$ | 0.585 | 0.563 | 0.373 | \n|$L_{\\text{est}}=10$ | 0.585 | 0.563 | 0.373 | $\\\\eta_{\\text{FCM}} = 10^{-5}$ | 0.579 | 0.566 | 0.377 | \n|$L_{\\text{est}}=1$ | 0.588 | 0.564 | 0.376 | $\\\\eta_{\\text{FCM}} = 10^{-4}$ | 0.587 | 0.561 | 0.382 | \n|$L_{\\text{est}}=2/3$ | 0.587 | 0.561 | 0.382 | $\\\\eta_{\\text{FCM}} = 10^{-3}$ | 0.586 | 0.565 | 0.370 | \n\n| $\\delta_0$ | EMD ($\\times10$) | CD ($\\times10$) | F-score |\n|:---|---|---|---|\n| $\\delta_0=5\\times10^{-2}$  | 0.644| 0.615 | 0.343 |\n| $\\delta_0=2\\times10^{-2}$  | 0.587 | 0.561 | 0.382 |\n| $\\delta_0=10^{-2}$ | 0.594 | 0.574 | 0.369 |\n| $\\delta_0=10^{-3}$ | 0.665 | 0.660 | 0.330 |\n\nWe used different hyperparameters for each dataset based on their characteristics: The CO3D dataset is derived from real-world captures and exhibits complex geometry. Accurately representing such complexity and recovering fine-grained details require a higher point density. In contrast, the ShapeNet dataset consists of synthetic objects with simpler geometry, allowing reconstructions to be achieved with a lower number of points. Given that denser point clouds inherently require more frequent and finer updates during sampling, we selected a smaller ($\\delta_0$) and increased the total number of sampling steps ($T$) to achieve optimal reconstruction performance under this experimental configuration.\n\nWe provide detailed ablation studies for key hyperparameters ($L_{\\text{est}}, \\eta_{\\text{FCM}}, \\delta_0$) in the above tables, conducted on single-view ShapeNet reconstruction. These show our method is robust to moderate parameter variations while indicating preferable configurations. We will ensure these justifications are clearly described in the camera-ready version."}, {"review_id": "JypKf5aYM1", "reviewer": "Reviewer_nEx8", "summary": "This paper introduces method for reconstructing 3D point clouds from images. The key contributions are:\n\n- To solve inverse problem, they combine prior models with likelihood updates instead of learning posterior distribution directly. Therefore, the proposed method is free to the number of inputs and inputs modalities.\n- Previous approach, which decomposes the posterior into a trainable prior and an updatable likelihood, struggles to determine step sizes for likelihood updates, leading to slow convergence and suboptimal quality. To address these limitations, they propose Forward Curvature-Matching optimization, which computes an adaptive step size.\n\nThe proposed method is evaluated on ShapeNet and CO3D datasets, demonstrating state-of-the-art performance in reconstructing high-quality point clouds without heuristic step size determination.", "questions": "1. Are there some reasons to choose point cloud as main representations? The qualitative results compared to previous approaches, which learns the posterior distribution directly, are not superior in Fig. 3 and Fig. 4. The authors already mention that the proposed method can be extended to Gaussian Splatting, and there is baseline method, GSD [23]. If the progress starts from GSD, it is much more clear to demonstrate the effectiveness of FCM compared to GSD. As I understand, there is no specific design for 3D point cloud and the proposed method can be general for diffusion-based methods.\n2. In Tab. 1, the geometric accuracy (EMD and CD) in chair and table is worse than BDM [34]. In appendices, the radius of the rasterizer might affect the performance in chair category. Table category also has similar issue? Or there are more reasons of these performance gap? Additionally, it might be better to mention the performance gap and reasons in the main paper.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. They decompose the posterior into a trainable prior and an updatable likelihood, which has several advantages. The model can train without conditioning signals and support single-view or multi-view images and various measurement modalities.\n2. Forward Curvature-Matching optimization determines optimal step sizes, enabling the complex optimization.\n3. The proposed framework might have potential in other domains, which are solving inverse problems with non-linear nature (e.g., Gaussian Splatting, Meshes as mentioned by authors).\n4. The proposed method outperforms fixed-step DPS approaches with varying step sizes, as shown in Fig. 7.\n5. They demonstrate the advantages of their contributions, versatility, in Fig. 5 and Fig. 6.\n\n### Weaknesses\n\nThere is no major weakness of the proposed method. They also mention their limitations.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I keep my original positive rating.", "rebuttal": "Thank you for the detailed review and constructive feedback. The following is our response to the specific comments and questions.\n\n> (Q1) Are there some reasons to choose point cloud as main representations? The qualitative results compared to previous approaches, which learns the posterior distribution directly, are not superior in Fig. 3 and Fig. 4. The authors already mention that the proposed method can be extended to Gaussian Splatting, and there is baseline method, GSD [23]. If the progress starts from GSD, it is much more clear to demonstrate the effectiveness of FCM compared to GSD. As I understand, there is no specific design for 3D point cloud and the proposed method can be general for diffusion-based methods.\n\nWe fully recognize the importance of clearly justifying our choice of the point-cloud representation. Upon careful consideration, we determined that Gaussian Splatting posed certain challenges regarding rigorous quantitative evaluations. Specifically, the absence of well-established baseline comparison groups and the inherent difficulty in defining standardized and objective evaluation metrics for Gaussian-based methods would hinder our ability to provide precise quantitative assessments.\n    \nConsequently, we opted for a point-cloud representation primarily due to its widespread use, well-defined evaluation protocols, and availability of clearly defined baseline methods for direct comparisons. Our goal was to establish an explicit and quantitative demonstration of the efficacy of the proposed method. \n    \nNevertheless, we agree that our method is not inherently limited to point clouds; rather, it represents a general framework applicable to other diffusion-based representations.\n\n> (Q2) In Tab. 1, the geometric accuracy (EMD and CD) in chair and table is worse than BDM [34]. In appendices, the radius of the rasterizer might affect the performance in chair category. Table category also has similar issue? Or there are more reasons of these performance gap? Additionally, it might be better to mention the performance gap and reasons in the main paper.\n\nYou are correct in noting that the use of a finite radius in the rasterizer can cause reconstructed structures—especially slender features like legs—to appear thinner compared to their ground-truth counterparts.\n\nHowever, we believe this alone does not fully account for the lower CD and EMD scores observed in the chair and table categories. These metrics may also reflect differences in the spatial distribution and geometric complexity characteristic of each object category. For instance, objects in the airplane category generally exhibit flatter shapes with limited variation along the vertical (y-axis) dimension. In contrast, objects in the chair and table categories tend to span more evenly across all three dimensions. As we noted in our limitations, utilizing gradient-based updates can introduce uneven density distributions in reconstructed point clouds. These density imbalances disproportionately affect metrics such as CD and EMD, particularly in categories with greater spatial coverage and complexity, such as chair and table. We believe this density imbalance issue significantly contributes to the performance gap observed in these two categories."}, {"review_id": "jvmfHmjqFf", "reviewer": "Reviewer_zdEn", "summary": "The paper focuses on reconstructing point clouds from images. Previous methods are limited to fixed-view inputs and typically require retraining the network when switching to a different input modality (e.g., from RGB images to depth maps). To address these issues, the authors propose decomposing ***p(X∣y)*** into a trainable prior ***p(X)*** and an updatable likelihood ***p(y∣X)***. They further introduce a Forward Curvature-Matching (FCM) optimization scheme to guide the update of ***p(y∣X)***. The paper provides a relatively rigorous theoretical derivation to support the proposed framework.", "questions": "See the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed method supports both single-view and multi-view inputs, using either RGB images or depth maps. It is supported by extensive experimental results and effectively broadens the application scope of point cloud diffusion models.\n2. The theoretical analysis and mathematical derivations in the paper are rigorous and appear convincing.\n3. The experiments in the paper are comprehensive and effectively support the conclusions.\n4. The paper is overall well-written and easy to follow.\n\n\n**Weaknesses:**\n\n1. The proposed method requires a rendering operation at each sampling step during diffusion to correct the optimization direction. This could be more time-consuming compared to baseline methods, and the authors should provide a runtime comparison to support the method's practicality.\n2. The point cloud representation does not explicitly include an alpha parameter. It is unclear how the alpha value for each point in Eq. (6) is computed.", "was_revised": true, "final_justification": "The authors' response has addressed my concerns, and I will maintain my positive rating.", "rebuttal": "Thank you for the detailed review and constructive feedback. The following is our response to the specific comments and questions.\n\n> The proposed method requires a rendering operation at each sampling step during diffusion to correct the optimization direction. This could be more time-consuming compared to baseline methods, and the authors should provide a runtime comparison to support the method's practicality.\n\n| **Component**        | **Ours**       | **PC$^2$**     | **BDM**                          |\n|----------------------|----------------|----------------|----------------------------------|\n| FCM update           | 5.339 ms      | --             | --                               |\n| Local Conditioning   | --             | 3.486 ms       | (same as PC$^2$)                 |\n| Sampling steps       | 256            | 1000           | 1080 (1000+80)                   |\n\n| $k$    | EMD ($\\times10$) | CD ($\\times10$)  | F-score | Time (s/sample)  |\n|--------|------------------|------------------|---------|------------------|\n| $k=4$  | 0.587            | 0.561            | 0.382   | 6.03             |\n| $k=3$  | 0.588            | 0.567            | 0.372   | 4.63             |\n| $k=2$  | 0.622            | 0.606            | 0.359   | 3.97             |\n\n| **Single Armijo Check**  | **Strong Wolfe Line Search**  |\n|--------------------------|-------------------------------|\n| 0.889 ms                 | 6.589 ms                      |\n\nPC$^2$ employs local conditioning, where the point cloud at the current timestep undergoes surface projection to be fed into the model input. It is true that the likelihood update in our method takes approximately 5.339 ms, which is somewhat more computationally intensive than the local conditioning step in PC$^2$ (3.486 ms).\n\nHowever, our method uses only 256 sampling steps compared to PC$^2$'s 1000 steps and BDM's 1080 steps (actually NFEs). This substantial reduction in total steps leads to significantly faster overall inference times, as demonstrated in Appendix Table 6, where our method achieves 32.1\\% to 42.9\\% faster runtime than existing approaches while simultaneously delivering superior reconstruction accuracy in EMD and F-score metrics.\n\nMoreover, as shown in the second table, the number of FCM iterations (k=4 in our experiments) is a user-defined parameter that allows flexible trade-offs between computational cost and reconstruction quality. Users can reduce k and increase sampling steps, or vice versa, providing flexibility for different computational constraints and quality requirements.\n\nIn addition, our single Armijo check approach is computationally efficient compared to traditional optimization strategy. As shown in the third table, our single Armijo check requires only 0.889 ms compared to 6.589 ms for strong Wolfe line search methods—approximately 86.5\\% faster. This efficiency is crucial since traditional strong Wolfe conditions would require dozens of forward evaluations across hundreds of diffusion steps, making them prohibitively expensive for this application.\n\nWe believe these empirical results and design choices clearly demonstrate the practicality and efficiency of our proposed FCM sampling framework. (All experiments were conducted with a batch size of 16, and the time evaluation was measured per sample.)\n\n> The point cloud representation does not explicitly include an alpha parameter. It is unclear how the alpha value for each point in Eq. (6) is computed.\n\nYou are correct that point-cloud representations inherently lack explicit alpha values, and we acknowledge that this detail was not clearly addressed in the current version. In our implementation, we resolve this by assigning an alpha value to each point based on its spatial proximity to the target pixel in the image space. \n\nSpecifically, each point’s alpha value is computed using the following relationship: \n\n$\\alpha_i = 1-\\frac{d_{i}^2}{r^2}$\n\nHere, $r$ represents the rasterizer radius, and $d_i$ is the Euclidean distance between the pixel center and the projected position of the point within the image space. Through this proximity-based weighting, our approach effectively captures each point’s relative influence on the final rendered pixel. These computed alpha values are subsequently used in the compositing process described by Equation (6) to determine the final pixel color."}, {"review_id": "t6qlbOuI6a", "reviewer": "Reviewer_QN2v", "summary": "The paper addresses point cloud reconstruction from one or multiple view images.\nThe basic approach involves a diffusion-based model that uses rendering as the operator to transform a noisy point cloud into a 2D representation.\nThe authors present a new approach for adaptive step size, which aims to improve point cloud reconstruction quality and lead to a faster process.", "questions": "1. Can you present the full diffusion step of your method?\n2. Why did BDM reconstruct entirely different chair legs and table? Even when the reference image is completely different, and the EMD and CD also present better scores?\n3. The plane seems to be worse with your approach from the novel's view, why?\n4. The step size is the minimum in equation 10, did you noticed objects that did not converged because of too low step size?", "limitations": "1. The point cloud reconstruction appears to prioritize satisfying the reference view, potentially at the expense of the novel views. This results in a strong reconstruction of the reference view but weaker results for novel views.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The reconstruction of the reference view demonstrates strong performance compared to baseline methods.\n2. The proposed approach for approximating the Hessian without expensive computations is noteworthy.\n\nWeaknesses:\n1. A key observation is that the presented method does not explicitly leverage the 3D nature of the problem. The adaptive step size mechanism appears applicable to any gradient-based optimization problem, not specific to 3D reconstruction.\n2. The real-world reconstruction of novel views remains somewhat limited, showing only marginal improvement over baseline methods.\n3. The amount of points in the reconstructed point cloud is fixed, so i assume that more points are assigned to the reference view and less to the novel view.", "was_revised": true, "final_justification": "i will keep my rating", "rebuttal": "Thank you for the detailed review and constructive feedback. The following is our response to the specific comments and questions.\n\n> A key observation is that the presented method does not explicitly leverage the 3D nature of the problem. The adaptive step size mechanism appears applicable to any gradient-based optimization problem, not specific to 3D reconstruction.\n\nYou are correct that FCM, as an optimization framework, is not exclusively 3D-specific. However, FCM was developed to address a fundamental challenge in 3D reconstruction: determining optimal step sizes for likelihood updates when working with expensive, non-linear forward operators like neural renderers.\n\nIn conventional optimization with linear operators, optimal step sizes can often be computed analytically using adjoint operations. However, for non-linear operators like 3D rendering processes, no such closed-form solution exists. Traditional approach—line search with Wolfe conditions (requiring 10-20 forward evaluations)—becomes computationally prohibitive when each evaluation involves an expensive rendering operation.\n\nFCM addresses this by approximating directional curvature using only forward operations: $\\mathbf{h}_k = (\\mathbf{g}_k - \\mathbf{g}_k')/\\delta_k$. This provides sufficient information for robust step size determination, making it practical for expensive forward operators where multiple evaluations would be infeasible. \n\n> The real-world reconstruction of novel views remains somewhat limited, showing only marginal improvement over baseline methods.\n\nWhile novel view improvements on CO3D appear visually modest, our F-score shows meaningful gains: 0.281 vs PC$^2$'s 0.244 (15.2\\% improvement). This better reflects the perceptual quality improvements that may not be immediately obvious in rendered views.\n \nThe visual subtlety occurs because real-world data presents unique challenges—complex lighting, cluttered backgrounds, and partial occlusions make substantial improvements difficult. Our method excels at preserving geometric boundaries (evident in the hydrant's base and teddy bear contours in Figure 4), though synthesizing textures in unseen regions remains challenging for all single-view methods.\n \nNotably, these improvements require no additional training—achieved purely through better optimization during inference. Our multi-view results demonstrate stronger gains (F-score: 0.281→0.423 with 5 views), showing our optimization effectively leverages additional information when available.\n\n> The amount of points in the reconstructed point cloud is fixed, so i assume that more points are assigned to the reference view and less to the novel view.\n\nIt is true that the number of points in the reconstructed point cloud is fixed in our current experimental setup. As you pointed out, the qualitative results may sometimes give the impression that the reconstructed point cloud is denser in the reference view and sparser in the novel view.\n\nHowever, we believe this phenomenon is not primarily due to an uneven allocation of points across different viewpoints. Rather, it stems from the fact that the FCM update explicitly aligns the point cloud to minimize rendering error from the reference view. Since the novel view is not directly involved in the optimization process, the reconstructed points may not be as well aligned from that perspective, occasionally resulting in sparse or incomplete appearances when viewed from novel angles.\n\n> (Q1) Can you present the full diffusion step of your method?\n\nWe understand you're looking for a comprehensive view of our method. Currently, Figure 1 (left) shows the visual progression and Algorithm 1 (Appendix A.1) provides the mathematical details. However, we recognize this may not fully convey the complete process.\n \nDue to rebuttal format constraints, we cannot provide additional visualizations here, but we commit to adding a more detailed figure in the camera-ready version showing step-by-step visualization at finer intervals (e.g., every 32 steps).\n\n> (Q2) Why did BDM reconstruct entirely different chair legs and table? Even when the reference image is completely different, and the EMD and CD also present better scores?\n\nYou correctly observe that BDM reconstructs geometry that differs from the reference image while achieving better EMD/CD scores. This occurs because BDM learns the posterior distribution $p(\\mathbf{X}|\\mathbf{y})$ with strong category-level priors. During inference, these priors can guide reconstructions toward typical category shapes rather than matching the specific input geometry, particularly evident in the chair and table examples in Figure 3. \n    \nAdditionally, the disconnect between better EMD/CD scores and visual accuracy reveals how these metrics operate. EMD measures average point transport distance, favoring smooth, uniformly distributed point clouds regardless of whether they match the input. CD similarly rewards regular point spacing over geometric accuracy. A well-distributed but incorrect shape can thus score better than an accurate but slightly irregular reconstruction. Our method directly optimizes the rendering loss $\\|\\mathbf{y}-\\mathcal{R}(\\mathbf{X})\\|_2$, ensuring the reconstruction matches the input image. While this can create slight density variations in the point cloud (more pronounced in spatially extended objects like chairs and tables), it preserves input-specific geometric features that BDM's prior-driven approach might smooth away. This explains why F-score better captures reconstruction quality—it evaluates whether the correct geometry is present rather than point distribution uniformity. Our better F-score (0.382 vs 0.343) confirms we reconstruct the actual input geometry more faithfully, even when EMD/CD metrics favor BDM's smoother but less accurate reconstructions.\n\n> (Q3) The plane seems to be worse with your approach from the novel's view, why?\n\nAs mentioned in a previous response, our method may produce slight density imbalances in the reconstructed point cloud due to the nature of gradient-based updates. This can occasionally lead to less uniformly distributed points, which may appear visually noisier compared to methods such as PC$^2$ or BDM that produce more uniformly sampled outputs.\n\nHowever, it is important to note that while PC$^2$ and BDM often generate plausible shapes, they slightly deviate from the actual geometry of the input image. In contrast, our method prioritizes capturing structurally accurate details by closely aligning the reconstruction with the input image through iterative refinement. Even if the resulting point distribution may appear less uniform, we find that our reconstructions better preserve category-specific fine-grained structures—such as the wing and tail shapes in airplanes—reflecting a closer adherence to the reference image.\n\nAlso, our quantitative metrics across the airplane category still show competitive performance (F-score: 0.543 vs PC$^2$: 0.498 and BDM: 0.543), suggesting that while some views may look less visually smooth, our reconstructions capture overall geometry more accurately.\n\n> (Q4) The step size is the minimum in equation 10, did you noticed objects that did not converged because of too low step size?\n\n| **$1/L_{\\text{est}}$**       | EMD ($\\times10$) | CD ($\\times10$) | F-score |\n|----------------------------|------------------|------------------|---------|\n| $1/L_{\\text{est}}=0.01$       | 0.731            | 0.770            | 0.307   |\n| $1/L_{\\text{est}}=0.1$        | 0.585            | 0.563            | 0.373   |\n| $1/L_{\\text{est}}=1$         | 0.588            | 0.564            | 0.376   |\n| $1/L_{\\text{est}}=1.5$       | 0.587            | 0.561            | 0.382   |\n\nIn Eq.(10), we introduced a capping mechanism that limits the step size by taking the minimum between the raw step size and the reciprocal of the estimated Lipschitz constant. This mechanism is specifically designed to ensure stable convergence while maintaining theoretical guarantees. As shown in the empirical ablation study in the table, we analyzed different choices for the estimated Lipschitz constant ($L_{\\text{est}}$) and their impact on reconstruction performance metrics.\n\nFrom the results, we observed that choosing a value too small ($1/L_{\\text{est}} =0.01$) indeed results in overly conservative updates, significantly reducing performance (EMD: 0.731, CD: 0.770, F-score: 0.307). However, moderately larger values ($1/L_{\\text{est}}=0.1,1, 1.5$) demonstrated consistently improved and stable convergence."}]

Author rebuttal:

Meta-review: The paper presents Forward Curvature-Matching (FCM) that can determine optimal step size dynamically in the diffusion-based 3D point cloud reconstruction problem from single images. The novelty and efficacy of the proposed method are acknowledged by all reviewers. All reviewers recommend acceptance, explicitly noting that all concerns have been addressed during the rebuttal and author-reviewer discussion phase. 

Given this strong consensus of reviewers and the clear strengths of the paper, the AC recommends accepting this paper.
Decision: Accept (spotlight)