Title: Adaptive 3D Reconstruction via Diffusion Priors and Forward Curvature-Matching Likelihood Updates
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D reconstruction

point cloud generation

diffusion model
Abstract: Reconstructing high-quality point clouds from images remains challenging in computer vision. Existing generative models, particularly diffusion models, based approaches that directly learn the posterior may suffer from inflexibility—they require conditioning signals during training, support only a fixed number of input views, and need complete retraining for different measurements. Recent diffusion-based methods have attempted to address this by combining prior models with likelihood updates, but they rely on heuristic fixed step sizes for the likelihood update that lead to slow convergence and suboptimal reconstruction quality. We advance this line of approach by integrating our novel Forward Curvature-Matching (FCM) update method with diffusion sampling. Our method dynamically determines optimal step sizes using only forward automatic differentiation and finite-difference curvature estimates, enabling precise optimization of the likelihood update. This formulation enables high-fidelity reconstruction from both single-view and multi-view inputs, and supports various input modalities through simple operator substitution—all without retraining. Experiments on ShapeNet and CO3D datasets demonstrate that our method achieves superior reconstruction quality at matched or lower NFEs, yielding higher F-score and lower CD and EMD, validating its efficiency and adaptability for practical applications. Code is available at https://github.com/Seunghyeok0715/FCM.

Adaptive 3D Reconstruction via Diffusion Priors and Forward Curvature-Matching Likelihood Updates

Abstract

Reconstructing high-quality point clouds from images remains challenging in computer vision. Existing generative models, particularly diffusion models, based approaches that directly learn the posterior may suffer from inflexibility-they require conditioning signals during training, support only a fixed number of input views, and need complete retraining for different measurements. Recent diffusion-based methods have attempted to address this by combining prior models with likelihood updates, but they rely on heuristic fixed step sizes for the likelihood update that lead to slow convergence and suboptimal reconstruction quality. We advance this line of approach by integrating our novel Forward Curvature-Matching (FCM) update method with diffusion sampling. Our method dynamically determines optimal step sizes using only forward automatic differentiation and finite-difference curvature estimates, enabling precise optimization of the likelihood update. This formulation enables high-fidelity reconstruction from both single-view and multi-view inputs, and supports various input modalities through simple operator substitution-all without retraining. Experiments on ShapeNet and CO3D datasets demonstrate that our method achieves superior reconstruction quality at matched or lower NFEs, yielding higher F-score and lower CD and EMD, validating its efficiency and adaptability for practical applications. Code is available at here.

1 Introduction

Three-dimensional reconstruction has become increasingly important across diverse applications including robotics, autonomous driving, augmented reality, and virtual environments. Among various 3D representations, point clouds serve as a fundamental data structure for representing objects and scenes due to their simplicity and flexibility. However, generating high-quality point clouds that accurately capture intricate details remains challenging, particularly when working with limited input information such as single-view images.

Recent advances in deep generative models, particularly diffusion models, have shown remarkable success in generating high-fidelity images [11, 14] and 3D data. Diffusion models use an iterative denoising process to progressively transform random noise into structured outputs, making them effective for capturing complex geometric patterns. In the domain of point cloud generation, researchers have begun exploring diffusion-based approaches with promising results [19, 39, 38, 24, 20, 21, 34].

While diffusion models offer powerful generative capabilities, applying them to 3D reconstruction presents unique challenges due to its nature as an inverse problem. In typical inverse problems (formulated as y = A x ), iterative optimization methods solving least-squares objectives can determine optimal step sizes analytically using gradients that incorporate A ⊤ (the adjoint of A ). However, 3D object rendering represents a particularly challenging case where the rendering operator is complex and non-linear, making the computation of the adjoint operation intractable. Since classical step-size

෩

𝐗0|𝑡

𝑇 = 0

𝑇 = 32

𝑇 = 96

𝑇 = 160

𝑇 = 224

𝑇 = 256

𝐗𝑇

Input

Point  Cloud

Output

BDM

(NFE=276)

PC

²

(NFE=256)

Ours

(NFE=256)

Ground

Truth

F-score :

0.310

0.343

0.382

formulas require an adjoint of the forward operator, the absence of a tractable renderer adjoint complicates step-size selection. This fundamental challenge impacts how researchers approach diffusion-based point cloud generation, especially when incorporating image-based guidance.

Current image-to-point-cloud methods predominantly learn the score of the posterior distribution ∇ log p ( X | y ) directly, where X represents the point cloud and y represents image measurements. This direct approach incurs significant limitations: it necessitates including images as conditioning signals during training [20], restricts models to a fixed number of input views without specialized encoders [8], and requires computationally expensive retraining whenever measurement types change (e.g., from RGB images to depth maps).

A promising alternative approach [22] decomposes the posterior p ( X | y ) into a trainable prior p ( X ) and an updatable likelihood p ( y | X ) , employing Diffusion Posterior Sampling (DPS) [6] with gradient updates via ∇ log p ( y | X ) for Gaussian splatting-based 3D reconstruction. While this decomposition is conceptually straightforward and modular, current implementations struggle with a critical limitation: determining appropriate step sizes for the likelihood updates. The non-linear nature of 3D rendering prevents analytical step size determination, leading existing methods to rely on heuristic, fixed step sizes [22]. This results in slow convergence, suboptimal reconstruction quality, and often necessitates additional 2D diffusion models for refinement, further complicating the pipeline.

To address these limitations, we present a novel approach that combines a point cloud diffusion model with Forward Curvature-Matching (FCM) optimization. Our approach, illustrated in Fig. 2, computes an adaptive step size using a Barzilai-Borwein rule and refines it with an Armijo backtracking condition, enabling more precise control. Our key insight is that by incorporating FCM's principled, curvature informed step size determination into the diffusion sampling process without any adjoint operations, we can effectively navigate the complex optimization landscape of 3D reconstruction.

Unlike previous DPS-based methods that rely on heuristic step sizes for the likelihood update, our approach employs FCM optimization to dynamically determine optimal step sizes. The key innovation is our reliance solely on the differentiable forward pass for curvature-informed step-size determination, obviating the adjoint. This enhancement enables significantly more efficient and accurate optimization during the diffusion sampling process. The technical contributions of our work include:

We integrate the FCM method with the reverse process of diffusion models, enabling high-fidelity point cloud reconstruction that accurately matches input images.

Our gradient-based updates are not constrained by the number of input images, allowing for point cloud reconstruction from either single-view or multi-view images without modifying the base model.

Our method can be applied to various measurement modalities (such as RGB image to 3D object or depth map to 3D object) by simply substituting the appropriate operator rather than retraining the entire model, significantly enhancing flexibility and efficiency.

We demonstrate the effectiveness of our approach by reconstructing colored point clouds from both synthetic and real-world datasets. Our method achieves more accurate reconstruction with fewer neural function evaluations (NFEs) compared to existing techniques, validating the efficiency of our FCM-based likelihood optimization. We further demonstrate the adaptability of our approach by applying it to both multi-view reconstruction and depth map to point cloud generation without retraining, highlighting its potential for diverse applications. The remainder of this paper is organized as follows: Section 2 reviews related work, Section 3 presents the proposed method, Section 4 details our experimental results, and Section 5 concludes with a discussion of future directions.

2 Related Work

3D Reconstruction from Images. As interest in 3D content creation continues to grow, research on reconstructing 3D shapes from 2D observations has advanced significantly. This challenging task requires inferring complete 3D structures, including both visible and occluded regions, from limited viewpoints. The difficulty is compounded by the scarcity of large-scale 3D datasets.

Various 3D representations have been explored for reconstruction, each with distinct advantages: mesh-based methods [13, 32, 3] offer compact representation but struggle with topological complexity; voxel-based approaches [15] provide a regular structure but face resolution limitations; point cloud methods [20, 34, 16] offer flexibility with additional rendering requirements; implicit functions [23, 10, 5, 12] enable high-quality rendering but are computationally intensive; and Gaussian splatting techniques [31, 30, 22] balance quality and efficiency.

Point cloud generative models have evolved from early GAN-based [1, 9, 28] and VAE-based [36] approaches to more recent diffusion-based methods. Diffusion models offer several advantages: stable training dynamics, high-quality generation capabilities, flexibility in conditioning, and a strong probabilistic foundation. The seminal work by Luo et al. [19] introduced diffusion models for point cloud generation, with subsequent research extending these methods for various applications [38, 17].

Building on these advancements in 3D generative modeling, recent diffusion-based approaches have significantly advanced image-to-point cloud reconstruction. PC 2 [20] performs single-view reconstruction by denoising a point cloud with projection conditioning, which ensures geometric consistency between the reconstruction and input view. However, it directly learns the posterior distribution, requiring images during training and limiting adaptability to varying input conditions. Bayesian Diffusion Models (BDM) [34] offer a complementary perspective by factorizing the 3D reconstruction task into a learned score of the prior ∇ log p ( X ) trained solely on 3D shapes and a learned score of the posterior ∇ log p ( X | y ) trained with paired image-shape data. During inference, the prior and posterior models exchange intermediate outputs over multiple denoising steps. While this "fusion-with-diffusion" paradigm is effective, BDM relies on a PC 2 -like trained posterior score function that requires images during training, thus limiting its adaptability to varying input modalities.

Diffusion Posterior Sampling. Diffusion Posterior Sampling (DPS) [6] proposes a framework for solving inverse problems using diffusion models without retraining for each new measurement type. This approach decomposes the posterior p ( X | y ) into a pre-trained prior p ( X ) and an adaptable likelihood term p ( y | X ) . During sampling, the intermediate predictions are adjusted using gradient updates from the likelihood term.

Recent works applying DPS to 3D reconstruction include GSD [22], which uses DPS with Gaussian Splatting for view-guided 3D generation. However, these methods rely on heuristic, manually-tuned step sizes for the likelihood update, which often requires careful calibration for each task and can lead to suboptimal convergence or reconstruction quality.

Sampling

Phase

FCM

Likelihood Update

Diffusion Model

𝜖𝜃 (𝐗

𝑡

)

𝑿𝑡

𝐗

: Point Cloud

𝐲

ℛ(⋅)

:Rendering Operator

FCM

෩

𝐗0|𝑡

𝐗𝑡-1

𝐲

DDIM

sampling

෡

𝐗0|𝑡

ℛ(𝐗)

𝐲

-

2

ℒ(𝐗) =

𝛿 = 𝛿0

෩

𝐗0|𝑡

2

𝒈𝑘

2

𝒈𝑘 = ∇ℒ ෩

𝐗0|𝑡

𝑘

𝒈′𝑘

= ∇ℒ( ෩

𝐗0|𝑡

𝑘

-𝛿𝒈𝑘)

𝛼𝑘

𝑟𝑎𝑤 =

𝒈𝑘

2

2

𝒈𝑘

⊤ 𝒉 𝑘 +𝜖

෩

𝐗0|𝑡

𝑘+1

= ෩

𝐗0|𝑡

𝑘

-𝛼𝑘

𝒈𝑘

෩

𝐗0|𝑡

𝑘+1

= ෩

𝐗0|𝑡

𝑘

- 𝛼𝑘

2

𝒈𝑘

If,

ℒ

(

෩

𝐗0|𝑡

𝑘+1

)>

ℒ ෩

𝐗0|𝑡

-𝜂𝐹𝐶𝑀𝛼𝑘𝒈𝑘

𝒈𝑘+1 = ∇ℒ

(

෩

𝐗0|𝑡

𝑘+1

)

𝑘 ← 𝑘 + 1

𝒉𝑘 =

𝒈𝑘 - 𝒈′𝑘

𝛿

Else,

Adaptive Step Size Methods in Optimization. Our FCM approach has roots in several foundational optimization techniques while introducing novel algorithmic elements. In numerical optimization, determining appropriate step sizes is a well-studied challenge with various classical solutions. Quasi-Newton methods [25] approximate the Hessian using rank-one or rank-two updates (e.g., BFGS, L-BFGS [18]), but require matrix storage and operations. Barzilai-Borwein (BB) methods [4] provide scalar approximations to the secant equation using the differences of consecutive iterates and gradients. Line search techniques with Armijo [2] or Wolfe conditions [33] ensure sufficient descent but typically involve multiple function evaluations.

Building on these foundations, FCM introduces several innovations specifically for diffusion-based 3D reconstruction: (1) a scale-adaptive curvature probe ( δ k = δ 0 · ∥ x k ∥ ∥ g k ∥ ) that automatically calibrates to the geometry of point clouds and gradient magnitudes, (2) a forward-difference directional curvature estimate that requires no adjoint operations of the renderer-critical for complex neural renderers where adjoint computation is intractable, (3) a robust BB-inspired step-size computation combined with principled capping that offers theoretical guarantees, and (4) a "once-only" Armijo check.

3 Method

Our goal is to perform high-quality, flexible 3D reconstruction by decomposing the posterior distribution p ( X | y ) into a learned prior p θ ( X ) and a likelihood update p ( y | X ) that does not require separate training. We train only the score of the prior ∇ log p θ ( X ) on unlabeled 3D data. Then, at inference, we incorporate the measurement information (e.g., single-view or multi-view images, depth maps) through an adaptive Forward Curvature-Matching (FCM) update, which approximates ∇ log p ( y | X ) .

Any forward operator R (e.g., a differentiable renderer for images or a map from 3D to depth measurements) can be plugged in to guide the generation of point clouds via the same trained diffusion prior. This design separates the learned model from the measurement modality, eliminating the need for retraining whenever the measurement operator changes. In this section, we detail our method in four parts. First, we describe how we train the diffusion model ∇ log p θ ( X ) . Next, we present our differentiable renderer R for the image-based scenario. We then introduce the FCM-based likelihood update, highlighting why FCM is needed in non-linear settings and how step sizes are optimally determined through a principled approach. Finally, we extend the method to the multi-view setting.

3.1 Diffusion Prior for Point Clouds

We begin by training a diffusion model p θ ( X ) on a large dataset of colored point clouds. Following the standard DDPM [11] framework, we define a forward diffusion process that corrupts a clean point cloud X 0 into X T with Gaussian noise over T timesteps. The reverse process is modeled by a neural network that estimates the noise at each timestep. Formally, in the forward process:

where β t is a variance schedule. This process can be written in closed form from X 0 :

with ¯ α t = ∏ t s =1 (1 -β s ) . The reverse process approximates p θ ( X t -1 | X t ) via a learned Gaussian:

During training, we minimize the simplified loss:

where ϵ ∼ N ( 0 , I ) .

Once trained, we use the DDIM sampler [29] for inference, generating point clouds from noise in fewer steps. Let ϵ ( t ) θ ( X t ) be the noise estimate at step t . Then the DDIM update from X t to X t -1 is:

where and

is a variance term controlling the sampling stochasticity. This DDIM sampler, combined with our trained model, provides a 3D prior that can generate plausible point clouds.

3.2 Differentiable Renderer as the Measurement Operator

Our method only requires that R be differentiable, so both R ( X ) and its gradient ∇ X ∥ y -R ( X ) ∥ 2 can be computed. In this section we introduce a forward operator R that projects a point cloud X into 2D measurements.

A point cloud X comprises points { ( x i , y i , z i , f i ) } , where ( x i , y i , z i ) are 3D coordinates and f i includes attributes such as color. Each point is projected onto the 2D image plane using known camera parameters. At each pixel ( u, v ) , R collects the K points with the smallest depth values z i (i.e., the nearest points along the viewing direction) and blends their colors via alpha compositing:

Here, the opacity α i is computed from the image space footprint as

where r is the radius of the rasterizer and ρ i is the Euclidean distance between the center of the pixel and the projected position of the point in the image space. The product term ∏ i -1 j =1 (1 -α j ) ensures that closer points dominate the final color, while partially occluded points contribute less. Repeating this calculation for each pixel ( u, v ) yields a 2D image matching the resolution of y .

In addition to color-based rendering, R can produce a depth map by applying inverse-square weighting to each point's distance. At each pixel ( u, v ) , the depth is computed from the same set of K nearest points:

Reference

Image

BDM

PC²

Ours

Ground

Truth

Ground

Truth

(Novel View)

PC²

(Novel View)

BDM

(Novel View)

Ours

(Novel View)

1

so that points closer to the camera have a larger influence on the final depth. Repeating this process for each pixel yields a 2D depth map matching the resolution of y .

Because we do not learn a dedicated score function for ∇ log p ( y | X ) , different operators R can be swapped in with minimal effort. If y is a single-view image, then R = R color with a single camera. For multi-view input, each view is rendered separately and their pixel or feature errors are averaged, as described in Section 3.4. If y is a depth map, then R = R depth from Eq. (8).

3.3 Likelihood Update via Forward Curvature-Matching

In standard diffusion posterior sampling (DPS) [6], one iteratively updates the current sample X t with a term proportional to the gradient ∇ X log p ( y | X ) . However, for complex, non-linear forward operators R , determining an appropriate step size is non-trivial. Previous approaches resort to heuristics [6] or empirically tuned factors [22] to balance the data fidelity term with the learned diffusion prior. While this can be effective, it may hamper convergence speed or degrade reconstruction quality if not carefully tuned.

To address these limitations, we propose Forward Curvature-Matching (FCM), a novel algorithm designed specifically for diffusion-based 3D reconstruction. The development of FCM was guided by key requirements: working without adjoint operations (intractable for neural renderers), maintaining predictable computational cost, and using universal parameters across different reconstruction tasks.

Our approach relies on a key insight: we can estimate curvature information through a scaled directional probe without requiring full Hessian approximations [25]. For the measurement loss L ( x ) = ∥ y -R ( x ) ∥ 2 , given the current estimate x k and gradient g k = ∇L ( x k ) , we compute:

Reference

View

Novel

View

Reference

View

Novel

View

Ground

Truth

PC²

Ours

Image

Image

Ground

Truth

PC²

Ours

This h k approximates ∇ 2 L ( x k ) · g k along the gradient direction. The scale-adaptive probe ( δ k ) automatically calibrates to the geometry of the point cloud, a crucial advantage over traditional finite-difference approaches [7].

We then compute a Barzilai-Borwein-inspired [4] step size, modified for robustness:

where ε = 10 -12 and L is the Lipschitz constant of ∇L . The capping mechanism ensures stability while maintaining theoretical guarantees. Unlike classical line searches that require multiple function evaluations [25], we incorporate a single Armijo check [2]: if L ( x k -α k g k ) > L ( x k ) -η FCM · α k · ∥ g k ∥ 2 , we halve α k once and accept.

This design yields a fixed computational cost of exactly two backward and three forward passes per step-significantly more efficient than traditional optimization methods like L-BFGS [18] or Wolfe line searches [33] with unpredictable evaluation counts.

3.3.1 Theoretical Guarantees

Our approach is built on the following assumptions, which are typically satisfied in the context of 3D reconstruction:

Assumption 3.1 (Smoothness) . L is L -smooth: ∥∇L ( u ) -∇L ( v ) ∥ ≤ L · ∥ u -v ∥ .

Assumption 3.2 (Lower bound) . L inf := inf x L ( x ) > -∞ .

Assumption 3.3 (Local convexity) . L is convex on the set of iterates (which is typically small or "benign" in practice).

This approach provides theoretical guarantees on convergence and optimality, as captured in the following theorem:

Theorem 3.4 (Guaranteed Loss Decrease) . Let c = min { η FCM 2 L , 1 8 L } . Our FCM algorithm ensures:

When integrated into the DDIM sampling process, FCM preserves the contraction properties of diffusion models:

Reference

Ours

Ground

Truth

(Novel view)

Ours

Depth

Ground

Truth

(Novel View)

5 Views

1 View

3 Views

Proposition 3.5 (Contraction Preservation) . Under Assumptions 3.1-3.3 and α k ≤ 1 /L , the combined DDIM+FCM operator remains a contraction in expectation, thus preserving the diffusion contraction property.

Our FCM method uses fixed constant η FCM = 10 -4 for all tasks, this principled approach leads to faster convergence and higher-quality reconstructions compared to methods that rely on heuristic step sizes. Detailed proofs and additional theoretical analysis are provided in the Appendix.

3.4 Multi-View Reconstruction

The same FCM-based likelihood update extends naturally to multi-view reconstruction. Suppose we have N images { y i } N i =1 with known camera parameters. We define

where R i is the differentiable renderer for the i -th viewpoint. The gradient ∇ X L MV ( X ) can be used in Algorithm 1 (replacing the single-view line ∥ y - R ( · ) ∥ with the multi-view average). As the number of views grows, reconstruction quality improves, yet the diffusion prior remains the same, illustrating the modality-agnostic nature of our approach.

By training only the diffusion prior on unlabeled 3D shapes and introducing an FCM-based likelihood update with an arbitrary forward operator R , we achieve a flexible, adaptive 3D reconstruction pipeline. The FCM approach ensures stable and fast convergence even with non-linear rendering operators, outperforming fixed-step DPS approaches.

4 Experiments

We evaluate the reconstructed point clouds using three different metrics: Earth Mover's Distance (EMD), L-1 Chamfer Distance (CD), and F-score at a threshold of 0.01. Details of the implementation are provided in the appendix.

ShapeNet. In our method, colors are essential during the rendering process. However, sampling colored point clouds from mesh-based objects is a challenging task. To address this, we train our model using the dataset provided by KeypointNet [37]. The color information in the KeypointNet point cloud does not correspond to the actual mesh color in ShapeNet. Instead, the model assigns colors according to object parts.

We perform our evaluation using the categories { airplane, car, chair, table } from the ShapeNet rendered image dataset [35].

900

Ours(NFE=256)

800

DDIM+DPS(y=0.04)

DDIM+DPS(y=0.05)

700

DDIM+DPS(y=0.06)

norm

DDPM+DPS(y=0.04)

600

2

DDPM+DPS(y=0.05)

Difference

500

DDPM+DPS(y=0.06)

400

300

200

1.0

0.8

0.6

0.4

0.2

0.0

Timestep

CO3D. The CO3D dataset is a large-scale collection of real-world multi-view images from common object categories. It provides a colored point cloud obtained using COLMAP from multi-view images, which is then used for model training and evaluation. We perform our evaluation using the categories hydrant and teddybear from the CO3D dataset.

4.1 Quantitative Results

We evaluate the performance of reconstruction in the ShapeNet dataset. In Tab. 1, our method is compared with PC 2 [20] and BDM [34]. In the original paper, BDM is evaluated using 4,096 points, whereas PC 2 is evaluated using 8,192 points. In this work, we adopt the evaluation approach of PC 2 for quantitative experiments. For BDM, we adopted the blending method that achieved the best results in their study and used PC 2 as the reconstruction model. Our method achieves the best results in all metrics. In this experiment, we ensured that the NFEs for all other models were set similarly for a fair comparison. Detailed comparisons with the settings proposed by their studies and quantitative results on CO3D are provided in the appendix.

4.2 Qualitative Results

In Fig. 3 we show the reconstructed point clouds of different models using the ShapeNet dataset. In Fig. 4 we present a comparison of the rendered results of reconstructed colored point clouds using the CO3D dataset. Other models fail to accurately follow the given image in their rendering results for the reference view, instead focusing on generating a plausible object within the learned category. However, our method achieves the highest level of detail for the reference image.

4.3 Adaptivity Analysis

Our method has the advantage of performing various tasks without requiring retraining of the model. In this section, we demonstrate this capability through multi-view reconstruction and depth map reconstruction. The models used in this section are the same as those used in the previous section for the ShapeNet dataset. Fig. 6 and Tab. 2 illustrate the effectiveness of our method in multi-view reconstruction. As the number of views increases, the generated point cloud becomes more refined, demonstrating the improved quality of reconstruction. Fig. 5 presents the results of applying our method to depth maps rendered using Eq. 8. The results show high fidelity to the reference depth map and the ability to generate natural-looking objects.

4.4 Ablation Study

To show the effectiveness of our method, we compare with other DPS-based methods. Fig. 7 and Tab. 3 compare our method with DPS-based approaches. Fig. 7 presents the plot of the difference

in L2 norm between the reference image and the rendered image during the sampling process over timesteps. We observed that both DDPM+DPS and DDIM+DPS methods achieve their best performance at the step size of 0.05. The reason DPS-based methods struggle to follow the reference image is that they update with a fixed step size, leading to suboptimal convergence. It demonstrates that our method converges more optimally compared to other approaches. As shown in Tab. 3, our method achieves the best point cloud reconstruction performance. Since the DDPM sampling process does not approximate ˆ X 0 , and the iterative FCM updates from noisy X t using measurement y are not ideal, we exclude the DDPM+FCM scheme from our comparison. Qualitative comparisons with DPS-based methods are provided in the appendix.

5 Conclusion

In this paper, we proposed the novel point cloud diffusion sampling approach for adaptive 3D reconstruction. Our method reconstructs the colored point cloud by updating it using likelihood ∇ log p ( y | X ) with given images through FCM during the reverse process of the point cloud diffusion model. In our experiments, we qualitatively demonstrate high-fidelity reconstruction of reference images with color, generating high-quality point cloud structures compared to prior works. Moreover, we quantitatively surpass previous works in point cloud reconstruction performance. Our method is applicable to various tasks, demonstrating its versatility. Additionally, it can be extended to different domains (e.g., Gaussian Splatting, meshes, etc.), highlighting its adaptability. As future work, we are interested in exploring larger datasets across diverse domains.

References

Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J. Guibas. Learning representations and generative models for 3D point clouds. In International Conference on Machine Learning , 2017.

Larry Armijo. Minimization of functions having Lipschitz continuous first partial derivatives. Pacific Journal of Mathematics , 16:1-3, 1966.

Mohammad Arshad and William J. Beksi. LIST: Learning implicitly from spatial transformers for single-view 3D reconstruction. In International Conference on Computer Vision , 2023.

Jonathan Barzilai and Jonathan Michael Borwein. Two-point step size gradient methods. Ima Journal of Numerical Analysis , 8:141-148, 1988.

Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Haoran Su. Single-stage diffusion NeRF: A unified approach to 3D generation and reconstruction. In International Conference on Computer Vision , 2023.

Hyungjin Chung, Jeongsol Kim, Michael T. McCann, Marc Louis Klasky, and J. C. Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference on Learning Representations , 2023.

John E. Dennis and Bobby Schnabel. Numerical methods for unconstrained optimization and nonlinear equations , volume 16 of Classics in Applied Mathematics . SIAM, 1996.

Yu Feng, Xing Shi, Mengli Cheng, and Yun Xiong. DiffPoint: Single and multi-view point cloud reconstruction with ViT based diffusion model. ArXiv , abs/2402.11241, 2024.

Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. A PapierMache approach to learning 3D surface generation. In Conference on Computer Vision and Pattern Recognition , 2018.

Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M. Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. NerfDiff: Single-image view synthesis with NeRF-guided distillation from 3D-aware diffusion. In International Conference on Machine Learning , 2023.

Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , 2020.

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3D. In International Conference on Learning Representations , 2024.

Tao Hu, Liwei Wang, Xiaogang Xu, Shu Liu, and Jiaya Jia. Self-supervised 3D mesh reconstruction from single images. In Conference on Computer Vision and Pattern Recognition , 2021.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems , 2022.

Vladimir Vladimirovich Kniaz, Vladimir Alexandrovich Knyaz, Fabio Remondino, A.N. Bordodymov, and Petr Moshkantsev. Image-to-voxel model translation for 3D scene reconstruction and segmentation. In European Conference on Computer Vision , 2020.

Jae Joong Lee and Bedrich Benes. RGB2Point: 3D point cloud generation from single RGB images. In Winter Conference on Applications of Computer Vision , 2025.

Yukun Li and Li-Ping Liu. Enhancing diffusion-based point cloud generation with smoothness constraint. ArXiv , abs/2404.02396, 2024.

Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming , 45:503-528, 1989.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3D point cloud generation. In Conference on Computer Vision and Pattern Recognition , 2021.

Luke Melas-Kyriazi, C. Rupprecht, and Andrea Vedaldi. PC 2 : Projection-conditioned point cloud diffusion for single-image 3D reconstruction. In Conference on Computer Vision and Pattern Recognition , 2023.

Shentong Mo, Enze Xie, Ruihang Chu, Lewei Yao, Lanqing Hong, Matthias Nießner, and Zhenguo Li. DiT-3D: Exploring plain diffusion transformers for 3D shape generation. In Advances in Neural Information Processing Systems , 2023.

Yuxuan Mu, Xinxin Zuo, Chuan Guo, Yilin Wang, Juwei Lu, Xiaofeng Wu, Songcen Xu, Peng Dai, Youliang Yan, and Li Cheng. GSD: View-guided Gaussian splatting diffusion for 3D reconstruction. In European Conference on Computer Vision , 2024.

Norman Muller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. DiffRF: Rendering-guided 3D radiance field diffusion. In Conference on Computer Vision and Pattern Recognition , 2023.

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-E: A system for generating 3D point clouds from complex prompts. ArXiv , abs/2212.08751, 2022.

Jorge Nocedal and Stephen J. Wright. Numerical optimization. Springer Series in Operations Research and Financial Engineering , 2006.

Nikhila Ravi, Jeremy Reizenstein, David Novotný, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D deep learning with PyTorch3D. In SIGGRAPH Asia , 2020.

Yi Rong, Haoran Zhou, Kang Xia, Cheng Mei, Jiahao Wang, and Tong Lu. RepKPU: Point cloud upsampling with kernel point representation and deformation. In Conference on Computer Vision and Pattern Recognition , 2024.

Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3D point cloud generative adversarial network based on tree structured graph convolutions. In International Conference on Computer Vision , 2019.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Stanislaw Szymanowicz, C. Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3D reconstruction. In Conference on Computer Vision and Pattern Recognition , 2024.

Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: Large multi-view Gaussian model for high-resolution 3D content creation. In European Conference on Computer Vision , 2024.

Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, W. Liu, and Yu-Gang Jiang. Pixel2Mesh: Generating 3D mesh models from single RGB images. In European Conference on Computer Vision , 2018.

Philip Wolfe. Convergence conditions for ascent methods. II: Some corrections. SIAM Review , 13:185-188, 1971.

Haiyang Xu, Yu Lei, Zeyuan Chen, Xiang Zhang, Yue Zhao, Yilin Wang, and Zhuowen Tu. Bayesian diffusion models for 3D shape reconstruction. In Conference on Computer Vision and Pattern Recognition , 2024.

Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomír Mˇ ech, and Ulrich Neumann. DISN: Deep implicit surface network for high-quality single-view 3D reconstruction. In Advances in Neural Information Processing Systems , 2019.

Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge J. Belongie, and Bharath Hariharan. PointFlow: 3D point cloud generation with continuous normalizing flows. In International Conference on Computer Vision , 2019.

Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, Liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. KeypointNet: A large-scale 3D keypoint dataset aggregated from numerous human annotations. In Conference on Computer Vision and Pattern Recognition , 2020.

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: Latent point diffusion models for 3D shape generation. In Advances in Neural Information Processing Systems , 2022.

Linqi Zhou, Yilun Du, and Jiajun Wu. 3D shape generation and completion through point-voxel diffusion. In International Conference on Computer Vision , 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claims regarding the novel approach to 3D reconstruction using diffusion priors and FCM-based likelihood updates, which are validated by the experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section A.5 discusses limitations including sensitivity to thin structures and lack of colored point clouds in ShapeNet, as well as trade-offs between CD and shapepreserving metrics like F-score.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Appendix A.3 states all assumptions (e.g., L-smoothness, local convexity/monotone gradient map, and step-size bounds tied to BB+Armijo) and provides complete, self-contained proofs (lemmas → main theorem) establishing descent/contraction of the FCM update.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper specifies the datasets used (ShapeNet and CO3D), the evaluation metrics (EMD, CD, and F-score), and the implementation details. It also provides information on how the baseline models were evaluated for a fair comparison.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The abstract provides a GitHub link for the code: https://github.com/Seunghyeok0715/FCM .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Training and test details are provided in Section A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper presents quantitative results in tables, but it does not include error bars or statistical significance tests. The paper provides quantitative results using EMD, CD, and F-score, but does not include error bars or statistical significance tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper reports inference time per sample (Table. 6), and Section A.2 specifies the use of NVIDIA RTX 6000 Ada Generation GPU and batch size. Although memory is not explicitly reported, the provided information is sufficient for reproduction.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper does not describe any deviations from the NeurIPS Code of Ethics. The research appears to be aligned with ethical guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The paper does not explicitly discuss the broader societal impacts of the work. The paper focuses on the technical aspects of the method and its performance.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not discuss the release of data or models that would require specific safeguards. The paper introduces a novel method for 3D reconstruction, but does not discuss releasing models or datasets that would require safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cites the original papers for the datasets and models used, and it appears to properly credit the original owners of the assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new datasets or models. The paper focuses on a novel method for 3D reconstruction.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects. The research is focused on 3D reconstruction using existing datasets.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects, so IRB approval is not applicable. The research is focused on 3D reconstruction using existing datasets.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper does not utilize LLMs as a core component of its methodology. The paper focuses on a novel method for 3D reconstruction.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/ LLM ) for what should or should not be described.

A.1 Algorithm

A.2 Implementation Details

To model the reverse process p θ , we employed a Diffusion Transformer, originally introduced in Point-E's unconditional model [24], as the neural network parameterized by θ , which predicts both µ θ and Σ θ . All images were set to a resolution of 224×224. For the ShapeNet dataset, 2,048 points were sampled and then upsampled to 8,192 points for comparison [27]. In the case of CO3D, 8,192 points were directly sampled. To sufficiently refine the point cloud, we perform four FCM updates per DDIM sampling step. We set the hyperparameters as follows: η FCM = 10 -4 , L = 2 / 3 . For ShapeNet, we set T = 256 and δ 0 = 2 × 10 -2 . For CO3D, we set T = 512 and δ 0 = 6 × 10 -3 . We use point cloud rendering processes provided by PyTorch3D [26]. For ShapeNet, we set the radius of the point cloud rasterizer to 0.018 for airplane category and 0.027 for the other categories. For CO3D, we set the radius to 0.013. All experiments were performed using an NVIDIA RTX 6000 Ada Generation with a batch size of 16.

A.3 Theoretical Analysis of FCM

In this appendix, we provide detailed proofs for the theoretical guarantees of our Forward CurvatureMatching (FCM) method. We begin by formally establishing the properties of FCM step sizes, followed by proofs of loss decrease and convergence guarantees. Finally, we analyze how FCM integrates with DDIM sampling while preserving its contraction properties.

A.3.1 Bounds on FCM Step Sizes

We first establish that the FCM step size is guaranteed to lie within a well-behaved range, ensuring stable iterations. Our FCM approach relies on a directional curvature estimate:

This h k approximates the directional curvature along the gradient direction. Specifically, h k is an approximation of the Hessian-vector product ∇ 2 L ( x k ) g k .

Lemma A.3.1 (Step Size Bounds) . Assume ε ≤ L ∥ g k ∥ 2 in the calculation of α raw k . Then the FCM step size α k (prior to any Armijo halving) satisfies:

Proof. From the finite difference approximation with h k = ( g k -g ′ k ) /δ k , we analyze ⟨ g k , h k ⟩ :

Under Assumption 3.1 ( L -smoothness), we can establish that:

where the last inequality holds given our assumption that ε ≤ L ∥ g k ∥ 2 . This implies:

Since we cap α k = min { α raw k , 1 /L } , we ensure α k ≤ 1 /L while maintaining the lower bound α k ≥ 1 / (2 L ) .

Remark A.3.2. If ∥ g k ∥ ≈ 0 , the raw step size α raw k could become very large. However, in such cases, the Armijo condition will catch insufficient decrease and halve the step size once, still ensuring stable updates.

This implies:

Therefore:

A.3.2 Firm Non-Expansiveness of the Gradient Step

Next, we establish that a gradient step with the FCM step size is firmly non-expansive, which is crucial for integrating with the diffusion process.

Lemma A.3.3 (Firmly Non-Expansive Gradient Step) . Let T k ( u ) = u -α k ∇L ( u ) be the gradient step operator with fixed α k > 0 . Under Assumptions 3.1-3.3, if 0 < α k < 2 /L , then T k is firmly non-expansive:

Hence, T k is in particular non-expansive: ∥ T k ( u ) -T k ( v ) ∥ ≤ ∥ u -v ∥ .

Proof. Let ∆ = u -v and ∆ g = ∇L ( u ) -∇L ( v ) . Then:

By the Baillon-Haddad theorem (which applies when L is convex and L -smooth), ∇L is 1 /L -cocoercive, meaning:

Substituting this into our expression:

Since 0 < α k ≤ 1 /L in our FCM algorithm (as established in Lemma A.3.1), the factor 2 L -α k > 0 . Thus, T k is firmly non-expansive, and consequently, ∥ T k ( u ) -T k ( v ) ∥ ≤ ∥ u -v ∥ .

A.3.3 Guaranteed Loss Decrease

We now prove Theorem 3.4 from the main paper, which guarantees that FCM decreases the measurement loss at each iteration.

Theorem A.3.4 (Guaranteed Loss Decrease) . Let c = min { η FCM 2 L , 1 8 L } . The FCM algorithm ensures:

Proof. We consider two cases:

Case 1 (Armijo condition satisfied): When the initial step satisfies the Armijo condition, we have:

where we used the lower bound α k ≥ 1 2 L from Lemma A.3.1.

Case 2 (Armijo halving required): If the initial step fails the Armijo condition and we halve α k , then α k ≥ 1 4 L remains. By the descent lemma for L -smooth functions:

Since α k ≤ 1 2 L after halving, we have 1 -Lα k 2 ≥ 1 2 . Combined with α k ≥ 1 4 L , this gives:

Taking the minimum of the guaranteed decrease in both cases, we get:

Corollary A.3.5 (Gradient Norm Convergence) . Under FCM iterations, ∥∇L ( x k ) ∥ → 0 as k →∞ , and every cluster point is stationary.

Proof. By Theorem A.3.4 and Assumption 3.2 (lower bound on L ), we have:

Since c > 0 , we must have ∑ ∞ k =0 ∥∇L ( x k ) ∥ 2 < ∞ , which implies ∥∇L ( x k ) ∥ → 0 as k → ∞ . This means that every cluster point of the sequence { x k } is a stationary point of L .

A.3.4 FCMIntegration with DDIM

Finally, we analyze how FCM integrates with the DDIM sampling process and prove Proposition 3.5 from the main paper.

Proposition A.3.6 (Contraction Preservation) . Let Φ t be a DDIM step that is ρ -contractive (with ρ < 1 ) in mean-square sense:

Define Ψ t ( u ) = T k (Φ t ( u )) , where T k ( u ) = u -α k ∇L ( u ) . Under Assumptions 3.1-3.3 and α k ≤ 1 /L , Ψ t is also ρ -contractive in expectation, thus preserving the diffusion contraction property.

Proof. From Lemma A.3.3, we know that T k is non-expansive: ∥ T k ( a ) -T k ( b ) ∥ 2 ≤ ∥ a -b ∥ 2 . Therefore, for any u , v :

Thus, Ψ t remains a ρ -contraction in mean-square sense.

A.3.5 Robustness to Non-Convexity

While Assumption 3.3 (local convexity) is used in our theoretical analysis, FCM shows empirical robustness even when this assumption is violated.

Remark A.3.7 (Behavior Under Non-Convexity) . If local convexity fails, the firm non-expansiveness of T k may break. However, Theorem A.3.4 and Corollary A.3.5 remain valid, guaranteeing that the FCM step decreases L and drives ∥∇L ( x k ) ∥ → 0 . This makes FCM robust in practice even for non-convex L .

A.3.6 Practical Parameter Settings

Over-estimating L in the algorithm only tightens the cap α k ≤ 1 /L and preserves all theoretical guarantees. Under-estimating L triggers the single Armijo halving, which prevents divergence while maintaining efficiency.

This combination of theoretical guarantees and practical robustness makes FCM an ideal choice for likelihood updates in diffusion-based 3D reconstruction, enabling high-quality results.

Ground

Truth

(Novel View)

5 Views

1 View

3 Views

Image

(Novel View)

A.4 Additional Experiments

Comparison with other methods proposed in their original papers. Tab. 4 shows a comparison of different models evaluated on the ShapeNet dataset, where each model is applied according to the method originally proposed in its respective study. PC 2 uses an NFEs of 1000, while BDM uses a total of 1080 NFEs-1000 for its reconstruction model and an additional 80 for its prior model.

Table 6: Time efficiency analysis of different methods. We report average scores of ShapeNet dataset and total sampling time (seconds per sample). The highest scores are marked in bold , and the second-highest scores are underlined. Our method achieves higher reconstruction accuracy with respect to F-score and EMD than prior approaches while maintaining comparable inference speed.

Table 8: Runtime breakdown. Left: component costs and NFE counts for each method (ours: 256 NFEs; PC 2 : 1000; BDM: 1080 = 1000 + 80). Right: cost of a single Armijo check (ours) versus a strong Wolfe line search. Our once-only Armijo strategy is substantially cheaper while preserving reliable descent, contributing to the overall speedup.

Despite using fewer NFEs, our method still achieves the highest scores in terms of both EMD and F-score.

More experiments on CO3D dataset. Tab. 5 presents a comparison with PC 2 on the CO3D dataset and additionally provides scores for the multi-view setting. These results demonstrate the improved performance of our method over existing approaches. Furthermore, Fig. 8 illustrates the qualitative results of multi-view reconstruction on CO3D. To demonstrate broader applicability, Fig. 10 presents qualitative comparisons on two additional CO3D categories-remote and vase-against PC 2 .

Hyperparameter Experiments. As shown in Tab. 7, the sampler behaves robustly once each knob is kept within a reasonable range. In particular, as discussed in A.3.6, an overly conservative choice of 1 /L (i.e., taking L too large) activates the clamping in Eq. 11, so the effective update becomes overly damped and Armijo progress can stall, leading to slow or failed convergence. Nevertheless, the table shows substantial tolerance: the sampler remains reliable for L ∈ [2 / 3 , 10] .

Time analysis. As shown in Tab. 6, our method achieves the best performance in both F-score and EMD score, highlighting its effectiveness even with a 32.1% reduction in runtime. As shown in Tab. 8, our once-only Armijo check is dramatically cheaper than a conventional strong Wolfe line search. While a single step in PC 2 or BDM can be faster than our FCM step, our sampler deliberately opts for far fewer steps: 256 NFEs versus 1000 (PC 2 ) and 1080 (BDM). This NFE gap dominates the end-to-end runtime-reducing the number of denoiser/forward evaluations-and leads to overall faster and more efficient reconstructions, without sacrificing accuracy.

Qualitative comparison with DPS-based methods. Fig. 9 shows a comparison between our method and different methods. Here, the step size for the DPS-based method is set to the optimal value of γ = 0 . 05 as identified in Fig. 7. While the DPS-based method captures the overall shape reasonably well, it fails to recover accurate color information. This limitation is attributed to the use of a fixed step size, which leads to suboptimal updates. In contrast, our method produces results that are more optimal with respect to the given measurements.

More examples and failure cases. Additional qualitative results are shown in Figs. 11- 18, and failure cases of our method are presented in Figs. 19- 21. Most failure cases occur when the object has a complex structure, which can be attributed to the diffusion prior being misled by unfamiliar or uncommon data distributions.

Ground

Truth

Ours

DDPM+DPS

DDIM+DPS

Reference

View

Novel

View

Reference

View

Novel

View

A.5 Limitations

While the rendered image may resemble the reference image, the structure of the point cloud appears slightly thinner than the ground truth point cloud due to the radius of the rasterizer. This effect is particularly noticeable in thin structures, such as the legs of a chair. Additionally, due to the unavailability of a colored point cloud dataset for ShapeNet, we used a dataset generated by KeypointNet. However, since KeypointNet does not assign the actual mesh colors, this may lead to a degradation in the quality of the reconstruction.

Direct control of point cloud positions for likelihood updates might limit CD metric performance compared to reconstructions using only the learned prior(or posterior). However, considering the objective of our task 'image(s) to 3D reconstruction", metrics such as F-score and EMD, which measure the shape similarity, are more aligned with the task's purpose than measuring the distance between individual points.

Reference

View

Novel

View

Reference

View

Novel

View

Ground

Truth

PC²

Ours

Image

Image

Ground

Truth

PC²

Ours

Generation Process

Reconstruction

Generation Process

Reconstruction

7

Reference

Image

Reference

View

Novel

Views

K

Reference

Image

Reference

View

Novel

Views

Reference

Image

Reference

View

Novel

Views

Reference

Image

Reference

View

Novel

Views

MAMI

Y

1

Reference

Image

Reference

View

Novel

Views

Reference

Image

Reference

View

Novel

Views

区

Reference

Image

Reference

View

Novel

Views

R

Reference

Image

Reference

View

Novel

Views

1

Reference

Image

Reference

View

Novel

Views

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses point cloud reconstruction from one or multiple view images.\nThe basic approach involves a diffusion-based model that uses rendering as the operator to transform a noisy point cloud into a 2D representation.\nThe authors present a new approach for adaptive step size, which aims to improve point cloud reconstruction quality and lead to a faster process.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The reconstruction of the reference view demonstrates strong performance compared to baseline methods.\n2. The proposed approach for approximating the Hessian without expensive computations is noteworthy.\n\nWeaknesses:\n1. A key observation is that the presented method does not explicitly leverage the 3D nature of the problem. The adaptive step size mechanism appears applicable to any gradient-based optimization problem, not specific to 3D reconstruction.\n2. The real-world reconstruction of novel views remains somewhat limited, showing only marginal improvement over baseline methods.\n3. The amount of points in the reconstructed point cloud is fixed, so i assume that more points are assigned to the reference view and less to the novel view.\n\nQUESTIONS:\n1. Can you present the full diffusion step of your method?\n2. Why did BDM reconstruct entirely different chair legs and table? Even when the reference image is completely different, and the EMD and CD also present better scores?\n3. The plane seems to be worse with your approach from the novel's view, why?\n4. The step size is the minimum in equation 10, did you noticed objects that did not converged because of too low step size?\n\nLIMITATIONS:\n1. The point cloud reconstruction appears to prioritize satisfying the reference view, potentially at the expense of the novel views. This results in a strong reconstruction of the reference view but weaker results for novel views.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-10"}

comments: SUMMARY:
The paper focuses on reconstructing point clouds from images. Previous methods are limited to fixed-view inputs and typically require retraining the network when switching to a different input modality (e.g., from RGB images to depth maps). To address these issues, the authors propose decomposing ***p(X∣y)*** into a trainable prior ***p(X)*** and an updatable likelihood ***p(y∣X)***. They further introduce a Forward Curvature-Matching (FCM) optimization scheme to guide the update of ***p(y∣X)***. The paper provides a relatively rigorous theoretical derivation to support the proposed framework.

STRENGTHS AND WEAKNESSES:
**Strengths:**

1. The proposed method supports both single-view and multi-view inputs, using either RGB images or depth maps. It is supported by extensive experimental results and effectively broadens the application scope of point cloud diffusion models.
2. The theoretical analysis and mathematical derivations in the paper are rigorous and appear convincing.
3. The experiments in the paper are comprehensive and effectively support the conclusions.
4. The paper is overall well-written and easy to follow.


**Weaknesses:**

1. The proposed method requires a rendering operation at each sampling step during diffusion to correct the optimization direction. This could be more time-consuming compared to baseline methods, and the authors should provide a runtime comparison to support the method's practicality.
2. The point cloud representation does not explicitly include an alpha parameter. It is unclear how the alpha value for each point in Eq. (6) is computed.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces method for reconstructing 3D point clouds from images. The key contributions are:\n\n- To solve inverse problem, they combine prior models with likelihood updates instead of learning posterior distribution directly. Therefore, the proposed method is free to the number of inputs and inputs modalities.\n- Previous approach, which decomposes the posterior into a trainable prior and an updatable likelihood, struggles to determine step sizes for likelihood updates, leading to slow convergence and suboptimal quality. To address these limitations, they propose Forward Curvature-Matching optimization, which computes an adaptive step size.\n\nThe proposed method is evaluated on ShapeNet and CO3D datasets, demonstrating state-of-the-art performance in reconstructing high-quality point clouds without heuristic step size determination.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. They decompose the posterior into a trainable prior and an updatable likelihood, which has several advantages. The model can train without conditioning signals and support single-view or multi-view images and various measurement modalities.\n2. Forward Curvature-Matching optimization determines optimal step sizes, enabling the complex optimization.\n3. The proposed framework might have potential in other domains, which are solving inverse problems with non-linear nature (e.g., Gaussian Splatting, Meshes as mentioned by authors).\n4. The proposed method outperforms fixed-step DPS approaches with varying step sizes, as shown in Fig. 7.\n5. They demonstrate the advantages of their contributions, versatility, in Fig. 5 and Fig. 6.\n\n### Weaknesses\n\nThere is no major weakness of the proposed method. They also mention their limitations.\n\nQUESTIONS:\n1. Are there some reasons to choose point cloud as main representations? The qualitative results compared to previous approaches, which learns the posterior distribution directly, are not superior in Fig. 3 and Fig. 4. The authors already mention that the proposed method can be extended to Gaussian Splatting, and there is baseline method, GSD [23]. If the progress starts from GSD, it is much more clear to demonstrate the effectiveness of FCM compared to GSD. As I understand, there is no specific design for 3D point cloud and the proposed method can be general for diffusion-based methods.\n2. In Tab. 1, the geometric accuracy (EMD and CD) in chair and table is worse than BDM [34]. In appendices, the radius of the rasterizer might affect the performance in chair category. Table category also has similar issue? Or there are more reasons of these performance gap? Additionally, it might be better to mention the performance gap and reasons in the main paper.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method for efficiently and accurately reconstructing 3D point clouds from various types of measurements, such as RGB images and depth maps, by combining a pretrained diffusion prior with Forward Curvature-Matching (FCM)-based adaptive likelihood updates. The approach begins by generating an initial 3D point cloud using DDIM sampling. It then uses a differentiable renderer to compute gradients and curvature of the observation loss, and determines a dynamic step size in a Barzilai–Borwein style, updating the likelihood accordingly. This allows for faster and more stable convergence compared to previous methods that rely on fixed step sizes, and the method can be applied to RGB, depth, and multi-view inputs without retraining the prior model. Experimental results on ShapeNet and CO3D demonstrate superior performance over prior methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts\n1) The paper is well-written and easy to follow.\n2) The use of FCM along with Barzilai–Borwein-style dynamic step size estimation, combined with a one-time Armijo condition check, is both novel and practical. This overcomes the limitations of fixed step sizes in previous DPS methods while allowing adaptation to various modalities (RGB, depth, multi-view) without retraining the prior model.\n3) The modularity of the method—only requiring the renderer to be changed for different modalities—suggests it could be highly useful in practice.\n\nWeaknesses\n1) While the use of a differentiable renderer reduces the need for retraining the prior model, this also means that the overall reconstruction quality heavily depends on the renderer's performance. It is unclear how well the method performs under varying lighting conditions across views in multi-view input scenarios.\n2) On the real-world CO3D dataset, the evaluation is limited to only two categories: 'hydrant' and 'teddy bear'. A broader evaluation across more categories would strengthen the claims.\n3) The proposed method involves several hyperparameters (e.g., $\\eta_{FCM}$, $L_\\text{est}$, $T$, $\\delta_{0}$), but no detailed analysis is provided. Since different hyperparameters are used for different datasets (e.g., $T=256$, $\\delta_{0}=2×10^{-2}$ for ShapeNet; $T=512$, $\\delta_{0}=6×10^{-3}$ for CO3D), a justification or ablation study is needed to explain how these values were chosen and how sensitive the method is to these choices.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}