Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image
Authors: Kailu Wu
Fangfu Liu
Zhihan Cai
Runjie Yan
Hanyang Wang
Yating Hu
Yueqi Duan
Kaisheng Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: image to 3d
3d generation
mesh generation
Primary area: generative_models
TL;DR: A novel high quality and efficient image-to-3D framework.

Abstract: In this work, we introduce Unique3D, a novel image-to-3D framework for efficiently generating high-quality 3D meshes from single-view images, featuring state-of-the-art generation fidelity and strong generalizability. Previous methods based on Score Distillation Sampling (SDS) can produce diversified 3D results by distilling 3D knowledge from large 2D diffusion models, but they usually suffer from long per-case optimization time with inconsistent issues. Recent works address the problem and generate better 3D results either by finetuning a multi-view diffusion model or training a fast feed-forward model. However, they still lack intricate textures and complex geometries due to inconsistency and limited generated resolution. To simultaneously achieve high fidelity, consistency, and efficiency in single image-to-3D, we propose a novel framework Unique3D that includes a multi-view diffusion model with a corresponding normal diffusion model to generate multi-view images with their normal maps, a multi-level upscale process to progressively improve the resolution of generated orthographic multi-views, as well as an instant and consistent mesh reconstruction algorithm called ISOMER, which fully integrates the color and geometric priors into mesh results. Extensive experiments demonstrate that our Unique3D significantly outperforms other image-to-3D baselines in terms of geometric and textural details.

Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image
Kailu Wu Tsinghua University
Fangfu Liu Tsinghua University
Zhihan Cai Tsinghua University
Runjie Yan Tsinghua University
Hanyang Wang Tsinghua University
Yating Hu AVAR Inc.
Yueqi Duan Tsinghua University
Kaisheng Ma Tsinghua University
Unique3 3D
Figure 1: Gallery of Unique3D. from single-view wild images within 30 seconds.
and diverse textured mesh generated by Unique3D
Abstract
In this work, we introduce Unique3D, novel image-to-3D framework for ef- ficiently generating high-quality 3D meshes from single-view images, featuring state-of-thee -ert--rr--h--ttttt art generation fidelity and strong generalizability. Previous methods based on Score Distillation Sampling (SDS) can produce diversified 3D results by distilling 3D knowledge from large 2D diffusion model. but they usually suffer from long per-case optimization time with inconsistent issues. Recent works ad- dress the problem and generate better 3D results either by finetuning multi-view diffusion model or training fast feed-forward model. However, they still lack intricate textures and complex geometries due inconsistency and limited gener-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ated resolution. To simultameously achieve high fidelity, consistency, and efficiency in single image-to-3D, we propose a novel framework Unique3D that includes a multi-view diffusion model with a corresponding normal diffusion model to generate multi-view images with their normal maps, multi-level upscale process to progressively improve the resolution of generated orthographic multi-views, as well as an instant and consistent mesh reconstruction algorithm called ISOMER, which fully integrates the color and geometric priors into mesh results. Exten- sive experiments demonstrate that our Unique3D significantly outperforms other image-to-33 baselines in terms of geometric and textural details. Project page: https: https://wukailu, github. io So/UniqueDD/.
1 Introduction
Automatically generating diverse and high-qualin 3D content from single-view images is fun-
damental task in 3D Computer Vision [13 13
which can facilitate a wide range of versatile applications [25 49], including gaming, architecture, art, and animation. However, this task
60
63
is challenging and ill posed due to the underlying ambiguity of 3D geometry in a single view.
Recently, the rapid development of diffusion models 12
52 39 has opened up new perspectives for
3D content creation. Powered by the strong prior of 2D image diffusion models, DreamFusion [36
proposes Score Distillation Sampling (SDS) to address the limitation of 3D data by distilling 3D
knowledge from 2D diffusions [41], inspiring the progress of SDS-based 2D lifting methods 220
37 [59 24
[]. Despite their diversified compelling results, they usually suffer from long per-case optimization time for hours, poor geometry, and inconsistent issues (e.g., Janus problem (36)), thus not practical for worldw applications. To overcome the problems, a series of works leverage larger- scale open-world 3D datasets 7 | 6 either to fine-tune a multi-view diffusion model [29 28 [5]
and recover the 3D shapes from the generated multi-view images or train large reconstruction model
(LRM) 13 64 60 63 by directly mapping image tokens into 3D representations (e-g.. triplane or
3D Gaussian (6]). However, due local inconsistency in mesh optimization 29
65 and limited
resolution of the generative process with expensive computational overhead 13
63 they struggle to
produce intricate textures and complex geometric details with high resolution.
In this paper, we present a novel image-to-3D framework for efficient 3D mesh generation, coined Unique3D, to address the above challenges and simultancously achieve high-fidelity, consistency, and generalizability. Given an input image, Unique3D first generates orthographic multi-view images from a multi-view diffusion model. Then we introduce multi-level upscale strategy progressively improve the resolution of generated multi-view images with their corresponding normal maps from a normal diffusion model. Finally, we propose an instant and consistent mesh reconstruction (ISOMER) algorithm to reconstruct high-quality 3D meshes from the multiple RGB images and normal maps, which fully integrates the color and geometric priors into mesh results. Both diffusion models trained on a filtered version of the Objaverse dataset [7] with 50k 3D data. To enhance the quality and robustness, we design series of strategies into our framework, including the noise offset channel in the multi-view diffusion training process to correct the discrepancy between training and inference 21). stricter dataset filtering policy, and an expansion regularization to avoid normal collapse in mesh reconstruction. Overall, our method can generate view consistent meshes from single-view wild images within 30 seconds, as shown in Figure
diverse, and multi-
We conduct extensive experiments on various wild 2D) images with different styles. The experiments verify the efficacy of our framework and show that Unique3D outperforms existing methods for high fidelity, geometric details, high resolution, and strong generalizability.
In summary, contributions are:
We propose a novel image-to-3D framework called Unique3D that holistically archives a leading level of high-fidelity, efficiency, and generalizability among current methods.
We introduce a multi-level upscale strategy to progressively generate higher-resolution RGB images with the corresponding normal maps.
We design a novel instant and consistent mesh reconstruction algorithm (ISOMER) to reconstruct 3D meshes with intricate geometric details and texture from RGB images and normal maps.
2

---

Extensive experiments on image-to-3D tasks demonstrate the efficacy and generation fidelity of our method, unlocking new possibilities for real-world deployment in the field of 3D generative AI.
2
Related Work
Mesh Reconstruction. Despite the significant advancements in various 3D representations (e.g., SDF [4], NeRF [2], 3D Gaussian (T6)), meshes remain the most widely used 3D format in popular 3D engines (e.g., Blender, Maya) with mature rendering pipeline. Reconstructing high- quality 3D meshes efficiently from multi-view or single-view images is a daunting task in graphics and 3D computer vision. Early approaches usually adopt laborious and complex photogrammetry pipeline with multiple stages, with techniques like Structure from motion (SfM) 1 42 51, Multi- View Stereo (MVS) 43,, and mesh surface extraction 35 30]. Powered by deep learning and powerful GPUs, recent works [44 45 [4, 67
60 64 have been proposed to pursue higher efficiency and quality with gradient-based mesh optimization or even training large feed-forward reconstruction network. However, their pipeline still suffers from heavy computational costs and struggles to adapt complex geometry. To balance efficiency and quality, we propose novel instant and high-quality mesh reconstruction algorithm in this paper that can reconstruct complex 31) meshes with intricate geometric details from sparse views.
[3
Score Distillation for 3D Generation. Recently, data-driven large-scale 2D diffusion models have 41
achieved notable success in image and video generation 39
66 50] However, transferring it
to 3D generation is non-trivial due to curating large-scale 3D datasets. Pioneering works DreamFu- sion (36] proposes Score Distillation Sampling (SDS) (also known Score Jacobian Chaining (55)) to distill 3D geometry and appearance from pretrained 2D diffusion models when rendered from different viewpoints. The following works continue enhance various aspects such fidelity, prompt alignment, consistency, and further applications (20) 37 59 5
19
54 (99] However, such
optimization-based 2D lifting methods are limited by long per-case optimization time and multi-face
problem [48] due to lack of explicit 3D prior. As Zero 123 (27) proves that Stable Diffusion [39] can be finetuned generate novel views by conditioning on relative camera poses, one-2-3-45 [26
directly produce plausible 3D shapes from generated images in Zero 123. Though it achieves high efficiency, the generated results show poor quality with a lack of texture details and 3D consistency.
Multi-view Diffusion Models for 3D Generation. To achieve efficient and 3D consistent results, 47
48 fine-tune the 2D diffusion models with large-scale 3D data 7 to generate multi-view consistent images and then create 3D contents using sparse view reconstruction. leverages attention layers to produce consistent multi-view color images and then use NeuS (56] for reconstruction. Wonder3I (29) explicitly encodes the geometric information into 3D results and improves quality by cross-domain diffusion. Although these methods generate reasonable results, they are still limited by local inconsistency from multi-views generated by out-domain input images and limited generated resolution from the architecture design, producing coarse results without nighuresolution textures and geometries. In contrast, our method can generate qugher-quality textured 3D meshes with more complex geometric details within just 30 seconds.
some works 29 28
57
For example, SyncDreamer 28
Method
In this section, we introduce our framework, i.e., Unique3D, for high-fidelity, efficient, and gen- eralizable 3D mesh generation from single in-the-wild image. Given an input image, we first generate four orthographic multi-view images with their corresponding normal maps from multi- view diffusion model and a normal diffusion model. Then, we lift them to high-resolution space progressively, (Sec 3.1). Given high-resolution multi-view RGB images and normal maps, we finally reconstruct high-quality 3D meshes with our instant and consistent mesh reconstruction algorithm ISOMER, (Sec 3.2). ISOMER directly handles the case where the global normal of the same vertex is inconsistent across viewpoints to enhance the consistency. An overview of our framework is depicted in Figure
3.1 High-resolution Multi-view Generation
We first explain the design of our bigh-resolution multi-view generation model that generates four orthographic view images from single input image. Instead of directly training high-resolution (2K)
3

---

2048?
X
2048"
256
Consistent Upicale
Multi-view
Diffusion
Multi-iiew
Normal Diffusion
Input Image
Multi-view Images
Upscaled Multi-view Images
Multi-view Normal Maps
Mesh Reconstruc & &tefine
Mesh Colorize
Mesh Init Initialize
Textured Mesh
Refined Mesh
Initialized Mesh
ISOMER
Figure 2: Pipeline of our Unique3D. Given single in-the-wiid wild image imput, first generate four orthographic multi-view images from multi-view diffusion model. Then, we progressively improve the resolution of generated niews through multi-level upscale process. Given generated color images, we train a normal diffusion model to generate normal maps corresponding to multi-view images and utilize similar strategy to lift it to high-resolution space. Finally, reconstruct high-quality 3D meshes from high-resolution color images and normal maps with our instant and consistent mesh reconstruction algorithm ISOMER.
multi-view difffusion that would consume excessive computational resources, we adopt a multi-level generation strategy to upscale the generated resolution progressively.
High-resolution Multi-view Image Generation. Instead of training from scratch, we start with the
initialization the pre- pre-trained 2D diffusion model using the checkpoint of Stable Diffusion [40] and encode multi-view dependencies to fine-tune it to obtain multi-view diffusion model that is able to generate four orthographic view images (256 resolution) from a single in-the-wild image. It is worth noting that the images generated in this step have relatively low resolution and suffer from multi-view inconsistency in out-of-the-domain data. This significantly limits the quality of
47). In contrast, address the multi-view consistency issue during the reconstruction phase (Sec 3.2) Given the generated four orthographic view images, we then finetune multi-view aware ControlNet 770 improve the resolution of images. This model leverages the four collocated RGB images as control information to generate corresponding clearer and more precise multi-view results. enhances the details and ameliorates unclear regions, leading the resolution of images from 256 to 512. Finally, we employ a single-view super-resolution model [58] to further upscale the image by factor of four, achieving resolution of 2048 that offers sharper edges and details without disrupting the multi-view consistency.
recent works 64
60
63
29
High-resolution Normal Map Prediction. Using pure RGB images makes it extremely hard reconstruct correct geometry. To effectively capture the rich surface details of the target 3D shape, we finetune normal diffusion model to predict normal maps corresponding to multi-viww color images. Similar to the above high-resolution image generation stage, we also employ the super-resolution model (58 to quadruple the normal resolution, which enables our method recover high-fidelity geometric details, especially the accuracy of the edges.
To enhance the capability of the image generation model and the standard normal prediction model in producing high-quality images with uniform backgrounds, we adopt a channel wiiee noise offset strategy 22 This alleviate the problem caused by the discrepancy between the initial Gaussian noise during sampling and the noisiest training sample.
3.2 ISOMER: An Efficient Method for Direct Mesh Reconstruction
Despite impressive results generated by recent popular image-to-3D methods 22
23
that
follow the field-based reconstruction 44 45 46), they have limited potential for higher-resolutte

---

applications as their computational load is proportional to the cube of the spatial resolution. In contrast, we design a novel reconstruction algorithm directly based on mesh, where the computational load scales with only the square of the spatial resolution and relates to the number of faces, thus achieving a fundamental improvement. This enables our model to efficiently reconstruct meshes with tens of millions of faces within seconds.
We now move to introduce our instant and consistent mesh reconstruction algorithm (ISOMER), which is a robust, accurate, and efficient approach for direct mesh reconstruction from high-resolution multi-view images. Specifically, the ISOMER consists of three main steps: (a) estimating the rough topological structure of the 3D object and generating an initial mesh directly; (b) employing a coarse-to- coure-tooffiee strategy further approximate the target shape; (c) explicitly addressing inconsistency across multiple views to reconstruct high-fidelity and intricate details. Notably, the entire mesh reconstruction process takes no more than 10 seconds.
Initial Mesh Estimation. Unlike popular reconstruction methods based on signed distance fields 771 or occupancy fields [3]], mesh-based reconstruction methods [IO] 62] struggle with changing topo- logical connectivity during optimization, which requires correct topological construction during initialization. Although initial mesh estimation can be obtained by existing methods like DMTet [. they cannot accurately reconstruct precise details (e.g.. small holes or gaps). To address the problem, we utilize front and back views to directly estimate the initial mesh, which is fast for accurate recovery of all topologically connected components visible from the front. Specifically, we integrate the normal map from the frontal view to obtain a depth map by
d(i.j) Σ (t.z(
where R2 (t, is the normal vector of the t-th pixel in the j-th row. Although the diffusion process generates pseudo normal maps, these maps do not yield a real normal field which is irrotational. To address this, we introduce random rotation to the normal map before integration. The process is repeated several times, and the mean value of these integrations is then utilized to calculate the depth, providing reliable estimation. Subsequently, we map each pixel its respective spatial location using the estimated depth, creating mesh models from both the front and back views of the object. The two models are seamlessly joined through Poisson reconstruction, which guarantees smooth connection between them. Finally, we simplify them into 2000 fewer faces for our mesh initialization.
Coarse-to-Fine Mesh Optimization. Building upon the research in inverse rendering 12, 33 18). we iteratively optimize the mesh model to minimize a loss function. During each optimization step, the mesh undergoes differentiable rendering to compute the loss and gradients, followed by vertex movement according to the gradients. Finally, the mesh is corrected after iteration through edge collapse, edge split, and edge flip maintain a uniform face distribution and reasonable edge lengths. After several hundred voarse-to-fine fine iterations, the model converges to a rough approximation of the target object's 's shape. The loss function for this part includes mask-based loss
L Lmask
Σ
M Mpred 2
where M, is the rendered mask under view and M.red is the predicted mask from previous subsection under view i. The mask-based loss regulates the mesh contour. Additionally, it includes a normal-based loss
Enormal
Σ Marod NN Npreed 2
concerning the rendered normal map N; of the object and the predicted normal map Nprred optimizing the normal direction in the visible areas, where @ denotes element-wise production. We compute the final loss function as:
Crecon = Lmask + Enormat-
To address potential surface collapse issues under limited- -iet normal supervision shown in Figure (b)) we employ a regularization method called Expansion. At each step, vertices are moved small distance in the direction of their normals, akin to weight decay.
ExplicitTarget Optimization for Multi-view Inconsistency and Geometric Refinement. Due to inherent inconsistencies in generated multi-view images from distrii-ittiiitonnnn (OOD) in-the-wild
5

---

input, no solution can perfectly align with every viewpoint. After the above steps, we can only reconstruct model that roughly matches the shape but lacks detail, falling short of our pursuit of high-quality mesh. Therefore, we cannot use the common method that minimizes differences in all views, which would lead to significant wave-pattern flaws, as shown in Figure (a). To overcome this challenge, finding a more suitable optimization target becomes crucial. Under single-view supervision, although a complete model cannot be reconstructed, the mesh shape within the visible area of that view meet the supervision requirements with highly detailed structures. Based on this, we propose novel method that assigns unique optimization target for each vertex to guide the optimization direction. In contrast to the conventional implicit use of multi-view images as optimization targets, we explicitly define the optimization target with better robustness. We call this explicit optimization target ExplicitTarget and devise it as follows:
(ExplicitTarget). Let (V, W) Σ, V,W; represent the weighted average function, and (M, (N+,N-) + (0, 1} represent the visibility of vertex e in mesh M under view i. Colas (v, i) Indicate the color of vertex in viewpoint i. We compute the ExplicifTarget ET of each vertex in mesh M as
W;
[ugg( (C, i), VM (1, ETM(U) = {
if E,E,AO VM(U:)) otherwise,
(5)
whereWar (v, = cos(( (M( N ((liew)) is weighting factor that is the vertex normal of in mesh M, and N N-(view) is the view direction view
In the function (I, the predicted color of vertex is computed as the weighted sum of supervised views, with weights determined by the square of cosine angles. This is because the projected area is directly proportional to the cosine value, and the prediction accuracy is also positively correlated with the cosine value. The object loss function for ExplicitTarget defined as
LET Σ Mfred
N -N-NN.T
(6)
where N,TT" is the rendering result of mesh M with {ETM (I, Nered w)lp E M) under the i-th viewpoint. The final optimization loss function is
Lrefine Lmmas LET-
Towards this end, we finish the introduction of the ISOMER reconstruction process, which includes three stages: Initialization, Reconstruction, and Refinement.
Upon generating precise geometric structures, it is necessary to colorize them based on multi-view images. Given the inconsistencies across multi-view images, the colorizing process adopts the same method used in the refinement stage. Specifically, the colors mesh . is {ETM (I, (I, Ipred v)lu e M). Moreover, certain regions of the model may remain unobservable from the multi-view per- spective, necessitating the coloring of these invisible areas. To address this, we utilize efficient smoothing coloring algorithm to complete the task. More detailed and specific algorithmic procedures can be found in the Appendix.
4 Experiments
4.1 Experimental Setting
Utilizing a subset of the Objaverse dataset as delincated by LGM [53], we apply a rigorous filtration process to exclude scenes containing multiple objects, low-resolution imagery, and unidirec- tional faces, leading to a refined dataset of approximately 50k objects. To address surfaces without thickness, we render eight orthographic projections around each object horizontally. By examining the epipolar lines corresponding to each horizontal ray, we identify 13k instances of illegitimate data. For rendering, we employ random environment maps and lighting to augment the dataset, thereby enhancing the model's 's robustness. To ensure high-quality generation, all images are rendered at a resolution of 2048 x 2048 pixels.
Dataset:
Network Architecture: The initial level of image generation initialized with the weight of the Stable Diffusion Image Variations Model [40), while the subsequent level employs upscaled

---

Input Image
Ours
GRM
Instantivlesh
CRM
SyncDreamer
OpenLRM
Wonder3D
One-2-3-45
Figure 3: Qualitative Comparison. Our approach provides superior geometry and texture.
version fine-tuned from ControlNet-Tile (70). The final stage the pre-trained Real-ESRGAN model [58]. Similarly, the initial stage of normal map prediction is initialized from the aforementioned
Stable Diffusion Image Variations. Details of these networks are provided in the Appendix.
Reconstruction Details: The preliminary mesh structure is inferred from a normal map with resolution of x 256, which is then simplified to mesh comprising 2, 000 faces. The reconstruction process involves 300 iterations using the SGD optimizer [3) with a learning rate of 0.3. The weight of expansion regularization is set 0.1. Subsequent refinement takes 100 iterations, maintaining the same optimization parameters.
The entire training takes around 4 days on 8 NVIDIA RTX4090 GPUs. The primary level of multiview image generation uses 30k training iterations with batch size of 1, 024. The training of viet image upscaling involves 10k iterations with batch size of 128. Normal
Training Details:

---

Input Image
Ours
InstantMesh
CRM
OpenLRM
Figure 4: Detailed Comparison. We compare our model with InstantMesh 63
CRM
60
and
OpenLRM 111. Our models generates accurate geometry and detailed texture.
Table 1: Quantitative comparison results for mesh visual and geometry quality. We report the metrics of PSNR, SSIM, LPIPS and Clip-Similarity (38. ChamferDistance (CD), Volume IoU and F-score
on GSO dataset.
Method
PSNR1
SSIMt
I.PISS
Clip-Simt
CDJ
Vol. IoUf
F-Scoref
One-2-3-45
16.1058
0.8874
0.1812
0.7782
0.0313
0.4142
0.5518
OpenLRM
18.0433
0.8957
0.1560
0.8416
0.0336
0.3947
0.5354
Wonder3D
18.0932
0.8995
0.1536
0.8535
0.0261
0.4663
0.6016
InstantMesh
18.8262
0.9111
0.1283
0.8795
0.0161
0,5083
0.6491
CRM
18.4407
0,9088
0.1366
0.8639
0.0141
0.5218
0.6574
Unique3D
20.0611
0.9222
0.1070
0.8787
0.0143
0.5416
0.6696
Unique3D w/o ET
20.0383
0.9199
0.1129
0.8675
0.0158
0.5320
0.6594
WSSMERR
18.6131
0.9026
0.1470
0.8621
0.0244
0.4743
0.6088
map prediction is trained for 10k iterations at a batch size of 128. Additional training specifics are accessible the Appendix.
4.2 Comparisons
Qualitative Comparison:
To highlight the advantages of our methodology, we perform a compre-
hensive comparison with existing works, including CRM (60), one-2-3-45 (26), Wonder3D (29].
OpenLRM [11), and InstantMesh [63]. For fair quality comparison, we choose to present samples
previously selected the referenced papers, originating from Wonder3D 29], SyncDreamer (2)
CRM 60), and InstantMesh [63] The results are shown in Figure 3 Our results clearly surpass the existing works in both geometric and material quality, thereby emphasizing the benefits of our approach in achieving high resolution and intricate details in both geometry and material. In addition to the above overall quality comparison, we further show the comparison the details in Figure 4 highlighting the advantage of our method in high resolution. The reconstruction process of ISOMER completed under seconds, while the entire procedure from the input image to high-precision mesh is accomplished in less than 30 seconds
RTX4090.

---

Input
w/o Explicit Target w/ Explicit Target
Input
w/o Expansion
w/ Expansion
Figure Ablation Study on ISOMER. (a) Without ExplicitTarget, the output mesh result has obvious defects. (b) Without expansion regularization, the output result collapses in some cases.
Colorize with Explicit Target
Colorize w/o Explicttarget
Figure 6: Ablation on Colorize. We show a comparison of whether or not to apply ExplicitTarget in coloring, and can that the group that does not use ExplicitTarget has significant artifacts, as there is no precise consistency across multiple views.
Quantitative Comparison: In line with previous work, we evaluate our results using the Google Scanned Objects (GSO) [8] dataset. We render frontal views at resolution of 1024 x 1024 with Blender EEVEE input for all methods. All generated mesh results normalized the bounding box 0.5, 0.5] to ensure aligmment. The geometric quality is assessed by calculating the distance the ground truth mesh using metries such as Chamfer Distance (CD), Volume IoU, and F-Score. Concurrently, we render 24 views around the object, selecting one of [o. 15, 30] for elevation angles and 8 evenly distributed azimuth angles spanning a full 360-degree rotation. We employ PSNR, SSIM, LPIIS, and Clip-Similarity [8) to evaluate the visual quality. The results are presented in evidenced in table, both our geometric and material quality outperform those of existing
Table T methods. We find that ISOMER can even be used to improve the consistency of other methods. For example, in Table [ we replace Wonder3D's reconstruction method with ISOMER, which is not only faster but also of higher quality.
4.3 Ablation Study and Disscussion
We analyze the importance of ExplicitTarget and expansion regularization in ISOMER. We compare samples with and without ExplicifTarget and Expansion Regularization in figure We clearly show the improvement of ExplicitTarget for geometry and the necessity of expansion regularization for reconstruction. SsplicitTargge notably improves reconstruction results in challenging cases, while expansion regularization avoids some possible collapses.
Mirroring the approach used for geometry, we will include additional experimental results in Figure that illustrate the impact of the Explicit Target method on texture quality. Without the Explicit Target, the results are obviously flawed.

---

Table 2: Quantitative comparison results for ablation on 100 random samples with random rotation on GSO dataset.
Method
PSNRT
SSIMT
LPIPSL
Clip-Simt
CDL
Vol. ToUt
F-Scoret
U
19.6744
0.9217
0.1101
0.8864
0.0118
0.5463
0.6833
Input image
256 Reselution
512 Resalution
2048 Resolution
Figure 7: Ablation on Resolution. The visualization of the generated multi-views images at different stages is shown. Multi-level super-resolution does not change the general structure, but only improves the detail resolution, allowing the model to remain well-detailed.
Additionally. we added new test with randomly rotated objects sampled from azimuth E U|,180, 180), elevation E U|-30, 30] Table 2 to test robustness in non-front-facing views. The test results show that Unique3D still performs well in this case, and even the geometry prediction more accurate.
We expand our study to include qualitative comparison across various resolutions in order to demonstrate the differences between different resolutions Figure The results demonstrate the necessity of high resolution maps in generating high resolution meshes.
The majority of ex- isting common samples are overly simplistic effectively demonstrate the advantages of our study. Consequently, we select two com- plex samples: object featuring detailed text and a photograph of a human, as shown in Figure It is observable that our method exhibits exceptional mesh materials and ge- ometry, even capable of sculpting geometric structures with textual detail. In the context of photographs, our reconstruction results are nearly par with specialized image-to- character mesh generation methods.
Challenging Examples:
Input Image
Genersteal Textured
Figure Challenging examples.
Conclusion
In this paper, we introduce Unique3D, pioneering image-to-3D framework that efficiently generates high-quality 3D meshes from single-view images with unprecedented fidelity and consistency. By integrating advanced diffusion models and the powerful reconstruction method ISOMER, Unique3D generates detailed and textured meshes within 30 seconds, significantly advancing the state-of-the-an in 3D content creation from single images.
Limitation and Future Works. Our method, while capable of generating high-fidelity textured meshes rapidly, faces challenges. The multi-view prediction model may produce less satisfactory predictions for skewed or perspeesee inputs. Furthermore, the geometric coloring algorithm currently does not support texture maps. In the future, aim to enhance the robustness of the multi-view prediction model by training on a more extensive and diverse dataset.
10

---

References
[1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, lan Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome a day. Communications of the ACM, 14(10):105-112,22) 2011.
[2] Mario Botsch and Leif Kobbelt. A remeshing approach to multiresolution modeling. In Sean-Daniel Boissonnat and Pierre Alliez, editors, Second Earographics Symposium on Geometry Processing, Nice, France, July 8-10, 2004, volume 71 of ACM International Conference Proceeding Seriex, pages 185-192. Eurographics Association, 2004.
[3] Léon Bottou, Large-scale machine learning with stochastic gradient descent. In Yves Lechevallier and Gilbert Saporta, editors, 19th International Conference on Computational Statistics, COMPSTAT 2010, Paris, France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177-186. Physica-Veriie 2010.
[4] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-ricl 3d model repository. arXiv preprint 251.0:251,, 2015.
[5] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. CoRR, abs/2303.13873 2023.
[6] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
[7] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 13142-13153. IEEE, 2023.
[8] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas Barlow McHugh, and Vincent Vanhoucke. Google scanned objects: high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 2553-2560. IEEF, 2022.
[9] Yasutaka Furukawa, Carlos Hernández, al. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision, 9112))1-14 2015.
[10] Shubham Goel, Georgia Gkioxari, and Jitendra Malik. Differentiable stereopsis: Meshes from multiple views using differentiable rendering. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 8625-8888008 2654863,, IEEE, 2022.
[11] Zexin He and Tengfei Wang. OpenIrm: Open-source large reconstruction models. httts::/gitte com/ httn//DDopaa/OpenLRR 2023.
[12] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.
[13] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: large reconstruction model for single image to 3d. CoRR, abs/2311 1112231110440 2023.
[14] Po-Han Huang, Kevin Matzen, Johannes Kopf. Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-vicw stereopsis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1821-28300 2018.
[15] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint 22Xiv:2305.0246: 2023.
[16] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139:1-1338 2023.
[17] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
11

---

[18] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen. and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Trans. Gruph., 3966):194:1-199:14 2020.
[19] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent lext-to-3d. arXiv preprint 10.0059, 2023.
[20] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung- Lin. Magic3d: High-resolution text-to-3d content creation. CoRR, abs/2211 1. 10.442, 2022.
[21] Shanchuan I. Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, 2024.
pages 2024.
[22] Shanchuan Lin, Bingehen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps flawed. In IEEE/CVF Winter Conference Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA, January 3-8, 2024, pages 5392 5399, IEEE, 2024.
[23] Fangfu Liu, Hanyang Wang, Weiliang Chen, Haowen Sun, and Yueqi Duan. Make-your-3d: Fast and consistent subject-driven 3d content generation. arXiv preprint arXiv:2403.0966 2024.
[24] Fangfu Liu, Diankun Wu, Yi Wei, Yongming Rao, and Yueqi Duan. Sherpa3d: Boosting high-fidelityy text-to-3d 3d generation via coarse 3d prior. arXiv preprint arXiv:2312.06655, 2023.
[25] Jian Liu, Xiaoshui Huang. Tianyu Huang, Lu Chen, Yuenan Hou, Shixiang Tang, Ziwei Liu, Wanli Ouyang, Wangmeng Zuo, Junjun Jiang, et al. A comprehensive survey on 3d content generation. arXiv preprint arXiv:2402.00 01.00....11000 166, 2024.
[26] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T. Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[27] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-t0-3: Zero-shot one image to 3d object. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 9264 9275. IEEE, 2023.
[28] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview--onsistent images from a single-view image. arXiv preprint arXiv:2309.03453. 2023.
[29] Xiaoxiao Long, Yuan- Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song Hai Zhang. Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross domms-n diffusion, arXiv preprint arXiv:2310. 15008, 2023.
[30] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphicx: pioneering efforts that shaped the field, pages 347-353, 1998.
[31] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jouathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: represeentngg scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1):99-106, 2022.
[32] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM Trans. Graph., 41(4):10:111102:111 2022.
[33] Werner Palfinger. Continuous remeshing for inverse rendering. Comput. Animat. Virtual Worlds, 33(5). 2022.
[34] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 165-174. Computer Vision Foundation / IEEE, 2019.
[35] Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. ACM Trans. Graph., 22(3):313- 318, 2003.
[36] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3h 3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023.
12

---

[37] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin. Bing Li, Hsin- Ying Lee. Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, al. Magic123; One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint wXXv:23306.77843 2023.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 2021.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695 10695, 2022.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resoltiion image synthesis with latent diffusion models. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674-10685. IEEE, 2022.
[41] Chitwan Saharia, William Chan, Saurahh Saxena, Lala Li, Jay Whang, Emily I., Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
[42] Johannes I. Schonberger and Jan-Michael Frahm. Stracture-from-mottinn revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4 4104-4113, 2016
[43] Johannes L Schönberger, Enliang Zheng, Jan-Michacl Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. Computer Vision- 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 501-518. Springer, 2016.
[44] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yo Yu I Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-rcsolution 3d shape synthesis. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 14, 2021, virtual, pages 6087-6101, 2021.
[45] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojeie, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh 2023.
optimization. ACM Trans. Graph., 42(4):37:1-337:16
[46] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang. Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Trans. Graph., 42(4):37:1-37:16 2023.
[47] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123+44 single image consistent multi-view diffusion base model. CoRR, abs/2310.15110, 2023.
[48] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for generation. CoRR, abs/2308. 16512, 2023.
[49] Zifan Shi, Sida Peng. Yinghao Xu, Yiyi Liao, and Yujun Shen. Deep generative models 3d representa- tions: A survey. CoRR, abs/2210. 15663, 2022.
[50] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et Make-a-video: Text-to-vided generation without tixt-video data, arXiv preprint arXiv:2209.1 14000
[51] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections 3d. In ACM siggruph 2006 papers, pages 835-846. 2006.
[52] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[53] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: large niet gaussian model for high-resolution 3d content creation. CoRR, abs/2402.05054, 2024.
13

---

[54] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint MkKv:2300.16653, 2023.
[55] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. CoRR, abs/2212.00774, 2022.
[56] Peng Wang, Lingje Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
[57] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. CoRR. abs/2312.02201.. 2023.
[58] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, October 11-17, 2021, pages 1905-1914, IEEE, 2021.
[59] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. CoRR, abs/2305.16213, 2023.
[60] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. CRM: single image to 3d textured mesh with convolutional reconstruction model. CoRR, abs/2403.05034, 2024.
[61] Wikipedia. Signed distance function - Wikipedia, the free encyclopedia. http://en /n. wikipedia ottg wttp://aawaaaaaaaeooooooeooaaaaa accessed 05 05-May-2024)
2024. [Online;
[62] Markus Worchel, Rodrigo Diaz, Weiwen Hu, Oliver Schreer, Ingo Feldmann, and Peter Eisert. Multi-view mesh reconstruction with neural deferred shading. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 6177-6187 6187. IEEE, 2022.
[63] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024. 2024.
[64] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. GRM: large gaussian reconstruction model for efficient 3d reconstruction and generation. CoRR, 1b4/2403.14621, 2024.
[65] Fan Yang. Jianfeng Zhang, Yichun Shi, Bowen Chen, Chenxu Zhang, Huichao Zhang, Xiaofeng Yang. Jiashi Feng, and Guosheng Lin. Magic-boost: Boost 3d generation with mutli-view conditioned diffusion. arXiv preprint arXiv:2404.06442, 2024.
[66] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hlsan Yang. Diffuusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[67] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pages 767-783, 2018.
[68] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. CoRR, abs/2308.06721, 2023.
[69] Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang, Xinzhou Wang, Yuegi Duan, and Jun Zhu. Dreamreward: Text-to-3d generation with human preference. arXiv preprint arXiv:2403. 14613, 2024.
[70] Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. CoRR, abs/2302.05543, 2023.
[71] Fuqiang Zhao, Yuheng Jiang, Kaixin Yao, Jiakai Zhang, Liao Wang, Haizhao Dai, Yuhui Zhong, Yingliang Zhang, Minye Wu, Lan Xu, and Jingyi Yu. Human performance modeling and rendering via neural animated mesh. ACM Trans. Graph., 11:5::235::-23::111,, 2022.
14

---

Input Image
Generated Textured Mesh
Input Image
Generated Textured Mesh
Figure More generated results of method from a single image.
A
More Results
We provide more generation results of our method from single image in Figure
15

---

B Network Architecture and Training Details
In this part, we develop model based on the architecture of Stable Diffusion Image Variation 400 with two main modifications: (1). The use of class embedding. which takes an integer from 0 to 3 as input, indicating the corresponding view indexes. (2). The simultameous forward of four perspectives, where they are concatenated in the self-attention layers to achieve multi-view consistency.
Multi-view Image Generation:
For the training of the network, we utilize the following parameters:
A learning rate of 10-4
A batch size of 1024.
A noise offset of 0.1.
An SNR gamma 5.0.
An 8-bit Adam optimizer 17 with betas set to (0.9,0.999) 0.999).
An Adam weight decay of 0.01.
An Adam epsilon of 10
Gradient clipping with norm of 1 to ensure training stability.
Multi-view Image Upscale: In this part, we aim upscale merged low-resolution four-view images to high resolution of 1024 pixels. To achieve this, we fine-tune the ControlNet- -ile network, leveraging StableDiffusion 1.5 its backbone. Unlike traditional methods, we do not use text input; instead, we feed an empty text. Concurrently, we pass the input image through an IP-Adapter [68] This approach allows the network be guided in enhancing the multi-view details and achieving the desired resolution.
For the training of this network, we use the following parameters:
A learning rate of 5 x 10-6
A batch size of 128.
A noise offset of 0.1.
An SNR gamma of 5.0.
An 8-bit Adam optimizer 17 with betas set to 0.999).
An Adam weight decay of 0.01.
An Adam epsilon of 10 10-8
Gradient clipping with norm of 1 to ensure training stability.
Freeze parameters except for the ControlNet.
In this part, we train diffusion model that takes an RGB image input and produces its corresponding normal map output. This model is based on the Stable Diffusion Image Variation [40]. with one key modification: a reference U-Net has been incorporated, which has an identical network structure and initialization as the original network. This reference U-Net provides pixel-wise reference attention to the main network exclusively at a new attention layer that added to the self-attention.
Normal Prediction Diffusion:
For the training of the network, we utilize the following parameters:
A learning rate of 10-4 for main network.
A learning rate of 10-5 for reference network.
A batch size of 128.
A noise offset of 0.1.
An SNR gamma of 5.0.
An 8-bit Adam optimizer 17 with betas set (0.9, 0.999).
An Adam weight decay of 0.01.
16

---

An Adam epsilon of 10
Gradient clipping with norm of 1 to ensure training stability.
Freeze parameters except for the self-attention in reference attention.
Train all parameters in the main network.
C Efficient Invisible Region Color Completion Algorithm
In our approach to mesh coloring using multiple viewpoints, we encounter a minor yet noteworthy
challenge: the need to color regions that are not directly visible. Although these regions typically sparse and inconspicuous. In field-based representations such as Signed Distance Fields [61], they often the color of neighboring visible areas upon completion of the field optimization. To address this, we opt for straightforward yet efficient algorithm that seamlessly spreads the colors of nearby visible regions into the invisible ones.
Our methodology employs a straightforward. multi-step color propagation algorithm, which stands out for its simplicity, swift exccution, and reliability in delivering reasonably detailed and nuanced color complement. This approach outperforms more complex, aasource-intensiig and less stable techniques like using pre-trained inpainting diffusion models. The algorithm leverages the surrounding colors to gently fill in the invisible regions, with the detailed process outlined in Algorithm [
A critical aspect of the algorithm to consider is the potential for stark color demarcation line if the process is halted immediately after all nodes have been colored. For example, one-dimensional scenario, if red is on the left and blue is on the right, separated by uncolored section, stopping immediately will result a high-contrast boundary. To mitigate this, we extend the color propagation process through number of iterations ensure smooth color gradient. This allows the colors to gradually permeate throughout the entire connected component the mesh that requires coloring, thus achieving harmonious and visually coherent result.
Algorithm 1 Color Completion Algorithm
Input: Mesh M, list of invisible vertices Inv, list of color of all vertices C
Output: The completed color list
1: ent +-0 0
2: stage2 + False
3: colored + 0
4: for all vertices e in M do
5: if v Ire then
Append True visible_vertices
7: else
8:
Append False to visible_vertices
9:
end
10:
end for
while stage2 == False or end > 0 do
12:
for all in Inu do
13:
colored neighbors list of vertices directly connected to in M that have colored = True if colored, neighhors! then
14:
colored[i] +- True
16:
Cli) + mean meaanCl(clloaooo ([
17:
else
18:
colored[i] +- False end
19:
20:
end for
21:
if all elements of colored are True then
22:
stage2 t- True ent cnt
23:
24: else
25:
ent + cnt +
26:
end if
27:
end while
28:
return
17

---

Algorithm 2 ExplicitTarget Algorithm
Input: Multi-view image list imgs, initial mesh model M
Output: Model M' with vertex colors set to ExplicitTarget
1: M" t- M
2: Set the color of M' to the vertex normals of M
3: for all vertices in the vertex set of M' do
4: tot tot_weight +-
5: tot_color - 0
Initialize to zero vector
6:
for all images im in imgs do
7:
if vertex v not visible in the viewpoint of im then continue
8:
9:
end if
10:
ci f- the color of vertex v in image im 4- the square of the cosine of the angle between the vertex normal of and the view direction from im to v tot tot_weight < too_tttuu weight + wi tol_color +- lol tol_color + ci end for
12:
14:
15: if ttt_ttteetee weigwehhht> >> then
16:
Set the color of vertex in M' to tot tot_ttoloooooteeeeoooooooteo color/tottttuueeeeeeeeoooooo follorgtttweightt end
17:
18:
end for
19:
return M'
D ExplicitTarget algorithm
Direction Error of the Prediction
162
140
20-
60
Cosine of Angles
Figure Correlation between prediction value and prediction errors.
In Algorithm we demonstrate the detailed computation of ExplicitTarget. Specifically, we set an optimization target for each vertex, which is weighted of the supervised signals from the visible views of the vertex. The weights are determined by two factors: the projected area of the nearby surface and the confidence level in the accuracy the normals, which are used calculate the weights. In Figure I0] we show the relationship between the normal results predicted by multi View diffusion and the accuracy of the predictions on the Objaverse [7] validation set. The results indicate that the closer the angle between the predicted normal and the vertical to the current viewpoint is, the lower the accuracy of the prediction. There negative correlation between these two factors, with Pearson correlation coefficient of -0.304.
18

---

E More on Mesh Optimizations
This operation used to avoid and heal defects in the mesh. It involves selecting an edge within a triangle and collapsing it to the other edge, effectively merging the two triangles into single triangle. This process can help to eliminate narrow triangles that might be causing issues in the mesh such as those that are too thin to accurately represent the surface they approximating. Edge collapse can prevent the creation of topological artifacts and maintain the quality of the mesh. Edge Split: This is the opposite of edge collapse. In edge split, an edge that is longer than specified maximum length is divided into two, creating new vertices at the midpoint of the edge. This operation is used to refine the mesh, ensuring that the local edge length is kept close to the optimal length. It helps to maintain the quality of the mesh by avoiding edges that are too long. which could lead to inaccurate representation of the surface. Edge Flip: Edge flip is an operation that adjusts the connectivity of the mesh to improve its quality. It involves flipping an edge within a triangle to connect two non-adjacent vertices, effectively changing the triangulation of the mesh. This can help to maintain the degree of the vertices close to their optimal value, which is typically six for internal vertices (or four for boundary vertices). The goal of these operations to improve the mesh quality while avoiding defects and ensuring that the mesh accurately represents the target geometry.
Edge Collapse:
F
Ablation Study on Mesh Initialization
Input Image
Our Initialization
Ball Initialization
Figure 11: Ablations on Mesh Initialization. We compare the results of using our fast initialization method, versus using a sphere as an initialization.
We compare the different mesh initialization methods and their results. One is our proposed fast initialization method, and the other is using spheres as initialization objects, a common practice in reconstruction techniques. Figure F Fillustrates the problem of the mesh reconstruction method that fails to modify its topological structure. For example, in the first row, the model cannot achieve a hollow structure by direct optimization because their topologies are inherently different. However, as shown in the second row, even though the topologies are different, the optimization method can still provide approximate results. For instance, the sphere-basee initialization can shape the handle on the right side, even though the handle is incomplete. The sphere- based initialization can sometimes produce even more accurate results than our proposed method, as seen in the third row. These experiments demonstrate that our method is robust to initialization. Aiming for better ability to generalize, chose use our fast initialization.
19

---

G
User Study
For user study, we render 360-degree videos of subject-driven 3D models and show each volunteer with five samples of rendered video from random method. They can rate in four aspects: 3D consistency, subject fidelity, prompt fidelity, and overall quality on scale of 1-10, with higher scores indicating better performance. We collect results from 30 volunteers shown in Table We find our method is significantly preferred by users over these aspects.
Table 3: Quantitative comparison results the multi-view consistency, subject fidelity (related to geometric and texture details), prompt fidelity (related to the alignment of input single image), and overall quality score in study, rated range of 1-10, with higher scores indicating better performance.
Method
Multi view Consistency
Subject Fidelity
Prompt Fidelity
Overall Quality
One-2- One-2-3-45
26
5,46
4.78
6.93
5.79
OpenI.RM
K
6.72
7.16
6.92
7.15
SyncDreamer
5.71
7.52
4.06
5.92
Wonder3D
29
8.67
7.80
7.39
8,14
Mest
63
8.31
7.68
7.91
8.43
GRM 64
6.93
7.42
6.02
7.38
CRM 60
7.95
8.53
8.03
8.25
Ours
9.26
8.74
8.52
9.02
H
Social Impact
The Unique3D framework can democratize 3D content creation, making it easier for artists and designers to produce 3D models from single images, which can lead to increased innovation and surge in creative applications across various industries including gaming, film, and education.
Positive Impacts:
Negative Impacts: On the flip side, the ease of generating high-quality 3D models raises concerns about potential misuse, such as creating deepfakes, and could lead job displacement for traditional 3D modelers. Additionally, there may be challenges related to intellectual property and privacy if the technology used irresponsibly.
Licenses for Used Assets
Stable Diffusion
40 under CreativeML Open RAIL M License.
Objaverse under ODC-By v1.0 license.
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We prove our claim through quantitative and qualitative experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
There is no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, released the inference code GitHub.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: Yes, the paper provides open access to the data and code, along with compre- hensive instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https i
//nips //nips.c//pil cc/public/gut
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, see the Experiments section and Appendix for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No, error bars are not available.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp.....ae...ooo/////////a
Answer: [Yes]
Justification: Yes, the research in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The license of assets are listed in Appendix.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to amonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper involves user study with human subjects. We invited 30 volunteers from the college to participate in our user study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: In the study described in this paper, no potential risks were identified for participants. The research design was carefully crafted ensure the safety and well-being of all individuals involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "h4YhqJUHQn", "reviewer": "Reviewer_Bw7b", "summary": "This paper introduces Unique3D, a framework aiming to generate 3D meshes from single-view images with high quality and fidelity. Driven by the observation that 2D image pixel and normal priors with higher resolution can be crucial in generating intricate textures and complex geometries, Unique3D integrates a multi-level upscale process to progressively improve the resolution. It also proposes an instant and consistent mesh reconstruction algorithm called ISOMER to lower the computation complexity and improve reconstruction quality.", "questions": "See weakness 1, 3, 4.", "limitations": "Limitations are discussed very briefly in the conclusion section. Addressing the questions in the above weaknesses could involve further discussions on the limitations. Societal impacts are included in the supplementary materials.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Unique3D addresses the limitations of previous methods by generating high-resolution 2D images and normal maps and optimizing the color mesh according to these 2D guidances. The idea of increasing resolution in 2D is straightforward and powerful as the experiments suggest. The pipeline is efficient and can generate mesh within a short time frame (about 30 seconds per mesh). This makes it practical for many real-world applications. The paper provides solid and detailed results for the effectiveness of its methods and delivers rather convincing results. The proposed ISOMER is novel and could inspire further studies in object color mesh reconstruction. The paper includes a thorough analysis of the ISOMER algorithm which provides good insights into the module.", "weaknesses": "1. This pipeline features an image diffusion model, two super-resolution modules, and a normal diffusion model. All work in a sequential manner without interaction. This may bring significant compounding errors in multi-view/multi-resolution consistency before reconstructing mesh with ISOMER. The author should include a more thorough analysis on error patterns with particular focus on multi-view/multi-resolution consistency.\n2. The subsection \"ExplicitTarget Optimization for Multi-view Inconsistency and Geometric Refinement\" (L183-) is hard to read and understand. Improvements in writing and equations are needed.\n3. Based on my understanding of the ISOMER algorithm, it could be prone to inconsistencies, especially around the 2D boundaries of the generated multi-view images. Could you provide more insights into this aspect? Also, how well is the model working with non-front-facing input views? Adding some discussion for these could further improve the coverage of the experiments and help people understand the limitations.\n4. 30 random objects (L257) are too few to be a meaningful quantitative evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and valuable comments, some of which are fundamental and in-depth suggestions that help us greatly improve our paper. To address your concerns, we present the point-to-point response as follows.\n\n**Comment 1: Multi-view Consistency Analysis**\n\nWe appreciate the reviewer bringing this comment up. To answer your concern, first, our choice to adopt a sequential design is based on the following observations: 1) Integrating the image diffusion model with two super-resolution models into a 2048-resolution diffusion model would result in an enormous computational load due to the high resolution. 2) We noticed that methods combining both image and normal diffusion models (e.g., Wonder3D) frequently yield normal results lacking sufficient details, and the guidance scale does not effectively control the output. Our experiments on this hybrid architecture were in alignment with these observations. By decoupling the two models, we have notably enhanced the accuracy of the normals, as the normal diffusion model does generations based on clear images rather than noisy image latents. Therefore, we have chosen to use separate image and normal diffusion models to improve the fidelity of the normals and the subsequent mesh reconstruction.\nISOMER can even be used to improve the consistency of other methods. For example, in Table 1, we replaced Wonder3D's reconstruction method with ISOMER, which is not only faster but also of higher quality.\n\n**Comment 2: Clarity in \"ExplicitTarget Optimization\" Subsection**\n\nWe value your insightful suggestions. To address your concern, we will revise the explanation and equations in \"ExplicitTarget Optimization\" subsection for better comprehension. Here is our improved version:\n\n (***ExplicitTarget***). Let  $Avg(V, W) = \\frac{\\sum_i{V_i W_i}}{W_i}$ represent the weighted average function, and $V_{M}(v, i): (\\mathbb{N}^+, \\mathbb{N}^+) \\rightarrow \\{0, 1\\}$ represent the visibility of vertex $v$ in mesh $M$ under view $i$. $Col_M(v,i) $ Indicate the color of vertex v in viewpoint i.  We compute the ExplicitTarget $ET$ of each vertex in mesh $M$ as\n$$\\\\begin{equation}\n    ET_{M}(v) = \\\\begin{cases}\nAvg(Col_M(v,i), V_{M}(v, i) W_M(v, i)^2) \n & \\text{, if } \\sum_{i} V_{M}(v, \\mathcal{i}) > 0 \\\\\\\\ \n \\mathbf{0} & \\text{, otherwise, } \n\\\\end{cases} \n\\end{equation}$$\nwhere $W_M(v, i) = -\\cos(N_{v}^{(M)}, N_{i}^{(view)})$ is a weighting factor, $N_{v}^{(M)}$ is the vertex normal of $v$ in mesh $M$, and $N_{i}^{(view)}$ is the view direction of view $i$. \n\n$ET_M$ computes the results with view direction weighting so that each viewpoint does not introduce significant errors for surfaces that are too skewed in the current viewpoint (because predictions in these places are usually inaccurate).\n\n**Comment 3: ISOMER Boundary Inconsistencies and Non-front-facing Views**\n\nThanks for your thorough comment. We agree that the key insight of ISOMER could be better clarified. Following your suggestions, we will **revise our paper** to emphasize that **ISOMER directly handles the case where the global normal of the same vertex is inconsistent across viewpoints.**\n\nAs demonstrated in our ablation study (Figure 5), previous reconstruction algorithms often yield poor results with blurriness or wavy patterns when faced with inconsistent inputs, while adopting ISOMER notably improves the reconstruction results. We **include a new evaluation experiment** comparing Wonder3D with and without ISOMER to provide a clearer understanding of the benefits in Table 1.\n\nAdditionally, we added **a new test with rotated objects** in Table 2 to test robustness in non-front-facing views. The test results show that Unique3D still performs well in this case, and even the geometry prediction is more accurate (Because the shape of an object is better estimated, e.g. a book).\n\n**Comment 4: Insufficient Quantitative Evaluation**\n\nWe chose 30 random objects following the standards set by previous work (CRM, SyncDreamer, Wonder3D). Given that SyncDreamer takes up to 50 hours to generate 30 objects and results from other papers show that SyncDreamer performs significantly worse than existing methods. We exclude SyncDreamer and conduct a thorough experiment on the full GSO dataset. During our experiments, we found that GRM’s online samples were unavailable and lacked open-source code, preventing further testing of GRM’s results. The results for all methods are provided in Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oz6n5iEUnG", "reviewer": "Reviewer_7xHF", "summary": "This paper proposes a novel method for converting a single image to 3D. The method mainly consists of two stages: multi-view RGB and normal generation, and multi-view guided mesh optimization and texturing. The key innovation of the paper is a multi-view-normal-based 3D mesh reconstruction module. Specifically, given a single input image, the method first generates multi-view RGB images by fine-tuning a 2D diffusion model. It then employs a ControlNet-tile model and a super-resolution model to increase the resolution of the multi-view images from 256 to 2048. Additionally, a multi-view normal model is fine-tuned to generate the corresponding multi-view normal maps. Following this, a module called ISOMER is proposed to first convert the multi-view normal maps into a 3D mesh by fusing the normal maps to depth and then applying Poisson surface reconstruction. The mesh geometry is then refined according to a specialized multi-view normal loss. Finally, the mesh vertex color is optimized through a similar approach.", "questions": "1. When converting normal maps to depth maps, how do you handle discontinuity issues? For example, there may be occluded regions, leading to sudden jumps in the normal map.\n\n2. The method uses two models to upscale the resolution, first from 256 to 512 and then to 2048. Do we really need two models? Can we directly upscale from 256 to 1024? Or can we change their order?\n\n3. It seems that the mesh vertex colors are directly computed as a weighted sum of the projected colors. Am I correct? Will this cause any inconsistency issues or other artifacts?\n\n4. Line 133: \"we adopt a channel-wise noise offset strategy\" is not very clear to me. Could you provide more details?\n\n5. Line 160: \"not yield a real normal field which is irrotational. To address this ...\" is not clear to me. Could you explain this further?\n\n6. Line 181: What does Figure 3(b) refer to?\n\n7. Line 224: \"By examining the epipolar lines corresponding to each horizontal ray, we identify 13k instances of illegitimate data.\" This is not very clear to me. Could you provide more details?\n\n8. Line 241: What does \"second level of training\" refer to?", "limitations": "The authors briefly mention the limitations in the final section.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is well-motivated. Previous methods attempt to directly convert multi-view RGB images to 3D through a feed-forward model, which is harder to train and requires more computing resources. The proposed method leverages the multi-view normal information from a 2D diffusion model to directly optimize 3D geometry, avoiding expensive training.\n\n2. The proposed multi-view normal to 3D module is novel and interesting.\n\n3. The paper is generally well-written and easy to follow.", "weaknesses": "1. While I agree that the proposed method is technically sound and may be a nice supplement to the community, my major concern is that the experimental section is quite lightweight. For instance, the quantitative experiment is only conducted on 30 objects from the GSO dataset. Since the proposed method and the comparing baseline are not expensive to run compared to SDS-based methods, it is not acceptable to base quantitative results on only 30 objects. It's very easy to get biased results, and the conclusions are likely to change if we choose another set of 30 objects. As a result, I strongly urge the authors to follow the conventions of previous papers and rigorously compare the methods on the entire GSO dataset with careful alignment between prediction and ground truth before calculating metrics. Additionally, it is highly suggested to include evaluations on other datasets, especially some real-world object datasets, and user studies. Otherwise, I cannot support the acceptance of the paper without convincing justification.\n\n2. The high-level pipeline of \"multi-view normal generation with 2D diffusion models and 3D reconstruction with normal-based optimization\" is not first proposed in this paper. For example, Wonder3D shares a very similar high-level pipeline, which greatly limits the contribution or novelty of the paper. While the paper proposes a novel and interesting reconstruction module, ISOMER, it has not been carefully analyzed separately. For instance, I would like to know whether the performance gains come from better multi-view prediction or a better multi-view normal-to-3D reconstruction module. An interesting experiment to include would be a direct comparison between the reconstruction modules of Wonder3D and ISOMER given the same multi-view normal maps. Only after a detailed ablation study can readers choose a better multi-view normal generation module and a better reconstruction module.\n\n3. The multi-view generation part lacks significant novelty, mainly consisting of existing known techniques related to 2D diffusion models.\n\n4. The ablation study is quite lightweight, and many important experiments are missing:\n\n   (a) The ISOMER module includes multiple stages. I suggest the authors include both the intermediate and final results of the module. A quantitative evaluation would also be beneficial.\n\n   (b) The method utilizes multiple 2D diffusion models to upscale the multi-view RGB (and normal?) images. It would be interesting to see the multi-view results before and after upscaling (resolutions at 256, 512, and 2048) and their impact on the final 3D models.\n\n   (c) For the \"explicit target,\" only the geometry results are shown. What is the effect on texture?\n\n5. The paper claims that \"generate tens of millions of faces within seconds\" as an advantage. However, I don't believe so. First, it's not difficult for existing methods to generate dense meshes efficiently. Also, dense meshes themselves are not required by applications but rather detailed and sharp geometry. In fact, many downstream applications prefer meshes with more compact faces and cannot tolerate tens of millions of faces.\n\n6. Line 170 states, \"Finally, the mesh is corrected after iteration through edge collapse, edge split, and edge flip to maintain a uniform face distribution and reasonable edge lengths.\" This introduction is too brief as it involves many operations but lacks the motivation and implementation details for each operation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and thorough comments. Your main suggestions on the experimental setting and ablation studies help us refine our paper. To address your concerns, we present the point-to-point response as follows.\n\n**Comment 1: Narrow Evaluation Scope**\n\nThanks for your thorough comment. GSO (Google Scanned Objects) is a set of real-world objects scanned by Google using specialized equipment for research. We chose 30 random objects in alignment with the standards set by previous work (CRM, SyncDreamer, Wonder3D). Given that SyncDreamer takes up to 50 hours to generate 30 objects and results from other papers show that SyncDreamer performs significantly worse than existing methods, we exclude SyncDreamer and **conduct an experiment** on the entire GSO dataset. The results for all methods are provided in Table 1. Additionally, following your suggestions, we will add more experiments on other real-world object datasets, such as MVImageNet, and provide a detailed evaluation in the revised paper. \n\nFor the user study, as shown in Appendix F, we rendered 360-degree videos of subject-driven 3D models and presented each volunteer with five samples of rendered videos from a random method.  Volunteers rated the samples on four aspects: 3D consistency, subject fidelity, prompt fidelity, and overall quality on a scale of 1-10, with higher scores indicating better performance. We collected results from 30 volunteers, as shown in Table 2 in the Appendix. Our findings indicate that users significantly preferred our method across these aspects.\n\n**Comment 2: Insufficient Analysis of ISOMER**\n\nWe appreciate your insightful comment. Addressing the concern about the novelty in the multi-view generation aspect, we will provide a unified response in the subsequent question. To resolve your concern regarding ISOMER, we will **revise its introduction** to emphasize the key insight of each component within ISOMER. The first step of ISOMER ensures topological consistency, while the second step efficiently reconstructs rough geometries, and the third step improves surface reconstruction accuracy by handling multi-view inconsistencies with ExplicitTarget.  \nOur experiments indicate that ISOMER is capable of enhancing consistency as demonstrated in Figure 5. We **include a new comparative evaluation** experiment that contrasts Wonder3D with and without ISOMER to provide a clearer understanding of the benefits in Table 1.\n\n**Comment 3: Lack of Novelty in Multi-view Generation**\n\nWe thank the reviewer for raising this concern. The main focus of our paper is on addressing current limitations within current multi-view reconstruction approaches, exemplified by Wonder3D. Existing methods often suffer from low resolution, multi-view inconsistency, and low generation speed. Our methodology is specifically tailored to address each of these issues within the existing framework, rather than proposing an entirely new pipeline. \n\nHere's a concise summary of our approach:\n\n- Noval Reconstruction Algorithm: For the first time in such a pipeline, we have introduced a mesh-based reconstruction algorithm, which has gained significant improvements in speed and quality.\n\n- Quality Enhancement: We have developed techniques to improve the resolution of reconstructed models, providing higher fidelity and detail.\n\n- Consistency Improvement: Our approach includes mechanisms to reduce inconsistencies across different views, leading to more coherent and reliable 3D reconstructions.\n\n**Comment 4: Lightweight Ablation Studies**\n\nWe are grateful for your comprehensive comments. Recognizing the need for a more robust ablation study, we have taken the following steps to enhance our original analysis:\n\n(a) Quantitative Evaluation of the ISOMER Module: We have incorporated a detailed quantitative assessment of the ISOMER module to better demonstrate its contribution to overall performance.\n\n(b) Resolution Impact Analysis: We have expanded our study to qualitative comparisons across various resolutions to see how different resolutions impact the final 3D reconstruction outcomes.\n\n(c)  Explicit Target's Effect on Texture: Mirroring the approach used for geometry, we will include additional experimental results that illustrate the impact of the Explicit Target method on texture quality in the revised manuscript. We show the difference in Figure 1.\n\n**Comment 5: Advantage of Generating Dense Meshes**\n\nWe thank the reviewer for raising this concern. We believe that detailed and sharp geometry does require a sufficient number of surfaces. As illustrated in Figure 6 of the main text, the intricacies of text engraving cannot be achieved without enough surfaces. For scenarios that prefer more compact meshes, such as in gaming or other graphics-intensive applications, various existing mesh decimation and retopology techniques can be applied to achieve the desired outcome. Since this process is quite engineering-oriented, we did not include it within the scope of our paper. \nIt is worth noting that while simplifying a high-precision dense mesh model to a more compact one is feasible, the inverse—enhancing a model with fewer details to a high-precision one—is significantly more challenging.  This consideration is a primary reason our paper emphasizes the optimization of generation accuracy and surface count, aiming to address the complexities involved in achieving high-fidelity geometric detail.\n\n**Comment 6: Lack of Detailed Explanations**\n\nThanks for pointing out the issue. We will **modify our statement** in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qxmf7Om34W", "reviewer": "Reviewer_GBd8", "summary": "The paper focuses on single-image-to-3D. Given a single image, it first finetunes Stable Diffusion Image Variations models to generate orthogonal multi-view RGB/normal images and ControlNet-Tile models to enhance resolution. Then, it proposes the instant and consistent mesh reconstruction algorithm (ISOMER). It has three stages: 1) Use the estimated depth (from predicted normal maps) of the front and back views to initialize the geometry. 2) Use differentiable rendering to optimize geometry with mask and normal loss. 3) Optimize geometry and vertex colors to fit the view-weighted color, where each view’s weight is the square of cosine between vertex normal and view direction. The total time required is claimed to be within 30 seconds. It compares with some recent approaches on 30 GSO objects and achieves better performance in terms of both visual and geometry quality.", "questions": "1. Please properly cite, acknowledge, and compare with previous works if similar ideas are applied. For example, in the Multi-view Image Generation, two key points (L499-502) seem to be the same as ImageDream [15]. And, it also uses IP-Adapter (L515). A reference U-Net is also used in Zero123++ [46]. Please cite related work and explain the differences when introducing the proposed method. This helps readers understand how ideas are inherited and improved.\n2. No notations in Eq. 1 are explained. I can understand the point. But it is not reader-friendly. Please delete it or explain it. The formula itself is also over-simplified.\n3. Wrong references of figures, and algorithms. L181, L188, L557, etc.\n4. In Algo. 1, the variable cnt is useless and not explained. The set ‘colored’ is initialized as empty but assumed as initialized by C (L13). The authors may consider putting it more clearly: For invisible vertex coloring, it applies the flood fill algorithm, where each vertex is colored by the average color of the neighboring colored vertices. Use proper terms and make your writing concise.\n5. Is Expansion (L181) referring to applying normal-based loss L_normal?\n6. L204: the result for vertex v → the predicted color of vertex v. Make it more clear.\n7. Time cost of each ISOMER step.", "limitations": "Many significant limitations of the approach are not discussed. Please refer to the Weaknesses.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- **High-Resolution Appearance at Input View:**\n    \n    It demonstrates improvements in the resolution of the generated appearances, especially at the input view. Unlike previous works that produced lower-resolution multi-view outputs, resulting in a lack of clarity and detail, this paper utilizes diffusion models to achieve effective super-resolution. This allows the generated views to maintain high resolution and exhibit more details (reimagined though).\n    \n- **Enhanced Geometry:**\n    \n    The paper showcases impressive geometry, particularly in examples of garage kit figures. Technically, this work better leverages predicted normal map information to optimize geometry and color, surpassing previous models like Wonder3D, which also predicted normal maps. This enhanced utilization of normal maps carves more geometric details.\n    \n- **Many Details and Supporting Materials:**\n    \n    It provides many network parameters, predicted meshes, and rendering videos, which are valuable for assessing the performance and effectiveness of the proposed methods.", "weaknesses": "- **Limited Qualitative Results to Frontal View, Worse Side Views:**\n    \n    The majority of qualitative results presented in the paper are focused on the frontal view, as it is prioritized by the approach. However, for non-frontal views, there are issues such as visible seams and artifacts. For example, the video in supplementary material shows several instances where characters have duplicated ears. Although the frontal view quality is enhanced, this improvement comes at the expense of other views, which suffer from more noticeable artifacts. \n    \n- **Potential Bottleneck by the Two-view Initialization of First Step**\n    \n    Following the previous point, those artifacts are potentially influenced by the algorithm’s initialization process since only front and back views are used. Artifacts like dual-ear, if introduced in the first step of ISOMER, may not be easily corrected via subsequent refinement. What if the input view is not exactly frontal and more ill-posed? The artifacts can be even more severe.\n    \n- **Narrow Evaluation Scope, Limited Geometry Complexity:**\n    \n    The paper predominantly evaluates small cartoon figures and animal models in terms of qualitative evaluation, with limited representation of real-world objects encountered in daily life. Real-world objects may have more complex geometry, where four orthogonal views are not enough to cover. Even for simple objects like a mug, the method may fail to generate a solid bottom and an empty interior given that they are unseen.\n    \n- **Pose Sensitivity and Distortion Issues:**\n    \n    The network outputs exhibit significant distortions when input images are not taken from a frontal view with the elevation equal to zero. For instance, a machine example in the supplementary video shows clear distortion when the input image is taken from an elevated side angle. This indicates a lack of robust data augmentation during training, leading to heavy restrictions on input poses.\n    \n- **Oversimplified Texture Handling of Occluded Region:**\n    \n    The approach to unseen vertex coloring is overly simplistic, using flood-fill colors to interpolate occluded areas. This may result in incorrect color transitions for complex geometries with significant occlusions. Even for simple cases, the bottoms or tops of objects are often unseen from 4 orthogonal views. Vertex colors in those regions are roughly smoothed out.\n    \n- **Small Evaluation Set:**\n    \n    The evaluation set consists of only 30 objects, which is insufficient, especially considering all of those baselines are not purely optimization-based, whose inference time costs per image are no longer than a few minutes. A few hundred objects will be necessary. For example, GRM [63] uses 250 shapes for single-view reconstruction evaluation, providing more robust and reliable assessments. \n    \n- **Unclear Writing:**\n    \n    The paper contains several sections where the writing is unclear. Please refer to the points listed in the Questions.\n    \n- **Lack of Quantitative Ablation Studies:**\n    \n    It does not include any quantitative evaluation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and valuable comments, some of which are fundamental and in-depth suggestions that help us greatly improve our paper. To address your concerns, we present the point-to-point response as follows.\n\n**Comment 1: Limited Qualitative Results to Frontal View, Worse Side Views**\n\nThanks for bringing this comment up, and we acknowledge the presence of this issue. Our investigation indicates that it stems from occasional inconsistencies in the multi-view predictions, which is a challenge inherent in all current multi-view reconstruction solutions. We are actively exploring ways to mitigate these inconsistencies to enhance the robustness and reliability of our reconstruction process.\n\n**Comment 2: Potential Bottleneck by the Two-view Initialization of First Step**\n\nThanks for your insightful question. As introduced in our main text, the initialization process is specifically designed to guarantee the correct overall topological structure and is unrelated to reconstruction details. The majority of artifacts are primarily derived from inconsistencies in multi-view predictions rather than the reconstruction process. As shown in Figure 9, our method can yield satisfactory results even with a sphere as the initial shape. Furthermore, Figure 5(a) demonstrates that the ISOMER reconstruction step is capable of correcting some of the artifacts introduced by multi-view predictions. This showcases the resilience and corrective capabilities of our approach in refining the final reconstruction output. \n\n**Comment 3: Narrow Evaluation Scope, Limited Geometry Complexity**\n\nThanks for your insightful question. For a consideration of fairness, the samples we have chosen for qualitative evaluation are largely based on previous methods, thus not incorporating diverse real-world objects. However, in our quantitative evaluation using the GSO dataset, a set of real-world objects scanned by Google, our method significantly outperforms existing methods. Our approach excels in capturing fine details, showcasing our proficiency in reconstructing complex geometries with high fidelity.\n\nRegarding geometry complexity, our method is comparable to existing works such as Wonder3D, OpenLRM, and InstantMesh. It's important to note that, like these methods, we cannot generate unseen features such as \"a solid bottom and an empty interior of a mug\" when these aspects are not present in the input views. This limitation is inherent to the field and represents a challenge for all methods dealing with incomplete view data.\n\n**Comment 4: Pose Sensitivity and Distortion Issues**\n\nThanks for pointing out the issue. To resolve your concerns, we **add a new test** with rotated objects in Table 2 to test robustness in non-front-facing views. The test results show that our method still performs well in this case, and even the geometry prediction is more accurate.\n\n**Comment 5: Oversimplified Texture Handling of Occluded Region**\n\nWe appreciate your detailed comments. We agree that our current method for handling colors in non-visible areas is quite straightforward. The simplicity of this approach allows for extremely efficient processing times, ensuring the overall efficiency and stability of our workflow. We are committed to exploring more sophisticated coloring techniques in future work to enhance the visual output of our reconstructions.\n\n**Comment 6: Small Evaluation Set**\n\nThanks for your thorough comment. We chose 30 random objects in alignment with the standards set by previous work (CRM, SyncDreamer, Wonder3D). Given that SyncDreamer takes up to 50 hours to generate 30 objects and results from other papers show that SyncDreamer performs significantly worse than existing methods, we exclude SyncDreamer and **conduct an experiment** on the entire GSO dataset. The results for all methods are provided in Table 1.\n\n**Comment 7: Lack of Quantitative Ablation Studies**\n\nWe are grateful for your comprehensive comments. Recognizing the need for a more robust ablation study, we have taken the following steps to enhance our original analysis:\n\n(a) ISOMER Module Analysis: We will incorporate a comparative experiment of the ISOMER module to better demonstrate its superiority over existing reconstruction algorithms.\n\n(b) Explicit Target Algorithm Analysis: We will show qualitatively how the ExplicitTarget algorithm improves the final performance.\n\n(c) Robustness analysis: we will quantitatively study the performance and differences of our method under non-front inputs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FRk264iTUD", "reviewer": "Reviewer_inWh", "summary": "This paper introduces a novel method to reconstruct 3D from a single image. The method is two-stage: (i) 4 images corresponding to the orthographic views are first generated along with normal maps, (ii) then, the multi-view maps are used to initialize and optimize a mesh using differentiable rendering techniques which are adapted to this sparse-view setting. This second stage contrasts with recent image-to-3D methods which typically train large feed-forward reconstruction models. Extensive comparisons are conducted on (i) samples from original papers for a qualitative evaluation and (ii) on the standard Google Scanned Objects (GSO) dataset for quantitative results. In both cases, the proposed approach performs significantly better than the presented baselines.", "questions": "Questions:\n- Eq (2): how is the ground-truth mask computed?\n- What happens when the input view does not correspond to the frontal orthographic view, e.g. say with elevation=20 and azimuth=45? \n- L171 what are the details behind edge collapse/split/flip?\n\nRemarks:\n- the terms \"wild image\", \"wild views\" do not sound correct, the appropriate term would be \"in-the-wild image\" but this typically corresponds to random real-world images that one could find on internet or social medias, which is not the case here. I would suggest removing this aspect which is not necessary for the paper storyline\n- the term \"ExplicitTarget optimization\" is not crystal clear out of context, I would suggest finding another term that clearly conveys what is under the hood, e.g. visibility-aware supervision", "limitations": "Yes, there is a brief limitation section. Including visuals illustrating failure cases would greatly help the readers better understand the model limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] **Novelty.** The proposed approach is interesting and novel in different aspects. The first stage aims at generating multi-view images and normal maps from a single image; this idea is not novel per se and was introduced/studied by prior works like Wonder3D. However, it is worth noting that it technically differs a bit from Wonder3D and there is an additional effort to generate high resolution outputs. The second stage about 3D mesh reconstruction using differentiable rendering is elegant, effective and rather novel. In particular, it contrasts with recent image-to-3D methods which typically train large feed-forward reconstruction models tailored for this task.\n\n[S2] **SOTA performances.** For both the qualitative and quantitative evaluations, the proposed approach showcases results that are significantly better than prior works. Visually, the method not only generates more detailed textures but also more accurate geometry.\n\n[S3] **Sound experiments.** The authors conducted extensive experiments that are sound and validate the proposed method. They not only compare the method to an exhaustive set of state-of-the-art competitors both qualitatively and quantitatively, but also conducted ablation studies on some method components.", "weaknesses": "[W1] **Unrigorous technical presentation.** While the high-level overview of the approach is clear, some parts of the technical presentation lack clarity and contain incoherences. For example:\n- Eq (1) looks wrong: a depth map can be obtained by integrating gradients of normals not the normals themselves\n- Eq (5) is incoherent and lacks clarity: is $i$ an integer or an image? If $\\mathcal{I}$ is a set of images, $i \\in \\mathcal{I}$ is an image, but $V_M(v, img)$ is undefined. The rightmost part of the equation $\\sum_i V_M(v, \\mathcal{I})$ does not make sense either. This part need revisions, $i$ can not be at the same time: an integer indexing the ground-truth views $\\mathcal{I_m}$, a sample view image from $\\mathcal{I}$ or an integer indexing the sample views $\\mathcal{I}$. I would define this process for a single target views $I_t$ given the source views $I_s^1, ... I_s^N$, and then define the loss in Eq (6) using the sum over the pseudo ground-truth target views.\n\nThis lack of rigour harms readability and resolving these issues would greatly strengthen the paper quality.\n\n[W2] **Missing ablation studies.** I would expect more ablation studies to better understand the impact of each component that are different from prior works. Currently, the two ablation experiments correspond to two very technical aspects which are a regularization term (called Expansion) and the explicit target technique. Experiments assessing the impact of the following components are missing:\n- the image-to-multiview stage (e.g., compare results by replacing it with Wonder3D image-to-multiview stage)\n- the super-resolution (e.g., compare results with/without super-resolution network)\n- the multiview-to-3D stage (e.g., compare results by replacing it with Wonder3D multiview-to-3D stage)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and valuable comments, some of which are fundamental and in-depth suggestions that help us greatly improve our paper. To address your concerns, we present the point-to-point response as follows.\n\n**Comment 1: Unrigorous technical presentation**\n\nWe appreciate your observation regarding the clarity of the notations and explanations in Equations (1) and (5). In response to your feedback, we will make the revisions in the updated manuscript to enhance understanding.\n\nFor Equation (1), here $d(i,j)$ denotes the value of coordinate (i,j) in the depth map, while $x$ is the integration process along the vertical line $y=j$, $\\vec{n}(x)$ denotes the vector of the input normal field at the position $x$. $n_x$ is the component of the normal field $\\vec{n}$ along the direction of the x-axis (which is a scalar function). The confusion might arise because the letter $x$ is used in two different contexts. To clarify, we’ll make the following adjustments: We will use the formula $d(i,j)=∑t=0inx(t,j)d(i,j)=∑t=0inx(t,j)$, where $d$ and $n_x$ are considered as matrices in the discrete version. This change aligns better with how the variables are handled in the code.\n\n For Equation (5) (***ExplicitTarget***). Let  $Avg(V, W) = \\frac{\\sum_i{V_i W_i}}{W_i}$ represent the weighted average function, and $V_{M}(v, i): (\\mathbb{N}^+, \\mathbb{N}^+) \\rightarrow \\{0, 1\\}$ represent the visibility of vertex $v$ in mesh $M$ under view $i$. $Col_M(v,i) $ Indicate the color of vertex v in viewpoint i.  We compute the ExplicitTarget $ET$ of each vertex in mesh $M$ as\n$$\\\\begin{equation}\n    ET_{M}(v) = \\\\begin{cases}\nAvg(Col_M(v,i), V_{M}(v, i) W_M(v, i)^2) \n & \\text{, if } \\sum_{i} V_{M}(v, \\mathcal{i}) > 0 \\\\\\\\ \n \\mathbf{0} & \\text{, otherwise, } \n\\\\end{cases} \n\\end{equation}$$\nwhere $W_M(v, i) = -\\cos(N_{v}^{(M)}, N_{i}^{(view)})$ is a weighting factor, $N_{v}^{(M)}$ is the vertex normal of $v$ in mesh $M$, and $N_{i}^{(view)}$ is the view direction of view $i$. \n\n$ET_M$ computes the results with view direction weighting so that each viewpoint does not introduce significant errors for surfaces that are too skewed in the current viewpoint (because predictions in these places are usually inaccurate).\n\n**Comment 2: Missing ablation studies**\n\nWe appreciate the depth of your feedback. To address your concern, we will enrich our paper with the following **additional ablation studies** to elucidate the contributions of each component within our model:\n\n(a) ISOMER Module Analysis: We will **incorporate a comparative experiment** of the ISOMER module to better demonstrate its superiority over existing reconstruction algorithms.\n\n(b) Explicit Target Algorithm Analysis: We will show **qualitatively how the ExplicitTarget algorithm improves** the final performance.\n\n(c) Robustness analysis: we will **quantitatively study** the performance and differences of our method under non-front inputs.\n\n(d) Resolution Impact Analysis: We will expand our study to **include a qualitative comparison** across various resolutions in order to demonstrate the differences between different resolutions.\n\n**Question 1.** Thanks for your insightful question. The ground truth mask is determined based on the normal map predicted by the model. Predicted pixels with an RGB magnitude far from 1 are considered background.\n\n**Question 2.** Thanks for your insightful question. To resolve your concerns, we **add a new test** with rotated objects in Table 2 to test robustness in non-front-facing views. The test results show that our method still performs well in this case, and even the geometry prediction is more accurate.\n\n**Question 3.** We appreciate your attention to this matter. In response, we will **enhance the clarity and details** of our explanation in the revised version as follows:\n\n- Edge Collapse: This operation is used to avoid and heal defects in the mesh. It involves selecting an edge within a triangle and collapsing it to the other edge, effectively merging the two triangles into a single triangle. This process can help to eliminate narrow triangles that might be causing issues in the mesh, such as those that are too thin to accurately represent the surface they are approximating.  It prevents the creation of topological artifacts and maintains mesh quality.\n\n- Edge Split: This is the opposite of edge collapse. In the edge split, an edge that is longer than a specified maximum length is divided into two, creating new vertices at the midpoint of the edge. This operation is used to refine the mesh, ensuring that the local edge length is kept close to the optimal length. It helps to maintain the quality of the mesh by avoiding edges that are too long, which could lead to an inaccurate representation of the surface.\n\n- Edge Flip: Edge flip is an operation that adjusts the connectivity of the mesh to improve its quality. It involves flipping an edge within a triangle to connect two non-adjacent vertices, effectively changing the triangulation of the mesh. This can help to maintain the degree of the vertices close to their optimal value, which is typically six for internal vertices (or four for boundary vertices).\n\nThese operations aim to improve mesh quality, avoid defects, and ensure an accurate representation of the target geometry.\n\n**Remark 1.** Thanks for pointing out the issue. We will **modify the statement** in our revision.\n\n**Remark 2.** Thanks so much for your great suggestions. We are definitely considering adopting a name that is more straightforward and easily comprehended.\n\nWe appreciate your detailed comments. We will thoroughly revise our paper. Thanks again for your time and in-depth suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their patience in reviewing, and we will respond to each of them individually, with additional experiments added in the Appendix PDF.

Meta-review: This paper was reviewed by four experts in the field. Although the initial review is mixed, after rebuttal and discussion, all reviewers converge to acceptance, and the final rating is 5, 6, 6, 7. All reviewers agreed that this is a high-quality work, with clear presentation, simple but effective ideas, and high-quality results. AC also agrees that this is well-presented and important work and thus would suggest acceptance.

Still, we recommended the authors to carefully read all reviewers’ final feedback, and revise the manuscript as suggested in the final camera-ready version, particularly about ablation study and more diverse experiment. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)