Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention
Authors: Peng Li
Yuan Liu
Xiaoxiao Long
Feihu Zhang
Cheng Lin
Mengfei Li
Xingqun Qi
Shanghang Zhang
Wei Xue
Wenhan Luo
Ping Tan
Wenping Wang
Qifeng Liu
Yike Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D generation
multiview attention
epipolar
Primary area: generative_models

Abstract: In this paper, we introduce **Era3D**, a novel multiview diffusion method that generates high-resolution multiview images from a single-view image. Despite significant advancements in multiview generation, existing methods still suffer from camera prior mismatch, inefficacy, and low resolution, resulting in poor-quality multiview images. Specifically, these methods assume that the input images should comply with a predefined camera type, e.g. a perspective camera with a fixed focal length, leading to distorted shapes when the assumption fails. Moreover, the full-image or dense multiview attention they employ leads to a dramatic explosion of computational complexity as image resolution increases, resulting in prohibitively expensive training costs. To bridge the gap between assumption and reality, Era3D first proposes a diffusion-based camera prediction module to estimate the focal length and elevation of the input image, which allows our method to generate images without shape distortions. Furthermore, a simple but efficient attention layer, named row-wise attention, is used to enforce epipolar priors in the multiview diffusion, facilitating efficient cross-view information fusion. Consequently, compared with state-of-the-art methods, Era3D generates high-quality multiview images with up to a 512×512 resolution while reducing computation complexity of multiview attention by 12x times. Comprehensive experiments demonstrate the superior generation power of Era3D- it can reconstruct high-quality and detailed 3D meshes from diverse single-view input images, significantly outperforming baseline multiview diffusion methods.

Era3D: High-Resolution Multiview Diffusion Using Efficient Row-wise Attention
Peng Li
Yuan Liu
Xiaoxiao Long" 1.21
Feihu Zhang"
Cheng Lin
Mengfei Li'
Xingqun Qi'
Shanghang Zhang
Wenhan Luo'
Ping Tan 1.5
Wenping Wang"
Qifeng Liu' Yike Guo
"Equal contribution 'Corresponding author
HKUST HKKU DreamTecch 4PKU LightIllusion
Abstract
In this paper, we introduce Era3D, a novel multiview diffusion method that gen- erates high-resolution multiview images from single-view image. Despite sig- nificant advancements in multiview generation, existing methods still suffer from camera prior mismatch, inefficacy, and low resolution, resulting in poor-quality multiview images. Specifically, these methods assume that the input images should comply with predefined camera type, e.g. perspective camera with fixed focal length, leading to distorted shapes when the assumption fails. Moreover, the full- image dense multiview attention they employ leads a dramatic explosion of computational complexity as image resolution increases, resulting in prohibitively expensive training costs. To bridge the gap between assumption and reality, Era3D first proposes diffusion based camera prediction module to estimate the focal length and elevation of the input image, which allows our method to generate im- ages without shape distortions. Furthermore, a simple but efficient attention layer, named row-wise attention, used to enforce epipolar priors in the multiview diffu- sion, facilitating efficient cross-vies view information fusion. Consequently, compared with state-of-the -tettth-rtt methods, Era3D generates high-quality multiview images with up a 51 512 resolution while reducing computation complexity of multiview attention by 12x times. Comprehensive experiments demonstrate the superior gener- ation power of Fra3D- it can reconstruct high-quality and detailed 3D meshes from diverse single-view input images, significantly outperforming baseline multiview diffusion methods. Project page: lnttps://penhhyyx, github 1o/ Era3D/
1 Introduction
3D reconstruction from single-view images an essential task in computer vision and graphies due to its potential applications in game design, virtual reality, and robotics. Early research [42 44 53 78
28 mainly relies on direct 3D regression on voxels [39] 58 99, which often leads to oversmoothed
results and has difficulty in generalizing to real-world unseen objects due limited 3D training
data [ Recently, diffusion models (DMs) [16] 50 show strong generation ability on image or video
synthesis by training extremely large-scale datasets 47 48). These diffusion models are promising
tools for single-view 3D reconstruction because is possible to generate novel-view images from the given image to enable 3D reconstruction.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input images
Generated images
Generated normal maps
Output mesh
Figure Given single-view image with arbitrary intrinsic and viewpoints, Era3D can generate high-quality multiview images with resolution of 512 x 512 on the orthogonal camera setting, which can be used in mesh reconstruction by NeuS (68)
To utilize image DMs for single-view 3D reconstruction, pioneer work DreamFusion [46] tries to
distill a 3D representation like NeRF 36 or Gaussian Splatting [62] from a 2D image diffusion by
Score Distillation Sampling (SDS) loss and many follow-up works improve the distillation-basse
methods in quality (71) and efficiency 62). However, these methods suffer from unstable convergence
and degenerated quality. Alternatively, recent works such as MVDream [53]. SyncDreamer [33]
Wonder3D (34] and 13++ (54) explicitly generate multiview images by multiview diffusion [64]
33 and then reconstruct 3D models from the generated images by neural reconstruction methods [68
or large reconstruction models (LRMs) [19] 27). Explicitly generating multiview images makes
these methods more controllable and efficient than SDS methods and thus is more popular in the single-view 3D reconstruction task.
Despite impressive ad advanementss in multiview diffusion methods [55]
34
33
32
31
64 63
27.. efficiently generating novel-view images for high-quality 3D reconstruction remains an open challenge. There three noticeable chal- lenges in the current approaches. (1) Inconsis- tent predefined camera type. Most multiview diffusion methods assume that the input images are captured by camera with predefined focal length. This leads to unwanted distortions when input images are captured by cameras with dif- ferent camera types or intrinsics, as exemplified in Fig, (e.g.. Wonder3D's assumption of an or- thogonal camera leads to distorted meshes when the input image is captured by perspective camera with a small focal length). (2) Inefficiency of multiview diffusion. Multiview diffusion methods usually rely on multiview attention layers to exchange information among different views to generate multiview-consistent images. However, these multiview attention layers are usually implemented by extending the self-attention in Stable
Input
Generated images and mesh
Figure 2: (top) Perspective input images for Won- der3D produce extreme distortion in the generation. (bottom) Era3D can handle images of commonly used intrinsics.

---

54
52
52
(a) Dense Attention
(b) General Camera Setting
(c) Epipolar Attentonn O( 5 K)
(d) Canonical Camera Setting
(e) Row-wise Atiention
((N25*)
O(N S*)
Figure 3: Different types of multiview attention layers. (a) dense multiview attention layer, all feature vectors of multiview images are fed into attention block. For a general camera setting (b) with arbitrary viewpoints and intrinsics, utilizing epipolar constraint to construct an epipolar attention (c) needs to correlate the features on the epipolar line. This means that need to sample K points along each epipolar line to compute such an attention layer. In our canonical camera setting (d) with orthogonal cameras and viewpoints on elevation of 0%, epipolar lines align with the row of the images across different views (e), which eliminates the need to resample epipolar line to compute epipolar attention. We assume the latent feature map has resolution of H x W and H = W = S. In such a N-view camera system, row-wise attention reduces the computational complexity to (SN
all multiview images, which is called dense multiview attention in Fig 3 and results in significant increase in computation complexity and memory consumption. (3) Low resolution of generated images. The limitation above restricts most existing multiview diffusion models to resolutions of 256 x 256, preventing them from reconstructing detailed meshes. Addressing above challenges is crucial for developing practical and scalable multiview diffusion methods.
Diffusion [52]
In this paper, we introduce Era3D, a novel multiview diffusion method that efficiently generates high-resolution (5 (512x512) consistent-multiview images for single-view 3D reconstruction. Unlike existing methods, Era3D allows images of commonly used camera types as inputs while mitigating the unwanted distortion brought by different camera models.
To this end, we employ a unique approach: using different camera models for input images and the generated ones for training, meaning that the input images are allowed have arbitrary focal lengths and elevations while generated images are with orthogonal cameras and fixed viewpoints of OP elevations. However, this requires DMs to implicitly infer and rectify the focal lengths and viewpoints of input images in the generation process, which is a challenging task and degrades the generation quality. To overcome this challenge and improve generation quality, we propose novel regression and condition scheme and utilize the low-level feature maps of UNet at each denoising step to predict camera information of input images. We find that such regression and condition scheme facilitates much more accurate camera pose prediction than existing methods (31) and Icads more details in the generation. As shown in Fig. [ Era3D successfully avoids the above distortion problem brought by the different camera types and focal lengths.
Moreover, drawing inspiration from epipolar attention [65]. Era3D enables efficient training for nigh-resolution multiview generation by introducing a novel row-wise multiview attention. Epipolar constraint can be utilized to constrain the attention regions across views and thus improve attention efficiency. However, directly applying such epipolar attention [65] for a general camera setting (Fig. ()) is still memory and computationally inefficient because one would have to sample multiple points on epipolar lines for attention. This would require to construct a 3D grid of features in the view frustums for multiview images, which is too slow and memory-consumingg In contrast, since Era3D generates images with orthogonal cameras on viewpoints of 0", we find that epipolar lines in our camera setting are aligned with pixel rows of images across different views (Fig. d)), which enables us to propose an efficient row-wise attention layer. Compared with dense multiview attention. row-wise attention significantly reduces memory consumption (35.32GB v.s. 1.66GB), and the computation complexity (220.41ms v.s. 2.23ms) of multiview attention (Fig.Be)) e)). Even with Xformers 266,, an accelerating library for attention, the efficiency of row-wise attention still outperforms existing methods by approximately twelve-fold as evident in Tab. 3 Consequently, the proposed row-wise attention allows us to easily scale Era3D to a high resolution of 512x512 to reconstruct more detailed 3D meshes.
3

---

Overall, our main contributions are summarized as follows: (1) Era3D is the first method that tries to solve the distortion artifacts brought by the inconsistent camera intrinsic 3D generation; (2) we design a novel regression and condition scheme to enable diffusion models to take images of arbitrary cameras as inputs while outputring the orthogonal images on the canonical camera setting; (3) we propose now-wise multiview attention, an efficient attention layer for high-resolution multiview image generation; (4) our method achieves state-of-the-ar performance for single-view 3D generation [10].
2
Related Works
7 which concentrate on per-scene optimization based on Score Distillation Sampling 46 67 emphasize feed-forward 3D) generation.
Our study is primarily centered on the domain of -mage-to-3D. Unlike early works [6 49 62
29
we
Image to 3D. Generating 3D assets from images has been extensively researched, paralleling the 18] A stream of these
development of GAN 13
23 38) and diffusion models (DMs) [57 17
works 20 79 5
19 76
directly produce 3D representations, like SDF
43
9 411 NeRF [36] [4]
D. Triplane 3
32 and subsequent works (44 3T represent the scene diffusion model conditioned on reference image and camera
15
56], Gaussian [24] 59 or 3D volume (60). Zero-I
pose. LRM- methods [19 27 70 74 employ a large transformer architecture to train a triplane representation with data-driven approach. Another technical line involves generating consistent multiview images first [55 69 54 72), and then robustly reconstructing 3D shapes with NeuS [3], Despite significant advancements, challenges remain in reconstruction quality, training resolution, or efficiency.
Gaussian Splatting 61 19 75 or LRM
Multiview diffusion. Cross- consistency is critical in 3D reconstruction and generation, relying on multiview feature correspondence to estimate 3D structures. MVffusion [64] first proposes generating multiview images in parallel with correspondence-awant attention, facilitating cross-view information interaction, and applies to texture scene meshes.
(65] [2] introduces epipolar features into DM to enhance fusion between viewpoints. Zero [54] tiles multi-views into a single image and performs a single pass for multiview generation, which is also used in Direct2.5 and Instant3D.
MVDream [55] and Wonder3D
34 also design multiview self-attention to improve multiview composes multiview features into 3D volumes, conducting 3D-aware fusion in 3D noise space. All of the aforementioned methods share the same idea: modeling 3D generation with multiview joint probability distribution. Other works [8 66] explore the priors from video diffusion model to achieve consistent multiview generation. Following widely used multiview self-attention, we propose more efficient row-wise attention to reduce computation workloads, but without loss of multiview feature interaction.
consistency. Syncdreamer 33
55] are trained with fixed focal lens. When inference, one also needs provide elevation of input for better performance. Further research seeks to mitigate this issue by leveraging fixed poses [34] [54] or incorporating additional elevation prediction modules 30 LEAP (20] uses pixel-wise similarity, rather than estimated poses, to aggregate multiview features. However, none of these methods consider the distortion eгror caused by cameras, which can severely affect the reconstruction of real-world data. To overcome this, we opt to generate images at fixed views in the canonical orthogonal space, with simultaneous predictions of elevation and focal distortion.
Camera pose. For 3D generation, early works 32
3 Methods
Era3D is proposed to generate a 3D mesh from a single-view image. The overview is shown in Fig. 4
consisting of three key components. Given input image with commonly used focal length and arbitrary viewpoint, Era3D generates multiview images in a canonical camera setting as introduced in Sec. [3.]] To improve the generation quality, Sec. [3.2] we propose a regression and condition scheme, enabling diffusion models to predict accurate camera pose and focal lengths and guiding the denoising process. Finally, we considerably reduce memory consumption and improve computation efficiency by proposing row-wise multiview attention (Sec. (.3). which exchanges information among multiview images to maintain multiview consistency. Finally, we reconstruct the 3D mesh from the generated images and normal maps using neural reconstruction methods like NeuS (68).

---

Max Pooling
Gravity Direction
EFReg
-90-
180
Position Embedding
a,f
90
Neu$
MVSD
Image on Various Intrinsies Viewpoints
MV Imuges in Camonical Camern Setting
Textured Mesh
Figure 4: Overview. Given single-view image as input, Era3D applies multiview diffusion to generate multiview consistent images and normal maps the canonical camera setting, which enables us to reconstruct 3D meshes using NeuS 68 37
3.1 Camera Canonicalization
Perspective distortion problem. Existing multiview diffusion methods 33 54 32 assume that the input image and the generated images share the same fixed intrinsic parameters. However, this assumption is often violated in practice, as input images may be captured by arbitrary cameras with varying focal lengths. When the input image has a different intrinsic matrix from the assumed one, the generated multiview images and reconstructed 3D meshes will exhibit perspective distortion, as illustrated in Fig. 2 This issue arises because these models are trained on renderings of the Objaverse [0] dataset with a fixed intrinsic and thus these models are severely biased towards the geometry patterns present in this fixed intrinsic matrix.
Canonical camera setting. To address this problem, Era3D uses different intrinsic parameters for input and generated images. Regardless of the focal length and pose of the input image, we consistently generate orthogonal images with an elevation of 0% For example, when processing an input image captured at elevation a and azimuth 8, Era3l) produces set of multiview images at an azimuth of {13 3 + 45% B8 + 90", , 45 90", 3 180%) and an elevation of 09. We refer to the setup of these output images as a Canonical camera setting.
3.2 Regression and Condition Scheme
Given an image with arbitrary viewpoint and focal length, generating novel-view images in the canonical camera setting is challenging because this implicitly puts an additional task on the diffusion model to infer the focal lengths and elevation of the input image. To make it easier, previous methods 32
33 2 rely on additional elevation inputs or predictions as the conditions for the diffussonn model. However, pose estimation from single image is inherently ill- ill-posed due the lack of geometry information. Moreover, even though estimating a rough elevation is possible, it is almost impossible for users to estimate the focal length of the input image.
To address this problem, we propose incorporating an Elevation and Focal length Regression mod- ule (EFReg) into the diffusion model. We use the feature maps of UNet to predict the camera pose in the diffusion process. Our motivation stems from the fact that the feature maps of UNet not only contain the input images but also include the current generation results which provide richer and more informative features for predicting the camera pose.
Specifically, within the middle-level transformer block of the UNet, we apply global average pooling to the hidden feature map H. yielding a feature vector that subsequently fed into three Multilayer Perceptron (MLP) layers R, and R2 to regress the elevation ā and focal lens f
ā = R, (Av)) f -aa==RRaAaaaaa (2(AAgPool(H))
The regressed elevation à and focal length f are supervised by the ground-truth elevation a and focal length f by
fregress MSE(G, a: + MSE(J.
Then, the regressed & and f are used as conditions in the diffusion process. We apply positional encoding ā and and concatenate them with the time embeddings of the Stable Diffusion model.
5

---

The concatenated feature vectors are used in all the upsampling layers of the UNet, providing information about the estimated focal lengths and elevations for better denoising.
3.3 Row-wise Multiview Attention (RMA)
To generate multiview-consistent images, multiview diffusion models typically rely on multiview attention layers to exchange information among generated images. Such layers are often implemented by extending the existing self-attention layers of Stable Diffusion conduct the attention all the generated multiview images (34 55 69 63]. However, this dense multiview attention can be computationally expensive and memory- intensive, processes all pixels of all multiview images. This limitation hinders the scalability of multiview diffusion models to high resolutions, such as 512 x 512.
Since the pixels of multiview images are related by epipolar geometry, considering epipolar lines in multiview attention could possibly reduce computational and memory complexity. However, strictly considering epipolar attention 665 in general two-view camera setting still consumes massive computation and memory. As shown in Fig. B), for pixel on camera O₁, find its corresponding epipolar line in camera O2 by the relative camera pose. Then, we need to sample K points on the epipolar lines conduct cross attention between these sample points of O2 and the input pixel of O1- In the following, we propose an efficient and compact row-wise multiview attention, which is a special epipolar attention tailored our canonical camera setting.
In our canonical camera setting, cameras are distributed at an elevation of 0º around an object. easily demonstrate the following proposition.
Therefore,
Proposition 1 If two orthogonal cameras look at the origin with their y coorlinate aligned with gravity direction and their elevations 0 as shown in Fig. Bd), then for a pixel with coordinate (xc, = (u, v) on one camera, its corresponding epipolar line on other views is =.
We leave the proof in the supplementary material. Proposition T is a simplification of epipolar constraint, revealing that all epipolar lines correspond to rows in the generated multiview images. Building on this insight, we leverage the epipolar constraint by applying new self attfnnttnnionn layers on the same row across generated images to learn multiview consistency. By exploiting this constraint, we avoid the computational expense of dense multiview attention and instead accurately focus attention on epipolar lines. Moreover, our row-wise attention layer only involves elements from the same row, rather than sampling multiple points on the epipolar line, thereby significantly reducing computational complexity and facilitating training even on high-resolution inputs such 512 x 512.
4 Experiments
To construct training images, we render 16 ground-truth images using orthogonal cameras with evenly distributed azimuth from 0 to 360 and a fixed elevation of 0 Subsequently, for each azimuth, we render more images using perspective cameras and one image using an orthogonal camera, both of which have random elevations sampled from the range [-20, 40] degrees. The perspective camera has a focal length randomly selected from the set (35, 50, 85, 105, 135) mm, which are commonly used camera parameters. All the renderings have the resolution of 512 512. Following the previous methodologies (32 33 we evaluate the performance of Era3D on the Google Scanned Object [12 dataset, widely regarded as standard benchmark for 3D generation tasks. Moreover, we also evaluate our methods on in-the-wild images collected from the Internet or generated by image diffusion models [52) to show the generalization ability. The same as previous methods [55] 69]. we remove backgrounds and center objects on these in-the-wild images.
Datasets. We trained Era3D on subset of Objaverse 10]
Metrics. Our methodology is evaluated in two tasks, novel view synthesis (NVS) and 3D reconstruc- tion. The NVS quality is evaluated by the Leamed Perceptual Image Patch Similarity (LPIPS) [77] between the generated and ground-truth images. LPIPS evaluates the perceptual consistency because there may be slight misalignment between generated and ground-truth images. The 3D reconstruction quality evaluated by the Chamfer Distance (CD) and the Volume IOU between the reconstructed meshes and the ground truth ones.

---

Input
Wonder3D
I.GM
One-2-3-45
Shap-F
Ours
Figure 5: Qualitative comparison of 3D reconstruction results on the GSO dataset produces the most high-quality 3D meshes with more details than baseline methods.
[12
Era3D
"Classic musketeer hat."
bulidog with a black pirate hat."
pig weoring backpock."
"A beautiful cyborg with brown hair.'
Ghoot eating burger
Input
LGM
Wonder3D
Magic123
Ours
Figure 6: Qualitative comparisons of novel view synthesis quality of reconstructed 3D meshes with single-view images generated by SDXL. 45

---

GSO
In-the-wild
Input
Ours
Wonder3I
Unique3D
Input
Ours
Wonder3D
Unique3D
Figure 7: More comparisons w.r.t distortion problem.
Implementation details. Our implementation is built upon the ppen-source text-to-image model, SID2. -uncllpp [51] We train Era3D on 16 H800 GPUs (each with 80 GB) using a batch size of 128 for 40, 000 steps. We set the initial learning rate as 1e-4 and decreased to 5e-5 after 5, 000 steps. The training process takes approximately 30 hours. To conduct classifier-free guidance (CFG) [18]. we randomly omit the clip condition at rate of 0.05. During inference, we employ the DDIM sampler [57 with 40 steps and a CFG scale of 3.0 for the generation. Following Wonder3D [34,, we first perform reconstruction with NeuS and then apply a texture refinement step. The whole pipeline requires approximately 4 minutes, comprising 13 seconds for the multiview diffusion, 3 minutes for the NeuS reconstruction, and 10 seconds for the texture refinement, For further details, please refer to the supplementary material.
4.1 Experimental Results
Novel view synthesis. First, several examples of the multiview images and normal maps generated by Era3D are shown in Fig. The results demonstrate that given input images with varying focal lengths and viewpoints, Era3D can generate high-quality and consistent multiview images and normal maps. When the input image is captured by perspective camera and its viewpoint is not 09, Era3D can correctly perceive the elevation of the viewpoint and the perspective distortion. Then, method learns generate images of the same object with high fidelity using orthogonal cameras on canonical viewpoints, effectively reducing the artifacts caused by the perspective distortion and improving the reconstruction quality. Moreover, Era3D can produce images in the 512 512 resolution, which enables generating much more details like the fine-grained texture on the "Armor" and the complex structures the "Mecha" in Fig. D
elevation of
Furthermore, we provide a quantitative comparison with other single-view reconstruction methods including RealFusion B35. Zero-1-to-3 (32,, Syeamer (33 and Wonder3D (34 in Tab, I The results show that our method outperforms previous approaches in terms of novel-view-synthesi quality by significant margin, showcasing the effectiveness of our designs.
Revonstruction. We further conduct experiments evaluate the quality of reconstructed 3D meshes. We compare our method with RealFusion
35
Zero-1-to-3 [32] One-2-3-45 [3], Shap-E [21],
Magic 12gic123[49] WD 34, SyneDreamer textures on the GSO dataset are shown in Fig. 5 while the renderings of the reconstructed meshes
33 and LGM 66 Reconstructed meshes and their
on text-generated images are shown Fig. 6 As shown in the results, Shap-E fails to generate completed structures. The meshes reconstructed by One-2-3-45 and LGM tend to be over-smoothed and lack details due to the multiview inconsistency in generated images by Zero-I-to-3 [32] or ImageDream 669. The results of Wonder3D tend to be distorted on these input images rendered with a focal length of 35mm because it assumes the input images are captured by orthogonal cameras. In contrast, results show significant improvements in terms of completeness and details than these baseline methods.
Quantitative comparisons of Chamfer Distance (CD) and Intersection over Union (IoU) are shown in Tab. T Era3D outperforms all other methods, exhibiting lower Chamfer Distance and higher Volume loU. suggesting that the meshes it generates align more closely with the actual 3D models.

---

Distortion problem We finally provide the comparisons with SOTA methods, like Unique3D, w.r.t distortion problem in Fig. / which shows that they suffers from severe perspective distortion while our method greatly alleviates this problem.
Table I: Quantitative evaluation of Chamfer distance, IoU (for reconstruction), and LPIPS (for NVS).
Table 2: Comparison of pose estimation accu-
racy. ã: elevation, f: normlized focal length.
Method CD J
InU]
1.PIPS 4
SSIM 1
PSNR 1
Method
à /
f / mm
RealFusion
0.0819
0.2714
0.283
0.722
15.26
Zero-1-to-3
0.0339
0.5035
0.166
0.779
18.93
Dino 10.24
0.28
0:3-2:3:45
0.0629
0.4086
Error
One-2-3-45
10.14
Shap-E
0.0436
0.3584
Ours
2.69
0.13
Magic123
0.0516
0.4528
SyncDreamer
0.0261
0.5421
0.146
0.798
20.05
Dino
377.07
0.058
Wonder3D
0.0248
0.5678
0.141
0.811
20.83
Variance
One-2-3-45
267.44
LGM
0.0259
0.5628
Ours
0.0217
0.5973
0.126
0.837
22,74
Ours
112.30
0.036
4.2 Accuracy of Estimated Elevations and Focal Lengths
Beyond the tasks already discussed, we further evaluate the pose prediction of Era3D on the GSO dataset. We render the images with an elevation of [-10, 40] degrees and focal lengths of {35, 50, 85, 105, 135, ).), respectively. As baseline method, we employ dinov2_ vitb 14 feature [40] to predict the pose and train it with the same dataset. We compare our predictions with this baseline method and One-2-3-45, As shown in Tab. 2 Era3D achieves superior performance in error and variance. A more detailed analysis is provided in the supplementary materials.
4.3 Ablations and Discussions
Regression and condition scheme. In Fig. g we remove the EFReg and compare the results with our full model to demonstrate the effectiveness of our design. Without regressing a focal length and elevation as conditions during the denoising process, the resulting shape is distorted and fails to generate reasonable novel views in the canonical camera setting. In comparison, adding our EFReg module and conditioning on the predicted elevations and focal lengths provides effective guidance to generate undistorted cross-domain images, thereby resulting more accurate 3D reconstructions.
Input
w/ EFReg
w/o EFReg
Figure 8: Ablation study of EFReg.
Row-wise multiview attention. As illustrated in Fig. our proposed RMA effectively facilitates information exchange among multiview images, yielding consistent results comparable to those achieved by dense multiview attention layers in [69 34]. In N-view camera system, assuming a latent feature with size of S x S, our RMA design significantly improves training efficiency by reducing the computational complexity of attention layers from 5t) to O(2233) as shown in Fig. 3 While epipolar attention also achieves a complexity reduction to O S2 where K the sample number, it does so at the cost of increased memory and time consumption due to the sampling process. To further highlight the efficiency of RMA over dense multiview attention, we present the memory usage and running time of both 256 and 512 resolutions. We use the epipolar attention implementation in (22. As listed in Tab. 33] the advantage of RMA becomes increasingly obvious as the resolution grows. At a resolution of 512, RMA achieves a thirty-fold reduction in memory usage and nearly hundred-fold reduction in running time. Even with xFormers (26). our method substantially improves training efficiency by a large margin (22.9 ms vs. 1.86 ms). This efficiency enables training models on higher resolutions with denser views without significantly increasing computational efficiency and demand, thereby maintaining a lightweight framework.
Finally, we conduct performance comparisons between dense, epipolar, and our row-wise attention. Considering that our orthogonal setup with the same elevation, epipolar attention is equivalent to row-wise attention (except for implementation differences), we compared dense and row-wise attention at a resolution of 256. Era3D achieves comparable performance with dense multiview attention, as evidenced in Tab. 4 Our row-wise setting outperforms the baseline for the Chamfer

---

Table Memory usage and running time of multiview attention with resolution 256 and 512.
w/o xFormers
w/
xFormers
Multiview attention
Memory usage (G)
Running time (ms)
Memory usage (G)
Running time (ms)
256
512
256
512
256
512
256
512
Dense
3.02
35.32
8.88
220.41
0.99
1.42
1.77
22.96
Epipolar
2.43
24.20
3.57
60.89
1.02
1.71
1.78
20.03
Row-wise
0.95
1.66
0.91
2.23
0.99
1.08
0.28
1.86
Table 4: Performance of dense and row-wise attention resolution of 256.
Method
CD +
IoU t
LPIPS 4
PSNR t
SSIM 1
dense
0.0239
0.5877
0.140
20.73
0.819
rowwise
0.0232
0,5831
0.137
20.92
0.813
distance, LPIPS, and PSNR. We attribute this to row-wise attention reducing the number of attention tokens, allowing the model to focus more on valuable tokens.
Limitation and Conclusion
Limitations. Though Era3D achieves improvements on the multiview generation task, our method struggles to generate intricate geometries like thin structures because we only generate 6 multiview images and such sparse generated images have difficulty in modeling complex geometries. Since the reconstruction algorithm is based on Neural SDF, Era3D cannot reconstruct meshes with open surfaces. In future works, we could integrate our framework with other 3D representations, such as Gaussian splatting, to improve both rendering and geometry quality.
Conclusion. In this paper, we present Era3D, a high-quality multiview generation method for single- view 3D reconstruction. In Era3D, we propose to generate images in a canonical camera setting while allowing imput images to have arbitrary camera intrinsics and viewpoints. improve the generation quality, we design regression and condition scheme to predict the focal length and elevation of input images, which are further conditioned to the diffusion process. Additionally, we employ row-wise multiview attention to replace dense attention, significantly reducing computational workloads and facilitating nigh-resolution cross-view generation. Compared with baseline methods, Era3D achieves superior geometry quality in single-view 3D reconstruction.
6 Ethics Statement
The objective of Era3D is to equip users with powerful tool for creating detailed 3D) models. Our method allows users to generate 3D objects based on a single image. However, there is potential risk that these generated models could be misused to deceive viewers. It is important to note that this issue not unique to our methodology but prevalent other generative model methodologies. Therefore, it is absolutely essential for current and future research in the field of 3D generative modeling to address and reassess these considerations consistently.
Acknowledgments and Disclosure of o Funding
The research supported by Theme-based Research Scheme (T45-205/21-N) from Hong Kong RGC, and Generative AI Research and Development Centre from InnoHK.
References
[1] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470 5479,, 2022.
10

---

[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffuuson models to large datasets, arXiv preprint arX1v:2311.11127, 2023.
[3] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay. Sameh Khamis, et al. Efficient geometry- 3d generative adversarial networks. In Proceedings of the IERE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 151203311,, 2015.
[5] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312. 2023.
[6] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IERF/CVF International Conference on Computer Vision, pages 22246-22256, 2023.
[7] Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint ar arXiv:2301. 2023.
[8] Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang. and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint wrX.v22403.06638, 2024.
[9] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction. and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[10] Matt Deitke, Dustin Schwenk. Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of annotated objects. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-
13153, 2023.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-879 2021.
[12] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned houschold items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022.
[13] Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[14] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint urXiv:2110.08955 2021.
[15] Anchit Gupta, Wenhan Xiong. Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffuusin for textured mesh generation. arXiv preprint M.Xix:233300377. 2023.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems. 33:6840 6851, 2020.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020.
[18] Jonathan Ho and Tim Salimans. Classifier- diffusion guidance. arXiv preprint arXiv:2207.12599 2022.
[19] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.044 114400, 2023.
[20] Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate vperse-view 3d modeling from camera poses. arXiv preprint 11000:2310001410, 2023.
[21] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv:2305. 02463, 2023.
arXiv preprint
11

---

[22] Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem. Sergey Tulyakov, Igor Gilitschenski, and Aliaksandr Siarohin. Spad: Spatially aware multiview diffusers. arXiv preprint 20225.00002.,. 2024.
[23] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[24] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[25] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high- perforrefe differentiable rendering. ACM Transactions on Graphics, 39(6). 2020.
[26] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: modular and hackable transformer modelling library. https://gi http:///ttttacceccceeeeeeeeeeeee wtt.://accebbboeeeerrcc//aaaeaaa oxwerrr 2022.
[27] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d 3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.0622 2023.
[28] Peng Li, Jiayin Zhao, Jingyao Wu, Chao Deng, Yuqi Han, Haoqian Wang, and Tao Yu. Opal: Occlusion pattern aware loss for unsupervised light field disparity estimation. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[29] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d 3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[30] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T. Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[31] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. 2-e-2-3-45: Any single image to 3d mesh in 45 seconds without per- shape optimization. arXiv preprint arXiv:2306. 169288 2023.
[32] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zem- 1-to-3: Zero-shot one image to object. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 9298-9309, 9009099300000000 2023.
[33] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang- Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 2023.
[34] Xiaoxiao Long, Yuan-Cher Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjic Liu, Yucxin Ma. Song Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross cross-domain diffusion, arXiv preprint arXiv:2310. 2023.
[35] Luke Melas-Kyriazi Christian Rupprecht, Tro I Laina, and Andrea Vedaldi, Realfusion: 360 reconstruction of any object from single image. In Arxiv, 2023.
[36] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM. 65(1):99- 1009 2021.
[37] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4: 2022.
[38] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In CVPR, 2021.
[39] Matthia Nießner, Michael Zollhöfer, Shahram Izadi, and Mare Stamminger. Real- time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG), 32 2013.
12

---

[40] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov. Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Pao Huang, Hu Xu, Vasu Sharma, Shang-Weit Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Learning robust visual features without supervision, 2023.
Dinov2:
[41] Roy Or-EI, Xuan Luo. Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher Shllzerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR, 2022.
[42] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[43] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. CVPR, 2019.
[44] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision ECCV 2020: 16th European Conference. Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pages 523 540. Springer, 2020.
[45] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high- image synthesis. arXiv preprint arXiv:2307.0199 2023.
[46] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 1498889888 2022.
[47] Xingqun Qi, Jiahao Pan, Peng Li, Ruibin Yuan, Xiaowei Chi. Mengfei Li, Wenhan Luo, Wei Xue, Shanghang Zhang. Qifeng Liu, et al. Weakly-supervised emotion transition learning for diverse 3d co- speech gesture generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Patter Recogmition, pages 10424-10434, 2024.
[48] Xingqun Qi, Hengyuan Zhang, Yatian Wang, Jiahao Pan, Chen Liu, Peng Li, Xiaowei Chi, Mengfei Li, Qixun Zhang, Wei Xue, et al. Cocogesture: Toward coherent co-speech 3d gesture generation in the wild. arXiv preprint Xiv::2240 16874, 2024.
[49] Guocheng Qian, Jinjie Mai. Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov. Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high qugh-quality 3d object generation using both 2d and 3d diffusion priors, arXiv preprint arXiv:2306. 17843, 2023.
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-rcsolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[53] Shunsuke Saito, Zeng Huang, Ryota Natsume. Shigeo Morishima, Angjoo Kanazawa, and Hao Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2304-2314, 2019.
[54] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng. and Hao Su. Zero123++: a single image to consistent niet diffusion base model. arXiv preprint 231. 2023.
[55] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
[56] J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In CVPR, 2023.
[57] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint ar 0rX2v2200..02502, 2020.
[58] Cheng Sun, Min Sun, and Hwann- Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
13

---

[59] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. arXiv preprint arXiv:2312.13150, 2023.
[60] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(-) image- conditioned 3d generative models from 2d data. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 8863-8873, 2023.
[61] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.0505: 2024.
[62] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653. 2023.
[63] Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang. Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mvdiffusion+++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. arXiv preprint arXiv:2402.12712, 2024.
[64] Shitao Tang, Fuayng Zhang, Jiacheng Chen, Peng Wang, and Furukawa Yasutaka. Mvdiffusion: Enabling holistic multi-view image generation with correspondence diffusion. arXiv preprint 2307.01097, 2023.
[65] Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16773-16783, 2023.
[66] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. arXiv preprint arXiv:2403. 240..22038.. 2024.
[67] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Ych, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proccedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12619-12629, 2023.
[68] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint ar Xixx:22166. 10689, 2021.
[69] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:23 022011 2023.
[70] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun I Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint 1xxiv2211..1000 2023.
[71] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[72] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu, Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.0500
2024.
[73] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su. and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint MxXiv2204411335 2024.
[74] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang. Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07199 2024.
[75] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ccyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiy preprint ar 2rXiv:2403.14m0 14621, 2024.
[76] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang. Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:231 100917, 2023.
14

---

[77] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018
[78] Ruichen Zheng, Peng Li. Haoqian Wang. and Tao Yu. Learning visibility field for detailed 3d human reconstruction and relighting. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 216-226, 2023.
[79] Zi-Xin Zou, Zhipeng Yu, Tuan-Chen Guo, Yangguang Ding Liang, Yan-Pei Cao, and Song Hai Zhang Triplane meets gaussian splatting: Fast and generalizable single-view reconstruction with transformers. arXiv preprint arXiv:2312 12.09147. 2023.
15

---

Distance of
Length
Inf
105mm
85mm
Orthogonna Scale
Camera Center
Object
Camera plane
Object
Camera plane
f
f
(a) Orthogonal Camera
(b) Perspective Camera
(c) Renderings with different focal lengths
Figure 9: Equivalence between orthogonal and perspective camera models and our rendering samples
in GSO dataset 112
A
Supplementary Material
A.1
Implementation Details
Data preparation. To render training data, we randomly choose a focal length for the input image from predefined set of focal lengths and always the orthogonal camera for all generated images. To train Era3D, we need to construct orthogonal camera render generation images and the equivalent perspective cameras to render input images. The equivalence here means that the renderings from these two kinds of cameras have almost the same size, for reducing training bias. As shown in Fig.
9 (a) and (b), given a predefined
orthogonal scale and given focal length f. we compute the distance d by d f/s. where is the orthogonal scale to scale the size of the rendered image. We will adjust the distance from the camera center the object the value to make rendered image as similar as possible.
Normalized focal length. our experiment, we choose several discrete focal lengths, (24, 35, 50, 85, 105, 135) mm, and an orthogonal camera to render multiview images for training. Due to the broad range focal lengths, we regress the normalized focal lengths instead of the actual ones. Specifically, we normalize them with minimum focal,
24/f.
(3)
For the orthogonal views, we set 0.
Cross-domain attention. In contrast to Wonder3D 34 we do not incorporate an additional atten- tion layer to fuse information between cross-domain images. Instead, in carlier experiments, we com- bined both the cross-domain fusion modulc and the multiview fusion module within the same row- wise attention block to enhance training efficiency. This initial setup, however, leads subopti- mal outcomes, as shown Fig. [ where tends produce overly smoothed normal maps. hypothesize that the significant differences between domain images are the primary cause of this is- sue, as the row-wise information alone is insufficient for learning fine-grained features across domains. In contrast, integrating the cross domain module within the self- attention block enables the utilization complete image features. allowing the model to better handle these disparities and achieve improved generalization. Our attention block consists of self- cross-domain attention, row-wise multiview attention, and cross- domain attention, which allows merge the two-stage training into single stgglestage joint training, effectively aligning images of both domains.
Reconstruction and texture refinement. We perform recon- struction from the generated multiview normal and color images using NeuS. However, since only generate sparse views, the rendcred images of the reconstructed meshes may attain the quality of the generated ones, particularly for objects with complex textures. To address this issue, we employ differen- tial rendering to refine the textures the reconstructed meshes. Specifically, we preserve the geometry of Neus and initialize the texture with vertex colors. We utilize 25 render multiview images in predefined views optimize the vertex colors with the corresponding generated images. This process takes less than second and significantly improves the texture quality. For the NeuS reconstruction, we use the same settings as Wonder3D. During texture refinement, we optimize the appearance for 200 iterations with resolution of 1024 and learning rate of le-3.
Input
in Self Attn.
in RMA
Table Ablation study on the position of the cross domain-block.
16

---

Noise scheduler. The original SD2.1-Unchin base model employs scaled linear noise scheduler during training. which is effective for smaller sample sizes, such 256. However, this scheduler tends restrict the generative capabilities of models at larger sizes, such as 512. Drawing inspiration from recent research by Chen [2]. we implemented a linear noise scheduler throughout our experiments to enhance generation quality and accelerate the convergence of the background. This adjustment proves critical in supporting the model's performance and efficiency.
A.2 Proof of Proposition
Given two cameras O₁ and O₂ with relative rotation R and translation t, let æ1 and 2a denote the homogeneous coordinates of corresponding points in the image planes, respectively, epipolar constraint can be expressed as
2 Ex 0,
(4)
where is the fundamental matrix. Then, the epipolar line in O₂ can be represented as
I 1 [t] R
(5)
where x denotes the skew-symmetric matrix generated by t. However. epipolar attention needs to query dense points epipolar lines, which leads significant computational inefficiency, due to arbitrary directions and varying lengths of epipolar lines.
Row-wise multiview attention a special cpipolar attention our defined canonical camera setting, as depicted Fig. Bj.. Assuming the y y-axis represents the gravity direction, the x-axis denotes the right-hand direction, and the z-axis points away from the camera to the origin of the object. The relative R and t between two cameras can be represented as
cos (#)
sin(O)
1 _ 0,,,0,,'"
R
(6)
(()
cos(O)
where is azimuth angle of the cameras. Considering the point (1 yr, #1) camera Oi, the coordinates of P. camera O2 are given by
cos(0) sin 21 + Lx
Pa y2, 22) = RP +t=
y1 cos(0)= sin (0) (0) + f
It is observed that the scene points from different views share the same y-coordinate. Assuming the identical orthographic scale, the y-coordinates of projections the image plane are also equivalent. Extending single sample line of points with the same y-coordinate, all projections of those points two image planes are on the same row.
A.3 Pose Estimation
During inference, we obtain the final pose by averaging the class-free guidance results from all denoising steps,
ā
(1 + w c) wa)(
(8)
i 1= (1 (1+ (z))
where 2 is the latent, e is the condition, w is the CFG scale, is parameters of UNet and ML.P and the denoising step. We observe that the averaged class-free guidance predictions achieve the highest accuracy. We attribute this to the random dropping of condition images during training. We do not rely solely the prediction at the final step because we constrain the estimated elevation and focal length with the ground pose rather than the noisy ones at each denoising step. We conduct extensive experiments to evaluate the accuracy of the estimated pose on the GSO datasct.
Elevation. As listed in Tab. 6 Dino and One-2-3-45 only learn the relative trend as the clevation increases, resulting in large errors for cases with high clevations. This is because our baseline only employs the fcature of single image, which is insufficient for predicting the absolute elevation. The prediction of One-2-3-45 mainly depends the multiview images generated by Zero-I-to-3, The inconsistency of generated images leads to ambiguity prediction. In comparison, our results are closer to the ground truth elevation. In most cases, the variance of our method is significantly lower than that of Dino and One-2-3 1ne-2-3-43 demonstrating remarkable robustness.
Focal. We report the error and variance of pormalized focal predictions for the baseline and our method in Tab. 7 It is observed that the baseline cannot distinguish the differences between various focal lengths. In contrast,
17

---

Table 6: Elevation accuracy GSO dataset.
Elevation /
Method
-10
10
20
30
40
Pred
Err
Pred
Err
Prod
Prod
Er
Pred
Em
Pred
Ein
3.58
4.63
5.37
10.21
16.72
14.28
18.41
21.59
Mean
One-2-3-45
-11.93
1.93
-5.23
5.23
-0.06
10.06
11.17
8.83
15.5
14.5
19.73
20.27
Ours
99.71
0.29
-2.20
2.20
5.59
4.41
23.85
3.85
34.03
4.03
38.67
1.33
Dino
202.37
252.51
153.67
365.32
463.92 92
824.63
One-2-3-45
168.61
200.97
326.05
630.32
Ours
32.1
83.63
163.38
113.55
134.69
146.46
Table 7: Focal length accuracy on GSO dataset.
Normalized focal length (focal length mm)
Method
0.68 (35)
0.48 (50)
0.28 (85)
0.22 (105)
0.17 (135)
0.17
0.0 (ortho) Pred
Pred
Er
Pred
Pred
Dino
0.75
0.07
0.46
0.02
0.53
0.27
0,56
0.34
0.57
0.4
0.59
0.59
Mean
Ours
0.69
0.01
0.54
0.06
0.37
0.09
0.33
0.11
0.34
0.17
0.32
0.32
Dino Ours
0.069
0.073
0.054
0.052
0.049
var
0.041
0.035
0.024
0.02
0.069
0.032
Rlevation -10
Elevation = 10
Flevation = 20
Flevation = 0
Figure Generation results of various elevation. We use reconstructions from the view Elevation 0 as the reference. Different inputs generate consistent results.
Focal 35mm
Focal 85mm
Focal 135mm
Ortho
Figure 11: Generation results of various distortions. We employ reconstructions from orthogonal inputs as the reference. For each case, we illustrate the input along with the generated color, normal, and mesh.
18

---

Table 8: Quantitative evaluation on images with various focal lengths. CD: Chamfer Distance.
a=0
f=00
a=0
Pose
f=35
f=
8=
f=105
f=135
am-10
crm 10
a=
a=
am44
f = 00
CD
0.0223
0.0219
0.0216
0.0214
0.0214
0.0217
0.0216
0.216
0.0219
0.0217
0.0213
model can predict the large distortion (c.g., focal=35, 50) of input images, which is advantagcous for correcting them to some extent. Simultancously, our method exhibits smaller errors for settings with large focal lengths than the baseline. We believe our method could be further explored and improved in the future.
Furthermore,
orthogonal renderings at an elevation of 0 degree the reference. We vary the elevation from - 10 10 degrees and select focal lengths from (35, 50, 85, 105, 135, oc} assess the system's 's robustness elevations and focal distortions by reconstructed meshes.
indicated in Tab. elevation achieves the best performance CD, and other settings are on the same bar with the reference setting. suggesting that our method effectively handles these distortions, producing meshes that align well with the reference ones, under significant focal distortion. We visualize some samples in Fig. 10 and Fig. 1T
orthogonal setting
19

---

Figure 12: More results of images from the Internet.
20

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All the claims supported by the results of experiments.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our method has difficulty in modeling complex geometries and open meshes.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well spec--seecciaion asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work do not contain theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
21

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We report the experiment details and we will release related codes.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution or the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does not require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g.. large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to deseribe the particular way they provide for reprodueibilitty In the case of closcd-sourc source models, it may be that access to the model is limited some way (c.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [No]
Justification: Our evaluation the public datasets. We do not provide code in Supplementary Material. But they will made publicly available once they have been fully prepared.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines (https://nips... cc/pulicc/ haaeee//oooSuubiiiiiiiieee
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nnis///////////////e:::eee cc/public/ qul.dess/Cocdeuubiieessoooooo... for more details.
22

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should relcase anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommenedd, but including URLs to data and code is permitted.
6. Experimental Setting/Detan Details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discuss the training and test details in the section on experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the error and variance of predicted elevation and focal lens.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We train Era3D with 16 H800 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
23

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics http://uurrppaccppppllll/////eea cctppaallic/fthhhaeauli.illliiia
Answer: [Yes]
Justification:
This work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However. if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
24

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification: This paper cites the related datasets and codes used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection. curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that procedures for this may vary significantly between institutions and Iocations, and we expect authors adhere to the NeurIPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
26


Reviews:
[{"review_id": "ydbCzWKWyO", "reviewer": "Reviewer_5Kpg", "summary": "This work introduces a novel take on multiview diffusion models, highlighting the potential to realize high-resolution images from one image. The method comes with a new design for the diffusion-based camera prediction module, focal length, and elevation of the input image elevation, together with row-wise attention in enforcing epipolar priors in the MV diffusion. The results show that the approach is very high in quality, detailed in 3D meshing ability, and multiview image generation with larger resolution ability while consuming much less computation compared to other presented approaches.", "questions": "1. The abstract (line 16) states, \"Era3D generates high-quality... up to a 512×512 resolution while reducing computation complexity by **12x times**.\" I was really confused about this context. In the paper, the authors claim the design of row-wise multiview attention results in a 12-fold reduction in running time compared with Dense MV Attention. However, the author didn't show the complexity of other parts' time in their pipeline. Further Claims are needed. \n2. Why did you choose viewpoints on an elevation of 0? Will it affect the generation for a specific side (e.g., bottom side)?\n3. Have the authors tried to perform any sparse view reconstruction? And why you choose 6 views: {β, β + 45◦, β + 90◦, β − 45◦, β − 90◦, β + 180◦}? More ablation studies or claims may be needed.\n4. I checked your demo video, but it only shows your result. Could you also compare it with other results in the video for better display?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Clear Motivation and Innovative Module Design**\nIn this paper, the authors address several significant challenges associated with MV Diffusion in 3D content generation. These challenges include issues such as low resolution, inefficient generation processes, and inconsistent camera settings. For each of these problems, the authors propose novel designs aimed at providing effective solutions.\n\n**State-of-the-Art Results**\nThe authors claim to have achieved state-of-the-art performance in single-view image generation tasks, as evidenced by the results in Table 1 and Table 2. However, due to the rapid advancements in 3D generation, their work does not compare their generation quality with recent methods such as LRM.\n\n**Novel Contribution to 3D MV Diffusion Generation**\nTo the best of my knowledge, this paper is the first to address and propose solutions for the distortion problem specifically in 3D MV Diffusion generation.", "weaknesses": "### Missing Critical Details\nThe paper lacks several essential points, including optimization time for inference, details on hyperparameters, and robustness testing.\n\n### Insufficient Ablation Study\nThe ablation study does not adequately justify the architectural design choices. More comprehensive experiments are needed to support these decisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:\n\n**Q1: Comparison with LRM-based methods.**  \nTab.1 showcases the comparisons of Era3D with OpenLRM and CRM on the GSO dataset, which shows that Era3D exhibits better performance than these two methods. Our method can be incorporated with LRM-based methods because recent LRM-based methods rely on multiview generation as the input while Era3D can provide generated multiview images.\n\n**Table 1**: Quantitative comparison with LRM-based methods:  \n\n| Method | CD↓ | IoU↑ | LPIPS↓ | SSMI↑ | PSNR↑ |\n|--------|-----|------|--------|-------|-------|\n| openLRM | 0.0302 | 0.5243 | 0.158 | 0.738 | 19.03 |\n| CRM | 0.0237 | 0.5693 | 0.146 | 0.803 | 20.78 |\n| Ours | 0.0217 | 0.5973 | 0.126 | 0.873 | 22.74 |\n\n**Q2: Missing details of inference time and hyperparameters.**  \nThe whole process requires approximately 4 minutes, comprising 13 seconds for the multiview diffusion, 3 minutes for the NeuS reconstruction, and 10 seconds for the texture refinement. Our diffusion model employs a 6-layer MLP for pose estimation. We use the same hyperparameters as SD2.1, except what is explicitly stated in the 'Implementation Details' section. For the NeuS reconstruction, we use the same settings as Wonder3D. During texture refinement, we optimize the appearance for 200 iterations with a resolution of 1024 and a learning rate of 1e-3. We will add these details to the revision. \n\n**Q3: Absence of robustness testing.**  \nWe comprehensively evaluate the robustness of our pose prediction on the GSO dataset and the generation quality for in-the-wild objects in Tab.5, Tab.6, and Fig.11 of ' Supplementary Material'. We will also release our code for public testing and evaluation. \n \n**Q4: Why select generation viewpoints of elevation 0? Would this affect the reconstruction of the bottom?**  \nIn contrast to the setup with various elevations employed in Zero123++, Era3D is trained to generate six views at an elevation of 0. This choice is because row-wise attention requires a fixed elevation. Intuitively, models perceive an elevation of 0 more readily than a random elevation. While this setting may affect the objects with planar bottoms, such cases are uncommon. Note that the prior works such as SyncDreamer and Wonder3D also use similar pre-defined viewpoints.\n\n**Q5: Why only generate 6 views?**  \nGenerating dense views necessitates substantial memory for training. Fig.1 in global response illustrates sparse-view reconstruction results using 2, 4, and 6 views. Our method, utilizing 6 views, consistently produces complete and plausible 3D shapes. \n\n**Q6: Insufficient ablation of EFReg.**  \nWe utilize the intermediate feature map with the lowest resolution in UNet for EFReg prediction since it could provide high-level global information. The effectiveness of EFReg is evaluated qualitatively in Fig.7 of the main paper. To provide quantitative support, we additionally report the Chamfer Distance in Tab.2. These results confirm that EFReg facilitates robust pose estimation and enhances reconstruction accuracy.  \n\n**Table 2**: Ablation of EFReg on GSO datasets with various elevation (α) and focal lengths (f). We report the Chamfer Distance (↓).\n\n| Pose | α=0 | | | | | f=∞ | | | | | α=0, f=∞ |\n|------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|------------|\n|      | f=35 | f=50 | f=85 | f=105 | f=135 | α=-10 | α=10 | α=20 | α=30 | α=40 | |\n| w/o EFReg | 0.0237 | 0.0233 | 0.020 | 0.022 | 0.0217 | 0.0221 | 0.0217 | 0.0225 | 0.0231 | 0.0228 | 0.0217 |\n| w EFReg | 0.0223 | 0.0219 | 0.0216 | 0.0214 | 0.0214 | 0.0217 | 0.0216 | 0.0216 | 0.0219 | 0.0217 | 0.0213 |\n\n**Q7: Provide additional visual comparison with baselines in the demo video.**  \nWe appreciate the reviewer's suggestion. We will incorporate comparisons in the demo video to provide a more comprehensive visual presentation of our method's performance relative to existing approaches.\n\n**Q8: Confusing claim about computation complexity reduction of row-wise attention.**  \nCompared to Dense MV attention, our row-wise attention reduces the computation complexity by 12 times. We will clarify this in the Abstract. In Tab.3, we list the memory usage and running time of each part of our pipeline, in which other parts include self attn, cross attn, and feed-forward layers. Notably, Dense MV attention layers constitute approximately 60% of the memory footprint and 75% of the running time in the overall pipeline. Our row-wise attention substantially mitigates these computational demands, with particularly remarkable improvements in execution time.\n\n**Table 3**: Memory usage and running time of the pipeline with 512 resolution and xFormer.\n| | Memory usage (G) | | | Running time (ms) | | |\n|----------|----------|----------|----------|----------|----------|----------|\n| | MV Attn | Other parts | Total | MV Attn | Other parts | Total |\n| Dense | 1.42 | ~1.0 | 2.40 | 22.96 | ~6.5 | 29.13 |\n| Epipolar | 1.71 | ~1.0 | 2.81 | 20.03 | ~6.5 | 26.75 |\n| Ours | 1.08 | ~1.0 | 2.09 | 1.86 | ~6.5 | 8.31 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YdtOVjw7rL", "reviewer": "Reviewer_a9Xx", "summary": "The authors proposed a method that can estimate the camera intrinsic matrix of the render of a given object, which attempted to solve the problem of other image-to-3d methods that only trained on fixed camera intrinsic matrix.", "questions": "1. Authors mentioned that they used the feature maps of middle-level transformer block for intrinsic estimation. Can you specify it? Which level did you use? How many levels did you use? Any experiments about the impact of different levels?\n2. Is it possible to directly generate images under perspective cameras?", "limitations": "1. I am not sure the novelty of the paper is enough for publication. The only contribution that I can tell is the camera intrinsics estimation. Row-wise attention is borrowed from other works. The authors mentioned that the proposed work can generate high resolution image, but sv3d can also do this.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method is the first work that considers the change of the camera intrinsic matrixes for the image-to-3D generation, and proposed an attention strategy to reduce the computational cost.", "weaknesses": "1. I do not think row-wise attention should be a contribution as it is a special case of the epipolar attention from another paper.\n2. Lacks the comparison with SV3D which is released before the deadline of the submission and can generate high resolution videos. Although Unique3D is released after the deadline, but it would be better if authors can also optionally include the comparison with it.\n3. There is only a demo video in the supplementary material. If authors want refer reviewers to the arxiv version, it actually violates the double-blind policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:  \n\n**Q1: Row-wise attention is just epipolar attention and is not the contribution of this paper.**   \nWe respectfully disagree with this characterization. As discussed in Line290 of the submission, the vanilla Eipolar attention adopted by previous methods significantly increases memory and time consumption due to their need for sampling points along the epipolar lines, which makes them even slower than the dense multiview attention. Era3D is the first attempt to simplify epipolar attention, which makes it extremely efficient for generating high-quality 3D objects. Thus, we consider it a core contribution of our work.   \n\n**Q2: Lack of comparison with SV3D and Unique3D.**  \nWhile we acknowledge the importance of providing comprehensive comparisons, Unique3D and SV3D are unpublished technical reports on arXiv. Both of them are concurrent works that released their codes near or after our submission. Despite it, we provide an additional comparison with Unique3D in Fig.2 of global response, which shows that Unique3D still suffers from perspective distortion while our method greatly alleviates this problem. We appreciate the reviewer's feedback regarding these comparisons.\n\n**Q3: There are no other supplementary materials other than the video and response to the reviewer's comment regarding the violation of the double-blind policy.**  \nWe respectfully disagree with the assessment. We strictly follow the NeurIPS 2024 guideline to attach the supplementary materials at the end of the main paper, including additional experiments and descriptions of our method. We also included an anonymous link at the end of the abstract to show our results without compromising the double-blind review process. We did not encourage or invite reviewers to search for our submission, and thus, we maintain that we have adhered to the double-blind reviewing policy as stipulated by the conference guidelines. We appreciate the reviewer's feedback and clarification on this matter.\n\n**Q4: Selection of features for EFReg.**  \nWe apologize for any lack of clarity in our presentation. For EFReg prediction, we utilize the final layer feature map of the intermediate UNet block with the lowest resolution since it could provide high-level global information.  \n\n**Q5: Can the proposed method directly generate perspective images?**  \nOur method is designed for 3D generation, which generates images around an object with orthogonal cameras. Thus, generating perspective images falls outside the scope of our paper. Moreover, generating perspective images will introduce additional challenges, such as varying scale prediction across multiple views, which would significantly increase the complexity of 3D generation. That's why existing works, including ours, focus on orthogonal setups rather than perspective ones.\n\n**Q6: Not sure the novelty of the paper is enough for publication.**  \nAs elaborated in Q1, our row-wise attention is distinct from the custom epipolar attention in previous studies. The proposed row-wise attention demonstrates a significant enhancement in training efficiency. SV3D is our concurrent work and requires considerably more resources for training. Therefore, we restate our contributions as follows:\n1. **Era3D** is the first method that tries to solve the distortion artifacts in general 3D generation tasks;\n2. We design the novel **EFReg** to enable diffusion models to take distorted images as inputs while outputting the orthogonal images on the canonical camera setting;\n3. We propose row-wise multiview attention, an efficient attention layer for high-resolution multiview image generation;\n\nWe believe our designs could advance the field and inspire other 3D generation models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "azn8erW6Po", "reviewer": "Reviewer_aLmz", "summary": "This paper develops a new image-to-multiview diffusion model with two key highlights. First, it proposes a novel method for estimating the focal length and elevation. Second, it introduces a new cross-view attention mechanism. Experiments demonstrate that the method outperforms previous SOTAs.", "questions": "(1) I think Wonder3D is a highly related paper, as both deal with the image-to-multiview problem. Given that the authors claim one of Era3D's superiorities is its high resolution, I'm wondering why they did not use PSNR and SSIM to evaluate the synthetic novel views as Wonder3D did.\n(2) As explained by the authors in L207, the RMA is based on a simplification. Table 3 shows the benefits of RMA. However, my intuition is that RMA sacrifices quality to some degree compared to Dense and Epipolar. I did not find an ablation study regarding this in the main paper or supplementary material. I think the authors should demonstrate that the quality of RMA, Dense/Epipolar are comparable. For example, if the quality deterioration of RMA exists, it should be shown to be acceptable. Otherwise, the authors should demonstrate that RMA outperforms Dense and Epipolar in terms of quality and explain the reason.", "limitations": "This paper addresses the issue of varying focal lengths in real-world data, which is a good motivation in my view. However, I do not think Era3D fully closed this problem, given that only a limited number of focal lengths are considered when preparing the training data. The authors should include this as a limitation or discuss it as a possible future work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Previous image-to-3D methods struggle when the object in the image is distorted due to a small focal length. Therefore, this paper focuses on a practical problem. Given the development of previous image-to-multiview methods, it is common to apply a cross-view attention mechanism to address the multiview consistency problem. This paper reconsiders the shortcomings of previous cross-view attention methods and proposes a better solution.", "weaknesses": "My main concern is with the presentation of experiments. As mentioned in L136, the motivation is to improve performance on real-world data, where small focal-length cases exist. Fig. 2 shows a good example. Before reading the experiments, I expected to see more results like Fig. 2. However, I think only the toaster example in Fig. 5 demonstrates the superiority of Era3D in this regard. The main paper should present more results. \n\nBy comparison, the results in Fig. 6 seem more focused on demonstrating the generalization ability of Era3D and how other methods suffer from out-of-distribution issues. Although I understand that the robust generalization ability of Era3D might be due to EFReg, given that other methods are also trained on Objaverse, I think this presentation is not explicit. In particular, the focal length of the images output by SDXL cannot be controlled, so it is unclear whether Era3D's better performance is due to EFReg or better preprocessing of the training data.\n\nMoreover, I disagree with the claim in L97 that Era3D is the first method to solve distortion artifacts. For example, \"Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion\" (CVPR23) also explores this issue by training directly on Pscal3D+, a real-world dataset with varying focal lengths. VQA-Diff (ECCV24) explores a similar problem and proposes to utilize prior knowledge of LLMs. Despite this, I find the proposed EFReg to be an interesting and novel solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:  \n\n**Q1: More results of perspective input.**  \nWe appreciate this suggestion and have accordingly expanded our results section. In Figure 2 of the global response,  we include more results on GSO datasets and cases from the Internet. Compared with Wonder3D and more recent Unique3D, our method demonstrates superior performance in mitigating distortion and reconstructing plausible 3D shapes. As per your recommendation, we will incorporate more results into the main paper.\n\n**Q2: The presentation about EFReg is explicit. Is the better performance from EFReg or preprocessing of training data?**  \nIn Fig.7 of the main paper, we evaluate the effectiveness via qualitative comparison by removing EFReg. Notably, without EFReg, the resulting shape is distorted and\nfails to generate reasonable novel views in the canonical camera setting. To provide quantitative support for these observations, we additionally report the Chamfer Distance in Tab.1. We use orthogonal renderings at an elevation of 0° as the reference (last column) and vary the elevation from −10° to 40° and select focal lengths from {35, 50, 85, 105, 135, ∞} to assess the system’s robustness to distortions. These results consistently confirm that EFReg significantly contributes to robust pose estimation and enhances overall reconstruction accuracy.\n\n**Table 1**: Ablation of EFReg on GSO datasets with various elevation (α) and focal lengths (f). We report the Chamfer Distance (↓).\n\n| Pose | α=0 | | | | | f=∞ | | | | | α=0, f=∞ |\n|------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|------------|\n|      | f=35 | f=50 | f=85 | f=105 | f=135 | α=-10 | α=10 | α=20 | α=30 | α=40 | |\n| w/o EFReg | 0.0237 | 0.0233 | 0.020 | 0.022 | 0.0217 | 0.0221 | 0.0217 | 0.0225 | 0.0231 | 0.0228 | 0.0217 |\n| w EFReg | 0.0223 | 0.0219 | 0.0216 | 0.0214 | 0.0214 | 0.0217 | 0.0216 | 0.0216 | 0.0219 | 0.0217 | 0.0213 |\n\n**Q3: Missing metrics of PSNR and SSIM.**  \nWe apologize for the missing reports of PSNR and SSIM. Following your valuable suggestion, we have now incorporated these results in Tab.2 to provide a more comprehensive evaluation following your suggestion. Our method significantly outperforms others, consistent with our performance on other geometry metrics.\n\n**Table 2:** Quantitative evaluation of SSIM and PSNR.\n| Method | RealFusion | Zero-1-to-3 | SyncDreamer | Wonder3D | Ours |\n|--------|------------|-------------|-------------|----------|------|\n| SSIM(↑) | 0.722 | 0.779 | 0.798 | 0.811 | **0.837** |\n| PSNR(↑) | 15.26 | 18.93 | 20.05 | 20.83 | **22.74** |\n\n\n**Q4: Inaccurate claim of 'the first method to solve perspective distortion'.**  \nWe acknowledge that our initial statement requires refinement. The study in CVPR'23 attempts to predict the distortion in category-specific image reconstruction. VQA-Diff only explores vehicle distortion in Autonomous Driving. In contrast to them, our study first considers distortion artifacts for general 3D object generation. We will revise the statement in the manuscript.  \n\n**Q5: Does RMA lead to performance degradation compared with dense/epipolar MV attention?**   \nTheoretically, row-wise, epipolar, and dense MV attentions are equivalent in our orthogonal setup. However, both dense MV attention and epipolar attention consume a large amount of GPU memory in the training, which cannot scale up to high resolution 512$\\times$512 and largely reduces the training batch size. As observed by Zero123 and other works, a large training batch size is very important for training a high-quality diffusion model. Due to the memory limitation, we cannot conduct a fair comparison experiment to train the model with dense MV attention or epipolar line attention.\n\n**Q6: Era3D does not fully address the distortion issue. It should be included as a limitation.**  \nThanks for your kind reminder. We use the commonly used focus lengths for training Era3D, which significantly mitigates perspective distortions. We acknowledge that our work offers a promising avenue for addressing this challenge rather than fully resolving the issue. We will clarify this point and discuss it in the limitation section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable comments. In summary, the reviewers are positive about the novelty, performance, and potential of our method: **"address several significant challenges associated with MV Diffusion"**(R-5Kpg), **"the first work that considers the change of the camera intrinsic matrixes for the image-to-3D generation"**(R-a9Xx) and **"be an interesting and novel solution"**(R-aLmz). 

We include the necessary Figures and Tables in the attached PDF file. We respectfully direct the reviewers to the corresponding section for detailed response.

Meta-review: The paper introduces Era3D, a method for generating high-resolution multiview images from a single image using efficient row-wise attention and a diffusion-based camera prediction module. The method addresses key challenges in multiview diffusion, such as camera prior mismatch and computational inefficiency, and achieves state-of-the-art results in generating detailed 3D meshes.
We are glad to accept the paper because of its contribution to multiview image generation and all positive comments from reviewers.
The authors are still encouraged to include all concerns and suggestion from reviewers into their camera-ready version.
Decision: Accept (poster)