GeoGS3D: Single-view 3D Reconstruction via Geometric-aware Diffusion Model and Gaussian Splatting

## Abstract

Abstract We introduce GeoGS3D, a novel two-stage framework for reconstructing detailed 3D objects from single-view images. Inspired by the success of pre-trained 2D diffusion models, our method incorporates an orthogonal plane decomposition mechanism to extract 3D geometric features from the 2D input, facilitating the generation of multi-view consistent images. During the following Gaussian Splatting, these images are fused with epipolar attention, fully utilizing the geometric correlations across views. Moreover, we propose a novel metric, Gaussian Divergence Significance (GDS), to prune unnecessary operations during optimization, significantly accelerating the reconstruction process. Extensive experiments demonstrate that GeoGS3D generates images with high consistency across views and reconstructs high-quality 3D objects, both qualitatively and quantitatively. Further examples can be found at the website .

## 1 Introduction

Single-view 3D reconstruction aims to recover 3D geometry and appearance of an object from a single RGB image. This task holds immense importance as it allows machines to understand and interact with the real 3D world, enabling various applications in virtual reality (VR), augmented reality (AR) (Kopf et al., ; Li et al., ) and robotics (Wang et al., ). Despite broad applications of the task, it remains highly challenging due to the inherent complexity of inferring 3D structures from a single 2D image.

Poineering works in multi-view diffusion models have made strides in improving 3D reconstruction by finetuning pre-trained image or video diffusion models on 3D datasets to enable multi-view synthesis (Liu et al., ; Shi et al., ; Kwak et al., ; Huang et al., ; Voleti et al., ; Long et al., ). However, maintaining multi-view consistency and handling objects with complex geometries remain difficult. Another line of research (Hong et al., ; Tang et al., ; Xu et al., ; ) proposes generalizable reconstruction models, generating 3D representation from one or few views in a feed-forward process, but these models usually demand substantial computational resources. For instance, LGM (Tang et al., ) requires training on 32 NVIDIA A100 (80G) for 4 days.

Meanwhile, 3D Gaussian Splatting (Kerbl et al., ) has emerged as a promising technique, offering high-quality rendering with competitive training and inference time as it combines the benefits of neural network-based optimization and explicit, structured data storage. Hence, it is desirable to explore Gaussian Splatting for efficient 3D reconstruction. However, current methods (Szymanowicz et al., ; Tang et al., ) using 3D Gaussian Splatting often feed a single image into the model, ignoring the spatial correspondence of multiple views. Additionally, we observe that the original implementation of Gaussian Splatting neglects the distance between 3D Gaussians, causing many unnecessary split and clone operations.

To address these limitations, we propose GeoGS3D, a novel two-stage framework for single-image 3D reconstruction. Our approach consists of a geometric-aware multi-view generation stage, followed by an accelerated 3D reconstruction stage. In the generation stage, we aim to synthesize 3D-aware images that maintain multi-view consistency. To achieve the goal, 3D features are extracted as geometric conditions by decoupling the orthogonal planes, while the semantic condition is obtained with the CLIP (Radford et al., ) encoder. These conditions, together with the input image are then fed into the diffusion model (Rombach et al., ). In the reconstruction stage, we introduce epipolar attention to fuse the generated views, fully leveraging the geometric correlations between them. Furthermore, we accelerate the optimization process by introducing a novel metric, Gaussian Divergent Significance (GDS), to avoid unnecessary operations during 3D Gaussian Splatting.

Extensive experiments and ablations on Objaverse (Deitke et al., ) and Google Scanned Object (Downs et al., ) datasets demonstrate that our method is able to produce high-quality 3D objects with strong multi-view consistency and detailed geometry. Our key contributions can be summarized as following:

- •
We incorporate an orthogonal plane decomposition mechanism with a diffusion model to synthesize multi-view consistent and geometric-aware novel view images.
- •
In order to take full advantage of the consistent multi-view images, we introduce epipolar attention into the optimization process, allowing for efficient and effective communication between images.
- •
To accelerate the optimization of Gaussian splatting, we derive a novel metric named Gaussian Divergent Significance (GDS) to prune unnecessary split and clone operations during optimization.

## 2 Related Work

### 2.1 Multi-view Diffusion Models

Recent 2D diffusion models (Song et al., ; Ho et al., ; Song & Ermon, ; Song et al., ; Xing et al., ; ; ) make impressive advances in generating images from various conditions. As an intermediate 3D representation, the generation of multi-view images using diffusion models has been explored. The advantage of multi-view images is that they are batched 2D projections and can be directly processed by existing image diffusion models with minor changes. Zero-123 (Liu et al., ) injects camera view as an extra condition to the diffusion model for generating images from different perspectives. Additionally,  Shi et al. () proposes replacing self-attention with multi-view attention in the UNet to produce multi-view consistent images. Other works (Chan et al., ; Liu et al., ; Szymanowicz et al., ; Yang et al., ; Tseng et al., ; Sargent et al., ) follow a similar approach to make diffusion model 3D-aware and enhance generation consistency. However, most of these works are not designed for reconstruction tasks and still rely on SDS loss to derive 3D content. Recent efforts such as SyncDreamer (Liu et al., ) and Wonder3D (Long et al., ) generate multi-view consistent 2D representations, subsequently applying reconstruction methods to obtain 3D content. Despite these advancements, the resulting 3D outputs often lack comprehensive geometric information. In contrast, our method generates more consistent views with detailed geometric information, facilitating high-quality 3D content reconstruction in terms of texture and geometry.

### 2.2 3D Reconstruction Pipelines

Researchers have explored directly training diffusion models on 3D representations (Jun & Nichol, ; Müller et al., ; Shue et al., ). However, they require exhaustive 3D data and computation resources and are also limited to category-level shape generation with simple textures. Other works (Poole et al., ; Melas-Kyriazi et al., ; Kwak et al., ; Sun et al., ; Tang et al., ; ; Wang et al., ; Zhang et al., ) proposed to lift 2D diffusion models for 3D generation via Score Distillation Sampling (SDS), optimizing various 3D representations including NeRF, mesh, SDF and Gaussian Splatting. However, these approaches often face challenges such as prolonged optimization times, the Janus problem, and over-saturated colors. More recent works (Xu et al., ; Szymanowicz et al., ; Siddiqui et al., ; Charatan et al., ; Xu et al., ; Tochilkin et al., ; Li et al., ) train large reconstruction models to directly map sparse views to 3D representations. Notably, LRM (Hong et al., ), Instant3D (Li et al., ), and GRM (Xu et al., ) utilize large transformers to predict triplanes from single or sparse views. TriplaneGaussian (Zou et al., ) and LGM (Tang et al., ) instead map sparse views into more memory-efficient 3D Gaussian Splatting, allowing for much higher resolution in supervision. Our work aligns with Liu et al. (; ); Tang et al. (); Wen et al. (), where we first multi-view diffusion model to generate consistent images, and then utilize reconstruction methods to obtain the corresponding 3D model.

## 3 Methodology

Figure: Figure 2: Overview of our method. In generation stage, we extract 3D features from the single input image by decoupling the orthogonal planes, and feed them into the UNet to generate high-quality multi-view images. In reconstruction stage, we leverage the epipolar attention to fuse images with different viewpoints. We further leverage Gaussian Divergent Significance (GDS) to accelerate the adaptive density control during optimization, allowing competitive training and inference time.
Refer to caption: x2.png

The overview of GeoGS3D is shown in Figure 2. Starting from an input image, our geometric-aware diffusion model first generates multi-view images sequentially in the generation stage (refer to Section 3.2). Subsequently, in the reconstruction stage, epipolar attention is incorporated into the Gaussian Splatting process to reconstruct the high-quality 3D objects (refer to Section 3.3). Additionally, we introduce a metric to accelerate the adaptive density control during Gaussian Splatting (refer to Section 3.4).

### 3.1 Preliminaries

#### Notations

Given an input image $\mathcal{I}_{0}$, our method aims to generate multiview-consistent images $\{\mathcal{I}_{i}\}_{i=1}^{N}$ and then reconstruct high-quality 3D Gaussians $\mathcal{G}$ with them. The multi-view images are obtained through a diffusion model conditioned on the input image $\mathcal{I}_{0}$ and a set of corresponding relative camera pose change $\{\Delta\pi_{i}\}_{i=1}^{N}$ during the generation stage. In the following reconstruction stage, the 3D Gaussians $\mathcal{G}$ are then optimized with these multi-view images through an accelerated Gaussian Splatting.

#### 3D Gaussian Splatting

(Kerbl et al., ) is a learning-based rasterization technique for 3D scene reconstruction and novel view synthesis. Each Gaussian element is defined with a position (mean) $\bm{\mu}$, a full 3D covariance matrix $\bm{\Sigma}$, color $c$, and opacity $\sigma$. The Gaussian function $G(x)$ can be formulated as:

$$ $G(x)=exp(-\frac{1}{2}(\bm{x}-\bm{\mu})^{T}\bm{\Sigma}^{-1}(\bm{x}-\bm{\mu})).$ (1) $$

To ensure the positive semi-definiteness of $\bm{\Sigma}$, the covariance matrix $\bm{\Sigma}$ can be factorized into a scaling matrix $S$ represented by a 3D-vector $s\in\mathbb{R}^{3}$ and a rotation matrix $R$ expressed as a quaternion $q\in\mathbb{R}^{4}$ for the differentiable optimization: $\bm{\Sigma}=RSS^{T}R^{T}$.

The rendering technique of splatting, as initially introduced in Kerbl et al. (), is to project the Gaussians onto the camera image planes, which are employed to generate novel view images. Given a viewing transformation $W$, the covariance matrix $\bm{\Sigma}{{}^{\prime}}$ in camera coordinates is given as: $\bm{\Sigma}{{}^{\prime}}=JW\bm{\Sigma}W^{T}J^{T}$, where $J$ is the Jacobian matrix of the affine approximation of the projective transformation. After mapping 3D Gaussians to a 2D image space, we count 2D Gaussians that overlap with each pixel and calculate their color $c_{i}$ and opacity $\sigma_{i}$ contribution. Specifically, the color of each Gaussian is assigned to every pixel based on the Gaussian representation described in Equation 1. And the opacity controls the influence of each Gaussian. The per-pixel color $\hat{C}$ can be obtained by blending N ordered Gaussians: $\hat{C}=\sum_{i\in N}c_{i}\sigma_{i}\prod_{j=1}^{i-1}(1-\sigma_{i})$.

### 3.2 Geometry-aware Multi-view Generation

Considering the success of pre-trained diffusion models (Rombach et al., ), it is intuitive to finetune them for novel view synthesis under a given camera transformation. However, maintaining multi-view consistency across views remains a substantial challenge. One stream of methods (Woo et al., ; Ye et al., ) addresses the problem by conditioning on previously generated images, which tends to be susceptible to cumulative errors and reduced processing speeds. Another stream of methods (Liu et al., ; ) solely use the reference image and semantic guidance to generate novel views, but suffer from collapsed geometry and limited fidelity.

We argue that the key lies in fully utilizing the geometric information provided by the input image. However, directly extracting 3D information from a single 2D image is not feasible. Thus, it is imperative to effectively disentangle 3D features from the image plane (*i.e.* $xy$-plane) by decoupling orthogonal planes. We first employ a vision transformer to encode the input image and capture overall correlations in the image, generating high-dimensional latent $\bm{h}$. Then we leverage two decoders, an image-plane decoder and an orthogonal-plane decoder, to generate geometric-aware features from the latent. The image-plane decoder reverses the encoding operation, leveraging a self-attention mechanism on the encoder output and converting it into $F_{xy}$. In order to generate orthogonal-plane features while maintaining structural alignment with the image plane, a cross-attention mechanism is employed to decode $yz$ and $xz$ plane features $F_{yz}$ and $F_{xz}$. To facilitate the decoding process across different planes, we introduce a learnable embedding $\bm{u}$ that supplies additional information for decoupling new planes. The learnable embedding $\bm{u}$ is first processed through self-attention encoding and then used as a query in a cross-attention mechanism with the encoded image latent $\bm{h}$. The image features are converted into keys and values for the cross-attention mechanism as following:

$$ $\texttt{CrossAttn}(\bm{u},\bm{h})=\texttt{SoftMax}\bigg{(}\dfrac{(W^{Q}\texttt {SelfAttn}(\bm{u}))(W^{K}\bm{h})^{T}}{\sqrt{d}}\bigg{)}(W^{V}\bm{h}),$ (2) $$

where $W^{Q}$, $W^{K}$, and $W^{V}$ are learnable parameters and $d$ is the scaling coefficient. Finally, the features are combined as geometric conditions:

$$ $F=F_{xy}ⓒ(F_{yz}+F_{xz}),$ (3) $$

where ⓒ and $+$ are concatenation and summation operations, respectively.

#### Training Objective

Similar to previous works (Rombach et al., ; Ho et al., ), we use a latent diffusion architecture with an encoder $\mathcal{E}$, a denoising UNet $\epsilon_{\theta}$, and a decoder $\mathcal{D}$. Following Liu et al. () and Liu et al. (), the input view is channel-concatenated with the noisy target view as the input to UNet. We employ the CLIP image encoder (Radford et al., ) for encoding $\mathcal{I}_{0}$, while the CLIP text encoder (Radford et al., ) is utilized for encoding the relative camera pose $\Delta\pi$. The concatenation of their embeddings, denoted as $c(\mathcal{I}_{0},\Delta\pi_{i})$, forms the semantic condition in the framework. We can learn the network by optimizing the following objective:

$$ $\mathcal{L}_{mv}=\mathbb{E}_{z\sim\mathcal{E}(\mathcal{I}),t,\epsilon\sim \mathcal{N}(0,1)}\|\epsilon-\epsilon_{\theta}(z_{t},t,c(\mathcal{I}_{0},\Delta \pi_{i}))\|_{2}^{2}$ (4) $$

Figure: Figure 3: Illustration of epipolar line and epipolar attention The epipolar line for a given feature point in one view is the line on which the corresponding feature point in the other view must lie, based on the known geometric transformation.
Refer to caption: x3.png

### 3.3 Reconstruction with Epipolar Attention

During the reconstruction stage, we target to exploit the synthesized consistent multi-view images for restoring high-quality 3D objects. However, relying solely on cross-attention to communicate between images of multiple viewpoints is insufficient. Therefore, we propose epipolar attention to allow association between the features of different views. The epipolar line for a given feature point in one view is the line on which the corresponding feature point in the other view must lie, based on the known geometric relationship between two views. It acts as a constraint to reduce the number of potential pixels in one view that can attend to another view. We present the illustration of epipolar line and epipolar attention in Figure 3.

Consider the intermediate UNet feature $f_{s}$, we can compute its corresponding epipolar lines $\{l_{t}\}_{t\neq s}$ on the feature map of all other views $\{f_{t}\}_{t\neq s}$ (please refer to the Appendix B for the details). Each point $p$ on $f_{s}$ will only access the features that lie along the camera ray (in other views) as all points in its own views during rendering. We then estimate the weight maps for all positions in $f_{s}$, stack these maps, and get the epipolar weight matrix $M_{st}$. Finally, the output of the epipolar attention layer $\hat{f}_{s}$ can be formulated as:

$$ $\hat{f}_{s}=\texttt{SoftMax}\bigg{(}\frac{f_{s}M_{st}^{T}}{\sqrt{d}}\bigg{)}M_ {st}.$ (5) $$

Our proposed epipolar attention mechanism limits the search space for corresponding features across different views, facilitating the efficient and accurate association of features across multiple views. In this way, we effectively reduce the computation cost as well as eliminate potential artifacts.

#### Training Objective

Given the input image $\mathcal{I}_{0}$ and multi-view images $\{\mathcal{I}_{i}\}_{i=1}^{N}$ with relative camera pose change $\{\Delta\pi_{i}\}_{i=1}^{N}$, we feed them into the network, and minimize the average reconstruction loss:

$$ $\mathcal{L}_{rec}=\frac{1}{N}\sum_{s=1}^{N}\|\mathcal{I}_{i}-g(f(\mathcal{I}_{ 0}),\Delta\pi_{i})\|^{2},$ (6) $$

where $g$ is the renderer that maps the set of Gaussians to an image and $f$ is an inverse function that reconstructs the mixture of Gaussians from an image.

The efficiency of our method stems from the idea that it renders the entire image at each training iteration. Therefore, instead of decomposing the results into pixels, we can leverage image-level losses as a whole. In practice, we employ SSIM loss to ensure the structural similarity between ground truth and synthesized images, and LPIPS loss for image quality, *i.e.*

$$ $\mathcal{L}=\mathcal{L}_{rec}+\lambda_{1}\mathcal{L}_{SSIM}+\lambda_{2} \mathcal{L}_{LPIPS},$ (7) $$

where $\lambda_{1}$ and $\lambda_{2}$ are the hyper-parameters of loss weights. Empirically, we set $\lambda_{1}=0.02$ and $\lambda_{2}=0.01$ as default.

### 3.4 Accelerating the Reconstruction

The optimization of Gaussian Splatting is based on successive iterations of rendering and comparing the resulting image to the training views. 3D Gaussians are first initialized from either Structure-from-Motion (SfM) or random sampling. Inevitably, geometry may be incorrectly placed due to the ambiguities of 3D to 2D projection. The optimization process thus needs to be able to adaptively create geometry and also remove geometry (termed as split and clone) if it is incorrectly positioned.

However, the split and clone operations proposed by the original work (Kerbl et al., ) overlook the distance between 3D Gaussians, during the optimization process which significantly slows down the process. We observe that if two Gaussians are close to each other, even if the positional gradients are larger than a threshold, they should not be split or cloned since these Gaussians are updating their positions. Empirically, splitting or cloning these Gaussians has negligible influence on the rendering quality as they are too close to each other. For this reason, we propose Gaussian Divergent Significance (GDS) as a measure of the distance of 3D Gaussians to avoid unnecessary splitting or cloning:

$$ $\Upsilon_{GDS}(G(\bm{x}_{1}),G(\bm{x}_{2}))=\|\bm{\mu}_{1}-\bm{\mu}_{2}\|^{2}+ tr(\bm{\Sigma}_{1}+\bm{\Sigma}_{2}-2(\bm{\Sigma}_{1}^{-1}\bm{\Sigma}_{2}\bm{ \Sigma}_{1}^{-1})^{1/2}),$ (8) $$

where $\bm{\mu}_{1}$, $\bm{\Sigma}_{1}$, $\bm{\mu}_{2}$, $\bm{\Sigma}_{2}$ are the position and covariance matrix of two 3D Gaussians $G(\bm{x}_{1})$ and $G(\bm{x}_{2})$.

To be specific, we only perform the split and clone operations on the 3D Gaussians with large positional gradients and GDS. To avoid the time-consuming process of calculating GDS for every pair of 3D Gaussians, we further propose two strategies. Firstly, for each 3D Gaussian, we locate its closest 3D Gaussian by leveraging the k-nearest neighbor (k-NN) algorithm and calculate their GDS for each pair. As a result, the time complexity is reduced from $O(N^{2})$ to $O(N)$. Additionally, as mentioned in Section 3.1, the covariance matrix can be factorized into a scaling matrix $S$ and a rotation matrix $R$: $\bm{\Sigma}=RSS^{T}R^{T}$. We take advantage of the diagonal and orthogonal properties of rotation and scaling matrices to simplify the calculation of Equation 8. Details of GDS will be discussed in the Appendix A.

Figure: Figure 4: Qualitative comparisons of generated multi-view images from Objaverse dataset. The artifacts are marked with red boxes. Our method achieves better consistency and visual quality.
Refer to caption: x4.png

## 4 Experiments

### 4.1 Experimental Protocols

#### Datasets

We train our multi-view generation model on the recently released Objaverse (Deitke et al., ) dataset, which is a large-scale CAD dataset containing 800K high-quality objects. We directly employ the processed rendering data from Zero-1-to-3, which provides 12 random views of each object. During the training, images are resized to $256\times 256$ resolution.

We evaluate the synthesized multi-view images and reconstructed 3D Gaussian Splatting (3D-GS) on the test split of Objaverse where we randomly select 100 objects beyond the training set. In addition, to test the performance of our model on the out-of-distribution data, we also evaluate the Google Scanned Object (GSO) dataset (Downs et al., ), which contains high-quality scanned household items.

#### Implementation Details

For the generation stage, we initialize the network with Zero-1-to-3 (Liu et al., ) pre-trained weights for training efficiency. We utilize a Vision Transformer (ViT) model of depth 6 as the reference image encoder and generate an output of size $1024\times 256$. The decoding process involves two decoders, *i.e.* image plane decoder and orthogonal plane decoder, each with a depth of 3 and outputs a feature map $F\in\mathbb{R}^{128\times 128\times 64}$. After the multi-view generation, we directly adopt the implementation of Yang et al. () to select 32 views with the highest perceptual quality score. For the reconstruction stage, the network that maps the input images to the mixtures of Gaussians is architecturally identical to the UNet (Song et al., ). The last layer is replaced with a $1\times 1$ convolutional layer with 15 output channels. As mentioned in Section 3.3, in order to allow the network to coordinate and exchange information between views, we add epipolar attention blocks after residual blocks followed by the cross-attention layers. We use the AdamW optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ with a learning rate of $10^{-4}$.

#### Metrics

We compare generated multi-view images and rendered views from reconstructed 3D-GS with the ground truth frames, in terms of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) (Wang et al., ), and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., ).

#### Baselines

In terms of multi-view generation, we compare our GeoGS3D with Zero-1-to-3 (Liu et al., ), Syncdreamer (Liu et al., ), and Wonder3D (Long et al., ). Zero-1-to-3 finetunes Stable Diffusion to achieve novel view synthesis under a given camera pose. Syncdreamer follows a similar intuition but learns the joint distribution of all target views. Wonder3D proposes cross-domain attention to propagate information between normal and image, generating normal maps and color images simultaneously.

For image-to-3D reconstruction, we adopt DreamGaussian (Tang et al., ), LGM (Tang et al., ) and InstantMesh (Xu et al., ). Notably, LGM and InstantMesh also adopt two-stage frameworks to achieve image-to-3D creation, which generates multi-view images and then utilizes large reconstruction models to obtain 3D objects. DreamGaussian initializes geometry and appearance using single-step SDS loss and then extracts a textured mesh with refinement guided by MSE loss.

### 4.2 Comparison with Existing Methods

#### Image to Multi-view

We first compare the generation stage of our method against recent image-to-multiview generation methods (Liu et al., ; ; Long et al., ). The quantitative results are shown in Table 1, and the qualitative results are shown in Figure 4. GeoGS3D surpasses all baseline methods regarding PSNR, LPIPS, and SSIM, indicating it provides sharper and more accurate results. According to the qualitative results, the nearby views synthesized by GeoGS3D are geometrically and semantically similar to the reference view, while the views with large viewpoint change showcase reasonable hallucination. Furthermore, the orthogonal-plane decomposition mechanism enables our model to capture the details of the input image.

**Table 1: Quantitative comparisons on the quality of generated multi-view images for image to multi-view task.**
|  | Objaverse | GSO |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| Zero-1-to-3 | 18.68 | 0.883 | 0.189 | 18.37 | 0.877 | 0.212 |
| SyncDreamer | 20.06 | 0.816 | 0.159 | 20.03 | 0.817 | 0.166 |
| Wonder3D | 23.52 | 0.895 | 0.130 | 22.69 | 0.883 | 0.147 |
| GeoGS3D (generation) | 23.97 | 0.921 | 0.113 | 22.98 | 0.899 | 0.146 |

#### Image to 3D

For the single-image 3D reconstruction task, we show the results in Table 2 and Figure 5. Statistically, GeoGS3D outperforms competing approaches by a substantial margin. From the visual results, our method is able to generalize to unseen data. This demonstrates the superiority of GeoGS3D over the current state-of-the-art methods and its capacity to generate high-quality 3D objects even with complex structures.

Figure: Figure 5: Qualitative comparisons for image-to-3D. The first two rows are from the Objaverse dataset, the next two are from the GSO dataset, and the final row is from an in-the-wild image. Our method demonstrates superior performance in terms of both visual fidelity and accuracy compared to existing approaches.
Refer to caption: x5.png

**Table 2: Quantitative comparisons on the quality of 3D representation for image-to-3D.**
|  | Objaverse | GSO |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| DreamGaussian | 19.97 | 0.860 | 0.205 | 20.24 | 0.856 | 0.179 |
| LGM | 20.44 | 0.861 | 0.222 | 18.72 | 0.842 | 0.231 |
| InstantMesh | 21.63 | 0.891 | 0.148 | 20.95 | 0.871 | 0.147 |
| GeoGS3D (reconstruction) | 22.35 | 0.894 | 0.143 | 21.75 | 0.896 | 0.141 |

### 4.3 Ablations and Analyses

The ablation studies below are conducted on the GSO dataset (Downs et al., ).

#### Ablations of multi-view generation

Our multi-view generation stage mainly consists of geometric and semantic guidance. Removing them respectively or simultaneously gives us four different combinations. As shown in Table 3 and Figure 6, the orthogonal-plane decomposition mechanism contributes the most to the geometric accuracy and consistency, bringing about visual enhancement to a great extent. The semantic guidance further increases the metric score and slightly improves visual consistency.

**Table 3: Ablation studies of multi-view generation.**
| id | geometric cond. | CLIP embedding | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| a | ✓ | ✓ | 22.98 | 0.899 | 0.146 |
| b | ✓ | ✗ | 20.79 | 0.878 | 0.175 |
| c | ✗ | ✓ | 18.37 | 0.877 | 0.212 |
| d | ✗ | ✗ | 17.05 | 0.801 | 0.203 |

Figure: Figure 6: Qualitative ablations of our key innovations.
Refer to caption: x6.png

#### Acceleration of the optimization

As mentioned in Section 3.4, we propose to use the Gaussian Divergent Significance (GDS) metric to further regularize the split and clone process. Table 5 demonstrates that this strategy has significantly reduced the optimization time while not sacrificing the reconstruction quality. Selecting the threshold carefully, our method leads to at most 15$\times$ faster convergence speed when compared with the original split and clone operation proposed in Kerbl et al. ().

**Table 4: Quantitative results of ablating GDS metric.**
| Threshold | LPIPS$\downarrow$ | recon. time ^4.3 |
| --- | --- | --- |
| w/o GDS | 0.214 | 15min |
| 0.01 | 0.168 | 93s |
| 0.1 | 0.146 | 55s |
| 0.5 | 0.147 | 78s |

#### Ablations of epipolar attention

To further validate the effectiveness of the proposed epipolar attention, we conduct ablation studies on it. We still adopt LPIPS to evaluate the quality of reconstruction. As presented in Table 5 and Figure 6, epipolar attention enables accurate association of features across views, fully utilizing the consistent multi-view images to synthesize high-quality 3D Gaussian Splatting.

### 4.4 Limitations and Future Works

While GeoGS3D shows promising results in reconstructing 3D objects from single-view images, there are still some limitations that the current framework does not entirely address. First, the number of generated views is fixed in our method. Adaptively generating different numbers of views for objects with different topological symmetries might further reduce the total reconstruction time. Additionally, our current method is restricted to single-object 3D reconstruction. It remains to be extended to complex scenes or multi-object reconstruction in the future.

## 5 Conclusions

In this work, we propose a two-stage model, GeoGS3D, to reconstruct 3D objects from single-view images. This method first synthesizes consistent and 3D-aware multi-view images via a diffusion model under the guidance of an orthogonal-plane decomposition mechanism. During the following 3D-GS reconstruction, epipolar attention is leveraged to communicate between relating multi-view images. A novel metric, *i.e.* Gaussian Divergent Significance (GDS), is proposed to accelerate optimization. Qualitative and quantitative results show that the proposed method reconstructs 3D Gaussian representations that 1) are consistent in different viewpoints, 2) are high fidelity to the reference image, and 3) display plausible creativity in the unseen areas.

## Appendix A Details of Gaussian Divergent Significance (GDS)

In this paper, we propose Gaussian Divergent Significance (GDS) as a measure of the distance of 3D Gaussians to avoid unnecessary split or clone operations:

$$ $\Upsilon_{GDS}(G(\bm{x}_{1}),G(\bm{x}_{2}))=\|\bm{\mu}_{1}-\bm{\mu}_{2}\|^{2}+ tr(\bm{\Sigma}_{1}+\bm{\Sigma}_{2}-2(\bm{\Sigma}_{1}^{-1}\bm{\Sigma}_{2}\bm{ \Sigma}_{1}^{-1})^{1/2}).$ (9) $$

The first term $\|\bm{\mu}_{1}-\bm{\mu}_{2}\|^{2}$ can be expressed as $(\bm{\mu}_{1}-\bm{\mu}_{2})^{T}(\bm{\mu}_{1}-\bm{\mu}_{2})$. Additionally, the covariance matrix can be factorized into a scaling matrix $S$ and a rotation matrix $R$: $\bm{\Sigma}=RSS^{T}R^{T}$. Since the scaling matrix is a diagonal matrix and the rotation matrix is orthogonal, we can thus simplify the computation of the second term.
Given:

$$ $\displaystyle\bm{\Sigma}_{1}=R_{1}S_{1}S_{1}^{T}R_{1}^{T}$ (10) $\displaystyle\bm{\Sigma}_{2}=R_{2}S_{2}S_{2}^{T}R_{2}^{T},$ $$

we have

$$ $\displaystyle\quad\;tr(\bm{\Sigma}_{1}+\bm{\Sigma}_{2}-2(\bm{\Sigma}_{1}^{-1} \bm{\Sigma}_{2}\bm{\Sigma}_{1}^{-1})^{1/2})$ (11) $\displaystyle=tr(\bm{\Sigma}_{1})+tr(\bm{\Sigma}_{2})-tr(2(\bm{\Sigma}_{1}^{-1 }\bm{\Sigma}_{2}\bm{\Sigma}_{1}^{-1})^{1/2})$ $\displaystyle=tr(R_{1}S_{1}S_{1}^{T}R_{1}^{T})+tr(R_{2}S_{2}S_{2}^{T}R_{2}^{T} )-tr(2(\bm{\Sigma}_{1}^{-1}\bm{\Sigma}_{2}\bm{\Sigma}_{1}^{-1})^{1/2})$ $\displaystyle=tr(S_{1}S_{1}^{T})+tr(S_{2}S_{2}^{T})-tr(2(\bm{\Sigma}_{1}^{-1} \bm{\Sigma}_{2}\bm{\Sigma}_{1}^{-1})^{1/2})$ $$

In this way, we can save the computation time for Gaussian Divergent Significance.

## Appendix B Details of Epipolar Line

Given the point $p^{t}$ on the target view image $\mathcal{I}_{t}$ and the relative camera pose change $\Delta\pi$, we show the computation of the epipolar line at the source view image $\mathcal{I}_{s}$. The relative pose change $\Delta\pi$ can be represented by the rotation and transformation from view $t$ to $s$ as $R^{t\rightarrow s}$ and $T^{t\rightarrow s}$. We first project the point $p^{t}$ onto the source view image plane as $p^{t\rightarrow s}$, namely

$$ $p^{t\rightarrow s}=\pi(R^{t\rightarrow s}(p^{t})+T^{t\rightarrow s}),$ (12) $$

where $\pi$ is the projection function. We also project the camera origin $o^{t}=[0,0,0]^{T}$ at the target view $t$ onto the source view image plane as $o^{t\rightarrow s}$:

$$ $o^{t\rightarrow s}=\pi(R^{t\rightarrow s}([0,0,0]^{T})+T^{t\rightarrow s}).$ (13) $$

Then the epipolar line of the point $p$ on the source view image plane can be formulated as

$$ $p_{epi}=o^{t\rightarrow s}+c(p^{t\rightarrow s}-o^{t\rightarrow s})\quad c\in \{-\infty,+\infty\}\in\mathbb{R}.$ (14) $$

Finally, the distance between a point $p^{s}$ on the source view image plane and the epipolar line can be computed as

$$ $d(p_{epi},p^{s})=\frac{\|(p^{s}-o^{t\rightarrow s})\times(p^{t\rightarrow s}-o ^{t\rightarrow s})\|}{\|p^{t\rightarrow s}-o^{t\rightarrow s}\|},$ (15) $$

where $\times$ and $\|\cdot\|$ indicate vector cross-product and vector norm, respectively. According to the epipolar line, we compute the weight map, where higher pixel values indicate a closer distance to the line

$$ $m_{st}(p^{s})=1-\texttt{sigmoid}(60(d(p_{epi},p^{s})-0.06))\quad\forall p^{s} \in\mathcal{I}_{s}.$ (16) $$

We use the constant 60 to make the sigmoid function steep and use the constant 0.06 to include the pixels that are close to the epipolar line. After estimating the weight maps for all positions in $f_{s}$, we stack these maps and reshape them to get the epipolar weight matrix $M_{st}$, which is used to compute the epipolar attention described in the paper.

## Appendix C Potential Social Impact

Our method mainly focuses on object reconstruction and does not involve the use of human data, ensuring that it does not violate human privacy or raise concerns regarding personal data misuse. However, despite this safeguard, it’s imperative to acknowledge potential negative social impacts. We are committed to ensuring that our technology is not misused to generate fake data or facilitate deceptive practices, such as counterfeiting or cheating. Ethical considerations and responsible deployment are paramount in our research and development efforts.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: -	Geometry-aware multi-view

 - How beneficial is the summation operation compared to multi-head attention? Could authors provide an ablation between multi-head attention and the proposed "cat-sum" operation in Eqn3? 
 - Can authors provide more evidence to back up the claim that the model learns to decompose 3D? For examples, visualization or ablation to strengthen the claim of the geometric decomposition.

-	Clarification on reconstruction model
 - Could authors provide a clear, step-by-step description of the entire reconstruction process, including how sections 3.3 and 3.4 relate to each other temporally and functionally?

- Novelty about epipolar attention
 - Could authors clarify what are the differences between the proposed epipolar attention and prior work?
 - Could authors provide citation to these approaches?
soundness: 1
strengths: -	The problem is of interest to the research community.
-	The paper is easy to follow.
summary: The manuscript introduces a framework named GeoGS3D, which can reconstruct 3D from a single image. It proposes to perform a decomposed attention mechanism for multi-view image generation. It also uses a reconstruction scheme to lift the multivew to 3D. The result of GeoGS3D was compared with prior works on GSO.
weaknesses: I am particularly concerned about the scientific contribution and clarity of the paper.

For contribution, my main concern is the claimed “geometry aware multi-view generation”. The paper claims to be able to decompose the multi-view generation into three planes, XY is the image plane, YZ and XZ are 3D planes to enhance the result. However, when looking at the formulation, this is only a variant of multi-head attention, where the QKV are split into three heads. The only difference from multi-head attention is the operation of summation in Eqn 3. The manuscript did not present any evidence of the ability to actually decompose the scene orthogonally. 

For the clarity issue, the reconstruction stage is very confusing. Is the reconstruction a feedforward model? If so, what’s the input and output? Why is 3.4 iterative optimization? Is this done after the feedforward model described in 3.3?

Epipolar attention has been proposed in the research community for a long time, especially in the multi-view stereo literature. The manuscript should not claim this as novelty and should provide citation to prior work, such as [a][b][c][d]. Currently, the work sounds like a simple extension of Zero123 and LGM, with the exception of epipolar attention and potentially a final refinement stage. I am not sure if this is significant enough as scientific discovery.

[a] Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., & Su, H. (2021). Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 14124-14133).
[b] Long, X., Lin, C., Wang, P., Komura, T., & Wang, W. (2022, October). Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision (pp. 210-227). Cham: Springer Nature Switzerland.
[c] Li, P., Liu, Y., Long, X., Zhang, F., Lin, C., Li, M., ... & Guo, Y. (2024). Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention. arXiv preprint arXiv:2405.11616.
[d] Li, Y., He, X., Jiang, Y., Liu, H., Tao, Y., & Hai, L. (2022, October). MeshFormer: High‐resolution Mesh Segmentation with Graph Transformer. In Computer Graphics Forum (Vol. 41, No. 7, pp. 37-49).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: Please clarify all points w.r.t. lack of clarity in the weaknesses section. To be more precise the questions are:
1. Are you using per-object optimization, feed-forward prediction, or a combination?
2. If it is a combination, how exactly do optimization and feed-forward components interact?
3. How are the epipolar attention and cross attention integrated into the used reconstruction process?
4. How is the geometric conditioning incorporated into the diffusion model's architecture?
5. How do you make use of the fact that the features should correspond to different planes in 3D?
6. What is the exact format or representation of the camera poses as input to the CLIP text encoder?
7. Why is the proposed epipolar attention supposed to be more efficient, if it is a weighted dense attention?
8. Do all reconstruction methods in table 2 use the same set of generated multi-view images as input or not?
soundness: 3
strengths: - The paper tackles an interesting and very challenging task of single-view 3D object reconstruction.
- The experimental evaluation shows strong performance with advantages over baselines in terms of
  - multi-view generation with a diffusion model (novel views quantitatively closer to ground truth (table 1) and qualitatively more reasonable (figure 4) and
  - 3D reconstruction (quantitatively better metric scores (table 2) and more detailed qualitative results (figure 5).
- The evaluation comprises both synthetic (Objaverse) as well as real (Google Scanned Objects) datasets with the later being out of training distribution.
- Ablation studies w.r.t. the proposed components (geometric conditioning, epipolar attention, and Gaussian Divergent Significance) validate the effectiveness of the contributions.
summary: The paper proposes a two-stage approach for single-view 3D object reconstruction via multi-view generation with a diffusion model first, followed by fitting of 3D Gaussians to the generated views to obtain the final 3D reconstruction. Compared to previous works, the authors propose a specific network architecture based on orthogonal plane decomposition to encode the image into a conditioning for the multi-view diffusion model. Furthermore, for the second 3D reconstruction stage, the authors claim to introduce epipolar attention into the optimization process and define an additional heuristic for the adaptive density control of 3DGS for faster fitting. An evaluation on Objaverse and Google Scanned Objects shows empirical advantages over baselines, both quantitatively and qualitatively. Ablation studies show the effectiveness of the proposed components.
weaknesses: - Lack of clarity: I do not understand major parts of the method.
  - In the reconstruction stage, it is not clear to me whether the method uses per object fitting of 3D Gaussians or a network that predicts Gaussians in a feed-forward manner or a combination of both and if so how, as I do not see how the proposed components are compatible.
    - In lines 70ff., authors name Splatter-Image [1] and DreamGaussians [2] as two current 3DGS-based reconstruction methods, i.e., one feed-forward approach for generalizable reconstruction [1] and one per object fitting with score distillation sampling [2].
    - In lines 133ff., the authors say that their work in terms of 3D reconstruction aligns with One-2-3-45 [3] and MVDiffusion++ [4], again one feed-forward approach [3] and one per object fitting [4]. However, both do not use 3D Gaussian Splatting.
    - Figure 2 shows some combination of the usual optimization loop of 3DGS with epipolar and cross attention, which does not provide a detailed explanation of how these are combined (also not in text).
    - Clues suggesting per object fitting:
      - In lines 150ff., the atuhors write " In the following reconstruction stage, the 3D Gaussians G are then optimized with these multi-view images through an accelerated Gaussian Splatting."
      - The use of adaptive density control with the proposed Gaussian Divergent Significance (GDS) as additional heuristic for when to apply split and clone operations. If Gaussians would be predicted in a feed-forward manner, I do not see how these non-differentiable operations could still be used.
      - The reconstruction time comparison in table 4: 15 minutes reconstruction time (without the GDS metric) should be too much for feed-forward prediction.
    - Clues suggesting feed-forward prediction:
      - The use of epipolar and cross attention. In lines 267ff., the authors say that these modules are applied to "the intermediate UNet feature[s]".
      - The training objective in equation 6 includes f as "an inverse function that reconstructs a mixture of Gaussians from an image" (285f.).
      - Lines 375ff.: "For the reconstruction stage, the network that maps the input images to the mixtures of Gaussians is architecturally identical to the UNet [...], we add epipolar attention blocks after residual blocks followed by the cross-attention layers."
  - The geometric conditioning in section 3.2 is missing an explanation of how exactly the features for the three different planes are used by the diffusion model. 
    - The subsection about the training objective (lines 234ff.) does not contain the geometric conditioning at all.
  - If the plane features are aggregated via concatenation and summation (equation 3) and then presumably used as conditioning of the diffusion model via cross attention, then It is unclear why this design choice should be effective:
    - There is no enforcement that the "orthogonal plane features" have anything to do with the three orthogonal planes in 3D.
    - If that is the case, F_yz and F_xz should be even equivalent and therefore redundant.
    - Lines 221f.: "The learnable embedding u is first processed through self-attention ...". Self-attention on learnable embeddings is unnecessary.
  - It is unclear how the relative camera poses are encoded by the CLIP text encoder (cf. 238f.).
  - The epipolar attention is claimed to "reduce the computation cost" (line 277), while it is not clear why this weighted attention should be more efficient.
  - In lines 288ff., the authors say that "the efficiency of our method stems from the idea that it renders the entire image at each training iteration...". All 3DGS approaches usually render entire images at once.

- Lack of related work and clarity regarding novelty of reconstruction with epipolar attention:
  - If epipolar attention is used in an architecture (UNet) that predicts Gaussians in a feed-forward manner, then this has been done before, e.g., by pixelSplat [5] and latentSplat [6].
  - If epipolar attention is used by the multi-view diffusion model itself, then this also has been done before, e.g., by Epidiff [7] and MVDiff [8].

- It is not clear whether the evaluation of the 3D reconstruction in table 2 is disentangled from the multi-view generation, i.e., whether they use the same generated images as input or not.
  - If not, that would be an interesting setup in order to better compare the different reconstruction techniques.
- The qualitative comparisons in figures 4 and 6 miss ground truth for reference.

Minor comments:
- The introduction is a bit repetitive w.r.t. the contributions (first in text form (lines 74ff.), then same again as a list (88ff.).
- As 3D Gaussian Splatting is mainly used as it is (except for the additional heuristic in the density control), I do not think that the extensive recap of it in preliminaries is really needed.

References:
- [1] Splatter Image: Ultra-Fast Single-View 3D Reconstruction. CVPR 2024
- [2] DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. ICLR 2024
- [3] One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. NeurIPS 2023
- [4] MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction. ECCV 2024
- [5] pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CVPR 2024
- [6] latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction. ECCV 2024
- [7] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion. CVPR 2024
- [8] MVDiff: Scalable and Flexible Multi-view Diffusion for 3D Object Reconstruction from Single-View. CVPR 2024

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: - In Figure 2, the authors utilize the standard 3D Gaussian Splatting, optimizing Gaussian points attributes through cloning and splitting for individual scenes. However, Line 377 describes using Unet and multiview images to output a fixed number of Gaussian points, which doesn't include point quantity control (similar to LGM). These two processes cannot exist in the same stage. So how is the point splitting and cloning work in this process, and how is the GDS works to  avoid unnecessary operation in splitting and cloning?

- Is the GDS metric a universal acceleration solution, or is it more suitable for object-level reconstruction?
soundness: 3
strengths: - The two-stage method presented in this paper performs remarkably well both in quantitative metrics and visual quality.
- The proposed GSD metric significantly improves the reconstruction speed of GS while enhancing the quality of results.
summary: GeoGS3D introduces a two-stage framework for reconstructing detailed 3D objects from single-view image. In generation stage, the framework innovatively replaces traditional single-view feature with 3D geometric feature, facilitating the generation of multi-view consistent images. In reconstruction stage,  the authors implement Gaussian Divergent Significance (GDS), which accelerates the optimization process by eliminating unnecessary operations during 3D Gaussian Splatting. Experiments demonstrate that GeoGS3D generates images with high consistency across views and reconstructs high-quality 3D objects.
weaknesses: - In Line 320, the complexity of KNN is not $O(N)$. For each point, we need to query its nearest neighbors using KD-tree: $O(log N)$. Therefore, the total time complexity is: $N * O(log N) = O(N log N)$ 
- In Line 221, the authors mention a learnable embedding $u$, however, this element is absent from Figure 2, creating confusion in the presentation. The authors should provide a more explicit explanation of how the output from $CrossAttn(u, h)$ is transformed into the triplane representations $(F_{xy}, F_{yz}, F_{xz})$.
- The paper lacks proper citations for epipolar attention, despite numerous prior works implementing this technique.
  - Generalizable Patch-Based Neural Rendering
  - pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction
  - EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. My main concern is the missing reference from Weaknesses (pt 1 and 2) and how is it different from the current method's epipolar attention mechanism as it is claimed to be new.
2. I would suggest the authors to reduce their metric novelty claim (Weaknesses pt 3).
3. Points 4 to 6 in weaknesses (technical details + writing) should be addressed.
soundness: 3
strengths: 1. Good demonstration of the combination of papers: Zero123, Epipolar attention (EpiDiff) and 3DGS to reconstruct 3D objects from single views. Uses them as components well.
2. Qualitative evaluations look promising
3. Sufficient ablation studies are performed to support the claimed motivation behind components.
summary: The paper proposes a single view to consistent 3D reconstruction method using geometrically consistent diffusion to produce multiple views for optimizing an explicit 3D reconstruction algorithm (3DGS). The paper proposed epipolar attention in the generation pipeline to achieve better multi-view consistency (over zero123's pretrained weights) and accelerate 3DGS using a so called novel metric Gaussian Divergence Significance.
weaknesses: 1. Lacks novelty. The epipolar attention mechanism is too similar to the recent paper: EpiDiff (CVPR 2024; 10 citations; [Link to paper](https://openaccess.thecvf.com/content/CVPR2024/papers/Huang_EpiDiff_Enhancing_Multi-View_Synthesis_via_Localized_Epipolar-Constrained_Diffusion_CVPR_2024_paper.pdf)) 

2. Missing direct reference & baseline: EpiDiff (from the point above) is not cited or compared against as a methodology. In my opinion, this is a directly related work and should ideally have been a baseline at the very least.

3. The GDS metric claim can not be called novel by any means. It is just the Wasserstein distance between two Guassians! Giving an existing distance measure a new name doesn't count as novelty. Maybe the authors should re-consider wording their strategy (as novel) for not pruning/splitting Gaussians based on a Wasserstein-distance threshold. 

4. Minor detail: The authors claim that GDS reduces the time complexity from O(N^2) to O(N) by not finding distances for all pairs using kNN. This is not a valid claim. Imagine a super noisy & compact scene. Here every Gaussian is close to each other due to compactness and the noise leads to large gradients - This will include all Gaussians again into consideration. Note that big-O notation is about worst case complexity and not average run-time complexity.

5. Table captions are not the most informative. For ex - the authors could improve table 3 by specifying what variants a, b, c and d are. It's hard to jump around the paper back and forth and retain the numbers at the same time. 

6. Why is GDS' formulation (a major claim) in the appendix? Similarly for epipolar attention (appendix B)? These should be a part of the main text since they are the main claims.

7. No in-the-wild object reconstructions are shown. Prior literature and direct usage component: Zero123 did that.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. Are $F_{yz}$ and $F_{xz}$ generated by the same $u$? Are the image plane decoder and orthogonal plane decoder jointly trained with the multi-view generative model?

2. How many views does the multi-view diffusion generate? How many are generated in total before selecting the 32 views with the highest quality scores?

3. The paper mentions a fixed number of generated views. Are the relative camera poses also fixed? 

4. How does the elevation of the input image affect the quality of generated views? For example, would the model fail if the input image is captured from a high elevation angle?

5. How long does the generation stage take, including the selection of 32 views?
soundness: 1
strengths: 1. The method is thoroughly evaluated on the Objaverse and GSO datasets, with quantitative results surpassing baseline approaches.

2. The mechanism of epipolar attention is clearly explained, and its application in reconstruction tasks is innovative.

3. The proposed GDS metric significantly reduces optimization time.
summary: This paper introduces a novel framework for generating 3D objects from single images. The proposed approach follows a two-stage process: first, generating multi-view images, and then reconstructing 3D Gaussians from these images. In the generation stage, it employs both geometric and semantic conditioning to guide a diffusion model for generating the target novel views. For the reconstruction stage, epipolar attention is utilized to fuse images from different perspectives, and a GDS is introduced to accelerate the reconstruction process.
weaknesses: 1. The design of the geometric condition is questionable. The goal appears to be the decoupling of features into three orthogonal planes. However, there is no mechanism to ensure that the decoupled features actually lie on orthogonal planes. In other words, there is no rationale for why these design choices could extract **3D** information. It seems to process 2D features solely through 2D attention modules.

2. Section 3.2 lacks an explanation of how the generated images maintain multi-view consistency, or if any module is introduced to enforce it. This is confusing, as the training objective in Equation 4 involves a single target view, and Figure 2 also displays only one target view. Does the model output only one target view per diffusion process, or does it generate multiple views, using cross-attention to ensure consistency (similar to Wonder3D)?

3. Although the calculations involved in epipolar attention are clearly explained, it’s not clear how this module is incorporated into the optimization process. Including an intuitive explanation would improve clarity.

4. The paper could include stronger baseline methods for comparison. For example, in multi-view generation, methods such as Era3D [1] and Zero123++ [2] could be added, and for image-to-3D, methods like TripoSR [3] and SF3D [4] could be relevant. The current baselines do not reflect the state-of-the-art in these fields.

5. The ablation on orthogonal-plane decomposition is unclear. The authors remove geometric guidance to evaluate the effect of the orthogonal-plane decomposition mechanism. However, the correct approach would be to remove the decomposition module itself and use the latent vector from a pre-trained ViT as a condition to isolate the impact of orthogonal-plane decomposition. Since previous works all condition the UNet on the input image feature, removing the condition entirely will certainly degrade performance.

6. The writing is unclear and omits essential details (see above weaknesses and questions sections).

Minor:

1. In Eq.4, the geometric condition is missing.

2. In Eq.6, the subscript $s$ doesn’t appear in the summation, which makes this equation confusing. 

[1] Li, Peng, et al. "Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention."

[2] Shi, Ruoxi, et al. "Zero123++: a single image to consistent multi-view diffusion base model."

[3] Tochilkin, Dmitry, et al. "Triposr: Fast 3d object reconstruction from a single image." 

[4] Boss, Mark, et al. "Sf3d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement."