Large Point-to-Gaussian Model for Image-to-3D Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 03:37:11

Abstract: Recently, image-to-3D approaches have significantly advanced the generation quality and speed of 3D assets based on large reconstruction models, particularly 3D Gaussian reconstruction models. Existing large 3D Gaussian models directly map 2D image to 3D Gaussian parameters, while regressing 2D image to 3D Gaussian representations is challenging without 3D priors. In this paper, we propose a large Point-to-Gaussian model, that inputs the initial point cloud produced from large 3D diffusion model conditional on 2D image to generate the Gaussian parameters, for image-to-3D generation. The point cloud provides initial 3D geometry prior for Gaussian generation, thus significantly facilitating image-to-3D Generation. Moreover, we present the Attention mechanism, Projection mechanism, and Point feature extractor, dubbed as APP block, for fusing the image features with point cloud features. The qualitative and quantitative experiments extensively demonstrate the effectiveness of the proposed approach on GSO and Objaverse datasets, and show the proposed method achieves state-of-the-art performance.

Main paper: 
 introduction: 
 Generating high-quality 3D assets from images is a pivotal task in numerous fields, notably in gaming, film production, and VR/AR, etc. The learning-based 3D generation algorithms [12, 33] allow rapid generation of high-quality 3D assets free of tedious manual processes and complex computer graphics tools.Recently, the realm of 3D generation has witnessed a surge of innovative techniques, with particular prominence given to two main approaches: 2D-lifting-based generation and feed-forward generation, driving the field's progress. Following the pioneering work [33] , the 2D-lifting approaches [17, 33, 50] leverages Score Distillation Sampling (SDS) [33] to distill 3D implicit representations (e.g., NeRF [30] ) from large diffusion models [20, 21] , while feed-forward generation approaches [12, 15, 45, 51] straightforward reconstruct 3D implicit representations from 2D image without iterative optimization. Compared to 2D-lifting approaches that require extensive optimization time, the feed-forward generation techniques can generate 3D assets within a few seconds. Besides, inspired by recent 3D Gaussian Splatting (3D-GS) [14] with promising rendering quality in the novel view synthesis (NVS) and fast rendering speed, the latest approaches improve the speed and quality of 3D generation by integrating the 3D-GS into 2D-lifting generation [6, 41, 52] or feed-forward generation [40, 49, 55] .The previous feed-forward approaches with 3D-GS [40, 49, 55 ] build mappings directly from implicit image features to Gaussian parameters. However, this regression based method is non-trivial for 3D-GS learning, given that the input 2D image does not always contain efficient 3D information for the corresponding object. By contrast, Point Cloud as an effective 3D representation is capable of providing informative geometry priors for the generation of explicit 3D Gaussians, whilst the current large 3D diffusion model (e.g. Point-E [31] ) can generate diverse and satisfying initial point cloud for the input image. Therefore, we present to generate Gaussian parameters from point cloud, dubbed as Point-to-Gaussian, to advance the 3D-GS learning for image-to-3D generation as shown in Fig. 2 .Nevertheless, it is still non-trivial to convert the coarse point cloud from 3D large models to Gaussian representations given that the generated point cloud might sparse and noisy, which carrying insufficient appearance feature and inaccurate geometry structural information to provide informative prior for precise Gaussians generation. We thus first utilize a point cloud upsampler to densify it and thus enhance the features, and then utilize a point feature extractor to extract the feature from the point cloud. Besides, considering that the input image contains rich appearance information, we present to enhance the corss modality features to fuse the 2D image representations to the 3D point cloud for facilitating the 3D representations. Specifically, we project the 3D point to 2D image according to camera pose to obtain the features. However, there are occlusions in the point cloud during the projection. We introduce an attention mechanism to selectively query features for 3D points, enhancing their representations, particularly under occlusion. Subsequently, we integrate the 3D representations obtained from the Attention mechanism, pose-aware Projection mechanism, and Point cloud feature extractor, termed APP Block, to conduct cross modality enhancement for more effective Gaussian learning.The Gaussian parameters are finally generated using a multi-head Gaussian decoder, and the novel view images are rendered by conventional Gaussian splatting.Our method converges quickly, which is trained only on the objaverse-LVIS [10] subset and achieves the comparable results of previous state-of-the-art methods trained with much more data, which indicates the effectiveness of the proposed method.In summary, our main contributions are as follow:• We propose a novel framework to generate high-quality 3D Gaussians with point cloud input. To our knowledge, our method is the first attempt to utilize the generalizable Point-to-Gaussian Generator for feed-forward image-to-3D generation.  
related work 2.1 diffusion priors for 3d generation: 
 Recent efforts for generating 3D contents are mostly inspired from the success of 2D generative models [35] . They usually rely on Score Distillation Sampling (SDS) proposed in DreamFusion [33] , which minimizes the difference between rendered images from the 3D object and the 2D diffusion with CLIP priors [34] . [43] interprets predictions from pretrained diffusion models as a score function of the data log-likelihood to optimize 3D representations via score matching. Subsequent work has made further enhancements based on SDS-based optimization. [17] increased the resolution of the generation, and introduced direct mesh optimizations. [47] proposed Variational Score Distillation (VSD) to enhance the quality and diversity of the generations. [50] introduced 3D shape prior from text-to-shape for better alignment between text and 3D shape. [4, 16, 36, 38] are proposed to achieve substantial improvement in quality, and further alleviate the Janus problem. [29, 41] aim to improve the optimization speed of 3D generation. Besides, many efforts [19-22, 24, 27, 37, 44 ] extend text-to-3D to single-view 3D generation. Theses methods focus on exploring non-optimization paradigms, and propose multi-view diffusion models which incorporates the information between different views for more consistent multi-view generation. Then the 3D reconstruction approaches are applied to obtain the 3D representations. SV3D and V3D [8, 42] are proposed to leverage video diffusion models for generating more consistent multi-view images.Unlike the SDS-based approach, the single-stage 3D generation method obtains 3D representation directly from image or text by a single feed-forward. Previous works attempt to train 3D diffusion models directly on point clouds or volumes [3, 13, 31] . However,Image InputPoint Cloud 3D GaussiansFigure 2 : Illustration of our full pipeline. Given a single-view image as input, the corresponding point cloud is first generated via pre-trained 3D diffusion model [13] . Then the point cloud is sent as the input of our proposed Point-to-Gaussian Generator.The input image is introduced as complementary condition to our Gaussian generator.they do not generalize well to other scenes and cannot provide satisfactory textures. Recently, LRM [12] first utilizes large transformers trained on large-scale 3D datasets [9, 10] to directly predict triplane NeRF from single view in a few seconds. [45] and [15] extend the input from single-view to sparse views. [15, 51] combine the large reconstruction model with diffusion priors to achieve text-to-3D and image-to-3D generation. However, these methods utilize volume rendering based triplane NeRF as the 3D representation, which still require massive forward inferences and computation.3D Gaussian Splatting (3D-GS) [14] showed marvelous performance in novel view synthesis for single scene optimization. This representation has been applied in many downstream tasks, such as generalizable reconstruction [1, 5, 39] , human reconstruction [54] , and 4D generation [18] , etc. Some concurrent works [7, 41, 52] adopt 3D-GS for SDS-based optimization to decrease generation time. In the field of single-stage 3D generation, [55] combines LRM with 3D-GS for faster rendering speed and superior rendering quality, [49] proposes two-stage optimization with 3D-GS for high-quality generation, [40] extends the single view Gaussian generation to sparse views for higher resolution. In this paper, we follow the feed-forward based image-to-3D generation scheme and propose a generalizable point-to-Gaussian model to advance the 3D-GS learning for image-to-3D generation. 
method: 
 In this section, we first present the fundamental background of 3D Gaussian splatting (Section 3.1). Then, the Point to Gaussian generator, which takes the sparse point cloud generated from the pretrained 3d diffusion model and the paired images condition as inputs and outputs the 3D Gaussian representations, is introduced in (Section 3.2). To employ image conditions for further enhancing the geometric and texture features of 3D Gaussians, we present to integrate the 3D representations obtained from the Attention, Projection, and Point feature extractor, termed APP Block in (Section 3.3) to enhance the cross modality features. Lastly, the loss function and data augmentation are introduced in (Section 3.4) for optimization.3D Gaussian Splatting (3D-GS) [14] shows high-fidelity rendering quality and real-time speed in novel view synthesis (NVS), which has gained a lot of popularity. 3D-GS renders images via splatting instead of volume rendering that is commonly used in implicit representation like NeRF [30] . Specifically, 3D-GS represents scenes with a set of explicit anisotropic 3D Gaussians. Each Gaussian distribution is defined by a 3D covariance matrix 𝚺 and a center position at point (mean) X. A 3D Gaussian distribution 𝐺 (X) is formulated as follows:EQUATIONFor more effectively optimized by gradient decent, the 3D covariance matrix Σ can be decomposed into a rotation matrix R and a scaling matrix S:EQUATIONwhere R and S are two learnable parameters. During optimization, the rotation matrix 𝑅 is transformed to quaternion 𝑟 . Each Gaussian consists of an opacity 𝜎 and spherical harmonics (SH) coefficients 𝑐 for rendering. Therefore, the complete Gaussian parameters are defined by G = {(X 𝑖 , S 𝑖 , R 𝑖 , 𝜎 𝑖 , 𝑐 𝑖 )} 𝑛 𝑖=0 . The Gaussians are projected from 3D space to 2D image plane for rasterization with viewing transform W, then the 2D covariance matrix 𝚺 ′ can be computed asEQUATIONwhere J is the Jacobian of the affine approximation of the projection transformation. Finally, the color 𝐶 of each pixel is accumulated by blending the overlapping Gaussians:EQUATIONwhere 𝛼 𝑖 is 𝜎 𝑖 multiplied by 𝚺 ′ . The tile-based rasterizer is utilized for efficient forward and backward pass. In this paper, we reduce the degree of the SH coefficients in Gaussians to zero which represents only the diffuse color. We also remove the densification and pruning proposed in conventional per-scene 3D-GS optimization to adapt amortized optimization.In this section, we introduce the architecture of our Point to Gaussian Generator. As shown in Fig. 3 , the Point to Gaussian Generator shares the encoder-decoder structure, which converts the point cloud to 3D Gaussians. Specifically, we leverage the point cloud generated by a pretrained diffusion model [31] for initialization, and then upsample the points with densification operation. Meanwhile, the conditional images (could be single or multi) are also incorporated to enrich the Gaussian features. Finally, a multi-head gaussian decoder is incorporated to decode the features into Gaussian parameters for splatting. Denote that the point cloud with color is of 𝑃 𝑁 * 6 , conditioning images {𝐼 𝑣 } 𝑉 𝑣=1 and camera parameters {𝐶 𝑣 } 𝑉 𝑣=1 , the output can be formulated asEQUATIONHere, the G = {(X 𝑖 , S 𝑖 , R 𝑖 , 𝜎 𝑖 , 𝑐 𝑖 )} 𝑁 𝑖=0 represents the 𝑁 Gaussians.To simplify the learning of 3D Gaussians, we utilize the point cloud as input. In single-scene optimization, as reported in the primitive 3D-GS [14] , pruning and densification techniques are employed to adjust the Gaussian numbers.Generally, sufficient number of 3D Gaussians can fairly represent the corresponding 3D objects, but an excess of Gaussians can also introduce computational and storage overheads. However, in the generalized 3D Gaussian framework, the gradient operates on the network, instead of the Gaussian itself, making it challenging to control and adjust the number of Gaussians dynamically. To strike a balance between performance and overhead, we initially perform a densification operation by upsampling the point cloud generated from the pretrained 3D diffusion model, thereby augmenting the number of Gaussians in the network's final output. Specifically, we rely on the methodology outlined in [48] to implement a dense sampling operation on the point cloud.3.2.2 Multi-Scale Gaussian Decoder. The Gaussian decoder's architecture adopts a U-Net structure, akin to that described in [23] . Following densification, as detailed in Section 3.2.1, the point cloud is inputted into the network. During downsampling, the number of point clouds progressively decreases, and the current layer's point cloud is derived via farthest point sampling (FPS) from the preceding shallower layer, thereby generating multi-scale point cloud features and expanding the receptive field. To further enrich the point cloud features and Gaussian attributes, we introduce projection and attention mechanisms for cross modality enhancement. For further elaboration, please refer to Section 3.3.After obtaining the enhanced features, we introduce the multi linear heads, which utilizes multiple decoder heads for different attributes in the Gaussian. Since the coordinates of the point cloud and the Gaussian are not exactly the same in space, we take inspiration from [55] and learn the positional offset of the Gaussians concerning the point cloud, instead of the centers themselves, to simplify the learning process. In this section, we present the core component of the our Point to Gaussian generator, which integrate the 3D representations obtained from the Attention mechanism, Projection mechanism, and Point feature extractor, termed as APP, for cross modality enhancement, as shown in Fig. 3 . The point feature extractor we employ combines an efficient fusion of point-based methods [2] with the strong spatial inductive bias of voxel-based methods [56] , which extracts the geometry and texture feature from the colored point cloud. Despite employing a multi-scale feature extraction module to provide a larger receptive field, the features extracted from point cloud input remain carries insufficient appearance feature and inaccurate geometry structural information to provide informative prior for precise GS generation. To further integrate the rich texture from the image modality into the point cloud tokens, we designed projection and attention modules which will be discussed in the following sections., where an efficient and intuitive approach is projection.Inspired by this, we fuse multi-scale point cloud tokens with image tokens using projection, complementing each point cloud with a feature from the image modality based on its position in space and the view of the input image. Specifically, drawing on insights from [28] , the projection considers the self-occlusion of the point cloud in space and employs a fast rasterization technique to map the pixel-wise image features to the visible points. However, considering that the point cloud is assumed to be volumetric and non-transparent during the rasterization process, it is inevitable that points opposing the current camera viewpoint will be invisible during this process. As illustrated in Fig. 4 , in the two provided viewpoints on the left side, we can only observe one side of the character, implying that the occluded point cloud on the other side will not participate in the projection process.To further compensate for the point clouds that are against the current projection view, we propose using the attention mechanism to enhance the point cloud features further. Specifically, the point cloud features and image features interact via cross-attention, which further fuses the features of both modalities. It is worth noting that we do not explicitly define the mapping between the point cloud and the image but encourage the network itself to model the positional relationship between them. This approach differs from the projection process, where we align the point cloud and image based on the camera parameters and then explicitly fuse them.This implicit fusion enables all points in the space to be augmented with image features obtained from DINOv2 [32] . Assuming the point cloud tokens are 𝑻 𝒑 , which are treated as the query, and the image tokens are 𝑻 𝒊 , which as key and value. The point cloud features interact with all the image tokens globally to obtain the output, which can be expressed as:𝑻 𝒂𝒕 𝒕 = 𝑪𝒓 𝒐𝒔𝒔𝑨𝒕𝒕 (𝑻 𝒑 , 𝑻 𝒊 , 𝑻 𝒊 ) (6)To enhance the features of the point cloud, we introduce projection and attention mechanisms for cross modality enhancement, respectively. Finally, we aggregate the enhanced features using average aggregation:EQUATIONLoss. The Gaussian generator is trained using rendering loss. In each iteration, we randomly select 𝑀 views, with one serving as the conditional input, while the remaining 𝑀 -1 views are employed for supervision. Following the methodology of prior research [40] , we compute the photometric loss and alpha loss between the rendered image and the ground truth image. Our objective is to minimize the following objective function.EQUATION)EQUATIONEQUATION)where 𝜆 𝐿𝑃𝐼 𝑃𝑆 and 𝜆 𝛼 are corresponding weight coefficient, and 𝐿 𝐿𝑃𝐼 𝑃𝑆 are perceptual image patch similarity [53] .During training, we use point cloud data from ground truth (GT), but when inference, the model inputs are generated from the 3D diffusion model. To mitigate the gap in data distributions, we perturb the point cloud data when training the model using data augmentation. Specifically, we jitter the coordinates and RGB values of the input point cloud by adding noise to them, thereby enhancing the robustness of the model to the input. 
experiments: 
 In this section, we initially delve into the specifics of the experiments, followed by a discussion on the dataset employed in our training and testing. Then we present our experimental results, offering both qualitative and quantitative analyses between oursLGM OursFigure 5 : Qualitative results among baselines of single view image-to-3D reconstruction on Objaverse [10] and Google Scanned Objects [11] dataset. Our method outperforms previous Gaussian based baselines in both geometry and texture representations, and in particular our method generates more consistent multi-view rendering results, thanks to the informative geometry prior of the point cloud input.and other methods. We conduct an ablation study to validate the effectiveness of our proposed modules lastly.The 3D diffusion model we deployed is Point-E [31] , which presents commendable performance. Alternative methods for obtaining point clouds, such as DUSt3R [46] , sparse reconstruction [25] , or Table 1 : Qualitative rendering results on single-view imageto-3D. We compare the reconstruction quality for 32 novel views with 100 random objects selected from GSO [11] . Our method outperformed relevant feed-forward based generation baselines with comparable inference speed. The best results are bolded.One-2-3-45 [20] 17 even 3D scanners, can also be utilized. The implementation of point cloud upsampler is modified form [48] , which takes a 4K input and generates a 16K point cloud output. The image encoder utilized is the pretrained DINOv2 [32] , and the architecture of our Point to Gaussian generator is based on the [23] , with both the encoder and decoder consisting of four layers. Besides, to conserve computational resources, we incorporate the APP module solely into the encoder during the training process. The loss weights for mask and LPIPS losses are both set to 1. Meanwhile, the loss weights for point clouds are gradually attenuated from 1 to 0.05 using cosine annealing. During the training process, the number of views per iteration was set to 4 (i.e., 𝑀 = 4). The experiment was conducted with 16 NVIDIA Tesla A100 GPUs for training, spanning approximately 3 days. The resolution of novel view rendering was 256, with a batch size of 128 (batch size 8 for each GPU).The model is trained on the Objaverse-LVIS [10] dataset. We render RGB images from 32 perspectives by rotating around the object's surface. Regarding the point cloud, we utilized the point cloud data provided by [26] . Through farthest point sampling, we obtained a 4k point cloud as the input. We evaluate our methods on the Objaverse and Google Scanned Objects [11] and randomly selected 100 test samples from each of the two datasets, respectively. For each case, akin to the training set, we rendered 32 different perspectives. 4.3.1 Qualitative Comparison. We predominantly compare our approach with recent Gaussian-based 3D generation methods on the Objaverse [10] and Google Scanned Objects [11] . The qualitative comparison of the results is depicted in Fig. 5 . As observed, our method demonstrates richer high-frequency details compared to other models for both datasets. This outcome can be attributed to the cross modality enhancement, which incorporates projection and attention to optimally utilize the input image.Reconstructing the backside view is a formidable challenge, as demonstrated by the results of LGM in the Second line. The results for the backside of the bottle appear inconsistent with the input view. However, our results exhibit remarkable consistency, and the Gaussians generated by our method more accurately recover the content from the input view, yielding more coherent and reasonable geometry and texture details. This improvement can be ascribed to the point cloud input, which simplifies the learning of Gaussians.Comparison. We also carried out a quantitative comparison with other methodologies and calculate PSNR, SSIM, and LPIPS [53] metrics to assess the quality of images. The quantitative results are presented in Table 1 . As evident from the table, our method surpasses other feed-forward methods in terms of generation quality across all metrics, while maintaining a comparable inference speed. The time intensity of our process is primarily attributed to the initial stage of point cloud acquisition, as can be deduced by comparing our time usage with that of Point-E. However, in the second stage, the Point to Gaussian Generator can be inferred in real-time due to the fast rendering speed of splatting.4.4.1 APP Block. We first trained a baseline model using the point feature extractor only. Based on it, we added projection and attention respectively. The quantitative experimental results are presented in Table 2 , which indicates the projection and attention we designed can significantly facilitate the learning of Gaussian attributes. Moreover, when both mechanism are employed, which further enhances the performance. A qualitative comparison of the results, is shown in Fig. 7 , which also demonstrates that the quality of the reconstructed images is significantly improved.With the assistance of existing image diffusion models, for example, MVDream [38] , our method can also support multi-image input. To be specific, we first convert the singleimage input into four consistent images and then feed them into To demonstrate the efficiency of Gaussian center offsets learning, we conducted a comparative analysis between involving Gaussian center offsets learning and direct Gaussian centers learning. We visualize the rendered images and 3D Gaussians at different iterations, as displayed in Fig. 6 . The top row presents the outcome of direct Gaussian centers learning. At the beginning of training, the 3D Gaussians are predominantly characterized by noise, and the model requires approximately 5k iterations before revealing the basic shape of the bag. In contrast, with offset learning, as depicted in the bottom row, the 3D Gaussians exhibit good geometry during the initial training and demonstrate faster convergence speed. After just 300 iterations, the model can reconstruct the basic geometry of the input object. With offset learning, the Gaussians can be initialized to a suitable position by the point cloud at the beginning of training. This approach allows the model to concentrate more on modeling other Gaussian attributes and is considerably simpler than mapping from images to Gaussians. 
conclusion: 
 In this paper, we present a large point-to-Gaussian model for imageto-3D generation. Point-to-Gaussian model inputs point cloud to generate Gaussian parameters, in which the input point cloud is generated from a large 3D diffusion model (e.g. Point-E) from the 2D image. With the geometry prior from point cloud, Point-to-Gaussian model is able to significantly improve image-to-3D generation. In addition, based on a multi-scale network, we further devise a APP block, which fuse the image features into point cloud representations with an attention mechanism and pose-aware projection mechanism. Extensive experiments demonstrate the effectiveness of the proposed approach on GSO and Objaverse datasets, and the proposed method achieves state-of-the-art performance.  
 
 Appendix:  


Decision: Pros: The paper is easy to follow, and the idea is relatively new.
Cons: There are some errors regarding the empirical results reported in Tables. As the topic is very hot, there have been many recent efforts, including e.g. TriplaneGaussian and many others, the authors are encouraged to follow closely and report comparisons with them, or at least the major ones. There is some concern of lack of details in method presentation, e.g. when describing the APP Up block operations, for readers to replicate the work. Moreover, there is some doubt of the empirical performance of the work.

Reviews: review: paper_summary: paper_summary: The authors propose a new approach: a large Point-to-Gaussian model. This model takes an initial point cloud (created from a 2D image) and uses it to generate the 3D model. The point cloud provides valuable 3D information that aids the generation process. Additionally, the paper introduces an APP block that merges image and point cloud features for improved results. Experiments show that this method achieves superior performance on standard datasets.
strength_weakness: strength_weakness: 1. Their method is the first attempt to utilize the generalizable Point-to-Gaussian Generator for feed-forward image-to-3D generation.
2. They introduce APP block into Point-to-Gaussian generator for Cross Modality Enhancement, which further integrate the geometric structural features with the 2D texture features for more effective learning.
limitations: limitations: 1. The experimental results are incorrect. Table 1 shows the results of **single-view** image-to-3D, but the results in Table 1 are the same as the results for **Multi View** in Table 3 Ablation study on the number of input. Therefore, Table 1 does not compare the results of single view.
2. The experimental results are insufficient. For the image-to-3D setting, the experimental section only compares the results of novel view synthesis, and does not compare reconstruction results, such as CD, IoU metrics that were compared in the baseline.
3. I have doubts about the final results of the paper. To my knowledge, the point clouds generated by Point-E from a single image are far inferior to the ground truth (GT) point clouds. The authors claim to have performed data augmentation on the GT point clouds during training: "we jitter the coordinates and RGB values of the input point cloud by adding noise to them, thereby enhancing the robustness of the model to the input." However, even such simple perturbations still result in significantly better results than the point clouds generated by Point-E, especially in invisible views.
4. LRM has a much larger training dataset than the one used in this paper. LGM builds upon MVDream, which utilizes 2D priors and more 3D data. In contrast, the proposed method is trained solely on the Objaverse-Lvis dataset. Given that Point-E struggles to accurately reconstruct the backside of input images, I am skeptical about the ability of the proposed method to generate accurate reconstructions from such point clouds, especially for the backside. The APP structure employed in the paper is a common approach. Therefore, I would like the authors to provide more convincing evidence to support their claim that the proposed method outperforms LGM and LRM in reconstructing the backside.
5. Here are some small writing errors I found in the paper: Line 890: "The results are presented in Section 4.4.2" should be "The results are presented in Table 3".
paper_summary: paper_summary: This work introduces a large Point-to-Gaussian model to generate high-quality 3D Gaussians with point cloud input. Firstly, the initial point cloud is produced from large 3D diffusion model conditional on 2D image to generate the Gaussian parameters, which provides initial 3D geometry prior for Gaussian generation, thus significantly facilitating image-to- 3D Generation. Moreover, the Attention mechanism, Projection mechanism, and Point feature extractor, dubbed as APP block, is proposed for fusing the image features with point cloud features. The proposed method was evaluated on GSO and Objaverse datasets for image-to-3D Generation and compared against recent works.
strength_weakness: strength_weakness: 1.Clarity and readability: the paper is well written, and the language is easy to follow. 
2.The proposed method is the first attempt to utilize the generalizable Point-to-Gaussian Generator for feed-forward image-to-3D generation. The Attention mechanism, Projection mechanism, and Point feature extractor as APP block was designed into Point-to-Gaussian generator for Cross Modality Enhancement. The proposed method is somewhat innovative in overall scheme design, but the core module has weak innovative.
3.Comparison against recent works.
limitations: limitations: 1. As noted in Sec3.2, the point cloud was generated by a pretrained diffusion model [31] for initialization. So, what if the initial point cloud generated by the diffusion model produces suboptimal results?
2. The Point Cloud Upsampler of Point-to-Gaussian Generator utilizes the methodology outlined in [48] to implement a dense sampling operation on the point cloud. However, the proposed method in [48] is aimed at point cloud completion. How to use it to generate dense point clouds? The technical details are not clearly explained in the paper. At the same time, the function of this module was not verified in the ablation experiment.
paper_summary: paper_summary: This paper proposes a novel generative model that transforms large point clouds into Gaussian representations. According to the complete pipeline diagram, the model takes a point cloud generated by a pre-trained 3D diffusion model and a two-dimensional image as input.   It employs the APP block (Attention mechanism, Projection mechanism, and Point feature extractor), as proposed by the author, to integrate image and point cloud features.   Ultimately, a 3D Gaussian image is produced using these integrated features, which is then converted into a multi-view image through traditional Gaussian scattering.
strength_weakness: strength_weakness: 1.The use of cross-modal information to enhance traditional Gaussian point cloud generation is highly innovative. 
2.The architecture diagram of the Point-to-Gaussian generator is detailed and clear. 
3.The experimental section demonstrates superior performance compared to LRM, Triplane Gaussian, and LGM. Results on the Objaverse and Google Scanned Objects datasets reveal richer high-frequency details.
limitations: limitations: 1.The paper lacks a detailed implementation description of the APP Up block operations following the Average Aggregation in Figure 3.
2.As the author mentioned in Section 4.3.1, reconstructing the back view is a difficult challenge, which the author can further think about how to improve the optimization.
3.This paper lacks a detailed description of the full English names of PSNR, SSIM, and LPIPS metrics in Section 4.3.2.
paper_summary: paper_summary: - Task: single img to 3DGS
- Contribution:
    - not 'img -> 3DGS', but 'img-> point cloud -> 3DGS'
    - propose APP block for fusing image and pc feature
    - SOTA performance, even trained with a much smaller dataset (objaverse-LVIS)

- Method：
    - Get initial pc from input img by PointE
    - Upsample the pc by SnowFlakeNet
    - Send the pc and img to a designed network called Point-to-Gaussian Generator, to get 3DGS
        - Point-to-Gaussian Generator mainly consists of some APP blocks. The final output heads output 3D Gaussian parameters。
        - An APP block contains point feature extractor, projection（project img features to 3D points？）and cross attention. The final result is the average of the three.

Summary:
This paper is well-written. The two main contributions are clearly stated, reasonable, and sound novel to me. My main concern is that the qualitative results in Figure 6 do not seem convincing enough. Some input images are from weird perspectives. Lack qualitative comparisons with the baseline method with the highest quantitative results (one-2-3-45). and sometimes the improvement compared to baseline methods does not seem significant. If the authors can provide more convincing qualitative results, and explain in detail on the evaluation strategy  (see the limitations part), I may be convinced. Also, there are some suggestions to make the 'Method' section clearer and easier to understand. Overall, I tend to regard this paper as 'borderline'. If the above concerns can be addressed, especially more convincing visual comparisons with baseline methods especially one-2-3-45, I may tend to accept it.
strength_weakness: strength_weakness: The novelty sounds sufficient to me, with the two main contributions: 

1) introducing 3D point cloud as a middle step instead of directly generating 3DGS from imgs, and 

2)  proposing a APP block for fusing image and point features. 

The motivation is clearly stated and convincing. 

Specially, the introduction and related work are well-written and easy to follow.


The overall language is good.
limitations: limitations: The main limitations lie in the experiment and method sections. Besides, some other small suggestions are provided.
## Experiment
**Experimental results not convincing enough.**
Among all 5 baselines (one-2-3-45, Point-E, LRM, TriplaneGaussian, and LGM) listed in Table 1, one-2-3-45 achieves the overall highest scores. However, some other listed baselines are actually more recent and they claim to be better than one-2-3-45. For example， the paper of TriplaneGaussian shows that it outperforms one-2-3-45 by a significant margin, both quantitative and qualitative. Notably, for evaluation, TriplaneGaussian also randomly selected 100 shapes from the Google Scanned Objects dataset, just like this paper. That is to say, the conclusion of TriplaneGaussian and this paper kind of disagree with each other. However, the visual comparisons in TriplaneGaussian seem more convincing than this paper, showing a more significant improvement of TriplaneGaussian compared to baseline methods. Although Figure 5 of this paper shows that the proposed method outperforms baseline methods to some degree, the superiority is not as significant. Another example would be LGM, which claims to outperform DreamGaussian, and DreamGaussian claims to outperform one-2-3-45.

It would be much more convincing if more qualitative comparisons could be provided, with a significant margin compared to baseline methods. However, the supplementary file only provides visual results of the proposed method, lacking comparison with baseline methods.


**Evaluation Strategy.** During evaluation, how did the authors choose the viewpoint of input single image? In line 787, it's said that 'For each case, akin to the training set, we rendered 32 different perspectives.' Does it mean that for each shape, 32 viewpoints are used as input to generate 3DGS respectively followed by evaluation, or only one viewpoint is used as input to generate 3DGS, but 32 viewpoints are used to render the generated 3D result for evaluation?

In Figure 5, some input images are from a side viewpoint, like rows 2,3, and 4. Under such side-view input circumstances, the proposed method shows a better performance, especially regarding the opposite side of the input. Is it that, the proposed method has an advantage for such cases with a not very ideal input viewpoint? When the input image is from the front or maybe 2 o'clock direction, does it still show an advantage?

Considering the application of reconstructing a 3D model from a single image, one is likely to take a photo of the target from the front or maybe 2 o'clock direction, so that the target can be clearly seen in the image. Using a side-view image as input is indeed more challenging, but does not seem as necessary, since it's often not any harder to take a photo from the front.


**About the multi-view experiment.** The proposed method is an img-to-3DGS method. MVDream is an img-to-NerF method. They focus on a quite similar task. But section 4.4.2 proposes to first use MVDream to generate 4 novel view images, then use all images as input of the proposed method. Why don't we directly use MVDream, why bother using the proposed method? Compared to directly using MVDream (further optimizing a NerF to fit the generated multi-view images), does the proposed method show a better performance?

**Some other suggestions about experiments**
- In Figure 7 (ablation study of the proposed APP block)， it's suggested to add GT, since the presented object is a little abstract and the readers don't really know what it's supposed to look like. If GT was added, it would be clearer for the readers to see how the APP module outperforms other settings.


- Add GT in Figure 5 (main qualitative comparison results)
- In Figure 5 (main qualitative comparison results), Why are there only 3DGS-based baselines? one-2-3-45 seems stronger according to Table 1, so why not compare with it qualitatively?


## Method
The proposed method can be presented more clearly. See the following:
- it's suggested to add mathematical notations of the input and output of APP Block, especially the dimensions of point tokens and image tokens.
- **How are the multi-scale pcs used.** The multi-scale pc obtained from FPS, where are they sent to the network? In Figure 3, there are no lines to connect them to the network. Are they only sent to the point extractor for multi-scale feature extraction, or is the whole network repeated several times with different scales of pc as input?
- **Details about Point Feature Extractor.** It's only said in line 496 that the point feature extractor combines point-based methods [2] and voxel-based methods [56], but how? It's ok to refer the readers to the supplementary file, instead of not saying anything in the main paper.
- **Implementation of projection.** 
    - It's said in line 513 that 'drawing on insights from [28], the projection ....'. This sounds amphibolous. Is it exactly the same as [28]? If so, please just say 'we directly adopt the method in [28]'. Otherwise, please write clearly what modifications are there compared to [28].  
    - Besides, for readers not familiar with [28], the so-called 'self-occluded' can be a little confusing. We can see in Figure 4 that, if look from one side, the opposite side is with fewer points due to occlusion. But shouldn't it be no points at all on the opposite side since it's completely occluded and no points should be seen?
- **How 'projection' fuses the img and point features.** I see that 'projection' maps img pixels with 3D points. But what then? How are the img and point features fused? Does it cat the img features to the corresponding points? or maybe sum?
- **How are the point tokens obtained?** Figure 3 shows that image tokens are obtained from image features of ViT, but how about the point tokens? 
- **Details about the 32 perspectives.** In line 781, it's said that ''We render RGB images from 32 perspectives by rotating around the object's surface''. How to rotate in detail? It's ok to refer the readers to the supplementary file or say 'we follow XXX'


Some other small questions or suggestions:

- Data Augmentation： During training, why use perturbed GT as input, instead of using results of diffusion models like in inference? If it's because the diffusion model takes too long time, it would be better to add something like 'considering diffusion model's XXX'.
- In line 745, why is it OK to only incorporate the APP module into the encoder during training? 
- In Section 4.4.3, it's said that directly learning Gaussian centers instead of offsets would take about 5k iterations to show an overall shape, while learning offsets only takes 300. It's suggested to also tell the readers how many iterations in total would there be.


## Some details or suggestions regarding language or presentation
- In line 536: ''This implicit fusion enables all points in the space to be augmented with image features obtained from DINOv2 [32]''. It's a bit weird to suddenly talk about the image encoder in the Attention part. The image features are actually already used in the previous step 'projection', right? 
- When citing SnowFlakeNet, like in Line 740 or 441, it's suggested to cite it with its name instead of only a number. i.e. say something like ''SnowFlakeNet [48]'' instead of just '[48]'. As a module used in the proposed method, it would be more convenient for the readers who are familiar with this method, and they no longer need to click to see what is [48].
- It may be more clear for the readers, if the number of samples of each used dataset is presented when introducing the datasets.(objaverse-LVIS,GSO, 和objaverse)

- Line 501: ''remain carries insufficient appearance feature'', 'remain carries' seems incorrect regarding grammer
- Line 507: ''following sections'' -> ''following subsections''?
- Line 890: ''The results are presented in Section 4.4.2'', （it is Section 4.4.2 already. Do you mean Table 3 instead?）