HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction
Authors: Shengji Tang
Weicai Ye
Peng Ye
Weihao Lin
Yang Zhou
Tao Chen
Wanli Ouyang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D reconstruction
Gaussian Splatting
Generalizable Multi-View Reconstruction
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A hierachical 3D gaussian splatting model for generalizable multi-view resconstruction

Abstract: Reconstructing 3D scenes from multiple viewpoints is a fundamental task in stereo vision. Recently, advances in generalizable 3D Gaussian Splatting have enabled high-quality novel view synthesis for unseen scenes from sparse input views by feed-forward predicting per-pixel Gaussian parameters without extra optimization. However, existing methods typically generate single-scale 3D Gaussians, which lack representation of both large-scale structure and texture details, resulting in mislocation and artefacts. In this paper, we propose a novel framework, HiSplat, which introduces a hierarchical manner in generalizable 3D Gaussian Splatting to construct hierarchical 3D Gaussians via a coarse-to-fine strategy. Specifically, HiSplat generates large coarse-grained Gaussians to capture large-scale structures, followed by fine-grained Gaussians to enhance delicate texture details. To promote inter-scale interactions, we propose an Error Aware Module for Gaussian compensation and a Modulating Fusion Module for Gaussian repair. Our method achieves joint optimization of hierarchical representations, allowing for novel view synthesis using only two-view reference images. Comprehensive experiments on various datasets demonstrate that HiSplat significantly enhances reconstruction quality and cross-dataset generalization compared to prior single-scale methods. The corresponding ablation study and analysis of different-scale 3D Gaussians reveal the mechanism behind the effectiveness. Code is at https://github.com/Open3DVLab/HiSplat.

Published as a conference paper at ICLR 2025
HISPLAT: HIERARCHICAL 3D GAUSSIAN SPLATTING FOR GENERALIZABLE SPARSE-VIEW RECONSTRUC-
TION
Shengji Tang Weicai 22
223 Peng Ye Weihao Lin' Yang Zhou 2.2 Tao Chen' Wanli Ouyang State Key Lab of CAD&CG, Zhejiang University
Fudan University
Shanghai Al Laboratory
Reference Images
PixelSplat
MV
Ours
Ground Truth
Previous Methods
Ours
Rettur
ESTRET
A
Netm
Thage Ing
3D Grustere
Togs
ID Gezslasi
Figure 1: Comparison between HiSplat and previous methods. HiSplat constructs hicrarchical 3D Gaussians which can better represent large-scale structures (more accurate location and less crack), and texture details (fewer artefacts and less blurriness).
ABSTRACT
Reconstructing 3D scenes from multiple viewpoints is a fundamental task in stereo vision. Recently, advances in generalizable 3D Gaussian Splatting have enabled high-quality novel view synthesis for unseen scenes from sparse input views by feed-forward predicting per-pixel Gaussian parameters without extra optimiza- tion. However, existing methods typically generate single-scale 3D Gaussians, which lack representation of both large-scale structure and texture details, re- sulting in mislocation and artefacts. In this paper, propose novel frame- work, HiSplat, which introduces a hierarchical manner in generalizable 3D) Gaus- sian Splatting construct hierarchical 3D Gaussians via coarse-to-fine strategy. Specifically, HiSplat generates large coarse-grained Gaussians to capture large- scale structures, followed by fine-grained Gaussians enhance delicate texture details. To promote inter-scale interactions, we propose an Error Aware Mod- ule for Gaussian compensation and Modulating Fusion Module for Gaussian repair. Our method achieves joint optimization of hierarchical representations, al- lowing for novel view synthesis using only two-view reference images. Compre- hensive experiments hances reconstruction quality and cross-dataset generalization compared to prior single-scale methods. The corresponding ablation study and analysis of different- scale 3D Gaussians reveal the mechanism behind the effectiveness. Code is http:///jitubbcom///eenDDLLoaa
various datasets demonstrate that HiSplat significantly en-
"Corresponiigg author

---

Published as a conference paper at ICLR 2025
INTRODUCTION
As a crucial task in stereo vision, reconstructing 3D scenes from multiple viewpoints has been extensively explored. Recently, 3D Gaussian Splatting (Kerbl et al.) 2023 Chen et al. (024a) has emerged, which utilizes 3D Gaussians as explicit scene representation and adopts gradient descent to optimize the corresponding primitives. Different from previous methods based on ray-marching based volume rendering, e.g., Neural Radiance Fields (NeRF) (Mildenhall et al. 2020), 3D Gaussian splatting (3D-GS) implements an efficient splatting rendering pipeline, enabling faster and higher- quality scene reconstruction without extra depth information. Nevertheless, the original 3D-GS requires a substantial number of multi-view images for per-scene optimization, which hinders its implementation under limited resources and sparse obversed images. To boost its generalization and transferability, data-driven generalizable 3D-GS (Szymanowicz et al., 2024] Charatan et al. 2024 Chen et al. 2024b; Zhang et al. 2024 Wewer et al. 2024) have been proposed. Generalizable 31>- GS leverages networks to feed-forward predict per- pix-p Gaussian splatting parameters for unseen scenes, which can be described as mapping each 2D pixel to a fixed number of 3D Gaussians, namely splatter images. By generating splatter images, generalizable 3D-GS allows for high high-quality novel view synthesis using only sparse views, especially the most challenging two views, of the new scene without additional optimization during inference.
However, current generalizable 3D-GS methods generate exed-resolution splatter images using ex- tracted single-scale features. The uniform 3D Gaussians result in a lack of hierarchical represen- tation, making challenging to simultaneously capture large-scale structures and delicate texture details. Consequently, it causes issues such as blurriness, cracks, mislocation, and artefacts as il- lustrated in Fig. [ In 2D visual perception tasks such as segmentation and detection, hierarchical and multi-scale representations (He et al. 2014 Liu et al. 2015 Lin et al. 2016 Kong et al. 2018 Ghiasi et al. 2019) play an essential role in effectively capturing high-level semantic information for large-scale objects and structures while preserving lew-level details. Inspired by the success of 2D multi-scale features, a natural question arises: as an explicit 3D representation akin to 2D features, can 3D Gaussians benefit from a hierarchical structure to unleash the potential of representa- tional capabilities? Nonetheless, applying the hierarchical manner to generalizable 3D-GS is not straightforward. A simple preliminary experiment that extracts multi-scale features to generate hier- archical 3D Gaussians and mixes them for rendering has been conducted. As shown in the first line of Table 3] compared with the previous methods, constructing a vanilla hierarchical 3D Gaussians cannot obtain improvement. The core issue lies in that the multi-scale 3D Gaussians are generated independently and suffer from the inability to capture inter-scale information.
Different from predicting multi-scale Gaussians independently, we propose a novel hierarchical gen- eralizable 3D-GS framework, namely HiSplat, for producing 3DD Gaussians in a coarse-to-fing man- ner. Specifically, HiSplat generates large coarse-grained Gaussians to establish the large-scale struc- ture as a skeleton, followed by fine-grained Gaussians around the coarse Gaussians to gradually enhance texture details decoration. To prompt the interaction of different scales, we propose an attention- based Error Aware Module that allows finr-ngrined Gaussians to focus on compensating for errors generated by the coarse- grained Gaussians, referred to as Gaussian compensation. Addi- tionally, to facilitate further correction of erroneous Gaussians, we propose a Modulating Fusion Module reweight the opacities of larger-scale Gaussians based the rendering quality of in- put reference images and the current joint features, termed Gaussian repair. By integrating Gaussian Compensation and Repair, the joint optimization of hierarchical 3D Gaussians can be achieved. Dur- ing training, supervision is applied to the rendering images in every stage to introduce more gradient flow and provide richer depth information, enabling faster convergence and improved localization. During inference, only the final fusion Gaussians are utilized for rendering.
Extensive experiments demonstrate that, compared to previous methods with single-scal 3D Gaus- sian representations, HiSplat significantly enhances the quality of novel view synthesis, e.g., sur- passing the leading open-source method +0.82 PSNR on RealEstate 10K, while exhibiting stronger generalization capabilities diverse unseen scenes, c.g., improving +3.19 PSNR for zero-shot test- ing on Replica. In summary, our contributions are as follows:
We first study and introduce hierarchical 3D Gaussians representation in generalizable 3D- GS. It can simultancously reconstruct higher-quality large-scale structures and more deli- cate texture details to significantly alleviate mislocation and artefacts.
2

---

Published as a conference paper at ICLR 2025
We construct novel generalizable framework named HiSplat. generating hierarchical 3D Gaussians to reconstruct the scene with only two-view reference images. To exploit the inter-scale information and achieve joint optimization, propose Error Aware Module for Gaussian compensation and Modulating Fusion Module for Gaussian repair.
Comprehensive experiments various mainstream datasets demonstrate that, compared with previous methods, HiSplat obtains superior reconstruction quality and cross-dataset generalization. Besides, ablation study and analysis on different scale Gaussians explain the effectiveness of HiSplat.
RELATED WORK
2.1 NOVEL VIEW SYNTHESIS
Novel view synthesis aims to generate photorealistic images from unseen viewpoints given set of input images. Neural Radiance Fields (NeRF) (Mildenhall et al. 2020 Yu et al. [2021] Pumarola al., 2020; Barron al., 2021 (022) marked a significant breakthrough by modeling scenes as continuous volumetric radiance fields parameterized with neural networks. Despite achieving high- quality results, NeRF suffers from slow training speeds and high memory usage due to extensive MLP evaluations and the storage of numerous point samples. In contrast, 3D Gaussian Splatting (3D-GS) (Kerbl et al., 2023 Yu et al. 2024 Yang et al. 2023) offers an explicit scene representation using anisotropic 3D Gaussians, defined by their positions, covariances, colors, and opacities. A differentiable renderer projects these Gaussians onto the image plane, allowing efficient rendering and gradient computation. However, traditional NeRF and 3DGS methods require dense multi- view images for single-scene optimization and lack generalization, performing poorly in sparse-view reconstruction tasks. Our approach addresses these limitations by achieving high generalization with only two-view reference images.
2.2 GENERALIZABLE 3D GAUSSIAN SPLATTING
Generalizable 3D Gaussian Splatting has become a key approach for efficient 3D scene represen- tation and novel view synthesis. Splatter Image (Szymanowicz et al. (022)) predicts 3D Gaussian parameters from a single image using an image-to-iman
neural network, enabling ultra-fast single- view 3D reconstruction. PixelSplat (Charatan et al.) 2024 extends this to sparse multi-view settings, using image pairs and an epipolar transformer to learn cross view correspondences and predict depth distributions. MVSplat (Chen et al.] (224bb) constructs cost volumes via plane sweeping to capture cross-view similarities, improving geometry reconstruction by predicting depth maps and unpro- jecting them to obtain Gaussian centers. TranSplat (Zhang al. (024) employs transformer- based model for sparse-view reconstruction, using depth-aware deformable matching and monocu- lar depth priors for refinement. Despite these advancements, existing methods often fail to capture fine-grained details due to single-scale features, leading to artefacts and reduced quality, especially in complex regions. To address these limitations, we propose a hierarchical 3D Gaussian splatting method for coarse-to-fine generalizable multi-view reconstruction.
2.3 HIERARCHICAL NEURAL REPRESENTATION
Hierarchical and multi-scale representations are fundamental in computer vision and graph- ics (Clark 1976 Burt & Adelson 1983 Adelson et al., 1984). In 2D visual perception, SPP-net (He
al. 2014) introduced multi-scale pooling layers for robust recognition across scales. SSD (Liu
ct al.
2015 utilized multi-resolution feature maps for efficient object detection, while FPN Lin
et al. 2016) constructed feature pyramids with top-down pathways and lateral connections, enabling
high-level semantic features at all scales. Subsequent works (Kong et al. [0188] Ghiasi et al.) 2019)
continued to refine multi-scale feature representations. In 3D novel view synthesis, hierarchical ap- proaches have enhanced efficiency and quality. NeRF (Mildenhall et al. (020) used hierarchical sampling with coarse and fine networks. Subsequent works like KiloNeRF (Reiser et al. (021). 1). and PyNeRF (Turki et al. 2023) employed strategies such spatial partitioning and multi-scale NeRF models to improve rendering speed and quality across varying scene complexities. Our work ex- tends hierarchical principles to generalizable 3D-GS, drawing inspiration primarily from 2D vision
3

---

Published as a conference paper at ICLR 2025
techniques. We are the first to apply hierarchical neural representation to explicit 3D Gaussian rep- resentations, unleashing the potential of 3D Gaussian representations in capturing rich scene details across scales.
METHOD
3.1 FRAMEWORK OVERVIEW
Festure Extract
Saged
Def D
Media
Foter Noahie
Bazkiurn
F
Reader
Reade
UeNet
Stugs?
Eri Anak Modian
Moddasaa Modak
U
Banke
Tanderl
Stagel
Duckriane
u=U
Render
Machulting Fuskon Middle
14
kelti-1
GS.ke[Li-1]
GS
D2
Modulate Plse
ke
Mulsppy
Figure 2: The overall framework of HiSplat. For simplicity, the situation with two input images is illustrated. HiSplat utilizes a shared U-Net backbone extract different-scale features. With these features, three processing stages predict pixel-aligned Gaussian parameters with different scales, respectively. Error aware module and modulating fusion module perceive the errors the early stages and guide the Gaussians in the later stages for compensation and repair. Finally, the fusing hicrarchical Gaussians can reconstruct both the large-scale structure and texture details.
The overall framework of HiSplat is illustrated in Figure 2 Given a set of sparse sequential input images with their corresponding and the target camera pose information, HiSplat aims to feed- forward render the photorealistic target images from unseen views. To achieve this, HiSplat adopts a shared U-Net CNN with a multi-view transformer to extract the hierarchical cross-view features. The subsequent processing can be divided into three interrelated stages. For the lowest stage with the lowest resolution, the cross-viee view features are fed into a CNN with cost volume formulation to predict the depth and Gassian features, which can be decoded into corresponding Gaussian parameters. For the other two higher stages, they utilize the information (depth, features, and image errors) from the lower stages predict their corresponding Gaussians via the Error Aware Module. Finally, the Modulating Fusion Module is used to adjust the Gaussian parameters and aggregate them to generate fusing multi-scale Gaussians. During training, fusing multi-scale Gaussians each stage will be used to render target views parallelly with the supervision of ground truth. For inference, only the fusing Gaussians the last stage are adopted.
3.2 HIERARCHICAL FEATURE EXTRACTION AND DEPTH ESTIMATION
Hierarchical Cross-view Feature Extraction. To extract hierarchical multi-scale features from input multi-view 2D images, construct CNN and Transformer mixed feature extraction back- bone network with a U-Net (Ronneberger et al. 2015) architecture. Specifically, the network is divided into encoder and decoder, each composed of three 3 x 3 standard residual blocks (He jet al, 2016). Each residual block performs upsampling/dowwsaapphhi scales, and residual connections used to integrate features of the same scale from the encoder and
to extract features at different

---

Published as a conference paper at ICLR 2025
decoder. At the end of the encoder, features are fed to multi-view Transformer (Xu et al. 2023 2022) that employs cross-attention and self-attention to extract mutual information from -ueti-view images and injects it into the subsequent decoding stream. For features extracted from different de- coder stages, a simple convolutional head is used to obtaín the corresponding scale features. To fur- ther enhance the generalization of feature extraction (see Table. (3), follow MVSFormer++ (Cao iet al. 2024) to introduce features of frozen DINOv2 (Oquab et al., 2023) and interpolate DINOv2 features to add with each scale features. Formally. Given N input images Iref E R xxxxxxx3 with image size x W, the cross-view feature for stage are denoted as
F3 (I in ;0x); + Interp(D(TTn)).0))
where F. E RR and DINOv2 respectively; 0 is the corresponding parameters of network; ((::)) denotes the output features from inh stage of U-Net backbone decoder; Interp(-) is the interpolating operation.
2 x C' is the channel number of N and D are the U-Net backbone
Depth Estimation. With the camera matrix, depth used to reproject the image pixel to 3D space, which is important for the correct location of Gaussians. For hierarchical Gaussians, the larger- scale Gaussians form the skeleton, while smaller Gaussians are near larger-scale Gaussians as a supplement. Therefore, the depth prediction principle is estimating the largest-scale Gaussian depth accurately, and using it as reference to predict the depth of smaller-scale Gaussians. For depth in the first stage, we utilize the cost volume matching in Multi-Viee Stereo (MVS) (Cao et al., 2022 2024| Yao et al. 2018) for accurate depth estimation. Since cost volume matching introduces unbearable computational load at high resolutions, we only apply
the first stage. Specifically, given the feature F' E Nx 4 x xC and the corresponding camera pose P E RRNxxx4, a plane- sweep stereo approach (Collins. 1996; Yao et al. 2018 Im al., 2019) is utilized to sample R different depth candidates V
{dl .2., dæj from [onear, dfarr. Then, the matching feature Fúj Farr, R 4 xC is sampled by warping the jth view feature F to ich view according to depth plane de, which is denoted as
FJj warp,k Warp(F Pd, Pd, dd) dx)
where Wa arp is the warping and sampling operation (aoo et al. 2018 Xu et al. 2023 Chen et al.) 2024b). The matching features from other views are utilized to generate the cost volume feature
1
cv { c P F Par,n by matching it with Fr, computed as
Fij "aarp.k .F
F
1
Σ j=Li
(3)
co.k
N-1
VC
The cost volume features are fed into lightweight CNN (Chen et al.
2024b) to obtain refined Fr
co.k
and the Gaussian feature GS The depth of dth view D E R Tx
is computed as
Di = softmarr (f FV.)V.
(4)
For Gaussians in subsequent stages, we employ an Error Aware Module to predict their relative depth offsets, enabling the placement of Gaussians in appropriate positions where the large-scale Gaussians lack details or are incorrect.
3.3 ERROR AWARE MODULE
To enable small-scale Gaussians to supplement the lacking details and correct structural errors of the large-scale Gaussians, we render the mixed Gaussians from the previous stage from input views and compute an error map with the input images. A lightweight 2D U-Net (Ronneberger et al., 2013 Zhou et al. (0188b with two predictors is used to aggregate and generate the depth offsets and Gaussian features. To ensure that the decorative Gaussians are always near the skeletal Gaussians, we have restricted the range of depth offsets. Formally, given the images T/ Nx rendered from Gaussians of stage Gaussian features FGS R N
the depth offsets degree 04 E [0, Nx and x go xCas can be computed as 2
(a;, R.G. u( rre| F;; Ou))
(5)
5

---

Published as a conference paper at ICLR 2025
where U and Bu are the lightweight U-Net and its corresponding weights. Given maximum depth coefficient 17 E [0, 1] (typical value 0.1), the depth D; in stage can be computed as
Dj AD; + Interp(D;-))); AD (2a; - 1) ()
where Interp is the interpolating operation aiming to upsameple D:-1 the same spacial size as D,. By introducing the AD; can be restricted in [-n)/ulerp( (1Tnerp(((.))
3.4 GAUSSIAN PARAMETER PREDICTION
After obtaining the Gaussian features FCS and corresponding depth D; in stage i, we follow Pixel- Splat (Charatan et al., 2024) to predict the pixel-aligned Gaussian parameters, including the Gaus- sian center, opacity, covariance, and spherical harmonics coefficients. Specifically, for the Gaussian center, we utilize the D, along the pixel-aligned ray unproject the 2D pixel the 3D space lo- cation as the Gaussian center. For other Gaussian parameters, stacked convolutional layers with the corresponding predictor adopted to generate them.
3.5 MODULATING FUSION MODULE
Simply fusing Gaussians of different scales struggles with achieving optimal joint optimization, and merely introducing smaller-scale Gaussians cannot fully correct the potential errors generated by large-scale Gaussians. Therefore, inspired by spatial attention (Jaderberg al.) 2015 Almahairi et al. 2016). we propose the Modulating Fusion Module. It focuses on modulating and reweighting the Gaussians' opacity in the areas with significant errors. Formally, given the Gaussian features from current and previous stages FGS and (FCS l E [1, - 1]}, we obtain the concatenate Gaus-
sians' features peat
{Feat A [1, - 1]}, denoted
F = cat(interp(E FGS).
(7)
where interp(-) is interploating pGS to the same corresponding spatial size as each FCS; cat(.,)) concatenating in the channel dimension. An MLP is used to squeeze the dimension of Feat to multiply the corresponding error maps spatially, following another MP2 with sigmoid out- puts the modulating coefficient & E N -i previous Gaussians' opacity O₂ [0, 1] Nx 2 23-2 denoted
E = Sigmoid (MLP2
0MLP)) interp(( fres - fref (MM)))) (8)
where Sigmoid(.) is the non-linear sigmoid operation; OMPP is the corresponding weights. The previous Gaussians' opacity {O₁, O2, .... 0 is updated following O₃ := O₃ Ex.
3.6 TRAINING OBJECTIVE
During training, we render the fusing hierarchical Gaussians in all stages to obtain images from target novel views and utilize the photometric losses (Charatan et al. 2024 Chen et al. 2024b), including Mean Squared Error (MSE) loss and Learned Perceptual Image Patch Similarity (LPIPS) losses (Zhang et al. 2018). to supervise each image. The all training objective is
3
Call
Amse Lmee (Itar par) +
(Itar faar).
(9)
where the loss coefficient Amse = 1, Appips = 0.05. Except for frozen feature extractor of DINOv2, other parameters are trainable and supervised by the training objective.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTING
Datasets. To comprehensively evaluate the reconstruction ability, we train and test models in two large-scale datasets, ReallEstate (Zhou al. 2018a) and ACID (Liu et al. 2021). The

---

Published as a conference paper at ICLR 2025
RealEstate 10K dataset comprises videos sourced from YouTube, divided into 67,477 training scenes and 7,289 testing scenes. The ACID dataset consists of nature scenes captured via aerial drones, with 11,075 scenes for training and 1,972 scenes for testing. Both datasets calibrated with Structure-from-Motiie (SfM) (Schonberger & Frahm 2016) algorithm to estimate camera intrinsic and extrinsic parameters for each frame. Following the novel view synthesis settings of previous works (Charatan et al.) 2024; Chen et al. 2024b] Zhang et al., 2024), two context images are as input, and three novel target views are rendered for each test scene. Besides, to compare the cross- dataset generalization ability, we select other two multi viel datasets, including real object-centrie dataset DTU (Jensen et al. (014) and synthetic indoor dataset Replica (Straub et al. 2019), for zero- shot test (without fine-tuning or training). Following (Chen et al. (024b) and (Zhi et al. 2021), we select sixteen scenes in DTU and eight scenes in Replica for testing. To quantitatively measure the rendering quality of novel views from different aspects, Peak Signal-to-Noise Ratio (PSNR), Struc- tural Similarity (SSIM) Wang et al. 2004) and Learned Perceptual Image Patch Similarity (LPIPS)
losses Zhang et al. 2018) are adopted as testing metrics.
Implementation Details. For a fair comparison, we follow the commonly used training set- tings (Chen et al. 2024b; Charatan et al.. 2024 Zhang al. 2024). Specifically, the input images are resized as 256 x 256, and the model is optimized by Adam (Kingma) (2044) for 300,000 iter- ations. The training experiments implemented in 8 RTX4090 with batch size 2 for two days. There are more implementation details in A.1
4.2
MAIN RESULTS
4.2.1
NOVEL VIEWS SYNTHESIS
To verify the effectiveness of the proposed HiSplat on novel view synthesis, train and test the model on lcage-scale multi-view datasets RealEstate 10K and ACID, respectively. We select four typical feed-forward NeRF-based methods, including pixelNeRF (Yu et al.) 202T), GPNR (Suhail (et af. 2022), Attneend (Du et al., 2023). MuRF (Xu et al.) 2024). and three recent Gaussian- Splatting-baser methods, including PixelSplat (Charatan al. 2024), MVSplat (Chen et. 2024b), TranSplat (Zhang et al., (2024) as comparisons. As shown in T compared with previous meth- ods, HiSplat can consistently achieve state-of-the-art (SOTA) performance on different datasets and metrics with significant improvement, which demonstrates the superiority of the proposed hierar- chical structure of Gaussian primitives. Specifically, on RealEstate10K dataset, HiSplat surpasses the latest state-of-the-an art (SOTA) open-source method MVSplat by +0.82 PSNR, and exceeds the leading method Transplat by +0.52 PSNR, achieving a new milestone by obtaining higher than 27 PSNR in the challenging tio-view reconstruction task. On ACID dataset, HiSplat outperforms other methods by +0.5 PSNR than MVSpalt and +0.4 PSNR than TranSplat. For the patch-level SSIM and feature-level LPIPS metrics, HiSplat also gains significant improvement, suggesting that HiS- plat can reconstruct details and large-scale structures with higher quality. Besides the performance, also report the inference time and peak GPU memory in A.2
Table I: Evaluation on RealEstatc10K and ACID, Compared with the previous NeRe-based and generalizable quality of unseen views.
based method, HiSplat can consistently obtain higher rendering
RealEstate
ACID
Method
PSNR
SSIM 1
LPIPS 1
PSNR
SSIM T
LPIPS I
RiX Yu et al.
2021
20.43
0.589
0.550
20.97
0.547
0.533
GPNR (Suhail al.
2022
24.11
0.793
0.255
25.28
0.764
0.332
AttnRend (Du et al.
2023
24.78
0,820
0.213
26,88
0.799
0.218
MuRF (Xu a 2024
26.10
0.858
0.143
28.09
0.841
0.155
PixelSplat Charatan et al.
2024
25.89
0.858
0.142
28. 18
0.839
0.150
MVSplat Chen al.
2024b
26.39
0.869
0.128
28.25
0.843
0.144
TranSplat /hhng al. 2024
26.69
0.875
0.125
28.35
0.845
0.143
HuSs)
27.21
0.881
0.117
28.75
0.853
0.133
4.2.2 CROSS-DATASET GENERALIZATION
To verify the generalization ability of HiSplat, train the model on RealEstate 1KK and directly test it on DTU, ACID, and Replica in a zero-shot setting. As depicted in Figure the images generated by HiSplat more aligned with human perception, featuring less edge blurriness, less artefacts
7

---

Published as a conference paper at ICLR 2025
and more accurate location. The quantitive results shown in Table 2 Compared with previ- ous single-scale Spalatiin- based methods, HiSplat performs a significant generalization improvement, e.g., obtaining +1.05 PNSR object-centric dataset DTU, +0.55 PSNR on outdoor dataset ACID and +3.19 PSNR on indoor dataset Replica over the suboptimal methods, suggest- ing that HiSplat is more effectively deployed in the practical open-world scenario with various data distribution. It is worth noting that on ACID, HiSplat's zero-shot performance even outperforms others trained specially on ACID significantly. The outstanding generalization ability can be ex- plained from two aspects: 1) the hierarchical Gaussian representation can reconstruct scenes of vastly different scales simultaneously 2) the error-aware mechanism is scale-invariant, which can benefit consistently across wide range of scenes.
Table 2: Evaluation on cross-dataset generalization. We train models on RealEstate 10K, and test them on object-centric dataset DTU, outdoor dataset ACID, and indoor dataset Replica in a zero-shot setting. Compared with previous methods, HiSplat can better handle various scenes with different distributions and scales.
Method
RealEstate 10K -
DTU
RealEstate
-CCID
RcalEstateiOK -Roplica
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPISS
PixelSppat Charatan et 2024
12.89
0.382
0.560
27.64
0.830
0.160
23.98
0.821
0.202
MVSplat CChe et al
202445
13.94
0.433
0.385
28.15
0.841
0.147
23.79
0.817
0.165
TranSplal Zhing
2024
14.93
0.531
0.326
28.17
0.842
0.146
OuSs)
16.05
0.671
0.277
28.06
0.850
0.137
27.17
0.899
0,113
4.3 ABLATION EXPERIMENT
To verify the effectiveness of each proposed technique of HiSplat, we conduct ablation exper- iments eliminate each component step by step, and compare them with previous methods on RealEsstate 10K. As illustrated in Table different degrees of performance degradation are observed of any removal, validating the necessity of each component. merits attention that the vanilla hi- erarchical 3D Gaussian representation cannot perform competitively compared with previous meth- ods (-0.21 1 PSNR compared with MVSpalt). Whereas, when combined with the proposed EAM and MFM, the potential of hierarchical manner is unleashed with prominent improvement (+0.82 PSNR). The reason is that the EAM and MFM promote the transfer of error-aware information and features across different scales, driving Gaussians in the later stages to compensate for lacking details and repair the error of earlier stages for joint optimization, as stated in Sec
Table 3: The ablation study on RealEstatelOK. Hier: Hierarchical Gaussian, EAM: Error Aware Module. MFM: Modulating Fusion Module, DINO: using DINOv2 feature. Only using vanilla hierarchical Gaussians cannot perform better than previous methods. Each proposed component
contributes to the final superior results.
Hier
EAM
MFM
DINO
PSNR
SSIMT
LPIPS
X
26.18
0.869
0.135
X
26.76
0.874
0.123
Variants of HiSplat(Ours)
x
27.02
0.879
0.120
27.21
0.881
0.117
PixelSplat (Charatan et al.. 2024
25.89
0.858
0.142
Other methods
MVSplat (Chen et al. 2024b
26.39
0.869
0.128
4.4 ANALYSIS OF HIERARCHICAL GAUSSIANS
In this section, we visually analyze the different stages during the inference process to reveal the mechanism behind the effectiveness of HiSplat.
To visually display the fusing Gaussians in different stages, we take the center of Gaussian primitives as the 3D position of point clouds and utilize the color of corresponding context images to draw the Gaussian primitives. As shown in Figure for the later stage, as the number Gaussian primitives increases, the rendering quality also gradually improves, demonstrating the effectiveness of adding Gaussian primitives. To further analyze the function of adding Gaussians of each stage, we report the statistical probability of opacity and mean scale of Gaussians. is noteworthy that because the fusing Gaussians in the later stage contain the Gaussians from the early stages, for a clear analysis, we individually report the statistical value of fusing Gaus- sians from each stage. It can be observed that the Gaussian primitives in stage 1 are sparse, solid,
Analysis of 3D Gassian Primitives.
8

---

Published as a conference paper at ICLR 2025
Reference Images
PixelSplat
Sp
Our
Ground Truth
DTU
Replica
CCID
Figure 3: Qualitative comparison of generalization ability. For the scenes out of training distribu- tion, HiSplat can generate higher-quality novel-view images. More comparison is provided in A.3
and large while the adding Gaussian primitives from later stages gradually become dense, trans- parent and small. This pattern of Gaussian attributes aligns with our hypothesis: larger and more solid Gaussian primitives form the skeleton of the scene, aiming to establish the large-scale basic strueture, while smaller and more transparent Gaussian primitives serve decoration, further refining the texture details, which can be described as "from bone to flesh". Following this hypothesis, we explain the function of the displayed hierarchical Gaussians that Gaussian primi- tives from stage and stage 2 are relatively solid, adhering closely to the surface of the scene, which gradually completes the outline and basic structure; as for the Gaussian primitives from stage 3, they are very tiny and transparent, introducing richer texture details and illumination. Besides, compared with other methods, HiSplat can render high-quality novel-view image with more accurate and consistent geometry, especially the toy bird's crest in Figure 4
Analysis of 2D Error Map. To analyze how hierarchical Gaussians reconstruct the scene step by step in detail, we show the colored error map of different stages, generated by comparing the rendering images with the ground truth in Figure 5 The brighter and redder pixels exhibit more significant errors and rectangle highlights the regions with complex and rich textures. It can be observed that as the stage index increases, the overall enror continuously decreases, especially in areas with complex textures. Besides, by directly examining the rendered images, observe

---

Published as a conference paper at ICLR 2025
Refermor finggs
Stagel PSNR: 17,53
Stuge2 PSNR:18.11
Curs Stagedi
Pladipplat
PSNR:1842
PENR:16.08
PSNR:
Gassiin Pamitives
Rendering
Iriges
Sugel
Sugje
Durs Stage?
66 stapel
@5utege3 c6 enged
G5 stage: Go raago3
craget
Distrbution
Opecity
Gbadit
Opecity
c5 stapet
stage
05 (5-ange2
05 stage?
Potsel kg
Protedd Poy
Postat bV
Os stape?
Scale
Distrhution
Maan Scale
Heen Scole
Vier Scale
Figure 4: Comparison of Gaussian primitives in different stages generate large-scale solid Gaussians as "bone" and small-scale transparent Gaussians as "flesh", confirming better rendering quality and geometry.
DTU. HisPlat can gradually
Stage1 PSNR: 28.05
Stage2 PSNR:30.18
Stage3 PSNR:31.49
Reference Images
Cround Truth
Figure 5: Comparison of rendering images from different stages on RealEstate HiSplat can perceive the error, and utilize Gaussians in the later stages to add details and correct errors gradually.
that there is less blurriness and more details. It suggests that HiSplat can perceive the error in the early stage, and utilize Gaussians in the later stage for repair and compensation.
CONCLUSION
In this paper, we propose novel generalizable 3D Gaussian Splatting framework, HiSplat, aiming to render novel-view images from sparse (only two) context images. Different from previous methods predicting single-seale Gaussians, HiSplat gradually generates multi-scale hierarchical Gaussians to reconstruct the targe-scale structure and texture details with higher quality. To facilitate the informa- tion interaction of different stages, we propose Error Aware Module and Modulating Fusion Module for Gaussian compensation and repair. Extensive experiments across multiple datasets demonstrate that HiSplat significantly enhances the quality of reconstructions and cross-dataset generalization, surpassing previous single-scale methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENIS
This work is supported by National Key Research and Development Program of China (No. 0022ZD0161001) Shanghai Natural Science Foundation (No. 232R 1402900... The computations in this research were performed using the CFFF platform of Fudan University.
REFERENCES
Edward H. Adelson, Peter J. Burt, Charles H. Anderson, Joan M. Ogden, and James R. Bergen. Pyramid methods in image processing, 1984. URL https::/app..c.
I:::/CorpuSI::133000
Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, and Aaron Courville. Dynamic capacity networks. In International conference on machine learning, pp- 2549-2558. PMLR, 2016.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin BrrtinnBruuall,. and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Peter J. Burt and Edward H. Adelson. The laplacian pyramid as a compact image code. 1EEE Trans. Commun., 31:532-540, 1983. URL https://api. www.sstticcooool...... orr
CorpusID. II: 8018433
Chenjie Cao, Xinlin Ren, and Yanwei Fu. Mysformer: Multi-view stereo by learning robust image features and temperature-based depth. arXiv preprint arXiv:2208.0254/ 2022.
Chenjie Cao, Xinlin Ren, and Yanwei Fu. Mysformer++: Revealing the devil in transformer's details for multi-view stereo. arXiv preprint arXiv:2401. 2024.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaus- sian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19457-19467, 2024.
Danpeng Chen, Hai Li, Weicai Ye, Yifan Wang, Weijian Xie, Shangjin Zhai, Nan Wang, Haomin Liu, Hujun Bao, and Guofeng Zhang. Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction. arXiv preprint arXiv:2406.0657. 2024a.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat- Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images, arXiv preprint arXiv:2403.1466: 2024b.
James H. Clark. Hierarchical geometric models for visible surface algorithms. Seminal graphics: pioneering efforts that shaped the field, 1976. URL /https://appt
8231831
Robert T Collins. space-sweep approach to true multi-image matching. In Proceedings CVPR IEEE computer society conference on computer vision and pattern recognition, pp. 358-363. leee, 1996.
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide- baseline stereo pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pp. 4970-4988 2023.
Golnaz Ghiasi, Tsung-Yr Lin, Ruoming Pang. and Quoc V. Le. Nas-fpn: Learning scal- able feature pyramid architecture for object detection. 2019 IEEE/CVF Conference Com- 2019. URL /https://ags
puter Vision and Pattern Recognition (CVPR), pp. semant cscho ar. org. htw.sc//l.....rooooorraaaaa11111
11

---

Published as a conference paper at ICLR 2025
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattemn Analysis and Machine Intelli- gence, 37:1904-1916, 2014. BRLhttps: ott.a///ea/////.o..aaaaaaaoaoaaa ortpoorp//oooii 436933
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sunghon Im, Hae-Gon Jeon, Stephen Lin, and In So Kweon. Dpsnet: End-to-end deep plane sweep stereo. arXiv preprint 1rXiv: 00538, 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 406-413,2011
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL httph:tpt:/eeeee sam. iirriiiii...........i fr hungraphhhh/dd--e gaussi an sp atting/
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Tao Kong, Fuchun Sun, Chuanqi Tan, Huaping Liu, and Wenbing Huang. Deep feature pyramid reconfiguration for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. 2017 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 936-944, 2016. URL hhtpss:/appi.
semant i cschol lobollarr lar owt./oorpss
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 14458-14467, 2021.
W. Liu, Dragomir Anguelov, D. Erhan, Christian Szegedy, Scott Reed, Cheng- Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In European Conference on Computer Vision, 2015. URL (tRl:tttps://// httttttss///pp....
semanticscholaa........ hwwpc/ccooooar.rrorrrrooeeeeeeee wotp//opp//rops:: 21:1177
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancík, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In International Conference Computer Viston (ICCV), 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and cosputer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 241. Springer, 2015.
12

---

Published as a conference paper at ICLR 2025
Johannes L Schonberger and Jan-Michael Frahm. Structure- fruuture-from--ie from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104 4113, 2016.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Brik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05773 2019.
Mohammed Suhail, Carlos Fiteves,, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision, pp. 156-174. Springer, 2022.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Haithem Turki, Michael Zollhöfer, Christian Richardt, and Deva Ramanan. Pynerf: Pyramidal neural radiance fields. In Thirty-Seventh Conference on Neural Information Processing Systems, 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Christopher Wewer, Kevin Raj, Eddy Ilg, Bernt Schiele, and Jan Eric Lenssen. latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction. arXiv preprint arXiv:2403. 16292, 2024.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8121-8130. 2022.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Marc Pollefeys, Andreas Geiger, and Fisher Yu. Murf: Multi-baseline radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20041-20050, 2024.
Ziyi Yang. Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309. 13101, 2023.
Yao Yao, Zixin Luo, Shiwei Li. Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pp. 767-783, 2018.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4578-4587, 2021.
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias- free 3d gaussian splatting. Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Chuanrui Zhang, Yingshuang Zou, Zhuoling Li, Minmin Yi, and Haoqian Wang. Transplat: Gen- eralizable 3d gaussian splatting from sparse multi-view images with transformers. arXiv preprint arXiv:2408. 13770, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 586-595, 2018.
13

---

Published as a conference paper at ICLR 2025
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene la- belling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15838-15847, 2021.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprini /805.0880000911111 2018a.
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain. September 20, 2018, Proceedings 4, pp. 3-11. Springer, 2018b.
14


Reviews:
[{"review_id": "BpKr3qBIFe", "reviewer": "Reviewer_Lyvm", "summary": "HiSplat introduces a coarse-to-fine strategy to construct hierarchical 3D Gaussians for generalizable 3D Gaussian Splatting. Additionally, HiSplat proposes an Error-Aware Module and a Modulating Fusion Module to enhance inter-scale interactions. Comprehensive experiments demonstrate that HiSplat achieves better results than single-scale methods.", "questions": "1. Discuss more about the importance of incorporating the DINO feature.\n2. Discuss more about the complexity analysis of the method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. HiSplat first introduces the hierarchical 3D Gaussian representation in the task of generalizable 3D Gaussian Splatting.\n2. The experiment effectively demonstrated the effectiveness of its hierarchical representation through Fig 4, which shows the primitives of Gaussians obtained from different stages.\n3. This paper show the effective ablation study, comparing the impact of different modules on the performance of Generalizable Sparse-View Reconstruction.", "weaknesses": "1. Several spelling errors have been identified in the document, specifically on lines 73, 106, and 508.\n2. The hierarchical representations appear to be memory-intensive and computationally inefficient. It is recommended that the paper discusses the implications for memory, FLOPs, and inference time.\n3. The rationale for incorporating the DINO feature seems underwhelming, and the provided experiments do not convincingly demonstrate the necessity of including the DINO feature.\n4. I hope to see additional experiments conducted to explore the impact of increasing the number of perspectives on the quality of reconstruction. It appears that the majority of the experiments presented in the article are limited to just two perspectives, which may not fully capture the potential benefits of a multi-perspective approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNegQp9OZ8", "reviewer": "Reviewer_gQRm", "summary": "This paper claims that previous generalizable 3D Gaussian Splatting methods utilize uniform 3D Gaussians, making it challenging to simultaneously capture large-scale structures and intricate texture details. The authors propose a hierarchical approach to generalizable 3D Gaussian Splatting, constructing hierarchical 3D Gaussians through a coarse-to-fine strategy. Additionally, they design an Error Aware Module and a Modulating Fusion Module to manage interactions among different hierarchical Gaussians.", "questions": "Apart from the weakness part, I also have the following questions.\n1. Will the number of Gaussian primitives change significantly after applying this hierarchical structure? A comparison with baseline methods on the primitive numbers will be nice.\n2. It seems that the first example in Fig.6. shows that HiSplat can generate less artifacts in the occluded area, as highlighted by the red bounding box. Does this just happen in coincidence or is it a prevalent phenomenon?\n\nI still have the aforementioned concerns and I would be glad to raise my rating if they can be addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of using different hierarchical Gaussians to simultaneously capture large-scale structures and delicate textures in scene generation sounds reasonable.\n2. The ablative study seems complete. \n3. The motivation is clear since not all points in a single image are equally important.", "weaknesses": "1. The paper introduces many modules, which slows down the process compared to MVSplat, which is also revealed in Tab.4, after adding all the stages together. \n2. The improvement in metrics compared with baseline method does not seem very significant from the reviewer's perspective. But the generalization capability to unseen datasets like DTU seems interesting. The authors can show more cases that demonstrates generalization capability. \n3. Some of the visualized cases does not seem to possess a very significant improvement compared with MVSplat. For example, the last line of Fig.6; the left red bounding box in the first example in Fig. 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jyrvmMLKl", "reviewer": "Reviewer_R9EA", "summary": "This work presents a new 3DGS-based generalizable sparse-view scene reconstruction method, HiSplat, which effectively integrates multi-scale hierarchical MVS features with proposed EAM and MFM modules to enhance the quality of feed-forward 3DGS models, especially in fine-grained details. Experimental results demonstrate the superior reconstruction quality compared to prior SoTA methods.", "questions": "1. Since it is unlikely we can always get accurate camera poses from sparse views in real life, can this work be extended to handle the pose-free sparse-view reconstruction for the captures from real life? If it is easy to implement, it would be great to see some examples.\n2. The depth coefficient $\\eta$ is a bit tricky. I assume it is a fixed value for each stage after the training, is this correct? If so, how do you choose the proper $\\eta$ values?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-motivated from the prior works on multi-scale hierarchical visual knowledge. It is well-written.\n2. The way the authors integrate the multi-scale features is clever and interesting. I love the Error Aware part, HiSplat wisely incorporates the reconstruction error refinement, usually through gradient optimizations, into this feed-forward reconstruction model.  The error maps are used to guide the refinement on the higher levels. I believe this idea can also be beneficial for other related tasks.\n3. The analysis in 4.4 is very informative and useful. The figures effectively demonstrate how multi-scale features can help improve the reconstruction quality.", "weaknesses": "1. The current model is pretty complex. It contains a lot of different neural models (MVSformer++, DINOv2, and a bundle of UNets and MLPs). I am not very clear which modules are trainable and which are not. The authors should clarify and summarize this in 3.6. Also, it would be even better, this is not required for rebuttal, if HiSplat could be a relatively simpler unified foundation model (e.g., like LRM).  \n2. Although the writing is generally good, the citation format of this paper is a bit disturbing. Please properly use `\\cite` and `\\citep` provided in the ICLR template.\n3. Minor typos:\n    * MVSpalt → MVSplat\n    * L298: fellow → follow", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j6cATWN3sl", "reviewer": "Reviewer_PsL3", "summary": "This paper proposes HiSplat to estimate the 3D Gaussian Splatting structure of two given images with known camera poses in a feedforward process. It aims to use multi-scale information to improve the quality and for which two novel modules are proposed. Extensive experiments on various datasets can show its SOTA performance and good generalizability, and validate the effects of each proposed component.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed Error Aware Module and Modulating Fusion Module are interesting, which can effectively filter the errors from photometric loss in a simple way.\n\n- Experiments are extensive. Both qualitative and quantitative results show that HiSplat achieves SOTA performance compared to the compared baselines. The results are convincing.\n\n- The paper is easy to follow.", "weaknesses": "- Some related methods are missed in the discussion, e.g., FreeSplat [1] and Splatt3R [2]. Especially, FreeSplat also uses a cost-volume-based structure to estimate the depth and then Gaussians with multi-scale strategy. A step further, the architecture of this work is also somehow similar to CasMVS [3] that uses a cascade structure to solve the MVS depth estimation task. The main differences lie in how the multi-scale features are used. Need more deeper discussions about these related works to show the technical differences and more valuable analysis.\n\n- The setting of this paper seems to be fixed at using only two images. Would like to see some discussion about if it's possible to extend the method to more input views, just like what FreeSplat does. Even if it is a part of limitation, it would still be much valuable.\n\n\n\n[1] Wang, Yunsong, et al. \"FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes.\" arXiv preprint arXiv:2405.17958 (2024).\n\n[2] Smart, Brandon, et al. \"Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs.\" arXiv preprint arXiv:2408.13912 (2024).\n\n[3] Gu, Xiaodong, et al. \"Cascade cost volume for high-resolution multi-view stereo and stereo matching.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)