DepthSplat: Connecting Gaussian Splatting and Depth

## Abstract

Abstract Gaussian splatting and single/multi-view depth estimation are typically studied in isolation. In this paper, we present DepthSplat to connect Gaussian splatting and depth estimation and study their interactions. More specifically, we first contribute a robust multi-view depth model by leveraging pre-trained monocular depth features, leading to high-quality feed-forward 3D Gaussian splatting reconstructions. We also show that Gaussian splatting can serve as an unsupervised pre-training objective for learning powerful depth models from large-scale unlabelled datasets.
We validate the synergy between Gaussian splatting and depth estimation through extensive ablation and cross-task transfer experiments. Our DepthSplat achieves state-of-the-art performance on ScanNet, RealEstate10K and DL3DV datasets in terms of both depth estimation and novel view synthesis, demonstrating the mutual benefits of connecting both tasks. Our code, models, and video results are available at https://haofeixu.github.io/depthsplat/ .

## 1 Introduction

Novel view synthesis (Buehler et al., ; Zhou et al., ) and depth prediction (Schönberger et al., ; Eigen et al., ) are two fundamental tasks in computer vision, serving as the driving force behind numerous applications ranging from augmented reality to robotics and autonomous driving. There have been notable advancements in both areas recently.

For novel view synthesis, 3D Gaussian Splatting (3DGS) (Kerbl et al., ) has emerged as a popular technique due to its impressive real-time performance while attaining high visual fidelity.
Recently, advances in feed-forward 3DGS models (Charatan et al., ; Chen et al., ; Szymanowicz et al., ) have been introduced to alleviate the need for tedious per-scene optimization, also enabling few-view 3D reconstruction.
The state-of-the-art sparse-view method MVSplat (Chen et al., ) relies on feature matching-based multi-view depth estimation (Xu et al., ) to localize the 3D Gaussian positions, which makes it suffer from similar limitations (*e.g.*, occlusions, texture-less regions, and reflective surfaces) as other multi-view depth methods (Schönberger et al., ; Yao et al., ; Gu et al., ; Wang et al., ; Duzceker et al., ).

On the other hand, significant progress has
been made in monocular depth estimation, with recent models (Yang et al., ; Ke et al., ; Yin et al., ; Fu et al., ; Ranftl et al., ; Eftekhar et al., )
achieving robust predictions on diverse in-the-wild data. However, these depths typically lack consistency across views, constraining their performance in downstream tasks (Wang et al., ; Yin et al., ).
In addition, both state-of-the-art multi-view (Cao et al., ; Xu et al., ) and monocular (Yang et al., ; Piccinelli et al., ; Yang et al., ) depth models are trained with ground truth depth supervision, which prevents exploiting large unlabelled datasets for more robust depth predictions.

The integration of 3DGS with single/multi-view depth estimation presents a compelling solution to overcome the individual limitations of each technique while at the same time enhancing their strengths. To this end, we introduce *DepthSplat*, which exploits the complementary nature of sparse-view feed-forward 3DGS and robust single/multi-view depth estimation to improve the performance for both tasks.

Specifically, we first contribute a robust multi-view depth model by integrating pre-trained monocular depth features (Yang et al., ) to the multi-view feature matching branch, which not only maintains the consistency of multi-view depth models but also leads to more robust results in situations that are hard to match (e.g., occlusions, texture-less regions and reflective surfaces). The predicted multi-view depth maps are then unprojected to 3D as the Gaussian centers, and we use an additional lightweight network to predict other remaining Gaussian parameters. They are combined together to achieve novel view synthesis with the splatting operation (Kerbl et al., ).

Thanks to our improved multi-view depth model, the quality of novel view synthesis with Gaussian splatting is also significantly enhanced (see Fig. 1 left).
In addition, our Gaussian splatting module is fully differentiable, which requires only photometric supervision to optimize all model components.
This provides a new, unsupervised way to pre-train depth prediction models on large-scale unlabeled datasets without requiring ground truth geometry information.
The pre-trained depth model can be further fine-tuned for specific depth tasks and achieves superior results over training from scratch (see Fig. 1 right, where unsupervised pre-training leads to improved performance).

We conduct extensive experiments on the large-scale TartanAir (Wang et al., ), ScanNet (Dai et al., ) and RealEstate10K (Zhou et al., ) datasets for depth estimation and Gaussian splatting tasks, as well as the recently introduced DL3DV (Ling et al., ) dataset, which features complex real-world scenes and thus is more challenging. Under various evaluation settings, our DepthSplat achieves state-of-the-art results. The strong performance on both tasks demonstrates the mutual benefits of connecting Gaussian splatting and depth estimation.

## 2 Related Work

Multi-View Depth Estimation.
As a core component of classical multi-view stereo pipelines (Schönberger et al., ), multi-view depth estimation exploits multi-view photometric consistency across multiple images to perform feature matching and predict the depth map of the reference image. Recently, many learning-based methods (Yao et al., ; Gu et al., ; Wang et al., ; Duzceker et al., ; Wang et al., ; Ding et al., ; Cao et al., ) have been proposed to improve depth accuracy. For example, MVSNet (Yao et al., ) uses the plane-sweep algorithm (Collins, ) to build a 3D cost volume, regularizes it with a 3D CNN, and then regresses the depth map. Though these learning-based methods significantly improve the depth quality when compared to traditional methods (Galliani et al., ; Schönberger et al., ; Xu & Tao, ), they cannot handle challenging situations where the multi-view photometric consistency assumption is not guaranteed, e.g., occlusions, low-textured areas, and non-Lambertian surfaces.

Monocular Depth Estimation. Recently, we have witnessed significant progress in depth estimation from a single image (Ranftl et al., ; Bhat et al., ; Yin et al., ; Yang et al., ; Ke et al., ), and existing methods can produce surprisingly accurate results on diverse in-the-wild data.
However, monocular depth methods inherently suffer from scale ambiguities and are not able to produce multi-view consistent depth predictions, which are crucial for downstream tasks like 3D reconstruction (Yin et al., ) and video depth estimation (Wang et al., ). In this paper, we leverage the powerful features from a pre-trained monocular depth model (Yang et al., ) to augment feature-matching based multi-view depth estimation, which not only maintains high multi-view consistency but also leads to significantly improved robustness in challenging situations such as low-textured regions and reflective surfaces.

Monocular and Multi-View Depth Fusion. Several previous methods (Bae et al., ; Li et al., ; Yang et al., ; Cheng et al., ) try to fuse monocular and multi-view depths to alleviate the limitations of feature matching-based multi-view depth estimations. However, they either fuse single- and multi-view depth predictions with additional networks or rely on sophisticated architectures. In contrast, we identify the power of off-the-shell pre-trained monocular depth models and propose to augment multi-view cost volumes with monocular features, which not only leads to a simple model architecture but also strong performance.

Feed-Forward Gaussian Splatting. Several feed-forward 3D Gaussian splatting models (Charatan et al., ; Szymanowicz et al., ; Chen et al., ; Wewer et al., ; Tang et al., ; Xu et al., ; Zhang et al., ) have been proposed in literature thanks to its efficiency and ability to handle sparse views. In particular, pixelSplat (Charatan et al., ) and Splatter Image (Szymanowicz et al., ) predict 3D Gaussians from image features, while MVSplat (Chen et al., ) encodes the feature matching information with cost volumes and achieves better geometry. However, it inherently suffers from the limitation of feature matching in challenging situations like texture-less regions and reflective surfaces. In this paper, we propose to integrate monocular features from pre-trained monocular depth models for more robust depth prediction and Gaussian splatting reconstruction. We additionally study the interactions between depth and Gaussian splatting tasks. Another line of work like LGM (Tang et al., ), GRM (Xu et al., ), and GS-LRM (Zhang et al., ) relies significantly on the training data and compute, discarding explicit feature matching cues and learning everything purely from data. This makes them expensive to train (e.g., GS-LRM (Zhang et al., ) is trained with 64 A100 GPUs for 2 days), while our model can be trained in 1 day with 8 GPUs.
Moreover, our Gaussian splatting module, in return, enables pre-training depth model from large-scale unlabelled datasets without the need for ground truth depths.

Depth and Gaussian Splatting. In addition to the experimental setup (feed-forward) studied in this paper, recently, there has been another line of work (Chung et al., ; Turkulainen et al., ) which applies an additional depth loss in the Gaussian splatting optimization process. We remark that these two approaches (feed-forward *vs.* per-scene optimization) are orthogonal. In particular, our approach focuses on exploring the advanced network architectures and the power of large-scale training data, while the optimization methods mainly study the effect of loss functions for regularizing the optimization process.

## 3 DepthSplat

Figure: Figure 2: DepthSplat connects depth estimation and 3D Gaussian splatting with a shared architecture, which enables cross-task transfer. In particular, DepthSplat consists of a multi-view branch to model feature-matching information and a single-view branch to extract monocular features. The per-view cost volumes and monocular features are concatenated for depth regression with a 2D U-Net architecture. For the depth estimation task, we train the depth model with ground truth depth supervision. For the Gaussian splatting task, we first unproject all depth maps to 3D as the Gaussian centers, and in parallel, we use an additional head to predict the remaining Gaussian parameters. Novel views are rendered with the splatting operation. The full model for novel view synthesis is trained with the photometric rendering loss, which can also be used as an unsupervised pre-training stage for the depth model.
Refer to caption: x2.png

Given $N$ input images $\{{\bm{I}}^{i}\}_{i=1}^{N},({\bm{I}}^{i}\in\mathbb{R}^{H\times W\times 3}$, where $H$ and $W$ are the image sizes) with corresponding projection matrices $\{\mathbf{P}_{i}\}_{i=1}^{N}$, ($\mathbf{P}_{i}\in\mathbb{R}^{3\times 4}$, computed from the intrinsic and extrinsic matrices),
our goal is to predict dense per-pixel depth $\mathbf{D}_{i}\in\mathbb{R}^{H\times W}$ and per-pixel Gaussian parameters $\{(\bm{\mu}_{j},\alpha_{j},\bm{\Sigma}_{j},\bm{c}_{j})\}_{j=1}^{H\times W
\times N}$ for each image, where $\bm{\mu}_{j}$, $\alpha_{j}$, $\bm{\Sigma}_{j}$ and $\bm{c}_{j}$ are the 3D Gaussian’s position, opacity, covariance, and color information. As shown in Fig. 2, the core of our method is a multi-view depth model augmented with monocular depth features, where we obtain the position $\bm{\mu}_{j}$ of each Gaussian by unprojecting depth to 3D with camera parameters, and other Gaussian parameters are predicted by an additional lightweight head.

More specifically, our depth model consists of two branches: one for modeling feature matching information using cost volumes, and another for extracting monocular features from a pre-trained monocular depth network. The cost volumes and monocular features are concatenated together for subsequent depth regression with a 2D U-Net and a softmax layer. For the depth task, we train our depth model with ground truth depth supervision. Our full model for novel view synthesis is
trained with the photometric rendering loss, which can also be used as an unsupervised pre-training stage for the depth model. In the following, we introduce the individual components.

### 3.1 Multi-View Feature Matching

In this branch, we extract multi-view features with a multi-view Transformer architecture and then build multiple cost volumes that correspond to each input view.

Multi-View Feature Extraction. For $N$ input images, we first use a lightweight weight-sharing ResNet (He et al., ) architecture to get $s\times$ downsampled features for each image independently. To handle different image resolutions, we make the downsampling factor $s$ flexible by controlling the number of stride-2 $3\times 3$ convolutions. For example, the downsampling factor $s$ is 4 if two stride-2 convolutions are used and 8 if three are used.
To exchange information across different views, we use a multi-view Swin Transformer (Liu et al., ; Xu et al., ; ) which contains six stacked self- and cross-attention layers to obtain multi-view-aware features $\{{\bm{F}_{i}}\}_{i=1}^{N}$, ${\bm{F}}^{i}\in\mathbb{R}^{\frac{H}{s}\times\frac{W}{s}\times C}$, where $C$ is the feature dimension. More specifically, cross-attention is performed between each reference view and other views. When more than two images ($N>2$) are given as input, we perform cross-attention between each reference view and its top-$2$ nearest neighboring views, which are selected based on their camera position distances to the reference view. This makes the computation tractable while maintaining cross-view interactions.

Feature Matching. We encode the feature matching information across different views with the plane-sweep stereo approach (Collins, ; Xu et al., ). More specifically, for each view $i$, we first uniformly sample $D$ depth candidates $\{d_{m}\}_{m=1}^{D}$ from the near and far depth ranges and then warp the feature ${\bm{F}}^{j}$ of view $j$ to view $i$ with the camera projection matrix and each depth candidate $d_{m}$. Then we obtain $D$ warped features $\{{\bm{F}}^{j\to i}_{d_{m}}\}_{m=1}^{D}$ that correspond to feature ${\bm{F}}^{i}$. We then measure their feature correlations with the dot-product operation (Xu & Zhang, ; Chen et al., ).
The cost volume ${\bm{C}}_{i}\in\mathbb{R}^{\frac{H}{s}\times\frac{W}{s}\times D}$ for image $i$ is obtained by stacking all correlations. Accordingly, we obtain cost volumes $\{{\bm{C}}_{i}\}_{i=1}^{N}$ for all input images $\{{\bm{I}}_{i}\}_{i=1}^{N}$.
For more than two input views, similar to the strategy in cross-view attention computation, we select the top-$2$ nearest views for each reference view and compute feature correlations with only the selected views. This enables our cost volume construction to achieve a good speed-accuracy trade-off and scale efficiently to a larger number of input views. The correlation values for the two selected views are combined with averaging.

### 3.2 Monocular Depth Feature Extraction

Despite the remarkable progress in multi-view feature matching-based depth estimation (Yao et al., ; Wang et al., ; Xu et al., ) and Gaussian splatting (Chen et al., ), they inherently suffer from limitations in challenging situations like occlusions, texture-less regions, and reflective surfaces.
Thus, we propose to integrate pre-trained monocular depth features into the cost volume representation to handle scenarios that are challenging or impossible to match.

More specifically, we leverage the pre-trained monocular depth backbone from the recent Depth Anything (Yang et al., ) model thanks to its impressive performance on diverse in-the-wild data. The monocular backbone is a ViT (Dosovitskiy et al., ; Oquab et al., ) model, which has a patch size of 14 and outputs a feature map that is $1/14$ spatial resolution of the original image.
We simply bilinearly interpolate the spatial resolution of the monocular features to the same resolution as the cost volume in Sec. 3.1 and obtain the monocular feature ${\bm{F}}^{i}_{\mathrm{mono}}\in\mathbb{R}^{\frac{H}{s}\times\frac{W}{s}\times C
_{\mathrm{mono}}}$ for input image ${\bm{I}}_{i}$, where $C_{\mathrm{mono}}$ is the dimension of the monocular feature. This process is performed for all input images in parallel and we obtain monocular features $\{{\bm{F}}^{i}_{\mathrm{mono}}\in\mathbb{R}^{\frac{H}{s}\times\frac{W}{s}
\times C_{\mathrm{mono}}}\}_{i=1}^{N}$, which are subsequently used for per-view depth map estimations.

### 3.3 Feature Fusion and Depth Regression

To achieve robust and multi-view consistent depth predictions, we combine the monocular feature ${\bm{F}}^{i}_{\mathrm{mono}}\in\mathbb{R}^{\frac{H}{s}\times\frac{W}{s}\times C
_{\mathrm{mono}}}$ and cost volume ${\bm{C}}_{i}\in\mathbb{R}^{\frac{H}{s}\times\frac{W}{s}\times D}$ via simple concatenation in the channel dimension. A subsequent 2D U-Net (Rombach et al., ; Ronneberger et al., ) is used to regress depth from the concatenated monocular features and cost volumes. This process is performed for all the input images in parallel and for each image, it outputs a tensor of shape $\frac{H}{s}\times\frac{W}{s}\times D$, where $D$ is the number of depth candidates. We then normalize the $D$ dimension with the softmax operation and perform a weighted average of all depth candidates to obtain the depth output.

We also apply a hierarchical matching (Gu et al., ) architecture where an additional refinement step at $2\times$ higher feature resolution is employed to improve the performance further. More specifically, based on the coarse depth prediction, we perform a correspondence search on the $2\times$ higher feature maps within the neighbors of the $2\times$ upsampled coarse depth prediction. Since we already have a coarse depth prediction, we only need to search a smaller range at the higher resolution, and thus, we construct a smaller cost volume compared to the coarse stage. Such a $2$-scale hierarchical architecture not only leads to improved efficiency since most computation is spent on low resolution, but also leads to better results thanks to the use of higher-resolution features (Gu et al., ). Similar feature fusion and depth regression procedure is used to get higher-resolution depth predictions, which are subsequently upsampled to the full resolution with a learned upsampler (Ranftl et al., ).

### 3.4 Gaussian Parameter Prediction

For the task of 3D Gaussian splatting, we directly unproject the per-pixel depth maps to 3D with the camera parameters as the Gaussian centers $\bm{\mu}_{j}$. We append an additional lightweight network to predict other Gaussian parameters $\alpha_{j},\bm{\Sigma}_{j},\bm{c}_{j}$, which are opacity, covariance, and color, respectively. With all the predicted 3D Gaussians, we can render novel view images with the Gaussian splatting operation (Kerbl et al., ).

### 3.5 Training Loss

We study the properties of our proposed model on two tasks: depth estimation and novel view synthesis with 3D Gaussian splatting (Kerbl et al., ). The loss functions are introduced below.

Depth estimation. We train our depth model (without the Gaussian splatting head) with $\ell_{1}$ loss and gradient loss between the inverse depths of prediction and ground truth:

$$ $L_{\mathrm{depth}}=\alpha\cdot|{\bm{D}}_{\mathrm{pred}}-{\bm{D}}_{\mathrm{gt}} |+\beta\cdot(|\partial_{x}{\bm{D}}_{\mathrm{pred}}-\partial_{x}{\bm{D}}_{ \mathrm{gt}}|+|\partial_{y}{\bm{D}}_{\mathrm{pred}}-\partial_{y}{\bm{D}}_{ \mathrm{gt}}|),$ (1) $$

where $\partial_{x}$ and $\partial_{y}$ denotes the gradients on the $x$ and $y$ directions, respectively. Following UniMatch (Xu et al., ), we use $\alpha=20$ and $\beta=20$.

View synthesis. We train our full model with a combination of mean squared error (MSE) and LPIPS (Zhang et al., ) losses between rendered and ground truth image colors:

$$ $L_{\mathrm{gs}}=\sum_{m=1}^{M}\left(\mathrm{MSE}(I^{m}_{\mathrm{render}},I^{m} _{\mathrm{gt}})+\lambda\cdot\mathrm{LPIPS}(I^{m}_{\mathrm{render}},I^{m}_{ \mathrm{gt}})\right),$ (2) $$

where $M$ is the number of novel views to render in a single forward pass.
The LPIPS loss weight $\lambda$ is set to 0.05 following MVSplat (Chen et al., ).

## 4 Experiments

Implementation Details.
We implement our method in PyTorch (Paszke et al., ) and optimize our model with the AdamW (Loshchilov & Hutter, ) optimizer and cosine learning rate learning rate schedule with warm up in the first $5\%$ of the total training iterations. We adopt the xFormers (Lefaudeux et al., ) library for our monocular ViT backbone implementation. We use a lower learning rate $2\times 10^{-6}$ for the pre-trained monocular backbone, and other remaining layers use a learning rate of $2\times 10^{-4}$. The feature downsampling factor $s$ in our multi-view branch (Sec. 3.1) is chosen based on the image resolution. More specifically, for experiments on the $256\times 256$ resolution RealEstate10K (Zhou et al., ) dataset, we choose $s=4$. For higher resolution datasets (e.g., TartanAir (Wang et al., ), ScanNet (Dai et al., ), KITTI (Geiger et al., ) and DL3DV (Ling et al., )), we choose $s=8$. Our hierarchical matching models in Sec. 3.3 use 2-scale features, i.e., $1/8$ and $1/4$, or $1/4$ and $1/2$ resolutions.

Training Details. For depth experiments, we mainly follow the setup of UniMatch (Xu et al., ) for fair comparisons. More specifically, we train our depth model on 4 GH200 GPUs for 100K iterations with a total batch size of 32. For Gaussian splatting experiments on RealEstate10K (Zhou et al., ), we use the same training and testing splits of pixelSplat (Charatan et al., ) and MVSplat (Chen et al., ), and train our model on 8 GH200 GPUs for 150K iterations with a total batch size of 32, which takes about 1 day. For experiments on the DL3DV (Ling et al., ) dataset, we evaluate on the official benchmark split with 140 scenes, while other remaining 9896 scenes are used for training. We fine-tune our RealEstate10K pre-trained model on 4 A100 GPUs for 100K iterations with a total batch size of 4, where the number of input views is randomly sampled from 2 to 6. We evaluate the model’s performance on different number of input views (2, 4, 6).
Our code and training scripts are available at [https://github.com/cvg/depthsplat](https://github.com/cvg/depthsplat).

**Table 1: DepthSplat model variants. We explore different monocular backbones and different multi-view models ($1$-scale and 2-scale features for hierarchical matching as described in Sec. 3.3), where both larger monocular backbones and $2$-scale hierarchical models lead to consistently improved performance for both depth estimation and view synthesis tasks.**
| Monocular | Multi-View | Depth (TartanAir) | 3DGS (RealEstate10K) | Param<br>(M) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Abs Rel $\downarrow$ | $\delta_{1}\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| ViT-S | $1$-scale | 8.46 | 93.02 | 26.76 | 0.877 | 0.123 | 37 |
| ViT-B | $1$-scale | 6.94 | 94.46 | 27.09 | 0.881 | 0.119 | 113 |
| ViT-L | $1$-scale | 6.07 | 95.52 | 27.34 | 0.885 | 0.118 | 354 |
| ViT-S | $2$-scale | 7.01 | 94.56 | 26.96 | 0.880 | 0.122 | 40 |
| ViT-B | $2$-scale | 6.22 | 95.31 | 27.27 | 0.885 | 0.120 | 117 |
| ViT-L | $2$-scale | 5.57 | 96.07 | 27.44 | 0.887 | 0.119 | 360 |

**Table 2: Ablations. We evaluate the contribution of the monocular feature branch and the cost volume branch, as well as different monocular features. Our results indicate that the monocular feature and cost volume are complementary, with large performance drops when removing either one. The pre-trained Depth Anything network weights achieve overall the best performance.**
| Module | Method | Depth (TartanAir) | 3DGS (RealEstate10K) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | Abs Rel $\downarrow$ | $\delta_{1}\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| Components | full | 8.46 | 93.02 | 26.76 | 0.877 | 0.123 |
| w/o mono feature | 12.25 | 88.00 | 26.27 | 0.866 | 0.130 |  |
| w/o cost volume | 11.34 | 90.02 | 23.09 | 0.761 | 0.187 |  |
| single branch | 11.26 | 90.84 | 25.99 | 0.858 | 0.134 |  |
| Monocular features | ConvNeXt-T | 10.50 | 91.13 | 26.28 | 0.867 | 0.130 |
| Midas | 9.53 | 91.61 | 26.40 | 0.869 | 0.129 |  |
| DINO v2 | 8.93 | 92.49 | 26.68 | 0.874 | 0.125 |  |
| Depth Anything v1 | 8.38 | 93.23 | 26.70 | 0.875 | 0.125 |  |
| Depth Anything v2 | 8.46 | 93.02 | 26.76 | 0.877 | 0.123 |  |

Figure: Figure 3: Effect of monocular features for depth on ScanNet. The monocular feature greatly improves challenging situations like texture-less regions (*e.g.*, the wall in the first example) and reflective surfaces (*e.g.*, the refrigerator in the second example).
Refer to caption: x3.png

Figure: Figure 4: Effect of monocular features for 3DGS on RealEstate10K. Without monocular features, the model struggles at predicting reliable depth for pixels that are not able to find correspondences (*e.g.*, the lounger chair highlighted with the read rectangle), which subsequently causes misalignment in the rendered image due to the incorrect geometry.
Refer to caption: x4.png

### 4.1 Model Variants

We first study several different model variants for both depth estimation and view synthesis tasks. In particular, we explore different monocular backbones (Yang et al., ) (ViT-S, ViT-B, ViT-L) and different multi-view models ($1$-scale and $2$-scale). We conduct depth experiments on the large-scale TartanAir (Wang et al., ) synthetic dataset, which features both indoor and outdoor scenes and has perfect ground truth depth. The Gaussian splatting experiments are conducted on the standard RealEstate10K (Zhou et al., ) dataset. Following community standards, we report the depth evaluation metrics (Eigen et al., ) of Abs Rel (relative $\ell_{1}$ error) and $\delta_{1}$ (percentage of correctly estimated pixels within a threshold) and novel view synthesis metrics (Kerbl et al., ) of PSNR, SSIM, and LPIPS. The results in Table 1 demonstrate that both larger monocular backbones and $2$-scale hierarchical models lead to consistently improved performance for both tasks.

From Table 1, we can also see that better depth network architecture leads to improved view synthesis. In Appendix A.1, we conduct additional experiments to study the effect of different initializations for the depth network to the view synthesis performance. Our results show that a better depth model initialization also contributes to improved rendering quality. In summary, both better depth network architecture and better depth model initialization lead to improved novel view synthesis results.

### 4.2 Ablation Study and Analysis

In this section, we study the properties of our key components on the TartanAir dataset (for depth estimation) and RealEstate10K dataset (for view synthesis with Gaussian splatting).

Monocular Features for Depth Estimation and Gaussian Splatting. In Table 2, we compare our full model (full) with the model variant where the monocular depth feature branch (with a ViT-S model pre-trained by Depth Anything v2 (Yang et al., )) is removed (w/o mono feature), leaving only the multi-view branch. We can observe a clear performance drop for both depth and view synthesis tasks. In Fig. 3, we visualize the depth predictions and error maps of both models on the ScanNet dataset. The pure multi-view feature matching-based approach struggles a lot at texture-less regions and reflective surfaces. At the same time, our full model achieves reliable results thanks to the powerful prior captured in monocular depth features. We also show the visual comparisons for the Gaussian splatting task in Fig. 4 with two input views. For regions (*e.g.*, the lounge chairs) that only appear in a single image, the pure multi-view method is unable to find correspondences. It thus produces unreliable depth predictions, leading to misalignment in the rendered novel views due to the incorrect geometry.

We also experiment with removing the cost volume (w/o cost volume) in the multi-view branch and observed a significant performance drop. This indicates that obtaining scale- and multi-view consistent predictions with a pure monocular depth backbone is challenging, which constrains achieving high-quality results in downstream tasks.

Fusion Strategy. We compare with alternative strategies for fusing the monocular features for multi-view depth estimation. In particular, we compare with MVSFormer (Cao et al., ) which constructs the cost volume with monocular features. More specifically, we replace our multi-view feature extractor with a weight-sharing ViT model and use the ViT features to build the cost volume as done in MVSFormer. This leads to a single-branch architecture (single branch in Table 2), unlike our two-branch design where the monocular features are not used to build the cost volume. We can observe that our fusion strategy performs significantly better than the single-branch design, potentially because our two-branch design disentangles feature matching and obtaining monocular priors, which makes the learning task easier.

Different Monocular Features. In Table 2, we also evaluate different monocular features, including the ConvNeXt-T (Liu et al., ) features used in AFNet (Cheng et al., ), and other popular monocular features including Midas (Ranftl et al., ) and DINO v2 (Oquab et al., ). The pre-trained Depth Anything monocular features achieve overall the best performance.

**Table 3: Unsupervised Depth Pre-Training with Gaussian Splatting. We first perform unsupervised pre-training with Gaussian splatting on RealEstate10K and then measure the depth performance on TartanAir, ScanNet and KITTI after fine-tuning for the depth task. Compared to previous supervised pre-trained models Depth Anything (for monocular ViT) and UniMatch (for multi-view Transformer), our new unsupervised pre-training improves performance consistently in all metrics. The improvements are especially significant on the challenging datasets like TartanAir and KITTI.**
| Initialization | TartanAir | ScanNet | KITTI |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Abs Rel $\downarrow$ | $\delta_{1}\uparrow$ | Abs Rel $\downarrow$ | $\delta_{1}\uparrow$ | Abs Rel $\downarrow$ | $\delta_{1}\uparrow$ |
| Depth Anything (only mono) | 11.12 | 89.97 | 6.78 | 96.13 | 11.67 | 85.96 |
| UniMatch + Depth Anything (mv & mono) | 10.86 | 90.55 | 6.70 | 96.14 | 11.56 | 87.27 |
| DepthSplat (full model) | 10.20 | 91.10 | 6.60 | 96.27 | 10.68 | 89.92 |

### 4.3 Unsupervised Depth Pre-Training with Gaussian Splatting

By connecting Gaussian splatting and depth, our DepthSplat provides a way to pre-train the depth model in a fully unsupervised manner. In particular, we first train our full model on the large-scale unlabelled RealEstate10K dataset (contains $\sim$ 67K Youtube videos) with only the Gaussian splatting rendering loss (Eqn. 2), without any direct supervision on the depth predictions. After pre-training, we take the pre-trained depth model and further fine-tune it to the depth task on the mixed TartanAir and VKITTI2 (Cabon et al., ) datasets with ground truth depth supervision.

In Table 3, we evaluate the performance on both in-domain TartanAir test set and the zero-shot generalization on unseen ScanNet and KITTI datasets. We compare with only initializing the monocular backbone with Depth Anything and additionally initializing the multi-view Transformer backbone with UniMatch (Xu et al., ). Our approach achieves the best results on all three datasets. It is also interesting to observe that our pre-training contributes more on more challenging datasets (i.e., TartanAir and KITTI which feature complex large-scale scenes unlike ScanNet only contains indoor scenes). We also note that both Depth Anything and UniMatch are trained with ground truth supervision, while our DepthSplat is trained with only photometric losses. Given the increasing popularity of view synthesis (Zheng & Vedaldi, ; Weng et al., ) and multi-view generative models (Shi et al., ; Blattmann et al., ), new multi-view datasets (Ling et al., ) and models (Voleti et al., ) are gradually introduced, our approach provides a way to pre-train depth models on large-scale unlabelled multi-view image datasets. This could potentially further improve the multi-view consistency and robustness of existing depth models (Yang et al., ; Piccinelli et al., ), which are usually trained with ground truth depth supervision.

### 4.4 Benchmark Comparisons

Comparisons on ScanNet and RealEstate10K. Table 5 and Table 5 compare the depth and novel view synthesis results on the standard ScanNet and RealEstate10K benchmarks, respectively. For both comparisons, the numbers of input images are two. We can observe that our DepthSplat outperforms previous methods Teed & Deng (); Yu et al. (); Xu et al. (); Chen et al. () by clear margins on both datasets for both tasks. The visual comparison with previous best methods is shown in Fig. 5 and Fig. 6, where our method significantly improves the performance on challenging scenarios like texture-less regions and occlusions.

Comparisons on DL3DV. To further evaluate the performance on complex real-world scenes, we conduct comparisons with the latest state-of-the-art Gaussian splatting model MVSplat (Chen et al., ) on the recently introduced DL3DV dataset (Ling et al., ). We also compare the results of different numbers of input views (2, 4 and 6) on this dataset. We fine-tune MVSplat and our RealEstate10K pre-trained models on DL3DV training scenes and report the results on the benchmark test set in Table 6. Our DepthSplat consistently outperforms MVSplat in all metrics. Visual comparisons with MVSplat on the DL3DV dataset are shown in Fig. 7, where MVSplat’s depth quality lags behind our DepthSplat due to matching failure, which leads to blurry and distorted view synthesis results. We show more visual comparison results in Appendix A.2. It is also worth noting that our method scales more efficiently to more input views thanks to our lightweight local feature matching approach (Sec 3.1), which is unlike the expensive pair-wise matching used in MVSplat.

We invite the readers to our project page [https://haofeixu.github.io/depthsplat/](https://haofeixu.github.io/depthsplat/) for the video results on different number of input views (6 and 12), where our DepthSplat is able to reconstruct larger-scale or 360 scenes from more input views, without any optimization.

**Table 4: Two-view Depth Estimation on ScanNet. Our DepthSplat outperforms all prior methods by clear margins.**
| Method | Abs Rel $\downarrow$ | RMSE $\downarrow$ | RMSElog $\downarrow$ |
| --- | --- | --- | --- |
| DeMoN | 0.231 | 0.761 | 0.289 |
| BA-Net | 0.161 | 0.346 | 0.214 |
| DeepV2D | 0.057 | 0.168 | 0.080 |
| NeuralRecon | 0.047 | 0.164 | 0.093 |
| DRO | 0.053 | 0.168 | 0.081 |
| UniMatch | 0.059 | 0.179 | 0.082 |
| DepthSplat (ours) | 0.044 | 0.119 | 0.059 |

Figure: Figure 5: Depth Comparison on ScanNet. Our DepthSplat performs significantly better than UniMatch (Xu et al., ) on challenging parts (e.g., edges of the couch and pillows).
Refer to caption: x5.png

Figure: Figure 6: Visual Synthesis on RealEstate10K. Our DepthSplat performs significantly better than pixelSplat (Charatan et al., ) and MVSplat (Chen et al., ) in challenging regions.
Refer to caption: x6.png

**Table 6: Comparisons on DL3DV. Our DepthSplat not only consistently outperforms MVSplat on different number of input views, but also scales more efficiently to more input views.**
| Method | #Views | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Time (s) |
| --- | --- | --- | --- | --- | --- |
| MVSplat | 2 | 17.54 | 0.529 | 0.402 | 0.072 |
| DepthSplat | 19.05 | 0.610 | 0.313 | 0.101 |  |
| MVSplat | 4 | 21.63 | 0.721 | 0.233 | 0.146 |
| DepthSplat | 22.82 | 0.766 | 0.188 | 0.124 |  |
| MVSplat | 6 | 22.93 | 0.775 | 0.193 | 0.263 |
| DepthSplat | 23.83 | 0.808 | 0.158 | 0.161 |  |

Figure: Figure 7: Visual Comparisons on DL3DV. Our DepthSplat performs significantly better than MVSplat (Chen et al., ) on regions that are hard to match (e.g., repeated patterns).
Refer to caption: x7.png

## 5 Conclusion

In this paper, we introduce DepthSplat, a new approach to connecting Gaussian splatting and depth to achieve state-of-the-art results on ScanNet, RealEstate10K and DL3DV datasets for both depth and view synthesis tasks.
We also show that our model enables unsupervised pre-training depth with Gaussian splatting rendering loss, providing a way to leverage large-scale unlabelled multi-view image datasets for training more multi-view consistent and robust depth models. Our current model requires camera pose information as input along with the multi-view images, removing this requirement would be exciting future work.

## Acknowledgement

We thank Yuedong Chen for his generous help with the DL3DV dataset. Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 - project number 390727645.

## Appendix A Appendix

### A.1 Depth Pre-Training for Gaussian Splatting

We further study the effect of depth pre-training for Gaussian splatting experiments. Unlike the pre-trained models Depth Anything (Yang et al., ) and UniMatch (Xu et al., ), which are trained for either monocular and multi-view features separately, we perform joint training of our full two-branch depth model on the depth datasets. We then compare the results of different initializations to the depth network for the full Gaussian splatting model. We can see from Table 7 that improved depth initialization leads to better view synthesis results.

**Table 7: Depth to Gaussian Splatting Transfer. We compare different pre-trained network weights for initializing the depth network when training our full DepthSplat model for view synthesis. Compared to using Depth Anything and UniMatch pre-trained monocular and multi-view network weights for initializing the monocular ViT and multi-view Transformer features, our jointly trained 2-branch depth model (full model) performs best on all metrics.**
| Initialization | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| --- | --- | --- | --- |
| Depth Anything (only mono) | 26.59 | 0.874 | 0.1256 |
| UniMatch + Depth Anything (mv & mono) | 26.76 | 0.877 | 0.1234 |
| DepthSplat (full model) | 26.81 | 0.878 | 0.1225 |

### A.2 More Visual Comparisons on DL3DV

We show more visual comparison results on DL3DV in Fig. 8 with different number of input views, where our DepthSplat consistently outperforms MVSplat in challenging regions.

Figure: Figure 8: Different number of input views on DL3DV. Our DepthSplat performs consistently better than MVSplat (Chen et al., ).
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 3
strengths: 1. Unified Architecture: The architecture of DepthSplat seamlessly integrates depth estimation and 3D Gaussian splatting, allowing for efficient cross-task knowledge transfer. This integration addresses the limitations of each technique while enhancing their complementary strengths.

2. Comprehensive Experiments: This paper validates its approach through extensive experiments, showcasing the model's robustness and generalizability.
summary: This paper introduces DepthSplat, a framework that bridges Gaussian splatting and depth estimation to improve both tasks. By incorporating pre-trained monocular depth features, the proposed method enhances 3D Gaussian splatting for novel view synthesis and enables unsupervised pre-training of depth models on unlabeled datasets. Extensive experiments demonstrate that DepthSplat achieves state-of-the-art performance in both depth estimation and view synthesis.
weaknesses: Although the proposed method achieves impressive performance, its technical contribution is relatively weak. 
The authors claim two main contributions: integrating pretrained monocular depth features into the multiview depth model and demonstrating the capability of Gaussian Splatting to serve as an objective for unsupervised depth learning. 

First, the approach to fuse multiview features and monocular features is straightforward, and the idea has been explored in previous works, such as [1, 2]. 

Second, while using Gaussian Splatting for unsupervised depth learning is interesting, it is not particularly innovative, as view-synthesis based approaches have been the mainstream for unsupervised depth estimation. Neural rendering methods, such as NeRF, have already been integrated into unsupervised depth estimation in previous works [3]. 

Finally, the feedforward framework proposed in this paper is largely similar to previous works, such as Pixelsplat, with the main difference being the architecture of the depth estimation network.

[1] MultiView Depth Estimation by Fusing Single-View Depth Probability With MultiView Geometry. CVPR 2022 

[2] Learning To Fuse Monocular and MultiView Cues for MultiFrame Depth Estimation in Dynamic Scenes. CVPR 2023 

[3] DeLiRa: Self-Supervised Depth, Light, and Radiance Fields. ICCV 2023

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: From the initial manuscript, I have some concerns and questions about the proposed method. With the questions properly addressed, I am eager to raise my ratings.

1. About the pre-trained Monocular Depth Estimation Module the authors are leveraging to initialize the weights of the monocular backbone, I am a bit confused if the authors are utilizing DepthAnythingv1[1] or DepthAnythingv2[2]. In most of the parts, the model is only specified as DepthAnything which is not clear which version of the model the authors are utilizing. Specifically, in Section 4.2, it is mentioned that DepthAnythingv2 is utilized but in Table 2, v1 is best in depth prediction and v2 is best in novel view synthesis and the authors simply mention that DepthAnything shows the best performance. I would suggest the authors to explicitly mention the version of the Depth Anything network they are utilizing as it is a bit confusing.

2. What is the architectural design for the Gaussian parameter prediction? It is only mentioned as “We append an additional lightweight network to
predict other Gaussian parameters” but I can’t not find any additional information about the lightweight network in the main paper or the appendix.

3. How does the pre-training stage only with the Gaussian Splatting rendering loss improve the performance of depth estimation? Specifically, I would like to see the comparison between the Depth prediction branch fine-tuned directly with the depth labels and with the pre-training stage.

4. As the authors claim that DepthSplat can achieve multi-view consistent depths, are there any evaluations that can show the superiority of DepthSplat on multi-view consistency? For example, a recent work called FreeSplat[3] mentions that when using per-pixel Gaussians, due to the slight inconsistent position of the Gaussians of corresponding pixels, this leads to quality degradation and proposes a Gaussian fusion method. I believe that with multi-view consistent depth, this type of additional method is not strictly required. I would appreciate if the authors could explain how the multi-view consistency can be understood or evaluated.

---

[1] Yang, Lihe, et al. "Depth anything: Unleashing the power of large-scale unlabeled data." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Yang, Lihe, et al. "Depth Anything V2." arXiv preprint arXiv:2406.09414 (2024).

[3] Wang, Yunsong, et al. "FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes." *arXiv preprint arXiv:2405.17958* (2024).
soundness: 4
strengths: - The overall paper is well-written and easy to understand, where most of the architectural designs and training details are also specifically mentioned, making the re-implementation of the paper easy.
- From the perspective of generalized feed-forward 3DGS, I believe that existing works still struggle to extend to multiple views whereas DepthSplat enables up to novel view synthesis with 6 views.
- Building up from a pre-trained Monocular Depth Estimation (MDE) Module, DepthSplat improves the performance of both Novel View Synthesis and Depth Estimation.
- The choice of the backbone MDE network and the components of the architecture is discussed through well-conducted ablation studies.
summary: The paper presents DepthSplat, a novel approach that connects 3D Gaussian Splatting (3DGS) with multi-view and monocular depth estimation to enhance both depth prediction and generalized novel view synthesis. Leveraging pre-trained monocular depth features from the Monocular Depth Estimation(MDE) model, DepthSplat improves the robustness of depth estimation, especially in challenging scenes with occlusions, texture-less regions, and reflective surfaces, which are typically problematic for multi-view methods alone. By integrating monocular features into a dual-branch model architecture DepthSplat achieves both multi-view consistency and high-quality reconstruction. Ablation studies show that monocular features are crucial for best performance, especially in difficult areas, and that DepthSplat's dual-branch design outperforms single-branch alternatives by simplifying the learning process. As a result, the presented DepthSplat archives state-of-the-art performance in multiple datasets.
weaknesses: Although the proposed contributions of DepthSplat are valuable, I believe that not all of the mentioned contributions are shown in the paper. Specifically,

- Some of the architectures are not mentioned explicitly in the paper, such as how the Gaussian parameters are estimated.
- Although the authors mention that the pre-training of the network only with the Gaussian Splatting rendering loss is a way to pre-train the depth model in a fully unsupervised manner, there is no explanation or ablation of how this pre-training improves the depth prediction performance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Pre-trained models like DUSt3R[1] and MASt3R[2], which perform well on both monocular and multi-view tasks, might serve as viable alternatives to DepthSplat's dedicated monocular branch. Have the authors considered using pre-trained models like DUSt3R or MASt3R, which could potentially simplify the architecture? What benefits does DepthSplat’s separate monocular branch provide in comparison? 
2. How well does DepthSplat generalize across datasets, such as when a model trained on RealEstate10K is tested on DL3DV? Would it maintain performance in depth estimation and novel view synthesis on previously unseen datasets?
3. Would it be possible if DepthSplat's depth estimation result is compared with 3DGS works like MVSplat or PixelSplat? This comparison would be very interesting to see.

[1] Wang, Shuzhe, et al. "Dust3r: Geometric 3d vision made easy." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Leroy, Vincent, Yohann Cabon, and Jérôme Revaud. "Grounding Image Matching in 3D with MASt3R." arXiv preprint arXiv:2406.09756 (2024).
soundness: 3
strengths: - The paper leverages pre-trained monocular depth models effectively to achieve strong and superior performance in both depth estimation and novel view rendering.
- DepthSplat achieves efficient memory usage and computational scalability, allowing it to handle large-scale scenes and even produce 360° views. This is a substantial improvement for feed-forward 3DGS models.
summary: The paper introduces DepthSplat, a novel model that integrates 3D Gaussian splatting (3DGS) with single- and multi-view depth estimation, thereby enhancing the performance of both tasks. By leveraging pre-trained monocular depth features, the model produces robust multi-view depth predictions that improve 3DGS outputs. This framework enables pre-training depth model from large scale unlabelled dataset without the ground truth depths. Through this approach, the model achieves state-of-the-art performance in both multi-view depth estimation and novel view synthesis tasks.
weaknesses: 1. Can the authors analyze how DepthSplat’s depth estimation performance varies with different numbers of input views? This would provide a better understanding of its robustness in sparse-view settings. Moreover, I would like to see how it performs with varying number of inputs in terms of PSNR, SSIM and LPIPS, when evaluated on RealEstate10K.
2.  I would like to see the efficiency comparison (memory consumption) to MVSplat [1], with varying the number of input views. Also, I am quite confused by the speed comparison in Table 6. If I understood correctly, the proposed method shares similar architecture to MVSplat, besides it adds monocular feature extraction part, which should increase the computation. Why does it infer faster than MVSplat?
3. For N-view evaluation, It seems to show different observation from [2, 3]. In [2, 3], MVSplat's synthesized image qualities drastically drop when the number of views increase. However, in the Table 6, it seems to increase. What is the difference that led to such differing results? If the number of gaussians increase, unless a module to control the number of Gaussians is proposed [2, 3] I was assuming that the results should not improve significantly. Is it because the authors use top-2 nearest views only for cost computation? However, even so, this does not explain the apparent improvements despite redundant 3D Gaussians depicting the same object surface.
4. The paper does not provide a detailed description of the architecture for the 3D Gaussian prediction module. This component is central to the view synthesis pipeline, so more information on its design would improve clarity and reproducibility.
5. The paper lacks results on the depth estimation performance after unsupervised pre-training alone, without additional fine-tuning. Having these results would help evaluate the standalone effectiveness of the unsupervised pre-training step for depth tasks.
6. As noted in the related works section, DepthSplat effectively augments multi-view cost volumes with pre-trained monocular depth estimation features. However, its approach closely resembles existing models that fuse monocular and multi-view depth estimations, such as those presented in [4, 5], particularly in how they handle the fusion of these estimations, aside from DepthSplat's use of pre-trained models. The biggest differences may be either concatenation or cross-attention between monocular branch and multi-view branch, and the use of pre-trained large model weights. To me, the novelty and effectiveness are questionable, since without pretrained weights, the performance would drop significantly, and the fusing part is highly similar to previous works. Providing more justification on how the integration of pre-trained models constitutes a novel contribution would help strengthen the paper.

[1] Chen, Yuedong, et al. "Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images." arXiv preprint arXiv:2403.14627 (2024).

[2] Fei, Xin, et al. "PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views." arXiv preprint arXiv:2410.18979 (2024).

[3] Wang, Yunsong, et al. "FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes." arXiv preprint arXiv:2405.17958 (2024).

[4] Li, Rui, et al. "Learning to fuse monocular and multi-view cues for multi-frame depth estimation in dynamic scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Cheng, Junda, et al. "Adaptive fusion of single-view and multi-view depth for autonomous driving." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. I have some doubts about the training process of DepthSplat. Is the model for predicting gaussian parameters and the model for predicting depth trained separately?
2. Are the depth and 3DGS results in Table 1 evaluated using the same model?
3. Can you provide metrics for DepthSplat pre-trained models using only photimetric loss in Table 3?


If the author can answer my question, I am willing to change my score.
soundness: 3
strengths: 1. DepthSplat combines the advantages of 3DGS reconstruction and depth estimation to enhance the performance in both tasks. I think this topic is innovative.
2. DepthSplat enhance the depth quality of low-textured areas and reflective surfaces by employing a pre-trained monocular depth model.
3. DepthSplat achieves state-of-the-art results on both tasks.
summary: This paper presents DepthSplat, which connects Gaussian splatting and depth estimation. Specifically, DepthSplat exploits the complementary nature of sparse view feed-forward 3DGS and robust single/multi-view depth estimation to enhance the performance in both tasks.
For 3DGS: DepthSplat leverages pre-trained monocular depth features (DINOv2) to improve the ability of generalizable of 3D Gaussian reconstructions.
For depth estimation: This paper provides an unsupervised method for pre-training depth prediction models, achieving superior results compared to training from scratch.
weaknesses: 1. Please answer the difference between Transplat[1] and DepthSplat. it seems that Transplat also improves the quality of reconstruction by using DINO features.
2. It seems similar to MVSFormer[2] when training the depth model, may I ask the difference between the two methods?

[1] Zhang C, Zou Y, Li Z, et al. Transplat: Generalizable 3d gaussian splatting from sparse multi-view images with transformers[J]. arXiv preprint arXiv:2408.13770, 2024.
[2] Cao C, Ren X, Fu Y. MVSFormer: Multi-view stereo by learning robust image features and temperature-based depth[J]. arXiv preprint arXiv:2208.02541, 2022.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Q1 - In the experiment in Section 4.3, it is unclear whether DepthAnything and UniMatch were fine-tuned on the test datasets, as this information is not clearly indicated in Lines 425-431.

Q2 - Table 3 appears inconclusive since these three methods utilize different architectures, making it difficult to directly assess the advantage of Gaussian splatting for pre-training. An additional comparison showing DepthSplat with and without pre-training on RE10k could be informative.

Q3 - More details on inference speed and memory cost across different configurations (e.g., depth model, hierarchical architecture) would enhance understanding.

Q4 - What are the key differences between DepthSplat and Transplat?
soundness: 3
strengths: S1 - The task addressed in this paper is meaningful, and its insights are interesting.

S2 - This paper contributes a robust multi-view depth model by leveraging pre-trained monocular depth features, resulting in high-quality feed-forward 3D Gaussian splatting reconstructions.

S3 - The study demonstrates that Gaussian splatting can serve as an unsupervised pre-training objective for learning powerful depth models from large-scale unlabeled datasets, which is valuable for large-scale pre-trained models.

S4 - Experimental results show that both depth estimation (Table 4) and reconstruction outcomes (Table 5) are significantly improved compared to the tested baselines.
summary: This paper introduces a model named DepthSplat, which integrates Gaussian splatting with depth estimation. 
The core of the pipeline is a 2D U-Net module that fuses multi-view depth information (from the cost volume) with monocular depth (derived from a pre-trained model, such as Depth Anything). 
Experiments conducted on both depth estimation and novel view synthesis tasks demonstrate that the proposed method achieves competitive results in both areas.
weaknesses: W1 - The experiment in Section 4.3 appears inconclusive, as it does not convincingly support the claim that "Gaussian splatting can serve as an unsupervised pre-training objective for learning robust depth models from large-scale unlabeled datasets."

W2 - The paper bears similarities to Transplat[1], which also uses monocular depth priors to enhance Gaussian splatting reconstruction. What are the main differences between these two approaches?

W3 - Additional details about DepthSplat, such as its inference speed and memory consumption compared to Mvsplat[2], would be valuable.

[1]Zhang C, Zou Y, Li Z, et al. Transplat: Generalizable 3d gaussian splatting from sparse multi-view images with transformers[J]. arXiv preprint arXiv:2408.13770, 2024.

[2]Chen Y, Xu H, Zheng C, et al. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images[J]. arXiv preprint arXiv:2403.14627, 2024.