Uncertainty modeling for fine-tuned implicit functions

## Abstract

Abstract Implicit functions such as Neural Radiance Fields (NeRFs), occupancy networks, and signed distance functions (SDFs) have become pivotal in computer vision for reconstructing detailed object shapes from sparse views. Achieving optimal performance with these models can be challenging due to the extreme sparsity of inputs and distribution shifts induced by data corruptions. To this end, large, noise-free synthetic datasets can serve as shape priors to help models fill in gaps, but the resulting reconstructions must be approached with caution. Uncertainty estimation is crucial for assessing the quality of these reconstructions, particularly in identifying areas where the model is uncertain about the parts it has inferred from the prior. In this paper, we introduce Dropsembles, a novel method for uncertainty estimation in tuned implicit functions. We demonstrate the efficacy of our approach through a series of experiments, starting with toy examples and progressing to a real-world scenario. Specifically, we train a Convolutional Occupancy Network on synthetic anatomical data and test it on low-resolution MRI segmentations of the lumbar spine. Our results show that Dropsembles achieve the accuracy and calibration levels of deep ensembles but with significantly less computational cost.

## 1 Introduction

Recent advancements in neural implicit functions have facilitated their use for 3D object representations and applications in novel views synthesis in computer vision. Neural Radiance Fields (NeRFs) [] have gained recognition for their ability to accurately synthesize novel views of complex scenes and became an important tool in applications of photorealistic rendering [], such as virtual reality and augmented reality. Signed Distance Functions (SDFs)[, ] is shown to be particularly useful in scenarios where precise boundary details are crucial, like industrial design and robotics. Occupancy Networks[], which model shapes as a probabilistic grid of space occupancy, excel in handling topological variations, making them ideal for medical imaging[] and animation. Advancing this concept, Convolutional Occupancy Networks [] integrate convolutional networks to enhance spatial learning, proving effective in detailed architectural modeling and complex reconstructions.

In practice, achieving optimal performance with these methods often requires densely sampled input data and a high degree of similarity between the training data and the target object. However, such ideal conditions are rarely met in practical applications like augmented reality, virtual reality, autonomous driving, and medical contexts, where inputs are typically sparse and less precise[, ]. For example, in medical applications, generating precise 3D representations from sparse inputs is particularly crucial for morphological analysis[]. Patient-specific anatomical modeling significantly enhances the assessment of a patient’s condition and aids in devising customized treatment plans[].

In addition to sparsity, real-world data usually suffers from noise and corruption that leads to distribution shifts with respect to the training data, and thus to significant performance degradation. Occlusion, noise, truncation, and lack of depth measurements [] greatly affect reconstruction from monocular observations. Noise in estimations of camera poses inevitably degrades reconstruction from sparse views[, , ]. Under input sparsity and distribution shifts, in safety-critical applications, such as medical[] or autonomous driving[], it is crucial that inferred information is transparently disclosed to the end user, as it may significantly influence the decision-making process. One approach to this end is to quantify uncertainty in the reconstruction. However, very few works focused on modeling uncertainty in these implicit representations[].

This paper addresses uncertainty quantification in 3D reconstruction from sparse and corrupted data using neural implicit functions. We propose *Dropsembles*,(^1^11[https://github.com/klanita/Dropsembles](https://github.com/klanita/Dropsembles)) a method that leverages a noise-free, densely sampled dataset to model a shape prior and reconstructs based on sparse and noisy input through fine-tuning the model.
Our approach aims to utilize the high-quality prior dataset to fill gaps and correct corruptions in the target data despite distribution shifts between prior and target datasets induced by noise and corruption. Crucially, the model is designed to recognize and highlight areas of high uncertainty. To the best of our knowledge, uncertainty quantification in 3D reconstruction for neural implicit functions has not been comprehensively addressed in the literature.

Monte Carlo (MC) dropout [] and deep ensembles [] are two commonly used baselines for estimating uncertainty in computer vision applications [], which could be readily applied to our task. MC dropout is simple to integrate and computationally efficient during training. However, it often underestimates uncertainty and requires multiple forward passes during inference, increasing computational cost[, ]. Deep ensembles involve training multiple neural networks independently and averaging their predictions, capturing a wider range of potential outputs. This method provides improved predictive performance and better-calibrated uncertainty estimates but is computationally costly and memory-intensive. This becomes especially demanding in our fine-tuning context, where each model in an ensemble needs to be trained on both datasets. Here we introduce Dropsembles, a method that creates ensembles based on the dropout technique. Dropsembles aim to moderate the computational demands associated with ensembles while attempting to maintain prediction accuracy and uncertainty calibration of deep ensembles. Combined with Elastic Weight Consolidation (EWC) [], Dropsembles is able to mitigate distribution shifts between source and target datasets.

Contributions
This paper introduces several contributions to address the gap in modeling uncertainties in fine-tuned neural implicit functions:

- •
We introduce Dropsembles (overview in Figure 1) to achieve the performance of vanilla ensembles with significantly reduced computation cost (subsection 4.1).
- •
We introduce EWC-inspired uncertainty adaptation to take into account the distribution shift in uncertainty modeling (subsection 4.2).
- •
We include a series of experiments to validate the proposed methods, in a controlled benchmark on synthetic data and in a real-world medical application. We demonstrate that it is possible to achieve high reconstruction quality and preserve patient-specific details when using synthetic data in the form of an anatomical atlas (section 5).

Figure: Figure 1: Occupancy network training with a dense prior and fine-tuning on a sparse dataset.
Refer to caption: x1.png

## 2 Related work

#### Implicit shape modeling from sparse input

Much attention has recently been directed towards the problem of novel view synthesis from sparse views enhancing this task using NeRFs [, , , , , , , , ], SDFs [, ], and occupancy networks [].
Implicit functions also found their applications in medical imaging, particularly in 3D shape reconstruction from sparse MRI slices [, , , , , ].
In this paper, we focus on occupancy networks due to their relevance in medical imaging applications but note that Dropsembles can be used with other neural implicit functions.

#### Uncertainty modeling

Estimating uncertainty in deep models has attracted significant interest in recent years [, , ].
A remaining challenge here is uncertainty estimation under distribution shifts, which in most real-world scenarios is essential for guaranteeing the reliability and resilience of predictions when confronted with out-of-distribution (OOD) examples [, , ].
In 3D reconstruction, this is important since corruption and sparsity pattern in test samples often deviates from those in training data.
While numerous methods have been devised to detect OOD cases [, , , , ] or enhance accuracy in unobserved domains [, ], the adaptation of uncertainty estimates has been relatively unexplored. In a comprehensive benchmark study, [] find that from all metrics considered, calibration transfers worst, meaning that models that are well calibrated on the training data are not necessarily well calibrated on OOD data. In an era dominated by LLMs and foundation models, addressing this issue becomes paramount, particularly given the tendency of such models for overconfidence as a result of fine-tuning [, ].
To overcome such issues, [] introduces a method for estimating uncertainty in fine-tuned LLMs using Low-Rank Adaptation. In computer vision, [] highlights that fine-tuning comes at the cost of robustness, and address this issues by ensembling the weights of the zero-shot and fine-tuned
models. [] propose uncertainty estimation in one-shot object detection, particularly focusing on fine-tuned models.

Closer to Dropsembels are [, , ] that address uncertainty of neural implicit representations. [] models uncertainty in the color and density output of a scene-level neural representation with conditional normalizing flows.
[] uses this method in robotics applications. [] estimates uncertainty for neural object representation from monocular images by propagating it from image space first to latent space, and consequently to 3D object shape. Dropsembels can complement all of the above approaches by modeling the uncertainty in the weights of implicit functions throughout fine-tuning, rendering them more robust to distribution shifts. By doing so, it improves both the performance and reliability of methods building on neural implicit representations.

## 3 Background

### 3.1 Implicit Shape Representations

Implicit functions are widely used in computer vision to represent complex shapes. Assuming a continuous encoding of the object domain $\mathcal{X}$, an SDF for a surface $S$ encoding a shape is defined as $f:\mathcal{X}\to\mathbb{R}$ where: $f(x)<0\Leftrightarrow x\text{ "inside" }S\quad f(x)=0\Leftrightarrow x\in S$.
Another popular way to represent a shape object is an occupancy function $g:\mathcal{X}\to\{0,1\}$ defined as $g(x)=\mathbb{I}_{\text{Shape Object}}(x)$ where $\mathbb{I}$ corresponds to the indicator function.
The primary advantage of both approaches lies in their ability to accurately model complex shapes from sparse observations using deep neural networks. This capability underpins methods such as "DeepSDF" [] and "Occupancy Network" [], which approximate functions by non-linearly regressing observations to the surface encoding the shape.
To facilitate learning from training sets, these techniques are often employed in conjunction with latent representations to capture diverse shape variations and at inference time yield continuous shape representations.
The latent representations can be trained directly from data using autoencoders [] or autodecoders []. Recent research has also explored using large pre-trained models like CLIP to obtain latent representations []. While the specific choice of the latent encoder is beyond the scope of this paper, we will refer to it as an "oracle."

Both methods can be described with the same formulation. We consider a dataset $\mathcal{D}:=\bigg{\{}\{x^{i}_{j},y^{i}_{j}\}_{j\in[K]},Z^{i}\bigg{\}}_{i\in[N]}$, where $N$ is the number of images, $K$ is the number of available pixels per image , $Z^{i}$ is a latent representation of image $i$ given by the "oracle", $x^{i}_{j}\in\mathcal{X}\subseteq\mathbb{R}^{3}$ is the position corresponding to the observed value $y^{i}_{j}$, which are obtained through an SDF or voxel occupancy following the definitions above. In both settings, the regression function is a neural network $f_{\theta}(x,Z),\theta\in\Theta_{NN}$.
At training, networks are trained using the following optimization objective.

$$ $\hat{R}_{\mathcal{D}}(\theta)=\frac{1}{NK}\sum_{j\in[K]\ \&\ i\in[N]}l(f_{ \theta}(x^{i}_{j},Z^{i}),y^{i}_{j})$ (1) $$

Here, the loss function $l$ depends on the method. For occupancy networks, $l$ is the binary cross-entropy, reflecting the binary nature of occupancy, while it is the $L_{2}$ norm for SDFs, aligning with the continuous nature of the distance function.

### 3.2 Uncertainty modeling

#### Deep Ensembles

Deep Ensembles[] apply ensemble methods to neural networks [, , , ].
Multiple independent networks are trained to determine a set of optimal weights $\{\hat{\theta}_{i}\}_{i\in[M]}$ with the subscript $i$ denoting different networks.
Ensembles mitigate the risk of selecting a single set of weights, which may not yield good results particularly when training data consist of fewer samples relative to the size of the parameter space. Instead, multiple solutions with comparable accuracy are determined, allowing the ensemble to average outputs and minimize the selection risk. Training of deep ensembles is associated with high computational demands due to training multiple networks.

#### Dropout

Dropout [] is a regularization technique that aims to reduce overfitting by randomly omitting subsets of features during each training step. By "dropping out" (i.e., setting to zero) a subset of activations within a network layer, it diminishes the network’s reliance on specific neurons, encouraging the development of more robust features.
Gal and Ghahramani [] demonstrated that dropout can also be interpreted from a Bayesian perspective and applied towards modeling uncertainty. Dropout at test time (referred to as Monte Carlo Dropout) is shown to perform approximate Bayesian inference, essentially through using randomness in dropout configurations for uncertainty modeling.

### 3.3 Elastic Weight Consolidation

Elastic Weight Consolidation (EWC) is a regularization technique introduced in continual learning to address catastrophic forgetting []. The underlying principle is to protect parameters crucial for previous tasks while learning new ones.
Assume two distinct tasks, $A$ and $B$, with their respective datasets $\mathcal{D}_{A}$ and $\mathcal{D}_{B}$, where $\mathcal{D}_{A}\cap\mathcal{D}_{B}=\emptyset$. The tasks are learned sequentially without access to previous tasks’ datasets.
First, task $A$ is learned by training a neural network on $\mathcal{D}_{A}$, resulting in a set of optimal weights $\hat{\theta}_{A}$. When learning task $B$ using dataset $\mathcal{D}_{B}$,
EWC regularizes the weights so they remain within a region in the parameter space that led to good accuracy for task $A$.

Next, during learning for task $B$, approximate posterior of weights obtained during learning for $A$ constrains the optimization.
A Gaussian approximation to $\log p(\theta|\mathcal{D}_{A})$ (see App. A for further details), serves as the regularizer

$$ $\hat{\theta}_{B}=\arg\min_{\theta}\hat{R}_{\mathcal{D}_{B}}(\theta)+\lambda( \theta-\hat{\theta}_{A})^{T}F(\hat{\theta}_{A})(\theta-\hat{\theta}_{A})$ (2) $$

where $\hat{R}_{\mathcal{D}_{B}}(\theta)$ corresponds to the likelihood term $\text{log }p(\mathcal{D}_{B}|\theta)$, $\lambda$ is a hyperparameter, $F$ is the diagonal of Fisher information matrix. Details can be found in Appendix A

## 4 Methods

We focus on shape reconstruction from sparsely sampled and corrupted inputs using occupancy networks. While the proposed method can be applied to both SDF and occupancy networks, we focus on the latter for demonstration. Given the sparse and corrupted nature of our input, we train an occupancy network with high-quality data and then fine-tune it on a test sample. Through fine-tuning, we expect the model to adapt to the input while transferring the prior information captured in training.

Assume access to datasets $\mathcal{D}_{A}$ and $\mathcal{D}_{B}$, defined in Section (3.1), with the number of points per image $K_{A}$ and $K_{B}$, and the number of images $N_{A}$ and $N_{B}$. The dataset $\mathcal{D}_{A}$ is assumed to be high quality and "dense," while the dataset $\mathcal{D}_{B}$ is "sparse" in terms of the number of points observed per image and potentially contains corruptions in individual images, i.e., $K_{A}\gg K_{B}$. Additionally, it is assumed that the number of images in $\mathcal{D}_{A}$ is greater than in $\mathcal{D}_{B}$, i.e., $N_{A}>N_{B}$ (in our experiments, we consider the case of a single image $N_{B}=1$). The two datasets can only be accessed successively and not simultaneously; that is, $\mathcal{D}_{A}$ is accessed first, followed by $\mathcal{D}_{B}$, without further access to $\mathcal{D}_{A}$. Note that the latent representation vectors ${Z^{i}}$ are assumed to be obtained by the same learning oracle for both datasets $\mathcal{D}_{A}$ and $\mathcal{D}_{B}$ described in Section 3.1.
Without loss of generality, in this paper, we consider the latent map $\mathcal{L}:\mathcal{X}\to\mathcal{Z}$ to be a "frozen" encoder pretrained on dataset $\mathcal{D}_{A}$.

The underlying parametric function class of the model is assumed to be a neural network $f_{\theta}:\mathcal{X}\times\mathcal{Z}\to\mathcal{Y}$ approximating a regression function as described in Section 3.1. The parameter $\theta$ represents the set of weight matrices of the network i.e $\theta:=\{W_{i}\}_{i\in[L]}$ where $L$ corresponds to the number of layers. The output space $\mathcal{Y}$ corresponds to $[0,1]$ for occupancy networks and $\mathbb{R}$ for signed distance functions. It is important to note that the versatility of the network model in terms of its output space is tailored to the specific modeling task. However, this flexibility does not limit the proposed method, which remains versatile across different tasks.

The procedures outlined in sections 4.1 and 4.2 involve two stages of training: initially on dataset $\mathcal{D}_{A}$, denoted as $\text{Task }A$, followed by training on dataset $\mathcal{D}_{B}$, referred to as $\text{Task }B$. The primary objective is to achieve high prediction accuracy in the second stage, while acknowledging the inherent challenges posed by the sparsity and smaller size of $\mathcal{D}_{B}$ relative to $\mathcal{D}_{A}$.

### 4.1 Dropsembles

Given the sparsity and corrupted nature of test samples, adaptation of the prior model is prone to uncertainties. Here we introduce Dropsembles, a technique that combines benefits of both dropout and deep ensembles, to capture and quantify this uncertainty. This approach aims to moderate the computational demands associated with ensembles while attempting to maintain reasonable prediction accuracy. Although ensembles are known for their reliable predictions, they are resource-intensive. In real-world applications, involving large training sets, this cost becomes a hindering factor.

In contrast, dropout involves training only a single model instance. Applying dropout to a neural network involves sampling a "thinned" network, effectively the same as applying a binary mask to the weights. However, unlike ensembles where each model is trained independently, dropout results in networks that are not independent; they share weights. Thus, training a neural network with dropout is akin to simultaneously training a collection of $2^{n}$ thinned networks, all sharing a substantial portion of their weights. However, often this comes at a price of less accurate predictions.

In Dropsembles, the model is first trained with dropout on dataset $\mathcal{D}_{A}$. Subsequently, $M$ thinned network $\{f_{\theta_{m}}\}_{m\in[M]}$ instances are generated by sampling binary masks. Each of these "thinned" instances is then fine-tuned on dataset $\mathcal{D}_{B}$ independently, effectively creating an ensemble of "thinned" networks initialized with correlated weights. For inference and uncertainty estimations, this ensemble is treated as a uniformly weighted mixture model, and the predictions are aggregated in the same manner as traditional ensembles: $p(y|x)=\frac{1}{M}\sum_{m=1}^{M}p_{\theta_{m}}(y|x,\theta_{m})$. Although ensembles typically benefit from networks being large and independent, our experiments show that this relaxation of independence does not significantly diminish prediction performance metrics or expected calibration error. The overall training procedure is summarized in Algorithm 1.

In the context of implicit functions, networks are optimized according to the objective defined in Equation (1). However, it is important to emphasize that Dropsembles is a versatile framework that can be adapted to train with any objective necessary for a specific task.

Figure: Algorithm 1 Dropsembles

### 4.2 Elastic Weight Consolidation regularization for implicit shape modeling

In the basic version of optimization described above, networks start their training initialized from the learned posterior of dataset $\mathcal{D}_{A}$.
However, there is no guarantee that during fine-tuning the network won’t diverge arbitrarily from initial weights.
This is particularly problematic for implicit shape modeling, where it is essential to retain information from a large and dense prior dataset $\mathcal{D}_{A}$ while adapting to sparse and noisy data $\mathcal{D}_{B}$ to avoid overfitting. To address this concern, we borrow developments from the continual learning literature, as they seamlessly fit into the framework described above.

In particular, when fine-tuning individual instances of thinned networks on dataset $\mathcal{D}_{B}$, we can apply the same reasoning to each network instance as described in EWC. Thus, the learning objective for part B is replaced by the objective described in equation (2), which includes an additional regularization term.

## 5 Experimental results

Our objective is to provide trustworthy predictions on dataset B without hurting performance. We do so by modeling the uncertainty of the weights of a fine-tuned model. In standard prediction tasks, models are evaluated with respect to accuracy and Expected Calibration Error (ECE). However, our specific setup (reconstruction from sparse views) calls for additional evaluation metrics suitable to computer vision tasks, such as Dice Score Coefficient (DSC) and Hausdorff distance. Besides ECE, we include reliability diagrams [] given our preference for a more conservative modeling approach. Details on evaluation metrics can be found in Appendix B.

### 5.1 Classification under distribution shift

#### Toy dataset

We first demonstrate our method on a toy data set for binary classification. This allows quantitative analysis and demonstrates the versatility of our method. The experiment focuses on fine-tuning a trained model on few samples with distribution shift compared to the original training samples.
We generated two-dimensional datasets A and B with a sinusoidal decision boundary and Gaussian noise. A moderate distribution shift was modeled between the datasets by adjusting the support values. We created 1000 train samples for dataset A, and 50 train samples for dataset B. Training details can be found in Appendix C.

All variations of Dropsembles and Ensembles were tested using just 4 network instances.
The best regularization parameter was selected for each method via an ablation study in Appendix D. All metrics in Table 1 were averaged across 3 random seeds. From the evaluations (Table 1, Figure 2), it is apparent that EWC improves the performance of Ensembles and Dropsembles on both datasets.

Figure: Figure 2: Toy classification example. a) Training data for binary classification task ("red" vs "blue") from datasets A (points, light) and B (crosses, dark) b) MC dropout trained only on Dataset A. c) Comparison of methods fine-tuned on Dataset B. Points are colored by the predicted class.
Refer to caption: x2.png

**Table 1: Comparison of methods fine-tuned on dataset B for toy classification and corrupted MNIST reconstruction experiments. The results are averaged over three samples. All metrics reported in [%].**
| Toy classification | MNIST |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | Acc-A $\uparrow$ | ECE-A $\downarrow$ | Acc-B $\uparrow$ | ECE-B $\downarrow$ | DSC $\uparrow$ | ECE $\downarrow$ |
| MCdropout | $59.0\pm 5.3$ | $40.0\pm 4.3$ | $90.5\pm 0.6$ | $11.5\pm 1.7$ | $58.2\pm 3.1$ | $8.0\pm 1.1$ |
| Dropsembles | $59.2\pm 4.3$ | $39.8\pm 4.3$ | $90.5\pm 0.6$ | $9.2\pm 1.3$ | $64.9\pm 6.6$ | $6.8\pm 1.4$ |
| Ensembles | $56.5\pm 2.4$ | $42.5\pm 1.7$ | $90.0\pm 0.0$ | $8.5\pm 0.6$ | $62.2\pm 3.5$ | $6.5\pm 0.6$ |
| MCdropout + EWC | $66.8\pm 5.6$ | $33.0\pm 4.8$ | $86.5\pm 2.1$ | $10.5\pm 1.9$ | $60.5\pm 4.8$ | $7.1\pm 1.5$ |
| Dropsembles + EWC | $\textbf{96.2}\pm 4.2$ | $\textbf{5.8}\pm 5.3$ | $\underline{93.5}\pm 3.8$ | $\underline{7.0}\pm 1.4$ | $\underline{70.3}\pm 3.9$ | $\underline{6.4}\pm 2.5$ |
| Ensembles + EWC | $\underline{95.8}\pm 2.5$ | $\underline{7.5}\pm 2.4$ | $\textbf{95.5}\pm 1.3$ | $\textbf{5.5}\pm 1.0$ | $\textbf{71.3}\pm 3.6$ | $\textbf{5.2}\pm 1.9$ |

### 5.2 Implicit shape modeling from sparse views

#### MNIST digit reconstruction

In our next experiment, we explore the reconstruction from sparse inputs using the MNIST dataset. To this end, we converted the images into binary masks through thresholding. To simulate sparse input conditions, we applied a grid mask to the images, masking out every third row and column. This masking strategy was consistently applied across both datasets A and B. For dataset A, we utilized all images for a single digit "7" from the MNIST training set.

To introduce a moderate distribution shift and mimic common real-world dataset corruptions, we rotated the images in dataset B by 15 degrees and obscured approximately 15 percent of the original pixels. In this section we are focusing on shape reconstruction, therefore "fine-tuning" and "testing" are applied per image and not per dataset. Given that the occupancy network is trained at the pixel level, each of these images effectively constitutes an individual "dataset B". We randomly selected three distinct images from the MNIST test split, which would give us three different variations of "dataset B". Fine-tuning and evaluating the whole test split of MNIST dataset would be computationally demanding, as each occupancy network is fine-tuned on an individual image.

Our occupancy network comprises an 8-layer MLP, designed to process the latent representation along with the 2-dimensional coordinates of each pixel, thereby facilitating pixel-wise predictions. For configurations utilizing dropout, we incorporated a dropout layer with a probability of $p=0.3$ following each linear layer in the network. Training details can be found in Appendix C.

We conducted an ablation study to identify the optimal regularization parameter for all methods, for which details are provided in the Appendix D. We demonstrate qualitative predictions and associated uncertainties for a single-image example in Figure 3. Visual inspection of the uncertainty estimates reveals that EWC-regularized ensembles exhibit the desired behavior: they not only deliver accurate predictions but also provide conservative uncertainty estimates, particularly noting high uncertainty in regions with data corruption. Reliability diagrams, along with quantitative evaluations presented in Table 1, confirm that elastic regularization enhances the performance across all methods. Notably, EWC-regularized Dropsembles achieve performance comparable to that of the Ensembles but with significantly reduced resource usage.

Figure: Figure 3: Corrupted MNIST reconstruction example. a) Example of training images. b) Comparison of fine-tuned methods on dataset B. A "perfectly calibrated" method would have reliability diagrams aligned on the diagonal. A "good conservative" method would have all bars above the diagonal.
Refer to caption: x3.png

#### Lumbar spine

In this section, we adapt the data preparation procedure from [] but replace the high-quality CT dataset with a synthetic dataset of anatomical shapes. As an anatomical shape prior, we utilize a rigged anatomical model from "TurboSquid". To model patient-specific variability and variations in poses during MRI acquisition, we generated 94 rigged deformations. Point-cloud models from the atlas were converted to voxels at 256 voxel resolution in order to correspond to the resolution of the target MRI dataset. We further applied random elastic deformations to mimic patient-specific shape variability. We created a paired "sparse" - "dense" dataset by selecting a consistent set of 17–21 sagittal slices and additionally applied two iterations of connected erosions to simulate patient-specific automatic MR segmentations []. A bicubic upsampling was employed on the "sparse" inputs to adapt them for the encoder. We trained ReconNet [] on the entire training split of the atlas dataset to obtain an encoder, which we subsequently kept frozen. For the implicit decoder, we employed only one of the rigged samples to accurately quantify performance on synthetic data and demonstrate the method’s robustness to medium and strong distribution shifts. For real-world applications, we recommend using the entire rigged transformations dataset.

For the occupancy network, we employed the MLP architecture described by [], which features eight linear layers, each of 128 dimensions (described in Appendix C). The initial training of the occupancy network was conducted on an anatomical atlas. This intensive training process required 48 hours on a single NVIDIA RTX A6000 GPU, equipped with 48 GB of memory. Given the significant resource demands, conducting a full ensemble comparison was deemed impractical. Such an analysis would require the training of multiple network instances, substantially increasing computational resource consumption. It is important to highlight that all models were trained at a very high resolution, which is critical in medical applications where precision is paramount.

As the target dataset B, we use a publicly available dataset from [], which comprises MR+CT images from 20 subjects. We employed the same segmentation network as [] to obtain segmentations of 5 vertebrae, 5 discs, and the spinal canal []. These automatically generated segmentations of high-quality MRI samples were used as the ground truth for the sparse 3D reconstruction task. To create sparse inputs, we removed the same set of sagittal slices as in the atlas dataset. We randomly selected 3 subjects for the consequent fine-tuning and testing.

The MR dataset described above lacks ground truth segmentations, which complicates numerical evaluation. The goal of the model is to impute missing or misclassified parts using anatomical atlas priors, which, while enhancing reconstruction, are typically marked as incorrect in standard segmentation metrics. To facilitate a more accurate assessment of reconstruction metrics, we introduce an intermediate benchmark. For this purpose, we utilize three random rigged deformations from the atlas that were not exposed during training and apply a doubled level of erosions. This strong augmentation provides a challenging input for dataset B and allows to model a significant distribution shift aimed at rigorously testing the model under adverse conditions.

Numerical evaluation in Table 2 demonstrates consistent improvement of our method upon MC-dropout. In order to investigate the performance of the proposed method on MR dataset, we perform a detailed qualitative analysis in Figure 4. A consistent and distinct pattern not captured by the metrics alone, stands out from the qualitative assessment: MC-dropout tends to produce noisy reconstructions, as illustrated in Figure 4c-d, and generally yields predictions that lack coherent, continuous shapes. This behaviour is partially captured in metric "DSC avg" in Table 2 - dice score evaluated on individual samples of the network instances. This observation is notable because, although models trained with conventional dropout generate reasonable predictions, their use in uncertainty estimation undermines the fundamental objective of 3D modeling. Samples drawn from the dropout distribution do not yield plausible shapes, as evidenced in Figure 4d. In contrast, our model, which draws from deep ensembles, does not exhibit these limitations. Furthermore, increasing the number of samples in MC-dropout does not resolve this issue but rather leads to higher computational demands, as documented in Table 2.

EWC did not show significant benefits in this experiment, despite its superior performance in previous examples. We believe the challenge lies in selecting the appropriate regularization strength, compounded by the heavy computational demands of high-resolution 3D reconstruction. Nevertheless, EWC did not degrade performance and can be applied safely even with suboptimal weights. This critical aspect of model tuning highlights a complex area that demands further investigation.

Figure: Figure 4: Lumbar spine reconstruction example on Subject 2. a) 3D-rendered views of sparse inputs (bicubic upsampling), dense ground truth (GT), and predictions by our method. b) Histograms of uncertainty (entropy) values, truncated between 0.1 and 1 for visibility. c) Examples of uncertainty estimates for two different sagittal slices of the 3D volume. d) For each method, network predictions are randomly sampled and the corresponding reconstructions are depicted.
Refer to caption: x4.png

**Table 2: Comparison of methods fine-tuned on dataset B of lumbar spine experiment. Evaluations are performed on the corrupted atlas. Metrics are reported for dataset B. Baseline is trained on dataset A.**
| Method | DSC [%] $\uparrow$ | DSC avg [%] $\uparrow$ | HD $\downarrow$ | ECE [%] $\downarrow$ | Inference [sec] $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| Baseline | $65.0\pm 2.0$ | $63.9\pm 1.8$ | $17.5\pm 1.2$ | $25.2\pm 0.4$ | 167 |
| MCdropout | $85.3\pm 2.4$ | $83.0\pm 2.0$ | $13.1\pm 2.6$ | $\underline{3.4}\pm 1.8$ | 164 |
| MCdropout + EWC | $85.3\pm 2.4$ | $83.1\pm 2.0$ | $12.3\pm 2.4$ | $\textbf{3.3}\pm 1.8$ | 117 |
| Dropsembles | $\underline{86.8}\pm 2.4$ | $\underline{86.3}\pm 2.3$ | $\underline{11.4}\pm 1.8$ | $4.6\pm 1.7$ | 22 |
| Dropsembles + EWC | $\textbf{86.9}\pm 2.4$ | $\textbf{86.4}\pm 2.3$ | $\textbf{10.9}\pm 2.1$ | $4.5\pm 1.7$ | 21 |

## 6 Discussion and conclusion

#### Strengths

In this study, we advanced sparse 3D shape reconstruction for high-precision applications by introducing uncertainty modeling. We developed a flexible framework that facilitates uncertainty-aware fine-tuning of models and showcased its utility in reconstructing the lumbar spine from sparse and corrupted MRI data. Our observations suggest that traditional uncertainty methods like MC dropout might not be ideal for implicit shape reconstruction, as they tend to undermine the basic principles of implicit functions. However, our Dropsemble method effectively addresses these limitations, providing a promising alternative for uncertainty modeling.

#### Limitations

Despite the efficiencies Dropsembles introduce increase in resource utilization during the initial training phase, they face the same computation demands in the fine-tuning stage as Ensemble models. Another drawback is that, while using only four network samples can yield good performance, this small number introduces a moderate level of variability in results.

#### Future directions

In this work, we focused on uncertainty in input weights, assuming the encoder to be frozen. An interesting future direction is to combine Dropsembles with the uncertainty propagation [] to comprehensively cover all aspects of uncertainty modeling in sparse 3D shape reconstruction.

## Appendix A Appendix: Elastic Weight Consolidation

Elastic Weight Consolidation (EWC) was introduced to address catastrophic forgetting in continual learning []. EWC is a regularization that protects crucial parameters in a network when learning a new task, in order to avoid catastrophic forgetting.

Assume two distinct tasks, $A$ and $B$, with their respective datasets $\mathcal{D}_{A}$ and $\mathcal{D}_{B}$, where $\mathcal{D}_{A}\cap\mathcal{D}_{B}=\emptyset$. The tasks are learned sequentially without access to previous task datasets.
First, task $A$ is learned by training a neural network on $\mathcal{D}_{A}$, resulting in a set of optimal weights $\hat{\theta}_{A}$. When learning task $B$ using dataset $\mathcal{D}_{B}$,
EWC regularizes the weights so they remain within a region in the parameter space that led to good accuracy for task $A$.
The justification of the method is based on probabilistic principles. Given a combined dataset $\mathcal{D}:=\mathcal{D}_{A}\cup\mathcal{D}_{B}$, applying Bayes’ rule yields:

$$ $\log p(\theta|\mathcal{D})=\log p(\mathcal{D}|\theta)+\log p(\theta)-\log p( \mathcal{D})$ (3) $$

where $p(\mathcal{D}|\theta)$ corresponds to the likelihood over the entire dataset and $p(\theta)$ is the user-defined prior over the weights of the network. Rearranging equation (3) yields

$$ $\log p(\theta|\mathcal{D})=\log p(\mathcal{D}_{B}|\theta)+\log p(\theta| \mathcal{D}_{A})-\log p(\mathcal{D}_{B})$ (4) $$

where it can be observed that all information from $\mathcal{D}_{A}$ is contained in the posterior $p(\theta|\mathcal{D}_{A})$, which is usually intractable. Therefore, the EWC method employs the Laplace approximation to the posterior, a process conducted during the training of task $A$. The resulting approximated posterior is modeled as a Gaussian distribution, with its mean represented by $\hat{\theta}_{A}$ and covariance matrix $\Sigma_{A}=(F(\hat{\theta}_{A})\circ I)^{-1}$ where $F(\hat{\theta}_{A})$ denotes the Fisher information matrix evaluated over data set $\mathcal{D}_{A}$ at $\hat{\theta}_{A}$.

Next, in stage $B$, the optimization of parameter $\theta$ incorporates the approximation of the posterior obtained during task $A$ as a constraint in the optimization process. Following the equation (4), this approximation, $\log p(\theta|\mathcal{D}_{A})$, serves as a regularizer in the learning objective:

$$ $\hat{\theta}_{B}=\arg\min_{\theta}\hat{R}_{\mathcal{D}_{B}}(\theta)+\lambda( \theta-\hat{\theta}_{A})^{T}(F(\hat{\theta}_{A})\circ I)(\theta-\hat{\theta}_{ A})$ (5) $$

where $\hat{R}(\theta)$ corresponds to the likelihood term $\text{log }p(\mathcal{D}_{B}|\theta)$, $\lambda$ is a hyperparameter, $I$ represents the identity matrix, and $\circ$ describes the Hadamard product.

## Appendix B Appendix: metrics

Uncertainty estimates for regression tasks involve using unbiased estimates of the modes of the approximated posterior predictive distribution. In neural networks, this corresponds to performing multiple stochastic forward passes. For classification tasks, three main methods for uncertainty estimates are variational ratios, predictive entropy, and mutual information []. In our experiments, we opt for predictive entropy.

#### Dice score

The Dice score, also known as the Dice Similarity Coefficient (DSC), measures the similarity between two data sets, commonly used in medical imaging to evaluate segmentation accuracy. Defined as $\text{DSC}=\frac{2\times|X\cap Y|}{|X|+|Y|}$, where $X$ and $Y$ represent the ground truth and predicted segmentation sets, respectively. The score ranges from 0 to 1, with 1 indicating perfect agreement and 0 representing no overlap.

#### Reliability diagrams and Expected Calibration Error

Reliability diagrams are graphical tools used in uncertainty modeling to assess the calibration of probabilistic predictions. They plot predicted probabilities against empirical frequencies, allowing for visual inspection of how well the predicted probabilities of a model correspond to the actual outcomes. A perfectly calibrated model would align closely with the diagonal line from the bottom left to the top right of the plot, indicating that the predicted probabilities match the observed probabilities. Reliability diagrams are computed by binning predicted probabilities into intervals. For each bin, the mean predicted probability is plotted against the observed frequency of the corresponding outcomes. This involves calculating the proportion of positive outcomes in each bin and plotting these against the average predicted probability for the bin. The closer the points lie to the diagonal line from the bottom left to the top right, the more calibrated the model is considered. Reliability diagrams are closely related to the Expected Calibration Error (ECE), which quantitatively assesses a model’s calibration. ECE is computed as a weighted average of the absolute differences between the predicted probabilities and the actual outcome frequencies across different bins used in reliability diagrams. Each bin’s weight corresponds to the number of samples it contains. Thus, while reliability diagrams provide a visual interpretation of model calibration, ECE offers a single numerical value summarizing the calibration error across all bins.

## Appendix C Appendix: training details

#### Toy experiment

We generated two-dimensional datasets A and B with a sinusoidal decision boundary and Gaussian noise. A moderate distribution shift was modeled between the datasets by adjusting the support values of $x_{1}$: $x_{1}\in[-0.75,0.7]$ for dataset A and $x_{1}\in[-0.5,2.0]$ for dataset B. We created 1000 training samples and 500 test samples for dataset A, and 50 train and 500 test samples for dataset B. Training details could be found in Appendix C.

We used a consistent model architecture across experiments—a straightforward 3-layer MLP with 256 hidden units in each layer. For methods using dropout, a dropout layer ($p=0.3$) followed each linear layer. For ensemble methods, 4 separate networks were trained on dataset A.
For ensembles and Dropsembles we used 4 network instances for fine-tuning and inference. For MC dropout we used 100 samples at inference. We trained for 800 epochs for training with a learning rate $1e-3$ and 600 epochs for tuning with a learning rate $5e-3$.

#### MNIST experiment

First, we trained a small autoencoder, consisting of three convolutional layers in the encoder, only on dataset A. The encoder does not have dropout layers. The encoder was trained with cross-entropy loss for 50 epochs with a learning rate $0.01$ and a cosine warmup scheduler. After this initial training, the encoder was kept fixed (frozen) for all subsequent experiments, and the decoder was discarded. This encoder now serves to generate a latent representation of the input data, which is then supplied to an occupancy network.

The 8-layer MLP occupancy network was trained with cross-entropy loss for 50 epochs on dataset A with a learning rate $0.005$ and a cosine warmup scheduler. For ensemble methods, 4 separate networks were trained on dataset A. For fine-tuning on dataset B we used same learning rate but tuned the networks for 30 epochs only. For ensembles and Dropsembles we used 4 network instances for fine-tuning and inference.

#### Lumbar spine experiment

All experiments in this section were performed on NVIDIA RTX A6000 GPU, equipped with 48 GB of memory. The networks were trained at 16-mixed precision due to memory constraints.

First ReconNet encoder was trained with cross-entropy loss on a full training split of rigged atlas dataset to obtain a "frozen" encoder. We used a learning rate $0.01$ and trained for 100 epochs with early stopping.

The occupancy network architecture incorporates skip connections and ReLU activations, with dropout layers ($p=0.2$) following each linear layer except the last. To effectively model the entire lumbar spine, we adapted the strategy from [] by replacing the learnable latent vector with an output from a pre-trained convolutional encoder. Specifically, we performed bilinear upsampling on the output of this frozen encoder to generate a detailed latent representation for each voxel. This representation, coupled with 3-dimensional voxel coordinates, was provided as input to the MLP. We used cross-entropy loss for training and fine-tuning the occupancy network. The network was trained for 100 epochs on dataset A with early stopping applied after 68 epochs. Learning rate $0.001$ and batch size 32 were used for training, where each batch we used only [64, 64, 64] random voxels. For fine-tuning on dataset B we used the same of parameters for all the methods: learning rate $0.001$ and tuned it for 50 epochs without early stopping. Results evaluated at the checkpoint of the last epoch are presented throughout the paper.

## Appendix D Appendix: selecting optimal regularization strength

Figure: Figure 5: Toy classification example ablation for EWC regularization strength. Averages and confidence over 3 random seeds. Red dot represents the optimal lambda selected for the main figure.
Refer to caption: x5.png

Figure: Figure 6: Corrupted MNIST reconstruction example ablation for EWC regularization strength. Dots correspond to different samples.
Refer to caption: x6.png

Figure: Figure 7: Corrupted atlas reconstruction example ablation for EWC regularization strength. Dots correspond to different samples. Due to high computational demand we were not able to evaluate all examples on the full grid.
Refer to caption: x7.png

## Appendix E Appendix: additional experimental results

**Table 3: Comparison of methods tuned on dataset B of lumbar spine MR dataset. Dice Score Coefficient (DSC), Hausdorff distance (HD), Dice Score Coefficient average per network sample (DSC avg), Expected Calibration Error (ECE), and total inference time (time) are reported for dataset B. Baseline is MC dropout trained on dataset A.**
|  | Method | DSC [%] $\uparrow$ | DSC avg [%] $\uparrow$ | HD $\downarrow$ | ECE [%] $\downarrow$ | time [sec] $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Subject 1 | Baseline | 75.8 | $74.0\pm 0.0$ | 18.2 | 4.7 | 161 |
| MCdropout | 93.5 | $92.3\pm 0.0$ | 6.3 | 3.6 | 159 |  |
| MCdropout + EWC | 93.7 | $92.5\pm 0.0$ | 7.1 | 3.5 | 156 |  |
| Dropsembles | 93.9 | $\textbf{93.5}\pm 0.1$ | 6.2 | 4.4 | 20 |  |
| Dropsembles + EWC | 93.9 | $\textbf{93.5}\pm 0.0$ | 6.0 | 4.2 | 21 |  |
| Subject 2 | Baseline | 35.9 | $36.4\pm 0.1$ | 34.2 | 33.6 | 30 |
| MCdropout | 91.3 | $89.7\pm 0.0$ | 15.2 | 4.9 | 160 |  |
| MCdropout + EWC | 91.2 | $89.7\pm 0.0$ | 18.2 | 4.8 | 157 |  |
| Dropsembles | 92.0 | $\textbf{91.5}\pm 0.1$ | 9.4 | 6.1 | 19 |  |
| Dropsembles + EWC | 92.0 | $\textbf{91.5}\pm 0.1$ | 9.5 | 6.2 | 19 |  |
| Subject 3 | Baseline | 76.5 | $74.2\pm 0.0$ | 19.5 | 3.7 | 159 |
| MCdropout | 92.5 | $91.3\pm 0.0$ | 7.1 | 4.9 | 158 |  |
| MCdropout + EWC | 92.5 | $91.3\pm 0.0$ | 6.6 | 4.7 | 156 |  |
| Dropsembles | 92.6 | $\textbf{92.2}\pm 0.0$ | 8.2 | 5.3 | 19 |  |
| Dropsembles + EWC | 92.6 | $\textbf{92.2}\pm 0.0$ | 7.9 | 5.3 | 19 |  |

**Table 4: Comparison of methods tuned on dataset B of lumbar spine experiment. Evaluations are performed on the corrupted atlas. Dice Score Coefficient (DSC), Hausdorff distance (HD), Dice Score Coefficient average per network sample (DSC avg), Expected Calibration Error (ECE), and total inference time (time) are reported for dataset B. Baseline is MC dropout trained on dataset A.**
|  | Method | DSC [%] $\uparrow$ | DSC avg [%] $\uparrow$ | HD $\downarrow$ | ECE [%] $\downarrow$ | time [sec] $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Corrupted<br>Atlas 1 | Baseline | 65.5 | $64.4\pm 0.0$ | 16.8 | 24.7 | 172 |
| MCdropout | 83.5 | $81.7\pm 0.0$ | 14.3 | 4.6 | 160 |  |
| MCdropout + EWC | 83.6 | $82.0\pm 0.0$ | 14.1 | 4.1 | 30 |  |
| Dropsembles | 85.0 | $\textbf{84.7}\pm 0.2$ | 13.2 | 5.7 | 21 |  |
| Dropsembles + EWC | 85.0 | $\textbf{84.7}\pm 0.2$ | 13.3 | 5.6 | 21 |  |
| Corrupted<br>Atlas 2 | Baseline | 67.2 | $65.8\pm 0.0$ | 16.5 | 25.6 | 164 |
| MCdropout | 88.6 | $85.7\pm 0.0$ | 9.4 | 0.8 | 170 |  |
| MCdropout + EWC | 88.7 | $85.8\pm 0.1$ | 8.9 | 0.8 | 161 |  |
| Dropsembles | 90.2 | $\textbf{89.6}\pm 0.0$ | 8.9 | 2.3 | 22 |  |
| Dropsembles + EWC | 90.3 | $\textbf{89.6}\pm 0.0$ | 8.1 | 2.1 | 22 |  |
| Corrupted<br>Atlas 3 | Baseline | 62.3 | $61.4\pm 0.0$ | 23.5 | 25.4 | 163 |
| MCdropout | 83.7 | $81.5\pm 0.0$ | 15.4 | 4.8 | 162 |  |
| MCdropout + EWC | 83.6 | $81.5\pm 0.1$ | 14.0 | 5.0 | 162 |  |
| Dropsembles | 85.2 | $\textbf{84.8}\pm 0.1$ | 12.0 | 5.9 | 22 |  |
| Dropsembles + EWC | 85.3 | $\textbf{84.8}\pm 0.2$ | 11.2 | 5.8 | 22 |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: **Q1**: Why is EWC necessary if Tasks A and B are similar?

Could you clarify why EWC is used in this context, given that Task A (dense dataset) and Task B (sparse/noisy dataset) do not appear to be significantly different? Is the intent to quantify uncertainty in a continual learning setting, or is EWC applied simply as a regularization tool?
What makes Dropsembles particularly suited to neural implicit representations?

**Q2**: Dropsembles seems like a standard dropout-ensemble approach. Are there specific adaptations that make it uniquely effective for implicit functions in 3D reconstructions, especially regarding spatial consistency or grid-based predictions?
Considering Dropsembles combines well-known methods, why weren’t additional baselines, like Bayesian neural networks or variational inference-based [1], tested to provide a clearer comparison?

**Q3**: How does Dropsembles handle spatial dependencies?

Neural implicit representations often require spatial coherence across grid or voxel predictions. 
Does Dropsembles include any mechanism to maintain this spatial consistency in its uncertainty estimation, like [2]?

Could you discuss the limitations of Dropsembles in more detail?

**Q4**: Given that Dropsembles simply ensembles thinned models from dropout, what are its limitations in terms of computational cost, reliability, or adaptability to different 3D tasks?

**Minor**: The term "thinned network" isn’t a standard term in the field and may lead to confusion. It could be misinterpreted, as it doesn't clearly indicate that dropout is being applied.

**References**:

[1] Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representations, 3DV 2021

[2] Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty, NeurIPS 2021
soundness: 2
strengths: Key contributions include:

(1) A method for modeling epistemic uncertainty in fine-tuned implicit functions.  

(2) The application of Elastic Weight Consolidation (EWC) to address distribution shifts. 

(3) Experimental validation across toy and real-world datasets.
summary: This paper introduces 'Dropsembles', an approach to uncertainty modeling in fine-tuned neural implicit functions. 'Dropsembles' combines dropout-based methods with ensemble learning to balance computational efficiency and robust uncertainty estimation.
The authors conduct experiments on synthetic anatomical datasets and MRI segmentation tasks, demonstrating that 'Dropsembles' maintains predictive accuracy comparable to deep ensembles while reducing computational demands.
weaknesses: (1) **Questionable motivation for EWC**:

**Lack of task distinction**: The dense and sparse datasets (Tasks A and B) are similar in content, which doesn’t align well with the typical use case for EWC in continual learning. EWC is usually applied to preserve knowledge across distinct tasks, so its application here appears unjustified or artificial.

(2) **Limited novelty in dropsembles**:

**Lack of specificity to NIR**:
Dropsembles combines dropout with ensembling in a straightforward way without tailoring it for neural implicit representations. 
This method may not provide significant innovation for uncertainty estimation in 3D reconstruction, particularly if it doesn’t leverage the unique characteristics of implicit functions.

(3) **Insufficient baseline comparisons**:

Dropsembles is not compared to other standard uncertainty estimation techniques such as Bayesian neural networks, or other methods tailored to implicit representations [1]. This limits the evidence for Dropsembles’ effectiveness.

(4) **Limited dataset diversity**:

**Narrow generalizability**: The experiments are focused on synthetic anatomical data and sparse MRI segmentation, which may not fully demonstrate Dropsembles' robustness across various domains or types of data shifts.

References:

[1] Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representations, 3DV 2021

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: - Could you provide more details on how the EWC regularization strength was selected? Is there a rationale beyond the empirical ablation studies, or could a more systematic method be developed?

- Have you tested Dropsembles on other types of neural implicit functions or datasets beyond those presented? How do you anticipate the method will perform in domains with significantly different characteristics?

- While the method reduces training costs, inference demands remain comparable to traditional deep ensembles. Are there potential strategies to also optimize inference efficiency?

- Given that synthetic data might not capture all real-world complexities, how do you ensure the generalization of the model to highly variable or unseen real-world data?

- Can you elaborate on the effectiveness of Dropsembles in handling extreme distribution shifts? Are there specific limitations or edge cases where the method struggles?

- How easily can Dropsembles be integrated into existing neural implicit networks? Are there any specific prerequisites or limitations that practitioners should be aware of?
soundness: 3
strengths: - The focus on uncertainty modeling in fine-tuned neural implicit functions addresses an underexplored area, contributing valuable insights into this challenging problem space.

- Dropsembles combines dropout and deep ensembles, offering a computationally efficient  training method for uncertainty estimation specifically tailored to neural implicit functions. This application of established uncertainty quantification and continual learning techniques to neural implicit functions is considered a novel contribution to neural implicit functions.

- The paper provides a rigorous experimental evaluation across synthetic and real-world spine datasets, effectively demonstrating that Dropsembles maintain prediction accuracy and uncertainty calibration under varying conditions.

- The method achieves significant reductions in computational costs during training compared to traditional deep ensembles, without compromising performance. However, computational demands during inference remain comparable to those of ensemble methods.

- The integration of Elastic Weight Consolidation (EWC) effectively mitigates the impact of distribution shifts, enhancing the model's robustness to noisy and sparse inputs.

- The approach is straightforward and versatile, capable of being integrated into various neural implicit networks and adapted to different task-specific training objectives, making it broadly applicable.

- The strategy of training on synthetic data while addressing distribution shifts between synthetic and real data is a pragmatic solution, particularly in scenarios where real-world data is scarce and noisy.

- The paper is clearly written, well-structured, and easy to follow.
summary: The paper presents Dropsembles, a novel uncertainty quantification technique for neural implicit functions such as Neural Radiance Fields (NeRFs) and Occupancy Networks. This method aims to achieve the high accuracy and calibration of deep ensembles while significantly reducing computational overhead. By integrating Monte Carlo dropout with ensemble strategies, and leveraging Elastic Weight Consolidation (EWC), Dropsembles effectively handles distribution shifts and sparse, noisy data.

The primary focus is on addressing the challenges of 3D shape reconstruction from sparse and corrupted inputs. The method leverages high-quality, densely sampled synthetic datasets as shape priors to fill gaps and correct corruptions in the target data, even when distribution shifts occur due to noise and corruption. The premise is that these priors can guide the reconstruction process, ensuring more reliable outputs.

Initially, a Convolutional Occupancy Network is trained on high-quality synthetic data. During testing, the network is fine-tuned on individual sparse and noisy samples using EWC, which regularizes the adaptation process to prevent forgetting the original training data and to transfer prior knowledge effectively. This method involves creating ensembles through dropout, where binary masks generate multiple network instances. Each instance is then fine-tuned independently on the testing sample, forming an ensemble of networks initialized with correlated weights.

The paper validates the method through extensive experiments on toy datasets and low-resolution MRI segmentations of the lumbar spine. Results demonstrate that Dropsembles maintains accuracy and reliable uncertainty estimation and efficiently handles computational constraints and distribution shifts.
weaknesses: - While the method shows promise in specific tasks like lumbar spine reconstruction, broader application beyond medical imaging is not thoroughly explored or validated.
- Despite reduced costs compared to deep ensembles, Dropsembles still require significant resources during fine-tuning, especially on high-resolution data.
- Although Dropsembles handle sparse inputs well, the effectiveness on highly varied real-world scenarios with extreme data corruption could have been examined more comprehensively.
- While Dropsembles generally outperform baselines, the improvements in some scenarios, particularly when using EWC, are not consistently significant across all metrics.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: Importance of encoder design / Application in auto-decoder setting:

Q1: Given the suitability of INRs for non-grid data, it is sometimes challenging to design an encoder for non-grid data. How was the encoder used/designed for the MNIST case, where some pixels are occluded? Did the encoder see the entire image (e.g., with a CNN-based encoder)? Sorry if I missed this; I would really appreciate an answer, even if it does not change the comparability of the different methods/baselines.

Q2: In the INR community, auto-decoder settings are quite common as well (see DeepSDF or Mehta et al.). Could the same method also be used in this case, given that the encoding of, e.g., the shape or instance stems from the same MLP, or are there any theoretical limitations that prohibit a meaningful uncertainty in this case?
soundness: 3
strengths: Originality: While the paper builds upon technical concepts from the uncertainty modeling community and does not introduce novel technical components, I believe the explored application and conclusions are novel and impactful.

Quality: The paper is very well structured and intuitive. The presentation and writing of the method are very clear, and in combination with the experiments, the method is sound.  

Clarity: The paper is well written in all sections, especially in related work and methods, making it a charm to read. It really flows. I also really enjoyed how well the method was put into the context of related work (i.e. very thoroughly research in different areas) and how the method was described in a simple and comprehensive way. Moreover, the authors detail every aspect of the experimental design in the main paper, and together with the supplementary this makes the evaluation of the presented method easy to follow. This, together with the accessibility of the source code, facilitates reproducability of the experiments. 

Significance: While implicit neural representations are often trained - individually - for (single) images and other signals, cohort-based training is quite common for shape completion in SDFs and occupancy representations. Given the prevalence of encoder-/decoder in these settings, this paper constitutes a meaningful exploratory study in the modeling of decoder uncertainty.
summary: This paper explores the topic of uncertainty modeling for decoder fine-tuning in implicit neural representations for occupancy representations. The presented method employs a combination of widely used uncertainty modeling concepts (i.e., dropout, ensembling) and the elastic weight consolidation concept stemming from continual learning, with the notion of fine-tuning, allowing to obtain multiple (fine-tuned) models at less training cost, with partially improved or comparable performance in modeling the uncertainty. This is demonstrated in experiments with toy examples (Sinusoidal decision boundary, MNIST) to medical shapes with the perseverance of reconstruction accuracy.
weaknesses: 1) Related Work:

As detailed above, the paper is very well written and conducts a very thorough related work section. However, I think it would be important to touch upon one more area - i.e. the more recent architectures used in cohort-based training that also use encoder frameworks. Specifically, it would be interesting to discuss the advancements in [1,2], given that these constitute more recent alternatives to the DeepSDF architecture. As both works [1,2] use encoder-/decoder settings in combination with weight modulation approaches, this would also add emphasis to the relevance of this work. 

2) Experimental Design: While the experimental design is sound, I believe the paper could have explored a higher variation of datasets and application scenarios.

(a)  Experiments not only for Re-LU-based networks but also for other activation functions:
While ReLU networks were routinely used in [3,4,5] medical applications, the INR community has shifted towards other activation functions [6] or embedding/projection functions [7,8] (particularly for images). While not all (novel) shape modeling applications use these (yet?), I believe it would have been interesting to explore if the same holds true for, e.g., SIREN [6]. This ablation would have been simple, as it merely requires a change in the MLP and DeepSDF [3], which also works well with sinusoid activation functions [6]. Moreover, it has been comparatively less explored if dropout and other anomaly detection frameworks are useful in this context. Thus, for decoder-based uncertainty modeling, for example, in the context of [2], such an ablation study would be ultimately interesting and relevant.

(b) Experiments on more common (shape) datasets, especially with SDFs: The authors state that their method is relevant for both SDF and occupancy representations but only conducts experiments for occupancy networks. Given that the training setup would not change much, I believe it would have been interesting to show experiments in an established shape dataset as well (e.g. (Med-) ShapeNet). 

References:

[1] Dupont E, Loya H, Alizadeh M, Goliński A, Teh YW, Doucet A. Coin++: Neural compression across modalities. arXiv preprint arXiv:2201.12904. 2022 Jan 30.

[2] Mehta I, Gharbi M, Barnes C, Shechtman E, Ramamoorthi R, Chandraker M. Modulated periodic activations for generalizable local functional representations. InProceedings of the IEEE/CVF International Conference on Computer Vision 2021 (pp. 14214-14223).

[3] Park JJ, Florence P, Straub J, Newcombe R, Lovegrove S. Deepsdf: Learning continuous signed distance functions for shape representation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2019 (pp. 165-174).

[4] Mescheder L, Oechsle M, Niemeyer M, Nowozin S, Geiger A. Occupancy networks: Learning 3d reconstruction in function space. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2019 (pp. 4460-4470).

[5] Amiranashvili T, Lüdke D, Li HB, Menze B, Zachow S. Learning shape reconstruction from sparse measurements with neural implicit functions. InInternational Conference on Medical Imaging with Deep Learning 2022 Dec 4 (pp. 22-34). PMLR.

[6] Sitzmann V, Martel J, Bergman A, Lindell D, Wetzstein G. Implicit neural representations with periodic activation functions. Advances in neural information processing systems. 2020;33:7462-73.

[7] Tancik M, Srinivasan P, Mildenhall B, Fridovich-Keil S, Raghavan N, Singhal U, Ramamoorthi R, Barron J, Ng R. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems. 2020;33:7537-47.

[8] Müller T, Evans A, Schied C, Keller A. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG). 2022 Jul 22;41(4):1-5.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the weakness part.
soundness: 2
strengths: 1.  The topic of this work is really interesting. By utilizing an implicit shape model, points with sparse shapes can be densified with refined shape representations.
2.  This is the first work to model epistemic uncertainty in the implicit decoder during finetuning from synthetic data.
summary: In this work, authors introduce Dropsembles, a novel method for uncertainty estimation in tuned implicit functions. They demonstrate the efficacy of this approach through a series of experiments, starting with toy examples and progressing to a real-world scenario. Our results show that Dropsembles achieve the accuracy and calibration levels of deep ensembles but with significantly less computational cost.
weaknesses: 1.  The core concept of this work is uncertainty modeling. There are two innovative designs proposed in this paper. For the Elastic weight consolidation strategy, it seems that no remarkable improvements are done. Authors simply employ EWC to adapt pretrained models from dataset A. The other component is Dropsembles, please provide a more detailed description on its contribution.
2.  Since Dropsembles is aimed for the tradeoff between computational costs and accuracy. However, no quantitative results are given in the experimental part. Authors should carefully list out the comparison between dropsembles and other methods on uncertainty estimation.
3.  The improvement on lumbar spine seems to be marginal, maybe a detailed analysis is required.
4.  The effectiveness of this method is highly required for the evaluation on more challenging datasets, such as Medshapenet, etc.

Meta-review: The paper proposes a method called Dropsembles for uncertainty estimation for implicit neural representations. The method combines dropout with model ensembles with a learned prior by fine-tuning a pre-trained model using elastic weight consolidation. The reviewers commented on both the simplicity of the approach and the fact that it constituted a general-purpose approach not necessarily tailored to implicit neural representations. However, they appreciated the importance of the application and (especially after revision) the breadth of evaluation in terms of datasets. There was some concern raised about the absence of comparison to Bayesian neural networks.

The paper received two borderline accepts and one strong accept, with these reviewers all suggesting that the paper will be of interest from the ICLR community.

Decision: Accept (Poster)