SEMF: Supervised Expectation-Maximization Framework for Predicting Intervals

## Abstract

Abstract This work introduces the Supervised Expectation-Maximization Framework (SEMF), a versatile and model-agnostic approach for generating prediction intervals in datasets with complete or missing data. SEMF extends the Expectation-Maximization algorithm, traditionally used in unsupervised learning, to a supervised context, leveraging latent variable modeling for uncertainty estimation. Extensive empirical evaluations across 11 tabular datasets show that SEMF often achieves narrower normalized prediction intervals and higher coverage rates than traditional quantile regression methods 1 1 1 Code: https://github.com/Unco3892/semf-paper . Furthermore, SEMF can be integrated with machine learning models like gradient-boosted trees and neural networks, highlighting its practical applicability. The results indicate that SEMF enhances uncertainty quantification, particularly in scenarios with complete data.

## 1 Introduction

In the evolving field of machine learning (ML), the quest for models able to predict outcomes while quantifying the uncertainty of their predictions is critical. The ability to estimate prediction uncertainty is particularly vital in high-stakes domains such as healthcare [], finance [], and autonomous systems [], where prediction-based decisions have important consequences. Traditional approaches have primarily focused on point estimates, with little to no insight into prediction reliability. This limitation underscores the need for frameworks that can generate both precise point predictions and robust prediction intervals. Such intervals provide a range within which the true outcome is expected to lie with a fixed probability, offering a finer understanding of prediction uncertainty. This need has spurred research into methodologies that extend beyond point estimation to include uncertainty quantification, thereby enabling more informed decision-making in applications reliant on predictive modeling [].

In this paper, we introduce the Supervised Expectation-Maximization Framework (SEMF) based on the Expectation-Maximization (EM) algorithm []. Traditionally recognized as a clustering technique, EM is used for supervised learning in SEMF, allowing for both point estimates and prediction intervals using any ML model (model-agnostic). SEMF generates representations for latent or missing modalities, which can be relevant for incomplete data and holds potential for multi-modal data applications, though multi-modal settings are left for future exploration. This paper details the methodology behind the framework and proposes a training algorithm based on Monte Carlo (MC) sampling, also used in variational inference for Variational Auto-Encoders (VAEs) [, ]. SEMF differs from prominent supervised EM approaches such as [], which focus on point prediction using Gaussian Mixture Models (GMMs). Additionally, our method operates in a frequentist paradigm, directly maximizing the likelihood function through iterative EM steps without integrating over posterior distributions. Although SEMF can be extended to a Bayesian framework as its likelihood component, this extension lies beyond the scope of this paper.

The remainder of this paper is organized as follows: Section 2 details the theory and the methodology of SEMF. Section 3 reviews related works in latent representation learning, uncertainty estimation, and handling of missing data. Section 4 describes the experimental setup, including datasets and evaluation metrics. Section 5 discusses the results, demonstrating the efficacy of SEMF. Lastly, Section 6 concludes the paper, and Section 7 outlines the limitations and potential research directions.

## 2 Method

This section presents the founding principles of SEMF from its parameters, training, and inference procedure with, at its core, the EM algorithm. This algorithm, first introduced by [], is an unsupervised method for handling latent variables and incomplete data. Invented to maximize the model likelihood, it builds a sequence of parameters that guarantee an increase in the log-likelihood [] by iterating between the Expectation (E) and the Maximization (M) steps. In the E-step, one computes

$$ $Q(p|p^{\prime})=\mathbb{E}_{Z\sim p^{\prime}(z|x)}\left[\log p(x,Z)\right]= \int\log p(x,z)p^{\prime}(z|x)dz,$ (1) $$

where $p^{\prime}$ stands for the current estimates, $\log p(x,z)$ is the log-likelihood of the complete observation $(x,z)$, and $z$ is a latent variable. The M-step maximizes this $Q$-function: $p^{\prime}\leftarrow\arg\max_{p}Q(p|p^{\prime})$ . The sequence is repeated until convergence.

### 2.1 Problem Scenario

Let $x=(x_{1},x_{2},\ldots,x_{K})$ denote $K$ inputs and the output be $y$. For simplicity, we limit $y$ to be numerical, although it could be categorical without loss of generality. Component $x_{k}$ is a source: a modality, a single or group of variables, or an unstructured input such as an image or text. For clarity, we limit to $K=2$, where $x_{1}$ and $x_{2}$ are single variables. We assume that only $x_{1}$ may contain missing values, either at random or partially at random.

Let $p(y|x)$ be the density function of the outcome given the inputs (the fact that $y$ is continuous can be easily relaxed). A founding assumption, in the spirit of VAE, is that $p(y|x)$ decomposes into $p(y|x)=\int p(y|z)p(z_{1}|x_{1})p(z_{2}|x_{2})dz_{1}dz_{2}$, where $z=(z_{1},z_{2})$ are unobserved latent variables. We assume that $p(y|z,x)=p(y|z)$, that is, $z$ contains all the information of $x$ about $y$, and that $p(z|x)=p(z_{1}|x_{1})p(z_{2}|x_{2})$, that is, there is one latent variable per source. These are independent conditionally on their corresponding source. Finally, if $x_{1}$ is missing, then $p(y|x_{2})=\int p(y|z)p(z_{1}|x_{1})p(x_{1}|x_{2})p(z_{2}|x_{2})dx_{1}dz_{1}dz
_{2}$.
The contribution to the log-likelihood of a complete observation $(y,z,x)$ is
$\log p(y,z|x)=\log p(y|z)+\log p(z|x)$. In the E-step, we compute

$$ $\displaystyle\int\log p(y,z|x)p^{\prime}(z|y,x)dz$ $\displaystyle=$ $\displaystyle\int\log p(y|z)p^{\prime}(z|y,x)dz+\int\log p(z|x)p^{\prime}(z|y, x)dz.$ (2) $$

where $p^{\prime}$ is our current estimate. Eq. 2 can be estimated by MC sampling. Since sampling from $p^{\prime}(z|y,x)$ can be inefficient, we rather rely on the decomposition $p^{\prime}(z|y,x)=p^{\prime}(y|z)p^{\prime}(z|x)/p^{\prime}(y|x)$. Thus, we sample $z_{r}$ from $p^{\prime}(z|x)$, $r=1,\ldots,R$, and, setting $w_{r}=p^{\prime}(y|z_{r})/\sum_{t}p^{\prime}(y|z_{t})$, approximate the right-hand side term of Eq. 2

$$ $\displaystyle\int\log p(y,z|x)p^{\prime}(z|y,x)dz$ $\displaystyle\approx$ $\displaystyle\sum_{r=1}^{R}\left\{\log p(y|z_{r})+\log p(z_{r}|x)\right\}w_{r}.$ (3) $$

If $x_{1}$ is missing, a similar development leads to

$$ $\displaystyle\int\log p(y,z,x_{1}|x_{2})p^{\prime}(z,x_{1}|y)dz$ $\displaystyle\approx$ $\displaystyle\sum_{r=1}^{R}\left\{\log p(y|z_{r})+\log p(z_{r}|x)+\log p(x_{1, r}|x_{2})\right\}w_{r},\quad$ (4) $$

where $x_{1,r}$ and $z_{r}$ are respectively sampled from $p^{\prime}(x_{1}|x_{2})$ and $p^{\prime}(z|x_{1,r},x_{2})$.

### 2.2 Objective Function

Adapting Eq. 3 and Eq. 4 for the observed data $\{(y_{i},x_{i})\}_{i=1}^{N}$, the overall loss function, $\mathcal{L}$, is

$$ $\mathcal{L}(\phi,\theta,\xi)=-\sum_{i=1}^{N}\sum_{r=1}^{R}\left\{\log p_{\phi} (z_{i,r}|x_{i,r})+\log p_{\theta}(y_{i}|z_{i,r})+\mathds{1}_{\{i\in I_{m}\}} \log p_{\xi}(x_{1,i,r}|x_{2,i})\right\}w_{i,r},$ (5) $$

where $I_{m}$ is the set of those $i$’s such that $x_{1,i}$ is missing. The models of $p(y|z)$, $p(z|x)$, and $p(x_{1}|x_{2})$ inherit parameters $\theta$, $\phi$, and $\xi$, respectively. Also, $x_{1,i,r}$ is sampled from $p_{\xi^{\prime}}(x_{1}|x_{2,i})$, if $x_{1,i}$ is missing, and $z_{1,i,r}$ and $z_{2,i,r}$ are sampled from $p_{\phi^{\prime}}(z_{1}|x_{1,i,r})$ and $p_{\phi^{\prime}}(z_{2}|x_{2,i})$. Furthermore, for compactness of notation, $x_{i,r}$ is $(x_{1,i},x_{2,i})$ if $x_{1,i}$ is observed, and $(x_{1,i,r},x_{2,i})$ if $x_{1,i,r}$ is missing. Finally, the weights are

$$ $w_{i,r}=\frac{p_{\theta^{\prime}}(y_{i}|z_{i,r})}{\sum_{t=1}^{R}p_{\theta^{ \prime}}(y_{i}|z_{i,t})}.$ (6) $$

Eq. 5 shows that ${\cal L}$ is a sum of losses associated with the encoder model, $p_{\phi}$, for each source, the decoder model, $p_{\theta}$, from the latent variables to the output, and, if applicable, the model handling missing data, $p_{\xi}$. At each M-step, ${\cal L}$ is minimized with respect to $\theta$, $\phi$, and $\xi$. Then, $\theta^{\prime}$, $\phi^{\prime}$, and $\xi^{\prime}$ are updated, as well as the weights and the sampling. Then the process is iterated until convergence.

#### 2.2.1 Example: ℒ ⁢ ( ϕ , θ , ξ ) ℒ italic-ϕ 𝜃 𝜉 \mathcal{L}(\phi,\theta,\xi) caligraphic_L ( italic_ϕ , italic_θ , italic_ξ ) under normality

Similar to [], we develop further ${\cal L}$ under normality assumptions for the encoder, $p_{\phi}$, and the decoder, $p_{\theta}$. These simple cases are illustrative, though any other distributions could be adopted, including non-continuous or non-numerical outcomes.

##### Encoder p ϕ ⁢ ( z | x ) subscript 𝑝 italic-ϕ conditional 𝑧 𝑥 p_{\phi}(z|x) italic_p start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( italic_z | italic_x ) .

Let $m_{k}$ be the length of the latent variable $z_{k}$, $k=1,2$. We assume a normal model for $Z_{k}$ given $X_{k}=x_{k}$,

$$ $Z_{k}|X_{k}=x_{k}\sim{\cal N}_{m_{k}}(g_{\phi_{k}}(x_{k}),\sigma_{k}^{2}J_{m_{ k}}),$ (7) $$

where $J_{m_{k}}$ is the $m_{k}\times m_{k}$ identity matrix. In particular,

$$ $\displaystyle\log p_{\phi}(z_{k}|x_{k})$ $\displaystyle=$ $\displaystyle-\frac{m_{k}}{2}\log 2\pi-\frac{1}{2}\log\sigma_{k}^{2}-\frac{1}{ 2\sigma_{k}^{2}}\sum_{j=1}^{m_{k}}\{z_{k,j}-g_{\phi_{k},j}(x_{k})\}^{2},\quad k =1,2.$ (8) $$

The mean $g_{\phi_{k}}(x_{k})$ can be any model, such as a neural network, with output of length $m_{k}$, $k=1,2$. The scale $\sigma_{k}$ can be fixed, computed via the weighted residuals, or learned through a separate set of models. It controls the amount of noise introduced in the latent dimension and is pivotal in determining the prediction interval width for $p(y|z)$. In this paper, $\sigma_{k}$ is fixed for simplicity.

##### Decoder p θ ⁢ ( y | z ) subscript 𝑝 𝜃 conditional 𝑦 𝑧 p_{\theta}(y|z) italic_p start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_y | italic_z ) .

We assume a normal model for $Y$ given $Z=z$,

$$ $Y|Z=z\sim\mathcal{N}(f_{\theta}(z),\sigma^{2}).$ (9) $$

This results in a log-likelihood contribution,

$$ $\log p_{\theta}(y|z)=-\frac{1}{2}\log 2\pi-\frac{1}{2}\log\sigma^{2}-\frac{1}{ 2\sigma^{2}}\{y-f_{\theta}(z)\}^{2}.$ (10) $$

Again, the mean $f_{\theta}(z)$ can be any model, such as a neural network.

##### Model for missing data p ξ ⁢ ( x 1 | x 2 ) subscript 𝑝 𝜉 conditional subscript 𝑥 1 subscript 𝑥 2 p_{\xi}(x_{1}|x_{2}) italic_p start_POSTSUBSCRIPT italic_ξ end_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) .

We use an empirical model for $X_{1}$ given $X_{2}=x_{2}$ where $p_{\xi}(x_{1}|x_{2})$ put masses only on those non-missing $x_{1}$’s in the training set. Let $I_{nm}$ be the set of indices such that $x_{1,j}$, $j\in I_{nm}$, are all the non-missing $x_{1}$ in the training set. Additionally, for a given $j\in I_{nm}$, $x_{1}[j]$ is the observed $x_{1}$ corresponding to $j$. For a given $x_{2}$, $p_{\xi}(x_{2})$ is a vector of length $|I_{nm}|$, the cardinality of $I_{nm}$, with components

$$ $p_{\xi}(j|x_{2})=\frac{\exp\{h_{\xi,j}(x_{2})\}}{\sum_{t\in I_{nm}}\exp\{h_{ \xi,t}(x_{2})\}},\quad j\in I_{nm},$ (11) $$

where $h_{\xi}$ is a vector of length $|I_{nm}|$, typically a neural network with input $x_{2}$ and output on $I_{nm}$. Now, the probability of $X_{1}=x_{1}$ given $X_{2}=x_{2}$ is

$$ $p_{\xi}(x_{1}|x_{2})=\sum_{j\in I_{nm}}p_{\xi}(j|x_{2})\cdot\mathds{1}\{x_{1}= x_{1}^{(nm)}[j]\}.$ (12) $$

##### Summary.

Overall, the M-step is

$$ $\phi_{k}^{*}=\arg\min_{\phi_{k}}\sum_{i,r}w_{i,r}\sum_{j=1}^{m_{k}}\{z_{k,i,r, j}-g_{\phi_{k},j}(x_{k,i,r})\}^{2},\quad k=1,2,$ (13) $$

$$ $\theta^{*}=\arg\min_{\theta}\sum_{i,r}w_{i,r}\{y_{i}-f_{\theta}(z_{i,r})\}^{2},$ (14) $$

$$ $(\sigma^{*})^{2}=\frac{1}{N}\sum_{i,r}w_{i,r}\{y_{i}-f_{\theta^{*}}(z_{i,r})\} ^{2},$ (15) $$

$$ $\xi^{*}=\arg\max_{\xi}\sum_{t\in D_{x}}w_{t}\log p_{\xi}(j_{t}|x_{2,t}).$ (16) $$

When $x_{1,i}$ is missing, the sampling is enriched by simulated $x_{1,i,r}$. Eq. 16 selects $j^{*}$ from $I_{nm}$ based on $p_{\xi^{\prime}}(j|x_{2})$ according to Eq. 12 and Eq. 11. The $D_{x}$ above is a subset of the data for $\xi$, learned on the missing data part. We also note that learning the parameters above is parallelizable.

### 2.3 Training

For efficiency purpose, the training set, $\{1,\ldots,N\}$, is segmented into batches $\{b_{1},\ldots,b_{L}\}$ on which the index $i$ runs (and thus the denominator of Eq. 15 must be adapted accordingly). The process iterates for each batch until the maximum number of steps is reached or an early stopping criterion is satisfied. The full details are given in Algorithm 1 (Appendix A). The framework requires tuning hyper-parameters such as the number of MC samples $R$, the number of latent nodes $m_{k}$, and the standard deviation $\sigma_{k}$ of $Z_{k}$. Monitoring the point prediction on a hold-out validation is important to combat overfitting and terminate the training early with a patience hyper-parameter. Moreover, due to the generative nature of SEMF, the variation resulting from the initial random seed is measured in Subsection 4.2. Additionally. the model-specific hyper-parameters ($p_{\phi}$, $p_{\theta}$ and $p_{\xi}$) are also discussed in the same Subsection.

### 2.4 Inference

The encoder-decoder structure of SEMF entails the simulations of $z_{r}$ during inference, as depicted in Figure 1. In theory, any inference can be performed for $\hat{y}$, for instance the mean value $\hat{y}=\frac{1}{R}\sum_{r=1}^{R}f_{\theta}(z_{r})$, where $z_{r}\sim p_{\phi}(z|x)$ (see Algorithm 2 in Appendix A). For prediction intervals, a double simulation scheme is used,

$$ $z_{r}\sim p(z|x),\quad\hat{y}_{r,s}\sim p_{\theta}(y|z_{r}),\quad r,s=1,\ldots ,R.$ (17) $$

Prediction interval at a given level of certainty $\alpha$ follows as

$$ $PI=\text{quantile}\left(\{\hat{y}_{r,s}\};\frac{\alpha}{2},1-\frac{\alpha}{2} \right).$ (18) $$

##### Remark.

We denote the $R$ for inference as $R_{\mathrm{infer}}$.

Figure: Figure 1: Inference procedure with the SEMF’s learnable parameters $\phi_{k},\theta$ and $\xi$. Here, we illustrate the number of inputs $k$ as $k=1,2$, assuming that $x_{1}$ may contain missing values
Refer to caption: x1.png

## 3 Related Work

### 3.1 Latent Representation Learning

Latent representation learning involves modeling hidden variables from observed data for various ML tasks, most notably Auto-Encoders (AEs) and VAEs. AEs are neural networks that reconstruct inputs by learning an intermediate latent representation. The encoder $g_{\phi}(\cdot)$ in an AE embeds the input $x$ into a latent variable $z$, which then passes through the decoder $f_{\theta}(\cdot)$, reconstructing $x$ as $\hat{x}=f_{\theta}(z)$. The training process optimizes the parameters $\phi$ and $\theta$ by minimizing the reconstruction loss. Unlike AEs, which focus on reconstruction, VAEs use variational methods to fit distributions of latent variables and the output []. Given a sample $x$ with a latent variable $z$, VAEs model the marginal likelihood $p_{\theta}(x)$. However, directly maximizing this likelihood is difficult []. Thus, alternative methods exist, such as maximizing the evidence lower bound (ELBO), which provides guarantees on the log-likelihood [].

For supervised and semi-supervised tasks, latent representation learning can include task-specific predictions []. More specifically, models such as AEs follow the classical encoder-decoder objective while training a predictor $h_{\psi}(z)$ through an additional layer or model to estimate the output $y$. This dual objective helps in learning more task-relevant embeddings [, ]. Semi-supervised VAEs are similar, with the distinction that they couple the reconstruction loss of the unlabeled data with a variational approximation of latent variables. This is effective even with sparse labels [, ].

The EM algorithm has already been used for supervised learning tasks using specific models [, , ], where the goal has been point prediction with GMMs. Similarly, the EM algorithm adapts well to minimal supervision [] and using labeled and unlabeled data in semi-supervised settings for both single and multiple modalities [, ]. Our work differs by modifying using MC sampling to generate prediction intervals with any ML model and, in theory, under any distribution.

### 3.2 Prediction Intervals

Crucial for estimating uncertainty, prediction intervals in regression are often derived using methods such as Bayesian approaches [, , ], ensemble techniques [, , ], or quantile regression [, ]. Additionally, these methods can be complemented with conformal prediction, a framework for calibrating any point predictor to produce prediction intervals [, ], making it highly relevant for enhancing reliability in applications requiring rigorous uncertainty quantification.

A key component of quantile regression is the pinball loss function, which effectively balances the residuals to capture the desired quantiles even for non-parametric models [], making it ideal for asymmetric distributions where tail behavior is of critical importance []. This loss function is pivotal not only for single model scenarios but also enhances ensemble methods by refining their quantile estimate []. Conformal prediction further extends the applicability of these intervals by providing a layer of calibration that adjusts intervals obtained from any predictive model, ensuring they cover the true value with a pre-specified probability [].

### 3.3 Missing Data

Managing missing values is a pivotal aspect when dealing with real-world data. Naive methods such as discarding instances or mean/median-imputation may be infeasible or carry the risk of changing the data distribution [, ]. The chosen technique for handling missing data should adhere to the dataset’s characteristics and mechanisms behind the missing data []. More advanced approaches, like the Iterative Imputer from scikit-learn [], an implementation of Multiple Imputation by Chained Equations (MICE) [], expand on the simple imputations by iteratively modeling each feature with missing values as a function of other features [, ]. Due to its complexity, MICE is best compared with alternatives such as K-means clustering and artificial neural networks (ANNs). K-means assigns missing values based on cluster centroids [] as opposed to ANNs, which learn complex, non-linear relationships between variables []. ANNs effectively predict or reconstruct missing values and thus are particularly useful in datasets where relationships between variables are intricate and not easily captured by straightforward imputation methods. Accordingly, ANNs have demonstrated superior performance over MICE and GMM (with EM for missing values) in scenarios where a large proportion of the data is missing [].

## 4 Experimental Setup

### 4.1 Datasets

We systematically curate a subset of datasets from the OpenML-CTR23 [] benchmark suite to evaluate and carry out our experiments. Initially comprising 35 datasets, we apply an exclusion criteria to refine this collection to 11 datasets. The details and overview are in Appendix B. We remove duplicated rows from all the datasets and carry out the standardization (scaling) of all predictors, including the outcome, which we transformed to have zero means and unit variances. The features of these datasets are then treated as separate inputs to SEMF. In the second stage of our experiments, we artificially introduce 50% missing values in our datasets for any predictor except for the first feature of a randomly chosen row, which emulates missing completely at random (MCAR) data. In all our datasets, 70% of the data is used to train all models, 15% as a hold-out validation set to monitor SEMF’s performance, and 15% to evaluate the models. To combat overfitting, baseline models that benefit from early stopping are allocated another 15% from the training data. Lastly, it is essential to note that all data in SEMF are processed batch-wise, without employing mini-batch training, to ensure consistency and stability in the training process.

### 4.2 Models

Our baseline consists of both point and quantile regression eXtreme Gradient Boosting (XGBoost) [], Extremely Randomized Trees (ET) [], and neural networks [], all summarized and depicted in Table 1. To ensure consistency in our experimental setup, we align the families and hyper-parameters of $p_{\phi}$ and $p_{\theta}$ with our baseline models. For example, in the case of XGBoost in SEMF, we use $K$ XGBoosts, $g_{\phi_{k}}(x_{k})$, one for each input $x_{k}$, $k=1,\ldots,K$, and one XGBoost for $f_{\theta}(z)$ with the same hyper-parameters. We refer to the SEMF’s adoption of these models as MultiXGBs, MultiETs, and MultiMLPs. When establishing prediction intervals, we conformalize our prediction intervals according to [] at an uncertainty tolerance of 5% for both the baseline and SEMF (Eq. 18). The missing data simulator, $p_{\xi}$, is constructed using a shallow neural network, which employs the $\mathrm{SELU}$ activation function []. It is experimented with two distinct node counts: 50 and 100.

To constrain the breadth of our parameter exploration, the simulator for the missing model adopts the optimal set of hyper-parameters identified from analyses involving complete datasets. We target (larger) $\sigma_{k}$ values that introduce more noise and produce better intervals than point predictions. The optimal models are then trained and tested with five different seeds, and the results are averaged. The point prediction, $\hat{y}$, uses the mean inferred values. We then study the performance of SEMF against mean and median imputation techniques, five nearest neighbors, and MICE from []. The imputers are used within the point and interval baseline models explained in the following subsection to form the missing value baseline. Appendix C contains more details on the hyper-parameters for each SEMF model and dataset.

**Table 1: SEMF models, baselines, and hyper-parameters.**
| SEMF | Point Prediction Baseline | Interval Prediction Baseline |
| --- | --- | --- |
| MultiXGBs | XGBoost []<br>Trees: 100, Maximum depth: 6, Early stopping steps: 10 | Quantile XGBoost<br>Same as point prediction baseline, XGBoost |
| MultiETs | Extremely Randomized Trees []<br>Trees: 100, Maximum depth: 10 | Quantile Extremely Randomized Trees []<br>Same as point prediction baseline, Extremely Randomized Trees |
| MultiMLPs | Deep Neural Network<br>Hidden layers: 2, Nodes per layer: 100, Activation functions: ReLU, Epochs: 1000 or 5000, Learning rate: 0.001, Batch training, Early stopping steps: 100 | Simultaneous Quantile Regression []<br>Same as point prediction baseline, Deep Neural Network |

### 4.3 Metrics

The evaluation of point predictions employs Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and R-squared ($\text{R}^{2}$). For prediction intervals, the chosen metrics, following [] and [], are the Prediction Interval Coverage Probability (PICP), and Normalized Mean Prediction Interval Width (NMPIW) as described in Subsection D.1. This paper also evaluates SEMF on our new metric, termed Coverage-Width Ratio (CWR),

$$ $\text{CWR}=\frac{\text{PICP}}{\text{NMPIW}},$ (19) $$

which evaluates the coverage probability ratio to the prediction interval’s width. CWR provides a refined understanding of the balance between an interval’s accuracy (coverage) and precision (width). Though a larger value of this metric is better in higher confidence levels, the marginal increase in NMPIW is likely higher than that of PICP, resulting in decreasing CWR.

In our case, measuring the performance of SEMF over the baseline models is far more critical than reviewing absolute metrics in isolation. For any metric above, except for ($\text{R}^{2}$), this is computed as

$$ $\text{Metric}_{\Delta}(\%)=\left(\frac{\text{Metric}_{\text{SEMF}}-\text{ Metric}_{\text{Baseline}}}{\text{Metric}_{\text{Baseline}}}\right)\times 100,$ (20) $$

on which we base our decisions for selecting the best hyper-parameters as explained in Subsection D.2.

## 5 Results

We trained and tested 330 models corresponding to the three model types—MultiXGBs, MultiETs, and MultiMLPs—across 11 datasets with both complete and 50% missing data, using five seeds for each combination. Table 2 and Table 3 present the mean and standard deviation for our metrics aggregated over the five seeds. Appendix E includes the results from each individual run. For comparison, we also present the non-conformalized prediction intervals in Appendix F, though we solely discuss the results for conformalized intervals on both complete and incomplete data.

### 5.1 Complete Data

The results of models with complete data are in Table 2. Overall, MultiXGBs and MultiMLPs performed generally well in producing intervals compared to their baselines, as shown by positive $\Delta$CWR and $\Delta$NMPIW while achieving similar $\Delta$PICP to the baselines. Notably, all models attained suitable intervals on naval_propulsion_plant and energy_efficiency. Interestingly, MultiMLPs also attained good relative performance improvements for point prediction, which can indicate that the chosen $\sigma_{k}$ was too low for generating performant prediction intervals. MultiETs attained mixed results, with significant improvements in $\Delta$CWR for other datasets such as naval_propulsion_plant, but its overall performance remains less conclusive.

**Table 2: Test results for all models with complete data at 95% quantiles aggregated over five seeds. For each metric, the mean and standard deviation of the performance across the seeds are separated by $\pm$. Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| MultiXGBs |  |  |  |  |  |  |  |  |
| space_ga | 6%$\pm$3% | -1%$\pm$0% | 7%$\pm$2% | 0.94$\pm$0.01 | 0.26$\pm$0.01 | -9%$\pm$1% | -10%$\pm$2% | 0.60$\pm$0.01 |
| cpu_activity | 16%$\pm$6% | 1%$\pm$1% | 12%$\pm$5% | 0.94$\pm$0.01 | 0.09$\pm$0.00 | 21%$\pm$1% | -1%$\pm$2% | 0.98$\pm$0.00 |
| naval_propulsion_plant | 172%$\pm$14% | 0%$\pm$1% | 63%$\pm$2% | 0.95$\pm$0.01 | 0.11$\pm$0.00 | -14%$\pm$4% | -12%$\pm$2% | 0.99$\pm$0.00 |
| miami_housing | -7%$\pm$3% | 0%$\pm$0% | -8%$\pm$3% | 0.95$\pm$0.00 | 0.13$\pm$0.00 | 4%$\pm$4% | 3%$\pm$3% | 0.91$\pm$0.01 |
| kin8nm | 6%$\pm$2% | 1%$\pm$0% | 5%$\pm$2% | 0.94$\pm$0.01 | 0.45$\pm$0.01 | -20%$\pm$1% | -22%$\pm$1% | 0.63$\pm$0.01 |
| concrete_compressive_strength | 41%$\pm$12% | -3%$\pm$2% | 31%$\pm$7% | 0.94$\pm$0.02 | 0.31$\pm$0.01 | -26%$\pm$6% | -40%$\pm$11% | 0.85$\pm$0.01 |
| cars | 40%$\pm$18% | -3%$\pm$1% | 30%$\pm$8% | 0.91$\pm$0.01 | 0.13$\pm$0.01 | -3%$\pm$3% | -1%$\pm$3% | 0.95$\pm$0.00 |
| energy_efficiency | 222%$\pm$45% | -4%$\pm$3% | 70%$\pm$3% | 0.92$\pm$0.02 | 0.05$\pm$0.01 | -15%$\pm$22% | -16%$\pm$17% | 1.00$\pm$0.00 |
| california_housing | -1%$\pm$3% | 0%$\pm$0% | -2%$\pm$4% | 0.95$\pm$0.00 | 0.42$\pm$0.01 | -1%$\pm$1% | -1%$\pm$1% | 0.81$\pm$0.00 |
| airfoil_self_noise | 21%$\pm$21% | -1%$\pm$2% | 15%$\pm$15% | 0.97$\pm$0.02 | 0.36$\pm$0.06 | -64%$\pm$32% | -73%$\pm$41% | 0.86$\pm$0.05 |
| QSAR_fish_toxicity | 34%$\pm$8% | -5%$\pm$3% | 29%$\pm$5% | 0.87$\pm$0.02 | 0.33$\pm$0.01 | 3%$\pm$3% | 1%$\pm$4% | 0.55$\pm$0.02 |
| MultiETs |  |  |  |  |  |  |  |  |
| space_ga | -6%$\pm$3% | 1%$\pm$1% | -7%$\pm$3% | 0.96$\pm$0.00 | 0.29$\pm$0.01 | -15%$\pm$2% | -18%$\pm$3% | 0.54$\pm$0.02 |
| cpu_activity | 11%$\pm$2% | -4%$\pm$0% | 13%$\pm$2% | 0.94$\pm$0.00 | 0.11$\pm$0.00 | -14%$\pm$1% | -20%$\pm$2% | 0.98$\pm$0.00 |
| naval_propulsion_plant | 137%$\pm$22% | 1%$\pm$0% | 57%$\pm$4% | 0.96$\pm$0.00 | 0.27$\pm$0.02 | -316%$\pm$45% | -406%$\pm$43% | 0.96$\pm$0.01 |
| miami_housing | -9%$\pm$1% | -1%$\pm$0% | -9%$\pm$1% | 0.95$\pm$0.00 | 0.15$\pm$0.00 | -10%$\pm$2% | -20%$\pm$2% | 0.90$\pm$0.00 |
| kin8nm | -10%$\pm$1% | 1%$\pm$1% | -12%$\pm$2% | 0.94$\pm$0.01 | 0.53$\pm$0.01 | -34%$\pm$2% | -38%$\pm$2% | 0.48$\pm$0.01 |
| concrete_compressive_strength | -8%$\pm$5% | -3%$\pm$2% | -6%$\pm$6% | 0.89$\pm$0.02 | 0.31$\pm$0.02 | -67%$\pm$6% | -94%$\pm$8% | 0.76$\pm$0.01 |
| cars | -25%$\pm$5% | 0%$\pm$3% | -34%$\pm$14% | 0.92$\pm$0.02 | 0.15$\pm$0.01 | 3%$\pm$4% | 1%$\pm$2% | 0.95$\pm$0.00 |
| energy_efficiency | 10%$\pm$6% | -4%$\pm$2% | 12%$\pm$6% | 0.94$\pm$0.02 | 0.06$\pm$0.00 | 3%$\pm$2% | 1%$\pm$1% | 1.00$\pm$0.00 |
| california_housing | 1%$\pm$2% | -2%$\pm$0% | 3%$\pm$2% | 0.95$\pm$0.00 | 0.55$\pm$0.01 | -15%$\pm$1% | -21%$\pm$1% | 0.71$\pm$0.01 |
| airfoil_self_noise | -16%$\pm$10% | -3%$\pm$2% | -17%$\pm$12% | 0.96$\pm$0.02 | 0.43$\pm$0.03 | -118%$\pm$43% | -141%$\pm$49% | 0.80$\pm$0.08 |
| QSAR_fish_toxicity | -6%$\pm$9% | -5%$\pm$2% | -2%$\pm$12% | 0.88$\pm$0.02 | 0.36$\pm$0.02 | -6%$\pm$1% | -11%$\pm$1% | 0.53$\pm$0.01 |
| MultiMLPs |  |  |  |  |  |  |  |  |
| space_ga | 6%$\pm$3% | -1%$\pm$1% | 6%$\pm$2% | 0.95$\pm$0.01 | 0.23$\pm$0.00 | 0%$\pm$1% | -1%$\pm$1% | 0.75$\pm$0.01 |
| cpu_activity | -7%$\pm$6% | 0%$\pm$1% | -9%$\pm$7% | 0.95$\pm$0.00 | 0.10$\pm$0.00 | 7%$\pm$2% | 5%$\pm$2% | 0.98$\pm$0.00 |
| naval_propulsion_plant | 4%$\pm$16% | 1%$\pm$0% | 0%$\pm$14% | 0.96$\pm$0.00 | 0.08$\pm$0.01 | -45%$\pm$25% | -36%$\pm$20% | 1.00$\pm$0.00 |
| miami_housing | -38%$\pm$3% | 0%$\pm$1% | -61%$\pm$9% | 0.95$\pm$0.00 | 0.15$\pm$0.01 | -3%$\pm$2% | 4%$\pm$2% | 0.91$\pm$0.00 |
| kin8nm | 8%$\pm$5% | 0%$\pm$1% | 8%$\pm$5% | 0.95$\pm$0.01 | 0.20$\pm$0.01 | 7%$\pm$2% | 7%$\pm$2% | 0.93$\pm$0.00 |
| concrete_compressive_strength | 11%$\pm$6% | -2%$\pm$2% | 11%$\pm$6% | 0.94$\pm$0.02 | 0.29$\pm$0.03 | 12%$\pm$5% | 15%$\pm$4% | 0.91$\pm$0.01 |
| cars | 3%$\pm$12% | -3%$\pm$3% | 4%$\pm$11% | 0.92$\pm$0.02 | 0.13$\pm$0.01 | -3%$\pm$3% | 0%$\pm$3% | 0.95$\pm$0.00 |
| energy_efficiency | 53%$\pm$21% | 0%$\pm$4% | 34%$\pm$10% | 0.96$\pm$0.02 | 0.05$\pm$0.00 | 32%$\pm$3% | 33%$\pm$3% | 1.00$\pm$0.00 |
| california_housing | -12%$\pm$3% | 0%$\pm$1% | -14%$\pm$4% | 0.95$\pm$0.00 | 0.43$\pm$0.01 | 7%$\pm$0% | 9%$\pm$1% | 0.82$\pm$0.00 |
| airfoil_self_noise | 68%$\pm$28% | -2%$\pm$1% | 40%$\pm$11% | 0.97$\pm$0.01 | 0.18$\pm$0.01 | 18%$\pm$4% | 16%$\pm$5% | 0.97$\pm$0.00 |
| QSAR_fish_toxicity | 5%$\pm$10% | 1%$\pm$1% | 3%$\pm$10% | 0.89$\pm$0.03 | 0.35$\pm$0.04 | 4%$\pm$5% | 7%$\pm$4% | 0.55$\pm$0.04 |

### 5.2 Missing Data

The results of models with 50% missing data are in Table 3. Note that relative metrics for SEMF are compared with the best result from any baseline imputer on that metric, regardless of how the imputer performed on the other metrics. Overall, the results are worse and less consistent than with complete data. MultiXGBs maintained good performance on some datasets, such as naval_propulsion_plant and energy_efficiency, but declined on others, like cpu_activity. MultiETs continued to exhibit mixed results, with only naval_propulsion_plant offering marginally better performance over the baseline. Similarly, MultiMLPs worked well on only one dataset, namely concrete_compressive_strength, while the other datasets had increased model uncertainty.

**Table 3: Test results for all models with missing data at 95% quantiles aggregated over five seeds. For each metric, the mean and standard deviation of the performance across the seeds are separated by $\pm$. Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| MultiXGBs |  |  |  |  |  |  |  |  |
| space_ga | 0%$\pm$3% | -2%$\pm$1% | 2%$\pm$3% | 0.94$\pm$0.01 | 0.32$\pm$0.01 | -7%$\pm$4% | -9%$\pm$3% | 0.45$\pm$0.06 |
| cpu_activity | -14%$\pm$9% | 0%$\pm$2% | -20%$\pm$14% | 0.94$\pm$0.03 | 0.18$\pm$0.02 | -1%$\pm$21% | -17%$\pm$13% | 0.83$\pm$0.07 |
| naval_propulsion_plant | 18%$\pm$23% | -3%$\pm$2% | 12%$\pm$18% | 0.93$\pm$0.04 | 0.59$\pm$0.17 | -14%$\pm$20% | -29%$\pm$35% | 0.77$\pm$0.05 |
| miami_housing | -18%$\pm$13% | -1%$\pm$1% | -25%$\pm$22% | 0.94$\pm$0.01 | 0.20$\pm$0.02 | -14%$\pm$11% | -18%$\pm$10% | 0.72$\pm$0.10 |
| kin8nm | -9%$\pm$3% | 1%$\pm$1% | -13%$\pm$5% | 0.96$\pm$0.01 | 0.63$\pm$0.03 | -3%$\pm$3% | -4%$\pm$4% | 0.40$\pm$0.02 |
| concrete_compressive_strength | 1%$\pm$9% | -2%$\pm$2% | 1%$\pm$10% | 0.95$\pm$0.01 | 0.57$\pm$0.02 | -16%$\pm$12% | -20%$\pm$14% | 0.58$\pm$0.08 |
| cars | 9%$\pm$27% | -7%$\pm$4% | 6%$\pm$23% | 0.90$\pm$0.04 | 0.34$\pm$0.02 | -30%$\pm$22% | -35%$\pm$24% | 0.66$\pm$0.15 |
| energy_efficiency | 26%$\pm$24% | -4%$\pm$4% | 18%$\pm$18% | 0.95$\pm$0.04 | 0.27$\pm$0.05 | -186%$\pm$265% | -153%$\pm$188% | 0.91$\pm$0.08 |
| california_housing | -6%$\pm$6% | -1%$\pm$1% | -8%$\pm$8% | 0.95$\pm$0.02 | 0.59$\pm$0.03 | -4%$\pm$4% | -5%$\pm$5% | 0.69$\pm$0.05 |
| airfoil_self_noise | -24%$\pm$6% | -1%$\pm$2% | -32%$\pm$12% | 0.95$\pm$0.03 | 0.76$\pm$0.07 | -37%$\pm$31% | -43%$\pm$36% | 0.41$\pm$0.12 |
| QSAR_fish_toxicity | 0%$\pm$8% | -4%$\pm$5% | 0%$\pm$11% | 0.91$\pm$0.05 | 0.50$\pm$0.11 | -4%$\pm$4% | -2%$\pm$3% | 0.36$\pm$0.04 |
| MultiETs |  |  |  |  |  |  |  |  |
| space_ga | -11%$\pm$2% | 0%$\pm$2% | -14%$\pm$5% | 0.95$\pm$0.01 | 0.34$\pm$0.02 | -12%$\pm$4% | -13%$\pm$5% | 0.40$\pm$0.06 |
| cpu_activity | -15%$\pm$8% | -3%$\pm$2% | -18%$\pm$13% | 0.94$\pm$0.02 | 0.20$\pm$0.02 | -21%$\pm$18% | -44%$\pm$14% | 0.82$\pm$0.07 |
| naval_propulsion_plant | 4%$\pm$16% | -4%$\pm$3% | 4%$\pm$18% | 0.95$\pm$0.03 | 0.73$\pm$0.13 | -37%$\pm$21% | -95%$\pm$48% | 0.71$\pm$0.06 |
| miami_housing | -33%$\pm$10% | 0%$\pm$3% | -55%$\pm$24% | 0.95$\pm$0.01 | 0.27$\pm$0.02 | -25%$\pm$10% | -39%$\pm$14% | 0.68$\pm$0.12 |
| kin8nm | -14%$\pm$1% | 1%$\pm$1% | -17%$\pm$2% | 0.95$\pm$0.01 | 0.66$\pm$0.02 | -11%$\pm$2% | -13%$\pm$3% | 0.32$\pm$0.03 |
| concrete_compressive_strength | -20%$\pm$3% | -2%$\pm$4% | -27%$\pm$9% | 0.95$\pm$0.02 | 0.61$\pm$0.02 | -37%$\pm$18% | -51%$\pm$22% | 0.48$\pm$0.06 |
| cars | -32%$\pm$12% | -4%$\pm$4% | -53%$\pm$32% | 0.93$\pm$0.02 | 0.38$\pm$0.05 | -38%$\pm$15% | -32%$\pm$14% | 0.62$\pm$0.16 |
| energy_efficiency | -30%$\pm$28% | -4%$\pm$2% | -82%$\pm$120% | 0.96$\pm$0.03 | 0.32$\pm$0.22 | -69%$\pm$60% | -75%$\pm$43% | 0.94$\pm$0.03 |
| california_housing | -6%$\pm$4% | -2%$\pm$1% | -4%$\pm$4% | 0.96$\pm$0.01 | 0.70$\pm$0.03 | -14%$\pm$3% | -17%$\pm$3% | 0.61$\pm$0.05 |
| airfoil_self_noise | -31%$\pm$4% | -3%$\pm$1% | -43%$\pm$9% | 0.94$\pm$0.02 | 0.75$\pm$0.05 | -62%$\pm$51% | -89%$\pm$66% | 0.35$\pm$0.10 |
| QSAR_fish_toxicity | -7%$\pm$7% | -3%$\pm$3% | -7%$\pm$6% | 0.92$\pm$0.04 | 0.50$\pm$0.08 | -9%$\pm$9% | -10%$\pm$9% | 0.39$\pm$0.09 |
| MultiMLPs |  |  |  |  |  |  |  |  |
| space_ga | -25%$\pm$11% | -3%$\pm$2% | -34%$\pm$25% | 0.94$\pm$0.01 | 0.38$\pm$0.06 | -40%$\pm$23% | -34%$\pm$15% | 0.21$\pm$0.23 |
| cpu_activity | -40%$\pm$9% | 0%$\pm$2% | -73%$\pm$24% | 0.95$\pm$0.01 | 0.25$\pm$0.02 | -15%$\pm$19% | -23%$\pm$17% | 0.75$\pm$0.12 |
| naval_propulsion_plant | -43%$\pm$11% | -1%$\pm$2% | -89%$\pm$44% | 0.95$\pm$0.02 | 1.33$\pm$0.19 | -108%$\pm$66% | -183%$\pm$195% | -0.03$\pm$0.47 |
| miami_housing | -36%$\pm$7% | -1%$\pm$1% | -57%$\pm$20% | 0.94$\pm$0.02 | 0.22$\pm$0.02 | -25%$\pm$8% | -28%$\pm$11% | 0.67$\pm$0.10 |
| kin8nm | -19%$\pm$3% | 0%$\pm$0% | -24%$\pm$5% | 0.96$\pm$0.01 | 0.70$\pm$0.02 | -16%$\pm$8% | -15%$\pm$9% | 0.37$\pm$0.04 |
| concrete_compressive_strength | 5%$\pm$10% | -3%$\pm$2% | 5%$\pm$10% | 0.94$\pm$0.02 | 0.54$\pm$0.04 | -12%$\pm$5% | -12%$\pm$6% | 0.54$\pm$0.05 |
| cars | -22%$\pm$17% | -1%$\pm$4% | -38%$\pm$30% | 0.95$\pm$0.03 | 0.35$\pm$0.06 | -28%$\pm$12% | -26%$\pm$13% | 0.66$\pm$0.10 |
| energy_efficiency | -7%$\pm$11% | -4%$\pm$3% | -11%$\pm$14% | 0.95$\pm$0.03 | 0.31$\pm$0.08 | 1%$\pm$9% | -1%$\pm$16% | 0.95$\pm$0.03 |
| california_housing | -24%$\pm$5% | 0%$\pm$1% | -34%$\pm$10% | 0.96$\pm$0.01 | 0.68$\pm$0.02 | -5%$\pm$2% | -3%$\pm$3% | 0.61$\pm$0.06 |
| airfoil_self_noise | -8%$\pm$6% | -4%$\pm$3% | -7%$\pm$8% | 0.93$\pm$0.04 | 0.61$\pm$0.07 | -15%$\pm$8% | -9%$\pm$8% | 0.55$\pm$0.18 |
| QSAR_fish_toxicity | -11%$\pm$5% | -2%$\pm$3% | -14%$\pm$8% | 0.92$\pm$0.04 | 0.53$\pm$0.11 | -13%$\pm$7% | -14%$\pm$8% | 0.34$\pm$0.06 |

### 5.3 Discussion

Our results indicate that SEMF, when combined with XGBoost and MLPs, performs strongly on datasets with complete data. Both models produce better prediction intervals than traditional quantile regression methods in the complete case. MultiMLPs also deliver good point predictions for some datasets despite the experimental design prioritizing interval estimation over point accuracy. This could indicate either the effectiveness of the chosen $\sigma_{k}$, which (indirectly) led to narrower prediction intervals or underfitting of the MLPs. The truth may lie somewhere in between; SEMF’s sampling operation, akin to cross-validation, partially helps combat overfitting, ensuring robust results despite variations in the data. In our preliminary experiments, we observed that increasing the depth of the baseline MLPs did not help and eventually led to overfitting. ETs exhibit more mixed performance, highly dependent on the dataset’s characteristics. One possible explanation behind the larger variations in predictive power compared to other models may be ETs’ reliance on randomized splits, which introduces significant variability in the prediction process. Consequently, ETs may not benefit from the iterative sampling and refinement of predictions inherent in SEMF.

The robustness of SEMF diminishes when applied to datasets with missing data. Despite not offering improvements over the baseline in the presence of missing data, we have presented these results to transparently illustrate our framework’s current capabilities of handling missing inputs. From a theoretical standpoint, we believe there is value in further investigating how a single loss function that leverages EM’s missing data capabilities can be effectively applied. Given that SEMF performs well with complete data, a practical alternative might involve using the best imputation method for the data at hand before applying SEMF, effectively treating the data as complete. We expect our ablation study to perform well, given the complete data results.

An important observation is the stability of SEMF across different random seeds, as indicated by consistent PICP and NMPIW metrics (full results in Appendix E), contrasting the significant variability observed in the baseline models. This suggests that SEMF offers a more reliable performance framework. Additionally, it is worth noting that our experiments did not precisely tune for conformalized prediction intervals but used the non-conformalized (raw) quantiles from SEMF. The non-conformalized intervals offer less coverage but produce better CWR and NMPIW than the baseline (Appendix F). Certain experimental choices—such as processing the columns separately, using the same hyperparameters for both complete and incomplete data, and fixing the number of latent nodes per input ($m_{k}$)—may have further constrained the models’ ability to tailor their performance to the varying complexities of different datasets.

## 6 Conclusion

This paper introduces the Supervised Expectation-Maximization Framework (SEMF), a novel model-agnostic approach for generating prediction intervals in datasets with missing values. SEMF draws from the EM algorithm for supervised learning to devise latent representations that produce better prediction intervals than quantile regression. Due to SEMF’s iterative simulation technique, training and inference can be done with complete and incomplete data. A comprehensive set of 330 experimental runs on 11 datasets with three different model types showed that SEMF, in the case of complete data, outperforms quantile regression, particularly on complete datasets and when using XGBoost, which intrinsically lacks latent representations. The results of the missing data are less positive and require further investigation. This research underscores SEMF’s potential in various application domains and opens new avenues for further exploration of supervised latent representation learning and uncertainty estimation.

## 7 Limitations & Future Work

The primary limitation of this study was its reliance on the normality assumption, which may not fully capture the potential of SEMF across diverse data distributions. Although in Appendix G we demonstrate that the framework can learn non-normal patterns, further investigation and exploration of SEMF under other distributions, such as uniform, log-normal, and generalized extreme value distributions, are needed. The computational complexity of the approach presents another significant challenge, as the current implementation can be optimized for large-scale applications. Additionally, while the CWR metric is useful, it implicitly assumes that a 1% drop in PICP equates to a 1% reduction in NMPIW, thus assuming a uniform distribution. Evaluating CWR under various distributional assumptions would provide a more comprehensive assessment of its implications. Additionally, SEMF has only been evaluated on MCAR data and does not address missing at random cases (MAR), which require further investigation for real-world applicability. Finally, applying the same hyper-parameters across all datasets without specific tuning for incomplete data likely contributes to the observed decline in accuracy and robustness.

Future work presents several intriguing avenues for exploration. A promising direction is the application of SEMF in multi-modal data settings, where the distinct $p_{\phi}$ components of the framework could be adapted to process diverse data types—from images and text to tabular datasets—enabling a more nuanced and powerful approach to integrating heterogeneous data sources. This capability positions the framework as a versatile tool for addressing missing data challenges across various domains and can also help expand it to discrete and multiple outputs. Another valuable area for development is the exploration of methods to capture and leverage dependencies among input features, which could improve the model’s predictive performance and provide deeper insights into the underlying data structure. These advancements can enhance the broader appeal of end-to-end approaches like SEMF in the ML community.

## Appendix A SEMF algorithm

Figure: Algorithm 1 SEMF Training: two input sources where $x_{1}$ can be missing

Figure: Algorithm 2 SEMF Inference

## Appendix B Datasets for tabular benchmark

OpenML-CTR23 [] datasets are selected in the following manner. The first criterion is to exclude datasets exceeding 30,000 instances or 30 features to maintain computational tractability. Moreover, we exclude the moneyball data [] to control for missing values and any datasets with non-numeric features, such as those with temporal or ordinal data not encoded numerically. We then categorize the datasets based on size: small for those with less than ten features, medium for 10 to 19 features, and large for 20 to 29 features. We apply a similar size classification based on the number of instances, considering datasets with more than 10,000 instances as large. To avoid computational constraints, we exclude datasets that were large in both features and instances, ensuring a varied yet manageable set for our experiments. This leads us to the final list of 11 datasets listed in Table 4.

**Table 4: Summary of benchmark tabular datasets retained from []**
| Dataset Name | N Samples | N Features | OpenML Data ID | Y [Min:Max] | Source |
| --- | --- | --- | --- | --- | --- |
| space_ga | 3,107 | 7 | [45402](https://www.openml.org/d/45402) | [-3.06:0.1] | [] |
| cpu_activity | 8,192 | 22 | [44978](https://www.openml.org/d/44978) | [0:99] | [] |
| naval_propulsion_plant | 11,934 | 15 | [44969](https://www.openml.org/d/44969) | [0.95:1.0] | [] |
| miami_housing | 13,932 | 16 | [44983](https://www.openml.org/d/44983) | [72,000:2,650,000] | [] |
| kin8nm | 8,192 | 9 | [44980](https://www.openml.org/d/44980) | [0.04:1.46] | [] |
| concrete_compressive_strength | 1,030 | 9 | [44959](https://www.openml.org/d/44959) | [2.33:82.6] | [] |
| cars | 804 | 18 | [44994](https://www.openml.org/d/44994) | [8,639:70,756] | [] |
| energy_efficiency | 768 | 9 | [44960](https://www.openml.org/d/44960) | [6.01:43.1] | [] |
| california_housing | 20,640 | 9 | [44977](https://www.openml.org/d/44977) | [14,999:500,001] | [] |
| airfoil_self_noise | 1,503 | 6 | [44957](https://www.openml.org/d/44957) | [103.38:140.98] | [] |
| QSAR_fish_toxicity | 908 | 7 | [44970](https://www.openml.org/d/44970) | [0.053:9.612] | [] |

## Appendix C Optimal set of hyper-parameters

The hyper-parameter tuning for SEMF is implemented and monitored using Weights & Biases []. A random search is done in the hyper-parameter space for a maximum of 500 iterations on all 11 datasets, focusing on tuning the models only on the complete datasets. Key hyper-parameters are varied across a predefined set to balance accuracy and computational efficiency. The following grid is used for hyper-parameter tuning: the number of importance sampling operations $R\in\{5,10,25,50,100\}$ (100 is omitted for MultiMLPS), nodes per latent dimension $m_{k}\in\{1,5,10,20,30\}$, and standard deviations $\sigma_{m_{k}}\in\{0.001,0.01,0.1,1.0\}$. Early stopping steps (patience) are set to five or ten, and $R_{\text{infer}}$ is explored at [30, 50, 70]. The option to run the models in parallel must be consistently enabled. Table 5 shows the optimal set of hyper-parameters. This table includes common hyper-parameters for complete and 50% datasets and another part showing $\xi_{\mathrm{nodes}}$, which is tuned manually and only relevant to the missing data.

**Table 5: Hyper-parameters for MultiXGBs, MultiETs, and MultiMLPs used for both complete and missing data.**
| Dataset | Complete and Missing | Missing |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| R | $m_{k}$ | $\sigma_{k}$ | Patience | $R_{\mathrm{infer}}$ | $\xi_{\mathrm{nodes}}$ |  |
| MultiXGBs |  |  |  |  |  |  |
| space_ga | 10 | 30 | 1.0 | 5 | 70 | 100 |
| cpu_activity | 5 | 30 | 1.0 | 5 | 70 | 50 |
| naval_propulsion_plant | 5 | 30 | 0.01 | 5 | 50 | 100 |
| miami_housing | 5 | 10 | 0.1 | 5 | 50 | 50 |
| kin8nm | 5 | 30 | 1.0 | 10 | 70 | 100 |
| concrete_compressive_strength | 25 | 30 | 1.0 | 5 | 70 | 100 |
| cars | 50 | 10 | 1.0 | 10 | 70 | 100 |
| energy_efficiency | 5 | 1 | 0.01 | 10 | 70 | 100 |
| california_housing | 5 | 10 | 0.1 | 10 | 50 | 100 |
| airfoil_self_noise | 25 | 1 | 0.01 | 10 | 70 | 100 |
| QSAR_fish_toxicity | 50 | 30 | 1.0 | 5 | 70 | 100 |
| MultiETs |  |  |  |  |  |  |
| space_ga | 10 | 30 | 1.0 | 10 | 70 | 100 |
| cpu_activity | 5 | 30 | 1.0 | 10 | 70 | 50 |
| naval_propulsion_plant | 5 | 30 | 0.01 | 10 | 50 | 100 |
| miami_housing | 10 | 10 | 0.1 | 10 | 50 | 50 |
| kin8nm | 5 | 30 | 1.0 | 10 | 70 | 100 |
| concrete_compressive_strength | 25 | 30 | 1.0 | 10 | 70 | 100 |
| cars | 100 | 5 | 0.1 | 10 | 100 | 100 |
| energy_efficiency | 5 | 1 | 0.01 | 10 | 70 | 100 |
| california_housing | 5 | 10 | 0.1 | 5 | 50 | 100 |
| airfoil_self_noise | 25 | 1 | 0.01 | 10 | 70 | 100 |
| QSAR_fish_toxicity | 50 | 30 | 1.0 | 10 | 70 | 100 |
| MultiMLPs |  |  |  |  |  |  |
| space_ga | 25 | 10 | 0.001 | 10 | 50 | 100 |
| cpu_activity | 5 | 20 | 0.001 | 5 | 50 | 50 |
| naval_propulsion_plant | 5 | 20 | 0.001 | 5 | 50 | 100 |
| miami_housing | 5 | 20 | 0.01 | 5 | 50 | 50 |
| kin8nm | 5 | 20 | 0.001 | 5 | 50 | 100 |
| concrete_compressive_strength | 5 | 30 | 0.001 | 10 | 50 | 100 |
| cars | 5 | 30 | 0.1 | 5 | 50 | 100 |
| energy_efficiency | 50 | 30 | 0.1 | 10 | 50 | 100 |
| california_housing | 5 | 20 | 0.01 | 5 | 50 | 100 |
| airfoil_self_noise | 25 | 10 | 0.01 | 10 | 50 | 100 |
| QSAR_fish_toxicity | 50 | 30 | 1.0 | 10 | 70 | 100 |

MultiXGBs and MultiMLPs benefit from early stopping to reduce computation time in complete and incomplete cases. Similarly, the baseline models for these instances use the same hyper-parameters for early stopping. Further, the number of epochs in the case of MultiMLPs is set as 1000, except for energy_efficiency and QSAR_fish_toxicity, where this is changed to 5000. Any model-specific hyperparameter we did not specify in this paper remains at the implementation’s default value (e.g., the number of leaves in XGBoost from []). Along with the supplementary code, we provide three additional CSV files: one for the results and hyperparameters of all 330 runs and the other two for the optimal hyperparameters of SEMF models, both raw (directly from SEMF) and conformalized.

Additional conditions are applied only for experiments with missing data. Datasets california_housing, cpu_activity, miami_housing, and naval_propulsion_plant—have a patience of five to expedite the training process. Additionally, for california_housing and cpu_activity, the $R_{\text{infer}}$ value is set to 30, while for all the other datasets, it is set to 50. We do this to ensure efficient computation, speed, and memory usage (especially for the GPU).

For training MultiXGBs and MultiETs, the computations are performed in parallel using CPU cores (Intel^® Core^™ i9-13900KF). For MultiMLPs, they are done on a GPU (NVIDIA^® GeForce RTX^™ 4090). The GPU is also consistently used for the missing data simulator and training $p_{\xi}$. All the computations are done on a machine with 32 GB of memory. The code provides further details on hardware and reproducibility.

## Appendix D Metrics for prediction intervals

### D.1 Common metrics

The most common metrics for evaluating prediction intervals [, ] are:

- •
Prediction Interval Coverage Probability (PICP): This metric assesses the proportion of times the true value of the target variable falls within the constructed prediction intervals. For a set of test examples ${(x_{1},y_{1}),\ldots,(x_{N},y_{N})}$, a given level of confidence $\alpha$, and their corresponding prediction intervals ${I_{1},\ldots,I_{N}}$, the PICP is calculated as:
$\text{PICP}=\frac{1}{N}\sum_{i=1}^{N}\mathds{1}(y_{i}\in[L_{i},U_{i}]),$
(21)
where $U_{i}$ and $L_{i}$ are the upper and lower bounds of the predicted values for the $i$-th instance. $y_{i}$ is the actual value of the $i$-th test example, and $\mathds{1}$ is the indicator function, which equals 1 if $y_{i}$ is in the interval $[L_{i},U_{i}]$ and 0 otherwise. $0\leq\text{{PICP}}\leq 1$ where PICP closer to 1 and higher than the confidence level $\alpha$ is favored.
- •
Mean Prediction Interval Width (MPIW): The average width is computed as
$\text{MPIW}=\frac{1}{N}\sum_{i=1}^{N}(U_{i}-L_{i}),$
(22)
which shows the sharpness or uncertainty, where $0\leq\text{{MPIW}}<\infty$ and MPIW close to 0 is preferred.
- •
Normalized Mean Prediction Interval Width (NMPIW): Since MPIW varies by dataset, it can be normalized by the range of the target variable
$\text{NMPIW}=\frac{\text{MPIW}}{\max(y)-\min(y)}$
(23)
where $\max(y)$ and $\min(y)$ are the maximum and minimum values of the target variable, respectively. The interpretation remains the same as MPIW.

### D.2 Impact of relative metrics for modeling

As our primary focus is on interval prediction, configurations demonstrating the most significant improvements in $\Delta$CWR and $\Delta$PICP are prioritized when selecting the optimal hyper-parameters. Furthermore, both $\Delta$PICP and $\Delta$CWR must be positive, indicating that we must at least have the same reliability of the baseline (PICP) with better or same interval ratios (CWR). In instances where no configuration meets the initial improvement criteria for both metrics, we relax the requirement for positive $\Delta$PICP to accept values greater than -5% and subsequently -10%, allowing us to consider configurations where SEMF significantly improves CWR, even if the PICP improvement is less marked but remains within an acceptable range for drawing comparisons.

## Appendix E Full conformalized results

**Table 6: Test results for MultiXGBs with complete data at 95% quantiles for seeds 0, 10, 20, 30, 40, with rows ordered by seed (ascending). Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | MPIW | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| space_ga | 2% | -2% | 4% | 0.95 | 2.40 | 0.26 | -10% | -13% | 0.60 |
| space_ga | 10% | -1% | 10% | 0.95 | 2.34 | 0.25 | -10% | -10% | 0.62 |
| space_ga | 8% | -2% | 9% | 0.93 | 2.24 | 0.24 | -9% | -8% | 0.60 |
| space_ga | 6% | -2% | 7% | 0.94 | 2.38 | 0.26 | -8% | -10% | 0.59 |
| space_ga | 7% | -1% | 7% | 0.95 | 2.55 | 0.28 | -8% | -10% | 0.59 |
| cpu_activity | 7% | 1% | 5% | 0.95 | 0.49 | 0.09 | 22% | 2% | 0.98 |
| cpu_activity | 20% | 1% | 17% | 0.93 | 0.45 | 0.09 | 20% | -5% | 0.98 |
| cpu_activity | 20% | 0% | 17% | 0.93 | 0.45 | 0.09 | 21% | 0% | 0.98 |
| cpu_activity | 21% | 1% | 16% | 0.94 | 0.45 | 0.09 | 23% | -2% | 0.98 |
| cpu_activity | 10% | 2% | 7% | 0.95 | 0.47 | 0.09 | 20% | -3% | 0.98 |
| naval_propulsion_plant | 163% | -1% | 62% | 0.95 | 0.36 | 0.11 | -15% | -12% | 0.99 |
| naval_propulsion_plant | 190% | 0% | 66% | 0.95 | 0.37 | 0.11 | -20% | -14% | 0.99 |
| naval_propulsion_plant | 151% | 0% | 60% | 0.96 | 0.40 | 0.12 | -12% | -9% | 0.99 |
| naval_propulsion_plant | 185% | -1% | 65% | 0.94 | 0.35 | 0.10 | -9% | -15% | 0.99 |
| naval_propulsion_plant | 169% | -1% | 63% | 0.96 | 0.37 | 0.11 | -15% | -11% | 0.99 |
| miami_housing | -2% | 0% | -2% | 0.95 | 0.99 | 0.12 | 4% | 1% | 0.91 |
| miami_housing | -10% | 0% | -10% | 0.95 | 1.03 | 0.13 | 7% | 7% | 0.90 |
| miami_housing | -6% | 1% | -8% | 0.95 | 0.99 | 0.12 | 9% | 5% | 0.91 |
| miami_housing | -7% | 0% | -8% | 0.95 | 0.99 | 0.12 | 3% | -1% | 0.91 |
| miami_housing | -10% | 0% | -12% | 0.95 | 1.05 | 0.13 | -4% | 0% | 0.90 |
| kin8nm | 6% | 0% | 6% | 0.94 | 2.27 | 0.45 | -21% | -24% | 0.62 |
| kin8nm | 4% | 1% | 3% | 0.95 | 2.39 | 0.47 | -19% | -21% | 0.64 |
| kin8nm | 9% | 1% | 8% | 0.94 | 2.20 | 0.43 | -17% | -20% | 0.64 |
| kin8nm | 9% | 1% | 7% | 0.95 | 2.36 | 0.46 | -21% | -22% | 0.63 |
| kin8nm | 3% | 1% | 2% | 0.94 | 2.33 | 0.46 | -20% | -23% | 0.62 |
| concrete_compressive_strength | 40% | -3% | 31% | 0.93 | 1.42 | 0.29 | -16% | -22% | 0.86 |
| concrete_compressive_strength | 47% | -6% | 36% | 0.91 | 1.48 | 0.30 | -29% | -52% | 0.84 |
| concrete_compressive_strength | 59% | -2% | 39% | 0.96 | 1.60 | 0.32 | -21% | -34% | 0.86 |
| concrete_compressive_strength | 35% | -3% | 28% | 0.95 | 1.48 | 0.30 | -28% | -39% | 0.83 |
| concrete_compressive_strength | 24% | 0% | 19% | 0.97 | 1.59 | 0.32 | -34% | -52% | 0.86 |
| cars | 56% | -4% | 38% | 0.91 | 0.70 | 0.12 | -7% | -4% | 0.95 |
| cars | 36% | -5% | 30% | 0.90 | 0.86 | 0.14 | -4% | -4% | 0.95 |
| cars | 30% | -3% | 25% | 0.91 | 0.76 | 0.13 | 2% | 5% | 0.96 |
| cars | 65% | -1% | 40% | 0.93 | 0.75 | 0.13 | -5% | -1% | 0.95 |
| cars | 16% | -4% | 17% | 0.91 | 0.87 | 0.15 | -2% | 0% | 0.95 |
| energy_efficiency | 165% | -5% | 64% | 0.92 | 0.20 | 0.06 | -50% | -29% | 1.00 |
| energy_efficiency | 288% | 0% | 74% | 0.91 | 0.14 | 0.04 | -3% | -13% | 1.00 |
| energy_efficiency | 217% | -1% | 69% | 0.96 | 0.18 | 0.05 | -21% | -31% | 1.00 |
| energy_efficiency | 253% | -3% | 73% | 0.92 | 0.16 | 0.04 | 17% | 16% | 1.00 |
| energy_efficiency | 185% | -10% | 68% | 0.91 | 0.23 | 0.07 | -17% | -22% | 1.00 |
| california_housing | -5% | 0% | -6% | 0.95 | 1.82 | 0.43 | -2% | -3% | 0.81 |
| california_housing | 4% | 0% | 4% | 0.95 | 1.72 | 0.41 | -1% | -2% | 0.81 |
| california_housing | -4% | 1% | -5% | 0.95 | 1.76 | 0.42 | 0% | -1% | 0.82 |
| california_housing | 1% | 0% | 1% | 0.95 | 1.74 | 0.41 | 0% | 0% | 0.81 |
| california_housing | -3% | 1% | -4% | 0.95 | 1.79 | 0.42 | -1% | -1% | 0.82 |
| airfoil_self_noise | 15% | -4% | 16% | 0.95 | 1.73 | 0.37 | -82% | -93% | 0.86 |
| airfoil_self_noise | 45% | 1% | 30% | 0.97 | 1.42 | 0.30 | -18% | -18% | 0.93 |
| airfoil_self_noise | 5% | 1% | 4% | 0.98 | 1.86 | 0.40 | -73% | -76% | 0.85 |
| airfoil_self_noise | -6% | 0% | -7% | 0.98 | 2.15 | 0.46 | -109% | -138% | 0.78 |
| airfoil_self_noise | 46% | -2% | 33% | 0.95 | 1.37 | 0.29 | -41% | -43% | 0.89 |
| QSAR_fish_toxicity | 21% | -2% | 19% | 0.88 | 2.12 | 0.33 | 3% | 3% | 0.53 |
| QSAR_fish_toxicity | 39% | -9% | 35% | 0.85 | 1.97 | 0.31 | 8% | 7% | 0.58 |
| QSAR_fish_toxicity | 43% | -3% | 32% | 0.89 | 2.21 | 0.34 | 4% | 3% | 0.57 |
| QSAR_fish_toxicity | 38% | -3% | 30% | 0.89 | 2.11 | 0.33 | -2% | -1% | 0.54 |
| QSAR_fish_toxicity | 29% | -6% | 28% | 0.85 | 2.11 | 0.33 | 0% | -6% | 0.52 |

**Table 7: Test results for MultiETs with complete data at 95% quantiles for seeds 0, 10, 20, 30, 40, with rows ordered by seed (ascending). Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | MPIW | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| space_ga | -8% | 2% | -11% | 0.95 | 2.59 | 0.28 | -13% | -15% | 0.55 |
| space_ga | -3% | 0% | -3% | 0.95 | 2.77 | 0.30 | -14% | -17% | 0.55 |
| space_ga | -2% | 1% | -4% | 0.96 | 2.61 | 0.28 | -13% | -13% | 0.57 |
| space_ga | -9% | 1% | -11% | 0.96 | 2.81 | 0.30 | -15% | -18% | 0.54 |
| space_ga | -7% | 0% | -8% | 0.96 | 2.79 | 0.30 | -19% | -24% | 0.51 |
| cpu_activity | 11% | -4% | 13% | 0.95 | 0.57 | 0.11 | -13% | -20% | 0.98 |
| cpu_activity | 14% | -5% | 16% | 0.94 | 0.54 | 0.10 | -13% | -18% | 0.98 |
| cpu_activity | 9% | -4% | 11% | 0.95 | 0.58 | 0.11 | -13% | -18% | 0.98 |
| cpu_activity | 10% | -4% | 13% | 0.94 | 0.57 | 0.11 | -15% | -22% | 0.97 |
| cpu_activity | 10% | -4% | 13% | 0.94 | 0.57 | 0.11 | -16% | -22% | 0.97 |
| naval_propulsion_plant | 131% | 1% | 56% | 0.96 | 0.96 | 0.28 | -320% | -418% | 0.96 |
| naval_propulsion_plant | 144% | 1% | 59% | 0.96 | 0.88 | 0.26 | -286% | -369% | 0.97 |
| naval_propulsion_plant | 174% | 0% | 63% | 0.96 | 0.81 | 0.24 | -257% | -348% | 0.97 |
| naval_propulsion_plant | 127% | 0% | 56% | 0.96 | 0.94 | 0.28 | -324% | -426% | 0.96 |
| naval_propulsion_plant | 108% | 1% | 52% | 0.96 | 1.03 | 0.30 | -391% | -469% | 0.95 |
| miami_housing | -7% | 0% | -7% | 0.95 | 1.22 | 0.15 | -11% | -20% | 0.90 |
| miami_housing | -10% | -1% | -10% | 0.95 | 1.24 | 0.15 | -6% | -17% | 0.90 |
| miami_housing | -8% | -1% | -8% | 0.95 | 1.25 | 0.15 | -13% | -23% | 0.89 |
| miami_housing | -9% | -1% | -10% | 0.95 | 1.22 | 0.15 | -9% | -19% | 0.90 |
| miami_housing | -9% | 0% | -9% | 0.94 | 1.24 | 0.15 | -10% | -22% | 0.89 |
| kin8nm | -10% | 1% | -12% | 0.95 | 2.75 | 0.54 | -36% | -40% | 0.46 |
| kin8nm | -8% | 0% | -9% | 0.94 | 2.66 | 0.52 | -33% | -36% | 0.49 |
| kin8nm | -11% | 3% | -15% | 0.95 | 2.76 | 0.54 | -32% | -35% | 0.50 |
| kin8nm | -9% | 2% | -12% | 0.94 | 2.67 | 0.52 | -36% | -40% | 0.48 |
| kin8nm | -10% | 1% | -13% | 0.95 | 2.73 | 0.53 | -33% | -36% | 0.49 |
| concrete_compressive_strength | -18% | -3% | -18% | 0.89 | 1.67 | 0.34 | -78% | -110% | 0.74 |
| concrete_compressive_strength | -5% | -3% | -3% | 0.89 | 1.51 | 0.30 | -71% | -97% | 0.76 |
| concrete_compressive_strength | -6% | -6% | 0% | 0.87 | 1.42 | 0.29 | -63% | -87% | 0.77 |
| concrete_compressive_strength | -6% | -3% | -4% | 0.90 | 1.47 | 0.30 | -63% | -91% | 0.77 |
| concrete_compressive_strength | -6% | 1% | -7% | 0.92 | 1.59 | 0.32 | -61% | -88% | 0.77 |
| cars | -35% | 5% | -60% | 0.95 | 1.01 | 0.17 | 6% | 3% | 0.95 |
| cars | -25% | 1% | -34% | 0.91 | 0.83 | 0.14 | 8% | 4% | 0.95 |
| cars | -19% | 0% | -24% | 0.92 | 0.80 | 0.14 | 4% | -2% | 0.95 |
| cars | -24% | -3% | -28% | 0.90 | 0.86 | 0.15 | 2% | 0% | 0.95 |
| cars | -21% | -3% | -22% | 0.91 | 0.91 | 0.15 | -3% | 1% | 0.94 |
| energy_efficiency | 16% | -8% | 21% | 0.90 | 0.18 | 0.05 | 0% | 0% | 1.00 |
| energy_efficiency | 6% | -2% | 6% | 0.97 | 0.21 | 0.06 | 4% | 1% | 1.00 |
| energy_efficiency | 8% | -4% | 11% | 0.95 | 0.21 | 0.06 | 5% | 2% | 1.00 |
| energy_efficiency | 3% | -3% | 5% | 0.96 | 0.21 | 0.06 | 1% | 0% | 1.00 |
| energy_efficiency | 17% | -4% | 18% | 0.95 | 0.19 | 0.06 | 5% | 0% | 1.00 |
| california_housing | -2% | -2% | 1% | 0.95 | 2.38 | 0.57 | -16% | -22% | 0.70 |
| california_housing | 3% | -2% | 5% | 0.95 | 2.31 | 0.55 | -13% | -19% | 0.72 |
| california_housing | 2% | -2% | 4% | 0.95 | 2.29 | 0.55 | -15% | -20% | 0.71 |
| california_housing | 0% | -3% | 3% | 0.95 | 2.32 | 0.55 | -16% | -22% | 0.71 |
| california_housing | 1% | -3% | 4% | 0.95 | 2.32 | 0.55 | -14% | -20% | 0.72 |
| airfoil_self_noise | -25% | -5% | -26% | 0.94 | 2.17 | 0.46 | -166% | -184% | 0.74 |
| airfoil_self_noise | 0% | -1% | 1% | 0.98 | 1.78 | 0.38 | -59% | -80% | 0.89 |
| airfoil_self_noise | -15% | -1% | -16% | 0.98 | 1.98 | 0.42 | -97% | -106% | 0.84 |
| airfoil_self_noise | -28% | -5% | -32% | 0.93 | 2.20 | 0.47 | -169% | -213% | 0.67 |
| airfoil_self_noise | -14% | -3% | -13% | 0.96 | 1.93 | 0.41 | -98% | -125% | 0.84 |
| QSAR_fish_toxicity | -16% | -2% | -17% | 0.88 | 2.45 | 0.38 | -6% | -10% | 0.53 |
| QSAR_fish_toxicity | -10% | -2% | -9% | 0.89 | 2.43 | 0.38 | -7% | -12% | 0.52 |
| QSAR_fish_toxicity | 3% | -6% | 9% | 0.88 | 2.38 | 0.37 | -8% | -12% | 0.50 |
| QSAR_fish_toxicity | -12% | -4% | -9% | 0.88 | 2.41 | 0.38 | -5% | -10% | 0.55 |
| QSAR_fish_toxicity | 8% | -9% | 16% | 0.84 | 2.03 | 0.32 | -6% | -13% | 0.53 |

**Table 8: Test results for MultiMLPs with complete data at 95% quantiles for seeds 0, 10, 20, 30, 40, with rows ordered by seed (ascending). Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | MPIW | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| space_ga | 5% | -2% | 7% | 0.95 | 2.11 | 0.23 | 0% | -1% | 0.74 |
| space_ga | 3% | -1% | 3% | 0.96 | 2.13 | 0.23 | -1% | -1% | 0.75 |
| space_ga | 6% | -1% | 7% | 0.94 | 2.04 | 0.22 | -3% | -3% | 0.74 |
| space_ga | 5% | -2% | 6% | 0.95 | 2.15 | 0.23 | 0% | 0% | 0.76 |
| space_ga | 10% | 0% | 9% | 0.95 | 2.06 | 0.22 | 1% | 0% | 0.76 |
| cpu_activity | -18% | 1% | -22% | 0.96 | 0.58 | 0.11 | 4% | 2% | 0.98 |
| cpu_activity | -3% | 0% | -2% | 0.95 | 0.51 | 0.10 | 8% | 7% | 0.98 |
| cpu_activity | -9% | 0% | -10% | 0.95 | 0.53 | 0.10 | 6% | 5% | 0.98 |
| cpu_activity | -6% | 0% | -7% | 0.95 | 0.54 | 0.10 | 6% | 5% | 0.98 |
| cpu_activity | -2% | 0% | -2% | 0.95 | 0.53 | 0.10 | 8% | 5% | 0.98 |
| naval_propulsion_plant | 32% | 1% | 24% | 0.95 | 0.23 | 0.07 | -44% | -48% | 1.00 |
| naval_propulsion_plant | 0% | 1% | -2% | 0.96 | 0.31 | 0.09 | -9% | -11% | 1.00 |
| naval_propulsion_plant | 7% | 0% | 5% | 0.95 | 0.24 | 0.07 | -35% | -13% | 1.00 |
| naval_propulsion_plant | -8% | 1% | -11% | 0.96 | 0.28 | 0.08 | -86% | -63% | 1.00 |
| naval_propulsion_plant | -13% | 2% | -16% | 0.96 | 0.34 | 0.10 | -51% | -45% | 0.99 |
| miami_housing | -38% | 0% | -60% | 0.95 | 1.18 | 0.15 | -6% | 1% | 0.91 |
| miami_housing | -37% | 0% | -59% | 0.95 | 1.16 | 0.14 | -2% | 7% | 0.91 |
| miami_housing | -43% | 0% | -74% | 0.95 | 1.25 | 0.15 | -4% | 6% | 0.91 |
| miami_housing | -38% | 1% | -62% | 0.95 | 1.15 | 0.14 | -2% | 3% | 0.92 |
| miami_housing | -33% | -1% | -47% | 0.95 | 1.10 | 0.14 | -1% | 4% | 0.91 |
| kin8nm | 10% | 1% | 9% | 0.95 | 1.03 | 0.20 | 4% | 5% | 0.93 |
| kin8nm | 2% | 1% | 1% | 0.96 | 1.07 | 0.21 | 10% | 10% | 0.94 |
| kin8nm | 4% | 1% | 3% | 0.95 | 1.06 | 0.21 | 10% | 6% | 0.94 |
| kin8nm | 10% | -3% | 12% | 0.93 | 0.99 | 0.19 | 6% | 5% | 0.93 |
| kin8nm | 15% | -1% | 13% | 0.94 | 1.03 | 0.20 | 7% | 7% | 0.93 |
| concrete_compressive_strength | 7% | -3% | 9% | 0.91 | 1.24 | 0.25 | 18% | 19% | 0.92 |
| concrete_compressive_strength | 18% | 1% | 14% | 0.97 | 1.39 | 0.28 | 9% | 12% | 0.91 |
| concrete_compressive_strength | 2% | 1% | 1% | 0.97 | 1.63 | 0.33 | 9% | 15% | 0.91 |
| concrete_compressive_strength | 7% | -5% | 11% | 0.94 | 1.62 | 0.33 | 4% | 10% | 0.90 |
| concrete_compressive_strength | 19% | -3% | 18% | 0.93 | 1.26 | 0.26 | 18% | 20% | 0.91 |
| cars | 5% | -9% | 13% | 0.88 | 0.66 | 0.11 | -5% | -2% | 0.95 |
| cars | 0% | -3% | 2% | 0.93 | 0.80 | 0.14 | -6% | -2% | 0.95 |
| cars | -9% | 0% | -10% | 0.93 | 0.80 | 0.14 | 1% | 1% | 0.96 |
| cars | -8% | -3% | -4% | 0.93 | 0.83 | 0.14 | -2% | -1% | 0.95 |
| cars | 25% | -1% | 21% | 0.94 | 0.79 | 0.13 | -4% | 5% | 0.95 |
| energy_efficiency | 76% | -3% | 46% | 0.94 | 0.16 | 0.04 | 31% | 31% | 1.00 |
| energy_efficiency | 46% | 7% | 25% | 0.99 | 0.19 | 0.06 | 35% | 32% | 1.00 |
| energy_efficiency | 34% | -1% | 27% | 0.95 | 0.17 | 0.05 | 28% | 32% | 1.00 |
| energy_efficiency | 29% | -3% | 25% | 0.95 | 0.16 | 0.05 | 31% | 32% | 1.00 |
| energy_efficiency | 79% | -3% | 45% | 0.96 | 0.17 | 0.05 | 37% | 38% | 1.00 |
| california_housing | -13% | 1% | -16% | 0.95 | 1.88 | 0.45 | 8% | 11% | 0.82 |
| california_housing | -8% | 0% | -9% | 0.95 | 1.78 | 0.42 | 7% | 8% | 0.81 |
| california_housing | -16% | 1% | -20% | 0.95 | 1.83 | 0.44 | 7% | 9% | 0.82 |
| california_housing | -11% | -1% | -12% | 0.94 | 1.82 | 0.43 | 7% | 8% | 0.81 |
| california_housing | -12% | 0% | -13% | 0.94 | 1.76 | 0.42 | 7% | 9% | 0.82 |
| airfoil_self_noise | 98% | -2% | 51% | 0.97 | 0.82 | 0.18 | 21% | 19% | 0.97 |
| airfoil_self_noise | 33% | 0% | 24% | 0.98 | 0.90 | 0.19 | 17% | 12% | 0.97 |
| airfoil_self_noise | 36% | -3% | 28% | 0.95 | 0.79 | 0.17 | 11% | 9% | 0.97 |
| airfoil_self_noise | 83% | -4% | 47% | 0.95 | 0.76 | 0.16 | 22% | 22% | 0.97 |
| airfoil_self_noise | 93% | -1% | 49% | 0.98 | 0.88 | 0.19 | 18% | 18% | 0.97 |
| QSAR_fish_toxicity | -11% | 2% | -16% | 0.91 | 2.53 | 0.39 | 5% | 11% | 0.51 |
| QSAR_fish_toxicity | 3% | 2% | 1% | 0.91 | 2.38 | 0.37 | 0% | -1% | 0.57 |
| QSAR_fish_toxicity | 7% | -1% | 7% | 0.91 | 2.31 | 0.36 | 6% | 11% | 0.54 |
| QSAR_fish_toxicity | 18% | 2% | 14% | 0.90 | 2.10 | 0.33 | -4% | 6% | 0.52 |
| QSAR_fish_toxicity | 9% | -1% | 9% | 0.83 | 1.84 | 0.29 | 10% | 9% | 0.61 |

**Table 9: Test results for MultiXGBs with 50% missing data at 95% quantiles for seeds 0, 10, 20, 30, 40, with rows ordered by seed (ascending). Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | MPIW | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| space_ga | 4% | -4% | 7% | 0.93 | 2.82 | 0.31 | -11% | -13% | 0.35 |
| space_ga | 2% | -1% | 3% | 0.94 | 2.97 | 0.32 | -6% | -6% | 0.45 |
| space_ga | -4% | -3% | -2% | 0.94 | 3.06 | 0.33 | -12% | -11% | 0.44 |
| space_ga | -1% | -1% | -1% | 0.95 | 2.97 | 0.32 | -1% | -6% | 0.46 |
| space_ga | 2% | 0% | 2% | 0.95 | 2.77 | 0.30 | -5% | -11% | 0.54 |
| cpu_activity | -9% | 1% | -11% | 0.95 | 0.89 | 0.17 | -7% | -23% | 0.94 |
| cpu_activity | -26% | 1% | -44% | 0.93 | 1.02 | 0.19 | -1% | -26% | 0.81 |
| cpu_activity | -15% | -2% | -14% | 0.89 | 0.80 | 0.15 | -30% | -30% | 0.73 |
| cpu_activity | -1% | -1% | -3% | 0.95 | 0.86 | 0.16 | 36% | 5% | 0.88 |
| cpu_activity | -19% | 2% | -26% | 0.98 | 1.12 | 0.21 | -2% | -11% | 0.81 |
| naval_propulsion_plant | -8% | 0% | -11% | 0.98 | 2.93 | 0.86 | -45% | -90% | 0.73 |
| naval_propulsion_plant | 31% | -2% | 24% | 0.93 | 1.50 | 0.44 | 5% | -6% | 0.84 |
| naval_propulsion_plant | 18% | -2% | 17% | 0.96 | 2.06 | 0.61 | -30% | -42% | 0.83 |
| naval_propulsion_plant | 53% | -5% | 37% | 0.87 | 1.27 | 0.37 | 6% | 6% | 0.76 |
| naval_propulsion_plant | -5% | -3% | -6% | 0.93 | 2.22 | 0.65 | -7% | -11% | 0.70 |
| miami_housing | -10% | -2% | -11% | 0.94 | 1.45 | 0.18 | -7% | -12% | 0.80 |
| miami_housing | -38% | 0% | -63% | 0.93 | 1.86 | 0.23 | -35% | -35% | 0.54 |
| miami_housing | -26% | -1% | -33% | 0.95 | 1.74 | 0.22 | -10% | -19% | 0.76 |
| miami_housing | 1% | -3% | 2% | 0.95 | 1.35 | 0.17 | -14% | -20% | 0.79 |
| miami_housing | -17% | 1% | -22% | 0.94 | 1.64 | 0.20 | -4% | -4% | 0.72 |
| kin8nm | -6% | 0% | -8% | 0.97 | 3.36 | 0.66 | 0% | 1% | 0.40 |
| kin8nm | -15% | 1% | -22% | 0.97 | 3.46 | 0.68 | -6% | -8% | 0.39 |
| kin8nm | -8% | 3% | -12% | 0.95 | 3.00 | 0.59 | 1% | 2% | 0.41 |
| kin8nm | -9% | 2% | -14% | 0.96 | 3.11 | 0.61 | -5% | -7% | 0.44 |
| kin8nm | -8% | 1% | -9% | 0.95 | 3.23 | 0.63 | -5% | -6% | 0.37 |
| concrete_compressive_strength | -8% | -3% | -8% | 0.95 | 2.81 | 0.57 | -19% | -22% | 0.48 |
| concrete_compressive_strength | 7% | -5% | 11% | 0.93 | 2.79 | 0.56 | -37% | -44% | 0.57 |
| concrete_compressive_strength | 14% | -3% | 13% | 0.97 | 2.69 | 0.55 | -4% | -9% | 0.64 |
| concrete_compressive_strength | 4% | -1% | 3% | 0.95 | 2.83 | 0.57 | -15% | -24% | 0.50 |
| concrete_compressive_strength | -11% | 0% | -14% | 0.97 | 3.01 | 0.61 | -4% | -3% | 0.69 |
| cars | 31% | -3% | 24% | 0.93 | 1.91 | 0.32 | -9% | -16% | 0.75 |
| cars | 51% | -7% | 38% | 0.93 | 1.84 | 0.31 | -37% | -25% | 0.81 |
| cars | -16% | -11% | -17% | 0.89 | 2.17 | 0.37 | -68% | -80% | 0.55 |
| cars | -13% | -3% | -21% | 0.94 | 2.18 | 0.37 | -8% | -14% | 0.76 |
| cars | -5% | -12% | 5% | 0.83 | 2.00 | 0.34 | -28% | -39% | 0.40 |
| energy_efficiency | -11% | 0% | -13% | 1.00 | 1.15 | 0.33 | -15% | -24% | 0.97 |
| energy_efficiency | 20% | -10% | 25% | 0.89 | 0.84 | 0.24 | -705% | -520% | 0.76 |
| energy_efficiency | 42% | -1% | 23% | 0.95 | 0.78 | 0.23 | -37% | -39% | 0.94 |
| energy_efficiency | 61% | -4% | 40% | 0.94 | 0.70 | 0.20 | -16% | -44% | 0.96 |
| energy_efficiency | 20% | -3% | 16% | 0.97 | 1.12 | 0.33 | -155% | -137% | 0.93 |
| california_housing | -14% | 0% | -17% | 0.96 | 2.67 | 0.64 | -3% | -3% | 0.65 |
| california_housing | 0% | -2% | 0% | 0.96 | 2.41 | 0.57 | -11% | -13% | 0.73 |
| california_housing | -7% | -1% | -8% | 0.96 | 2.48 | 0.59 | 0% | 1% | 0.73 |
| california_housing | 2% | -3% | 3% | 0.92 | 2.28 | 0.54 | -4% | -7% | 0.61 |
| california_housing | -12% | -1% | -16% | 0.97 | 2.63 | 0.63 | -1% | -3% | 0.73 |
| airfoil_self_noise | -30% | -2% | -43% | 0.97 | 3.95 | 0.84 | -94% | -110% | 0.50 |
| airfoil_self_noise | -18% | -1% | -22% | 0.91 | 2.98 | 0.64 | -19% | -29% | 0.38 |
| airfoil_self_noise | -23% | -4% | -25% | 0.91 | 3.49 | 0.75 | -43% | -48% | 0.19 |
| airfoil_self_noise | -31% | 1% | -49% | 0.97 | 3.61 | 0.77 | -14% | -8% | 0.52 |
| airfoil_self_noise | -17% | -1% | -21% | 0.97 | 3.81 | 0.81 | -14% | -22% | 0.43 |
| QSAR_fish_toxicity | -8% | 3% | -14% | 0.98 | 4.22 | 0.66 | 4% | 4% | 0.43 |
| QSAR_fish_toxicity | -6% | 1% | -7% | 0.97 | 3.93 | 0.61 | -9% | -3% | 0.35 |
| QSAR_fish_toxicity | 8% | -5% | 10% | 0.88 | 2.59 | 0.40 | -4% | -5% | 0.38 |
| QSAR_fish_toxicity | -5% | -5% | -5% | 0.88 | 2.83 | 0.44 | -7% | -3% | 0.35 |
| QSAR_fish_toxicity | 12% | -11% | 15% | 0.85 | 2.58 | 0.40 | -5% | -3% | 0.30 |

**Table 10: Test results for MultiETs with 50% missing data at 95% quantiles for seeds 0, 10, 20, 30, 40, with rows ordered by seed (ascending). Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | MPIW | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| space_ga | -7% | -2% | -6% | 0.94 | 3.08 | 0.33 | -18% | -19% | 0.29 |
| space_ga | -12% | 3% | -18% | 0.97 | 3.41 | 0.37 | -7% | -7% | 0.40 |
| space_ga | -13% | 0% | -16% | 0.95 | 3.05 | 0.33 | -13% | -11% | 0.42 |
| space_ga | -13% | 0% | -18% | 0.96 | 3.21 | 0.35 | -8% | -8% | 0.41 |
| space_ga | -10% | 0% | -11% | 0.96 | 2.91 | 0.32 | -15% | -18% | 0.46 |
| cpu_activity | -18% | -3% | -23% | 0.95 | 1.03 | 0.19 | -40% | -56% | 0.92 |
| cpu_activity | -12% | -6% | -11% | 0.91 | 1.04 | 0.20 | -15% | -44% | 0.81 |
| cpu_activity | -5% | -5% | -2% | 0.92 | 1.00 | 0.19 | -43% | -59% | 0.75 |
| cpu_activity | -13% | -2% | -14% | 0.96 | 0.97 | 0.18 | 2% | -19% | 0.87 |
| cpu_activity | -30% | -1% | -42% | 0.96 | 1.31 | 0.25 | -10% | -43% | 0.76 |
| naval_propulsion_plant | -19% | 0% | -23% | 0.98 | 3.17 | 0.94 | -63% | -158% | 0.71 |
| naval_propulsion_plant | 11% | -5% | 13% | 0.94 | 2.24 | 0.66 | -31% | -78% | 0.73 |
| naval_propulsion_plant | 14% | -2% | 14% | 0.97 | 2.26 | 0.67 | -61% | -147% | 0.80 |
| naval_propulsion_plant | 24% | -9% | 25% | 0.89 | 1.99 | 0.59 | -13% | -42% | 0.66 |
| naval_propulsion_plant | -10% | -2% | -11% | 0.95 | 2.75 | 0.81 | -19% | -52% | 0.64 |
| miami_housing | -15% | -2% | -17% | 0.94 | 1.86 | 0.23 | -11% | -25% | 0.79 |
| miami_housing | -44% | 5% | -87% | 0.96 | 2.39 | 0.30 | -43% | -64% | 0.47 |
| miami_housing | -41% | 0% | -69% | 0.96 | 2.30 | 0.29 | -27% | -42% | 0.72 |
| miami_housing | -35% | -2% | -52% | 0.96 | 2.18 | 0.27 | -21% | -33% | 0.79 |
| miami_housing | -31% | 1% | -48% | 0.95 | 2.24 | 0.28 | -22% | -30% | 0.64 |
| kin8nm | -15% | -1% | -19% | 0.96 | 3.55 | 0.69 | -8% | -8% | 0.29 |
| kin8nm | -14% | 2% | -19% | 0.96 | 3.43 | 0.67 | -10% | -14% | 0.32 |
| kin8nm | -11% | 0% | -13% | 0.94 | 3.20 | 0.63 | -9% | -12% | 0.32 |
| kin8nm | -14% | 0% | -18% | 0.95 | 3.30 | 0.65 | -14% | -18% | 0.37 |
| kin8nm | -14% | 2% | -19% | 0.96 | 3.42 | 0.67 | -13% | -16% | 0.30 |
| concrete_compressive_strength | -20% | 4% | -35% | 0.96 | 3.01 | 0.61 | -30% | -43% | 0.41 |
| concrete_compressive_strength | -16% | -8% | -10% | 0.92 | 3.08 | 0.62 | -72% | -94% | 0.47 |
| concrete_compressive_strength | -18% | -3% | -24% | 0.96 | 2.99 | 0.60 | -21% | -33% | 0.56 |
| concrete_compressive_strength | -25% | -4% | -34% | 0.94 | 2.86 | 0.58 | -26% | -36% | 0.41 |
| concrete_compressive_strength | -22% | 2% | -31% | 0.96 | 3.05 | 0.62 | -35% | -50% | 0.53 |
| cars | -24% | -6% | -27% | 0.92 | 1.99 | 0.34 | -22% | -23% | 0.73 |
| cars | -49% | 4% | -103% | 0.97 | 2.70 | 0.46 | -20% | -17% | 0.80 |
| cars | -14% | -5% | -12% | 0.93 | 2.02 | 0.34 | -59% | -56% | 0.60 |
| cars | -36% | -6% | -54% | 0.93 | 2.44 | 0.41 | -37% | -37% | 0.66 |
| cars | -37% | -6% | -68% | 0.89 | 2.12 | 0.36 | -50% | -25% | 0.34 |
| energy_efficiency | 2% | -3% | 5% | 0.97 | 0.62 | 0.18 | -18% | -32% | 0.98 |
| energy_efficiency | -42% | -6% | -62% | 0.94 | 0.85 | 0.25 | -159% | -104% | 0.90 |
| energy_efficiency | -4% | -6% | 2% | 0.93 | 0.69 | 0.20 | -8% | -20% | 0.95 |
| energy_efficiency | -29% | -4% | -41% | 0.94 | 0.79 | 0.23 | -39% | -82% | 0.95 |
| energy_efficiency | -75% | 0% | -316% | 1.00 | 2.61 | 0.76 | -122% | -136% | 0.92 |
| california_housing | -2% | -2% | 0% | 0.96 | 2.88 | 0.69 | -11% | -16% | 0.60 |
| california_housing | -9% | -2% | -8% | 0.96 | 2.90 | 0.69 | -19% | -23% | 0.64 |
| california_housing | -5% | -2% | -4% | 0.97 | 3.02 | 0.72 | -11% | -13% | 0.64 |
| california_housing | -1% | -3% | 1% | 0.93 | 2.77 | 0.66 | -15% | -18% | 0.50 |
| california_housing | -10% | -1% | -10% | 0.97 | 3.07 | 0.73 | -13% | -17% | 0.64 |
| airfoil_self_noise | -33% | -3% | -46% | 0.97 | 3.62 | 0.77 | -163% | -218% | 0.34 |
| airfoil_self_noise | -36% | -2% | -56% | 0.93 | 3.42 | 0.73 | -27% | -36% | 0.42 |
| airfoil_self_noise | -31% | -5% | -40% | 0.92 | 3.42 | 0.73 | -50% | -83% | 0.17 |
| airfoil_self_noise | -32% | -2% | -45% | 0.97 | 3.91 | 0.83 | -45% | -57% | 0.43 |
| airfoil_self_noise | -22% | -2% | -30% | 0.94 | 3.29 | 0.70 | -27% | -51% | 0.36 |
| QSAR_fish_toxicity | 2% | 1% | 0% | 0.98 | 4.12 | 0.64 | 4% | 2% | 0.51 |
| QSAR_fish_toxicity | -10% | -4% | -7% | 0.93 | 3.43 | 0.53 | -12% | -12% | 0.40 |
| QSAR_fish_toxicity | -7% | 1% | -11% | 0.93 | 2.87 | 0.45 | -6% | -8% | 0.45 |
| QSAR_fish_toxicity | -17% | -7% | -16% | 0.86 | 2.99 | 0.47 | -23% | -26% | 0.25 |
| QSAR_fish_toxicity | 0% | -3% | -1% | 0.90 | 2.63 | 0.41 | -4% | -6% | 0.34 |

**Table 11: Test results for MultiMLPs with 50% missing data at 95% quantiles for seeds 0, 10, 20, 30, 40, with rows ordered by seed (ascending). Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | MPIW | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| space_ga | -25% | -3% | -32% | 0.95 | 3.45 | 0.37 | -41% | -40% | 0.15 |
| space_ga | -22% | -3% | -27% | 0.96 | 3.58 | 0.39 | -28% | -25% | 0.35 |
| space_ga | -45% | -2% | -82% | 0.95 | 4.52 | 0.49 | -85% | -61% | -0.21 |
| space_ga | -13% | -2% | -13% | 0.95 | 3.12 | 0.34 | -21% | -20% | 0.37 |
| space_ga | -20% | -6% | -18% | 0.92 | 3.07 | 0.33 | -25% | -26% | 0.38 |
| cpu_activity | -22% | -2% | -32% | 0.95 | 1.14 | 0.21 | 14% | 1% | 0.92 |
| cpu_activity | -44% | 2% | -80% | 0.95 | 1.40 | 0.27 | -30% | -34% | 0.72 |
| cpu_activity | -49% | 1% | -100% | 0.94 | 1.44 | 0.27 | -37% | -49% | 0.54 |
| cpu_activity | -45% | 2% | -89% | 0.96 | 1.23 | 0.23 | -20% | -17% | 0.78 |
| cpu_activity | -38% | -1% | -61% | 0.96 | 1.29 | 0.24 | -3% | -16% | 0.78 |
| naval_propulsion_plant | -36% | -4% | -52% | 0.95 | 5.40 | 1.59 | -215% | -562% | -0.85 |
| naval_propulsion_plant | -44% | -2% | -78% | 0.95 | 4.14 | 1.22 | -99% | -94% | 0.16 |
| naval_propulsion_plant | -64% | -1% | -175% | 0.98 | 4.66 | 1.37 | -141% | -163% | 0.50 |
| naval_propulsion_plant | -35% | 2% | -67% | 0.92 | 3.53 | 1.04 | -24% | -27% | 0.27 |
| naval_propulsion_plant | -38% | -2% | -70% | 0.94 | 4.75 | 1.40 | -62% | -68% | -0.21 |
| miami_housing | -32% | -1% | -46% | 0.94 | 1.48 | 0.18 | -21% | -20% | 0.74 |
| miami_housing | -46% | 1% | -88% | 0.96 | 1.98 | 0.25 | -41% | -48% | 0.53 |
| miami_housing | -25% | -2% | -31% | 0.95 | 1.69 | 0.21 | -18% | -19% | 0.75 |
| miami_housing | -34% | -2% | -49% | 0.96 | 1.82 | 0.23 | -23% | -22% | 0.77 |
| miami_housing | -42% | -2% | -72% | 0.91 | 1.91 | 0.24 | -24% | -32% | 0.57 |
| kin8nm | -14% | 1% | -17% | 0.98 | 3.68 | 0.72 | -3% | 0% | 0.43 |
| kin8nm | -18% | 0% | -22% | 0.97 | 3.67 | 0.72 | -20% | -22% | 0.36 |
| kin8nm | -17% | 1% | -22% | 0.96 | 3.56 | 0.70 | -12% | -10% | 0.39 |
| kin8nm | -24% | 0% | -32% | 0.96 | 3.46 | 0.68 | -26% | -24% | 0.36 |
| kin8nm | -20% | 0% | -25% | 0.95 | 3.51 | 0.69 | -20% | -19% | 0.31 |
| concrete_compressive_strength | -10% | 2% | -13% | 0.95 | 2.80 | 0.57 | -9% | -6% | 0.51 |
| concrete_compressive_strength | 9% | -3% | 8% | 0.95 | 2.68 | 0.54 | -15% | -13% | 0.59 |
| concrete_compressive_strength | -1% | -4% | 3% | 0.95 | 2.86 | 0.58 | -20% | -23% | 0.52 |
| concrete_compressive_strength | 16% | -5% | 16% | 0.91 | 2.49 | 0.50 | -10% | -10% | 0.46 |
| concrete_compressive_strength | 13% | -3% | 11% | 0.96 | 2.40 | 0.48 | -5% | -9% | 0.60 |
| cars | -23% | -3% | -33% | 0.95 | 1.93 | 0.33 | -3% | -5% | 0.75 |
| cars | -28% | -2% | -43% | 0.97 | 1.94 | 0.33 | -36% | -38% | 0.75 |
| cars | -27% | -3% | -39% | 0.94 | 2.06 | 0.35 | -34% | -33% | 0.70 |
| cars | -41% | 6% | -85% | 0.97 | 2.70 | 0.46 | -32% | -37% | 0.63 |
| cars | 8% | -4% | 10% | 0.90 | 1.68 | 0.28 | -32% | -15% | 0.48 |
| energy_efficiency | -12% | 0% | -14% | 0.99 | 1.44 | 0.42 | -8% | 1% | 0.97 |
| energy_efficiency | -4% | -2% | -2% | 0.97 | 0.69 | 0.20 | 14% | 18% | 0.98 |
| energy_efficiency | -24% | -3% | -37% | 0.94 | 1.23 | 0.36 | 8% | 2% | 0.92 |
| energy_efficiency | 10% | -3% | -1% | 0.96 | 1.07 | 0.31 | -8% | 6% | 0.96 |
| energy_efficiency | -7% | -10% | 0% | 0.89 | 0.84 | 0.24 | -1% | -32% | 0.93 |
| california_housing | -28% | 1% | -42% | 0.95 | 3.00 | 0.71 | -8% | -9% | 0.55 |
| california_housing | -20% | -2% | -25% | 0.97 | 2.85 | 0.68 | -1% | 0% | 0.68 |
| california_housing | -30% | 0% | -42% | 0.96 | 2.90 | 0.69 | -7% | -3% | 0.63 |
| california_housing | -16% | -1% | -19% | 0.93 | 2.70 | 0.64 | -4% | -3% | 0.54 |
| california_housing | -28% | 0% | -40% | 0.96 | 2.83 | 0.67 | -2% | -1% | 0.65 |
| airfoil_self_noise | -14% | 0% | -16% | 1.00 | 3.46 | 0.74 | -20% | -11% | 0.84 |
| airfoil_self_noise | 2% | -6% | 7% | 0.89 | 2.60 | 0.56 | -22% | -13% | 0.31 |
| airfoil_self_noise | -12% | -8% | -9% | 0.88 | 2.56 | 0.55 | -23% | -20% | 0.44 |
| airfoil_self_noise | -13% | -5% | -12% | 0.93 | 2.94 | 0.63 | -9% | -1% | 0.62 |
| airfoil_self_noise | -5% | 1% | -8% | 0.94 | 2.77 | 0.59 | -3% | 1% | 0.54 |
| QSAR_fish_toxicity | -16% | 1% | -20% | 0.99 | 4.78 | 0.74 | -8% | -9% | 0.44 |
| QSAR_fish_toxicity | -7% | -5% | -7% | 0.92 | 3.31 | 0.51 | -21% | -28% | 0.28 |
| QSAR_fish_toxicity | -5% | -2% | -8% | 0.95 | 3.17 | 0.49 | -6% | -8% | 0.39 |
| QSAR_fish_toxicity | -8% | -5% | -8% | 0.86 | 2.74 | 0.43 | -21% | -17% | 0.28 |
| QSAR_fish_toxicity | -17% | 2% | -28% | 0.90 | 3.03 | 0.47 | -10% | -9% | 0.32 |

## Appendix F Aggregated results for non-conformalized predictions

**Table 12: Test results for all models with complete-raw data at 95% quantiles aggregated over five seeds. For each metric, the mean and standard deviation of the performance across the seeds are separated by $\pm$. Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| MultiXGBs |  |  |  |  |  |  |  |  |
| space_ga | -1%$\pm$5% | 6%$\pm$2% | -7%$\pm$6% | 0.89$\pm$0.01 | 0.20$\pm$0.01 | -9%$\pm$1% | -10%$\pm$2% | 0.60$\pm$0.01 |
| cpu_activity | 25%$\pm$6% | 9%$\pm$1% | 12%$\pm$4% | 0.89$\pm$0.01 | 0.07$\pm$0.00 | 21%$\pm$1% | -1%$\pm$2% | 0.98$\pm$0.00 |
| naval_propulsion_plant | 156%$\pm$13% | 8%$\pm$1% | 58%$\pm$2% | 0.96$\pm$0.00 | 0.11$\pm$0.00 | -14%$\pm$4% | -12%$\pm$2% | 0.99$\pm$0.00 |
| miami_housing | 61%$\pm$5% | -7%$\pm$2% | 43%$\pm$3% | 0.82$\pm$0.01 | 0.06$\pm$0.00 | 4%$\pm$4% | 3%$\pm$3% | 0.91$\pm$0.01 |
| kin8nm | 10%$\pm$3% | 5%$\pm$2% | 4%$\pm$4% | 0.90$\pm$0.01 | 0.38$\pm$0.01 | -20%$\pm$1% | -22%$\pm$1% | 0.63$\pm$0.01 |
| concrete_compressive_strength | 13%$\pm$4% | 26%$\pm$6% | -11%$\pm$7% | 0.91$\pm$0.01 | 0.25$\pm$0.00 | -26%$\pm$6% | -40%$\pm$11% | 0.85$\pm$0.01 |
| cars | 18%$\pm$8% | 15%$\pm$4% | 2%$\pm$8% | 0.89$\pm$0.02 | 0.12$\pm$0.01 | -3%$\pm$3% | -1%$\pm$3% | 0.95$\pm$0.00 |
| energy_efficiency | 247%$\pm$89% | 22%$\pm$6% | 60%$\pm$17% | 0.88$\pm$0.02 | 0.05$\pm$0.01 | -15%$\pm$22% | -16%$\pm$17% | 1.00$\pm$0.00 |
| california_housing | 31%$\pm$5% | 0%$\pm$1% | 23%$\pm$4% | 0.88$\pm$0.01 | 0.28$\pm$0.00 | -1%$\pm$1% | -1%$\pm$1% | 0.81$\pm$0.00 |
| airfoil_self_noise | 61%$\pm$41% | 4%$\pm$9% | 31%$\pm$16% | 0.81$\pm$0.07 | 0.19$\pm$0.04 | -64%$\pm$32% | -73%$\pm$41% | 0.86$\pm$0.05 |
| QSAR_fish_toxicity | 10%$\pm$3% | 11%$\pm$3% | -1%$\pm$3% | 0.76$\pm$0.01 | 0.23$\pm$0.00 | 3%$\pm$3% | 1%$\pm$4% | 0.55$\pm$0.02 |
| MultiETs |  |  |  |  |  |  |  |  |
| space_ga | 9%$\pm$3% | -2%$\pm$1% | 9%$\pm$4% | 0.91$\pm$0.01 | 0.22$\pm$0.01 | -16%$\pm$2% | -18%$\pm$3% | 0.54$\pm$0.02 |
| cpu_activity | 27%$\pm$3% | -8%$\pm$0% | 27%$\pm$2% | 0.90$\pm$0.00 | 0.09$\pm$0.00 | -14%$\pm$1% | -20%$\pm$2% | 0.98$\pm$0.00 |
| naval_propulsion_plant | 219%$\pm$32% | -9%$\pm$0% | 71%$\pm$3% | 0.91$\pm$0.00 | 0.19$\pm$0.02 | -320%$\pm$41% | -409%$\pm$36% | 0.96$\pm$0.01 |
| miami_housing | 69%$\pm$3% | -13%$\pm$0% | 48%$\pm$1% | 0.86$\pm$0.00 | 0.08$\pm$0.00 | -10%$\pm$2% | -20%$\pm$1% | 0.90$\pm$0.00 |
| kin8nm | 10%$\pm$2% | -9%$\pm$0% | 17%$\pm$2% | 0.88$\pm$0.01 | 0.44$\pm$0.01 | -35%$\pm$2% | -38%$\pm$2% | 0.48$\pm$0.01 |
| concrete_compressive_strength | 9%$\pm$9% | -12%$\pm$2% | 19%$\pm$8% | 0.81$\pm$0.01 | 0.24$\pm$0.02 | -67%$\pm$7% | -94%$\pm$9% | 0.76$\pm$0.01 |
| cars | 11%$\pm$6% | 16%$\pm$2% | -5%$\pm$7% | 0.83$\pm$0.02 | 0.09$\pm$0.01 | 3%$\pm$3% | 1%$\pm$2% | 0.95$\pm$0.00 |
| energy_efficiency | -4%$\pm$5% | 0%$\pm$7% | -5%$\pm$14% | 0.65$\pm$0.03 | 0.02$\pm$0.00 | 3%$\pm$3% | 0%$\pm$1% | 1.00$\pm$0.00 |
| california_housing | 31%$\pm$2% | -7%$\pm$0% | 29%$\pm$1% | 0.90$\pm$0.00 | 0.41$\pm$0.01 | -15%$\pm$1% | -21%$\pm$1% | 0.71$\pm$0.01 |
| airfoil_self_noise | 18%$\pm$25% | -15%$\pm$4% | 25%$\pm$17% | 0.83$\pm$0.04 | 0.26$\pm$0.06 | -117%$\pm$45% | -141%$\pm$51% | 0.80$\pm$0.08 |
| QSAR_fish_toxicity | 12%$\pm$2% | -9%$\pm$1% | 19%$\pm$2% | 0.82$\pm$0.01 | 0.28$\pm$0.00 | -6%$\pm$1% | -11%$\pm$1% | 0.53$\pm$0.01 |
| MultiMLPs |  |  |  |  |  |  |  |  |
| space_ga | 8%$\pm$1% | 0%$\pm$3% | 7%$\pm$3% | 0.81$\pm$0.01 | 0.14$\pm$0.00 | 0%$\pm$1% | -1%$\pm$1% | 0.75$\pm$0.01 |
| cpu_activity | 8%$\pm$3% | -9%$\pm$4% | 15%$\pm$6% | 0.72$\pm$0.02 | 0.05$\pm$0.00 | 7%$\pm$2% | 5%$\pm$2% | 0.98$\pm$0.00 |
| naval_propulsion_plant | 21%$\pm$18% | 0%$\pm$3% | 16%$\pm$13% | 0.92$\pm$0.01 | 0.07$\pm$0.01 | -45%$\pm$25% | -36%$\pm$20% | 1.00$\pm$0.00 |
| miami_housing | 9%$\pm$4% | 2%$\pm$3% | 6%$\pm$6% | 0.81$\pm$0.02 | 0.05$\pm$0.00 | -3%$\pm$2% | 4%$\pm$2% | 0.91$\pm$0.00 |
| kin8nm | 2%$\pm$7% | 8%$\pm$5% | -6%$\pm$10% | 0.81$\pm$0.02 | 0.13$\pm$0.01 | 7%$\pm$2% | 7%$\pm$2% | 0.93$\pm$0.00 |
| concrete_compressive_strength | 98%$\pm$13% | -23%$\pm$7% | 61%$\pm$5% | 0.56$\pm$0.04 | 0.06$\pm$0.00 | 12%$\pm$5% | 15%$\pm$4% | 0.91$\pm$0.01 |
| cars | 0%$\pm$14% | 6%$\pm$12% | -9%$\pm$30% | 0.76$\pm$0.08 | 0.07$\pm$0.01 | -3%$\pm$3% | 0%$\pm$3% | 0.95$\pm$0.00 |
| energy_efficiency | 82%$\pm$18% | 25%$\pm$23% | 31%$\pm$11% | 0.62$\pm$0.04 | 0.02$\pm$0.00 | 32%$\pm$3% | 33%$\pm$3% | 1.00$\pm$0.00 |
| california_housing | -14%$\pm$2% | 8%$\pm$1% | -26%$\pm$3% | 0.89$\pm$0.01 | 0.31$\pm$0.01 | 7%$\pm$0% | 9%$\pm$1% | 0.82$\pm$0.00 |
| airfoil_self_noise | 74%$\pm$41% | 0%$\pm$6% | 39%$\pm$15% | 0.75$\pm$0.02 | 0.08$\pm$0.00 | 18%$\pm$4% | 16%$\pm$5% | 0.97$\pm$0.00 |
| QSAR_fish_toxicity | -6%$\pm$4% | 11%$\pm$3% | -19%$\pm$5% | 0.76$\pm$0.03 | 0.23$\pm$0.01 | 4%$\pm$5% | 7%$\pm$4% | 0.55$\pm$0.04 |

**Table 13: Test results for all models with missing-raw data at 95% quantiles aggregated over five seeds. For each metric, the mean and standard deviation of the performance across the seeds are separated by $\pm$. Performance over the baseline is highlighted in bold.**
|  | Interval Predictions | Point Predictions |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Relative | Absolute | Relative | Absolute |  |  |  |  |
| Dataset | $\Delta$CWR | $\Delta$PICP | $\Delta$NMPIW | PICP | NMPIW | $\Delta$RMSE | $\Delta$MAE | $\text{R}^{2}$ |
| MultiXGBs |  |  |  |  |  |  |  |  |
| space_ga | -5%$\pm$6% | 4%$\pm$2% | -12%$\pm$7% | 0.86$\pm$0.02 | 0.22$\pm$0.01 | -7%$\pm$4% | -9%$\pm$3% | 0.45$\pm$0.06 |
| cpu_activity | -5%$\pm$19% | 12%$\pm$8% | -37%$\pm$46% | 0.89$\pm$0.06 | 0.14$\pm$0.04 | -1%$\pm$21% | -17%$\pm$13% | 0.83$\pm$0.07 |
| naval_propulsion_plant | 105%$\pm$29% | -16%$\pm$6% | 56%$\pm$6% | 0.70$\pm$0.06 | 0.18$\pm$0.02 | -14%$\pm$20% | -29%$\pm$35% | 0.77$\pm$0.05 |
| miami_housing | 16%$\pm$13% | -5%$\pm$5% | 10%$\pm$16% | 0.82$\pm$0.06 | 0.10$\pm$0.02 | -14%$\pm$11% | -18%$\pm$10% | 0.72$\pm$0.10 |
| kin8nm | -5%$\pm$2% | 3%$\pm$2% | -14%$\pm$5% | 0.88$\pm$0.01 | 0.48$\pm$0.01 | -3%$\pm$3% | -4%$\pm$4% | 0.40$\pm$0.02 |
| concrete_compressive_strength | 2%$\pm$3% | 8%$\pm$4% | -10%$\pm$3% | 0.77$\pm$0.03 | 0.30$\pm$0.01 | -16%$\pm$12% | -20%$\pm$14% | 0.58$\pm$0.08 |
| cars | -15%$\pm$14% | 26%$\pm$9% | -63%$\pm$26% | 0.85$\pm$0.04 | 0.27$\pm$0.03 | -30%$\pm$22% | -35%$\pm$24% | 0.66$\pm$0.15 |
| energy_efficiency | 10%$\pm$26% | 11%$\pm$6% | -24%$\pm$43% | 0.92$\pm$0.02 | 0.22$\pm$0.05 | -186%$\pm$265% | -153%$\pm$188% | 0.91$\pm$0.08 |
| california_housing | 26%$\pm$7% | -2%$\pm$3% | 21%$\pm$6% | 0.88$\pm$0.03 | 0.40$\pm$0.03 | -4%$\pm$4% | -5%$\pm$5% | 0.69$\pm$0.05 |
| airfoil_self_noise | -2%$\pm$18% | -3%$\pm$4% | -4%$\pm$17% | 0.73$\pm$0.04 | 0.34$\pm$0.06 | -37%$\pm$31% | -43%$\pm$36% | 0.41$\pm$0.12 |
| QSAR_fish_toxicity | -2%$\pm$3% | 0%$\pm$5% | -6%$\pm$7% | 0.73$\pm$0.03 | 0.26$\pm$0.02 | -4%$\pm$4% | -2%$\pm$3% | 0.36$\pm$0.04 |
| MultiETs |  |  |  |  |  |  |  |  |
| space_ga | 4%$\pm$3% | -5%$\pm$1% | 7%$\pm$4% | 0.88$\pm$0.02 | 0.24$\pm$0.01 | -12%$\pm$4% | -14%$\pm$4% | 0.40$\pm$0.05 |
| cpu_activity | -16%$\pm$10% | -5%$\pm$1% | -20%$\pm$18% | 0.93$\pm$0.01 | 0.20$\pm$0.03 | -30%$\pm$23% | -54%$\pm$20% | 0.79$\pm$0.10 |
| naval_propulsion_plant | 58%$\pm$5% | -17%$\pm$4% | 46%$\pm$3% | 0.82$\pm$0.05 | 0.41$\pm$0.03 | -36%$\pm$21% | -93%$\pm$48% | 0.71$\pm$0.05 |
| miami_housing | 20%$\pm$14% | -10%$\pm$4% | 21%$\pm$13% | 0.87$\pm$0.05 | 0.14$\pm$0.03 | -27%$\pm$11% | -37%$\pm$11% | 0.67$\pm$0.11 |
| kin8nm | 5%$\pm$2% | -8%$\pm$2% | 12%$\pm$4% | 0.87$\pm$0.02 | 0.50$\pm$0.02 | -11%$\pm$2% | -13%$\pm$3% | 0.32$\pm$0.03 |
| concrete_compressive_strength | 1%$\pm$5% | -14%$\pm$3% | 13%$\pm$2% | 0.80$\pm$0.02 | 0.37$\pm$0.01 | -35%$\pm$21% | -50%$\pm$25% | 0.49$\pm$0.06 |
| cars | -23%$\pm$6% | -8%$\pm$5% | -35%$\pm$14% | 0.83$\pm$0.04 | 0.24$\pm$0.02 | -33%$\pm$18% | -31%$\pm$18% | 0.66$\pm$0.09 |
| energy_efficiency | -15%$\pm$16% | 0%$\pm$9% | -36%$\pm$42% | 0.85$\pm$0.05 | 0.15$\pm$0.01 | -73%$\pm$74% | -84%$\pm$64% | 0.94$\pm$0.04 |
| california_housing | 21%$\pm$7% | -7%$\pm$2% | 22%$\pm$5% | 0.91$\pm$0.02 | 0.52$\pm$0.03 | -14%$\pm$3% | -17%$\pm$3% | 0.61$\pm$0.05 |
| airfoil_self_noise | -7%$\pm$18% | -20%$\pm$2% | 8%$\pm$16% | 0.76$\pm$0.03 | 0.42$\pm$0.08 | -65%$\pm$54% | -93%$\pm$70% | 0.33$\pm$0.11 |
| QSAR_fish_toxicity | 10%$\pm$7% | -9%$\pm$3% | 16%$\pm$3% | 0.82$\pm$0.03 | 0.33$\pm$0.03 | -8%$\pm$8% | -10%$\pm$9% | 0.39$\pm$0.09 |
| MultiMLPs |  |  |  |  |  |  |  |  |
| space_ga | -11%$\pm$5% | -15%$\pm$2% | 1%$\pm$6% | 0.68$\pm$0.03 | 0.16$\pm$0.01 | -40%$\pm$23% | -34%$\pm$15% | 0.21$\pm$0.23 |
| cpu_activity | -50%$\pm$16% | 22%$\pm$14% | -193%$\pm$118% | 0.86$\pm$0.10 | 0.18$\pm$0.07 | -15%$\pm$19% | -23%$\pm$17% | 0.75$\pm$0.12 |
| naval_propulsion_plant | -52%$\pm$19% | -14%$\pm$24% | -102%$\pm$56% | 0.63$\pm$0.18 | 0.34$\pm$0.07 | -108%$\pm$66% | -183%$\pm$195% | -0.03$\pm$0.47 |
| miami_housing | -33%$\pm$15% | 13%$\pm$10% | -92%$\pm$70% | 0.85$\pm$0.06 | 0.12$\pm$0.04 | -25%$\pm$8% | -28%$\pm$11% | 0.67$\pm$0.10 |
| kin8nm | -11%$\pm$5% | -5%$\pm$2% | -8%$\pm$6% | 0.77$\pm$0.01 | 0.35$\pm$0.02 | -16%$\pm$8% | -15%$\pm$9% | 0.37$\pm$0.04 |
| concrete_compressive_strength | -15%$\pm$13% | -13%$\pm$28% | -10%$\pm$48% | 0.60$\pm$0.22 | 0.22$\pm$0.11 | -12%$\pm$5% | -12%$\pm$6% | 0.54$\pm$0.05 |
| cars | -51%$\pm$4% | 43%$\pm$7% | -211%$\pm$34% | 0.93$\pm$0.02 | 0.29$\pm$0.03 | -28%$\pm$12% | -26%$\pm$13% | 0.66$\pm$0.10 |
| energy_efficiency | -13%$\pm$18% | 9%$\pm$15% | -30%$\pm$25% | 0.66$\pm$0.13 | 0.09$\pm$0.04 | 1%$\pm$9% | -1%$\pm$16% | 0.95$\pm$0.03 |
| california_housing | -17%$\pm$6% | 4%$\pm$3% | -28%$\pm$14% | 0.88$\pm$0.03 | 0.44$\pm$0.04 | -5%$\pm$2% | -3%$\pm$3% | 0.61$\pm$0.06 |
| airfoil_self_noise | 2%$\pm$16% | -2%$\pm$5% | -1%$\pm$21% | 0.75$\pm$0.07 | 0.25$\pm$0.08 | -15%$\pm$8% | -9%$\pm$8% | 0.55$\pm$0.18 |
| QSAR_fish_toxicity | -17%$\pm$5% | 6%$\pm$6% | -30%$\pm$11% | 0.76$\pm$0.04 | 0.28$\pm$0.03 | -13%$\pm$7% | -14%$\pm$8% | 0.34$\pm$0.06 |

## Appendix G Example of learning with non-normality

To illustrate how SEMF adapts to non-normal outcomes, we provide an example from the naval_propulsion_plant dataset []. Figure 2 shows the distribution of the ground-truth $y$ variable which in this case is gt_compressor_decay_state_coefficient. The values are uniformly distributed, and we only standardize the values without changing the shape of the distribution.

Figure: Figure 2: Distribution of the standardized outcome ($y$) variable for the naval_propulsion_plant dataset which shows that $y$ is uniformly distributed prior to any training.
Refer to caption: extracted/5895273/assets/npp_standardized_y_dist.png

After training our SEMF model under the normality assumption with the ideal hyper-parameters (and a seed of 0), sampling from a normal distribution for the $z$ dimension, we infer on some randomly sampled test instances that provide us with the prediction intervals in Figure 3. The ‘SEMF intervals’ can be compared with XGBoost quantile regression, constituting our baseline. This figure shows that SEMF’s predicted intervals are better than the baseline. This plot alone does not tell us much about the predicted output distribution. Therefore, we provide Figure 4. The last plot shows that for a handful of the instances, the predicted values can take any shape and are not necessarily normal.

Figure: Figure 3: Predicted intervals and true values on 100 randomly selected test samples for SEMF MultiXGBs and XGBoost quantile regression with complete data at 95% quantiles (according to Eq 13).
Refer to caption: extracted/5895273/assets/npp_predicted_quantiles.png

Figure: Figure 4: Violin plot for the first ten instances of the test set where both the ground truth and 50 values inferred by SEMF (according to Eq 12) are generated for each instance. We have added jitter here, so the points do not perfectly align along the x-axis.
Refer to caption: extracted/5895273/assets/npp_violin_plot.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: [W2] Can you elaborate on why you have decided to propose a new score CWR instead of using the existing interval score as suggested in the above comments? 

[W4] Can you please comment on why the results of SEMF on missing data are inferior to results on complete data when compared to baseline methods.
soundness: 2
strengths: Importance: Developing methods and models for improving predictions with their uncertainty estimates is significant and important for applications of predicted models especially when incorrect predictions may carry a high risk. Model agnostic framework the work aims to develop would be a great advantage as it does not have to be tailored to specifics of individual models. 

Novelty: The proposed EM framework is novel in context of uncertainty assessment. The methodology of EM and its use in unsupervised and semi-supervised tasks settings is not new.  In addition, the paper introduces a new evaluation metric Coverage-Width Ratio (CWR) that accounts for both the coverage and the precision of the prediction intervals.

Experiments: the experiments are done on 11 problems. Three different prediction methods, and their existing prediction interval estimates are considered. However, the number of baseline models considered does not appear to be sufficient to support the model-agnostic objective. 

Code: Authors provide the code for the reproducibility of the reported results
summary: Quantification of uncertainty of a given prediction made by the models is critical for a variety of downstream applications. One way to measure the uncertainty of a prediction is to use quantile estimates and corresponding prediction intervals. The paper develops and explores SEMF: a Semi-supervised EM Framework for generating prediction intervals that are model agnostic and can work with incomplete data. The gist of the framework is to convert inputs to a latent space, that is in turn used to predict the outputs, much like autoencoder architectures. The proposed interval estimation framework is tested on 11 problems and three different baseline prediction models.
weaknesses: [W1] Intuition. Lack of intuition and argument supporting the benefit of the SEMF framework. The paper says it aims to leverage latent variable modeling framework for uncertainty estimation in predictions. The intuition and justification of this step and design is however not very well argued in the paper. Adding the text covering the intuition aspect would greatly enhance the readability and clarity of the paper and its steps. Along the same lines it would be great to see arguments or intuition why this approach could improve upon alternative quantile estimation methods. 


[W2] Experiment – baselines. The evaluation is limited as it only considers quantile regression as the baseline of comparison that generates prediction interval. Perhaps [1,2], or other relevant methods can be considered as additional baselines. Also, the evaluation is presented on relatively simple datasets: number of features in [7,22] and number of samples in [768, 21K]. It is hard to justify the applicability of the model at scale i.e., with a greater number of features and/or number of samples.

[W3] Experiment- metrics. The results for interval estimates consider PICP, PIW (NPIW) and new CWR metrics, where CWR attempts to combine two different aspects of the interval estimates,  However, there is another existing combined interval score (Gneiting , 2007) that attempts to combine two aspects of the interval estimates and could have been used instead   
T. Gneiting, F. Balabdaoui, AE Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69 (2):243–268, 2007 

[W4] Interpretation of results and conclusions: The results and their discussion are limited, and it is unclear whether the objectives of the development are supported by the experiments and results.  First, using only three model baselines is limiting in terms of support of the model agnostic objective. Second, somewhat surprising are missing data experiments where latent variables in the model should adapt and handle better the data missingness.  What is the insight for these results?  

[W5] Running time: As authors note in the limitations, the usefulness of this approach is limited by its high computation complexity. However, the paper does not report the train and inference time. For completeness, the paper can benefit from including these times to assess the extent of the slowdown caused by SEMF.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - In L153, why is the $x_1[j]$ notation needed? Isn't this already expressed as $x_{1,j}$?
- In equation 12, what is $x_{1}^{(nm)}[j]$? I don't think $x_{1}^{(nm)}$ is defined.
- I don't quite understand how the missing values were implemented - in L299, when you say "except for the first feature", does that mean the first feature is never missing, and half of the rest of the features are masked out, chosen at random?
soundness: 3
strengths: - Adapting the EM algorithm seems pretty novel and the algorithm is well-motivated and sound. 
- There have been many methods introduced in the uncertainty literature which aim to produce prediction intervals, and this work would fit alongside those methods. 
- Consideration for missing data is an interesting application setting which prior works in uncertainty to not consider often. 
- Overall, the writing quality is good and mostly easy to follow
summary: This paper introduces an algorithm for producing prediction intervals (PI) by adapting the EM algorithm in a supervised learning framework. The modeling takes into consideration possible missing input features. The empirical evaluation considers the 95% PI (its coverage and width) against baseline models.
weaknesses: - The proposed method incorporates conformal prediction at the end, which guarantees correct coverage. the only other metric is the PI width. In that case, is the benefit of the method just in producing more tightly clustered samples which lead to tighter PI?
- Is it correct to understand the proposed method as just producing samples from the modeled underlying distribution? There are other metrics which are sample-based that could provide a more holistic evaluation of the quality of the predictive distribution, like the energy score.
- As I understand it, this method requires a separate model per input feature dimension, which seems prohibitively expensive, either for large models or large input dimensions.
- Despite the emphasis on adaptability to missing data in the methods sections, its performance on missing data seems a bit unfortunate, but I appreciate the authors' frankness in providing the results.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: - The authors assume independent distributions among the variables. Based on this assumption, is the hypothesis of the latent variable $z$ not that important? Could we establish the relationship between 
$y$ and $x$ directly instead?
- There is a significant amount of research on interval prediction (which the authors also mention), but the authors do not seem to compare their method with these existing approaches (in the absence of missing data). For cases with missing data, using simple methods (such as interpolation) for comparison would also be straightforward.
- The SEMF method seems to require tuning multiple hyperparameters, such as the number of Monte Carlo samples, the number of latent nodes, and the standard deviation. This may demand considerable experimental and computational resources. Are there any general empirical settings that could be recommended?
soundness: 2
strengths: - The SEMF proposed by the authors is a model-agnostic method, meaning it can be integrated with various machine learning models, providing high applicability and flexibility.
- The problem addressed by the authors is often overlooked in the real world, specifically the presence of missing data and the uncertainty estimation of provided predictions.
- The authors conducted experiments on a large number of datasets to validate the effectiveness of their method.
summary: This paper presents a novel prediction interval generation framework called SEMF (Supervised Expectation-Maximization Framework). SEMF is a general, model-agnostic approach that can be applied to complete datasets or datasets containing missing data. It extends the traditional EM (Expectation-Maximization) algorithm, which is typically used for unsupervised learning, by applying it to supervised learning for uncertainty estimation through latent variable modeling.
weaknesses: - The writing in this paper is unclear. I suggest that the authors introduce the research problem setting either before Section 2 or at the beginning of Section 2, rather than listing formulas.
- The theoretical analysis in the paper assumes independent distributions among the variables, but real-world situations are often more complex. I believe the authors' investigation of this issue is not sufficiently thorough.
- There appears to be a substantial amount of prior research [1, 2, 3] on interval data prediction, which is not mentioned in this paper.

[1] Billard, L. and Diday, E. Regression analysis for interval valued data. In Data Analysis, Classification, and Related Methods, pp. 369–374. Springer, 2000.


[2] Sadeghi, J., De Angelis, M., and Patelli, E. Efficient training of interval neural networks for imprecise training data. Neural Networks, 118:338–351, 2019.


[3] Yang, Z., Lin, D. K., and Zhang, A. Interval-valued data prediction via regularized artificial neural network. Neurocomputing, 331:336–345, 2019.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - Why do we need the double index r, s when sampling first z and then y in EQ 17?  Just sampling z_r and then y_r would be conceptually enough I believe.
- Why does mini-batching make your results unstable? Is that a common reason or something specific to your method?
soundness: 3
strengths: - The approach model agnostic and so can in theory easily applied on any supervised learning baseline model.
- The overall algorithm is well described and easy to follow, at least for someone who has worked on EM algorithms.
summary: This paper proposes an Expectation Maximization (EM) Approach for supervised learning problems. The algorithm is described and evaluated empirically on several benchmark models and datasets. I am not yet convinced by the approach but happy to change my mind during the discussion period.
weaknesses: - Using MC sampling to approximate the posterior over the latent variables z is a potentially inaccurate approach, especially as z increases in dimensionality. This is the reason, in Bayesian inference, we would e.g. use MCMC sampling not MC sampling from the prior. Can the authors comment on why this is not a problem in their approach? Asked differently, how large did the number of samples R have to be in their cases to produce good results? How does R affect the quality of the results?
- The results not super convincing across the board. Especially, I was surprised to see the point estimation performance to drop when using the EM approach. Can the authors explain that?
- The missing value setting is strange to me. Why investigate an MCAR setting, which implies ignorable missingness. I believe a MAR setting where missingness is in principle recoverable would be a much more relevant scenario.