# Aggregated Conformal Prediction

Lars Carlsson, Martin Eklund, Ulf Norinder

► To cite this version:

Lars Carlsson, Martin Eklund, Ulf Norinder. Aggregated Conformal Prediction. 10th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Sep 2014, Rhodes, Greece. pp.231-240, 10.1007/978-3-662-44722-2_25. hal-01391050

HAL Id: hal-01391050

https://inria.hal.science/hal-01391050

Submitted on 2 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Aggregated conformal prediction

Lars Carlsson¹*, Martin Eklund²**, and Ulf Norinder³***

¹ AstraZeneca Research and Development, SE-431 83 Mölndal, Sweden

² Department of Surgery, University of California San Francisco (UCSF), 1600 Divisadero St, San Francisco CA 94143, USA

³ H. Lundbeck A/S, Ottiliavej 9, 2500 Valby, Denmark

**Abstract.** We present the aggregated conformal predictor (ACP), an extension to the traditional inductive conformal prediction (ICP) where several inductive conformal predictors are applied on the same training set and their individual predictions are aggregated to form a single prediction on an example. The results from applying ACP on two pharmaceutical data sets (CDK5 and GNRHR) indicate that the ACP has advantages over traditional ICP. ACP reduces the variance of the prediction region estimates and improves efficiency. Still, it is more conservative in terms of validity than ICP, indicating that there is room for further improvement of efficiency without compromising validity.

## 1 Introduction

Quantitative Structure-Activity Relationship (QSAR) modeling for predicting properties, e.g. solubility or toxicity, of chemical compounds using statistical learning techniques is a widespread approach within the pharmaceutical industry to prioritize compounds for experimental testing or to alert for potential toxicity. In making informed decisions based on predictions from QSAR models, the confidence in such predictions is of vital importance and conformal predictors have been successfully applied to the drug discovery setting [1]. In particular, we have shown that a Mondrian inductive conformal predictor is efficient (i.e. informative) and almost valid when applied to binary categorical data; exact validity was not achieved due to deviations from the exchangeability assumption [2]. Furthermore, we have demonstrated that an inductive conformal predictor (ICP) applied to regression data from the pharmaceutical industry was valid. However, for regression the prediction regions were in many cases as wide or wider than the possible ranges of the true responses (i.e. the range of the experimental assay that generates the true label). An important problem is thus to improve the efficiency of the ICPs when applied in the QSAR domain.

There are many ways to improve efficiency of conformal predictions, for example by using improved nonconformity scores, choosing different machine-learning methods, or using a transductive approach [3]. The transductive approach is the

<footnote>* lars.a.carlsson@astrazeneca.com</footnote>

<footnote>** martin.eklund@farmbio.uu.se</footnote>

<footnote>*** ulfn@lundbeck.com</footnote>
most appealing in terms of validity, but is often computationally costly and the nonconformity scores can be difficult to compute.

Another interesting approach to improve efficiency is the cross conformal predictor (CCP) [4]. Here the training data is divided into separate non-overlapping folds and each fold is used as a calibration set and the remainder of the data is used as a proper training set. This division allows for more data to be used for calibration and p-values are averaged over all folds. Similarly, the bootstrap conformal predictor (BCP) bootstraps datasets and uses the out-of-bag examples as a calibration set. p-values are then averages across all bootstrap replications.

In this paper we attempt to generalize the BCP and the CCP in what we term the aggregated inductive conformal predictor (ACP). We will empirically assess the ACP using data from the pharmaceutical domain and we show through a theoretical argument and experiments that ACP seems to have advantages over the standard ICP.

## 2 Aggregated conformal predictor

Consider the standard prerequisite for a description of a conformal predictor (CP), a bag of examples {$z_1, ..., z_i, ..., z_l$} drawn from an exchangeable distribution $Q$. Each example $z_i = (x_i, y_i)$ can be described by its object $x_i \in \mathbf{X}$ and its label $y_i \in \mathbf{Y}$. The labels can be either categorical or continuous. For an inductive conformal predictor (ICP) the bag {$z_i$} is partitioned into two different bags, one holding the proper training examples {$z_1, ..., z_m$} and the other holding the calibration examples {$z_{m+1}, ..., z_l$}. The ICP p-value is then computed as

$$p = \frac{|\{j = m + 1, ..., l : \alpha_j \ge \alpha_{l+1}\}|}{l - m + 1}$$

The prediction region of an ICP is determined by the "borderline" p-value, $p_t$, i.e. the smallest value p can obtain and still satisfy $p > \epsilon$. We can thus view $p_t$ as the $\epsilon$th sample quantile (estimated from above)

$$p_t = U_{l-m}^{-1}(\epsilon),$$

where $U_{l-m}$ is the empirical cumulative probability distribution of the p-values defined by

$$U_{l-m}(p) = \frac{1}{l-m+1} \sum_{j=m+1}^{l} I(p_j < p),$$

where $I(\cdot) = 1$ if $p_j < p$ and $I(\cdot) = 0$ otherwise. We now introduce definitions of Exchangeable resampling and Consistent resampling, after which we define what we mean by an aggregated conformal predictor (ACP).

**Definition 1 (Exchangeable resampling).** Let {$z_1^*, ..., z_n^*$} be a bag of examples resampled from the empirical distribution $Q_l$. We call this resampling
exchangeable if

$$P\{(z_1^*, \dots, z_n^*)\} = P\{(z_{\pi(1)}^*, \dots, z_{\pi(n)}^*)\},$$

where $\pi$ is any permutation of $\{1, \dots, n\}$.

**Definition 2 (Consistent resampling).** Let $T = T(z_1, \dots, z_l, Q)$ be a statistic and $T^* = T(z_1^*, \dots, z_n^*, Q_l)$ be an exchangeably resampled version of $T$. Further, let $G_l$ and $G_l^*$ be the probability distributions of $T$ and $T^*$, respectively. We call the sampling process consistent (with respect to $T$) if

$$\sup_z |G_l - G_l^*| \to 0 \text{ as } l \to \infty \text{ and } n \to \infty.$$

**Definition 3 (ACP: Aggregated Conformal Predictor).** The following procedure is repeated $B$ times, for $b = 1, \dots, B$: Resample a bag $\{z_1^*, \dots, z_{n_b}^*\}$ of examples from $\{z_1, \dots, z_l\}$ using a consistent resampling procedure with respect to $\alpha_t$. Compute the ICP p-value using the resampled bag,

$$p_b^* = \frac{|\{j = m_b + 1, \dots, n_b : \alpha_j^* \ge \alpha_{n_b+1}^*\}|}{n_b - m_b + 1}, \quad (1)$$

where $\alpha_j^*$ are the nonconformity scores computed using $\{z_1^*, \dots, z_{n_b}^*\}$ ($m_b$ and $n_b$ are indexed with $b$ to make explicit that they may differ for different values of $b$). We define the ACP p-value as

$$p_B = \frac{1}{B} \sum_{b=1}^{B} p_b^* \quad (2)$$

and the corresponding prediction region as

$$\Gamma^\epsilon(z_1, \dots, z_l, x_{l+1}) := \{y | p_B > \epsilon\}. \quad (3)$$

A smoothed ACP can be defined analogously.

We note that the cross-conformal predictor and the bootstrapped conformal predictor suggested by Vovk [5] are two examples of ACPs.

**Proposition 1.** The aggregated conformal predictor is conservatively valid.

*Proof.* Since we use an exchangeable resampling procedure to construct the ACP and since an ICP is conservatively valid (Proposition 4.1, [3]), each resampled ICP in the ACP is conservatively valid (by symmetry). From this follows that the ACP also is conservatively valid.

*Remark 1.* Proposition 1 only holds unconditionally. The situation is different conditional on the particular dataset we have observed.
## 2.1 How does the ACP improve on the ICP?

For a $p$ in an ICP, $p_t$ is a hard threshold in the sense that the label $y$ corresponding to $p$ is either inside or outside the prediction region. Heuristically, the ACP averages over thresholds varying around $p_t$ (since $p_t^*$ based on a resampled bag $\{z_1^*, ..., z_l^*\}$ fluctuates around $p_t$), resulting in a smoothed threshold estimate with decreased variance compared to the estimate $p_t$.

We can use the method used by Bühlmann and Yu [6] to analyze this in a bit more detail. Consider the function

$$ \delta(p) = I(p_t < p), $$

which indicates whether a $p$ is smaller or larger than the borderline value $p_t$ in an ICP (and thus if its corresponding label $y$ either is inside or outside the prediction region). The sample quantile estimate $p_t$ follows a normal distribution with mean $\epsilon = U^{-1}(\epsilon)$ and variance

$$ \sigma^2 = \frac{(1 - \epsilon)\epsilon}{(l - m + 1)[f(q)]^2} = \frac{1 - \epsilon}{(l - m + 1)\epsilon}, $$

where $F$ is the population cumulative distribution function with density function $f$ [7]. For a $p$ in the neighborhood of $\epsilon$

$$ p = p(c) = \epsilon + c\sigma\sqrt{l - m + 1} \quad (4) $$

we have the approximation

$$ \delta(p(c)) \approx I(W < c), \quad W \sim N(0, 1), \quad (5) $$

where $W$ is the limiting random quantity from the asymptotic distribution of $p_t$ (because of the construction of $p(c)$ in Equation (4)). For a fixed $c$, this is a hard threshold function of $W$. It follows that

$$ \begin{aligned} \mathrm{E}[\delta(p(c))] & \rightarrow P(W < c) = \Phi(c) \quad \text{as } l \rightarrow \infty \text{ and } l - m \rightarrow \infty \\ \mathrm{Var}[\delta(p(c))] & \rightarrow \Phi(c)(1 - \Phi(c)) \quad \text{as } l \rightarrow \infty \text{ and } l - m \rightarrow \infty, \end{aligned} \quad (6) $$

where $\Phi(\cdot)$ is the cumulative distribution function of the standard normal distribution. Note that the variance does not converge to zero; $\delta(p(c))$ assumes the values 0 and 1 with a positive probability even as $l$ tends to infinity. However, for the ACP the situation looks different,

$$ \begin{aligned} \delta_B(p(c)) &= \mathrm{E}^*[I(p_t^* \le p(c))] \\ &= \mathrm{E}^*\left[I\left(\sqrt{l-m+1}(p_t^* - p_t)/\sigma \le \sqrt{l-m+1}(p(c) - p_t)/\sigma\right)\right] \\ &= \Phi(\sqrt{l-m+1}(p(c) - p_t)) + o_p(1) \approx \Phi(c-W), \quad W \sim N(0, 1). \end{aligned} $$

where the first approximation over the second equal sign follows because we by definition of the ACP have a consistent resampling process. Comparing with
Equation (5) for an ICP, the ACP produces a smoothed decision function of Z and therefore reduces variance. Again, following Bühlmann and Yu [6], we can study the case $p = p(0) = \epsilon$, i.e. when we are right at the population threshold and therefore has maximum variance. Then

$$ \delta_B(p(0)) \rightarrow \Phi(-W) \sim U[0, 1] $$

and, therefore,

$$ \begin{aligned} E[\delta_B(p(0))] &\rightarrow E[U] = 1/2 \text{ as } l \rightarrow \infty \\ \text{Var}[\delta_B(p(0))] &\rightarrow \text{Var}(U) = 1/12 \text{ as } l \rightarrow \infty. \end{aligned} \quad (7) $$

Comparing Equation (6) to Equation (7), we see that the variance is reduced to one third for ACP compared to ICP.

## 3 Empirical results of ACP

We used two different machine learning methods; the support vector machine (SVM) [8] implemented in the Java library version of libsvm [9] with a Gaussian radial basis kernel function

$$ K(x, x') = \exp(-\gamma \|x - x'\|^2), $$

with $\gamma = 0.002$ and $C = 50$, and Random Forest (RF) [10], for which the default settings were used and each ensemble contained 100 trees.

Following Equation (16) in [11], we defined the nonconformity measure used in combination with SVM according to

$$ \alpha_i = \frac{|y_i - \hat{y}_i|}{\exp(\hat{\mu}_i)}, \quad (8) $$

where $\hat{\mu}_i$ is the prediction of the value $\ln(|y_i - \hat{y}_i|)$ produced by a support vector regression machine trained on the proper training sets. After training the underlying SVM of the ICP, we calculate the residuals $|y_j - \hat{y}_j|$ for all proper training examples $j = 1, ..., m$ and train an SVM on the pairs $(x_i, \ln(|y_i - \hat{y}_i|))$. Measure (8) normalizes the absolute prediction error with the predicted accuracy of the SVM on a given example. The nonconformity measure used with RF was

$$ \alpha_i = \frac{|y_i - \hat{y}_i|}{\hat{\nu}_i}, $$

where $\hat{\nu}_i$ is the RF prediction of the value $|y_i - \hat{y}_i|$ with the same settings as for the other RF model.

Two public dataset (CDK5 and GNRHR) from the pharmaceutical domain was used [12]. The datasets consist of 230 and 198 examples, respectively. Each example (chemical compound) was described (characterised) by so-called signature descriptors [13] in the same way as described in [2].
The data was randomly split into two parts: A training set (80% of the original data) and a working set (20%). This procedure was repeated 50 times as to generate 50 training and working set, respectively. Furthermore, each training set was then, subsequently, randomly split into a proper training set (70% of the training set) and a calibration set (30%), similar to the 2:1 recommendation in [5]. This random selection of proper training and calibration examples was, in turn, repeated 100 times enabling the construction of 100 inductive conformal predictors for each working set. This sampling procedure is often called the *m-n* sampling or non-replacement subsampling in the bootstrap literature and consistent with Definition 2 [14].

The results are presented in Tables 1-8 and in Figures 1 and 2.

<table><caption>**Table 1.** The fraction of true labels within the prediction ranges for an ACP and a, for each run, randomly selected ICP at different significance levels for the 50 runs on the CDK data using SVM. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.4</td><td>0.6522</td><td>0.7391</td><td>0.7717</td><td>0.7717</td><td>0.8043</td><td>0.8913</td></tr><tr><td>ICP</td><td>0.4</td><td>0.5100</td><td>0.5622</td><td>0.5997</td><td>0.5990</td><td>0.6303</td><td>0.6826</td></tr><tr><td>ACP</td><td>0.3</td><td>0.7174</td><td>0.8261</td><td>0.8478</td><td>0.8474</td><td>0.8696</td><td>0.9348</td></tr><tr><td>ICP</td><td>0.3</td><td>0.6191</td><td>0.6535</td><td>0.6879</td><td>0.6896</td><td>0.7202</td><td>0.7876</td></tr><tr><td>ACP</td><td>0.2</td><td>0.8043</td><td>0.8913</td><td>0.9130</td><td>0.9170</td><td>0.9565</td><td>0.9783</td></tr><tr><td>ICP</td><td>0.2</td><td>0.7296</td><td>0.7712</td><td>0.7923</td><td>0.7989</td><td>0.8282</td><td>0.8848</td></tr><tr><td>ACP</td><td>0.1</td><td>0.8913</td><td>0.9565</td><td>0.9565</td><td>0.9643</td><td>0.9783</td><td>1.0000</td></tr><tr><td>ICP</td><td>0.1</td><td>0.8378</td><td>0.8626</td><td>0.8933</td><td>0.8890</td><td>0.9101</td><td>0.9537</td></tr></tbody></table>

<table><caption>**Table 2.** The fraction of true labels within the prediction ranges for an ACP and a, for each run, randomly selected ICP at different significance levels for the 50 runs on the GNRHR data using SVM. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.4</td><td>0.3590</td><td>0.5962</td><td>0.6667</td><td>0.6505</td><td>0.7179</td><td>0.8205</td></tr><tr><td>ICP</td><td>0.4</td><td>0.4297</td><td>0.5560</td><td>0.5960</td><td>0.5947</td><td>0.6569</td><td>0.7321</td></tr><tr><td>ACP</td><td>0.3</td><td>0.5128</td><td>0.6731</td><td>0.7179</td><td>0.7227</td><td>0.7628</td><td>0.8718</td></tr><tr><td>ICP</td><td>0.3</td><td>0.5215</td><td>0.6428</td><td>0.6736</td><td>0.6750</td><td>0.7208</td><td>0.7926</td></tr><tr><td>ACP</td><td>0.2</td><td>0.7179</td><td>0.7949</td><td>0.8205</td><td>0.8286</td><td>0.8462</td><td>0.9744</td></tr><tr><td>ICP</td><td>0.2</td><td>0.6782</td><td>0.7466</td><td>0.7749</td><td>0.7791</td><td>0.8126</td><td>0.8797</td></tr><tr><td>ACP</td><td>0.1</td><td>0.8205</td><td>0.8974</td><td>0.9231</td><td>0.9258</td><td>0.9487</td><td>1.0000</td></tr><tr><td>ICP</td><td>0.1</td><td>0.8072</td><td>0.8633</td><td>0.8745</td><td>0.8835</td><td>0.9104</td><td>0.9605</td></tr></tbody></table>
<table><caption>**Table 3.** The fraction of true labels within the prediction ranges for an ACP and a, for each run, randomly selected ICP at one significance level for the 50 runs on the CDK data using RF. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.2</td><td>0.7174</td><td>0.8478</td><td>0.8696</td><td>0.8696</td><td>0.9130</td><td>0.9565</td></tr><tr><td>ICP</td><td>0.2</td><td>0.6926</td><td>0.7786</td><td>0.8071</td><td>0.7982</td><td>0.8249</td><td>0.8930</td></tr></tbody></table>

<table><caption>**Table 4.** The fraction of true labels within the prediction ranges for an ACP and a, for each run, randomly selected ICP at one significance level for the 50 runs on the GNRHR data using RF. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.2</td><td>0.7179</td><td>0.8205</td><td>0.8718</td><td>0.8564</td><td>0.8974</td><td>1.0000</td></tr><tr><td>ICP</td><td>0.2</td><td>0.6854</td><td>0.7706</td><td>0.7919</td><td>0.8013</td><td>0.8271</td><td>0.9162</td></tr></tbody></table>

<table><caption>**Table 5.** The efficiency for an ACP and an ICP at one significance level for the 50 runs on the CDK5 data using SVM. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.2</td><td>0.5874</td><td>1.2190</td><td>1.6880</td><td>3.2000</td><td>3.3220</td><td>67.6700</td></tr><tr><td>ICP</td><td>0.2</td><td>0.1550</td><td>0.9128</td><td>1.2620</td><td>3.5760</td><td>1.8970</td><td>398.3000</td></tr></tbody></table>

# 4 Discussion

The results presented in Tables 1-8 and in Figures 1 and 2 indicate that the ACP methodology has advantages over traditional ICP. The former adds stability and robustness to the predictions, which is particularly clear in Figures 1-2 where the variance in the prediction ranges is smaller than from an individual ICP. This is of considerable importance within the pharmaceutical domain where precision as well as robustness in predictions are key elements for successful application in ongoing discovery projects. Although the traditional ICP is valid on average,
<table><caption>**Table 6.** The efficiency for an ACP and an ICP at one significance level for the 50 runs on the CDK5 data using RF. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.2</td><td>0.4752</td><td>1.0060</td><td>1.2640</td><td>1.3750</td><td>1.5870</td><td>3.9780</td></tr><tr><td>ICP</td><td>0.2</td><td>0.0653</td><td>0.8659</td><td>1.1830</td><td>1.3680</td><td>1.6380</td><td>8.5950</td></tr></tbody></table>

<table><caption>**Table 7.** The efficiency for an ACP and an ICP at one significance level for the 50 runs on the GNRHR data using SVM. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.2</td><td>1.2770</td><td>1.625</td><td>1.733</td><td>1.907</td><td>1.903</td><td>25.830</td></tr><tr><td>ICP</td><td>0.2</td><td>0.5486</td><td>1.457</td><td>1.704</td><td>1.957</td><td>2.000</td><td>62.400</td></tr></tbody></table>

<table><caption>**Table 8.** The efficiency for an ACP and an ICP at one significance level for the 50 runs on the GNRHR data using RF. Here, $Q_1$ and $Q_3$ are the lower and upper quartile, respectively.</caption><thead><tr><th>Method</th><th>ε</th><th>min</th><th>Q<sub>1</sub></th><th>median</th><th>mean</th><th>Q<sub>3</sub></th><th>max</th></tr></thead><tbody><tr><td>ACP</td><td>0.2</td><td>0.7481</td><td>1.6050</td><td>1.9620</td><td>2.1290</td><td>2.5490</td><td>6.3040</td></tr><tr><td>ICP</td><td>0.2</td><td>0.07138</td><td>1.3940</td><td>1.8230</td><td>2.0980</td><td>2.4840</td><td>10.3100</td></tr></tbody></table>

the variance is very large. This means that there is a relatively large proportion of very tight predictions regions that in fact are far from valid (for example, for the CDK5 data, an ICP with a confidence of 80% produces 46% error in the quartile with tightest prediction regions; the corresponding figure for the ACP is 21%. These tight prediction regions are offset by some prediction regions that are very wide (as wide or wider than the possible range of the response value) that are *always* valid, which produces a conformal predictor that is valid on average. Neither the too tight prediction regions that cannot be trusted nor non-informative regions are helpful for the researcher using the model.

The results in Tables 1-8 show that the ACP (as used in this paper) is too conservative on datasets with a relatively small number of examples, which is clear from Equations (1) and (2). This indicates that improved efficiency can be achieved without compromising the validity of the ACP, e.g. by more clever resampling (a large body of literature exists that address this problem, see e.g. [15]) or smoothing of p-values on either side of $\epsilon$.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The distribution of prediction interval size for an ACP and a randomly sampled ICP for all 50 runs on the CDK5 data. The results are shown at $\epsilon = 0.2$.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** The distribution of prediction interval size for an ACP and a randomly sampled ICP for all 50 runs on the GNRHR data. The results are shown at $\epsilon = 0.2$.</figcaption></figure>

To conclude: We have introduced the ACP, a generalization of the BCP and CCP introduced by Vovk [5] and we have shown that it improves on classical ICP by reducing the variance in the estimated prediction regions (analogously to how a bagged predictor improves the prediction by reducing variance). ACP
seems to represent a pragmatic and useful way forward for obtaining models and predictions with good precision and small prediction ranges.

Future ways to develop the ACP include to (i) study ACP for categorical labels (e.g. aggregation through voting); (ii) other resampling schemas.

## References

1. Eklund, M., Norinder, U., Boyer, S., Carlsson, L.: Application of Conformal Prediction in QSAR. In: AIAI (2). (2012) 166–175

2. Eklund, M., Norinder, U., Boyer, S., Carlsson, L.: The application of conformal prediction to the drug discovery process. Annals of Mathematics and Artificial Intelligence (2013)

3. Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. 1 edn. Springer (2005)

4. Vovk, V.: Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence (2013) 1–20

5. Vovk, V.: Cross-conformal predictor. Working Paper 6. (2013) http://alrw.net.

6. Bühlmann, P., Yu, B.: Analyzing bagging. The Annals of Statistics 30(4) (2002) 927–961

7. Ruppert, D.: Statistics and Data Analysis for Financial Engineering (Springer Texts in Statistics). 1., st edition. edn. Springer, Berlin (2010)

8. Vapnik, V.N.: Statistical learning theory. 1 edn. Wiley (1998)

9. Chang, C.C., Lin, C.J.: LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2 (2011) 27:1–27:27 Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

10. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011) 2825–2830

11. Papadopoulos, H., Haralambous, H.: Reliable prediction intervals with regression neural networks. Neural Networks 24(8) (2011) 842–851

12. Chen, H., Carlsson, L., Eriksson, M., Varkonyi, P., Norinder, U., Nilsson, I.: Beyond the scope of free-wilson analysis: Building interpretable qsar models with machine learning algorithms. Journal of Chemical Information and Modeling 53(1324-1336) (2013)

13. Faulon, J.L., Collins, M.J., Carr, R.D.: The signature molecular descriptor. 4. Canonizing molecules using extended valence sequences. J Chem Inf Comput Sci 44(2) (2004) 427–36

14. Politis, D., Romano, J., Wolf, M.: Subsampling. Springer-Verlag, New York (1999)

15. Egloff, D., Leippold, M.: Quantile estimation with adaptive importance sampling. The Annals of Statistics 38(2) (2010) 1244–1278