<figure><img src="image_1.png" /></figure>

HAL
open science

# Osteoporosis Risk Assessment with Well-Calibrated Probabilistic Outputs

Antonis Lambrou, Harris Papadopoulos, Alexander Gammerman

► To cite this version:

Antonis Lambrou, Harris Papadopoulos, Alexander Gammerman. Osteoporosis Risk Assessment with Well-Calibrated Probabilistic Outputs. 9th Artificial Intelligence Applications and Innovations (AIAI), Sep 2013, Paphos, Greece. pp.432-441, 10.1007/978-3-642-41142-7_44. hal-01459680

HAL Id: hal-01459680

https://inria.hal.science/hal-01459680

Submitted on 7 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Osteoporosis Risk Assessment with
well-calibrated probabilistic outputs

Antonis Lambrou<sup>1,2</sup>, Harris Papadopoulos<sup>1,2,3</sup>, and Alexander Gammerman<sup>1</sup>

<sup>1</sup>Frederick Research Center, Nicosia, Cyprus

<sup>2</sup>Computer Learning Research Centre, Computer Science Department, Royal Holloway, University of London, England.

<{A.Lambrou, A.Gammerman}@cs.rhul.ac.uk

<sup>3</sup>Computer Science and Engineering Department, Frederick University, Cyprus.
H.Papadopoulos@frederick.ac.cy

**Abstract.** Osteoporosis is a disease of bones that results in an increased risk of bone fracture. The diagnosis of Osteoporosis is usually performed by measuring the Bone Mineral Density (BMD) using Dual-Energy X-ray Absorptiometry (DEXA) scanning. In this work, we introduce the use of Venn Prediction in order to assess the risk of Osteoporosis before a DEXA scan, based on known risk factors. Unlike other probabilistic methods, Venn Predictors can provide well-calibrated probabilistic outputs under the assumption that the data used are identically and independently distributed (i.i.d.). Our contribution is two-fold: Firstly, we have collected real-world data from various clinic centres in Cyprus which based on their locality can be used for analysis of Osteoporosis risk factors specifically for Cypriot patients. To the best of our knowledge, local data in Cyprus for Osteoporosis risk assessment have not been collected before. Secondary, our results demonstrate that our method can provide probabilistic outputs that may be practical and trustful to physicians.

**Keywords:** well calibrated probabilities, osteoporosis, risk assessment, Venn Predictor, Machine Learning.

# 1 Introduction

Osteoporosis is a disease of bones that results in an increased risk of bone fracture. The diagnosis of Osteoporosis is usually performed by measuring the Bone Mineral Density (BMD) using Dual-Energy X-ray Absorptiometry (DEXA) scanning. A result of BMD that is lower than 2.5 standard deviations from the average of young healthy adults is defined by the World Health Organisation (WHO) as Osteoporosis [11].

We introduce the use of Venn Prediction in order to predict the risk of Osteoporosis before a DEXA scan, based on known risk factors. Unlike other probabilistic methods, Venn Predictors can provide well-calibrated probabilistic outputs under the assumption that the data used are identically and independently distributed (i.i.d.). We have collected real-world data from various clinic centres
in Cyprus which based on their locality can be used for analysis of Osteoporosis risk factors specifically for Cypriot patients. To the best of our knowledge, local data in Cyprus for Osteoporosis risk assessment have not been collected before. Moreover, our results show that Venn Predictors (VPs) can provide probabilistic outputs that may be practical and trustful to physicians.

The rest of the paper is structured as follows. In Section 2, we outline related work. In Section 3, we give a detailed explanation of the Osteoporosis data that we have collected, and we also explain how Venn Prediction works. In Section 4, we show our experimental results on the Osteoporosis data and discuss. Finally, in Section 5, we conclude and outline our future work.

## 2 Related Work

The World Health Organisation (WHO) has defined the disease of Osteoporosis as a Bone Mineral Density (BMD) which is lower than 2.5 standard deviations from the average of young healthy adults. Furthermore, BMD that is 1 standard deviation lower is defined as Osteopenia, which is a precursor to Osteoporosis [11]. DEXA stands for Dual Energy X-ray Absorptiometry, and is a standard test for BMD. DEXA scanners throw an X-ray beam at the lumbar vertebrae and measure the shadow cast by the bones. In Fig. 1 we include a sample image of the lumbar spine of a DEXA scan. Software in the machine estimates the amount of calcium in the bone based on the darkness of the shadow. The result is expressed as a number of grams per square centimeter, which is defined as the Bone Mineral Density (BMD). In Table 1, we show how the BMD is mapped to a t-score value compared against the average of young healthy adults.

The Venn Prediction (VP) framework is based on the Conformal Prediction (CP) framework. CP is a novel technique for obtaining reliable confidence measures. The technique is proposed in [7] and later improved in [22] and [24]. CPs are built using classical machine learning algorithms, called underlying algorithms. CPs complement the predictions of the underlying algorithms with measures of confidence. Many CPs have been built to date, based on various algorithms such as Support Vector Machines [22], k-Nearest Neighbours for classification [21] and for regression [18], Random Forests [4], and Genetic Algorithms [8]. The computational efficiency of CPs has also been greatly improved using Inductive Conformal Prediction (ICP) [12], as demonstrated in applications to Ridge Regression [17], and more recently in applications to Neural Networks [19]. The CP framework has been successfully applied to medical problems, such as evaluation of the risk of stroke [9], breast cancer diagnosis [6], classification of leukaemia subtypes [1], and acute abdominal pain diagnosis [15]. Additionally, CPs have been applied to other problems such as Software Effort Estimation in [16].

Venn Prediction has been introduced in [23] where the interested reader can find a detailed description of the framework. Since then, VPs have been developed based on k-Nearest Neighbours [3], Nearest Centroid [2] and Neural Networks [13, 14]. Furthermore, VPs based on SVMs have been developed in [10,
<table><caption>**Table 1.** Young Adult (YA) T-score based on the Bone Mineral Density (BMD) according to the World Health Organisation (WHO).</caption><tr><td>BMD</td><td>1.44</td><td>1.32</td><td>1.20</td><td>1.08</td><td>0.96</td><td>0.84</td><td>0.72</td><td>0.60</td></tr><tr><td>YA T-Score</td><td>2</td><td>1</td><td>0</td><td>-1</td><td>-2</td><td>-3</td><td>-4</td><td>-5</td></tr></table>

27], and have been compared with Platt's method [20], Binning [5] and Isotonic Regression [26]. As it is shown in [10], such methods do not guarantee that the probabilistic outputs will be well-calibrated.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Image of the Lumbar Spine AP (Anterior Posterior) from a DEXA Scan.</figcaption></figure>

# 3 Material and methods

## 3.1 Osteoporosis data

We have collected data from various clinics in Cyprus. In particular, we have collected 389 cases of female patients that have performed a DEXA scan. The data are constructed based on a questionnaire that is given to patients to complete. The patients may have previous history of osteoporosis and may already follow therapy. The questionnaire was constructed by physicians and contains questions that are relevant to Osteoporosis risk factors. Each case is classified as
"Normal" or "Risk of Osteoporosis" based on the patient's spine t-score that is given by the DEXA scan. According to the WHO, patients with a t-score above -1 are diagnosed as healthy, therefore we have classified patients into two classes: "Normal" for patients with t-score above -1, and "Risk of Osteoporosis" otherwise. From the 389 patients, 174 have a t-score above or equal to -1, and 215 have a t-score below -1. In Table 2, we give the list of attributes of our dataset.

## 3.2 Venn Prediction

In this section, we explain how Venn Prediction (VP) works. Typically, we have a training set<fnref n="1" /> of the form {$z_1, ..., z_{n-1}$}, where each $z_i \in Z$ is a pair $(x_i, y_i)$ consisting of the object $x_i$ and its classification $y_i$. For a new object $x_n$, we intend to estimate its probability of belonging to each class $Y_j \in \{Y_1, ..., Y_c\}$. The Venn Prediction framework assigns each one of the possible classifications $Y_j$ to $x_n$ and divides all examples {$(x_1, y_1), ..., (x_n, Y_j)$} into a number of categories based on a taxonomy. A taxonomy is a sequence $A_n, n = 1, ..., N$ of finite measurable partitions of the space $Z^{(n)} \times Z$, where $Z^{(n)}$ is the set of all multisets of elements of $Z$ of length $n$. We will write $A_n(\{z_1, ..., z_n\}, z_i)$ for the category of the partition $A_n$ that contains $(\{z_1, ..., z_n\}, z_i)$. Every taxonomy $A_1, A_2, ..., A_N$ defines a different VP. In this work, we define three VPs based on taxonomies that use the classification output of three underlying algorithms, namely, the J48 decision tree, Random Forests (RF), and Sequential Minimal Optimisation (SMO). Examples are categorized according to the underlying algorithm classifications that are given to them.

After partitioning the examples into categories using a taxonomy, the empirical probability of each classification $Y_k$ in the category $\tau_{new}$ that contains $(x_n, Y_j)$ will be

$$p^{Y_j}(Y_k) = \frac{|\{(x^*, y^*) \in \tau_{new} : y^* = Y_k\}|}{|\tau_{new}|} \qquad (1)$$

This is a probability distribution for the class of $x_n$. So after assigning all possible classifications to $x_n$ we get a set of probability distributions $P_n = \{p^{Y_j} : Y_j \in \{Y_1, ..., Y_c\}\}$ that compose the multi-probability prediction of the VP. As proved in [25], these are automatically well calibrated, regardless of the taxonomy used.

The maximum and minimum probabilities obtained for each label $Y_k$ amongst all distributions {$p^{Y_j} : Y_j \in \{Y_1, ..., Y_c\}$}, define the interval for the probability of the new example belonging to $Y_k$. We denote these probabilities as $U(Y_k)$ and $L(Y_k)$, respectively. The VP outputs the prediction $\hat{y}_n = Y_{kbest}$, where

$$k_{best} = \arg \max_{k=1,...,c} \overline{p(k)}, \qquad (2)$$

and $\overline{p(k)}$ is the mean of the probabilities obtained for label $Y_k$ amongst all probability distributions. The probability interval for this prediction is $[L(Y_k), U(Y_k)]$.

<footnote n="1">¹ The training set is in fact a multiset, as it can contain some examples more than once.</footnote>
<table><caption>**Table 2.** Table of attributes in the Osteoporosis dataset.</caption><thead><tr><th>#</th><th>Attribute name</th><th>Type</th><th>#</th><th>Attribute name</th><th>Type</th></tr></thead><tbody><tr><td>1</td><td>Sex</td><td>Binary</td><td>35</td><td>Receive Thyroxine</td><td>Binary</td></tr><tr><td>2</td><td>Age</td><td>Numeric</td><td>36</td><td>Receive Estrogens</td><td>Binary</td></tr><tr><td>3</td><td>Weight</td><td>Numeric</td><td>37</td><td>Neurogenic Anorexia</td><td>Binary</td></tr><tr><td>4</td><td>Height</td><td>Numeric</td><td>38</td><td>Malabsorption syndrome</td><td>Binary</td></tr><tr><td>5</td><td>Start of Menstruation</td><td>Numeric</td><td>39</td><td>Chronic liver diseases</td><td>Binary</td></tr><tr><td>6</td><td>End of Menstruation</td><td>Numeric</td><td>40</td><td>Inflammatory bowel diseases</td><td>Binary</td></tr><tr><td>7</td><td>Pregnancies</td><td>Numeric</td><td>41</td><td>Transplantation</td><td>Binary</td></tr><tr><td>8</td><td>Smoking now</td><td>Binary</td><td>42</td><td>Chronic renal failure</td><td>Binary</td></tr><tr><td>9</td><td>Smoking in the past</td><td>Binary</td><td>43</td><td>Prolonged immobilization</td><td>Binary</td></tr><tr><td>10</td><td>No smoking</td><td>Binary</td><td>44</td><td>Cushing's syndrome</td><td>Binary</td></tr><tr><td>11</td><td>Years of past smoking</td><td>Numeric</td><td>45</td><td>Epilepsy</td><td>Binary</td></tr><tr><td>12</td><td>Years of current smoking</td><td>Numeric</td><td>46</td><td>Insulin Dependent</td><td>Binary</td></tr><tr><td>13</td><td>Cigarettes per day</td><td>Numeric</td><td>47</td><td>Ovariectomy before menopause</td><td>Binary</td></tr><tr><td>14</td><td>Alchohol intake per day</td><td>Numeric</td><td>48</td><td>Chronic gastrointestinal disorders</td><td>Binary</td></tr><tr><td>15</td><td>Caffeine intake per day</td><td>Numeric</td><td>49</td><td>Paget's Disease</td><td>Binary</td></tr><tr><td>16</td><td>History of fracture</td><td>Binary</td><td>50</td><td>Hyperthyroidism</td><td>Binary</td></tr><tr><td>17</td><td>Hip fracture</td><td>Binary</td><td>51</td><td>Parathyroid gland disease</td><td>Binary</td></tr><tr><td>18</td><td>Spine fracture</td><td>Binary</td><td>52</td><td>Receive Steroids</td><td>Binary</td></tr><tr><td>19</td><td>Wrist fracture</td><td>Binary</td><td>53</td><td>Receive Thyroxine</td><td>Binary</td></tr><tr><td>20</td><td>Low energy</td><td>Binary</td><td>54</td><td>Anticonvulsants (for seizures, epilepsy)</td><td>Binary</td></tr><tr><td>21</td><td>High energy</td><td>Binary</td><td>55</td><td>Diuretics</td><td>Binary</td></tr><tr><td>22</td><td>Sports</td><td>Binary</td><td>56</td><td>Heparin</td><td>Binary</td></tr><tr><td>23</td><td>History of osteoporosis</td><td>Binary</td><td>57</td><td>Chemotherapy</td><td>Binary</td></tr><tr><td>24</td><td>Osteoporosis in family</td><td>Binary</td><td>58</td><td>Treatment of osteoporosis</td><td>Binary</td></tr><tr><td>25</td><td>Loss of height</td><td>Binary</td><td>59</td><td>Alendronati</td><td>Binary</td></tr><tr><td>26</td><td>Kyphosis</td><td>Binary</td><td>60</td><td>Risedronati</td><td>Binary</td></tr><tr><td>27</td><td>End of menstrual bleeding</td><td>Binary</td><td>61</td><td>Zoledronati</td><td>Binary</td></tr><tr><td>28</td><td>Arthritis</td><td>Binary</td><td>62</td><td>Raloxifeni</td><td>Binary</td></tr><tr><td>29</td><td>Secondary Osteoporosis</td><td>Binary</td><td>63</td><td>Strontio</td><td>Binary</td></tr><tr><td>30</td><td>Breast feeding</td><td>Binary</td><td>64</td><td>Parathormoni</td><td>Binary</td></tr><tr><td>31</td><td>Avoidance of milk</td><td>Binary</td><td>65</td><td>Denosoymapi</td><td>Binary</td></tr><tr><td>32</td><td>Avoidance of sex</td><td>Binary</td><td>66</td><td>Kalsitonini</td><td>Binary</td></tr><tr><td>33</td><td>Diarrhea</td><td>Binary</td><td>67</td><td>Calcium + Vitamin D</td><td>Binary</td></tr><tr><td>34</td><td>Receive Cortisone</td><td>Binary</td><td>68</td><td>Calcium</td><td>Binary</td></tr></tbody></table>
<table><caption>**Table 3.** Results of the six algorithms on the Osteoporosis dataset. We compare the accuracy, the lower and upper probability outputs of the VPs, and the percentages and accuracy rates of examples which have probabilities of at least 75%, 70%, and 60%.</caption><thead><tr><th>Predictors</th><th>J48</th><th>RF</th><th>SMO</th><th>J48-VP</th><th>RF-VP</th><th>SMO-VP</th></tr></thead><tbody><tr><td>Accuracy</td><td>70.18%</td><td>68.89%</td><td>67.10%</td><td>67.38%</td><td>65.17%</td><td>65.71%</td></tr><tr><td>Lower Probability</td><td>-</td><td>-</td><td>-</td><td>64.27%</td><td>57.93%</td><td>64.21%</td></tr><tr><td>Upper Probability</td><td>-</td><td>-</td><td>-</td><td>80.62%</td><td>78.09%</td><td>71.83%</td></tr><tr><td>Min probability ≥ 75%</td><td>-</td><td>-</td><td>-</td><td>54.73%</td><td>34.27%</td><td>7.61%</td></tr><tr><td>Min probability ≥ 70%</td><td>-</td><td>-</td><td>-</td><td>67.10%</td><td>60.51%</td><td>39.49%</td></tr><tr><td>Min probability ≥ 60%</td><td>-</td><td>-</td><td>-</td><td>75.53%</td><td>64.88%</td><td>84.52%</td></tr><tr><td>Accuracy at ≥ 75% min. prob.</td><td>-</td><td>-</td><td>-</td><td>73.61%</td><td>73.43%</td><td>74.05%</td></tr><tr><td>Accuracy at ≥ 70% min. prob.</td><td>-</td><td>-</td><td>-</td><td>72.71%</td><td>71.51%</td><td>69.01%</td></tr><tr><td>Accuracy at ≥ 60% min. prob.</td><td>-</td><td>-</td><td>-</td><td>72.06%</td><td>71.44%</td><td>67.93%</td></tr></tbody></table>

# 4 Experiments and results

## 4.1 Offline experiments

We perform 10-fold cross validation experiments on our Osteoporosis dataset with the J48 decision tree, Random Forests (RF), Sequential Minimal Optimization (SMO), J48-VP, RF-VP, and SMO-VP algorithms. Each algorithm performs a Correlation Based Feature Selection (CBFS) during each fold of the experiment. For the RF, we chose the number of trees to 20, and the depth of each tree to 4 nodes. For the SMO, we chose the RBF kernel with a spread parameter of 0.43. These parameters were chosen after several experimental settings. In the results, we show the average accuracy, and for the VPs we also show the average lower probabilities and upper probabilities. Since VPs provide well-calibrated probabilistic outputs, the accuracy of the VPs is expected to fall within the lower and upper probability intervals. We demonstrate that an amount of predictions have significant probabilities of being correct, and such predictions may be practical for physicians. Moreover, we show that the accuracy rates of the predictions do not drop below the minimum probabilities given by the VPs (up to statistical fluctuations).

In Table 3, we compare the average accuracy of the four algorithms on the Osteoporosis dataset. The J48 algorithm provides the highest accuracy which is 70.18%. The corresponding J48-VP provides a slightly lower accuracy which is 67.38%. The VPs provide extra information for each prediction, which is the lower and upper probability interval. We show the average lower probabilities and upper probabilities given by the three VPs, and the corresponding accuracy rates of such predictions. The accuracy rates demonstrate the validity of the probabilistic outputs of the VPs. For example, at 75% probability, the accuracy rates should be at least (up to statistical fluctuations) at the 75% which is expected. In our case, the offline results give accuracies of 73.61%, 73.43%, and 74.05%, which are acceptable. In the table, we also show the percentage of examples that have a minimum probability of 75%, 70%, and 60%. The J48-VP
<table><caption>**Table 4.** Confusion matrices of the three algorithms J48, RF, and SMO on the Osteoporosis dataset.</caption><thead><tr><th rowspan="2"></th><th colspan="2">J48</th><th colspan="2">RF</th><th colspan="2">SMO</th></tr><tr><th>Normal</th><th>Risk of Ost.</th><th>Normal</th><th>Risk of Ost.</th><th>Normal</th><th>Risk of Ost.</th></tr></thead><tbody><tr><th>Normal</th><td>113</td><td>61</td><td>107</td><td>67</td><td>89</td><td>85</td></tr><tr><th>Risk of Ost.</th><td>55</td><td>160</td><td>54</td><td>161</td><td>43</td><td>172</td></tr></tbody></table>

gives 54.73% of predictions with a probability of at least 75%. These predictions can be considered significant, since the error rate will be at most 25% in the long run.

In Table 4, we show the confusion matrices of the three algorithms on the Osteoporosis dataset. The confusion matrices demonstrate that the algorithms misclassify examples at a balanced rate between the two classes.

## 4.2 Online experiments

In order to show the validity of the probability estimates of the VPs, we conduct experiments in the on-line mode. Initially all examples are test examples and they are added to the training set one by one after a prediction for each one is made. We graph the Cumulative Lower Accuracy Probability (CLAP), the Cumulative Upper Accuracy Probability (CUAP), and the Cumulative Accuracy (CA) curves:

$$CLAP(t) = \frac{1}{t} \sum_{i=1}^{t} U_i(Y_{kbest}), \qquad (3)$$

$$CUAP(t) = \frac{1}{t} \sum_{i=1}^{t} L_i(Y_{kbest}), \qquad (4)$$

$$CA(t) = \frac{1}{t} \sum_{i=1}^{t} Acc_i, \qquad (5)$$

where *t* is the number of test examples that have been added to the training set, and *Acc* = 1 when the prediction for example *x*<sub>*i*</sub> is correct and 0 otherwise. Since VPs provide well calibrated probabilistic outputs, it is expected that the CA curve will fall within or near the CLAP and CUAP curves.

In Fig. 2, we show the online results of the three VPs, J48-VP (left), RF-VP (right), and SMO-VP (bottom) on the Osteoporosis dataset. The results demonstrate the validity of the VPs, since the actual accuracy of the predictors always falls within the lower and upper probability outputs of the VPs. Comparing the three VPs, we can see that the RF-VP gives the wider probabilistic outputs, while the SMO-VP gives the narrowest outputs.
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Online experiments with J48-VP (left), RF-VP (right), and SMO-VP (bottom) on the Osteoporosis dataset.</figcaption></figure>

# 5 Conclusion

In this work, we have applied Venn Prediction to the problem of Osteoporosis Risk Assessment. We have evaluated our method on real-world data that we have collected from various clinics in Cyprus. Our results, demonstrate that our method provides well-calibrated probabilistic outputs in the predictions that can be useful in practice. Precisely, patients may get a prognosis based on Osteoporosis risk factors before the performance of a DEXA scan. In the future, we aim to collect more data and perform supplementary analysis, in order to improve and evaluate further our VPs. Furthermore, we are in the process of building a tool for physicians that will enable them to use our VPs for assessing the risk of Osteoporosis.

**Acknowledgments.** This work was co-funded by the European Regional Development Fund and the Cyprus Government through the Cyprus Research Promotion Foundation “DESMI 2009-2010”, TPE/ORIZO/0609(BIE)/24 (“Development of New Venn Prediction Methods for Osteoporosis Risk Assessment”).
References

1. Tony Bellotti, Zhiyuan Luo, and Alexander Gammerman. Reliable classification of childhood acute leukaemia from gene expression data using confidence machines. In *Proceedings of IEEE International Conference on Granular Computing (GRC '06)*, pages 148–153, 2006.

2. Mikhail Dashevskiy and Zhiyuan Luo. Reliable probabilistic classification and its application to internet traffic. In *Advanced Intelligent Computing Theories and Applications. With Aspects of Theoretical and Methodological Issues*, volume 5226 of LNCS, pages 380–388. Springer, 2008.

3. Mikhail Dashevskiy and Zhiyuan Luo. Predictions with confidence in applications. In Petra Perner, editor, *Machine Learning and Data Mining in Pattern Recognition*, volume 5632 of LNCS, pages 775–786. Springer, 2009.

4. Devetyarov Dmitry and Nouretdinov Ilia. Prediction with confidence based on a random forest classifier. In Harris Papadopoulos, Andreas Andreou, and Max Bramer, editors, *Artificial Intelligence Applications and Innovations*, volume 339, pages 37–44. Springer Boston, 2010.

5. Joseph Drish. Obtaining calibrated probability estimates from Support Vector Machines, 1998.

6. Alexander Gammerman, Ilia Nouretdinov, Brian Burford, Alexey Chervonenkis, Volodya Vovk, and Zhiyuan Luo. Clinical mass spectrometry proteomic diagnosis by conformal predictors. *Statistical applications in genetics and molecular biology*, 7(2), 2008.

7. Alexander Gammerman, Volodya Vovk, and Vladimir Vapnik. Learning by transduction. In *In Uncertainty in Artificial Intelligence*, pages 148–155. Morgan Kaufmann, 1998.

8. Antonis Lambrou, Harris Papadopoulos, and Alexander Gammerman. Reliable confidence measures for medical diagnosis with evolutionary algorithms. *IEEE Transactions on Information Technology in Biomedicine*, 15(1):93–99, 2011.

9. Antonis Lambrou, Harris Papadopoulos, Efthyvoulos Kyriacou, Constantinos S. Pattichis, Marios S. Pattichis, Alexander Gammerman, and Andrew Nicolaides. Evaluation of the risk of stroke with confidence predictions based on ultrasound carotid image analysis. *International Journal on Artificial Intelligence Tools*, 21(04):1240016, 2012.

10. Antonis Lambrou, Harris Papadopoulos, Ilia Nouretdinov, and Alexander Gammerman. Reliable probability estimates based on support vector machines for large multiclass datasets. In Lazaros Iliadis, Ilias Maglogiannis, Harris Papadopoulos, Kostas Karatzas, and Spyros Sioutas, editors, *Artificial Intelligence Applications and Innovations*, volume 382 of IFIP Advances in Information and Communication Technology, pages 182–191. Springer Berlin Heidelberg, 2012.

11. World Health Organisation. *Prevention and management of Osteoporosis*. Geneva, 2003.

12. Harris Papadopoulos. Inductive Conformal Prediction: Theory and application to Neural Networks. In Paula Fritzsche, editor, *Tools in Artificial Intelligence*, chapter 18, pages 315–330. InTech, Vienna, Austria, 2008.

13. Harris Papadopoulos. Reliable probabilistic prediction for medical decision support. In *Proceedings of the 7th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI 2011)*, volume 364 of IFIP AICT, pages 265–274. Springer, 2011.
14. Harris Papadopoulos. Reliable probabilistic classification with neural networks. Neurocomputing, 107:59 – 68, 2013.

15. Harris Papadopoulos, Alex Gammerman, and Volodya Vovk. Reliable diagnosis of acute abdominal pain with conformal prediction. *International Journal of Engineering Intelligent Systems for Electrical Engineering and Communications*, 17(2-3):127–137, 2009.

16. Harris Papadopoulos, Efi Papatheocharous, and Andreas S. Andreou. Reliable confidence intervals for software effort estimation. In *Proceedings of the 2nd Workshop on Artificial Intelligence Techniques in Software Engineering (AISEW 2009)*, volume 475 of *CEUR Workshop Proceedings*, pages 211–220. CEUR WS.org, 2009.

17. Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In *Proceedings of the 13th European Conference on Machine Learning (ECML'02)*, volume 2430 of *Lecture Notes in Computer Science*, pages 345–356. Springer, 2002.

18. Harris Papadopoulos, Vladimir Vovk, and Alex Gammerman. Regression conformal prediction with nearest neighbours. *Journal of Artificial Intelligence Research*, 40:815–840, 2011.

19. Harris Papadopoulos, Volodya Vovk, and Alex Gammerman. Conformal prediction with neural networks. In *Proceedings of the 19th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'07)*, volume 2, pages 388–395. IEEE Computer Society, 2007.

20. John C. Platt. Probabilistic outputs for Support Vector Machines and comparisons to regularized likelihood methods. In *Advances in large margin classifiers*, pages 61–74. MIT Press, 1999.

21. Kostas Proedrou, Ilia Nouretdinov, Volodya Vovk, and Alex Gammerman. Transductive confidence machines for pattern recognition. In *Proceedings of the 13th European Conference on Machine Learning (ECML'02)*, volume 2430 of *Lecture Notes in Computer Science*, pages 381–390. Springer, 2002.

22. Craig Saunders, Alex Gammerman, and Volodya Vovk. Transduction with confidence and credibility. In *Proceedings of the 16th International Joint Conference on Artificial Intelligence*, volume 2, pages 722–726, Los Altos, CA, 1999. Morgan Kaufmann.

23. Vladimir Vovk, Glenn Shafer, and Ilia Nouretdinov. Self-calibrating probability forecasting. In Sebastian Thrun, Lawrence K. Saul, and Bernhard Schölkopf, editors, *Advances in Neural Information Processing Systems 16*, pages 1133–1140, Cambridge, MA, 2004. MIT Press.

24. Volodya Vovk, Alex Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. In *Proceedings of the Sixteenth International Conference on Machine Learning*, pages 444–453. Morgan Kaufmann, 1999.

25. Volodya Vovk, Alexander Gammerman, and G. Shafer. *Algorithmic Learning in a Random World*. New York, Springer, 2005.

26. Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In *Proceedings of the 8th ACM International Conference on Knowledge Discovery and Data Mining*, pages 694–699, 2002.

27. Chenzhe Zhou, Ilia Nouretdinov, Zhiyuan Luo, Dmitry Adamskiy, Luke Randell, Nick Coldham, and Alex Gammerman. A comparison of Venn Machine with Platt's method in probabilistic outputs. In *Proceedings of the 7th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI 2011)*, volume 364 of *IFIP AICT*, pages 483–490. Springer, 2011.