# Exploiting Machine Learning for Predicting Nodal Status in Prostate Cancer Patients

Mauro Vallati, Berardino De Bari, Roberto Gatta, Michela Buglione, Stefano M. Magrini, Barbara A. Jereczek-Fossa, Filippo Bertoni

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mauro Vallati, Berardino De Bari, Roberto Gatta, Michela Buglione, Stefano M. Magrini, et al. Exploiting Machine Learning for Predicting Nodal Status in Prostate Cancer Patients. 9th Artificial Intelligence Applications and Innovations (AIAI), Sep 2013, Paphos, Greece. pp.61-70, 10.1007/978-3-642-41142-7_7. hal-01459665

HAL Id: hal-01459665

https://inria.hal.science/hal-01459665

Submitted on 7 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Exploiting Machine Learning for Predicting Nodal Status in Prostate Cancer Patients

Mauro Vallati<fnref n="1" />, Berardino De Bari<fnref n="2" />, Roberto Gatta<fnref n="3" />, Michela Buglione<fnref n="2" />, Stefano M. Magrini<fnref n="2" />, Barbara A. Jereczek-Fossa<fnref n="4" />, and Filippo Bertoni<fnref n="5" />

**Abstract.** Prostate cancer is the second cause of cancer in males. The prophylactic pelvic irradiation is usually needed for treating prostate cancer patients with Subclinical Nodal Metestases. Currently, the physicians decide when to deliver pelvic irradiation in nodal negative patients mainly by using the Roach formula, which gives an approximate estimation of the risk of Subclinical Nodal Metestases.

In this paper we study the exploitation of Machine Learning techniques for training models, based on several pre-treatment parameters, that can be used for predicting the nodal status of prostate cancer patients. An experimental retrospective analysis, conducted on the largest Italian database of prostate cancer patients treated with radical External Beam Radiation Therapy, shows that the proposed approaches can effectively predict the nodal status of patients.

**Keywords:** Machine Learning, Classification, Medicine Applications

## 1 Introduction

Prostate cancer is the second cause of cancer in males [14]. External Beam Radiation Therapy (EBRT) has a well established role in the radical treatment of prostate cancer [1, 2], but some issues still remain to be uniformly defined amongst Radiation Oncologists. One of these issues is the role of the pelvic irradiation. The irradiation of pelvic nodes remains controversial in the treatment of intermediate and high risk nodes-negative prostate cancer. The main problem is that microscopic Subclinical Nodal Metestases (SNM) can not be detected by the available technology: the low sensitivity of computed tomography (CT) and magnetic resonance imaging (MRI) limits their usefulness in clinical staging [20]. This results in prostate cancer patients that are considered as nodes-negative even though, in fact, they have SNM.

Even if some retrospective studies [23, 24] showed a significant advantage in terms of biochemical control for patients having an extended primary tumor and

<footnote n="1">¹ School of Computing and Engineering, University of Huddersfield, UK</footnote>

<footnote n="2">² Department of Radiation Oncology, University of Brescia, Italy</footnote>

<footnote n="3">³ Department of Information Engineering, University of Brescia, Italy</footnote>

<footnote n="4">⁴ Department of Radiotherapy, European Institute of Oncology and University of Milan, Italy</footnote>

<footnote n="5">⁵ Department of Radiation Oncology, Hospital of Modena, Italy</footnote>
a risk > 15% (estimated with Roach formula [22]), other retrospective studies, that specifically addressed to establish the role of the pelvic irradiation, [3, 16, 18] failed in showing any significant advantage in favour of the pelvic irradiation.

In order to help the clinician in the decision to treat or not pelvic nodes, many predictor tools have been proposed in the literature, most of them based on linear regression analyses [22, 20, 17]. One the most diffused methods to predict the SNM of prostate cancer patients is the Roach formula [22]. The reason of its success is principally its simplicity, as it is based on only 2 parameters: the Gleason Score sum (GSsum) and the Prostate Specific Antigen (PSA) level. GSsum is obtained by the analysis of the specimens of the biopsy: the pathologist assigns a grade to the most common tumor pattern, and a second grade to the next most common tumor pattern. The two grades are added together to get a Gleason Score sum. The PSA is a protein secreted by the epithelial cells of the prostate gland. PSA is present in small quantities in the serum of men with healthy prostates, but it is often elevated in the presence of prostate cancer or other prostate disorders.

The Roach formula is shown in Equation 1. In its last formulation it gives the estimated risk of SNM and when this risk is bigger than 15-25%, Radiation Oncologists often deliver EBRT to the pelvic nodes. Despite that, this method presents some important limits in the more updated and used formulation, the principal one being the absence of information about the clinical stage of the tumor, that has been already showed to be an important factor influencing the risk of nodal involvement [17].

$$ \frac{2}{3} \times PSA + ([GSsum - 6] \times 10) \quad (1) $$

Other methods, as the Partins Tables [20], do not share the same simplicity of the Roach formula, but they consider the clinical stage of the tumor. On the other hand their accuracy is still an argument of discussion [21, 11]. Moreover, some important limits exists also for the Partins Tables: they consider only a limited part of the population of prostate cancer patients and, furthermore, it has been showed that their algorithm overestimate the presence of lymph node and seminal vesicle involvement in patients with a predicted risk of 40% or more [15]. This could mean that a large part of patients would be uselessly treated on the pelvis and exposed to the potential side effects of the pelvic irradiation (acute and/or chronic colitis, diarrhea, cystitis).

Last but not least, all the proposed algorithms estimates only a rate of risk of SNM (sometimes with large confidence intervals) and their outputs are not based on a clear dichotomic yes/not criteria. This leads to different interpretation of their outcome, that results in different treatments delivered to similar patients.

In this paper we study the exploitation of three Machine Learning (ML) classification algorithms to define patients presenting a SNM status, which can not be detected by MRI or CT, for helping the clinician in deciding the treatment of pelvic nodes. The generated models are based on several pre-treatment parameters of prostate cancer patients.
The possibility to have a reliable predictor to define the nodal status of prostate cancer patients could have a potential interest also in the field of Health Economy: it has been calculated that to not perform a staging abdominal CT in low risk patients –which would be avoided by exploiting an accurate prediction of the nodal status of the patients– could reduce medical expenditures for prostate cancer management by U.S. $ 20 - 50 million a year [20].

# 2 Methodology

## 2.1 Data

The Pattern of Practices II was retrospectively analyzed. It is the largest Italian database including clinical, diagnostic, technical and therapeutic features of prostate cancer patients treated with radical EBRT (+/- hormonal therapies) in 15 Italian Radiation Oncology centers between 1999 and 2003. A total of 1808 patients were included in this analysis.

The pre-treatment features were chosen considering the “real life” situation of the Radiation Oncologist visiting for the first time a prostate cancer patient. Their decision to treat or not the pelvic nodes would be based only on the clinical stage of the tumor, the total Gleason Score (e.g. 3+4) and the Gleason Score sum (e.g. 3+4=7), the initial PSA value, the age of the patient and the presence of an ongoing hormonal therapy. The initial clinical stage of the tumor has been defined with the TNM system, that describes the extent of the primary tumor (T stage), the absence or presence of spread to nearby lymph nodes (N stage) and the absence or presence of distant spread, or metastasis (M stage) [9]. The domains of the considered variables are defined as shown in Table 1. In this study the machine learning techniques were used for predicting the value of N: N+ indicates patients with nodal involvement, N0 indicates that patients that do not have any nodal involvement.

N+ patients were defined as those with a positive contrast enhanced pelvic magnetic resonance imaging (MRI) and/or computed tomography (CT) scan; those showing a nodal only relapse after RT were also classified as N+ (as none of them received pelvic RT). With these criteria, a total of 55 N+ patients were identified out of the 1808 patients considered.

Following the D'Amico [8] classification that takes into account the initial PSA, the GS and the clinical T stage, patients were classified in three prognostically different sets, namely: low, intermediate and high risk. Each category included, 317, 577 and 914 patients, respectively. Finally, 6/317 (1.9%), 10/577 (1.7%), 39/914 (4.2%) N+ patients were found in the 3 D'Amico sets.

## 2.2 Classification methods

The selection of the classification techniques to exploit is a fundamental part of this work. Since the objective of the study is to obtain an automated system for supporting the physicians in their work, we looked for something that is
<table><caption>**Table 1.** The domains of the considered variables on the selected patients. N represents the nodal status of patients, that is the variable that we want to predict.</caption><thead><tr><th>Variable name</th><th>Domain</th></tr></thead><tbody><tr><td></td><td>1+1, 1+2, 1+3, 2+1, 2+2, 2+3, 2+4, 2+5</td></tr><tr><td>GS</td><td>3+1, 3+2, 3+3, 3+4, 3+5, 4+2, 4+3</td></tr><tr><td></td><td>4+4, 4+5, 5+1, 5+2, 5+3, 5+4, 5+5</td></tr><tr><td>GSum</td><td>2 - 10</td></tr><tr><td>Age</td><td>46 - 86</td></tr><tr><td>Initial PSA</td><td>0.1 - 1066</td></tr><tr><td>T</td><td>1, 1a, 1b, 1c, 2, 2a, 2b, 2c, 3, 3a, 3b, 4</td></tr><tr><td>Hormonal therapy</td><td>9 possible therapies + no therapy</td></tr><tr><td>N</td><td>N+, NO</td></tr></tbody></table>

reliable, allows accurate predictions and, at the same time, it is easy to explain and to represent. After discussing with Radiation Oncologists and physicians we observed that they will trust and evaluate the suggestions of the automated system only if it is possible to understand how the decisions are taken. Mainly for the latter requirement, we focused our study on decision trees. A model generated by decision tree approaches can be easily represented, and it is intuitive to follow the flow of information and to understand how the suggested outcome is generated.

We considered three machine learning techniques for classification, based on decision trees, included in the well known tool WEKA [12]: J48 [19], Random Tree [10] and Random Forest [5]. Even if we are aware of the objective difficulty for humans to interpret the raw model generated by the Random Forest approach [4], we believe that it will be possible to represent the models generated by this classifier in an understandable format. Moreover they have demonstrated to achieve very good performance in classification [6].

## 2.3 Experimental design

The classification methods are used for classifying patients as NO (not affected by nodal metastasis) and N+ (affected by nodal metastasis). The classes of patients NO and N+ are very imbalanced through all the D'Amico categories of risk (low, medium and high), and in the whole database. The N+ population represents less than the 4% of the all entire set. For balancing the classes among the considered data sets we applied three different strategies: oversampling, undersampling, and a mix of them, as follows.

- A random undersampling of the NO patients.

- B random oversampling of the N+ patients.

- C random undersampling of the NO patients and random oversampling of the N+ patients.

Random oversampling and undersampling techniques were selected due to the improved overall classification performance, compared to a very imbalanced data
set, that base classifiers can achieve [13]. Given the fact that it is still not clear
which technique could lead to better performance, we decided to experimentally
evaluate three combinations of them.

The sampling techniques were applied on the different categories of risk, lead-
ing to nine different data sets. The resulting datasets had a distribution of about
40%-60% of, respectively, N+ and NO. Furthermore we generated a further dataset
that considers the whole population of the database. The sampling techniques
were applied also to this dataset.

Each of the selected classifying techniques was trained on the previously
described data sets separately, and the resulting predictive models were then
evaluated using a k-fold cross-validation strategy.

For the Roach formula we considered three different cut-offs: 15, 10 and 5
percent. Using a 5% cut-off corresponds to a pessimistic evaluation of the clinical
data, and leads to overestimate the risk of a patient to be N+, while the 15%
value corresponds to a more optimistic approach. The Roach formula does not
need any training on the data sets and is meant to be used as-is, for this reason
we applied it directly to the original datasets.

3 Results

In Table 2 the results of the comparison between Roach formula and the selected
machine learning algorithms are shown. For the comparison we considered four
different metrics: number of patients classified as false negative, specificity, sen-
sitivity and accuracy of the model. The number of false negative, i.e., patients
N+ classified as NO, is critical. These are patients that require to be treated with
pelvic irradiation but that, according to the classification, will not be treated.
Specificity, sensitivity and accuracy are showed as indexes of the overall quality
of the generated models. They indicate the proportion of patients classified as
NO which are actually NO (specificity), the proportion of patients classified as N+
which are actually N+ (sensitivity), and the proportion of the patients correctly
classified, regardless to their class (accuracy).

By analyzing the results shown in Table 2, we can derive some interesting
observations. Regarding the Roach formula, we can derive a precise behaviour.
Its ability in identifying affected patients increases with the category of risk, but
at the same time also the number of false positive increases dramatically. On the
other hand the accuracy, very good in low risk patients, decreases in medium
and high risk patients. The number of false negative rapidly decreases from low
risk to high risk categories, because in the latter the Roach formula is classifying
as affected a significantly large percentage of the population.

As expected, the performance of the machine learning approaches does not
significantly change between the risk categories. On the other hand the sampling
technique used on the data set has a great impact on the overall performance.
The data set obtained by applying only the undersampling (A) usually leads the
algorithms to poor performance, mainly because of the very small populations
of the resulting data sets. The only exception is the decision tree generated by
<table><caption>**Table 2.** The results, in terms of number of false negative (2nd column), specificity, sensitivity and accuracy, achieved by the Roach formula with the different cut-offs and the selected classification algorithms on low (upper), medium (middle) and high (lower) risk patients. The numbers in brackets indicate the population of the considered data set.</caption><thead><tr><th>Roach cut off</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr></thead><tbody><tr><td>15%</td><td>6</td><td>0.0</td><td><strong>100.0</strong></td><td>98.1</td></tr><tr><td>10%</td><td>6</td><td>0.0</td><td><strong>100.0</strong></td><td>98.1</td></tr><tr><td>5%</td><td>6</td><td>0.0</td><td>87.0</td><td>85.8</td></tr><tr><th>ML Approaches</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr><tr><td>J48 A (12)</td><td>6</td><td>0.0</td><td>50.0</td><td>25.0</td></tr><tr><td>J48 B (612)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>92.5</td><td>96.3</td></tr><tr><td>J48 C (354)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>94.7</td><td>97.2</td></tr><tr><td>RandomT A (12)</td><td>3</td><td>50.0</td><td>0.0</td><td>25.0</td></tr><tr><td>RandomT B (612)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>94.8</td><td>97.4</td></tr><tr><td>RandomT C (354)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>95.2</td><td>97.5</td></tr><tr><td>RandomF A (12)</td><td>6</td><td>0.0</td><td>33.3</td><td>16.7</td></tr><tr><td>RandomF B (612)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>98.7</td><td><strong>99.4</strong></td></tr><tr><td>RandomF C (354)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>98.4</td><td>99.2</td></tr></tbody></table>

<table><thead><tr><th>Roach cut off</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr></thead><tbody><tr><td>15</td><td>6</td><td>40.0</td><td>64.9</td><td>64.6</td></tr><tr><td>10</td><td>3</td><td>70.0</td><td>48.6</td><td>49.0</td></tr><tr><td>5</td><td>3</td><td>70.0</td><td>33.4</td><td>34.0</td></tr><tr><th>ML Approaches</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr><tr><td>J48 A (20)</td><td>1</td><td>90.0</td><td>40.0</td><td>65.0</td></tr><tr><td>J48 B (966)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>93.1</td><td>96.0</td></tr><tr><td>J48 C (433)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>97.7</td><td>98.9</td></tr><tr><td>RandomT A (20)</td><td>4</td><td>60.0</td><td><strong>100.0</strong></td><td>80.0</td></tr><tr><td>RandomT B (966)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>96.5</td><td>97.9</td></tr><tr><td>RandomT C (433)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>97.7</td><td>98.9</td></tr><tr><td>RandomF A (20)</td><td>4</td><td>60.0</td><td>90.0</td><td>75.0</td></tr><tr><td>RandomF B (966)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>97.7</td><td>98.7</td></tr><tr><td>RandomF C (433)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>99.1</td><td><strong>99.5</strong></td></tr></tbody></table>

<table><thead><tr><th>Roach cut off</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr></thead><tbody><tr><td>15</td><td>3</td><td>92.3</td><td>20.1</td><td>23.2</td></tr><tr><td>10</td><td>3</td><td>92.3</td><td>14.5</td><td>17.8</td></tr><tr><td>5</td><td>2</td><td>94.9</td><td>9.9</td><td>13.6</td></tr><tr><th>ML Approaches</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr><tr><td>J48 A (78)</td><td>1</td><td>97.4</td><td><strong>100.0</strong></td><td><strong>98.7</strong></td></tr><tr><td>J48 B (1460)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>88.6</td><td>93.2</td></tr><tr><td>J48 C (701)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>92.9</td><td>96.4</td></tr><tr><td>RandomT A (78)</td><td>11</td><td>71.8</td><td>79.5</td><td>75.6</td></tr><tr><td>RandomT B (1460)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>94.9</td><td>96.9</td></tr><tr><td>RandomT C (701)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>96.9</td><td>98.4</td></tr><tr><td>RandomF A (78)</td><td>2</td><td>94.9</td><td>97.4</td><td>96.2</td></tr><tr><td>RandomF B (1460)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>96.1</td><td>97.7</td></tr><tr><td>RandomF C (701)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>95.4</td><td>97.7</td></tr></tbody></table>
<table><caption>**Table 3.** The results, in terms of number of false negative (2nd column), specificity, sensitivity and accuracy, achieved by the Roach formula with the different cut-offs and the selected classification algorithms on the complete data set. The numbers in brackets indicate the population of the considered data set.</caption><thead><tr><th>Roach cut off</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr></thead><tbody><tr><td>15</td><td>15</td><td>75.4</td><td>44.2</td><td>45.1</td></tr><tr><td>10</td><td>12</td><td>80.3</td><td>33.4</td><td>34.7</td></tr><tr><td>5</td><td>11</td><td>81.2</td><td>32.0</td><td>32.7</td></tr></tbody><thead><tr><th>ML Approaches</th><th>False Neg</th><th>Specificity (%)</th><th>Sensitivity (%)</th><th>Accuracy (%)</th></tr></thead><tbody><tr><td>J48 A (110)</td><td>10</td><td>81.8</td><td>81.8</td><td>81.8</td></tr><tr><td>J48 B (3039)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>92.1</td><td>95.5</td></tr><tr><td>J48 C (1489)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>91.1</td><td>95.5</td></tr><tr><td>RandomT A (110)</td><td>14</td><td>74.6</td><td>78.2</td><td>76.4</td></tr><tr><td>RandomT B (3039)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>96.2</td><td>97.8</td></tr><tr><td>RandomT C (1489)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>94.8</td><td>97.4</td></tr><tr><td>RandomF A (110)</td><td>9</td><td>83.6</td><td>78.2</td><td>80.9</td></tr><tr><td>RandomF B (3039)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td><strong>96.9</strong></td><td><strong>98.2</strong></td></tr><tr><td>RandomF C (1489)</td><td><strong>0</strong></td><td><strong>100.0</strong></td><td>96.3</td><td>98.1</td></tr></tbody></table>

the J48 algorithm for the high risk patients, which achieved significant results in both accuracy and sensitivity. This is a surprising result, given the fact that the use of the cross-validation technique should avoid (or, at least, significantly limit) the overfitting issue [13, 7].

Usually the best performance are achieved by the classification models generated by the random forest algorithm on either B or C data sets. It is also important to note that *all* the models generated by the selected machine learning approaches lead to 0 false negative while exploiting B or C sampling techniques.

In Table 3 the results of the comparison between Roach formula and the selected machine learning algorithms, on the whole population, are shown. A striking result is that the number of false negative is still 0 for all the selected machine learning approaches on the data sets generated by the B or C sampling techniques. This results is confirmed by the 100.0% specificity value. It should also be noticed that a very few number of patients that are classified as N+ by the proposed techniques, are actually NO; this means only a small number of patients will receive a pelvic irradiation, and it is in contrast with the results obtained with the classical Roach method, even with the more optimistic cut-off (15%). Furthermore the overall accuracy is very high.

Regarding the Roach formula, the high number of high risk patients, that leads the formula to define as N+ a very high percentage of NO patients (as shown in Table 2), results in poor performance. It is also disappointing that many affected patients are classified as NO even by the most pessimistic cut-off of the 5%. We experimentally observed that by using smaller cut-offs, almost all the high risk patients will be considered as N+; considering that they are actually the 4.2%, this result is not relevant and the corresponding cut-off cannot be use by physicians for taking every day decisions.
# 4 Conclusions and future work

Prostate cancer is the second cause of cancer in males. Radiotherapy has a well established role in the radical treatment of prostate cancer, but some issues still remain to be uniformly defined amongst Radiation Oncologists. One of these issues is the role of the pelvic irradiation and, in particular, when a prostate cancer patient should or should not receive it. Currently, the decision regarding the pelvic irradiation is still taken mainly by estimating the risk through the Roach formula, which is very inaccurate. The main issue of this formula is related to the small number of pre-treatment parameters that it considers.

In this paper we proposed a different approach, based on three well known classification algorithms: J48, random tree and random forest. The decision trees algorithms have been chosen due to their intuitive representation, which could make them more trustable for the physicians. These techniques have been exploited for generating a predictive model that will support physicians in taking every day decisions. In order to test and experimentally evaluate the proposed approaches we used the largest Italian database of prostate cancer patients treated with radical EBRT. Given the fact that the classes to predict were very imbalanced, we applied random oversampling and undersampling techniques for preprocessing them; this lead to three different data set that were then compared with the Roach formula. The results obtained by the ML algorithms are significantly better; they show that the decision trees classification algorithms can be effectively exploited for supporting the physicians in every day work. The proposed models are usually able to correctly identify all the N+ patients and, furthermore, to limit the number of NO patients that will receive a pelvic irradiation. Moreover, it seems that the application of both undersampling and oversampling is the best strategy for handling this data.

We see several avenues for future work. Concerning the model generation, we are interested in evaluating the proposed approaches on different, and possibly larger, prostate cancer patients databases. We are also interested in studying the possible differences between different populations of patients (maybe gathered from different geographical regions) and the generated models. It could happen that very different populations lead to the generation of significantly different predictive models; in this case the proposed approach can be explicitly trained for handling different populations.

On the other hand, we are interested in studying a technique for selecting the model to exploit for supporting physicians decisions. A possibility could be the combination of the classifications provided by the models generated by the three exploited classifiers. Additionally, better results could be achieved by clustering the patients and selecting, for each cluster, the best performing approach.

Finally, we are planning to make available a decision support system based on the proposed study. The main features of the DSS that we are designing are: (i) let clinicians to be able to predict the nodal status of new patients; (ii) update the patients database by integrating new patients; (iii) keep the predictive model updated with the new added patients and, (iv) receive feedback from the community. The proposed enhanced DSS will have an intuitive presentation of
the models. It will be made available online and via applications for mobile
devices; they are currently extensively used by clinicians in every day work.

**Acknowledgement.** We acknowledge the support provided by the following colleagues: Cynthia Aristei, Enza Barbieri, Giampaolo Biti, Ines Cafaro, Dorian Cosentino, Sandro Fongione, Paola Franzone, Paolo Muto, Pietro Ponticelli, Riccardo Santoni, Alessandro Testolin, Giuseppina Apicella, Debora Beldi, Filippo De Rienzo, Pietro G. Gennari, Gianluca Ingrosso, Fernando Munoz, Andrea Rampini, Enzo Ravo, Ludovica Pegurri, Giovanni Girelli, Alessia Guarnieri, Icro Meattini, Umberto Ricardi, Monica Mangoni, Pietro Gabriele, Rita Bellavita, Marco Bellavita, Alberto Bonetta, Ernestina Cagna, Feisal Bunkheila, Simona Borghesi, Marco Signor, Adriano Di Marco, Marco Stefanacci.

References

1. National collaborating centre for cancer. CG58 prostate cancer: evidence review. http://www.nice.org.uk/nicemedia/pdf/CG58EvidenceReview.pdf (2008)

2. National cancer institute. PDQ cancer information summaries. http://www.cancer.gov/cancertopics/pdq/treatment/prostate/HealthProfessional/page4#Reference4.21 (2012)

3. Asbell, S.O., Martz, K.L., Shin, K.H., Sause, W.T., Doggett, R.L., Perez, C.A., Pilepich, M.V.: Impact of surgical staging in evaluating the radiotherapeutic outcome in RTOG #77-06, a phase III study for T1BN0M0 (A2) and T2N0M0 (B) prostate carcinoma. International Journal of radiation oncology, biology, physics 40(4), 769-782 (1998)

4. Berthold, M.R., Borgelt, C., Hppner, F., Klawonn, F.: Guide to Intelligent Data Analysis: How to Intelligently Make Sense of Real Data. Springer, 1st edn. (2010)

5. Breiman, L.: Random forests. Machine Learning 45, 5–32 (2001)

6. Caruana, R., Karampatziakis, N., Yessenalina, A.: An empirical evaluation of supervised learning in high dimensions. In: Proceedings of the 25th international conference on Machine learning. pp. 96–103. ACM (2008)

7. Cawley, G.C., Talbot, N.L.C.: On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research pp. 2079–2107 (2010)

8. D'Amico, A.V., Whittington, R., Malkowicz, S.B., Schultz, D., Blank, K., Broderick, G.A., Tomaszewski, J.E., Renshaw, A.A., Kaplan, I., Beard, C.J., Wein, A.: Biochemical outcome after radical prostatectomy, external beam radiation therapy, or interstitial radiation therapy for clinically localized prostate cancer. JAMA 280, 969–974 (1998)

9. Edge, S.B., Byrd, D.R., Compton, C.C., Fritz, A.G., Greene, F.L., Trotti, A.: AJCC Cancer Staging Manual, 7th ed. Springer (2010)

10. Fan, W., Wang, H., Yu, P., Ma, S.: Is random model better? on its accuracy and efficiency. In: Proceedings of the 3rd IEEE International Conference on Data Mining. pp. 51–58. IEEE (2003)

11. Graefena, M., Augustina, H., Karakiewiczb, P.I., Hammerera, P.G., Haesea, A., Palisaara, J., Blonskia, J., Fernandeza, S., Erbersdoblerc, A., Hulanda, H.: Can predictive models for prostate cancer patients derived in the united states of america be utilized in european patients? a validation study of the partin tables. European Urology 43(1), 6–11 (2003)
12. Hall, M., Holmes, F.G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: An update. SIGKDD Explorations 11(1), 10–18 (2009)

13. He, H., Garcia, E.A.: Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering 21(9), 1263–1284 (2009)

14. Jemal, A., Bray, F., Center, M.M., Ferlay, J., Ward, E., Forman, D.: Global cancer statistics. CA: A Cancer Journal for Clinicians 61(2), 69–90 (2011)

15. Kattan, M.W., Stapleton, A.M., Wheeler, T.M., Scardino, P.T.: Evaluation of a nomogram used to predict the pathologic stage of clinically localized prostate carcinoma. Cancer 79, 528–537 (1997)

16. Lawton, C.A., DeSilvio, M., Roach, M., Uhl, V., Kirsch, R., Seider, M., Rotman, M., Jones, C., Asbell, S., Valicenti, R., Hahn, S., Thomas, C.R.: An update of the phase III trial comparing whole pelvic to prostate only radiotherapy and neoadjuvant to adjuvant total androgen suppression: updated analysis of RTOG 94-13, with emphasis on unexpected hormone/radiation interactions. International Journal of Radiation Oncology, biology, physics 69, 646–655 (2007)

17. Partin, A.W., Kattan, M.W., Subong, E.N., Walsh, P.C., Wojno, K.J., Oesterling, J.E., Scardino, P.T., Pearson, J.D.: Combination of prostate-specific antigen, clinical stage, and Gleason score to predict pathological stage of localized prostate cancer. A multiinstitutional update. JAMA 277, 1445–1451 (1997)

18. Pommier, P., Chabaud, S., Lagrange, J.L., Richaud, P., Lesaunier, F., Le Prise, E., Wagner, J.P., Hay, M.H., Beckendorf, V., Suchaud, J.P., Pabot du Chatelard, P.M., Bernier, V., Voirin, N., Perol, D., Carrie, C.: Is there a role for pelvic irradiation in localized prostate adenocarcinoma? preliminary results of GETUG-01. International Journal of Radiation Oncology, biology, physics 25(34), 5366–5373 (2007)

19. Quinlan, J.R.: C4.5: programs for machine learning. Morgan Kaufmann (1993)

20. Reckwitz, T., Potter, S.R., Partin, A.W.: Prediction of locoregional extension and metastatic disease in prostate cancer: a review. World journal of urology 18, 165–172 (2000)

21. Regnier-Couderta, O., McCalla, J., Lothiana, R., Lamb, T., McClintonb, S., N'Dowb, J.: Machine learning for improved pathological staging of prostate cancer: A performance comparison on a range of classifiers. Artificial Intelligence in Medicine 55(1), 25–35 (2012)

22. Roach, M., Marquez, C., Yuo, H.S., Narayan, P., Coleman, L., Nseyo, U.O., Navvab, Z., Carroll, P.R.: Predicting the risk of lymph node involvement using the pre-treatment prostate specific antigen and gleason score in men with clinically localized prostate cancer. International Journal of Radiation Oncology, biology, physics 28, 33–37 (1994)

23. Seaward, S.A., Weinberg, V., Lewis, P., Leigh, B., Phillips, T.L., Roach, M.: Identification of a high-risk clinically localized prostate cancer subgroup receiving maximum benefit from whole-pelvic irradiation. The Cancer Journal from Scientific American 4(6), 370–377 (1998)

24. Seaward, S.A., Weinberg, V., Lewis, P., Leigh, B., Phillips, T.L., Roach, M.: Improved freedom from PSA failure with whole pelvic irradiation for high-risk prostate cancer. International Journal of Radiation Oncology, biology, physics 42(5), 1055–1062 (1998)