BACKGROUND
differential diagnosis of cancer plays a crucial role in addressing medical therapies and surgical interventions.
however, cancer diagnosis can become a very difficult task in the presence of nonspecific symptoms and different malignancies involving the same cancer site.
malignant pleural mesothelioma  is a rare highly fatal tumour, whose incidence is rapidly increasing in developed countries due to the widespread past exposure to asbestos in environmental and occupational settings  <cit> .
the correct diagnosis of mpm is often hampered by the presence of atypical clinical symptoms that may cause misdiagnosis with either other malignancies  or benign inflammatory or infectious diseases  causing pleurisies  <cit> .
cytological examination  may allow to identify malignant cells, but sometimes a very high false negative proportion may be encountered due to the high prevalence of non-neoplastic cells  <cit> .
moreover, in most cases a positive result from ce only does not allow to distinguish mpm from other malignancies  <cit> .
many tumour markers  have been demonstrated to be useful complementary tools for the diagnosis of mpm  <cit> .
in particular, a recent investigation, based on pairwise comparisons by standard roc analysis, analysed the concentrations of three tumour markers in pleural effusions, namely: the soluble mesothelin-related peptide , cyfra 21- <dig> and cea, and their association with a differential diagnosis of mpm, pleural metastasis from other tumours  and bd  <cit> .
smrp showed the best performance in separating mpm from both mtx and bd, while high values of cyfra 21- <dig> were associated to both mpm and mtx.
conversely, high concentrations of cea were mainly observed in patients with mtx.
taken together, these results indicate that information from the three considered markers and from ce might be combined together in order to obtain a classifier to separate mpm from both mtx and bd.
logic learning machine  is an innovative method of supervised data mining able to provide threshold-based rules for classification purposes  <cit> .
the present investigation is aimed at illustrating the application of llm for the differential diagnosis of mpm by identifying simple and intelligible rules based on ce and tm concentration.
preliminary results of the present study have been published as an extended abstract in the framework of the bioinformatics italian society annual meeting  <dig>  <cit> .
methods
data set description
a consecutive cohort of  <dig> patients admitted for diagnosis to two pulmonary departments in northern italy from  <dig> to  <dig> was considered as eligible.
concentration of smrp, cyfra 21- <dig> and cea tumour markers was measured in pleural effusion as described by filiberti et al.
<cit> .
all patients underwent ce, while  <dig> had at least one missing data for a considered tm, and were consequently excluded from the study, thus leaving  <dig> patients available for the analyses .
study design was carried out according to the protocol "research on pulmonary diseases" approved by the ethical committee of ao villa scassi hospital of genoa, italy, on  <dig> december  <dig>
an informed consent for analysis of pleural fluid was obtained from all patients.
descriptive statistics of the three considered tm and results of ce are resumed in table  <dig>  smrp concentration was higher among mpm than in the other two classes, whereas cyfra 21- <dig> showed very low values among bd and higher values among the two malignancies, with the highest median concentration observed for mpm.
cea showed high values among mtx and similar low values among the other two classes.
the corresponding interquartile ranges were largely overlapping, indicating that no considered tm can provide a perfect separation between mpm and the other two classes.
finally, ce provided a positive result in about one third of mpm and a half of mtx patients only, confirming the very low sensitivity of such technique  <cit> .
furthermore, a positive ce result was observed among bd, which corresponded to a very old patient who died after a short period of follow-up, as described in filiberti et al.
<cit> .
it remains unclear if it was due to the occurrence of some latent pleural malignancy or it actually represents a false positive result.
mpm = malignant pleural mesothelioma; mtx = metastasis; bd = benign disease ; iqr = interquartile range.
llm classification rules
information from tumour marker concentrations and ce was combined using a set of simple intelligible rules, automatically generated by the llm algorithm, which is an efficient implementation of the switching neural network model  <cit> .
in more details, let x ∈ℜd be a d-dimensional example in a classification problem to be assigned to one of q possible classes, labeled by the values of a categorical output y. starting from a training set s including n pairs , i =  <dig> ..., n, deriving from previous observation, llm has the aim of generating a classifier, i.e.
a model g that provides the correct answer y = g for most input patterns x. concerning the components xj two different situations can be devised: a) ordered variables: xj varies within an interval  of the real axis and an ordering relationship exists among its values; b) nominal  variables: xj can assume only the values contained in a finite set and there is no ordering relationship among them.
llm generates an intelligible model g described by a set of m rules rk, k =  <dig> ..., m, in the if-then form:
if<premise>then<consequence>
where <premise> is the logical product  of mk conditions ckl, with l =  <dig> ..., mk, on the components xj, whereas <consequence> gives a class assignment y = ỹ for the output.
in general, a condition ckl in the premise involving an ordered variable xj has one of the following forms xj >λ, xj ≤ µ, λ <xj ≤ µ, being λ and µ two real values, whereas a nominal variable xj leads to membership conditions xj ∈ {α, δ, σ}, being α, δ, σ admissible values for the j-th component of x.
for instance, if x <dig> is an ordered variable in the domain  <cit>  and x <dig> is a nominal component assuming values in the set {a, b, c}, a possible rule r <dig> is:
if x1> 40 and x2ϵ{a,b} then y= <dig>
where  <dig> denotes one of the q possible assignments .
the llm algorithm for rule extraction
intelligible classification rules described in the previous paragraph are generated by llm following the three steps illustrated in figure  <dig>  during the first step , data are binarized according to the inverse only one code, which allows to preserve ordering and distance when used to transform both ordered and nominal attributes  <cit> .
by means of binarisation, each example is therefore transformed into a string z ∈{ <dig> }v of binary values .
the length of these strings , denoted by v, depends on the number of inputs and on the number of values that each input assumes in the training set.
after this step the training set has been translated into a binary input-output matrix which can be seen as a portion of the truth table of a monotone boolean function.
the second step adopts a proper technique for digital synthesis capable of retrieving in a reasonable time a monotone boolean function consistent with a partially described truth table.
a method of this kind is the shadow clustering  algorithm  <cit> , which builds at each iteration a new logical product  to be added to the final and-or expression for the monotone boolean function and adopts specific approaches to increase the classification ability of each produced implicant.
in particular, the maximum covering shadow clustering criterion  <cit>  attempts to increase the number of training patterns covered by each implicant while keeping low its complexity.
extensive trials has shown that the resulting procedure leads to excellent intelligible models in the analysis of several real-world classification problems  <cit> .
finally, in the third step from every generated implicant an intelligible rule, including a logical product  of simple conditions, is automatically retrieved.
the resulting set of rules forms the classification model for the problem at hand.
more details about sc implementation and estimates of the related computational burden under different scenarios are provided in dedicated papers  <cit> .
quality measures of llm and class prediction
according to the output value included in their consequence part, the m rules rk describing a given model g can be subdivided into q groups g <dig>  g <dig> ..., gq.
considering the training set s, any rule r ∈ gl is characterized by four quantities: the numbers true positive ) and false positive ) of examples  with yi = yl and yi ≠ yl, respectively, that satisfy all the conditions in the premise of r, and the numbers false negative ) and true negative ) of examples  with yi = yl and yi ≠ yl, respectively, that do not satisfy at least one of the conditions in the premise of r.
starting from tp, fp, tn, and fn, other useful characteristic quantities, such as the covering c, the error e, and the precision p can be derived:
c=tptp+fn
e=fptn+fp
p=tptp+fp
c and p are also known as the sensitivity and the positive predictive value in clinical epidemiology setting, while e corresponds to  <dig> - specificity.
c and p are usually adopted as measures of relevance for a rule r. as a matter of fact, the greater is the covering and the precision, the higher is the generality and the correctness of the corresponding rule.
on the other hand, to obtain a measure of relevance r for a condition c included in the premise part of a rule r, one can consider the rule r' obtained by removing that condition from r. since the premise part of r' is less stringent, we obtain that e ≥ e so that the quantity:
r=c-e)
can be used as a measure of relevance for the condition c of interest.
since each condition c refers to a specific component of x, it is also possible to define a measure of relevance rj for every input variable xj:
rj=1-∏kr
where the product is computed on the rules rk that includes a condition ckl on the variable xj.
the model g generated by the llm task of rulex can be adopted to produce the output class for any input pattern x*, including those that do not verify any generated rule, provided that at least one condition inside at least one rule was verified.
to this aim the <premise> part of each of the m intelligible rules rk, k =  <dig> ..., m, describing the model g, is checked to analyze if it is verified by the considered sample x*.
let d be the number of conditions in the <premise> part of the rule rk that are not verified by the pattern x*.
then, for every output class yl we can determine the minimum value dl=minr∈gld and the subset hl of rules in the group gl characterized by that minimum:
hl={r∈gl|d=dl}
then, we choose as output value for the pattern x* the class l scoring the lowest dl and, in case of ties, the minimum value of the quantity wl defined as
wl= ∏r∈hl1-∑verifiedcinrr
where the summation is performed on all the conditions c in the <premise> part of the rule r that are verified by the sample x*.
llm performance assessment
in order to obtain an unbiased estimate of the llm performance, data were analysed according to a leave-one-out cross-validation .
rules were generated allowing a maximum error rate of 5% in the training set.
accuracy of llm classification applied to the test set was compared to that of selected competing standard methods of supervised analysis, namely: k-nearest neighbour classifier , artificial neural network , and decision tree .
in particular, dt, similarly to llm, is able to generate threshold-based intelligible rules.
for this reason, we performed a comparison between the rules generated by llm and those obtained by dt.
llm is implemented as part of the rulex software suite, developed and distributed by rulex inc .
competing methods
a brief description of competing methods  is here given; details regarding their use and implementation can be found in standard books of data mining  <cit> .
k-nearest-neighbor
although knn is one of the simplest technique for classifying previously unseen patterns x taking into account the information contained in a given training set s, it can achieve a good accuracy even in complex situations.
its approach is very straightforward: when an input vector x has to be classified, knn searches for the k nearest points x <dig>  x <dig> ..., xk in s according to a given definition of distance.
then, it assigns to x the most common class present in x <dig>  x <dig> ..., xk.
the value of k is usually chosen to avoid ties .
although the adopted definition of distance can affect the accuracy of the knn classifier, very often the standard euclidean distance is employed, after having normalized the components of x to avoid undesirable effects due to unbalanced domain intervals in different input variables.
in the reported trials the choice k =  <dig> was performed, which corresponded to assign to any previously unseen point x the class of its nearest neighbor in the training set s.
artificial neural network
building a classifier starting from a given training set s corresponds to determining a subset of the input domain for each output class or, equivalently, to constructing proper separating surfaces that delimit these subsets.
in general, each separating surface can be nonlinear and even complex, depending on the specific classification problem at hand.
a convenient way to manage this complexity is to build the separating surface through the composition of simpler functions.
this approach is followed by ann, a connectionist model formed by the interconnection of simple units, called neurons, arranged in layers.
each neuron performs a weighted sum of its inputs  and applies a proper activation function to obtain the output value that will be propagated to the following layer.
the first layer of neurons is fed by the components of the input vector x, whereas the last layer produces the output class to be assigned to x.
suitable optimization techniques are used to retrieve the weights for each neuron, which form the set of parameters for the ann.
by properly setting these weights we can obtain separating surfaces arbitrarily complex, provided that a sufficient number of neurons is included in the ann.
the choice of this quantity, together with the selection of the number of layers, must be performed at the beginning of the training process and affect the generalisation ability of the resulting model.
decision trees
an intelligible classifier can be obtained by generating a tree graph where each node is associated with a condition on a component of the input vector x  and each leaf corresponds to an assignment for the output class to be assigned to x. a model of this kind is called decision tree.
it is straightforward to retrieve an intelligible rule for the classification problem at hand by navigating the decision tree from a leaf to the root and by using as antecedent for the rule the logical product  of the conditions associated with the nodes encountered during the navigation.
rules obtained in these way are disjoint from each other.
although different learning algorithms have been proposed for building a dt, a basic divide-and-conquer strategy is followed by all of them.
at each iteration a new node is added to the dt by considering a subset of s  and by choosing the condition that subdivides it in the best way, according to a specific measure of goodness.
with this approach the size of the subset pertaining to added nodes decreases during the construction of the tree, which halts when a specific stopping criterion is reached .
proper pruning techniques are adopted to simplify the final dt with the aim of reducing its complexity and increasing its generalisation ability.
RESULTS
comparison between the performance of llm and that of the other supervised methods
mpm = malignant pleural mesothelioma; mtx = metastasis; bd = benign diseases; llm = logic learning machine; dt = decision tree; ann = artificial neural network; knn = k-nearest neighbour classifier.
classification rules obtained by llm
llm and dt analyses were repeated on the entire dataset in order to obtain stable rules for patients classification.
llm classifier included  <dig> rules, but  <dig> of them had a very low covering .
diag.
= diagnosis; mpm = malignant pleural mesothelioma; mtx = metastasis; bd = benign diseases ; ce = cytological examination
diag.
= diagnosis; w% = e - e; r% = relevance%; cov.
% = covering percent.
w and r are defined according to equation .
classification rules obtained by dt
dt classification was based on  <dig> rules.
discussion
llm is an innovative method that can provide useful classification rules by exploiting the complex multivariable correlation between the different analysed features.
llm has been recently successfully applied to a variety of datasets in biomedical settings  <cit> , but so far it has not been used for differential diagnosis of cancer patients based on tumour markers combination.
in the last decades many other methods of supervised data mining have been successfully applied to classification tasks in different biomedical fields, including oncology.
in particular, ann and knn have shown a good accuracy in many instances  <cit> .
however, they represent "black-box" methods that cannot provide useful insights about biological and clinical aspects of the disease under study.
for this reason, intelligible "and- type" and "or-type" rules are in general preferred, but methods for multi-class classification are scarce.
among them, dt is probably the most widely used tool for its simplicity and easily implementation, but in general it tends to show a low accuracy when compared to other supervised methods  <cit> .
however, in the present investigation, dt provided a quite good both total and class-specific accuracy that was higher than that obtained from the two black-box algorithms.
however, llm outperformed all competing methods including dt.
in particular, dt performance was slightly lower among bd patients, slightly higher for mpm class only and clearly lower for both mtx and the pool of malignancies.
classification rules obtained from both llm and dt were in good agreement with a priori knowledge about the considered tumour markers.
in particular, high values of cea were associated with mtx class with a covering of about 50% for both methods .
such a proportion roughly corresponds to the percentage of patients with pleural metastasis from lung adenocarcinoma inside the analysed cohort  <cit> .
this finding is in agreement with the characteristics of cea marker, which is largely expressed among cancers from epithelial origin  <cit> .
moreover, high smrp concentrations were associated with mpm classification.
this finding confirms previous observations from other independent cohorts reporting that smrp concentration in pleural fluid is specific in distinguishing mesothelioma from both benign and all other malignant effusions  <cit> .
finally, low values of cyfra 21- <dig> were associated with bd classification with a high coverage in both methods , in agreement with previous studies that have associated high values of this marker to a large variety of neoplastic diseases  <cit> .
rules extracted by llm and dt only partly overlapped, thus reflecting a sort of balance between the capability of the two methods of identifying useful information for classification purposes, and, on the other hand, the two very different algorithms for rule generation.
for instance, the rule with the highest covering for mpm classification was almost identical in the two methods, both including high values of smrp and cyfra 21- <dig> and low values of cea, with very similar cut-offs.
conversely, the best rule for mtx classification was rather different, including only high values of cea for dt, and a combination of high values of cea  and high cyfra 21- <dig> concentrations for llm.
finally, the best classification for bd was obtained from both methods by low values of both cyfra 21- <dig> and cea, but at different cut-offs.
furthermore, llm rule also included negative ce .
on the whole, our results indicate that both llm and dt are able to extract meaningful information from tumour markers and to combine them in simple rules for classification tasks.
dt also provides a simple plot that allows a very easy interpretation of the rules generated, whereas llm rules, being partly overlapping, provide a rather more complicated picture.
however, in our analysis, in agreement with results from previous investigations  <cit>  the presence of overlapping rules allowed llm to outperform dt classification.
furthermore, a non-ambiguous classification can always be obtained by using coverage and error rate parameters and by adopting a proper measure of relevance that allows to select the most probable class for the pattern at hand.
moreover, overlapping rules can be weighted in order to improve classification accuracy in the presence of severely unbalanced sample size  <cit> , thus conferring a high flexibility to llm based classification.
results of our investigation should be evaluated at the light of some unavoidable limits, in particular the rather small sample size.
mesothelioma is a rare cancer and, at least at our knowledge, larger datasets including all the three tms considered in the present study are not available.
the possibility that the comparison between the selected classification methods could have been influenced by the size of classes under investigation cannot be completely ruled out.
however, in some previous analyses, carried out in different biomedical fields, llm was demonstrated to outperform other methods of machine learning when applied to large datasets.
in particular, llm accuracy was higher than that of two competing methods  in a feature selection task using data from three real and three simulated databases from microarray experiments, each based on many thousands of gene expression profiles  <cit> .
furthermore, in a recent analysis of biomedical datasets of the statlog benchmark  <cit> , which included a large database of  <dig> cases of diabetes and  <dig> healthy controls, llm systematically outperformed four competing methods of learning machine   <cit> .
another limit of our investigation is the low accuracy for mtx classification, even if this latter was better classified by llm than by the considered competing methods.
finally, the set of rules generated by llm does not cover all the possible combinations between tumour markers and ce results, then making potentially difficult the classification of some additional patients.
such a limitation can be overcome by llm when the features associated to the subject at least fulfil a subset of one or more conditions inside a composite rule, by combining accuracy measures using equations  and .
CONCLUSIONS
results from the present study indicates that llm is a flexible and powerful tool for the differential classification of malignant mesothelioma patients.
dt performance was poorer, but, quite surprisingly, clearly better than the two selected "black-box" competing methods.
further studies on larger cohorts are needed in order to obtain stable and reproducible rules for mpm classification.
moreover, additional tumour markers should be tested to improve the classification of non-mesothelioma cancers with pleural metastasis.
list of abbreviations
llm: logic learning machine.
dt: decision tree.
knn: k-nearest neighbour classifier.
ann: artificial neural network.
mpm: malignant pleural mesothelioma.
bd: benign disease.
mtx: metastasis from non-mesothelioma cancers.
cea: carcino-embryonic antigen.
cyfra 21-1: soluble fragment of cytokeratin-19
smrp: soluble mesothelin-related peptide.
ce: cytological examination.
tm: tumour marker.
competing interests
the authors declare that they have no competing interests.
authors' contributions
sp, rf and mm  conceived the study and wrote the paper.
moreover, sp performed most analyses, while mm conceived and implemented the llm method.
pm, rl, gpi and mm  provided data of tumour marker concentrations and contributed in the interpretation of biological meaning of results and in writing the discussion section.
ef, cm and em implemented most routines for supervised analysis.
ef also supervised data analyses.

 differential diagnosis of cancer plays a crucial role in addressing medical therapies and surgical interventions. however, cancer diagnosis can become a very difficult task in the presence of nonspecific symptoms and different malignancies involving the same cancer site. malignant pleural mesothelioma is a rare highly fatal tumour, whose incidence is rapidly increasing in developed countries due to the widespread past exposure to asbestos in environmental and occupational settings. the correct diagnosis of mpm is often hampered by the presence of atypical clinical symptoms that may cause misdiagnosis with either other malignancies or benign inflammatory or infectious diseases causing pleurisies. cytological examination may allow to identify malignant cells, but sometimes a very high false negative proportion may be encountered due to the high prevalence of non-neoplastic cells. moreover, in most cases a positive result from ce only does not allow to distinguish mpm from other malignancies. many tumour markers have been demonstrated to be useful complementary tools for the diagnosis of mpm. in particular, a recent investigation, based on pairwise comparisons by standard roc analysis, analysed the concentrations of three tumour markers in pleural effusions, namely: the soluble mesothelin-related peptide, cyfra 21- and cea, and their association with a differential diagnosis of mpm, pleural metastasis from other tumours and bd. smrp showed the best performance in separating mpm from both mtx and bd, while high values of cyfra 21- were associated to both mpm and mtx. conversely, high concentrations of cea were mainly observed in patients with mtx. taken together, these results indicate that information from the three considered markers and from ce might be combined together in order to obtain a classifier to separate mpm from both mtx and bd. logic learning machine is an innovative method of supervised data mining able to provide threshold-based rules for classification purposes. the present investigation is aimed at illustrating the application of llm for the differential diagnosis of mpm by identifying simple and intelligible rules based on ce and tm concentration. preliminary results of the present study have been published as an extended abstract in the framework of the bioinformatics italian society annual meeting. methods data set description a consecutive cohort of patients admitted for diagnosis to two pulmonary departments in northern italy from to was considered as eligible. concentration of smrp, cyfra 21- and cea tumour markers was measured in pleural effusion as described by filiberti et al.. all patients underwent ce, while had at least one missing data for a considered tm, and were consequently excluded from the study, thus leaving patients available for the analyses. study design was carried out according to the protocol "research on pulmonary diseases" approved by the ethical committee of ao villa scassi hospital of genoa, italy, on december an informed consent for analysis of pleural fluid was obtained from all patients. descriptive statistics of the three considered tm and results of ce are resumed in table smrp concentration was higher among mpm than in the other two classes, whereas cyfra 21- showed very low values among bd and higher values among the two malignancies, with the highest median concentration observed for mpm. cea showed high values among mtx and similar low values among the other two classes. the corresponding interquartile ranges were largely overlapping, indicating that no considered tm can provide a perfect separation between mpm and the other two classes. finally, ce provided a positive result in about one third of mpm and a half of mtx patients only, confirming the very low sensitivity of such technique. furthermore, a positive ce result was observed among bd, which corresponded to a very old patient who died after a short period of follow-up, as described in filiberti et al.. it remains unclear if it was due to the occurrence of some latent pleural malignancy or it actually represents a false positive result. mpm = malignant pleural mesothelioma; mtx = metastasis; bd = benign disease; iqr = interquartile range. llm classification rules information from tumour marker concentrations and ce was combined using a set of simple intelligible rules, automatically generated by the llm algorithm, which is an efficient implementation of the switching neural network model. in more details, let x ∈ℜd be a d-dimensional example in a classification problem to be assigned to one of q possible classes, labeled by the values of a categorical output y. starting from a training set s including n pairs, i =..., n, deriving from previous observation, llm has the aim of generating a classifier, i.e. a model g that provides the correct answer y = g for most input patterns x. concerning the components xj two different situations can be devised: a) ordered variables: xj varies within an interval of the real axis and an ordering relationship exists among its values; b) nominal variables: xj can assume only the values contained in a finite set and there is no ordering relationship among them. llm generates an intelligible model g described by a set of m rules rk, k =..., m, in the if-then form: if<premise>then<consequence> where <premise> is the logical product of mk conditions ckl, with l =..., mk, on the components xj, whereas <consequence> gives a class assignment y = ỹ for the output. in general, a condition ckl in the premise involving an ordered variable xj has one of the following forms xj >λ, xj ≤ µ, λ <xj ≤ µ, being λ and µ two real values, whereas a nominal variable xj leads to membership conditions xj ∈ {α, δ, σ}, being α, δ, σ admissible values for the j-th component of x. for instance, if x is an ordered variable in the domain and x is a nominal component assuming values in the set {a, b, c}, a possible rule r is: if x1> 40 and x2ϵ{a,b} then y= where denotes one of the q possible assignments. the llm algorithm for rule extraction intelligible classification rules described in the previous paragraph are generated by llm following the three steps illustrated in figure during the first step, data are binarized according to the inverse only one code, which allows to preserve ordering and distance when used to transform both ordered and nominal attributes. by means of binarisation, each example is therefore transformed into a string z ∈{ }v of binary values. the length of these strings, denoted by v, depends on the number of inputs and on the number of values that each input assumes in the training set. after this step the training set has been translated into a binary input-output matrix which can be seen as a portion of the truth table of a monotone boolean function. the second step adopts a proper technique for digital synthesis capable of retrieving in a reasonable time a monotone boolean function consistent with a partially described truth table. a method of this kind is the shadow clustering algorithm, which builds at each iteration a new logical product to be added to the final and-or expression for the monotone boolean function and adopts specific approaches to increase the classification ability of each produced implicant. in particular, the maximum covering shadow clustering criterion attempts to increase the number of training patterns covered by each implicant while keeping low its complexity. extensive trials has shown that the resulting procedure leads to excellent intelligible models in the analysis of several real-world classification problems. finally, in the third step from every generated implicant an intelligible rule, including a logical product of simple conditions, is automatically retrieved. the resulting set of rules forms the classification model for the problem at hand. more details about sc implementation and estimates of the related computational burden under different scenarios are provided in dedicated papers. quality measures of llm and class prediction according to the output value included in their consequence part, the m rules rk describing a given model g can be subdivided into q groups g g..., gq. considering the training set s, any rule r ∈ gl is characterized by four quantities: the numbers true positive ) and false positive ) of examples with yi = yl and yi ≠ yl, respectively, that satisfy all the conditions in the premise of r, and the numbers false negative ) and true negative ) of examples with yi = yl and yi ≠ yl, respectively, that do not satisfy at least one of the conditions in the premise of r. starting from tp, fp, tn, and fn, other useful characteristic quantities, such as the covering c, the error e, and the precision p can be derived: c=tptp+fn e=fptn+fp p=tptp+fp c and p are also known as the sensitivity and the positive predictive value in clinical epidemiology setting, while e corresponds to - specificity. c and p are usually adopted as measures of relevance for a rule r. as a matter of fact, the greater is the covering and the precision, the higher is the generality and the correctness of the corresponding rule. on the other hand, to obtain a measure of relevance r for a condition c included in the premise part of a rule r, one can consider the rule r' obtained by removing that condition from r. since the premise part of r' is less stringent, we obtain that e ≥ e so that the quantity: r=c-e) can be used as a measure of relevance for the condition c of interest. since each condition c refers to a specific component of x, it is also possible to define a measure of relevance rj for every input variable xj: rj=1-∏kr where the product is computed on the rules rk that includes a condition ckl on the variable xj. the model g generated by the llm task of rulex can be adopted to produce the output class for any input pattern x*, including those that do not verify any generated rule, provided that at least one condition inside at least one rule was verified. to this aim the <premise> part of each of the m intelligible rules rk, k =..., m, describing the model g, is checked to analyze if it is verified by the considered sample x*. let d be the number of conditions in the <premise> part of the rule rk that are not verified by the pattern x*. then, for every output class yl we can determine the minimum value dl=minr∈gld and the subset hl of rules in the group gl characterized by that minimum: hl={r∈gl|d=dl} then, we choose as output value for the pattern x* the class l scoring the lowest dl and, in case of ties, the minimum value of the quantity wl defined as wl= ∏r∈hl1-∑verifiedcinrr where the summation is performed on all the conditions c in the <premise> part of the rule r that are verified by the sample x*. llm performance assessment in order to obtain an unbiased estimate of the llm performance, data were analysed according to a leave-one-out cross-validation. rules were generated allowing a maximum error rate of 5% in the training set. accuracy of llm classification applied to the test set was compared to that of selected competing standard methods of supervised analysis, namely: k-nearest neighbour classifier, artificial neural network, and decision tree. in particular, dt, similarly to llm, is able to generate threshold-based intelligible rules. for this reason, we performed a comparison between the rules generated by llm and those obtained by dt. llm is implemented as part of the rulex software suite, developed and distributed by rulex inc. competing methods a brief description of competing methods is here given; details regarding their use and implementation can be found in standard books of data mining. k-nearest-neighbor although knn is one of the simplest technique for classifying previously unseen patterns x taking into account the information contained in a given training set s, it can achieve a good accuracy even in complex situations. its approach is very straightforward: when an input vector x has to be classified, knn searches for the k nearest points x x..., xk in s according to a given definition of distance. then, it assigns to x the most common class present in x x..., xk. the value of k is usually chosen to avoid ties. although the adopted definition of distance can affect the accuracy of the knn classifier, very often the standard euclidean distance is employed, after having normalized the components of x to avoid undesirable effects due to unbalanced domain intervals in different input variables. in the reported trials the choice k = was performed, which corresponded to assign to any previously unseen point x the class of its nearest neighbor in the training set s. artificial neural network building a classifier starting from a given training set s corresponds to determining a subset of the input domain for each output class or, equivalently, to constructing proper separating surfaces that delimit these subsets. in general, each separating surface can be nonlinear and even complex, depending on the specific classification problem at hand. a convenient way to manage this complexity is to build the separating surface through the composition of simpler functions. this approach is followed by ann, a connectionist model formed by the interconnection of simple units, called neurons, arranged in layers. each neuron performs a weighted sum of its inputs and applies a proper activation function to obtain the output value that will be propagated to the following layer. the first layer of neurons is fed by the components of the input vector x, whereas the last layer produces the output class to be assigned to x. suitable optimization techniques are used to retrieve the weights for each neuron, which form the set of parameters for the ann. by properly setting these weights we can obtain separating surfaces arbitrarily complex, provided that a sufficient number of neurons is included in the ann. the choice of this quantity, together with the selection of the number of layers, must be performed at the beginning of the training process and affect the generalisation ability of the resulting model. decision trees an intelligible classifier can be obtained by generating a tree graph where each node is associated with a condition on a component of the input vector x and each leaf corresponds to an assignment for the output class to be assigned to x. a model of this kind is called decision tree. it is straightforward to retrieve an intelligible rule for the classification problem at hand by navigating the decision tree from a leaf to the root and by using as antecedent for the rule the logical product of the conditions associated with the nodes encountered during the navigation. rules obtained in these way are disjoint from each other. although different learning algorithms have been proposed for building a dt, a basic divide-and-conquer strategy is followed by all of them. at each iteration a new node is added to the dt by considering a subset of s and by choosing the condition that subdivides it in the best way, according to a specific measure of goodness. with this approach the size of the subset pertaining to added nodes decreases during the construction of the tree, which halts when a specific stopping criterion is reached. proper pruning techniques are adopted to simplify the final dt with the aim of reducing its complexity and increasing its generalisation ability.  comparison between the performance of llm and that of the other supervised methods mpm = malignant pleural mesothelioma; mtx = metastasis; bd = benign diseases; llm = logic learning machine; dt = decision tree; ann = artificial neural network; knn = k-nearest neighbour classifier. classification rules obtained by llm llm and dt analyses were repeated on the entire dataset in order to obtain stable rules for patients classification. llm classifier included rules, but of them had a very low covering. diag. = diagnosis; mpm = malignant pleural mesothelioma; mtx = metastasis; bd = benign diseases; ce = cytological examination diag. = diagnosis; w% = e - e; r% = relevance%; cov. % = covering percent. w and r are defined according to equation. classification rules obtained by dt dt classification was based on rules. discussion llm is an innovative method that can provide useful classification rules by exploiting the complex multivariable correlation between the different analysed features. llm has been recently successfully applied to a variety of datasets in biomedical settings, but so far it has not been used for differential diagnosis of cancer patients based on tumour markers combination. in the last decades many other methods of supervised data mining have been successfully applied to classification tasks in different biomedical fields, including oncology. in particular, ann and knn have shown a good accuracy in many instances. however, they represent "black-box" methods that cannot provide useful insights about biological and clinical aspects of the disease under study. for this reason, intelligible "and- type" and "or-type" rules are in general preferred, but methods for multi-class classification are scarce. among them, dt is probably the most widely used tool for its simplicity and easily implementation, but in general it tends to show a low accuracy when compared to other supervised methods. however, in the present investigation, dt provided a quite good both total and class-specific accuracy that was higher than that obtained from the two black-box algorithms. however, llm outperformed all competing methods including dt. in particular, dt performance was slightly lower among bd patients, slightly higher for mpm class only and clearly lower for both mtx and the pool of malignancies. classification rules obtained from both llm and dt were in good agreement with a priori knowledge about the considered tumour markers. in particular, high values of cea were associated with mtx class with a covering of about 50% for both methods. such a proportion roughly corresponds to the percentage of patients with pleural metastasis from lung adenocarcinoma inside the analysed cohort. this finding is in agreement with the characteristics of cea marker, which is largely expressed among cancers from epithelial origin. moreover, high smrp concentrations were associated with mpm classification. this finding confirms previous observations from other independent cohorts reporting that smrp concentration in pleural fluid is specific in distinguishing mesothelioma from both benign and all other malignant effusions. finally, low values of cyfra 21- were associated with bd classification with a high coverage in both methods, in agreement with previous studies that have associated high values of this marker to a large variety of neoplastic diseases. rules extracted by llm and dt only partly overlapped, thus reflecting a sort of balance between the capability of the two methods of identifying useful information for classification purposes, and, on the other hand, the two very different algorithms for rule generation. for instance, the rule with the highest covering for mpm classification was almost identical in the two methods, both including high values of smrp and cyfra 21- and low values of cea, with very similar cut-offs. conversely, the best rule for mtx classification was rather different, including only high values of cea for dt, and a combination of high values of cea and high cyfra 21- concentrations for llm. finally, the best classification for bd was obtained from both methods by low values of both cyfra 21- and cea, but at different cut-offs. furthermore, llm rule also included negative ce. on the whole, our results indicate that both llm and dt are able to extract meaningful information from tumour markers and to combine them in simple rules for classification tasks. dt also provides a simple plot that allows a very easy interpretation of the rules generated, whereas llm rules, being partly overlapping, provide a rather more complicated picture. however, in our analysis, in agreement with results from previous investigations the presence of overlapping rules allowed llm to outperform dt classification. furthermore, a non-ambiguous classification can always be obtained by using coverage and error rate parameters and by adopting a proper measure of relevance that allows to select the most probable class for the pattern at hand. moreover, overlapping rules can be weighted in order to improve classification accuracy in the presence of severely unbalanced sample size, thus conferring a high flexibility to llm based classification. results of our investigation should be evaluated at the light of some unavoidable limits, in particular the rather small sample size. mesothelioma is a rare cancer and, at least at our knowledge, larger datasets including all the three tms considered in the present study are not available. the possibility that the comparison between the selected classification methods could have been influenced by the size of classes under investigation cannot be completely ruled out. however, in some previous analyses, carried out in different biomedical fields, llm was demonstrated to outperform other methods of machine learning when applied to large datasets. in particular, llm accuracy was higher than that of two competing methods in a feature selection task using data from three real and three simulated databases from microarray experiments, each based on many thousands of gene expression profiles. furthermore, in a recent analysis of biomedical datasets of the statlog benchmark, which included a large database of cases of diabetes and healthy controls, llm systematically outperformed four competing methods of learning machine. another limit of our investigation is the low accuracy for mtx classification, even if this latter was better classified by llm than by the considered competing methods. finally, the set of rules generated by llm does not cover all the possible combinations between tumour markers and ce results, then making potentially difficult the classification of some additional patients. such a limitation can be overcome by llm when the features associated to the subject at least fulfil a subset of one or more conditions inside a composite rule, by combining accuracy measures using equations and.  results from the present study indicates that llm is a flexible and powerful tool for the differential classification of malignant mesothelioma patients. dt performance was poorer, but, quite surprisingly, clearly better than the two selected "black-box" competing methods. further studies on larger cohorts are needed in order to obtain stable and reproducible rules for mpm classification. moreover, additional tumour markers should be tested to improve the classification of non-mesothelioma cancers with pleural metastasis. list of abbreviations llm: logic learning machine. dt: decision tree. knn: k-nearest neighbour classifier. ann: artificial neural network. mpm: malignant pleural mesothelioma. bd: benign disease. mtx: metastasis from non-mesothelioma cancers. cea: carcino-embryonic antigen. cyfra 21-1: soluble fragment of cytokeratin-19 smrp: soluble mesothelin-related peptide. ce: cytological examination. tm: tumour marker. competing interests the authors declare that they have no competing interests. authors' contributions sp, rf and mm conceived the study and wrote the paper. moreover, sp performed most analyses, while mm conceived and implemented the llm method. pm, rl, gpi and mm provided data of tumour marker concentrations and contributed in the interpretation of biological meaning of results and in writing the discussion section. ef, cm and em implemented most routines for supervised analysis. ef also supervised data analyses. 
it is essentially an efficient implementation of the switching neural network model and reaches excellent classification accuracy while keeping low the computational demand. classification accuracy can be improved by combining information from different markers using standard data mining techniques, like decision tree, artificial neural network, and k-nearest neighbour classifier. concentration of three tumour markers was measured in the pleural fluid of each patient and a cytological examination was also carried out. the performance of llm and that of three competing methods was assessed by leave-one-out cross-validation. tumour markers are standard tools for the differential diagnosis of cancer. finally, llm provided classification rules in a very good agreement with a priori knowledge about the biological role of the considered tumour markers. however, the occurrence of nonspecific symptoms and different malignancies involving the same cancer site may lead to a high proportion of misclassifications. logic learning machine is an innovative method of supervised data analysis capable of building classifiers described by a set of intelligible rules including simple conditions in their antecedent part. llm is a new flexible tool potentially useful for the differential diagnosis of pleural mesothelioma. llm was applied to data from a consecutive cohort of patients admitted for diagnosis to two pulmonary departments in northern italy from to patients included malignant pleural mesotheliomas, pleural metastases from other tumours and benign diseases associated with pleurisies.
 tumour markers are standard tools for the differential diagnosis of cancer. however, the occurrence of nonspecific symptoms and different malignancies involving the same cancer site may lead to a high proportion of misclassifications. classification accuracy can be improved by combining information from different markers using standard data mining techniques, like decision tree, artificial neural network, and k-nearest neighbour classifier. unfortunately, each method suffers from some unavoidable limitations. dt, in general, tends to show a low classification performance, whereas ann and knn produce a "black-box" classification that does not provide biological information useful for clinical purposes. methods logic learning machine is an innovative method of supervised data analysis capable of building classifiers described by a set of intelligible rules including simple conditions in their antecedent part. it is essentially an efficient implementation of the switching neural network model and reaches excellent classification accuracy while keeping low the computational demand. llm was applied to data from a consecutive cohort of patients admitted for diagnosis to two pulmonary departments in northern italy from to patients included malignant pleural mesotheliomas, pleural metastases from other tumours and benign diseases associated with pleurisies. concentration of three tumour markers was measured in the pleural fluid of each patient and a cytological examination was also carried out. the performance of llm and that of three competing methods was assessed by leave-one-out cross-validation.  llm outperformed all other considered methods. global accuracy was % for llm, % for dt, % for knn, and % for ann, respectively. in more details, llm correctly classified 79% of mpm, 66% of mtx and 89% of bd. the corresponding figures for dt were: mpm = 83%, mtx = 55% and bd = 84%; for knn: mpm = 58%, mtx = 45%, bd = 62%; for ann: mpm = 71%, mtx = 47%, bd = 76%. finally, llm provided classification rules in a very good agreement with a priori knowledge about the biological role of the considered tumour markers.  llm is a new flexible tool potentially useful for the differential diagnosis of pleural mesothelioma. 26- february eleventh annual meeting of the bioinformatics italian society meeting rome, italy 