BACKGROUND
for decades, machine learning has been used as a tool in bioinformatics for predictive purposes.
a number of concepts have been implemented in order to estimate the predictive power of the individual methods.
the commonly used performance measures have been described in lundegaard et al.
<cit>  predictive power is generally estimated from a number of examples that have been excluded from the training process and an overall estimate of the accuracy of the method is calculated.
this, however, will not provide information regarding the reliability of each of the individual predictions.
for discrete qualitative predictions, the reliability is conventionally estimated as the difference between output scores of selected classes  <cit> .
however, many biological problems are quantitative in nature and are therefore more appropriately characterized by a real value than a discrete class.
real value predictions often provide a single output value and the estimation of the accuracy of a given prediction is more complicated than for predictions of discrete classes.
prediction of the solvent accessible surface area  of amino acid residues within a native folded protein is an example of a real value prediction problem, where the estimation of reliability scores is nontrivial.
the asa for experimentally solved structures is given in Å <dig> and the area is calculated by rolling a sphere the size of a water molecule over the protein surface  <cit> .
for comparative and predictive purposes, the asa is often transformed to a relative surface area , which is calculated as the asa of a given amino acid residue in the polypeptide chain, relative to the maximal possible exposure of that residue in the center of a tri-peptide flanked with either glycine  <cit>  or alanine  <cit> .
knowledge of the degree of surface exposure of an amino acid is valuable and it has been used to enhance the understanding of a variety of biological problems including protein-protein interactions  <cit> , structural epitopes  <cit> , active sites  <cit> , and prediction of disease-related single nucleotide polymorphisms  <cit> .
several methods for predicting surface accessibility from the primary protein sequence have been developed often inspired by the related field of protein secondary structure prediction as exemplified with  <cit>  implemented in  <cit> .
generally, the best methods involve the use of advanced machine learning algorithms such as artificial neural networks  or support vector machines  combined with evolutionary information  <cit> .
the surface accessibility has traditionally been predicted in two classes as either buried or exposed using various more or less arbitrary cut-offs.
recently, real value rsa predictors have been developed thus removing the need to define specific cut-offs  <cit> .
this change in focus from classifying towards quantitative systems has made it difficult to assess the reliability of a prediction.
previous studies have shown that prediction of the rsa is significantly more accurate for buried compared to exposed amino acids  <cit> .
however, the most biologically interesting residues are often exposed, as these are able to interact with the environment.
for this reason, it is important to have a good estimate of the reliability, especially for the more exposed amino acid residues.
the current best method available for real value surface exposure prediction is real-spine  <cit> .
this method exists in a web accessible form, which in addition to the predicted surface accessibility, also provides a score for each prediction that is a measure of the consistency between two predictors .
rs =  <dig> - |a - b| where a and b are the results from two predictors on solvent accessibility  <cit> .
as described this score is solely a consistency score and it has not previously been described to what degree such consistency measures provide information of the reliability of the individual predictions beyond the fact that the most exposed residues are predicted most unreliably.
here, we have developed a generic method that assigns a reliability score to each surface accessibility prediction as an inherent part of the training process.
the method is evaluated on a common set of sequences and compared to other state-of-the art prediction methods.
in particular, we investigate to what extent our method for residue-specific reliability prediction is able to discriminate between good and bad predictions in the entire range from completely buried to fully exposed amino acids.
RESULTS
a schematic overview of the netsurfp method is shown in figure  <dig>  the method consists of two neural network ensembles.
the primary networks are trained on sequence profiles and predicted secondary structure and have two outputs corresponding to buried or exposed, respectively.
the higher output defines the predicted category.
the secondary networks use these outputs as input together with sequence profiles and have been trained to predict the relative surface exposure of the individual amino acid residues.
the proposed reliability prediction method is applied to the secondary networks only.
primary networks
classification artificial neural networks  were trained to predict whether an amino acid was buried or exposed i.e., below or above 25% of asamax of the given amino acid.
input to the anns was sequence profiles and predicted secondary structures.
the prediction performance of the secondary structure prediction in terms of the straight q <dig> measure on the cb <dig> dataset was 81%.
secondary structure predictors were trained to predict h or e classes , which differs from the casp classification scheme used by many secondary structure prediction methods .
using 10-fold cross validation each spanning a series of different network architectures, an ensemble were constructed of the  <dig> best performing network architectures, determined by the cross validation leave-out test sets .
a test performance of  <dig> % accuracy and a matthews correlation coefficient  of  <dig>  were obtained.
this ann ensemble was also evaluated using the evaluation set cb <dig>  the performance values were  <dig> % correctly classified residues and a mcc of  <dig> .
these values are compared with the performance obtained by  <cit>  as shown in table  <dig>
evaluation of the best performing ann ensemble using the evaluation set cb <dig>  the columns are the overall %-correct prediction of buried and exposed amino acids and matthew's correlation coefficient .
dor and zhou gives the performance value published by  <cit> .
secondary networks
the output classification values from the primary networks were used together with sequence profiles in the form of position-specific scoring matrices  to train the secondary neural networks as also implemented by  <cit> .
a significant improvement was obtained compared to bare pssm input only with respect to linear as well as two-state correlations .
several neural network architectures were trained using 10-fold cross-validation.
the best cross-validation leave out test set performance was obtained by using a window size of  <dig> residues and a number of hidden neurons in the range 25– <dig>  the real-spine method  <cit>  has not previously been evaluated on the cb <dig> set.
we therefore submitted the sequences in the cb <dig> set to the real-spine  <dig>  web-server.
two sequences were not accepted by the server leaving us with a set of  <dig> sequences  used when comparing the performance of netsurfp and several other methods  <cit> .
the realspine and netsurfp methods perform equally well as shown in table  <dig>
performances are shown for  <dig> different approaches to predict absolute and relative  surface accessibility.
methods included in the benchmark are ahmad:  <cit> , yuan:  <cit> , nguyen:  <cit> , real-spine:  <cit> , netsurfp: this work.
train gives the training performance, and cb513/cb <dig> gives the evaluation performance on the cb <dig> data set.
train performance of the real-spine method and evaluation performances for the ahmad, yuan, and nguyen method are taken from the corresponding publications.
ann = artificial neural networks, svr = support vector regression.
pearson's correlation coefficients  are shown for all methods based on the absolute surface exposure of an amino acid.
also, pcc values are given for relative surface exposure for the two methods netsurfp and real-spine.
prediction and analysis of reliability scores
neural networks were trained as described in section 'secondary networks'.
real value predictions usually gives one output value between 0– <dig> per residue, however, our described method generates two output values for each prediction; the predicted surface accessibility and a reliability of this prediction for each amino acid residue.
this was implemented using a modified back-propagation procedure as described in the method section.
we evaluated the performance of this method on the cb <dig> data set and compared the results to those obtained with the method by dor and zhou  <cit> .
unless otherwise stated, the performance values were calculated from the rsa.
the overall predictive performance of the neural network was  <dig>  in terms of the mean error, e, and  <dig>  in terms of the pearson's correlation coefficient , which is similar to the values obtained earlier using the conventional networks .
from the network reliability score, we calculated a reliability value as a z-score as described in methods.
figure  <dig>  shows the variation in the mean error as a function of the z-score reliability from netsurfp.
from this figure, it is apparent that data points with high z-scores have lower predicted error compared to data points with low z-scores.
we found that the group of data points with positive z-scores, corresponding to 51% of all data points, achieved a pcc of  <dig> , whereas the data points with negative z-scores achieved a pcc of  <dig> .
this difference is highly significant .
the real-spine method provides a residue-specific consistency measure associated with each prediction.
the relationship between this value and the mean error is shown in the right panel of figure  <dig>  comparing these two plots suggests that both methods are able to identify the most reliable predictions.
it has previously been reported that amino acid residues, which are predicted to be highly buried tend to have lower predicted error compared to those predicted as exposed  <cit> .
to investigate how this might bias the reliabilities we examined the mean predicted error as a function of the predicted exposure when splitting the data in two groups with high  and low  reliability, respectively .
the plot visualizes how the predictions with a corresponding high z-score have a lower mean error compared to those with a low z-score.
this is valid for all ranges of predicted exposure.
this, on the other hand, is not the case for the consistency scores.
comparing the "high" and "low" reliability groups we see a difference only for residues that were predicted to be buried .
the same trend is observed when using a cut-off of top 25% and 75% highest predictions for both real-spine and netsurfp .
likewise, we tested to what degree the two reliability measures are capable of identifying reliable predictions independent of the degree of exposure.
the distribution of predicted rsa values for the 25%, 50%, 75% and 80% residues with highest consistency scores was shown for the real-spine  and highest z-score for netsurfp , respectively.
these figures reveal that the real-spine method predominantly assigns high consistency scores to buried residues, and when filtering out low consistency predictions mostly exposed residues are removed.
this can be seen on the insert for real-spine  where there is a bias against low rsa.
in contrast to this, high netsurfp z-score values are found for residues in all exposure ranges.
the curve in the insert for netsurfp , is close to horizontal meaning predictions are equally distributed over the different levels of exposure independent of z-score reliability threshold.
the predictive performance of the 80% residues with highest reliability of the two methods is  <dig>  and  <dig>  in terms of the pcc for the consistency and the derived z-score methods, respectively.
this difference in predictive performance is highly significant .
the above results could depend on the chosen cut-off for the fraction of most reliable predictions  that were included in the test.
to investigate this bias we took an increasing number of the z-score/consistency ranked predictions and calculated the average rsa of the selected sets both regarding predicted and measured rsa.
in table  <dig> it is shown that the predictions from the real-spine with the highest consistency have a strong bias towards buried residues.
using the netsurfp derived z-score, no such bias was observed and the ratio between buried/exposed residues was maintained for all levels of reliability, i.e.
the mean predicted relative accessibility  equals the mean measured  in each subset.
in addition, the pcc of the z-score filtered netsurfp predictions is better within nearly all of the most reliable subsets than that of the consistency filtered real-spine predictions, despite the fact that the two methods have close to identical overall performances.
furthermore, the subsets of reliable netsurfp predictions identified by the z-score method maintain a constant average of both the predicted surface exposure and the surface exposure calculated from experimentally solved structures independent of the degree of reliability.
however, using the consistency filter on real-spine predictions we saw that the average of the predicted or calculated surface exposure decreased  as the reliability increases.
the final implementation of the netsurfp method as a web-server was done by also including the sequences  that were previously only used as an evaluation set.
the secondary structure predictor is implemented as part of the netsurfp web-server.
the web-server is available at
%top and n give the percentage and number of residues selected.
rsa and asa give the pearson's correlation between predicted and target for relative and absolute surface areas, respectively.
p-rsa, and m-rsa give the mean predicted and mean measured rsa values, respectively, on the selected subset of residues.
discussion
the power of a prediction method is commonly evaluated as an overall estimate of the accuracy of the method in large-scale benchmark experiments.
such evaluation, however, provides no knowledge of the reliability of each of the individual predictions.
for discrete, qualitative predictions the reliability is conventionally estimated as the difference between output scores of selected classes.
for real value prediction this approach is unfeasible.
here, we have described a new reliability score method, useful for real value predictions.
we have designed and implemented the method in a way that assigns reliability scores for each single real value prediction.
as an example, the method has been implemented as part of a web-server to predict the relative surface accessible area of amino acids within the three dimensional structure of a protein.
by nature, the reliability method is different from other procedures where reliabilities most commonly are obtained by post-processing the output  <cit> .
this method was trained to assign a reliability output to each surface accessibility prediction as an inherent part of the network architecture.
this output was then recomputed to a z-score.
in tests to investigate the validity of the calculated z-score we found that the score could indeed successfully be used to filter out more reliable predictions resulting in a significantly better correlation between predicted and measured values.
the accessible surface area has been found more difficult to predict for exposed than buried amino acids and these findings are still valid  <cit> .
however, we see that netsurfp z-scores enable the identification of the most reliable/unreliable predictions for both buried and exposed amino acids.
this allows for identification of subsets of highly reliable predictions covering all ranges of surface exposure.
this is in contrast to the consistency score, the only other surface accessibility prediction associated reliability method  <cit> , where high reliability scores are predominantly associated with buried amino acids.
the prediction accuracy is compared to real-spine  <dig>   <cit>  as real-spine  <dig>  is the server that produces the consistency measures.
furthermore the newly published real-spine  <dig>   <cit>  was not available at the time of the evaluation.
CONCLUSIONS
in the present context, the developed reliability information is especially valuable when using the surface exposed predictions to estimate other protein structure related features such as fold, b cell epitopes, phosphorylation sites, and active sites.
however, the approach is generic and is potentially useful in other types of real value predictions where anns have been shown to produce good results.

 for decades, machine learning has been used as a tool in bioinformatics for predictive purposes. a number of concepts have been implemented in order to estimate the predictive power of the individual methods. the commonly used performance measures have been described in lundegaard et al. predictive power is generally estimated from a number of examples that have been excluded from the training process and an overall estimate of the accuracy of the method is calculated. this, however, will not provide information regarding the reliability of each of the individual predictions. for discrete qualitative predictions, the reliability is conventionally estimated as the difference between output scores of selected classes. however, many biological problems are quantitative in nature and are therefore more appropriately characterized by a real value than a discrete class. real value predictions often provide a single output value and the estimation of the accuracy of a given prediction is more complicated than for predictions of discrete classes. prediction of the solvent accessible surface area of amino acid residues within a native folded protein is an example of a real value prediction problem, where the estimation of reliability scores is nontrivial. the asa for experimentally solved structures is given in Å and the area is calculated by rolling a sphere the size of a water molecule over the protein surface. for comparative and predictive purposes, the asa is often transformed to a relative surface area, which is calculated as the asa of a given amino acid residue in the polypeptide chain, relative to the maximal possible exposure of that residue in the center of a tri-peptide flanked with either glycine or alanine. knowledge of the degree of surface exposure of an amino acid is valuable and it has been used to enhance the understanding of a variety of biological problems including protein-protein interactions, structural epitopes, active sites, and prediction of disease-related single nucleotide polymorphisms. several methods for predicting surface accessibility from the primary protein sequence have been developed often inspired by the related field of protein secondary structure prediction as exemplified with implemented in. generally, the best methods involve the use of advanced machine learning algorithms such as artificial neural networks or support vector machines combined with evolutionary information. the surface accessibility has traditionally been predicted in two classes as either buried or exposed using various more or less arbitrary cut-offs. recently, real value rsa predictors have been developed thus removing the need to define specific cut-offs. this change in focus from classifying towards quantitative systems has made it difficult to assess the reliability of a prediction. previous studies have shown that prediction of the rsa is significantly more accurate for buried compared to exposed amino acids. however, the most biologically interesting residues are often exposed, as these are able to interact with the environment. for this reason, it is important to have a good estimate of the reliability, especially for the more exposed amino acid residues. the current best method available for real value surface exposure prediction is real-spine. this method exists in a web accessible form, which in addition to the predicted surface accessibility, also provides a score for each prediction that is a measure of the consistency between two predictors. rs = - |a - b| where a and b are the results from two predictors on solvent accessibility. as described this score is solely a consistency score and it has not previously been described to what degree such consistency measures provide information of the reliability of the individual predictions beyond the fact that the most exposed residues are predicted most unreliably. here, we have developed a generic method that assigns a reliability score to each surface accessibility prediction as an inherent part of the training process. the method is evaluated on a common set of sequences and compared to other state-of-the art prediction methods. in particular, we investigate to what extent our method for residue-specific reliability prediction is able to discriminate between good and bad predictions in the entire range from completely buried to fully exposed amino acids.  a schematic overview of the netsurfp method is shown in figure the method consists of two neural network ensembles. the primary networks are trained on sequence profiles and predicted secondary structure and have two outputs corresponding to buried or exposed, respectively. the higher output defines the predicted category. the secondary networks use these outputs as input together with sequence profiles and have been trained to predict the relative surface exposure of the individual amino acid residues. the proposed reliability prediction method is applied to the secondary networks only. primary networks classification artificial neural networks were trained to predict whether an amino acid was buried or exposed i.e., below or above 25% of asamax of the given amino acid. input to the anns was sequence profiles and predicted secondary structures. the prediction performance of the secondary structure prediction in terms of the straight q measure on the cb dataset was 81%. secondary structure predictors were trained to predict h or e classes, which differs from the casp classification scheme used by many secondary structure prediction methods. using 10-fold cross validation each spanning a series of different network architectures, an ensemble were constructed of the best performing network architectures, determined by the cross validation leave-out test sets. a test performance of % accuracy and a matthews correlation coefficient of were obtained. this ann ensemble was also evaluated using the evaluation set cb the performance values were % correctly classified residues and a mcc of. these values are compared with the performance obtained by as shown in table evaluation of the best performing ann ensemble using the evaluation set cb the columns are the overall %-correct prediction of buried and exposed amino acids and matthew's correlation coefficient. dor and zhou gives the performance value published by. secondary networks the output classification values from the primary networks were used together with sequence profiles in the form of position-specific scoring matrices to train the secondary neural networks as also implemented by. a significant improvement was obtained compared to bare pssm input only with respect to linear as well as two-state correlations. several neural network architectures were trained using 10-fold cross-validation. the best cross-validation leave out test set performance was obtained by using a window size of residues and a number of hidden neurons in the range 25– the real-spine method has not previously been evaluated on the cb set. we therefore submitted the sequences in the cb set to the real-spine web-server. two sequences were not accepted by the server leaving us with a set of sequences used when comparing the performance of netsurfp and several other methods. the realspine and netsurfp methods perform equally well as shown in table performances are shown for different approaches to predict absolute and relative surface accessibility. methods included in the benchmark are ahmad:, yuan:, nguyen:, real-spine:, netsurfp: this work. train gives the training performance, and cb513/cb gives the evaluation performance on the cb data set. train performance of the real-spine method and evaluation performances for the ahmad, yuan, and nguyen method are taken from the corresponding publications. ann = artificial neural networks, svr = support vector regression. pearson's correlation coefficients are shown for all methods based on the absolute surface exposure of an amino acid. also, pcc values are given for relative surface exposure for the two methods netsurfp and real-spine. prediction and analysis of reliability scores neural networks were trained as described in section'secondary networks'. real value predictions usually gives one output value between 0– per residue, however, our described method generates two output values for each prediction; the predicted surface accessibility and a reliability of this prediction for each amino acid residue. this was implemented using a modified back-propagation procedure as described in the method section. we evaluated the performance of this method on the cb data set and compared the results to those obtained with the method by dor and zhou. unless otherwise stated, the performance values were calculated from the rsa. the overall predictive performance of the neural network was in terms of the mean error, e, and in terms of the pearson's correlation coefficient, which is similar to the values obtained earlier using the conventional networks. from the network reliability score, we calculated a reliability value as a z-score as described in methods. figure shows the variation in the mean error as a function of the z-score reliability from netsurfp. from this figure, it is apparent that data points with high z-scores have lower predicted error compared to data points with low z-scores. we found that the group of data points with positive z-scores, corresponding to 51% of all data points, achieved a pcc of, whereas the data points with negative z-scores achieved a pcc of. this difference is highly significant. the real-spine method provides a residue-specific consistency measure associated with each prediction. the relationship between this value and the mean error is shown in the right panel of figure comparing these two plots suggests that both methods are able to identify the most reliable predictions. it has previously been reported that amino acid residues, which are predicted to be highly buried tend to have lower predicted error compared to those predicted as exposed. to investigate how this might bias the reliabilities we examined the mean predicted error as a function of the predicted exposure when splitting the data in two groups with high and low reliability, respectively. the plot visualizes how the predictions with a corresponding high z-score have a lower mean error compared to those with a low z-score. this is valid for all ranges of predicted exposure. this, on the other hand, is not the case for the consistency scores. comparing the "high" and "low" reliability groups we see a difference only for residues that were predicted to be buried. the same trend is observed when using a cut-off of top 25% and 75% highest predictions for both real-spine and netsurfp. likewise, we tested to what degree the two reliability measures are capable of identifying reliable predictions independent of the degree of exposure. the distribution of predicted rsa values for the 25%, 50%, 75% and 80% residues with highest consistency scores was shown for the real-spine and highest z-score for netsurfp, respectively. these figures reveal that the real-spine method predominantly assigns high consistency scores to buried residues, and when filtering out low consistency predictions mostly exposed residues are removed. this can be seen on the insert for real-spine where there is a bias against low rsa. in contrast to this, high netsurfp z-score values are found for residues in all exposure ranges. the curve in the insert for netsurfp, is close to horizontal meaning predictions are equally distributed over the different levels of exposure independent of z-score reliability threshold. the predictive performance of the 80% residues with highest reliability of the two methods is and in terms of the pcc for the consistency and the derived z-score methods, respectively. this difference in predictive performance is highly significant. the above results could depend on the chosen cut-off for the fraction of most reliable predictions that were included in the test. to investigate this bias we took an increasing number of the z-score/consistency ranked predictions and calculated the average rsa of the selected sets both regarding predicted and measured rsa. in table it is shown that the predictions from the real-spine with the highest consistency have a strong bias towards buried residues. using the netsurfp derived z-score, no such bias was observed and the ratio between buried/exposed residues was maintained for all levels of reliability, i.e. the mean predicted relative accessibility equals the mean measured in each subset. in addition, the pcc of the z-score filtered netsurfp predictions is better within nearly all of the most reliable subsets than that of the consistency filtered real-spine predictions, despite the fact that the two methods have close to identical overall performances. furthermore, the subsets of reliable netsurfp predictions identified by the z-score method maintain a constant average of both the predicted surface exposure and the surface exposure calculated from experimentally solved structures independent of the degree of reliability. however, using the consistency filter on real-spine predictions we saw that the average of the predicted or calculated surface exposure decreased as the reliability increases. the final implementation of the netsurfp method as a web-server was done by also including the sequences that were previously only used as an evaluation set. the secondary structure predictor is implemented as part of the netsurfp web-server. the web-server is available at %top and n give the percentage and number of residues selected. rsa and asa give the pearson's correlation between predicted and target for relative and absolute surface areas, respectively. p-rsa, and m-rsa give the mean predicted and mean measured rsa values, respectively, on the selected subset of residues. discussion the power of a prediction method is commonly evaluated as an overall estimate of the accuracy of the method in large-scale benchmark experiments. such evaluation, however, provides no knowledge of the reliability of each of the individual predictions. for discrete, qualitative predictions the reliability is conventionally estimated as the difference between output scores of selected classes. for real value prediction this approach is unfeasible. here, we have described a new reliability score method, useful for real value predictions. we have designed and implemented the method in a way that assigns reliability scores for each single real value prediction. as an example, the method has been implemented as part of a web-server to predict the relative surface accessible area of amino acids within the three dimensional structure of a protein. by nature, the reliability method is different from other procedures where reliabilities most commonly are obtained by post-processing the output. this method was trained to assign a reliability output to each surface accessibility prediction as an inherent part of the network architecture. this output was then recomputed to a z-score. in tests to investigate the validity of the calculated z-score we found that the score could indeed successfully be used to filter out more reliable predictions resulting in a significantly better correlation between predicted and measured values. the accessible surface area has been found more difficult to predict for exposed than buried amino acids and these findings are still valid. however, we see that netsurfp z-scores enable the identification of the most reliable/unreliable predictions for both buried and exposed amino acids. this allows for identification of subsets of highly reliable predictions covering all ranges of surface exposure. this is in contrast to the consistency score, the only other surface accessibility prediction associated reliability method, where high reliability scores are predominantly associated with buried amino acids. the prediction accuracy is compared to real-spine as real-spine is the server that produces the consistency measures. furthermore the newly published real-spine was not available at the time of the evaluation.  in the present context, the developed reliability information is especially valuable when using the surface exposed predictions to estimate other protein structure related features such as fold, b cell epitopes, phosphorylation sites, and active sites. however, the approach is generic and is potentially useful in other types of real value predictions where anns have been shown to produce good results. 
an ensemble of artificial neural networks has been trained on a set of experimentally solved protein structures to predict the relative exposure of the amino acids. however, our implementation of reliability scores in the form of a z-score is shown to be the more informative measure for discriminating good predictions from bad ones in the entire range from completely buried to fully exposed amino acids. estimation of the reliability of specific real value predictions is nontrivial and the efficacy of this is often questionable. such an approach is not feasible for methods that predict a biological feature as a single real value rather than a classification. it is important to know if you can trust a given prediction and therefore the best methods associate a prediction with a reliability score or index. the method assigns a reliability score to each surface accessibility prediction as an inherent part of the training process. both methods associate a reliability score with the individual predictions. for discrete qualitative predictions, the reliability is conventionally estimated as the difference between output scores of selected classes. as a solution to this challenge, we have implemented a method that predicts the relative surface accessibility of an amino acid and simultaneously predicts the reliability for each prediction, in the form of a z-score.
 estimation of the reliability of specific real value predictions is nontrivial and the efficacy of this is often questionable. it is important to know if you can trust a given prediction and therefore the best methods associate a prediction with a reliability score or index. for discrete qualitative predictions, the reliability is conventionally estimated as the difference between output scores of selected classes. such an approach is not feasible for methods that predict a biological feature as a single real value rather than a classification. as a solution to this challenge, we have implemented a method that predicts the relative surface accessibility of an amino acid and simultaneously predicts the reliability for each prediction, in the form of a z-score.  an ensemble of artificial neural networks has been trained on a set of experimentally solved protein structures to predict the relative exposure of the amino acids. the method assigns a reliability score to each surface accessibility prediction as an inherent part of the training process. this is in contrast to the most commonly used procedures where reliabilities are obtained by post-processing the output.  the performance of the neural networks was evaluated on a commonly used set of sequences known as the cb set. an overall pearson's correlation coefficient of was obtained, which is comparable to the performance of the currently best public available method, real-spine. both methods associate a reliability score with the individual predictions. however, our implementation of reliability scores in the form of a z-score is shown to be the more informative measure for discriminating good predictions from bad ones in the entire range from completely buried to fully exposed amino acids. this is evident when comparing the pearson's correlation coefficient for the upper 20% of predictions sorted according to reliability. for this subset, values of and are obtained using our and the compared method, respectively. this tendency is true for any selected subset. 