BACKGROUND
a b-cell epitope is defined as a part of a protein antigen recognized by either a particular antibody molecule or a particular b-cell receptor of the immune system  <cit> .
the main objective of b-cell epitope prediction is to facilitate the design of a short peptide or other molecule that can be synthesized and used instead of the antigen, which in the case of a pathogenic virus or bacteria, may be harmful to a researcher or experimental animal  <cit> .
a  b-cell epitope may be continuous, that is, a short contiguous stretch of amino acid residues, or discontinuous, comprising atoms from distant residues but close in three-dimensional space and on the surface of the protein.
synthetic peptides mimicking epitopes, as well as anti-peptide antibodies, have many applications in the diagnosis of various human diseases  <cit> .
also, the attempts have been made to develop peptide-based synthetic prophylactic vaccines for various infections, as well as therapeutic vaccines for chronic infections and noninfectious diseases, including autoimmune diseases, neurological disorders, allergies, and cancers  <cit> .
the immunoinformatics software and databases developed to facilitate vaccine design have previously been reviewed  <cit> .
during the last  <dig> years b-cell epitope prediction methods have focused primarily on continuous epitopes.
they were mostly sequence-dependent methods based upon various amino acid properties, such as hydrophilicity  <cit> , solvent accessibility  <cit> , secondary structure  <cit> , and others.
recently, several methods using machine learning approaches have been introduced that apply hidden markov models   <cit> , artificial neural networks   <cit> , support vector machine   <cit> , and other techniques  <cit> .
recent assessments of continuous epitope prediction methods demonstrate that "single-scale amino acid propensity profiles cannot be used to predict epitope location reliably"  <cit>  and that "the combination of scales and experimentation with several machine learning algorithms showed little improvement over single scale-based methods"  <cit> .
as crystallographic studies of antibody-protein complexes have shown, most b-cell epitopes are discontinuous.
in  <dig>  the first attempts at epitope prediction based on 3d protein structure was made for a few proteins for which continuous epitopes were known  <cit> .
subsequently, thornton and colleagues  <cit>  proposed a method to locate potential discontinuous epitopes based on a protrusion of protein regions from the protein's globular surface.
however, until the first x-ray structure of an antibody-protein complex was solved in  <dig>  <cit> , protein structural data were mostly used for prediction of continuous rather than discontinuous epitopes.
in cases where the three-dimensional structure of the protein or its homologue is known, a discontinuous epitope can be derived from functional assays by mapping onto the protein structure residues involved in antibody recognition  <cit> .
however, an epitope identified using an immunoassay may be an artefact of measuring cross-reactivity of antibodies due to the presence of denatured or degraded proteins  <cit> , or due to conformational changes in the protein caused by residue substitutions that may even lead to protein mis-folding  <cit> .
therefore, structural methods, particularly x-ray crystallography of antibody-antigen complexes, generally identify b-cell epitopes more reliably than functional assays  <cit> .
b-cell epitopes can be thought of in a structural and functional sense.
structural epitopes  are defined by a set of residues or atoms in the protein antigen contacting antibody residues or atoms  <cit> .
in contrast, a functional epitope consists of antigen residues that contribute significantly to antibody binding  <cit> .
functional epitopes are determined through functional assays  or calculated theoretically using known structures of antibody-protein complexes  <cit> .
thus, functional and structural epitopes are not necessary the same.
functional epitopes in proteins are usually smaller than structural epitopes; only three to five residues of the structural epitope contribute significantly to the antibody-antigen binding energy  <cit> .
this work focuses on structural epitopes inferred from known 3d structures of antibody-protein complexes available in the protein data bank   <cit> .
antibody-protein complexes can be categorized as intermediate transient non-obligate protein-protein complexes  <cit> .
non-obligate complexes, implying that individual components can be found on their own in vivo, are classified as either permanent or transient depending on their stability under particular physiological and environmental conditions  <cit> .
for example, many enzyme-inhibitor complexes are permanent non-obligate complexes.
transient non-obligate complexes range from weak , to intermediate , and to strong   <cit> .
most antibodies demonstrate intermediate affinity for their specific antigens  <cit> .
based on this classification, general methods for the prediction of intermediate transient non-obligate protein-protein interactions have been applied to the prediction of structural epitopes  <cit> .
for example, jones and thornton, using their method for predicting protein-protein binding sites  <cit> , successfully predicted b-cell epitopes on the surface of the β-subunit of human chorionic gonadotropin   <cit> .
since the number of available structures of antibody-protein complexes remains limited, thus far only a few methods, cep   <cit>  and discotope  <cit> , for b-cell epitope prediction using a protein of a given three-dimensional structure have been developed.
in the near future, with growth in the number of available structures of antibody-protein complexes, extensive development in this area is expected.
existing and new methods for epitope prediction demand a benchmark which will set the standard for the future comparison of methods.
to facilitate the further development of this standard, we have developed b-cell epitope benchmark datasets inferred from existing 3d structures of antibody-protein complexes.
further, using the benchmark datasets, we evaluated cep, discotope, and six recently developed publicly available web-servers for generalized protein-protein binding site prediction using various approaches: protein-protein docking ; structure-based methods applying different principals and trained on different datasets , and residue conservation .
RESULTS
structural epitope definition
three definitions of an epitope inferred from the x-ray structures of antibody-protein complexes were considered:  the epitope consists of protein antigen residues in which any atom of the residue looses more than 1Å <dig> of accessible surface area  upon antibody binding.
asa was calculated using the program naccess  <cit> ;  the epitope consists of protein antigen residues in which any atom of the epitope residue is separated from any antibody atom by a distance ≤ 4Å;  the epitope consists of protein antigen residues in which any atom of the epitope residue is separated from any antibody atom by a distance ≤ 5Å.
these three definitions were used for two reasons.
first, the methods evaluated in this work use one of these three definitions, second, we wished to study how the epitope definition influenced the results.
results  indicated that the structural epitope definition did not influence the outcome.
hence, unless otherwise specified, results are based on the second epitope definition.
construction of the benchmark datasets
two benchmark datasets were derived from the 3d structures of antibody-protein complexes available from the pdb  <cit> :
• dataset # <dig> – representative 3d structures of protein antigens with structural epitopes inferred from 3d structures of antibody-protein complexes.
this dataset is intended for the study of the antigenic properties of proteins as well as for development and evaluation of the methods based on protein structure alone, or protein-protein unbound docking methods, that is, if the structure of the antibody is known or can be modeled.
here this dataset was used for the evaluation of scale-based methods .
the dataset contains  <dig> antigens,  <dig> of which are one-chain antigen proteins.
• dataset # <dig> – representative 3d structures of antibody-protein complexes presenting different epitopes.
this dataset is useful for the study of the properties of individual epitopes as well as for development and evaluation of protein-protein bound docking methods.
since the current work attempts to compare the methods of different types, including protein-protein docking methods, this dataset was used to compare the performance of all methods to each other.
the dataset contains  <dig> structures of proteins in complexes with two-chain antibodies and  <dig> structures of proteins in complexes with one-chain antibodies.
the flowchart describing the construction of the benchmark datasets is shown in figure  <dig>  steps from  <dig> to  <dig> relate to dataset #1; steps 1– <dig> relate to dataset # <dig>
step  <dig> – crystal structures of protein antigens of length ≥ <dig> amino acids at a resolution ≤ 4Å in complex with antibody fragments containing variable regions  were collected from the protein data bank   <cit> .
structures in which the antibody binds antigen but involves no cdr residues have been excluded from the analysis; there were four such structures .
if a structure contained several complexes in one asymmetric unit and there was no structural difference observed between these complexes, only one complex was selected.
in this way  <dig> structures containing  <dig> antibody-protein complexes were selected:  <dig> complexes were formed by one-chain antibody fragments and  <dig> complexes by two-chain antibody fragments.
step  <dig> – all antigen protein chains were structurally aligned to one another using the ce algorithm  <cit> .
two protein chains were considered similar if all the following conditions applied:  rmsd ≤3Å,  z-score ≥ <dig> ,  number of residue-residue matches relative to the length of the longest chain ≥80%,  sequence identity in the structural alignment  ≥80%.
the z-score takes into account overall structural similarity and number of gapped positions.
two protein molecules were considered similar if each chain in one protein had a similar chain in another protein.
figure  <dig> demonstrates how the last two parameters, number of matches and sequence identity in the structural alignment, are defined.
the structural alignment rather than sequence alignment was used because protein structure is more conserved than sequence, and there can be expected regions in proteins with low sequence similarity that cannot be aligned by sequence alone.
the structural alignment also avoids considering two proteins as similar if they have similar sequences but different structures .
the threshold values were chosen empirically based on previous experience working with the ce algorithm.
as a result, the chosen threshold values separated human and bird lysozymes  and neuraminidases of different influenza virus strains, h3n <dig> and h11n <dig> .
step  <dig> –  <dig> proteins were orphans represented by only one 3d structure.
of the remaining  <dig> proteins represented by more than one 3d structure, the structure with the best resolution was selected as the representative structure.
the final representative dataset contained  <dig> antigens ,  <dig> of which were one-chain antigen proteins.
step  <dig> – for each protein, epitopes inferred from the 3d structures of antibody-protein complexes were mapped onto the representative structure of the protein.
first, epitope residues were calculated for each complex structure using one of the aforementioned epitope definitions.
second, epitope residues defined for the represented structures were mapped onto the representative structure based on the structure alignments.
for example, the hemagglutinin ha <dig> chain of influenza a virus was represented by six 3d structures of the protein in complexes with fab fragments of antibodies hc <dig> , bh <dig> , hc <dig> , and hc <dig> .
figure  <dig> illustrates a representative structure  of hemagglutinin ha <dig> upon which epitopes are mapped having been inferred from six complex structures.
in this way, epitopes inferred from  <dig> structures of antibody-protein complexes were mapped onto the  <dig> representative protein structures.
the resulting dataset is denoted dataset # <dig>  data on mapped epitope residues are available upon request.
step  <dig> – to study the properties of individual epitopes and their prediction a dataset of representative epitopes, dataset # <dig> derived from 3d structures of antibody-protein complexes defining different epitopes was constructed.
an important question to consider is how to define individual epitopes yet avoid bias by over-presentation of particular epitopes?
for example , while hc <dig>  and bh <dig>  epitopes overlap, neither hc <dig>  nor hc <dig>  epitopes overlap, they are separated on the protein surface.
nevertheless, hc <dig> and bh <dig> epitopes share residues , as do hc <dig> and hc <dig> epitopes .
are hc <dig> and bh <dig> epitopes similar or different?
this question is answered by considering the degree of overlap.
two epitopes are deemed similar if, in addition to the aforementioned criteria for epitope definition, they belong to similar protein chains and have >75% residues in common for both epitopes.
a cut-off value of 75% for epitope similarity was chosen empirically.
thus, the hc <dig> and bh <dig> epitopes on influenza a virus hemagglutinin ha <dig>  share  <dig> residues, that make up 74% and 93% of the size of hc <dig> and bh <dig> epitopes, respectively.
a cut-off on epitope overlap of less than 75% would define hc <dig> and bh <dig> epitopes as similar even though they are known to be different.
hc <dig> and bh <dig> are antibodies from different germ-lines with variable domains sharing only 56% sequence similarity, their h <dig> cdr regions adopt distinct conformations and these antibodies are tolerant to different mutations in hemagglutinin  <cit> .
another example, x <dig> and 17b epitopes of gp <dig> share 75% of their residues yet x <dig> and 17b antibodies are from different genes  <cit> .
a cut-off value for epitope similarity equal to or less than 75% would erroneously define x <dig> and 17b epitopes as similar.
conversely, a cut-off value of 80% would make epitopes inferred from different structures of the same antibody-protein complex dissimilar.
for example, the h <dig> epitope of t cell receptor n <dig> is inferred from two complex structures of a single crystal asymmetric unit - and -, where the letters denote protein chain identifiers) would be dissimilar.
given a 75% empirical cut-off for epitope similarity, epitopes inferred from structures of complexes with two-chain antibody fragments were divided into  <dig> singletons and  <dig> groups; epitopes inferred from structures of complexes with one-chain antibody fragments were divided into ten singletons and two groups.
step  <dig> – for each group of similar epitopes, the representative 3d structure of the antibody-protein complex was selected based upon the following preferences.
first, the structure with no or a minimal number of heteroatoms  and other protein chains in the interface  was preferred.
second, preference was given to the structure with the largest epitope, i.e., maximum number of epitope residues.
third, the structure with the best resolution ≤ <dig> Å was preferred.
dataset # <dig> of representative structures of antibody-protein complexes  consisted of  <dig> structures of proteins in complexes with two-chain antibody fragments and  <dig> structures of proteins in complexes with one-chain antibody fragments.
web-servers performance evaluation
using the benchmark datasets introduced above we evaluated eight recently-developed and publicly available web-servers.
the servers use different methods yet all have the goal of predicting either b-cell epitopes, or more generally protein-protein binding sites.
the servers are listed in table  <dig>  any reference in the text to the method actually means the server which implements that method, e.g., the dot method running on the cluspro server is called cluspro.
the methods fall into two categories:
• scale-based methods – each protein residue is assigned a value reflecting the probability of that residue being part of the protein interface or epitope.
discotope, pier, promate and consurf fall into this category.
• patch prediction and protein-protein docking methods – each protein residue is predicted to be part of a surface patch of residues defining the protein interface or epitope.
discotope, promate, cep, ppi-pred, cluspro, and patchdock fall into this category.
two methods, discotope and promate, fall into both categories since they predict patches and assign score values to each protein residue.
the evaluation of the methods was performed as follows.
first, the scale-based methods were analyzed on how well the residue score values discriminate epitope versus non-epitope residues using dataset # <dig>  further, performance of all methods was evaluated on their ability to recognize representative epitopes from dataset # <dig>  the first step is obviously not essential; it was performed as an example of the application of dataset # <dig> that can be used for future methods development and for revealing properties of epitope residues beyond the fact that epitopes are sites on the protein surface.
scale-based methods: score value distributions
discotope, pier, promate and consurf assign to each protein residue a score reflecting the probability of that residue being a part of the protein interface or epitope.
details are provided in the methods section.
for the analysis of epitope residues versus non-epitope residues we used dataset # <dig>  that is, representative antigen structures with epitopes mapped onto them.
here an epitope residue is an antigen residue known to be part of an epitope in any complex of this antigen with any antibody.
conversely a non-epitope residue implies an antigen residue which is not known to be part of a structural epitope.
to simplify the calculation proteins with epitopes located on more than one protein chain were discarded from the analyses .
as a result  <dig> protein antigens were analyzed .
the score distributions for epitope, non-epitope and all protein residues were calculated for each method and are shown in figures  <dig>   <dig>   <dig>   <dig>  distributions taking into account only surface residues were similar for all methods .
the definition of a surface residue is given in the methods section.
discotope, promate and consurf scores discriminate epitope versus non-epitope and versus all protein residues, while pier and consurf confidence scores do not.
thus, as one can see in figure  <dig>  discotope discriminates epitope residues  from non-epitope residues   and all antigen residues  .
these distributions are significantly different  regardless of the epitope definition used.
the consurf conservation score also discriminates epitope residues  versus non-epitope residues  and versus all antigen residues  .
the same was true for epitope vs. all surface residues.
further, the confidence level did not change when the definition of surface residues and/or epitope residues was changed .
however, if only residues with consurf confidence score values were considered, no significant difference between epitope and other protein residues was observed .
for promate mean scores for epitope residues  were significantly higher than for all antigen residues  or non-epitope residues or all surface residues  .
the pier score does not discriminate epitope versus other antigen residues  .
these results suggest that epitope residues are less conservative according to the consurf evolutionary conservancy scores than protein surface residues in general at a  <dig> % confidence level .
pier, which is trained on 3d structures of all protein-protein complexes available in the pdb, could not distinguish epitopes from the rest of the protein surface.
one possible explanation of this failure is that epitope residues do share some properties with residues having transient non-obligate hetero-interactions with other proteins.
promate is trained using such complexes  <cit> .
criteria and dataset used in methods evaluation
there is no commonly acceptable standard for evaluating binding site prediction methods.
some authors measure performance on a per protein bases, measuring statistics across the dataset  <cit> , while others measure performance on a per residue basis  <cit> .
some authors report sensitivity and specificity and measure the performance from the area under the roc curve  <cit> , while others consider only the sensitivity and positive predictive values and measure the method performance from the relative number of successful predictions in the test dataset  <cit> .
approaching the task of evaluation and comparison of different methods, we encountered a number of questions.
how can we compare scale-based methods with patch prediction and docking methods?
discotope and promate predict one patch per protein, while other methods predict several patches, how can these be compared?
using a score value assigned by promate, discotope, or consurf to a residue, all epitopes in the protein are taken into account, so can we say that the method predicts one epitope per protein?
is not the direct comparison of protein docking methods , patchdock) versus patch-based prediction methods  questionable since the former methods are based on optimization of an interaction energy function, while the latter depend on training?
finally, docking methods require knowledge of the structures of both interacting proteins, antigen and antibody, while binding site prediction methods are based on the structure of the protein antigen alone and do not require knowledge of the antibody structure.
is this a fair comparison?
being aware of these questions and limitations, we applied various evaluation criteria in an attempt to provide a thorough and fair comparison of the methods.
the evaluation was performed on the dataset of representative epitopes, assuming any antigen residue which is not a part of a considered epitope is part of a non-epitope.
we didn't discard non-epitope residues, which we know belong to some other epitope in the protein, because we assumed that a prediction program will predict an epitope in an antigen for which it doesn't have any additional information except its sequence and structure – this is how all evaluated methods were constructed.
the analysis was performed using the representative epitopes from dataset # <dig> that were inferred from structures of one-chain  antigens in complexes with two-chain antibody fragments.
there were  <dig> such epitopes in  <dig> antigens .
'na' means that results for the protein were not obtained.
significant predictions  are shown in bold.
& – epitopes used in the discotope training set are indicated by an asterisk; those not used in the training set are indicated by a hyphen.
the following parameters were used to evaluate the methods:
sensitivity ) = tp/ – a proportion of correctly predicted epitope residues  with respect to the total number of epitope residues .
specificity ) =  <dig> - fp/ – a proportion of correctly predicted non-epitope residues  with respect to the total number of non-epitope residues .
positive predictive value   = tp/ – a proportion of correctly predicted epitope residues  with respect to the total number of predicted epitope residues .
accuracy  = / – a proportion of correctly predicted epitope and non-epitope residues with respect to all residues.
area under the roc curve  - a roc curve is a graph representing a dependency of tpr versus fpr, or sensitivity versus specificity.
the auc measure is a widely used measure for immunoinformatics and bioinformatics methods; it has also been recommended for methods comparison in the recent report  <cit> .
the auc gives the general performance of the method; it is "equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance"  <cit> .
success rate - the number of epitopes from the dataset that were successfully predicted.
while the auc is a convenient and commonly used measure in immunoinformatics since many protein-protein binding site prediction methods, as well as three methods evaluated in the current work, promate, ppi-pred, and cep, used success rate as a measure of their performance, we considered it necessary to also calculate the methods success rates.
while this measure is easily and naturally interpretable, it requires us to define the successful prediction and that can be done in many different ways.
for this reason, many scientists try to avoid using this measure.
the statistical significance of a prediction, that is, the difference between observed and expected frequencies of an actual epitope/non-epitope residue in the predicted epitope/non-epitope, was determined by fisher's exact test .
the prediction was considered significant if the significance level was ≥95%, that is, the p-value was ≤  <dig> .
the above parameters were applied in evaluating the methods as follows:
for the scale-based methods, promate, discotope, consurf, and pier, by varying the threshold values for score values classifying epitope residues from non-epitope residues, the auc values have been calculated for each epitope.
success rates for all methods were calculated on a per protein bases taking into account one epitope per protein predicted with the highest significance.
such an approach assumes that if the epitope in a protein was successfully predicted, the prediction for the protein is successful.
criteria used for definition of successful prediction are discussed further.
patch prediction methods and protein-protein docking methods fall in the category of discrete classifiers, that is, they classify a residue as an epitope or non-epitope residue with no score assigned.
therefore, a roc curve cannot be generated for these methods, only the auc value can be estimated.
other statistics have also been obtained for these methods by averaging statistical values over epitopes and then calculating the overall statistical values over epitope and non-epitope residues in the dataset.
prediction of individual epitopes
the results for each method in predicting  <dig> representative epitopes are given in tables  <dig>   <dig> and supplementary materials .
for scale-based methods only the auc values were computed , while for patch prediction and docking methods all other statistics were produced .
'na' means that results for the epitope/protein were not obtained.
& – epitopes used in the discotope training set are indicated by an asterisk; those not used in the training set are indicated by a hyphen.
discotope and promate predict only one epitope per protein.
cluspro and patchdock rank predicted models starting from the model with the best score.
for these methods, the first  prediction was considered.
if it was not significant , the next by rank significant prediction  was reported in table  <dig>  since the number of epitopes predicted by cep in a protein varies  and they are not ranked, the average prediction was reported for each epitope.
more detailed statistics on the prediction results is provided in the supplementary table  .
no one epitope was predicted by all methods .
some epitopes, for example, hyhel- <dig> on hel  and 8–18c <dig> on myelin oligodendrocyte glycoprotein , were predicted by all methods except cep .
two epitopes, cetuximab on egfr  and 7e2c50s on cytochrome c oxidase , appeared to be difficult to predict; they could probably be predicted using the consurf average score in combination with a patch generation method.
the extracellular region of egfr  is a large  loosely-packed multi-domain protein with a lot of loops and hence epitope recognition appears difficult.
similarly, recognition of epitopes on subunit ii of cytochrome c oxidase  appears problematic because the protein possesses long protruded α-helixes.
the lower specificity of cep and discotope  results from these methods predicting larger epitopes  –  <dig> and discotope  –  <dig> residues) in comparison with other methods.
the average size of predicted epitope size for patchdock is  <dig> residues, cluspro  is  <dig> residues, and ppi-pred is  <dig> residues.
the size of actual epitopes in the dataset varies from  <dig> to  <dig> residues .
however, it should be emphasized that if the most of the methods considered were designed to predict an epitope as a whole single entity, discotope focuses on the prediction of individual epitope residues that can be part of several different epitopes in the same protein.
therefore, the average size of the epitope predicted by discotope is large; moreover, the predicted epitope residues can be located too far from each on the protein surface to form a single epitope.
overall performance of each method
the overall performance of each method have been compared first using average auc values for all methods and then calculating all other statistics for patch prediction and protein-protein docking methods.
both comparisons were made on different subsets of representative epitopes from dataset # <dig>
calculating auc values for all methods, we discarded from the analysis the proteins for which any method didn't produced a result  were not able to predict epitopes for several proteins, see methods).
the final subset contained  <dig> epitopes from table  <dig> of which  <dig> epitopes were not used for discotope training.
all other methods didn't use any epitopes for training.
auc values averaged on subsets of  <dig> and  <dig> epitopes are shown in figure  <dig>  consurf, discotope, ppi-pred and docking methods, when the  <dig> best models were considered, demonstrated average auc values above  <dig> , that is, poor or mediocre performance.
patchdock was the best, giving an auc of  <dig> .
all other methods performed close to random .
discotope gave auc values of  <dig>  and  <dig>  on all representative epitopes and those that were not used by the method for training, respectively.
when discotope performance was evaluated by the authors of the method  <cit> , it gave an auc value of  <dig>  averaged over the five evaluation sets used for cross-validation.
for patch prediction and docking methods, to avoid the problematic comparison of methods predicting one epitope per protein with those that predict several epitopes, all epitopes from proteins with more than one epitope have been removed from dataset # <dig>  epitopes from proteins for which any method did not produce the prediction have also been discarded.
the following statistics were calculated on the resulting subset of epitopes.
first, fp, fn, tp, and tn values were summarized for the whole pool of epitopes, and sensitivity, specificity, accuracy, ppv, and auc values calculated for each method .
auc values obtained in this way were close to those demonstrated in fig.
<dig>  the best performers were docking methods patchdock and dot when the top ten models were considered, giving auc values of  <dig>  and  <dig> , respectively .
among the methods producing one prediction per protein, discotope was rated the best by with an auc of  <dig> .
second, statistics were averaged over epitopes .
the overall performance was poor for all methods.
the best performance demonstrated by docking methods  was 41% ppv  and 46% sensitivity  for cluspro and 30% ppv and 42% sensitivity for patchdock.
among the methods producing one prediction per protein, discotope was rated the best by sensitivity  and cluspro first model by ppv  .
comparison of success rates
since patch prediction methods used in the current analysis used success rate as a performance measure, we additionally calculated the methods success rate on the subset of  <dig> epitopes used for overall methods comparison above.
the prediction of each epitope was deemed successful if the auc value was above a threshold value of  <dig>  or  <dig> .
the results are presented in fig.
<dig>
the proportion of epitopes predicted with an auc ≥  <dig>  for the scale-based methods  and the methods providing only one prediction per protein  first model) was lower than 50% except for consurf and discotope methods, which showed success rates of approximately 60% .
among the methods predicting several epitopes per protein , and cep) patchdock performed best with >75% successful predictions at an auc ≥  <dig>  and 55% at an auc ≥  <dig>  .
ppi-pred predicted 75% of protein-protein binding interfaces successfully, with a specificity over 50% and sensitivity over 20%, values previously used to claim success  <cit> .
the promate's authors reported a success rate for protein-protein binding site prediction of 70%  <cit> , while application of the criteria used in ppi-pred gave promate's success rate as 36%  <cit> .
according to our data , the prediction with an auc ≥  <dig>  corresponded to a significant prediction  at a sensitivity >30%.
using an auc ≥  <dig>  as a criterion of successful prediction, ppi-pred gave 60% and promate 35% successful predictions, respectively .
neither promate nor ppi-pred used antibody-protein interfaces for their methods development; nevertheless, they predicted epitopes with a success rate comparable to those for prediction of protein interfaces.
epitopes and other protein-protein interfaces indeed share many properties.
thus, blythe  <cit>  compared  <dig> protein-protein binding interfaces of  <dig> proteins from the dataset used for promate development  <cit>  with epitopes and paratopes inferred from x-ray structures of  <dig> complexes calculating the following interface properties: amino acid composition, hydrophobicity by the eisenberg's scale  <cit> , amino acid contribution to form inter-molecular hydrogen bonds, residue evolutionary conservancy, and several geometrical parameters, such as planarity and complementarity of interfaces.
epitopes and non-obligate heterodimer interfaces were very similar considering all the aforementioned properties except residue conservancy; epitope residues were more variable than heterodimer interfaces  <cit> .
the current work additionally demonstrates that, on average, epitope residues are significantly less conservative than protein surface residues.
indeed, protein-protein interaction sites are under evolutionary pressure to be more conserved than protein surface residues on average.
while antibody-antigen interactions are not under evolutionary pressure, they are under the selection pressure from the host immune system.
this selection pressure is assumed to cause polymorphisms in pathogens and to explain the variability of immune epitopes.
CONCLUSIONS
benchmark datasets for use in b cell structural epitope prediction have been constructed and made available.
using these benchmark data, eight publicly available web servers and their associated methods were evaluated.
several schemes for methods evaluation were considered.
the overall performance was poor for all methods and did not exceed an average auc of  <dig>  and 40% positive predictive value  at 46% sensitivity .
the values of the area under the receiver operating characteristic  curve for the evaluated methods were about  <dig>  for consurf, discotope, and ppi-pred  and above  <dig>  but not exceeding  <dig>  for protein-protein docking methods when the best of the top ten models for the bound docking were considered.
certainly a best case, since under real conditions many more models would be presented.
other methods, pier, promate , cep, ppi-pred first patch, and the first models of docking methods, performed close to random.
despite the fact that structural epitopes and protein-protein non-obligate transient heterodimer interfaces share many properties, protein-protein binding site prediction methods were poor epitope predictors.
when the top ten models and bound docking were considered, the docking methods performed the best, especially patchdock, where success can be explained by application of the cdr filter, which the dot algorithm does not use.
independent evaluation of ppi-pred and four docking algorithms  made by martin blythe  <cit>  and not available to us until the peer-reviewing stage of the manuscript agrees with the results presenting in the current work.
using the matthew's correlation coefficient , blythe measured the correlation between predicted and structural epitopes and paratopes inferred from  <dig> antibody-protein complexes.
for the first models, all evaluated methods demonstrated near random correlations.
likewise, when the top ten models for each complex were considered, low and negative mcc values prevailed over positive values for all algorithms except patchdock.
further experiments demonstrated that using the cdr filter may improve the prediction.
thus, using predefined cdrs for antibodies, the dot method significantly improved and showed mcc values comparable with those for patchdock  <cit> .
obviously, unbound docking would have more practical value for epitope prediction than bound docking.
however, the performance of unbound docking for antibody-antigen interactions, as was shown by the authors of patchdock, was unsatisfactory in comparison to bound docking and other protein-protein interaction methods  <cit> .
while the bound docking considered in this work has no practical value for epitope prediction, it needs to be benchmarked to further improve unbound docking algorithms and tune them for modeling antibody-antigen complexes.
currently the problem of b-cell epitope prediction is far from solved: structure-based method for prediction of discontinuous epitopes perform on the same level as sequence-based methods for prediction of continuous epitopes giving the area under the receiver operating characteristic curve  values of approximately  <dig>   <cit> .
three definitions of an epitope inferred from the x-ray structure of antibody-protein complexes were considered, but this made no significant difference to the predictions.
hence, we finally considered an epitope residue as the protein antigen residue for which any atom is separated from any antibody atom by a distance ≤ 4Å.
currently, each method requires writing a separate parser taking into account different representations of the output data.
there is a need to develop a common format for output data generated by both scale-based and patch generation tools that is easily interpreted by both a human and computer.
given these shortcomings and current success rates, how can epitope prediction be improved?
the availability of larger datasets containing only well-defined epitopes inferred from x-ray structures of antibody-protein complexes, which are then used for training, would help.
this will come over time as the pdb continues to grow at a rapid rate.
this need, in the context of continuous epitope prediction, has been noted by others  <cit> .
the performance of docking algorithms might be improved by tuning them specifically for antibody-antigen complexes.
existing b-cell epitope prediction methods utilize only a few features characterizing epitopes .
therefore, another possible way for improving the prediction would be to utilize more features that discriminate epitopes from non-epitopes, for example, the evolutionary conservation score.
this assumes that an epitope is indeed a discreet entity based upon what we know about proteins today.
perhaps the more fundamental question is whether it makes sense to consider a b-cell epitope a discrete feature of a protein at all?
time will tell as more x-ray structures on antibody-protein complexes become available.

 a b-cell epitope is defined as a part of a protein antigen recognized by either a particular antibody molecule or a particular b-cell receptor of the immune system. the main objective of b-cell epitope prediction is to facilitate the design of a short peptide or other molecule that can be synthesized and used instead of the antigen, which in the case of a pathogenic virus or bacteria, may be harmful to a researcher or experimental animal. a b-cell epitope may be continuous, that is, a short contiguous stretch of amino acid residues, or discontinuous, comprising atoms from distant residues but close in three-dimensional space and on the surface of the protein. synthetic peptides mimicking epitopes, as well as anti-peptide antibodies, have many applications in the diagnosis of various human diseases. also, the attempts have been made to develop peptide-based synthetic prophylactic vaccines for various infections, as well as therapeutic vaccines for chronic infections and noninfectious diseases, including autoimmune diseases, neurological disorders, allergies, and cancers. the immunoinformatics software and databases developed to facilitate vaccine design have previously been reviewed. during the last years b-cell epitope prediction methods have focused primarily on continuous epitopes. they were mostly sequence-dependent methods based upon various amino acid properties, such as hydrophilicity, solvent accessibility, secondary structure, and others. recently, several methods using machine learning approaches have been introduced that apply hidden markov models, artificial neural networks, support vector machine, and other techniques. recent assessments of continuous epitope prediction methods demonstrate that "single-scale amino acid propensity profiles cannot be used to predict epitope location reliably" and that "the combination of scales and experimentation with several machine learning algorithms showed little improvement over single scale-based methods". as crystallographic studies of antibody-protein complexes have shown, most b-cell epitopes are discontinuous. in the first attempts at epitope prediction based on 3d protein structure was made for a few proteins for which continuous epitopes were known. subsequently, thornton and colleagues proposed a method to locate potential discontinuous epitopes based on a protrusion of protein regions from the protein's globular surface. however, until the first x-ray structure of an antibody-protein complex was solved in, protein structural data were mostly used for prediction of continuous rather than discontinuous epitopes. in cases where the three-dimensional structure of the protein or its homologue is known, a discontinuous epitope can be derived from functional assays by mapping onto the protein structure residues involved in antibody recognition. however, an epitope identified using an immunoassay may be an artefact of measuring cross-reactivity of antibodies due to the presence of denatured or degraded proteins, or due to conformational changes in the protein caused by residue substitutions that may even lead to protein mis-folding. therefore, structural methods, particularly x-ray crystallography of antibody-antigen complexes, generally identify b-cell epitopes more reliably than functional assays. b-cell epitopes can be thought of in a structural and functional sense. structural epitopes are defined by a set of residues or atoms in the protein antigen contacting antibody residues or atoms. in contrast, a functional epitope consists of antigen residues that contribute significantly to antibody binding. functional epitopes are determined through functional assays or calculated theoretically using known structures of antibody-protein complexes. thus, functional and structural epitopes are not necessary the same. functional epitopes in proteins are usually smaller than structural epitopes; only three to five residues of the structural epitope contribute significantly to the antibody-antigen binding energy. this work focuses on structural epitopes inferred from known 3d structures of antibody-protein complexes available in the protein data bank. antibody-protein complexes can be categorized as intermediate transient non-obligate protein-protein complexes. non-obligate complexes, implying that individual components can be found on their own in vivo, are classified as either permanent or transient depending on their stability under particular physiological and environmental conditions. for example, many enzyme-inhibitor complexes are permanent non-obligate complexes. transient non-obligate complexes range from weak, to intermediate, and to strong. most antibodies demonstrate intermediate affinity for their specific antigens. based on this classification, general methods for the prediction of intermediate transient non-obligate protein-protein interactions have been applied to the prediction of structural epitopes. for example, jones and thornton, using their method for predicting protein-protein binding sites, successfully predicted b-cell epitopes on the surface of the β-subunit of human chorionic gonadotropin. since the number of available structures of antibody-protein complexes remains limited, thus far only a few methods, cep and discotope, for b-cell epitope prediction using a protein of a given three-dimensional structure have been developed. in the near future, with growth in the number of available structures of antibody-protein complexes, extensive development in this area is expected. existing and new methods for epitope prediction demand a benchmark which will set the standard for the future comparison of methods. to facilitate the further development of this standard, we have developed b-cell epitope benchmark datasets inferred from existing 3d structures of antibody-protein complexes. further, using the benchmark datasets, we evaluated cep, discotope, and six recently developed publicly available web-servers for generalized protein-protein binding site prediction using various approaches: protein-protein docking; structure-based methods applying different principals and trained on different datasets, and residue conservation.  structural epitope definition three definitions of an epitope inferred from the x-ray structures of antibody-protein complexes were considered: the epitope consists of protein antigen residues in which any atom of the residue looses more than 1Å of accessible surface area upon antibody binding. asa was calculated using the program naccess; the epitope consists of protein antigen residues in which any atom of the epitope residue is separated from any antibody atom by a distance ≤ 4Å; the epitope consists of protein antigen residues in which any atom of the epitope residue is separated from any antibody atom by a distance ≤ 5Å. these three definitions were used for two reasons. first, the methods evaluated in this work use one of these three definitions, second, we wished to study how the epitope definition influenced the results. results indicated that the structural epitope definition did not influence the outcome. hence, unless otherwise specified, results are based on the second epitope definition. construction of the benchmark datasets two benchmark datasets were derived from the 3d structures of antibody-protein complexes available from the pdb: • dataset # – representative 3d structures of protein antigens with structural epitopes inferred from 3d structures of antibody-protein complexes. this dataset is intended for the study of the antigenic properties of proteins as well as for development and evaluation of the methods based on protein structure alone, or protein-protein unbound docking methods, that is, if the structure of the antibody is known or can be modeled. here this dataset was used for the evaluation of scale-based methods. the dataset contains antigens, of which are one-chain antigen proteins. • dataset # – representative 3d structures of antibody-protein complexes presenting different epitopes. this dataset is useful for the study of the properties of individual epitopes as well as for development and evaluation of protein-protein bound docking methods. since the current work attempts to compare the methods of different types, including protein-protein docking methods, this dataset was used to compare the performance of all methods to each other. the dataset contains structures of proteins in complexes with two-chain antibodies and structures of proteins in complexes with one-chain antibodies. the flowchart describing the construction of the benchmark datasets is shown in figure steps from to relate to dataset #1; steps 1– relate to dataset # step – crystal structures of protein antigens of length ≥ amino acids at a resolution ≤ 4Å in complex with antibody fragments containing variable regions were collected from the protein data bank. structures in which the antibody binds antigen but involves no cdr residues have been excluded from the analysis; there were four such structures. if a structure contained several complexes in one asymmetric unit and there was no structural difference observed between these complexes, only one complex was selected. in this way structures containing antibody-protein complexes were selected: complexes were formed by one-chain antibody fragments and complexes by two-chain antibody fragments. step – all antigen protein chains were structurally aligned to one another using the ce algorithm. two protein chains were considered similar if all the following conditions applied: rmsd ≤3Å, z-score ≥, number of residue-residue matches relative to the length of the longest chain ≥80%, sequence identity in the structural alignment ≥80%. the z-score takes into account overall structural similarity and number of gapped positions. two protein molecules were considered similar if each chain in one protein had a similar chain in another protein. figure demonstrates how the last two parameters, number of matches and sequence identity in the structural alignment, are defined. the structural alignment rather than sequence alignment was used because protein structure is more conserved than sequence, and there can be expected regions in proteins with low sequence similarity that cannot be aligned by sequence alone. the structural alignment also avoids considering two proteins as similar if they have similar sequences but different structures. the threshold values were chosen empirically based on previous experience working with the ce algorithm. as a result, the chosen threshold values separated human and bird lysozymes and neuraminidases of different influenza virus strains, h3n and h11n. step – proteins were orphans represented by only one 3d structure. of the remaining proteins represented by more than one 3d structure, the structure with the best resolution was selected as the representative structure. the final representative dataset contained antigens, of which were one-chain antigen proteins. step – for each protein, epitopes inferred from the 3d structures of antibody-protein complexes were mapped onto the representative structure of the protein. first, epitope residues were calculated for each complex structure using one of the aforementioned epitope definitions. second, epitope residues defined for the represented structures were mapped onto the representative structure based on the structure alignments. for example, the hemagglutinin ha chain of influenza a virus was represented by six 3d structures of the protein in complexes with fab fragments of antibodies hc, bh, hc, and hc. figure illustrates a representative structure of hemagglutinin ha upon which epitopes are mapped having been inferred from six complex structures. in this way, epitopes inferred from structures of antibody-protein complexes were mapped onto the representative protein structures. the resulting dataset is denoted dataset # data on mapped epitope residues are available upon request. step – to study the properties of individual epitopes and their prediction a dataset of representative epitopes, dataset # derived from 3d structures of antibody-protein complexes defining different epitopes was constructed. an important question to consider is how to define individual epitopes yet avoid bias by over-presentation of particular epitopes? for example, while hc and bh epitopes overlap, neither hc nor hc epitopes overlap, they are separated on the protein surface. nevertheless, hc and bh epitopes share residues, as do hc and hc epitopes. are hc and bh epitopes similar or different? this question is answered by considering the degree of overlap. two epitopes are deemed similar if, in addition to the aforementioned criteria for epitope definition, they belong to similar protein chains and have >75% residues in common for both epitopes. a cut-off value of 75% for epitope similarity was chosen empirically. thus, the hc and bh epitopes on influenza a virus hemagglutinin ha share residues, that make up 74% and 93% of the size of hc and bh epitopes, respectively. a cut-off on epitope overlap of less than 75% would define hc and bh epitopes as similar even though they are known to be different. hc and bh are antibodies from different germ-lines with variable domains sharing only 56% sequence similarity, their h cdr regions adopt distinct conformations and these antibodies are tolerant to different mutations in hemagglutinin. another example, x and 17b epitopes of gp share 75% of their residues yet x and 17b antibodies are from different genes. a cut-off value for epitope similarity equal to or less than 75% would erroneously define x and 17b epitopes as similar. conversely, a cut-off value of 80% would make epitopes inferred from different structures of the same antibody-protein complex dissimilar. for example, the h epitope of t cell receptor n is inferred from two complex structures of a single crystal asymmetric unit - and -, where the letters denote protein chain identifiers) would be dissimilar. given a 75% empirical cut-off for epitope similarity, epitopes inferred from structures of complexes with two-chain antibody fragments were divided into singletons and groups; epitopes inferred from structures of complexes with one-chain antibody fragments were divided into ten singletons and two groups. step – for each group of similar epitopes, the representative 3d structure of the antibody-protein complex was selected based upon the following preferences. first, the structure with no or a minimal number of heteroatoms and other protein chains in the interface was preferred. second, preference was given to the structure with the largest epitope, i.e., maximum number of epitope residues. third, the structure with the best resolution ≤ Å was preferred. dataset # of representative structures of antibody-protein complexes consisted of structures of proteins in complexes with two-chain antibody fragments and structures of proteins in complexes with one-chain antibody fragments. web-servers performance evaluation using the benchmark datasets introduced above we evaluated eight recently-developed and publicly available web-servers. the servers use different methods yet all have the goal of predicting either b-cell epitopes, or more generally protein-protein binding sites. the servers are listed in table any reference in the text to the method actually means the server which implements that method, e.g., the dot method running on the cluspro server is called cluspro. the methods fall into two categories: • scale-based methods – each protein residue is assigned a value reflecting the probability of that residue being part of the protein interface or epitope. discotope, pier, promate and consurf fall into this category. • patch prediction and protein-protein docking methods – each protein residue is predicted to be part of a surface patch of residues defining the protein interface or epitope. discotope, promate, cep, ppi-pred, cluspro, and patchdock fall into this category. two methods, discotope and promate, fall into both categories since they predict patches and assign score values to each protein residue. the evaluation of the methods was performed as follows. first, the scale-based methods were analyzed on how well the residue score values discriminate epitope versus non-epitope residues using dataset # further, performance of all methods was evaluated on their ability to recognize representative epitopes from dataset # the first step is obviously not essential; it was performed as an example of the application of dataset # that can be used for future methods development and for revealing properties of epitope residues beyond the fact that epitopes are sites on the protein surface. scale-based methods: score value distributions discotope, pier, promate and consurf assign to each protein residue a score reflecting the probability of that residue being a part of the protein interface or epitope. details are provided in the methods section. for the analysis of epitope residues versus non-epitope residues we used dataset # that is, representative antigen structures with epitopes mapped onto them. here an epitope residue is an antigen residue known to be part of an epitope in any complex of this antigen with any antibody. conversely a non-epitope residue implies an antigen residue which is not known to be part of a structural epitope. to simplify the calculation proteins with epitopes located on more than one protein chain were discarded from the analyses. as a result protein antigens were analyzed. the score distributions for epitope, non-epitope and all protein residues were calculated for each method and are shown in figures distributions taking into account only surface residues were similar for all methods. the definition of a surface residue is given in the methods section. discotope, promate and consurf scores discriminate epitope versus non-epitope and versus all protein residues, while pier and consurf confidence scores do not. thus, as one can see in figure discotope discriminates epitope residues from non-epitope residues and all antigen residues. these distributions are significantly different regardless of the epitope definition used. the consurf conservation score also discriminates epitope residues versus non-epitope residues and versus all antigen residues. the same was true for epitope vs. all surface residues. further, the confidence level did not change when the definition of surface residues and/or epitope residues was changed. however, if only residues with consurf confidence score values were considered, no significant difference between epitope and other protein residues was observed. for promate mean scores for epitope residues were significantly higher than for all antigen residues or non-epitope residues or all surface residues. the pier score does not discriminate epitope versus other antigen residues. these results suggest that epitope residues are less conservative according to the consurf evolutionary conservancy scores than protein surface residues in general at a % confidence level. pier, which is trained on 3d structures of all protein-protein complexes available in the pdb, could not distinguish epitopes from the rest of the protein surface. one possible explanation of this failure is that epitope residues do share some properties with residues having transient non-obligate hetero-interactions with other proteins. promate is trained using such complexes. criteria and dataset used in methods evaluation there is no commonly acceptable standard for evaluating binding site prediction methods. some authors measure performance on a per protein bases, measuring statistics across the dataset, while others measure performance on a per residue basis. some authors report sensitivity and specificity and measure the performance from the area under the roc curve, while others consider only the sensitivity and positive predictive values and measure the method performance from the relative number of successful predictions in the test dataset. approaching the task of evaluation and comparison of different methods, we encountered a number of questions. how can we compare scale-based methods with patch prediction and docking methods? discotope and promate predict one patch per protein, while other methods predict several patches, how can these be compared? using a score value assigned by promate, discotope, or consurf to a residue, all epitopes in the protein are taken into account, so can we say that the method predicts one epitope per protein? is not the direct comparison of protein docking methods, patchdock) versus patch-based prediction methods questionable since the former methods are based on optimization of an interaction energy function, while the latter depend on training? finally, docking methods require knowledge of the structures of both interacting proteins, antigen and antibody, while binding site prediction methods are based on the structure of the protein antigen alone and do not require knowledge of the antibody structure. is this a fair comparison? being aware of these questions and limitations, we applied various evaluation criteria in an attempt to provide a thorough and fair comparison of the methods. the evaluation was performed on the dataset of representative epitopes, assuming any antigen residue which is not a part of a considered epitope is part of a non-epitope. we didn't discard non-epitope residues, which we know belong to some other epitope in the protein, because we assumed that a prediction program will predict an epitope in an antigen for which it doesn't have any additional information except its sequence and structure – this is how all evaluated methods were constructed. the analysis was performed using the representative epitopes from dataset # that were inferred from structures of one-chain antigens in complexes with two-chain antibody fragments. there were such epitopes in antigens.'na' means that results for the protein were not obtained. significant predictions are shown in bold. & – epitopes used in the discotope training set are indicated by an asterisk; those not used in the training set are indicated by a hyphen. the following parameters were used to evaluate the methods: sensitivity ) = tp/ – a proportion of correctly predicted epitope residues with respect to the total number of epitope residues. specificity ) = - fp/ – a proportion of correctly predicted non-epitope residues with respect to the total number of non-epitope residues. positive predictive value = tp/ – a proportion of correctly predicted epitope residues with respect to the total number of predicted epitope residues. accuracy = / – a proportion of correctly predicted epitope and non-epitope residues with respect to all residues. area under the roc curve - a roc curve is a graph representing a dependency of tpr versus fpr, or sensitivity versus specificity. the auc measure is a widely used measure for immunoinformatics and bioinformatics methods; it has also been recommended for methods comparison in the recent report. the auc gives the general performance of the method; it is "equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance". success rate - the number of epitopes from the dataset that were successfully predicted. while the auc is a convenient and commonly used measure in immunoinformatics since many protein-protein binding site prediction methods, as well as three methods evaluated in the current work, promate, ppi-pred, and cep, used success rate as a measure of their performance, we considered it necessary to also calculate the methods success rates. while this measure is easily and naturally interpretable, it requires us to define the successful prediction and that can be done in many different ways. for this reason, many scientists try to avoid using this measure. the statistical significance of a prediction, that is, the difference between observed and expected frequencies of an actual epitope/non-epitope residue in the predicted epitope/non-epitope, was determined by fisher's exact test. the prediction was considered significant if the significance level was ≥95%, that is, the p-value was ≤. the above parameters were applied in evaluating the methods as follows: for the scale-based methods, promate, discotope, consurf, and pier, by varying the threshold values for score values classifying epitope residues from non-epitope residues, the auc values have been calculated for each epitope. success rates for all methods were calculated on a per protein bases taking into account one epitope per protein predicted with the highest significance. such an approach assumes that if the epitope in a protein was successfully predicted, the prediction for the protein is successful. criteria used for definition of successful prediction are discussed further. patch prediction methods and protein-protein docking methods fall in the category of discrete classifiers, that is, they classify a residue as an epitope or non-epitope residue with no score assigned. therefore, a roc curve cannot be generated for these methods, only the auc value can be estimated. other statistics have also been obtained for these methods by averaging statistical values over epitopes and then calculating the overall statistical values over epitope and non-epitope residues in the dataset. prediction of individual epitopes the results for each method in predicting representative epitopes are given in tables and supplementary materials. for scale-based methods only the auc values were computed, while for patch prediction and docking methods all other statistics were produced.'na' means that results for the epitope/protein were not obtained. & – epitopes used in the discotope training set are indicated by an asterisk; those not used in the training set are indicated by a hyphen. discotope and promate predict only one epitope per protein. cluspro and patchdock rank predicted models starting from the model with the best score. for these methods, the first prediction was considered. if it was not significant, the next by rank significant prediction was reported in table since the number of epitopes predicted by cep in a protein varies and they are not ranked, the average prediction was reported for each epitope. more detailed statistics on the prediction results is provided in the supplementary table. no one epitope was predicted by all methods. some epitopes, for example, hyhel- on hel and 8–18c on myelin oligodendrocyte glycoprotein, were predicted by all methods except cep. two epitopes, cetuximab on egfr and 7e2c50s on cytochrome c oxidase, appeared to be difficult to predict; they could probably be predicted using the consurf average score in combination with a patch generation method. the extracellular region of egfr is a large loosely-packed multi-domain protein with a lot of loops and hence epitope recognition appears difficult. similarly, recognition of epitopes on subunit ii of cytochrome c oxidase appears problematic because the protein possesses long protruded α-helixes. the lower specificity of cep and discotope results from these methods predicting larger epitopes – and discotope – residues) in comparison with other methods. the average size of predicted epitope size for patchdock is residues, cluspro is residues, and ppi-pred is residues. the size of actual epitopes in the dataset varies from to residues. however, it should be emphasized that if the most of the methods considered were designed to predict an epitope as a whole single entity, discotope focuses on the prediction of individual epitope residues that can be part of several different epitopes in the same protein. therefore, the average size of the epitope predicted by discotope is large; moreover, the predicted epitope residues can be located too far from each on the protein surface to form a single epitope. overall performance of each method the overall performance of each method have been compared first using average auc values for all methods and then calculating all other statistics for patch prediction and protein-protein docking methods. both comparisons were made on different subsets of representative epitopes from dataset # calculating auc values for all methods, we discarded from the analysis the proteins for which any method didn't produced a result were not able to predict epitopes for several proteins, see methods). the final subset contained epitopes from table of which epitopes were not used for discotope training. all other methods didn't use any epitopes for training. auc values averaged on subsets of and epitopes are shown in figure consurf, discotope, ppi-pred and docking methods, when the best models were considered, demonstrated average auc values above, that is, poor or mediocre performance. patchdock was the best, giving an auc of. all other methods performed close to random. discotope gave auc values of and on all representative epitopes and those that were not used by the method for training, respectively. when discotope performance was evaluated by the authors of the method, it gave an auc value of averaged over the five evaluation sets used for cross-validation. for patch prediction and docking methods, to avoid the problematic comparison of methods predicting one epitope per protein with those that predict several epitopes, all epitopes from proteins with more than one epitope have been removed from dataset # epitopes from proteins for which any method did not produce the prediction have also been discarded. the following statistics were calculated on the resulting subset of epitopes. first, fp, fn, tp, and tn values were summarized for the whole pool of epitopes, and sensitivity, specificity, accuracy, ppv, and auc values calculated for each method. auc values obtained in this way were close to those demonstrated in fig. the best performers were docking methods patchdock and dot when the top ten models were considered, giving auc values of and, respectively. among the methods producing one prediction per protein, discotope was rated the best by with an auc of. second, statistics were averaged over epitopes. the overall performance was poor for all methods. the best performance demonstrated by docking methods was 41% ppv and 46% sensitivity for cluspro and 30% ppv and 42% sensitivity for patchdock. among the methods producing one prediction per protein, discotope was rated the best by sensitivity and cluspro first model by ppv. comparison of success rates since patch prediction methods used in the current analysis used success rate as a performance measure, we additionally calculated the methods success rate on the subset of epitopes used for overall methods comparison above. the prediction of each epitope was deemed successful if the auc value was above a threshold value of or. the results are presented in fig. the proportion of epitopes predicted with an auc ≥ for the scale-based methods and the methods providing only one prediction per protein first model) was lower than 50% except for consurf and discotope methods, which showed success rates of approximately 60%. among the methods predicting several epitopes per protein, and cep) patchdock performed best with >75% successful predictions at an auc ≥ and 55% at an auc ≥. ppi-pred predicted 75% of protein-protein binding interfaces successfully, with a specificity over 50% and sensitivity over 20%, values previously used to claim success. the promate's authors reported a success rate for protein-protein binding site prediction of 70%, while application of the criteria used in ppi-pred gave promate's success rate as 36%. according to our data, the prediction with an auc ≥ corresponded to a significant prediction at a sensitivity >30%. using an auc ≥ as a criterion of successful prediction, ppi-pred gave 60% and promate 35% successful predictions, respectively. neither promate nor ppi-pred used antibody-protein interfaces for their methods development; nevertheless, they predicted epitopes with a success rate comparable to those for prediction of protein interfaces. epitopes and other protein-protein interfaces indeed share many properties. thus, blythe compared protein-protein binding interfaces of proteins from the dataset used for promate development with epitopes and paratopes inferred from x-ray structures of complexes calculating the following interface properties: amino acid composition, hydrophobicity by the eisenberg's scale, amino acid contribution to form inter-molecular hydrogen bonds, residue evolutionary conservancy, and several geometrical parameters, such as planarity and complementarity of interfaces. epitopes and non-obligate heterodimer interfaces were very similar considering all the aforementioned properties except residue conservancy; epitope residues were more variable than heterodimer interfaces. the current work additionally demonstrates that, on average, epitope residues are significantly less conservative than protein surface residues. indeed, protein-protein interaction sites are under evolutionary pressure to be more conserved than protein surface residues on average. while antibody-antigen interactions are not under evolutionary pressure, they are under the selection pressure from the host immune system. this selection pressure is assumed to cause polymorphisms in pathogens and to explain the variability of immune epitopes.  benchmark datasets for use in b cell structural epitope prediction have been constructed and made available. using these benchmark data, eight publicly available web servers and their associated methods were evaluated. several schemes for methods evaluation were considered. the overall performance was poor for all methods and did not exceed an average auc of and 40% positive predictive value at 46% sensitivity. the values of the area under the receiver operating characteristic curve for the evaluated methods were about for consurf, discotope, and ppi-pred and above but not exceeding for protein-protein docking methods when the best of the top ten models for the bound docking were considered. certainly a best case, since under real conditions many more models would be presented. other methods, pier, promate, cep, ppi-pred first patch, and the first models of docking methods, performed close to random. despite the fact that structural epitopes and protein-protein non-obligate transient heterodimer interfaces share many properties, protein-protein binding site prediction methods were poor epitope predictors. when the top ten models and bound docking were considered, the docking methods performed the best, especially patchdock, where success can be explained by application of the cdr filter, which the dot algorithm does not use. independent evaluation of ppi-pred and four docking algorithms made by martin blythe and not available to us until the peer-reviewing stage of the manuscript agrees with the results presenting in the current work. using the matthew's correlation coefficient, blythe measured the correlation between predicted and structural epitopes and paratopes inferred from antibody-protein complexes. for the first models, all evaluated methods demonstrated near random correlations. likewise, when the top ten models for each complex were considered, low and negative mcc values prevailed over positive values for all algorithms except patchdock. further experiments demonstrated that using the cdr filter may improve the prediction. thus, using predefined cdrs for antibodies, the dot method significantly improved and showed mcc values comparable with those for patchdock. obviously, unbound docking would have more practical value for epitope prediction than bound docking. however, the performance of unbound docking for antibody-antigen interactions, as was shown by the authors of patchdock, was unsatisfactory in comparison to bound docking and other protein-protein interaction methods. while the bound docking considered in this work has no practical value for epitope prediction, it needs to be benchmarked to further improve unbound docking algorithms and tune them for modeling antibody-antigen complexes. currently the problem of b-cell epitope prediction is far from solved: structure-based method for prediction of discontinuous epitopes perform on the same level as sequence-based methods for prediction of continuous epitopes giving the area under the receiver operating characteristic curve values of approximately. three definitions of an epitope inferred from the x-ray structure of antibody-protein complexes were considered, but this made no significant difference to the predictions. hence, we finally considered an epitope residue as the protein antigen residue for which any atom is separated from any antibody atom by a distance ≤ 4Å. currently, each method requires writing a separate parser taking into account different representations of the output data. there is a need to develop a common format for output data generated by both scale-based and patch generation tools that is easily interpreted by both a human and computer. given these shortcomings and current success rates, how can epitope prediction be improved? the availability of larger datasets containing only well-defined epitopes inferred from x-ray structures of antibody-protein complexes, which are then used for training, would help. this will come over time as the pdb continues to grow at a rapid rate. this need, in the context of continuous epitope prediction, has been noted by others. the performance of docking algorithms might be improved by tuning them specifically for antibody-antigen complexes. existing b-cell epitope prediction methods utilize only a few features characterizing epitopes. therefore, another possible way for improving the prediction would be to utilize more features that discriminate epitopes from non-epitopes, for example, the evolutionary conservation score. this assumes that an epitope is indeed a discreet entity based upon what we know about proteins today. perhaps the more fundamental question is whether it makes sense to consider a b-cell epitope a discrete feature of a protein at all? time will tell as more x-ray structures on antibody-protein complexes become available. 
it may be possible to improve epitope prediction methods through training on datasets which include only immune epitopes and through utilizing more features characterizing epitopes, for example, the evolutionary conservation score. as the number of structures of antibody-protein complexes grows, further interest in prediction methods using 3d structure is anticipated. two b-cell epitope benchmark datasets inferred from the 3d structures of antibody-protein complexes were defined. the second is a dataset of structures of antibody-protein complexes containing different structural epitopes. the first is a dataset of representative 3d structures of protein antigens with inferred structural epitopes. this work aims to establish a benchmark for 3d structure-based epitope prediction methods. using these experimental data computational methods exist for b-cell epitope prediction. the benchmark datasets are included as a supplement to this paper. the values of the area under the receiver operating characteristic curve for the evaluated methods were about for consurf, discotope, and ppi-pred methods and above but not exceeding for protein-protein docking methods when the best of the top ten models for the bound docking were considered; the remaining methods performed close to random.
 the ability to predict antibody binding sites for a given protein is a precursor to new vaccine design and diagnostics. among the various methods of b-cell epitope identification x-ray crystallography is one of the most reliable methods. using these experimental data computational methods exist for b-cell epitope prediction. as the number of structures of antibody-protein complexes grows, further interest in prediction methods using 3d structure is anticipated. this work aims to establish a benchmark for 3d structure-based epitope prediction methods.  two b-cell epitope benchmark datasets inferred from the 3d structures of antibody-protein complexes were defined. the first is a dataset of representative 3d structures of protein antigens with inferred structural epitopes. the second is a dataset of structures of antibody-protein complexes containing different structural epitopes. using these datasets, eight web-servers developed for antibody and protein binding sites prediction have been evaluated. in no method did performance exceed a 40% precision and 46% recall. the values of the area under the receiver operating characteristic curve for the evaluated methods were about for consurf, discotope, and ppi-pred methods and above but not exceeding for protein-protein docking methods when the best of the top ten models for the bound docking were considered; the remaining methods performed close to random. the benchmark datasets are included as a supplement to this paper.  it may be possible to improve epitope prediction methods through training on datasets which include only immune epitopes and through utilizing more features characterizing epitopes, for example, the evolutionary conservation score. notwithstanding, overall poor performance may reflect the generality of antigenicity and hence the inability to decipher b-cell epitopes as an intrinsic feature of the protein. it is an open question as to whether ultimately discriminatory features can be found. 