BACKGROUND
manual curation of information from the literature into electronic databases is of increasing importance in biomedical science.
prominent examples of databases incorporating curated data include the swiss-prot section of the universal protein resource knowledgebase   <cit> , the gene reference into function  system  <cit> , the mouse genome informatics database  <cit> , kegg  <cit> , dip  <cit>  and bind  <cit> .
these databases provide targeted query interfaces to access their data, which allow the performance of summary analysis that would be much harder or impossible to perform when relying on literature alone.
this study was motivated by the needs of the immune epitope database and analysis resource   <cit> .
the iedb catalogs molecular structures of immune epitopes recognized by the antigen specific receptors of t cells and b cells or binding to the major histocompatibility complex  molecules.
in addition, the iedb describes the biological context in which epitope recognition was recorded including information on the host organism, immunogen, antigen and assay.
together with the analytical tools hosted within it, the iedb provides resources for the development of epitope based techniques to detect, monitor, and prevent or treat diseases.
the iedb literature curation is a highly iterative and cooperative process  <cit> .
epitope-related references are first extracted from pubmed through complex queries consisting of multiple keywords and logical operators.
since the queries were designed to return all potentially relevant references, a substantial number of abstracts returned by the query are not actually in the scope of the iedb and need to be removed from the curation process.
examples for studies that are out of the iedb scope, even though their abstracts contain epitope-related keywords, include those describing a) epitope-tags for protein purification purposes, b) computational studies describing epitope predictions but not experiments, and c) studies which characterize immune responses without identifying the molecular structure of the targeted epitopes.
until recently, this abstract classification was performed manually by two experienced immunologists.
abstracts deemed to be outside of the iedb scope are discarded.
for abstracts within the scope, the full-text articles are retrieved and their epitope related content is entered into the database by a team of curators with expertise in biochemistry, microbiology and immunology.
the accuracy of the curation is reviewed by an independent panel of senior immunologists and structural biologists, which give final approval to make the data public on the iedb website.
manual literature curation is a resource and time-consuming process, which makes it highly desirable to automate any part of it.
one step that clearly lends itself for automation is determining which publication retrieved from pubmed is likely to contain relevant information.
this is referred to as text classification or relevance assessment and falls in the scientific domain of information retrieval  <cit> .
the most prominent platform to evaluate the performance of information retrieval techniques is the text retrieval conference  organized by the us national institute for standards and technology .
with the recent addition of trec's genomics track  <cit> , trec serves as a valuable resource to promote the advance and application of information retrieval techniques towards real world biomedical problems.
text classification is one of the simplest information retrieval procedures  <cit>  and can be directly applied to the iedb reference selection process.
early attempts of text classification were mainly carried out by constructing manually designed rules engineered from expert knowledge  <cit> .
with the increased availability of large amount of literature in digital format, machine learning methods became the dominant approaches.
a large body of literature has been published documenting the application of virtually every major machine learning algorithms in text classification  <cit> .
popular approaches include naïve bayes classifiers  <cit> , decision trees  <cit>  and support vector machines   <cit> .
automated text classification has been successfully applied to aid in biomedical database curation.
donaldson et al.
<cit>  used an svm model to distinguish abstracts containing information on protein-protein interaction before they were curated into the bind database.
in related work, a probabilistic latent categoriser  with kullback-leibler  divergence was used to re-rank pubmed references before they were curated into the swiss-prot database  <cit> .
miotto et al.
used artifical neural networks and classification trees to identify relevant abstracts containing allergen cross-reactivity information  <cit> .
in a more recent study, chen et al.
combined svm with a novel phrase-based clustering algorithm to classify papers about c. elegans  <cit> .
we here report the implementation of automated text classification via a naïve bayes classifier into the iedb curation process.
the naïve bayes classifier approach is a popular machine learning method for text classification because it is fast, easy to implement and performs well.
it has a long and distinguished record in text classification  <cit>  and it has been successfully applied to real world problems such as filtering spam emails  <cit> .
the classification algorithms were developed utilizing the large dataset of  <dig>  pubmed abstracts previously classified by experts.
several established text classification techniques were compared, and a naïve bayes classifier with information theory based feature selection performed best.
this classifier was further customized with domain-specific approaches to feature selection.
the final algorithm was put into practice classifying abstracts as "relevant", "irrelevant" and "uncertain", of which only the latter are further reviewed by an expert.
this greatly reduces the workload for the human expert without sacrificing classification performance.
RESULTS
overview of the dataset
the dataset used in our study are  <dig>  pubmed abstracts that have been manually evaluated by domain experts during the population of the iedb in the period from october  <dig> to october  <dig> .
the iedb is consecutively targeting different categories of epitope information based on different epitope sources: category a-c pathogens , emerging and re-emerging diseases , allergens, and epitope sources not in the previous three categories .
the abstracts were extracted from pubmed using complex keyword queries tailored for each category .
all abstracts are expert classified as either "relevant" or "irrelevant" to the scope of the iedb .
"relevant" abstracts are abstracts that contain epitope related information that can be extracted and curated into the iedb.
the majority of abstracts returned in the pubmed query are not relevant.
the relevant:irrelevant ratios range from 1: <dig> for ac pathogens to 1: <dig> for allergens and there are a total of  <dig>  relevant and  <dig>  irrelevant references.
of course, the human expert classifications are not always correct.
this is especially true as the abstract scanning is done very rapidly.
to estimate the inherent disagreement rate in this process, we compared the classifications made by two experts on the same dataset.
we found the rate of disagreement to be slightly less than 5%.
this inherent disagreement rate in our dataset limits the accuracy that can be achieved even from a perfect classification algorithm.
application of established text classification techniques
we first evaluated the performance of a number of standard text classification algorithms using their default implementation in the weka package  <cit> .
their performance was compared in 10-fold cross-validation  using area under receiver operating characteristic curves  as the performance metric.
the input features for classification are the raw words in each abstract.
we adopted a set of words approach and treat the features as binary attributes.
in other words, we take into account only the presence of a feature in an abstract, not the number of times it occurs.
all features were used except for "stop words", common words which occurred at identical frequency in relevant and irrelevant abstracts .
of the tested algorithms, the naïve bayes classifier performed best  for our dataset.
to increase our flexibility in modifying the algorithm and to avoid computer memory issues with the weka package in handling very large datasets, we implemented the classification and performance evaluation as a set of python scripts.
the classification performance was identical with that of the weka package.
the implemented classifier also has fast executing speed and can finish 10-fold cross-validation on the  <dig>  abstracts in about  <dig> minutes when tested on ubuntu linux system with a  <dig> ghz pentium  <dig> processor.
when the trained classifier is used to process new abstracts, it can classify  <dig>  abstracts in less than  <dig> seconds.
the features that can be extracted from pubmed include not only the words in the abstract and title, but also the authors, the journal and mesh headings.
we compared using all of these features with using only the words of the abstract.
the ten-fold cross-validation estimated performance improved from  <dig>  to  <dig>   which is significant with a p-value  <dig>  as determined by a paired t-test .
this improvement in classifier performance clearly demonstrates that the abstract contains most but not all of the information useful for reference classification.
while the t-test shows that the two methods produce different auc values, one can reasonably ask if a change of  <dig>  in auc has any practical relevance.
we can test the improvement by using the irrelevant abstracts in our dataset.
in practice we wish to remove as many as possible the irrelevant abstracts while retaining 95% of the relevant abstracts.
the classifier with auc of  <dig>  applied to our dataset of  <dig>  abstracts will identify  <dig>  false positive abstracts to achieve a 95% true positive rate, while the classifier with auc of  <dig>  will identify  <dig>  false positive abstracts.
this results in a net decrease of  <dig> false positive abstracts and is a performance improvement of 3%, which is a small but noticeable practical improvement.
the feature space in our dataset is now consisting of  <dig>  unique words.
to select features that are likely to be more relevant, we applied two well accepted feature selection methods: document frequency  and information gain  described in the methods section.
the performance curves of naïve bayes classifier after feature selection were plotted in figure  <dig>  both ig and df feature selection methods have similar effects on the classifier performance.
the best performances are achieved when around  <dig>  features are used.
both techniques permit removing up to 80% of the features while maintaining improved performance under cross validation.
applying the combined cutoff with df> <dig> and ig> <dig> e- <dig> selects  <dig>  features and increases the auc in ten-fold cross-validation from  <dig>  to  <dig>   .
while this improvement is not significant when applying the customary p <  <dig>  cutoff, it is a benefit in itself to remove features carrying little information or occurring rarely, leading to decreases in computation time and a reduction in the risk of over fitting.
an alternative approach to condense the feature space is to identify words by their stem, e.g.
reducing the words "binding, binds, bind" to their common stem "bind".
there are standard stemming algorithms designed to handle everyday english writing  <cit> , which are often applied for text classification including for bioinformatics.
when we applied the porter stemming algorithm instead of feature selection, we found that the classifier performance actually drops significantly from auc =  <dig>  to auc =  <dig>   .
this was in agreement with a previous study  <cit>  that suggests the standard porter stemmer may not be suitable to text classification for biomedical literature, as there is a large set of domain specific vocabulary which may be reduced to unsuitable stems.
we did not test alternative stemming algorithms, some of which are listed in the discussion section.
domain specific feature extraction
in order to reduce the dimensionality of the feature space while still capturing the essence of domain specific features, we introduced novel rules that specifically try to capture immune epitope related expressions, and group them together.
through this process, the information carried in individual expressions is combined and the information content is enriched which can lead to better performance.
the following four concepts were identified:
peptide sequences .
sequences of peptides are often included in abstracts describing epitopes, but make poor features for text classification as two sequences that are not identical letter by letter will be treated as separate features.
we identified peptide sequences as a) exclusively containing characters representing the twenty amino acids, b) in upper case c) length greater than seven and d) not one of the following words: "clinical", "material", "materials", "patients", "research" or "significance".
all identified peptide sequences are replaced with "~peptide~ ".
position ranges .
expressions that can indicate the location of an epitope in a protein sequence were identified and replaced with "~range<50~ " or "~range>50~ " depending on the length of the range specified.
as the iedb requires epitopes to be mapped to stretches of less than  <dig> amino acids, only ranges of less than  <dig> are good indicators of relevance.
mhc alleles  there are thousands of different mhc alleles described with different nomenclatures and numbering systems for different species.
we compiled a regular expression representing most mhc alleles from humans and mice and replaced them with "~mhc_allele~ ".
"x-mers" , which is a term referring to peptides of a specific length.
the x-mers were identified with regular expressions matching the following pattern: one or more digits followed by hyphen then followed by "mer", and were translated into "~#integer~-mer".
the addition of domain specific feature extraction consistently improved the performance of the classifier .
the naïve bayes classifier using all features from pubmed  and feature selection has an average auc of  <dig> .
incorporating domain specific feature extraction further improves its performance to an auc of  <dig> , which is highly significant .
classification of abstracts from a new sub-domain
as described above, the iedb consecutively curates abstracts from different sub-domains based on the source of the epitope.
this means that the cross-validation used here could overestimate the performance of the classifier whenever abstracts from a new sub-domain are being classified.
to test the performance of existing classifiers on newly acquired abstracts from a different sub-domain, we performed a series of tests.
in each of such test, we first trained a classifier based on three of the four available sets of abstracts.
we then tested the performance of classifiers learn on such sets on the fourth set of abstracts .
the tests demonstrated that naïve bayes classifiers learned from different categories of abstracts have significantly lower but still competitive performances and can achieve aucs in the range of  <dig>  to  <dig> .
these lower performance estimates have to be used when applying the classifier to a new sub-domain of articles, and were used to establish the cutoff values in the following section.
testing text classification in practice: the malaria abstracts
our goal for the classification is to remove as many irrelevant abstracts as possible while maintaining a false negative rate comparable to the human expert disagreement rate of 5%.
we also do not want to overload the curation queue with false positive abstracts, as placing articles in the queue and later removing them involves costs for e.g.
retrieving the full text manuscripts.
we therefore classify references in one of three categories: abstracts with very high predictive scores  predominantly relevant to the iedb and are directly placed into the curation pipeline.
abstracts with very low predictive scores  are predominantly irrelevant to the iedb and can be safely discarded.
those abstracts with intermediate predictive scores are manually classified by domain experts.
the thresholds determining what scores are very high and very low are chosen based on the performance in classifying abstracts from a new sub-domain described above.
the first test case of this scheme was the classification of abstracts from malaria epitopes, which constitutes a new sub-domain.
the initial pubmed query returned  <dig>  abstracts with malaria epitope specific keywords.
we classified the malaria abstracts using the naïve bayes classifier trained on all available abstracts and validated the predicted against manual classification results.
a roc plot and precision-recall graph of the classification performance is shown in figure  <dig>  there are  <dig> abstracts classified as "relevant" according to the cutoff determined from previous training.
a close examination shows that  <dig> out of those  <dig> abstracts are also classified as "relevant" by domain experts  which gives a positive prediction value of  <dig> %.
we next examined the  <dig> abstracts have been classified into the irrelevant category using cutoff determined from previous training .
a comparison shows that the negative predictive value for this cutoff is  <dig> % which is right on par with domain experts.
in summary, using classifiers built on existing abstracts, we were able to classify  <dig> % of newly acquired malaria abstracts with performance on par with human experts .
a close examination of miss-classified abstracts can reveal interesting insights.
for example, the abstract with pmid  <dig> was classified as relevant with a high score of  <dig>  while it is in fact irrelevant to the iedb.
the authors discuss the sequence and structure of mhc molecules of aotus monkeys and compare their peptide binding region with that of human mhc molecules.
however, no experimental data was generated.
this example shows the limitation of the independency assumption of the naïve bayes classifier and suggests that more sophisticated methods are required to classify such abstracts automatically.
discussion
we report here our implementation of text classification into the iedb curation process.
using cross-validation on existing data to evaluate classifier performances, we built a customized naïve bayes classifier to categorize if pubmed abstracts are within the scope of the iedb.
the final classification scheme was applied to a set of abstracts from a new sub-domain of epitopes, and successfully validated.
a number of lessons were learned that are generally applicable for similar projects and may not be immediately obvious.
first of all, we found classification performance was improved when using information beyond abstract and title of a study such as mesh terms and the names of authors conducting a study.
author names are not commonly included in text classification, but doing so makes sense as some scientists specialize in the application of methods, which makes their authorship of a paper indicative of its content.
potentially, there are many more sources of information linked to a manuscript which could be used as additional features for classifications.
for example, if an article is the primary reference for an entry in the pdb, the information contained in the pdb could be used as a source for features.
similarly, the references citing or cited by an article could be incorporated into the feature space.
secondly, we found that grouping together biomedical terms with the same meaning has a significant positive impact on classifier performance.
grouping not only reduced the dimensionality of the feature space but also created features with enriched information that contributed to better performance.
several of the features we extracted are likely to be relevant for other domains as well, e.g.
a project doing transplant research could benefit from our regular expressions used for identifying mhc alleles.
finally, we have implemented a hybrid categorization process using the automated classifier to pre-group abstracts into clearly relevant, clearly irrelevant, or uncertain.
the latter ones are then classified by a human expert.
this stream of expert classifications will be used to continuously update the classifier, which should result in improvements in performance that will further reduce the number of abstracts for which human classification is necessary.
there are a number of ways the classifier could be further improved.
for example, we only tested one word stemming algorithm, the porter stemmer.
the use of less stringent stemming algorithms  or a combination of stemming and part of speech tagging could help to improve classifier performance as this would tend to preserved domain specific terms and has been useful in information retrieval from pubmed  <cit> .
we could also incorporate techniques learned from information retrieval research in other domains  <cit> .
for example, potential improvements may be achieved by giving higher weight to documents having passages with high concentrations of high information content terms.
also, the high information content terms identified in this study could be applied to refine the construction of the initial pubmed queries.
many irrelevant abstracts are returned by our current queries; this could potentially be avoided if more discriminating search terms can be identified.
in addition to reporting what we believe to be valuable lessons learned, we also make the accompanying datasets of expert classified abstracts publicly available , which could be a valuable addition for existing resources benchmarking biomedical text classification.
the biocreative  provides several such benchmarks for information extraction and text mining  <cit> .
the pubmed abstracts annotated with mesh terms are another resource of expert classified abstracts.
comparing to benchmarks that are currently available, our abstracts offer a very large dataset for possible optimal training of classifiers to address a practical use-case.
it is critical for the development and evaluation of biomedical text mining and categorization tools to identify biologically significant problems and set up corresponding benchmarks for evaluation.
this will benefit the community just like what the casp evaluation has contributed to computational protein structure.
we want to strongly encourage others to utilize the iedb dataset as part of their benchmarking, and hope to learn from their experience to further improve our process.
CONCLUSIONS
in summary, we have successfully sped up the abstract selection process of iedb reference curation.
we achieved sensitivity and specificity comparable to that of the human expert classification on a subset of automatically classified abstracts by combining standard machine learning techniques and novel feature extraction method and using a hybrid machine/human classification scheme.
the insights learned from this study provide practical recommendations for users of text classification tools and the large dataset can serve as a benchmark to facilitate progress in tool development.

 manual curation of information from the literature into electronic databases is of increasing importance in biomedical science. prominent examples of databases incorporating curated data include the swiss-prot section of the universal protein resource knowledgebase, the gene reference into function system, the mouse genome informatics database, kegg, dip and bind. these databases provide targeted query interfaces to access their data, which allow the performance of summary analysis that would be much harder or impossible to perform when relying on literature alone. this study was motivated by the needs of the immune epitope database and analysis resource. the iedb catalogs molecular structures of immune epitopes recognized by the antigen specific receptors of t cells and b cells or binding to the major histocompatibility complex molecules. in addition, the iedb describes the biological context in which epitope recognition was recorded including information on the host organism, immunogen, antigen and assay. together with the analytical tools hosted within it, the iedb provides resources for the development of epitope based techniques to detect, monitor, and prevent or treat diseases. the iedb literature curation is a highly iterative and cooperative process. epitope-related references are first extracted from pubmed through complex queries consisting of multiple keywords and logical operators. since the queries were designed to return all potentially relevant references, a substantial number of abstracts returned by the query are not actually in the scope of the iedb and need to be removed from the curation process. examples for studies that are out of the iedb scope, even though their abstracts contain epitope-related keywords, include those describing a) epitope-tags for protein purification purposes, b) computational studies describing epitope predictions but not experiments, and c) studies which characterize immune responses without identifying the molecular structure of the targeted epitopes. until recently, this abstract classification was performed manually by two experienced immunologists. abstracts deemed to be outside of the iedb scope are discarded. for abstracts within the scope, the full-text articles are retrieved and their epitope related content is entered into the database by a team of curators with expertise in biochemistry, microbiology and immunology. the accuracy of the curation is reviewed by an independent panel of senior immunologists and structural biologists, which give final approval to make the data public on the iedb website. manual literature curation is a resource and time-consuming process, which makes it highly desirable to automate any part of it. one step that clearly lends itself for automation is determining which publication retrieved from pubmed is likely to contain relevant information. this is referred to as text classification or relevance assessment and falls in the scientific domain of information retrieval. the most prominent platform to evaluate the performance of information retrieval techniques is the text retrieval conference organized by the us national institute for standards and technology. with the recent addition of trec's genomics track, trec serves as a valuable resource to promote the advance and application of information retrieval techniques towards real world biomedical problems. text classification is one of the simplest information retrieval procedures and can be directly applied to the iedb reference selection process. early attempts of text classification were mainly carried out by constructing manually designed rules engineered from expert knowledge. with the increased availability of large amount of literature in digital format, machine learning methods became the dominant approaches. a large body of literature has been published documenting the application of virtually every major machine learning algorithms in text classification. popular approaches include naïve bayes classifiers, decision trees and support vector machines. automated text classification has been successfully applied to aid in biomedical database curation. donaldson et al. used an svm model to distinguish abstracts containing information on protein-protein interaction before they were curated into the bind database. in related work, a probabilistic latent categoriser with kullback-leibler divergence was used to re-rank pubmed references before they were curated into the swiss-prot database. miotto et al. used artifical neural networks and classification trees to identify relevant abstracts containing allergen cross-reactivity information. in a more recent study, chen et al. combined svm with a novel phrase-based clustering algorithm to classify papers about c. elegans. we here report the implementation of automated text classification via a naïve bayes classifier into the iedb curation process. the naïve bayes classifier approach is a popular machine learning method for text classification because it is fast, easy to implement and performs well. it has a long and distinguished record in text classification and it has been successfully applied to real world problems such as filtering spam emails. the classification algorithms were developed utilizing the large dataset of pubmed abstracts previously classified by experts. several established text classification techniques were compared, and a naïve bayes classifier with information theory based feature selection performed best. this classifier was further customized with domain-specific approaches to feature selection. the final algorithm was put into practice classifying abstracts as "relevant", "irrelevant" and "uncertain", of which only the latter are further reviewed by an expert. this greatly reduces the workload for the human expert without sacrificing classification performance.  overview of the dataset the dataset used in our study are pubmed abstracts that have been manually evaluated by domain experts during the population of the iedb in the period from october to october. the iedb is consecutively targeting different categories of epitope information based on different epitope sources: category a-c pathogens, emerging and re-emerging diseases, allergens, and epitope sources not in the previous three categories. the abstracts were extracted from pubmed using complex keyword queries tailored for each category. all abstracts are expert classified as either "relevant" or "irrelevant" to the scope of the iedb. "relevant" abstracts are abstracts that contain epitope related information that can be extracted and curated into the iedb. the majority of abstracts returned in the pubmed query are not relevant. the relevant:irrelevant ratios range from 1: for ac pathogens to 1: for allergens and there are a total of relevant and irrelevant references. of course, the human expert classifications are not always correct. this is especially true as the abstract scanning is done very rapidly. to estimate the inherent disagreement rate in this process, we compared the classifications made by two experts on the same dataset. we found the rate of disagreement to be slightly less than 5%. this inherent disagreement rate in our dataset limits the accuracy that can be achieved even from a perfect classification algorithm. application of established text classification techniques we first evaluated the performance of a number of standard text classification algorithms using their default implementation in the weka package. their performance was compared in 10-fold cross-validation using area under receiver operating characteristic curves as the performance metric. the input features for classification are the raw words in each abstract. we adopted a set of words approach and treat the features as binary attributes. in other words, we take into account only the presence of a feature in an abstract, not the number of times it occurs. all features were used except for "stop words", common words which occurred at identical frequency in relevant and irrelevant abstracts. of the tested algorithms, the naïve bayes classifier performed best for our dataset. to increase our flexibility in modifying the algorithm and to avoid computer memory issues with the weka package in handling very large datasets, we implemented the classification and performance evaluation as a set of python scripts. the classification performance was identical with that of the weka package. the implemented classifier also has fast executing speed and can finish 10-fold cross-validation on the abstracts in about minutes when tested on ubuntu linux system with a ghz pentium processor. when the trained classifier is used to process new abstracts, it can classify abstracts in less than seconds. the features that can be extracted from pubmed include not only the words in the abstract and title, but also the authors, the journal and mesh headings. we compared using all of these features with using only the words of the abstract. the ten-fold cross-validation estimated performance improved from to which is significant with a p-value as determined by a paired t-test. this improvement in classifier performance clearly demonstrates that the abstract contains most but not all of the information useful for reference classification. while the t-test shows that the two methods produce different auc values, one can reasonably ask if a change of in auc has any practical relevance. we can test the improvement by using the irrelevant abstracts in our dataset. in practice we wish to remove as many as possible the irrelevant abstracts while retaining 95% of the relevant abstracts. the classifier with auc of applied to our dataset of abstracts will identify false positive abstracts to achieve a 95% true positive rate, while the classifier with auc of will identify false positive abstracts. this results in a net decrease of false positive abstracts and is a performance improvement of 3%, which is a small but noticeable practical improvement. the feature space in our dataset is now consisting of unique words. to select features that are likely to be more relevant, we applied two well accepted feature selection methods: document frequency and information gain described in the methods section. the performance curves of naïve bayes classifier after feature selection were plotted in figure both ig and df feature selection methods have similar effects on the classifier performance. the best performances are achieved when around features are used. both techniques permit removing up to 80% of the features while maintaining improved performance under cross validation. applying the combined cutoff with df> and ig> e- selects features and increases the auc in ten-fold cross-validation from to. while this improvement is not significant when applying the customary p < cutoff, it is a benefit in itself to remove features carrying little information or occurring rarely, leading to decreases in computation time and a reduction in the risk of over fitting. an alternative approach to condense the feature space is to identify words by their stem, e.g. reducing the words "binding, binds, bind" to their common stem "bind". there are standard stemming algorithms designed to handle everyday english writing, which are often applied for text classification including for bioinformatics. when we applied the porter stemming algorithm instead of feature selection, we found that the classifier performance actually drops significantly from auc = to auc =. this was in agreement with a previous study that suggests the standard porter stemmer may not be suitable to text classification for biomedical literature, as there is a large set of domain specific vocabulary which may be reduced to unsuitable stems. we did not test alternative stemming algorithms, some of which are listed in the discussion section. domain specific feature extraction in order to reduce the dimensionality of the feature space while still capturing the essence of domain specific features, we introduced novel rules that specifically try to capture immune epitope related expressions, and group them together. through this process, the information carried in individual expressions is combined and the information content is enriched which can lead to better performance. the following four concepts were identified: peptide sequences. sequences of peptides are often included in abstracts describing epitopes, but make poor features for text classification as two sequences that are not identical letter by letter will be treated as separate features. we identified peptide sequences as a) exclusively containing characters representing the twenty amino acids, b) in upper case c) length greater than seven and d) not one of the following words: "clinical", "material", "materials", "patients", "research" or "significance". all identified peptide sequences are replaced with "~peptide~ ". position ranges. expressions that can indicate the location of an epitope in a protein sequence were identified and replaced with "~range<50~ " or "~range>50~ " depending on the length of the range specified. as the iedb requires epitopes to be mapped to stretches of less than amino acids, only ranges of less than are good indicators of relevance. mhc alleles there are thousands of different mhc alleles described with different nomenclatures and numbering systems for different species. we compiled a regular expression representing most mhc alleles from humans and mice and replaced them with "~mhc_allele~ ". "x-mers", which is a term referring to peptides of a specific length. the x-mers were identified with regular expressions matching the following pattern: one or more digits followed by hyphen then followed by "mer", and were translated into "~#integer~-mer". the addition of domain specific feature extraction consistently improved the performance of the classifier. the naïve bayes classifier using all features from pubmed and feature selection has an average auc of. incorporating domain specific feature extraction further improves its performance to an auc of, which is highly significant. classification of abstracts from a new sub-domain as described above, the iedb consecutively curates abstracts from different sub-domains based on the source of the epitope. this means that the cross-validation used here could overestimate the performance of the classifier whenever abstracts from a new sub-domain are being classified. to test the performance of existing classifiers on newly acquired abstracts from a different sub-domain, we performed a series of tests. in each of such test, we first trained a classifier based on three of the four available sets of abstracts. we then tested the performance of classifiers learn on such sets on the fourth set of abstracts. the tests demonstrated that naïve bayes classifiers learned from different categories of abstracts have significantly lower but still competitive performances and can achieve aucs in the range of to. these lower performance estimates have to be used when applying the classifier to a new sub-domain of articles, and were used to establish the cutoff values in the following section. testing text classification in practice: the malaria abstracts our goal for the classification is to remove as many irrelevant abstracts as possible while maintaining a false negative rate comparable to the human expert disagreement rate of 5%. we also do not want to overload the curation queue with false positive abstracts, as placing articles in the queue and later removing them involves costs for e.g. retrieving the full text manuscripts. we therefore classify references in one of three categories: abstracts with very high predictive scores predominantly relevant to the iedb and are directly placed into the curation pipeline. abstracts with very low predictive scores are predominantly irrelevant to the iedb and can be safely discarded. those abstracts with intermediate predictive scores are manually classified by domain experts. the thresholds determining what scores are very high and very low are chosen based on the performance in classifying abstracts from a new sub-domain described above. the first test case of this scheme was the classification of abstracts from malaria epitopes, which constitutes a new sub-domain. the initial pubmed query returned abstracts with malaria epitope specific keywords. we classified the malaria abstracts using the naïve bayes classifier trained on all available abstracts and validated the predicted against manual classification results. a roc plot and precision-recall graph of the classification performance is shown in figure there are abstracts classified as "relevant" according to the cutoff determined from previous training. a close examination shows that out of those abstracts are also classified as "relevant" by domain experts which gives a positive prediction value of %. we next examined the abstracts have been classified into the irrelevant category using cutoff determined from previous training. a comparison shows that the negative predictive value for this cutoff is % which is right on par with domain experts. in summary, using classifiers built on existing abstracts, we were able to classify % of newly acquired malaria abstracts with performance on par with human experts. a close examination of miss-classified abstracts can reveal interesting insights. for example, the abstract with pmid was classified as relevant with a high score of while it is in fact irrelevant to the iedb. the authors discuss the sequence and structure of mhc molecules of aotus monkeys and compare their peptide binding region with that of human mhc molecules. however, no experimental data was generated. this example shows the limitation of the independency assumption of the naïve bayes classifier and suggests that more sophisticated methods are required to classify such abstracts automatically. discussion we report here our implementation of text classification into the iedb curation process. using cross-validation on existing data to evaluate classifier performances, we built a customized naïve bayes classifier to categorize if pubmed abstracts are within the scope of the iedb. the final classification scheme was applied to a set of abstracts from a new sub-domain of epitopes, and successfully validated. a number of lessons were learned that are generally applicable for similar projects and may not be immediately obvious. first of all, we found classification performance was improved when using information beyond abstract and title of a study such as mesh terms and the names of authors conducting a study. author names are not commonly included in text classification, but doing so makes sense as some scientists specialize in the application of methods, which makes their authorship of a paper indicative of its content. potentially, there are many more sources of information linked to a manuscript which could be used as additional features for classifications. for example, if an article is the primary reference for an entry in the pdb, the information contained in the pdb could be used as a source for features. similarly, the references citing or cited by an article could be incorporated into the feature space. secondly, we found that grouping together biomedical terms with the same meaning has a significant positive impact on classifier performance. grouping not only reduced the dimensionality of the feature space but also created features with enriched information that contributed to better performance. several of the features we extracted are likely to be relevant for other domains as well, e.g. a project doing transplant research could benefit from our regular expressions used for identifying mhc alleles. finally, we have implemented a hybrid categorization process using the automated classifier to pre-group abstracts into clearly relevant, clearly irrelevant, or uncertain. the latter ones are then classified by a human expert. this stream of expert classifications will be used to continuously update the classifier, which should result in improvements in performance that will further reduce the number of abstracts for which human classification is necessary. there are a number of ways the classifier could be further improved. for example, we only tested one word stemming algorithm, the porter stemmer. the use of less stringent stemming algorithms or a combination of stemming and part of speech tagging could help to improve classifier performance as this would tend to preserved domain specific terms and has been useful in information retrieval from pubmed. we could also incorporate techniques learned from information retrieval research in other domains. for example, potential improvements may be achieved by giving higher weight to documents having passages with high concentrations of high information content terms. also, the high information content terms identified in this study could be applied to refine the construction of the initial pubmed queries. many irrelevant abstracts are returned by our current queries; this could potentially be avoided if more discriminating search terms can be identified. in addition to reporting what we believe to be valuable lessons learned, we also make the accompanying datasets of expert classified abstracts publicly available, which could be a valuable addition for existing resources benchmarking biomedical text classification. the biocreative provides several such benchmarks for information extraction and text mining. the pubmed abstracts annotated with mesh terms are another resource of expert classified abstracts. comparing to benchmarks that are currently available, our abstracts offer a very large dataset for possible optimal training of classifiers to address a practical use-case. it is critical for the development and evaluation of biomedical text mining and categorization tools to identify biologically significant problems and set up corresponding benchmarks for evaluation. this will benefit the community just like what the casp evaluation has contributed to computational protein structure. we want to strongly encourage others to utilize the iedb dataset as part of their benchmarking, and hope to learn from their experience to further improve our process.  in summary, we have successfully sped up the abstract selection process of iedb reference curation. we achieved sensitivity and specificity comparable to that of the human expert classification on a subset of automatically classified abstracts by combining standard machine learning techniques and novel feature extraction method and using a hybrid machine/human classification scheme. the insights learned from this study provide practical recommendations for users of text classification tools and the large dataset can serve as a benchmark to facilitate progress in tool development. 
we here report our experience in automating this process using naïve bayes classifiers trained on abstracts classified by domain experts. we have implemented the classifier into the curation process determining if abstracts are clearly relevant, clearly irrelevant, or if no certain classification can be made, in which case the abstracts are manually classified. this study provides both practical recommendations for users of text classification tools, as well as a large dataset which can serve as a benchmark for tool developers. like similar projects in other knowledge domains, significant effort is spent on identifying which articles are relevant for this purpose. by implementing text classification, we have sped up the reference selection process without sacrificing sensitivity or specificity of the human expert classification. the immune epitope database contains information on immune epitopes curated manually from the scientific literature.
 the immune epitope database contains information on immune epitopes curated manually from the scientific literature. like similar projects in other knowledge domains, significant effort is spent on identifying which articles are relevant for this purpose.  we here report our experience in automating this process using naïve bayes classifiers trained on abstracts classified by domain experts. improvements on the basic classifier performance were made by a) utilizing information stored in pubmed beyond the abstract itself b) applying standard feature selection criteria and c) extracting domain specific feature patterns that e.g. identify peptides sequences. we have implemented the classifier into the curation process determining if abstracts are clearly relevant, clearly irrelevant, or if no certain classification can be made, in which case the abstracts are manually classified. testing this classification scheme on an independent dataset, we achieve 95% sensitivity and specificity in the % of abstracts that were automatically classified.  by implementing text classification, we have sped up the reference selection process without sacrificing sensitivity or specificity of the human expert classification. this study provides both practical recommendations for users of text classification tools, as well as a large dataset which can serve as a benchmark for tool developers. 