BACKGROUND
databases are the cornerstone of bioinformatics analyses.
experimental methods keep advancing and high-throughput methods keep increasing in volume, the number of biological data repositories are growing rapidly  <cit> .
similarly, the quantity and complexity of the data are growing requiring both the refinement of analyses and higher resolution and accuracy of results.
in addition to the most commonly used biological data types such as sequence data , structural data, and quantitative data , the increasing amount of high-level functional annotations of biological sequences are needed to enable detailed studies of biological systems.
these high-level annotations are also captured in the databases, but to a much smaller degree than the essential data types.
the literature, however, is a rich source of functional annotation information, and combining these two types of sources provides a body of data, information, and knowledge needed for practical application in bioinformatics and clinical bioinformatics.
extraction of knowledge from these sources is facilitated through emerging knowledgebases  that enable not only data extraction, but also data mining, extraction of patterns hidden in the data, and predictive modeling.
thus, kb bring bioinformatics one step closer to the experimental setting compared to traditional databases since they are intended to enable summarization of hundreds of thousands of data points and in silico simulation of experiments all in one place.
a knowledge-based system  is a computational system that uses logic, statistics and artificial intelligence tools for support in decision making and solving complex problems.
the kbs include specialist databases designed for data mining tasks and knowledge management databases .
a kbs is a system comprising a kb, a set of analytical tools, a logic unit, and user interface.
the logic unit connects user queries and determines, using workflows, how analytical tools are applied to the knowledge base to perform the analysis and produce the results.
primary sources such as uniprot  <cit>  or genbank  <cit> , as well as specialized databases such as the influenza research database   <cit>  and the los alamos national laboratory hiv databases , offer a number of integrated tools and annotated data, but their analytical workflows are limited to basic operations.
examples of more advanced kbs include flavidb a kbs of flavivirus antigens,  <cit> , flukb a kbs of influenza antigens , and tantigen a kbs of tumor antigens .
kbs focus on a narrow domain, and a set of analytical tools to perform complex analyses and decision support.
kbs must contain sufficient data, and annotations to enable data mining for summarization, pattern discovery and building of models that simulate behavior of real systems.
for example flavidb, enables summarization of diversity of sequences for more than  <dig> species of flaviviruses.
it also enables the analysis of the complete set of predicted t cell epitopes for  <dig> common hla alleles and has the capacity to display the complete landscape of both predicted and experimentally verified hla associated peptides.
the extension of antigen analysis functionalities with flukb enables analysis of cross-reactivity of all entries for neutralizing antibodies.
both these examples focus on identification, prediction, variability analysis and cross-reactivity of immune epitopes.
the implementation of workflows in these kbs enables complex analyses to be performed by filling a single query form and results are presented in a single report.
to get high quality results, we must ensure that kbs are up to date and error-free .
since the information in kbs is derived from multiple sources, providing high quality updates is complex.
manual updating of kbs is impractical, so automation of the updating process is needed.
automated updating of data and annotation by extracting data from primary databases such as uniprot, genbank, or iedb is relatively simple since these sources enable export of data using standardized formats, mainly xml.
ideally, functional annotations will be deposited by direct submission to appropriate databases by the discoverers, but a historical lack of submission standards for higher-level biological data, has lead to the vast majority of this information being recording only in primary scientific literature.
the use of data embedded in primary scientific literature accessible through pubmed or google scholar, is markedly more complex.
the information stored in abstracts or full texts is, at best, semi-structured, but typically it is provided as free text.
given that as many as tens of thousands of articles may be published each year on a given topic, access to this information and assessment of its relevance require efficient methods for identification of publications of interest and rapid assessment of their suitability for inclusion in the kbs.
such analysis is facilitated through use of text mining techniques, ranging from simple statistical pattern learning based on term frequencies, to complex natural language processing techniques in order to produce text categorization, document summarization, information retrieval, and ultimately the data mining  <cit> .
a long-term solution for this issue invariably involves standardizing submission and storage of complex biological data, but the knowledge currently embedded in the literature remains available for extraction.
text mining operations have previously been applied for specific knowledge extraction for vaccine development  <cit> , as well as document classification for separation of abstracts by topic  <cit>  and for semi-automated extraction of allergen cross-reactivity information  <cit> .
in this article, we will define the conceptual framework for semi-automated updating of our tumor antigen knowledgebase, tantigen, using data parsing, basic text mining operations, and a standardized submission system.
RESULTS
conceptual framework
depending on the content of the kbs one wishes to update, there are issues pertaining to the complexity of biological data that require considerations.
particularly we must address the diversity of data types, diversity of data formats, dispersion of data across different sources, and size of data sets.
there are many biological data types - the most common include sequence data , molecular structures, expression data, and functional annotations.
data can be stored and retrieved either as structured text, table formats, semantic web formats , or non-structured text.
depending on the target data format, retrieval can be performed by direct extraction, parsing, text mining, or manual extraction.
text mining, manual extraction, or a combination of these two is common in extracting the high-level data, such as functional annotations.
data availability and individual entry size vary between different data types, presenting a computational challenge in terms of retrieval, handling, analysis, and storage.
additional factors that affect the complexity of the updating task are data heterogeneity, integration of multiple data types after retrieval, as well as provenance tracking for quality assessment  <cit> .
to address these issues we have formalized a number of common tasks pertaining to knowledgebase updating into a conceptual framework for updating biological kbs, shown in figure  <dig>
step 1: produce status report of current knowledgebase build.
this report will serve as the filter for the two main updating tasks: update of existing entries and update of data body by introduction of new entries.
step 2: automatic download of data from selected sources.
most biological data repositories enable full download of latest database build and most allow automated retrieval via gnu wget or ftp clients.
if automatic download is not possible, this step can be performed manually.
step 3: automatic data pre-processing.
depending on the data format, pre-processing steps can be automated in various ways.
for simple syntax-based formats such as xml, parsing of desired data is possible, where for non-standardized formats, such as raw text, pre-processing involves tasks derived from text mining, such as word stemming, stop word removal, and generation of document-term matrix   <cit> .
step 4: text categorization.
if the desired information is not available in a standardized format - for example that it is only available in primary scientific literature, the text mining or machine learning methods can be applied to direct and streamline the manual extraction.
a text corpus may contain documents that fall into two or more categories, of which only one or a few are of interest for a given task.
to maximize the efficiency of manual data extraction, it is helpful to classify documents before embarking on data extraction.
options for classification using machine learning methods include: unsupervised methods such as clustering and blind signal separation, or supervised methods such as artificial neural networks, support vector machines, nearest neighbor methods, naive bayes, decision trees, among others  <cit> .
for some of these algorithms, feature extraction using matrix factorization methods, such as principal component analysis  can be useful to reduce dimensionality of dtm, which can become quite large.
step 5: manually extract data and information from categorized texts.
some higher-level data types, such as functional annotations, are often found in tables, figures, legends, or supplementary materials of primary scientific articles, making automated extraction of this information highly complex or practically impossible  <cit> .
a manual extraction step may therefore be needed and simultaneously allow for quality control.
step 6: submission of new or updated entries to the kbs.
submission of extracted data to the kbs should be standardized to the highest degree possible in order to ensure the adherence to standardized format and quality of an entry.
the use of a standardized submission form allows non-experts to perform the task of updating.
automated extraction of related data from primary databases can minimize the manual entry of data and mismatches between existing entries addition to entries, provide automated error detection to be manually addressed.
step 7: refining categorization by increasing the training corpus.
each manually inspected document  represents a new addition to the training data used for documentation categorization.
in addition to refining the model and improving performance, a feedback loop to the classification module reduces the need for a large initial training corpus.
case study: tantigen tumor t cell antigen database
selection of useful tumor t cell antigens represents a major bottleneck to the study and design of cancer immunotherapies.
the methods of selecting immunotherapy targets involve the selection of antigens and the analysis of their immune epitopes.
this process has been greatly enhanced by the use of computational immunology methods  <cit> .
however, as computational efforts produce vast amounts of potential targets, the bottleneck is shifted to the wet lab, where the vaccine target candidates must be validated for both relevance and immunogenicity before they are included in potential vaccine constructs.
great advances have been made in techniques for high-throughput epitope validation  <cit> , but as computational methods grow ever more powerful, so does the need for post-analysis verification of results.
efficient cataloguing of experimentally validated epitopes for cross-referencing of new predictions with past experimental data is a valuable resource that could reduce the need for and streamline further experimentation.
several specialized resources for this and similar purposes have been established, for example: ird  <cit> , the hiv databases , human papillomavirus t cell antigen database for hpv , as well as general hla binder repositories such as syfpeithi  <cit>  and the immune epitope database   <cit> .
the tantigen database was established in  <dig> as a tumor-specific t cell antigen database.
it provides the scientific community with a curated repository of experimentally validated tumor t-cell antigens, and matched t-cell epitopes and hla binders.
each antigen entry contains detailed information about somatic mutations from the catalogue of somatic mutations in cancer   <cit> , splice isoforms from uniprot/swiss-prot, gene expression profiles from unigene, and known t-cell epitopes from secondary databases or literature.
additionally, tantigen is equipped with a number of analysis tools such as blast search  <cit> , multiple sequence alignment using mafft  <cit> , t-cell epitope/hla ligand prediction  <cit>  and visualization, and tumor antigen classification  <cit> .
updating tantigen
keeping up-to-date data in a kbs represents a major bottleneck in the maintenance of tantigen.
in  <dig>   <dig>  articles responding to the keywords "tumor antigen" were indexed in pubmed.
although many of these articles may not contain tumor t cell antigens, the growing quantities of literature represents a major bottleneck in the maintenance of curated databases  <cit> .
the data types to be updated in tantigen are experimentally characterized t cell epitopes and hla ligands, and expression and variability information for the proteins that harbor them.
in build  <dig> of tantigen, these data were collected from six different sources: manual collection from the literature, the peptide database: t cell-defined tumor antigens , the listing of human tumor antigens recognized by t cells by parmiani and colleagues  <cit> , and parsing from iedb, as well as four other public databases that are outdated or unavailable at present.
the primary resource for these data remains manual collection from the literature, as no primary database is actively collecting or curating tumor antigen data.
iedb offers some curated cancer data , but in their february  <dig> newsletter they announced that they will no longer curate cancer tumor epitope data.
preliminary filtering of literature
a simple keyword search for the terms "cancer or tumor or antigen or epitope" in pubmed, yielded > <dig>  results .
when keyword stringency increased the number of useful publications decreased to a workable level .
for this task, we decided to use the search term " and ", which yields  <dig>  hits in pubmed.
keyword search terms could be further expanded or refined, by reiterating either manually or using feature extraction of discriminative terms using machine-learning methods.
manually sorting of these articles is extremely laborious task.
pubmed is currently growing at approximately 4% per year  <cit> , so the issue will only increase.
it is therefore advantageous to automate the classification of publication content before manually extracting relevant information.
for this task, we employed an adapted version of the conceptual framework to update tantigen.
formal approach to updating
step 1: status report of tantigen build  <dig>  the status report for tantigen lists  <dig> unique proteins and corresponding uniprot accession numbers.
many of these proteins have multiple splice isoforms for which uniprot accession numbers are also listed.
all uniprot accession numbers are listed as these entries are subject to updating by direct parsing from uniprot data downloads.
similarly, pubmed ids are listed for all referenced articles.
these articles represent relevant literature and corresponding abstracts can be directly parsed from the pubmed abstract download to the training document set.
the build  <dig> of tantigen has  <dig>  curated antigen entries.
step 2: automatic data download.
the latest versions of uniprot and cosmic are downloadable as xml files from the database web sites.
pubmed results can be narrowed down by search term, in this case we used " and ", but this can be refined in later iterations if suitable.
due to the very high volume of abstracts in pubmed, query results can also be filtered by date, and we here filtered out articles published before the last tantigen update.
search results are downloadable in xml format.
step 3: automatic data pre-processing.
the cosmic and uniprot xml downloads needed no further pre-processing for parsing.
the pubmed abstracts were extracted from the xml and parsed into a text corpus format for pre-processing.
the following tasks were performed on the corpus: lower case transformation, removal of stop words, removal of general punctuation, word stemming, and white space stripping.
the numbers are usually removed in text mining preprocessing, but it was not done here because we needed to preserve the terms defining hla alleles, cd receptors, and other immunologically relevant descriptors.
step 4: abstract categorization.
the resulting dtm was tf-idf transformed, and each abstract was classified using a k-nearest neighbor  classifier trained on  <dig> manually pre-classified abstracts.
iterative refinement of the algorithm showed that a six nearest neighbors model yielded the best results.
each abstract in the corpus was given a probability score based on the ratio of relevant neighbors in the model.
the output list was ordered from most probable to least, thus eliminating the need to define a static threshold.
step 5: manually extract antigen data from literature.
the articles corresponding to each abstract classified as relevant were accessed through pubmed or publishing journal.
epitopes, hla ligands and related data, such as hla restriction and protein of origin, were extracted.
for tantigen build  <dig>  we manually searched the top  <dig> articles out of classified  <dig>  articles.
the cutoff of  <dig> articles was chosen when article relevance started decreasing drastically in the ordered list during manual data extraction.
step 6: submission of data.
submission was done by filling out a standardized tantigen submission form for each antigen.
additional information was parsed directly from the downloaded uniprot xml, based on the protein of origin.
similarly, mutation entries and splice variants were automatically linked by cross-referencing with cosmic xml.
entries in tantigen were automatically linked to each other where applicable .
updating of existing entries was performed by automated parsing form uniprot xml, as some entries were removed, assigned new accession, updated with more splice isoforms.
this step also serves as a error detection: if an existing entry in tantigen does not match the information entered in the standardized submission form, the user is notified and prompted to determine whether the existing entry, the submission, or both are erroneous.
similarly, if protein information extracted from uniprot does not match that in cosmic, the user will be prompted to resolve the issue, thus increasing data quality.
step 7: refine training set with new entries.
the tantigen submission form has an addition field, where the curator performing the manual submission is prompted to classify the article as "relevant" or "irrelevant".
this feature was used to feed manually inspected abstracts back into the training corpus, to increase its size and thus performance.
the false positives and false negatives were fed back, but only a randomly selected fraction of true positives and true negatives were fed back into the training corpus, as these may further bias a potentially already biased model.
results of tantigen update
accuracy of classification
the average accuracy in the five-fold cross-validation training of the k-nn model with  <dig> nearest neighbors was  <dig>  with sensitivity of  <dig>  and specificity of  <dig> .
model performance is likely to increase with the increase of training set size, and particularly the addition of false positives from the manual extraction step.
true positive should also be added to the training corpus, but including all true positives may further bias a potentially biased model.
special care should be taken in initial classification rounds to extract and include false negatives, as low sensitivity is highly detrimental to the quality and completeness of the update.
wrongfully discarding relevant literature will not only lead to, potentially permanent, loss of valuable data, but also negatively affect classifier performance, when misclassified training data is fed back into the model.
results of manual extraction of tumor t-cell antigens
manual extraction of new antigenic proteins and tumor t-cell antigens was performed from the classified literature.
since classification was based on the six nearest neighbors, the body of classified abstracts was divided in seven groups, corresponding to whether an abstract had from zero to six relevant neighbors in the training set.
out of the  <dig>  classified abstracts,  <dig> had six relevant neighbors,  <dig> had five,  <dig> had four,  <dig> had three,  <dig>  had two,  <dig>  had one, and  <dig>  abstracts had zero relevant neighbors.
we manually examined the top  <dig> scoring papers in which we found  <dig> new antigenic proteins harboring  <dig> new tumor t-cell epitopes.
additionally, we found more than  <dig> new t-cell epitopes discovered in proteins already recorded as tumor antigens in tantigen.
training set refinement iteratively increase classification accuracy
the performance of the document classification model is expected to gradually increase as the size of the training corpus is increased with each database update.
learning curves for accuracy, sensitivity, and specificity constructed by gradually increasing the training corpus for a test corpus fixed to  <dig> abstracts  supports this notion .
although the sensitivity and specificity show some fluctuations, accuracy is observed to steadily increase as the training set size is increased in increments of  <dig> abstracts .
the learning curves will likely plateau with the addition of further training abstracts, although any increase in sensitivity will add to data completeness, and increased specificity will minimize labor intensity.
abstract category signatures
the dtm of the training corpus contains more than  <dig>  terms.
most are very rare terms present in only one or a few abstracts, and have very little influence on abstract classification as corresponding to either relevant or irrelevant articles.
rare terms can be removed by setting a sparsity threshold if dtm dimensions become too large.
examining the top ten terms, most discriminative between abstracts of relevant and irrelevant articles , show a distinct signature and reveal particular emphasis on such terms as "immunotherapy", "epitope", "t cell", and "ctl" .
these terms are likely the main drivers of classification and may very well be sufficient to support the main task of classification.
notable is the fact that all discriminating terms are predominant in relevant abstracts, which may explain that sensitivity of classification is higher than specificity.
this is most likely due to the highly specific nature of the relevant abstracts, whereas irrelevant abstracts are a much broader class.
however, these terms are still represented in the corpus of irrelevant literature, so a machine learning approach to classification is highly likely to outperform a simple keyword search.
CONCLUSIONS
specialized biological databases are gradually moving from data repositories towards knowledge-based systems.
enriching basic biological data with higher-level functional annotations and facilitating specialized analyses in organized workflows enables extraction of higher-level knowledge.
currently, however, functional annotations are primarily stored in the literature, rather than in standardized formats of primary biological databases.
as the quantity of this information increases, easy access to multiple layers of biological data and information enables improved extraction of knowledge, thus increasing the value to the user.
we here present a conceptual framework for automating the process of updating biological databases and knowledgebases with standardized non-standardized data from both primary and secondary data repositories, as well as literature.
we deployed a text mining-based approach to categorize literature, based on defining term signatures of freely available article abstracts, which enable significantly faster manual extraction of relevant data.
we have applied this conceptual framework to literature for updating the tantigen kbs of tumor t cell antigens.
training of a k-nn classifier on  <dig> abstracts yielded classification accuracy of  <dig> , thus showing significant value in support of data extraction from the literature.

 databases are the cornerstone of bioinformatics analyses. experimental methods keep advancing and high-throughput methods keep increasing in volume, the number of biological data repositories are growing rapidly. similarly, the quantity and complexity of the data are growing requiring both the refinement of analyses and higher resolution and accuracy of results. in addition to the most commonly used biological data types such as sequence data, structural data, and quantitative data, the increasing amount of high-level functional annotations of biological sequences are needed to enable detailed studies of biological systems. these high-level annotations are also captured in the databases, but to a much smaller degree than the essential data types. the literature, however, is a rich source of functional annotation information, and combining these two types of sources provides a body of data, information, and knowledge needed for practical application in bioinformatics and clinical bioinformatics. extraction of knowledge from these sources is facilitated through emerging knowledgebases that enable not only data extraction, but also data mining, extraction of patterns hidden in the data, and predictive modeling. thus, kb bring bioinformatics one step closer to the experimental setting compared to traditional databases since they are intended to enable summarization of hundreds of thousands of data points and in silico simulation of experiments all in one place. a knowledge-based system is a computational system that uses logic, statistics and artificial intelligence tools for support in decision making and solving complex problems. the kbs include specialist databases designed for data mining tasks and knowledge management databases. a kbs is a system comprising a kb, a set of analytical tools, a logic unit, and user interface. the logic unit connects user queries and determines, using workflows, how analytical tools are applied to the knowledge base to perform the analysis and produce the results. primary sources such as uniprot or genbank, as well as specialized databases such as the influenza research database and the los alamos national laboratory hiv databases, offer a number of integrated tools and annotated data, but their analytical workflows are limited to basic operations. examples of more advanced kbs include flavidb a kbs of flavivirus antigens,, flukb a kbs of influenza antigens, and tantigen a kbs of tumor antigens. kbs focus on a narrow domain, and a set of analytical tools to perform complex analyses and decision support. kbs must contain sufficient data, and annotations to enable data mining for summarization, pattern discovery and building of models that simulate behavior of real systems. for example flavidb, enables summarization of diversity of sequences for more than species of flaviviruses. it also enables the analysis of the complete set of predicted t cell epitopes for common hla alleles and has the capacity to display the complete landscape of both predicted and experimentally verified hla associated peptides. the extension of antigen analysis functionalities with flukb enables analysis of cross-reactivity of all entries for neutralizing antibodies. both these examples focus on identification, prediction, variability analysis and cross-reactivity of immune epitopes. the implementation of workflows in these kbs enables complex analyses to be performed by filling a single query form and results are presented in a single report. to get high quality results, we must ensure that kbs are up to date and error-free. since the information in kbs is derived from multiple sources, providing high quality updates is complex. manual updating of kbs is impractical, so automation of the updating process is needed. automated updating of data and annotation by extracting data from primary databases such as uniprot, genbank, or iedb is relatively simple since these sources enable export of data using standardized formats, mainly xml. ideally, functional annotations will be deposited by direct submission to appropriate databases by the discoverers, but a historical lack of submission standards for higher-level biological data, has lead to the vast majority of this information being recording only in primary scientific literature. the use of data embedded in primary scientific literature accessible through pubmed or google scholar, is markedly more complex. the information stored in abstracts or full texts is, at best, semi-structured, but typically it is provided as free text. given that as many as tens of thousands of articles may be published each year on a given topic, access to this information and assessment of its relevance require efficient methods for identification of publications of interest and rapid assessment of their suitability for inclusion in the kbs. such analysis is facilitated through use of text mining techniques, ranging from simple statistical pattern learning based on term frequencies, to complex natural language processing techniques in order to produce text categorization, document summarization, information retrieval, and ultimately the data mining. a long-term solution for this issue invariably involves standardizing submission and storage of complex biological data, but the knowledge currently embedded in the literature remains available for extraction. text mining operations have previously been applied for specific knowledge extraction for vaccine development, as well as document classification for separation of abstracts by topic and for semi-automated extraction of allergen cross-reactivity information. in this article, we will define the conceptual framework for semi-automated updating of our tumor antigen knowledgebase, tantigen, using data parsing, basic text mining operations, and a standardized submission system.  conceptual framework depending on the content of the kbs one wishes to update, there are issues pertaining to the complexity of biological data that require considerations. particularly we must address the diversity of data types, diversity of data formats, dispersion of data across different sources, and size of data sets. there are many biological data types - the most common include sequence data, molecular structures, expression data, and functional annotations. data can be stored and retrieved either as structured text, table formats, semantic web formats, or non-structured text. depending on the target data format, retrieval can be performed by direct extraction, parsing, text mining, or manual extraction. text mining, manual extraction, or a combination of these two is common in extracting the high-level data, such as functional annotations. data availability and individual entry size vary between different data types, presenting a computational challenge in terms of retrieval, handling, analysis, and storage. additional factors that affect the complexity of the updating task are data heterogeneity, integration of multiple data types after retrieval, as well as provenance tracking for quality assessment. to address these issues we have formalized a number of common tasks pertaining to knowledgebase updating into a conceptual framework for updating biological kbs, shown in figure step 1: produce status report of current knowledgebase build. this report will serve as the filter for the two main updating tasks: update of existing entries and update of data body by introduction of new entries. step 2: automatic download of data from selected sources. most biological data repositories enable full download of latest database build and most allow automated retrieval via gnu wget or ftp clients. if automatic download is not possible, this step can be performed manually. step 3: automatic data pre-processing. depending on the data format, pre-processing steps can be automated in various ways. for simple syntax-based formats such as xml, parsing of desired data is possible, where for non-standardized formats, such as raw text, pre-processing involves tasks derived from text mining, such as word stemming, stop word removal, and generation of document-term matrix. step 4: text categorization. if the desired information is not available in a standardized format - for example that it is only available in primary scientific literature, the text mining or machine learning methods can be applied to direct and streamline the manual extraction. a text corpus may contain documents that fall into two or more categories, of which only one or a few are of interest for a given task. to maximize the efficiency of manual data extraction, it is helpful to classify documents before embarking on data extraction. options for classification using machine learning methods include: unsupervised methods such as clustering and blind signal separation, or supervised methods such as artificial neural networks, support vector machines, nearest neighbor methods, naive bayes, decision trees, among others. for some of these algorithms, feature extraction using matrix factorization methods, such as principal component analysis can be useful to reduce dimensionality of dtm, which can become quite large. step 5: manually extract data and information from categorized texts. some higher-level data types, such as functional annotations, are often found in tables, figures, legends, or supplementary materials of primary scientific articles, making automated extraction of this information highly complex or practically impossible. a manual extraction step may therefore be needed and simultaneously allow for quality control. step 6: submission of new or updated entries to the kbs. submission of extracted data to the kbs should be standardized to the highest degree possible in order to ensure the adherence to standardized format and quality of an entry. the use of a standardized submission form allows non-experts to perform the task of updating. automated extraction of related data from primary databases can minimize the manual entry of data and mismatches between existing entries addition to entries, provide automated error detection to be manually addressed. step 7: refining categorization by increasing the training corpus. each manually inspected document represents a new addition to the training data used for documentation categorization. in addition to refining the model and improving performance, a feedback loop to the classification module reduces the need for a large initial training corpus. case study: tantigen tumor t cell antigen database selection of useful tumor t cell antigens represents a major bottleneck to the study and design of cancer immunotherapies. the methods of selecting immunotherapy targets involve the selection of antigens and the analysis of their immune epitopes. this process has been greatly enhanced by the use of computational immunology methods. however, as computational efforts produce vast amounts of potential targets, the bottleneck is shifted to the wet lab, where the vaccine target candidates must be validated for both relevance and immunogenicity before they are included in potential vaccine constructs. great advances have been made in techniques for high-throughput epitope validation, but as computational methods grow ever more powerful, so does the need for post-analysis verification of results. efficient cataloguing of experimentally validated epitopes for cross-referencing of new predictions with past experimental data is a valuable resource that could reduce the need for and streamline further experimentation. several specialized resources for this and similar purposes have been established, for example: ird, the hiv databases, human papillomavirus t cell antigen database for hpv, as well as general hla binder repositories such as syfpeithi and the immune epitope database. the tantigen database was established in as a tumor-specific t cell antigen database. it provides the scientific community with a curated repository of experimentally validated tumor t-cell antigens, and matched t-cell epitopes and hla binders. each antigen entry contains detailed information about somatic mutations from the catalogue of somatic mutations in cancer, splice isoforms from uniprot/swiss-prot, gene expression profiles from unigene, and known t-cell epitopes from secondary databases or literature. additionally, tantigen is equipped with a number of analysis tools such as blast search, multiple sequence alignment using mafft, t-cell epitope/hla ligand prediction and visualization, and tumor antigen classification. updating tantigen keeping up-to-date data in a kbs represents a major bottleneck in the maintenance of tantigen. in articles responding to the keywords "tumor antigen" were indexed in pubmed. although many of these articles may not contain tumor t cell antigens, the growing quantities of literature represents a major bottleneck in the maintenance of curated databases. the data types to be updated in tantigen are experimentally characterized t cell epitopes and hla ligands, and expression and variability information for the proteins that harbor them. in build of tantigen, these data were collected from six different sources: manual collection from the literature, the peptide database: t cell-defined tumor antigens, the listing of human tumor antigens recognized by t cells by parmiani and colleagues, and parsing from iedb, as well as four other public databases that are outdated or unavailable at present. the primary resource for these data remains manual collection from the literature, as no primary database is actively collecting or curating tumor antigen data. iedb offers some curated cancer data, but in their february newsletter they announced that they will no longer curate cancer tumor epitope data. preliminary filtering of literature a simple keyword search for the terms "cancer or tumor or antigen or epitope" in pubmed, yielded > results. when keyword stringency increased the number of useful publications decreased to a workable level. for this task, we decided to use the search term " and ", which yields hits in pubmed. keyword search terms could be further expanded or refined, by reiterating either manually or using feature extraction of discriminative terms using machine-learning methods. manually sorting of these articles is extremely laborious task. pubmed is currently growing at approximately 4% per year, so the issue will only increase. it is therefore advantageous to automate the classification of publication content before manually extracting relevant information. for this task, we employed an adapted version of the conceptual framework to update tantigen. formal approach to updating step 1: status report of tantigen build the status report for tantigen lists unique proteins and corresponding uniprot accession numbers. many of these proteins have multiple splice isoforms for which uniprot accession numbers are also listed. all uniprot accession numbers are listed as these entries are subject to updating by direct parsing from uniprot data downloads. similarly, pubmed ids are listed for all referenced articles. these articles represent relevant literature and corresponding abstracts can be directly parsed from the pubmed abstract download to the training document set. the build of tantigen has curated antigen entries. step 2: automatic data download. the latest versions of uniprot and cosmic are downloadable as xml files from the database web sites. pubmed results can be narrowed down by search term, in this case we used " and ", but this can be refined in later iterations if suitable. due to the very high volume of abstracts in pubmed, query results can also be filtered by date, and we here filtered out articles published before the last tantigen update. search results are downloadable in xml format. step 3: automatic data pre-processing. the cosmic and uniprot xml downloads needed no further pre-processing for parsing. the pubmed abstracts were extracted from the xml and parsed into a text corpus format for pre-processing. the following tasks were performed on the corpus: lower case transformation, removal of stop words, removal of general punctuation, word stemming, and white space stripping. the numbers are usually removed in text mining preprocessing, but it was not done here because we needed to preserve the terms defining hla alleles, cd receptors, and other immunologically relevant descriptors. step 4: abstract categorization. the resulting dtm was tf-idf transformed, and each abstract was classified using a k-nearest neighbor classifier trained on manually pre-classified abstracts. iterative refinement of the algorithm showed that a six nearest neighbors model yielded the best results. each abstract in the corpus was given a probability score based on the ratio of relevant neighbors in the model. the output list was ordered from most probable to least, thus eliminating the need to define a static threshold. step 5: manually extract antigen data from literature. the articles corresponding to each abstract classified as relevant were accessed through pubmed or publishing journal. epitopes, hla ligands and related data, such as hla restriction and protein of origin, were extracted. for tantigen build we manually searched the top articles out of classified articles. the cutoff of articles was chosen when article relevance started decreasing drastically in the ordered list during manual data extraction. step 6: submission of data. submission was done by filling out a standardized tantigen submission form for each antigen. additional information was parsed directly from the downloaded uniprot xml, based on the protein of origin. similarly, mutation entries and splice variants were automatically linked by cross-referencing with cosmic xml. entries in tantigen were automatically linked to each other where applicable. updating of existing entries was performed by automated parsing form uniprot xml, as some entries were removed, assigned new accession, updated with more splice isoforms. this step also serves as a error detection: if an existing entry in tantigen does not match the information entered in the standardized submission form, the user is notified and prompted to determine whether the existing entry, the submission, or both are erroneous. similarly, if protein information extracted from uniprot does not match that in cosmic, the user will be prompted to resolve the issue, thus increasing data quality. step 7: refine training set with new entries. the tantigen submission form has an addition field, where the curator performing the manual submission is prompted to classify the article as "relevant" or "irrelevant". this feature was used to feed manually inspected abstracts back into the training corpus, to increase its size and thus performance. the false positives and false negatives were fed back, but only a randomly selected fraction of true positives and true negatives were fed back into the training corpus, as these may further bias a potentially already biased model. results of tantigen update accuracy of classification the average accuracy in the five-fold cross-validation training of the k-nn model with nearest neighbors was with sensitivity of and specificity of. model performance is likely to increase with the increase of training set size, and particularly the addition of false positives from the manual extraction step. true positive should also be added to the training corpus, but including all true positives may further bias a potentially biased model. special care should be taken in initial classification rounds to extract and include false negatives, as low sensitivity is highly detrimental to the quality and completeness of the update. wrongfully discarding relevant literature will not only lead to, potentially permanent, loss of valuable data, but also negatively affect classifier performance, when misclassified training data is fed back into the model. results of manual extraction of tumor t-cell antigens manual extraction of new antigenic proteins and tumor t-cell antigens was performed from the classified literature. since classification was based on the six nearest neighbors, the body of classified abstracts was divided in seven groups, corresponding to whether an abstract had from zero to six relevant neighbors in the training set. out of the classified abstracts, had six relevant neighbors, had five, had four, had three, had two, had one, and abstracts had zero relevant neighbors. we manually examined the top scoring papers in which we found new antigenic proteins harboring new tumor t-cell epitopes. additionally, we found more than new t-cell epitopes discovered in proteins already recorded as tumor antigens in tantigen. training set refinement iteratively increase classification accuracy the performance of the document classification model is expected to gradually increase as the size of the training corpus is increased with each database update. learning curves for accuracy, sensitivity, and specificity constructed by gradually increasing the training corpus for a test corpus fixed to abstracts supports this notion. although the sensitivity and specificity show some fluctuations, accuracy is observed to steadily increase as the training set size is increased in increments of abstracts. the learning curves will likely plateau with the addition of further training abstracts, although any increase in sensitivity will add to data completeness, and increased specificity will minimize labor intensity. abstract category signatures the dtm of the training corpus contains more than terms. most are very rare terms present in only one or a few abstracts, and have very little influence on abstract classification as corresponding to either relevant or irrelevant articles. rare terms can be removed by setting a sparsity threshold if dtm dimensions become too large. examining the top ten terms, most discriminative between abstracts of relevant and irrelevant articles, show a distinct signature and reveal particular emphasis on such terms as "immunotherapy", "epitope", "t cell", and "ctl". these terms are likely the main drivers of classification and may very well be sufficient to support the main task of classification. notable is the fact that all discriminating terms are predominant in relevant abstracts, which may explain that sensitivity of classification is higher than specificity. this is most likely due to the highly specific nature of the relevant abstracts, whereas irrelevant abstracts are a much broader class. however, these terms are still represented in the corpus of irrelevant literature, so a machine learning approach to classification is highly likely to outperform a simple keyword search.  specialized biological databases are gradually moving from data repositories towards knowledge-based systems. enriching basic biological data with higher-level functional annotations and facilitating specialized analyses in organized workflows enables extraction of higher-level knowledge. currently, however, functional annotations are primarily stored in the literature, rather than in standardized formats of primary biological databases. as the quantity of this information increases, easy access to multiple layers of biological data and information enables improved extraction of knowledge, thus increasing the value to the user. we here present a conceptual framework for automating the process of updating biological databases and knowledgebases with standardized non-standardized data from both primary and secondary data repositories, as well as literature. we deployed a text mining-based approach to categorize literature, based on defining term signatures of freely available article abstracts, which enable significantly faster manual extraction of relevant data. we have applied this conceptual framework to literature for updating the tantigen kbs of tumor t cell antigens. training of a k-nn classifier on abstracts yielded classification accuracy of, thus showing significant value in support of data extraction from the literature. 
we defined and applied a machine learning approach for literature classification to support updating of tantigen, a knowledgebase of tumor t-cell antigens. as the output of biological assays increase in resolution and volume, the body of specialized biological data, such as functional annotations of gene and protein sequences, enables extraction of higher-level knowledge needed for practical application in bioinformatics. whereas common types of biological data, such as sequence data, are extensively stored in biological databases, functional annotations, such as immunological epitopes, are found primarily in semi-structured formats or free text embedded in primary scientific literature. training and five-fold cross-validation of a k-nn classifier on abstracts yielded classification accuracy of, thus showing significant value in support of data extraction from the literature. we here propose a conceptual framework for semi-automated extraction of epitope data embedded in scientific literature using principles from text mining and machine learning. abstracts from pubmed were downloaded and classified as either "relevant" or "irrelevant" for database update. the addition of such data will aid in the transition of biological databases to knowledgebases. text miningmachine learningbiological databasesautomation20- september asia pacific bioinformatics network twelfth international conference on bioinformatics taicang, china
 as the output of biological assays increase in resolution and volume, the body of specialized biological data, such as functional annotations of gene and protein sequences, enables extraction of higher-level knowledge needed for practical application in bioinformatics. whereas common types of biological data, such as sequence data, are extensively stored in biological databases, functional annotations, such as immunological epitopes, are found primarily in semi-structured formats or free text embedded in primary scientific literature.  we defined and applied a machine learning approach for literature classification to support updating of tantigen, a knowledgebase of tumor t-cell antigens. abstracts from pubmed were downloaded and classified as either "relevant" or "irrelevant" for database update. training and five-fold cross-validation of a k-nn classifier on abstracts yielded classification accuracy of, thus showing significant value in support of data extraction from the literature.  we here propose a conceptual framework for semi-automated extraction of epitope data embedded in scientific literature using principles from text mining and machine learning. the addition of such data will aid in the transition of biological databases to knowledgebases. text miningmachine learningbiological databasesautomation20- september asia pacific bioinformatics network twelfth international conference on bioinformatics taicang, china 