BACKGROUND
post-translational modifications  of protein sequences regulate many aspects of protein behavior including protein activity, localization, interactions, and expression level, and their characterization is a crucial step towards the complete description of protein function.
databases such as the uniprot knowledgebase
<cit>  include information on ptms that is directly curated from the scientific literature.
the continuing and ever-accelerating growth of this literature presents a challenge for database curators  who wish to keep pace with the knowledge on ptms and their functions.
text mining tools may facilitate the work of database curators in three ways, through:  the identification and prioritization of relevant documents,  the identification of bio-entity mentions in the text of those documents, such as gene and disease names, as well as their normalization , and  the extraction of biological events, which necessitates determination of the relationships between entities
<cit> .
biological event extraction has been the subject of increasing interest by the text mining community in recent years, with typical challenges relating to the extraction of information on protein-protein interactions or ptms.
examples include the community-based evaluations of event extraction methods in the biocreative ii/iii challenges
<cit> , and the bionlp shared tasks, which included ptm event extraction for phosphorylation
<cit> , and  <dig> additional ptms
<cit> .
the results of these bionlp challenges were very promising, with participants achieving f-scores over 80% for single ptm type extraction
<cit>  and approaching 70% for multiple ptms
<cit> .
the highest-performing methods were based on natural language processing  and machine-learning techniques.
in the follow-up to the bionlp shared tasks, s pyysalo et al.
<cit>  extended the event extraction task to  <dig> ptms, but system performance decreased with the best systems achieving f-scores only slightly in excess of 50%.
ptm information extraction methods were also developed in other contexts.
rlims-p was the first tool for protein phosphorylation information extraction
<cit> .
this tool processes documents with a shallow parser and then extracts information by matching text with manually designed patterns.
efip is a system produced by the developers of rlims-p which combines various tools to identify abstracts which provide information on the functional context of protein phosphorylation, such as protein interaction induced by phosphorylation
<cit> .
another tool, minephos, uses enhanced rlims-p patterns along with support vector machines  combined with dictionary lookup to identify the modified protein
<cit> .
these methods achieved good performance with precision and recall over 80%, but they were not assessed on the same dataset as the tools participating in the bionlp challenges.
in this work, we developed a ptm information extraction procedure to assist curation of uniprotkb.
uniprotkb is a high-quality resource of protein sequence and functional information that includes substantial curated information on protein sequence features, including ptms
<cit> .
to support uniprot curation efforts we developed an in-house system that is able to extract the information that is needed for uniprotkb ptm annotation from pubmed abstracts.
the tool was developed in close collaboration with uniprotkb curators.
since ptm data are often not displayed in abstracts, we also explored the extraction of ptm information from full-text articles.
the tool is based on a pattern-matching and rule-based approach, and specifically targets those ptms that are most frequently annotated in uniprotkb, namely acetylation, amidation, disulfide bridge formation, glycosylation, methylation, phosphorylation, and sulfation.
our procedure is used to continually track new articles describing modified sites as they appear in pubmed.
it is also used in a targeted fashion, to find potential confirmatory evidence for sites that have been annotated in uniprotkb based on information from high-throughput mass spectrometry-based proteomic studies
<cit> .
since such studies do not normally include significant follow-up or functional studies on potential identified sites, we use our tool to scan the literature to find new studies which may provide these types of supporting information.
RESULTS
the system we developed is designed to facilitate the annotation of uniprotkb with information on post-translational modifications .
such annotation requires knowledge of  the type of modification,  the modified amino acid and its position in the protein sequence, and  the protein which is modified .
in uniprotkb, over  <dig> different ptms are described.
most of these are relatively rare, so for development purposes we limited the scope of the tool to the most frequently annotated ptms, namely acetylation, amidation, disulfide bond formation, glycosylation, methylation, phosphorylation, and sulfation.
using a pattern-matching and rule-based approach, the procedure retrieves information in a stepwise fashion, retaining those sentences which contain both information on the actual type of ptm and the identity of the modified site.
the type of ptm is detected in two steps.
first, the whole document  is screened for the presence of generic tokens specific for each ptm .
following this initial triage, each sentence of the document is checked with a set of fine-grained regular expressions related to the generic token that matches .
in the retained sentences, the modification site is sought for the mention of at least an amino acid which can be modified.
this is achieved using regular expressions and does not require any syntactic link between terms describing site and ptm type; hence we avoid the use of deep nlp-parsing and the design of complex extraction pattern templates specific for each ptm.
for the detection of the position in the sequence, however, we used preposition-based parsing
<cit>  which relies on finding specific grammatical elements, like preposition, punctuation or conjunction, which link amino acid mentions and figures.
for instance, in the sentence “lc-ms/ms analysis of pkcdelta-activated intact hbvr identified phosphorylated serine positions  <dig>   <dig>   <dig>  and 237…” , our elementary parsing technique would permit the detection of positions separated from the amino acid mention by linking terms, e.g.
“serine positions 21”, as well as suites of figures, e.g.
“ <dig>   <dig>   <dig>  and 237”.
extraction of the position of the modified site in the sequence is not mandatory, since this information is often found only in the full-text article.
the putative modified protein was detected using the gene mention tagger aiiagmt
<cit> , which performed well in the gene mention task of the biocreative ii assessment
<cit> .
aiiagmt returns a list of potential candidate proteins  which is ranked according to a simple scoring schema .
“filtering token” is the term used to select the abstracts, “# filtered abstracts” is the number of abstracts which contain these terms, and “# retrieved abstracts” is the number of abstracts selected by the complete sentence extraction procedure.
precision was estimated based on manual analysis of  <dig> positive abstracts.
method assessment
for each ptm tested we estimated the precision on a generic corpus composed of all abstracts cited in uniprotkb/swiss-prot.
this dataset was utilised both to refine the regular expressions used for ptm type and site identification and to define additional filtering rules - such as excluding dna methylation.
the precision was estimated by manual analysis of  <dig> retrieved abstracts which were not used for method refinement.
the accuracy of protein identification by aiiagmt was assessed independently of the ptm extraction, meaning that a ptm event was considered as correct although the actual protein target was not identified.
table
<dig> provides a summary of the performance of the tool.
precision varies according to ptm type, reaching 94% for disulfide bonds .
the lowest precision was observed for protein methylation, due to the frequent occurrence of the “methyl” token in chemical compound names.
for this particular ptm it is difficult to improve precision except by the time-consuming design of very fine-grained patterns and exclusion rules.
the most frequent source of false positive sentences was due to the detection of terms wrongly interpreted as protein modification, such as “o-acetylserine” – a metabolite of cysteine synthesis – or “sepharose 4b-tyrosine-sulfanilamide” – an affinity chromatography chemical.
other errors were due to the detection of non-specific information, such as “…proteome-wide lysine acetylation has been documented in prokaryotes,…” , although, in this case, the full-text article may contain ptm data for specific proteins.
the detection of documents describing enzymes catalazing a specific ptm was also a frequent source of errors, e.g.
“: oligosaccharyltransferase  transfers glycan to asparagine in the n-glycosylation sequon.” .
finally, the lack of any syntactic parsing in our method also led to several identifiable errors.
for instance, the title sentence: “overexpression of reactive cysteine-containing 2-nitrobenzoate nitroreductase  and its mutants alters the sensitivity of escherichia coli to reactive oxygen species by reprogramming a regulatory network of disulfide-bonded proteins.” , the system associates the cysteine mention to the disulfide bond bond although there is no dependency between these terms.
a strategy for improving the precision of identification would consist of further filtering the documents for information on the position of the ptm in the protein sequence.
we noticed that the precision increases significantly when the position is also mandatory , associated with a concomitant reduction in recall .
as the goal of the tool is to facilitate comprehensive annotation of ptms , and results are perused by human curators, we retained those abstracts which do not provide positional information for further checking.
the recall was estimated from the proportion of documents retrieved from a positive corpus that consisted of the set of abstracts cited in uniprotkb/swiss-prot as providing experimental evidence for the existence of the modification.
since uniprotkb/swiss-prot annotation is based on the full text of documents, it is likely that some of the information we are seeking is not available in the abstracts of this positive corpus.
we therefore examined those abstracts that were not retrieved by the tool, in order to check if the abstract contained information on modified residues, or whether this information was restricted to the full text of the article.
in the latter case, such “negative abstracts” were not scored as false negatives.
the evaluation of the gene/protein detection and ranking was performed by a manual check of  <dig> abstracts with ptm information.
in  <dig>   <dig>  and  <dig> documents the mention of the gene carrying the modification was ranked at the first, second and third position respectively.
in only  <dig> abstracts did aiiagmt fail to find the correct gene mention.
we also assessed our sentence extraction procedure on the genia project’s “ptm event corpus”  task
<cit> ) for three types of ptms, acetylation, glycosylation and methylation .
the performance was evaluated based on the tools ability to find the type of ptm and its site , but not the protein target  or the enzyme catalyzing the reaction .
for this reason, the assessment cannot be directly compared with the bionlp event extraction results.
#events is the number of acetylation, glycosylation or methylation events with site information on in the genia corpus, #documents is the number of abstracts positive for phosphorylation information in the rlimp-p corpus.
phosphorylation information extraction was also evaluated on the rlims-p corpus
<cit>  .
this corpus provides a list of abstracts classified as positive or negative for protein phosphorylation information.
our tool has comparable precision to rlims-p, which relies on sophisticated information extraction techniques
<cit> .
the lower recall can mainly be explained by our requirement for a site mention for a sentence to be selected.
actually in the rlims-p corpus, documents which contain only generic information on phosphorylation were classified as positive, such as the following title: “phosphorylation, a factor controlling the synthesis of l-erythrodihydrobiopterin .” .
we took advantage of the benchmarked gene normalization provided for each positive document of the rlims-p corpus to evaluate the gene/protein mention detection and ranking.
among the  <dig> positive documents our information extraction system classified correctly, the actual phosphorylated protein was mentioned at the first rank for  <dig> documents, at the second for  <dig>  and at the third for  <dig>  in only  <dig> cases was the correct protein not found because aiiagmt did not correctly identify the protein name boundaries.
system applications
the method we developed was designed to support the annotation of uniprotkb in two settings.
the first application is the automated tracking of new publications describing protein modifications involving any of the seven target ptms.
the second application is the targeted retrieval of documents that provide potential supporting information for phosphorylation sites that were previously annotated in uniprotkb/swiss-prot based on information from high-throughput mass spectrometry-based proteomic studies.
a pipeline for screening ptm information
we set up a pipeline to automatically retrieve abstracts from pubmed that contain information on each of the seven target ptms.
in this pipeline, pubmed is queried with the keyword “protein”, using the entrez programming utilities
<cit> , and the resulting documents are processed by the ptm information extraction procedure.
the selected abstracts  are presented to curators in the form of an html document with relevant information such as the putative modifications, the putative modified amino acids and their position, and gene mentions highlighted in the text .
the ranked list of gene/protein mentions completes each abstract display.
since protein phosphorylation is frequently mentioned in abstracts even when no  sites are experimentally characterized in the associated publication, we retrieved only those abstracts that also mentioned a sequence position in the sentence.
this reduced the number of extracted publications that provide information which is irrelevant to ptm annotation.
cumulative figures for monthly requests during a six month period are shown in table
<dig>  although the number of retrieved phosphosites remains high, we found that many of these were already annotated in uniprotkb/swiss-prot.
for the other types of ptms, the number of retrieved abstracts remains manageable for human curators.
“retrieved abstracts” are the result of querying pubmed with the keyword “protein”.
the number of ptms with positional information is shown in parentheses for each ptm type .
retrieval of supporting information for phosphosites annotated based on high-throughput proteomic studies
in uniprotkb/swiss-prot, over  <dig>  entries contain experimentally determined phosphorylation sites whose positions were annotated based on information from high-throughput mass spectrometry-based proteomics experiments.
these types of experiment often take the form of an exploratory survey, and do not include follow-up characterization to confirm the positions of the putative identified sites.
we therefore implemented a procedure to scan the literature to find any new publications that may provide such confirmation .
in brief, the procedure queries pubmed for each individual protein.
the retrieved abstracts are then treated to extract information on phosphorylated sites.
each extracted site is then checked against the list of annotated sites in the corresponding uniprotkb/swiss-prot entry.
the output of the pipeline is a tab-delimited file which displays the list of identified sites together with links to the corresponding uniprotkb/swiss-prot entries, and links to the abstracts from which the information was extracted.
when applied to the complete set of annotated phosphosites from high-throughput mass spectrometry-based proteomics experiments in uniprotkb/swiss-prot, our procedure retrieved abstracts matching  <dig>  annotated phosphosites in  <dig> proteins.
by manual analysis of the abstracts retrieved for  <dig> of these proteins, we estimated the precision of the procedure, calculated on unique sites, to be 66%.
most of the false positives were due to incorrect attribution of a modification of a conserved site found in a closely related homolog of the protein in question.
sentence extraction from full-text articles
in order to assess the benefits of analyzing full-text articles, we performed a preliminary analysis using the open-access subset of uk pubmed central
<cit> .
the papers were selected from the list of pubmed abstracts which were retained in the previous experiment following the first filtering step with the “phospho” token.
we gathered over  <dig>  articles from ukpmc that mention  <dig>  proteins with phosphosites detected by high-throughput proteomics experiments.
these articles were divided into sections and the sentence extraction procedure applied to each section.
to reduce the number of false positive identifications we required that extracted sentences include a mention of the actual queried protein.
the information extraction system identified  <dig> articles with sentences which matched  <dig> phosphorylated sites annotated based on high-throughput proteomics experiments in  <dig> proteins.
a precision level of 68% was calculated by manual analysis of the retrieved articles, with the majority of false positives again due to incorrect attribution of a modification to a homologous protein.
as much as 90% of all matching sites were only described outside the abstract.
within the full text, around 38% of the extracted sentences were found in the “results” sections, 29% in the “methods” section , 19% in the “discussion” and 11% in the “introduction”.
the remaining 3% of sentences were found in undefined sections, mostly due to unusual sectioning.
we also examined how a requirement that the queried gene/protein name and the phosphorylation site co-occurred in the same sentence would affect recall, by manually checking the extracted sentences from  <dig> articles.
from the  <dig> sentences with validated site information,  <dig> were lost when this restriction was applied.
however, for  <dig> of them, it did not result in a loss of information since the identified phosphosites were already found in other retained sentences of the same article.
this preliminary experiment demonstrates that full-text article analysis provides higher recall than simple abstract screening for ptm information extraction.
it is difficult to draw general conclusions from this result though, as open-access articles available in xml represent only around 2% of the biomedical publications referred in pubmed.
we therefore adapted the procedure to treat documents derived from the conversion in text format of the pdf version of articles.
we downloaded from ukpmc a subset of  <dig> articles in pdf format, among the articles which show matching sentences in the previous experiment.
from the  <dig> mentions of a phosphosite detected in these sentences, only  <dig> were not found in these converted articles, showing that our procedure is robust to errors due to pdf-text conversion.
we therefore carried out another experiment in the framework of a uniprotkb annotation task which consisted of trying to complete ambiguous citation content description where phosphorylation was mentioned without site specification.
we screened  <dig> articles cited in  <dig> uniprotkb entries and were able to find  <dig> positions described in  <dig> articles.
the precision was high, since there was no ambiguity concerning the modified protein; only  <dig> positions could not be added to the citation content description, mainly because of insufficient experimental evidence, a value judgment normally performed by a human curator.
this last application is a clear demonstration of the complementarity of the text mining tools and the manual curation process.
discussion
the tool we present in this paper is designed to facilitate the annotation of information on protein post-translational modifications in uniprotkb.
our procedure offers a means to track publications describing new ptms and can be used to link existing annotations to supporting evidence in the literature, which is vital for a number of applications including the cleaning of annotations derived from high throughput experiments and the development of evidence tags
<cit> .
one of the main goals of text mining development is to increase the speed of database curation while maintaining the high accuracy of expert-curated annotations.
targeted surveys of curators suggest that appropriate text mining tools should be simple, user-friendly, and adaptable to the annotation needs of the particular database
<cit> .
the role of the tool should be to facilitate the work of the curator; while it should retrieve relevant information at acceptable levels of precision and recall, it does not need to be perfect.
our system provides batch-processed results in a user-friendly format which facilitates visualization of the extracted information and access to additional resources.
the process is simple and could be integrated in any curation workflow.
our information extraction method integrates biological knowledge in the form of expert-curated patterns and domain-specific rules.
since it only relies on a specific vocabulary, the method can be easily extended to retrieve information on other type of ptms.
while such domain-specific approaches often achieve high precision at the expense of low recall, we purposely attempted to favor recall during the design of our tool, the output of which is provided to human curators.
this was achieved by allowing non-specific pattern matches for ptm type detection and filtering of the resulting matches with additional rules restricting context and site specificity.
in this context, we found that the identity of the actual modified position in the sequence significantly reduced the number of false positive identifications.
a similar procedure proved equally useful in the development of a tool to retrieve new publications on protein mutations
<cit> .
to detect phosphorylation events, rlims-p and minephos use similar pattern-matching approaches, but complement these with a prior nlp step consisting of part-of-speech tagging and phrase chunking.
although the performance of these tools were better than ours, a direct comparison was not feasible, as existing evaluations were performed on different datasets , and the tool specifications were different .
to extract ptm information, we deliberately chose to limit the use of natural language processing to very simple methods, such as propositional parsing, which does not significantly enhance the computational expense.
it was a requirement of our system to perform large-scale document analysis in a reasonable time.
our tool has been used to treat nearly  <dig>  million abstracts for validation of phosphosites detected by high throughput proteomics experiments.
however, we did not measure of the impact of techniques, such as shallow or deep parsing, on the tool either at the level of extraction accuracy or expended computational time.
direct comparisons with the highest-performing methods for event extraction assessed in the bionlp st challenges are also not feasible, since the methods have different task specifications and site information was often not present in event annotation.
nevertheless our tool performs appropriately as a useful curator aid in a real-world annotation workflow.
future developments will include consideration of methods to enhance the procedure for identification of the modified protein.
this could be done normalizing the identified mention.
such normalization will allow the tool  to check if the detected residue at the detected position corresponds to the one in the protein sequence  to filter sites which are already annotated in the database.
gene normalization is a difficult problem, especially when only abstracts are analyzed since species information is often found in the core article.
however, there has been progress in this task which was recently evaluated at the biocreative iii challenge
<cit> , and we will test current programs for possible inclusion in our pipeline.
CONCLUSIONS
this study presents a simple and robust procedure, based on domain-specific patterns and rules, to retrieve and extract from the biomedical literature information on protein post-translational modifications.
assessment of the method showed that it is competitive with other tools designed for the same purpose.
the procedure was set up to treat both pubmed abstracts and full-text articles and to handle specific annotation tasks of the uniprot knowldgebase.
it demonstrates that text mining techniques can be efficiently applied in database curation.

 post-translational modifications of protein sequences regulate many aspects of protein behavior including protein activity, localization, interactions, and expression level, and their characterization is a crucial step towards the complete description of protein function. databases such as the uniprot knowledgebase include information on ptms that is directly curated from the scientific literature. the continuing and ever-accelerating growth of this literature presents a challenge for database curators who wish to keep pace with the knowledge on ptms and their functions. text mining tools may facilitate the work of database curators in three ways, through: the identification and prioritization of relevant documents, the identification of bio-entity mentions in the text of those documents, such as gene and disease names, as well as their normalization, and the extraction of biological events, which necessitates determination of the relationships between entities. biological event extraction has been the subject of increasing interest by the text mining community in recent years, with typical challenges relating to the extraction of information on protein-protein interactions or ptms. examples include the community-based evaluations of event extraction methods in the biocreative ii/iii challenges, and the bionlp shared tasks, which included ptm event extraction for phosphorylation, and additional ptms. the results of these bionlp challenges were very promising, with participants achieving f-scores over 80% for single ptm type extraction and approaching 70% for multiple ptms. the highest-performing methods were based on natural language processing and machine-learning techniques. in the follow-up to the bionlp shared tasks, s pyysalo et al. extended the event extraction task to ptms, but system performance decreased with the best systems achieving f-scores only slightly in excess of 50%. ptm information extraction methods were also developed in other contexts. rlims-p was the first tool for protein phosphorylation information extraction. this tool processes documents with a shallow parser and then extracts information by matching text with manually designed patterns. efip is a system produced by the developers of rlims-p which combines various tools to identify abstracts which provide information on the functional context of protein phosphorylation, such as protein interaction induced by phosphorylation. another tool, minephos, uses enhanced rlims-p patterns along with support vector machines combined with dictionary lookup to identify the modified protein. these methods achieved good performance with precision and recall over 80%, but they were not assessed on the same dataset as the tools participating in the bionlp challenges. in this work, we developed a ptm information extraction procedure to assist curation of uniprotkb. uniprotkb is a high-quality resource of protein sequence and functional information that includes substantial curated information on protein sequence features, including ptms. to support uniprot curation efforts we developed an in-house system that is able to extract the information that is needed for uniprotkb ptm annotation from pubmed abstracts. the tool was developed in close collaboration with uniprotkb curators. since ptm data are often not displayed in abstracts, we also explored the extraction of ptm information from full-text articles. the tool is based on a pattern-matching and rule-based approach, and specifically targets those ptms that are most frequently annotated in uniprotkb, namely acetylation, amidation, disulfide bridge formation, glycosylation, methylation, phosphorylation, and sulfation. our procedure is used to continually track new articles describing modified sites as they appear in pubmed. it is also used in a targeted fashion, to find potential confirmatory evidence for sites that have been annotated in uniprotkb based on information from high-throughput mass spectrometry-based proteomic studies. since such studies do not normally include significant follow-up or functional studies on potential identified sites, we use our tool to scan the literature to find new studies which may provide these types of supporting information.  the system we developed is designed to facilitate the annotation of uniprotkb with information on post-translational modifications. such annotation requires knowledge of the type of modification, the modified amino acid and its position in the protein sequence, and the protein which is modified. in uniprotkb, over different ptms are described. most of these are relatively rare, so for development purposes we limited the scope of the tool to the most frequently annotated ptms, namely acetylation, amidation, disulfide bond formation, glycosylation, methylation, phosphorylation, and sulfation. using a pattern-matching and rule-based approach, the procedure retrieves information in a stepwise fashion, retaining those sentences which contain both information on the actual type of ptm and the identity of the modified site. the type of ptm is detected in two steps. first, the whole document is screened for the presence of generic tokens specific for each ptm. following this initial triage, each sentence of the document is checked with a set of fine-grained regular expressions related to the generic token that matches. in the retained sentences, the modification site is sought for the mention of at least an amino acid which can be modified. this is achieved using regular expressions and does not require any syntactic link between terms describing site and ptm type; hence we avoid the use of deep nlp-parsing and the design of complex extraction pattern templates specific for each ptm. for the detection of the position in the sequence, however, we used preposition-based parsing which relies on finding specific grammatical elements, like preposition, punctuation or conjunction, which link amino acid mentions and figures. for instance, in the sentence “lc-ms/ms analysis of pkcdelta-activated intact hbvr identified phosphorylated serine positions and 237…”, our elementary parsing technique would permit the detection of positions separated from the amino acid mention by linking terms, e.g. “serine positions 21”, as well as suites of figures, e.g. “ and 237”. extraction of the position of the modified site in the sequence is not mandatory, since this information is often found only in the full-text article. the putative modified protein was detected using the gene mention tagger aiiagmt, which performed well in the gene mention task of the biocreative ii assessment. aiiagmt returns a list of potential candidate proteins which is ranked according to a simple scoring schema. “filtering token” is the term used to select the abstracts, “# filtered abstracts” is the number of abstracts which contain these terms, and “# retrieved abstracts” is the number of abstracts selected by the complete sentence extraction procedure. precision was estimated based on manual analysis of positive abstracts. method assessment for each ptm tested we estimated the precision on a generic corpus composed of all abstracts cited in uniprotkb/swiss-prot. this dataset was utilised both to refine the regular expressions used for ptm type and site identification and to define additional filtering rules - such as excluding dna methylation. the precision was estimated by manual analysis of retrieved abstracts which were not used for method refinement. the accuracy of protein identification by aiiagmt was assessed independently of the ptm extraction, meaning that a ptm event was considered as correct although the actual protein target was not identified. table  provides a summary of the performance of the tool. precision varies according to ptm type, reaching 94% for disulfide bonds. the lowest precision was observed for protein methylation, due to the frequent occurrence of the “methyl” token in chemical compound names. for this particular ptm it is difficult to improve precision except by the time-consuming design of very fine-grained patterns and exclusion rules. the most frequent source of false positive sentences was due to the detection of terms wrongly interpreted as protein modification, such as “o-acetylserine” – a metabolite of cysteine synthesis – or “sepharose 4b-tyrosine-sulfanilamide” – an affinity chromatography chemical. other errors were due to the detection of non-specific information, such as “…proteome-wide lysine acetylation has been documented in prokaryotes,…”, although, in this case, the full-text article may contain ptm data for specific proteins. the detection of documents describing enzymes catalazing a specific ptm was also a frequent source of errors, e.g. “: oligosaccharyltransferase transfers glycan to asparagine in the n-glycosylation sequon.”. finally, the lack of any syntactic parsing in our method also led to several identifiable errors. for instance, the title sentence: “overexpression of reactive cysteine-containing 2-nitrobenzoate nitroreductase and its mutants alters the sensitivity of escherichia coli to reactive oxygen species by reprogramming a regulatory network of disulfide-bonded proteins.”, the system associates the cysteine mention to the disulfide bond bond although there is no dependency between these terms. a strategy for improving the precision of identification would consist of further filtering the documents for information on the position of the ptm in the protein sequence. we noticed that the precision increases significantly when the position is also mandatory, associated with a concomitant reduction in recall. as the goal of the tool is to facilitate comprehensive annotation of ptms, and results are perused by human curators, we retained those abstracts which do not provide positional information for further checking. the recall was estimated from the proportion of documents retrieved from a positive corpus that consisted of the set of abstracts cited in uniprotkb/swiss-prot as providing experimental evidence for the existence of the modification. since uniprotkb/swiss-prot annotation is based on the full text of documents, it is likely that some of the information we are seeking is not available in the abstracts of this positive corpus. we therefore examined those abstracts that were not retrieved by the tool, in order to check if the abstract contained information on modified residues, or whether this information was restricted to the full text of the article. in the latter case, such “negative abstracts” were not scored as false negatives. the evaluation of the gene/protein detection and ranking was performed by a manual check of abstracts with ptm information. in and documents the mention of the gene carrying the modification was ranked at the first, second and third position respectively. in only abstracts did aiiagmt fail to find the correct gene mention. we also assessed our sentence extraction procedure on the genia project’s “ptm event corpus” task ) for three types of ptms, acetylation, glycosylation and methylation. the performance was evaluated based on the tools ability to find the type of ptm and its site, but not the protein target or the enzyme catalyzing the reaction. for this reason, the assessment cannot be directly compared with the bionlp event extraction results. #events is the number of acetylation, glycosylation or methylation events with site information on in the genia corpus, #documents is the number of abstracts positive for phosphorylation information in the rlimp-p corpus. phosphorylation information extraction was also evaluated on the rlims-p corpus. this corpus provides a list of abstracts classified as positive or negative for protein phosphorylation information. our tool has comparable precision to rlims-p, which relies on sophisticated information extraction techniques. the lower recall can mainly be explained by our requirement for a site mention for a sentence to be selected. actually in the rlims-p corpus, documents which contain only generic information on phosphorylation were classified as positive, such as the following title: “phosphorylation, a factor controlling the synthesis of l-erythrodihydrobiopterin.”. we took advantage of the benchmarked gene normalization provided for each positive document of the rlims-p corpus to evaluate the gene/protein mention detection and ranking. among the positive documents our information extraction system classified correctly, the actual phosphorylated protein was mentioned at the first rank for documents, at the second for and at the third for in only cases was the correct protein not found because aiiagmt did not correctly identify the protein name boundaries. system applications the method we developed was designed to support the annotation of uniprotkb in two settings. the first application is the automated tracking of new publications describing protein modifications involving any of the seven target ptms. the second application is the targeted retrieval of documents that provide potential supporting information for phosphorylation sites that were previously annotated in uniprotkb/swiss-prot based on information from high-throughput mass spectrometry-based proteomic studies. a pipeline for screening ptm information we set up a pipeline to automatically retrieve abstracts from pubmed that contain information on each of the seven target ptms. in this pipeline, pubmed is queried with the keyword “protein”, using the entrez programming utilities, and the resulting documents are processed by the ptm information extraction procedure. the selected abstracts are presented to curators in the form of an html document with relevant information such as the putative modifications, the putative modified amino acids and their position, and gene mentions highlighted in the text. the ranked list of gene/protein mentions completes each abstract display. since protein phosphorylation is frequently mentioned in abstracts even when no sites are experimentally characterized in the associated publication, we retrieved only those abstracts that also mentioned a sequence position in the sentence. this reduced the number of extracted publications that provide information which is irrelevant to ptm annotation. cumulative figures for monthly requests during a six month period are shown in table  although the number of retrieved phosphosites remains high, we found that many of these were already annotated in uniprotkb/swiss-prot. for the other types of ptms, the number of retrieved abstracts remains manageable for human curators. “retrieved abstracts” are the result of querying pubmed with the keyword “protein”. the number of ptms with positional information is shown in parentheses for each ptm type. retrieval of supporting information for phosphosites annotated based on high-throughput proteomic studies in uniprotkb/swiss-prot, over entries contain experimentally determined phosphorylation sites whose positions were annotated based on information from high-throughput mass spectrometry-based proteomics experiments. these types of experiment often take the form of an exploratory survey, and do not include follow-up characterization to confirm the positions of the putative identified sites. we therefore implemented a procedure to scan the literature to find any new publications that may provide such confirmation. in brief, the procedure queries pubmed for each individual protein. the retrieved abstracts are then treated to extract information on phosphorylated sites. each extracted site is then checked against the list of annotated sites in the corresponding uniprotkb/swiss-prot entry. the output of the pipeline is a tab-delimited file which displays the list of identified sites together with links to the corresponding uniprotkb/swiss-prot entries, and links to the abstracts from which the information was extracted. when applied to the complete set of annotated phosphosites from high-throughput mass spectrometry-based proteomics experiments in uniprotkb/swiss-prot, our procedure retrieved abstracts matching annotated phosphosites in proteins. by manual analysis of the abstracts retrieved for of these proteins, we estimated the precision of the procedure, calculated on unique sites, to be 66%. most of the false positives were due to incorrect attribution of a modification of a conserved site found in a closely related homolog of the protein in question. sentence extraction from full-text articles in order to assess the benefits of analyzing full-text articles, we performed a preliminary analysis using the open-access subset of uk pubmed central. the papers were selected from the list of pubmed abstracts which were retained in the previous experiment following the first filtering step with the “phospho” token. we gathered over articles from ukpmc that mention proteins with phosphosites detected by high-throughput proteomics experiments. these articles were divided into sections and the sentence extraction procedure applied to each section. to reduce the number of false positive identifications we required that extracted sentences include a mention of the actual queried protein. the information extraction system identified articles with sentences which matched phosphorylated sites annotated based on high-throughput proteomics experiments in proteins. a precision level of 68% was calculated by manual analysis of the retrieved articles, with the majority of false positives again due to incorrect attribution of a modification to a homologous protein. as much as 90% of all matching sites were only described outside the abstract. within the full text, around 38% of the extracted sentences were found in the “results” sections, 29% in the “methods” section, 19% in the “discussion” and 11% in the “introduction”. the remaining 3% of sentences were found in undefined sections, mostly due to unusual sectioning. we also examined how a requirement that the queried gene/protein name and the phosphorylation site co-occurred in the same sentence would affect recall, by manually checking the extracted sentences from articles. from the sentences with validated site information, were lost when this restriction was applied. however, for of them, it did not result in a loss of information since the identified phosphosites were already found in other retained sentences of the same article. this preliminary experiment demonstrates that full-text article analysis provides higher recall than simple abstract screening for ptm information extraction. it is difficult to draw general conclusions from this result though, as open-access articles available in xml represent only around 2% of the biomedical publications referred in pubmed. we therefore adapted the procedure to treat documents derived from the conversion in text format of the pdf version of articles. we downloaded from ukpmc a subset of articles in pdf format, among the articles which show matching sentences in the previous experiment. from the mentions of a phosphosite detected in these sentences, only were not found in these converted articles, showing that our procedure is robust to errors due to pdf-text conversion. we therefore carried out another experiment in the framework of a uniprotkb annotation task which consisted of trying to complete ambiguous citation content description where phosphorylation was mentioned without site specification. we screened articles cited in uniprotkb entries and were able to find positions described in articles. the precision was high, since there was no ambiguity concerning the modified protein; only positions could not be added to the citation content description, mainly because of insufficient experimental evidence, a value judgment normally performed by a human curator. this last application is a clear demonstration of the complementarity of the text mining tools and the manual curation process. discussion the tool we present in this paper is designed to facilitate the annotation of information on protein post-translational modifications in uniprotkb. our procedure offers a means to track publications describing new ptms and can be used to link existing annotations to supporting evidence in the literature, which is vital for a number of applications including the cleaning of annotations derived from high throughput experiments and the development of evidence tags. one of the main goals of text mining development is to increase the speed of database curation while maintaining the high accuracy of expert-curated annotations. targeted surveys of curators suggest that appropriate text mining tools should be simple, user-friendly, and adaptable to the annotation needs of the particular database. the role of the tool should be to facilitate the work of the curator; while it should retrieve relevant information at acceptable levels of precision and recall, it does not need to be perfect. our system provides batch-processed results in a user-friendly format which facilitates visualization of the extracted information and access to additional resources. the process is simple and could be integrated in any curation workflow. our information extraction method integrates biological knowledge in the form of expert-curated patterns and domain-specific rules. since it only relies on a specific vocabulary, the method can be easily extended to retrieve information on other type of ptms. while such domain-specific approaches often achieve high precision at the expense of low recall, we purposely attempted to favor recall during the design of our tool, the output of which is provided to human curators. this was achieved by allowing non-specific pattern matches for ptm type detection and filtering of the resulting matches with additional rules restricting context and site specificity. in this context, we found that the identity of the actual modified position in the sequence significantly reduced the number of false positive identifications. a similar procedure proved equally useful in the development of a tool to retrieve new publications on protein mutations. to detect phosphorylation events, rlims-p and minephos use similar pattern-matching approaches, but complement these with a prior nlp step consisting of part-of-speech tagging and phrase chunking. although the performance of these tools were better than ours, a direct comparison was not feasible, as existing evaluations were performed on different datasets, and the tool specifications were different. to extract ptm information, we deliberately chose to limit the use of natural language processing to very simple methods, such as propositional parsing, which does not significantly enhance the computational expense. it was a requirement of our system to perform large-scale document analysis in a reasonable time. our tool has been used to treat nearly million abstracts for validation of phosphosites detected by high throughput proteomics experiments. however, we did not measure of the impact of techniques, such as shallow or deep parsing, on the tool either at the level of extraction accuracy or expended computational time. direct comparisons with the highest-performing methods for event extraction assessed in the bionlp st challenges are also not feasible, since the methods have different task specifications and site information was often not present in event annotation. nevertheless our tool performs appropriately as a useful curator aid in a real-world annotation workflow. future developments will include consideration of methods to enhance the procedure for identification of the modified protein. this could be done normalizing the identified mention. such normalization will allow the tool to check if the detected residue at the detected position corresponds to the one in the protein sequence to filter sites which are already annotated in the database. gene normalization is a difficult problem, especially when only abstracts are analyzed since species information is often found in the core article. however, there has been progress in this task which was recently evaluated at the biocreative iii challenge, and we will test current programs for possible inclusion in our pipeline.  this study presents a simple and robust procedure, based on domain-specific patterns and rules, to retrieve and extract from the biomedical literature information on protein post-translational modifications. assessment of the method showed that it is competitive with other tools designed for the same purpose. the procedure was set up to treat both pubmed abstracts and full-text articles and to handle specific annotation tasks of the uniprot knowldgebase. it demonstrates that text mining techniques can be efficiently applied in database curation. 
for ptm extraction, precision varies from 57% to 94%, and recall from 75% to 95%, according to the type of modification. to help curators cope with this growing body of information we have developed a system which extracts information from the scientific literature for the most frequently annotated ptms in uniprotkb. the procedure was used to track new publications on ptms and to recover potential supporting evidence for phosphorylation sites annotated based on the results of large scale proteomics experiments. the annotation of protein post-translational modifications is an important task of uniprotkb curators and, with continuing improvements in experimental methodology, an ever greater number of articles are being published on this topic. the information retrieval and extraction method we have developed in this study forms the basis of a simple tool for the manual curation of protein post-translational modifications in uniprotkb/swiss-prot.
 the annotation of protein post-translational modifications is an important task of uniprotkb curators and, with continuing improvements in experimental methodology, an ever greater number of articles are being published on this topic. to help curators cope with this growing body of information we have developed a system which extracts information from the scientific literature for the most frequently annotated ptms in uniprotkb.  the procedure uses a pattern-matching and rule-based approach to extract sentences with information on the type and site of modification. a ranked list of protein candidates for the modification is also provided. for ptm extraction, precision varies from 57% to 94%, and recall from 75% to 95%, according to the type of modification. the procedure was used to track new publications on ptms and to recover potential supporting evidence for phosphorylation sites annotated based on the results of large scale proteomics experiments.  the information retrieval and extraction method we have developed in this study forms the basis of a simple tool for the manual curation of protein post-translational modifications in uniprotkb/swiss-prot. our work demonstrates that even simple text-mining tools can be effectively adapted for database curation tasks, providing that a thorough understanding of the working process and requirements are first obtained. this system can be accessed at http://eagl.unige.ch/ptm/. 