BACKGROUND
the accelerating expansion of biomedical research outpaces most individual attempts at comprehensive review even in relatively narrow fields.
just as the vast sequence data available for the human  <cit>  and additional organisms  <cit>  require sophisticated genomic browsing tools  <cit> , computational methods are required to thoroughly explore the corpus of biomedical literature.
many computational methods for interrogating the scientific literature have been developed  <cit> .
these programs can be broadly defined as methods for information retrieval and those for information extraction  <cit> .
existing methods can identify significant association between individual genes and terms from the medical subject heading  index and gene ontology  databases  <cit> , manually curated biological lists  <cit> , or disease-specific lists  <cit> .
these prior methods demonstrate that genes with disease-specific differential expression can be strongly correlated with key terms within the medical literature  <cit> .
in addition, gene-gene associations within the literature have been combined with multiple available databases to extend associations beyond the literature alone  <cit> .
while these and similar approaches have underscored the potential of automated literature searching to facilitate discovery, few have provided both methods for assessing the statistical strength of identified associations and supported their methods with experimental validation.
here, we describe and apply "literature lab", a method of automated data retrieval confined to publicly available citations and abstracts.
literature lab statistically assesses identified associations within the corpus of medical literature between sets of experimentally derived genes and key terms derived from curated or mesh lists.
we demonstrate that our methodology can identify previously reported relationships and can result in discovery.
molecular methods
gene expression sets
literature lab was applied to three gene sets derived from previously published microarray data and one gene set from as yet, unpublished dataset to determine if literature mining identified important metabolic, physiologic, or pathway activity.
these gene sets include, 1) the top  <dig> genes with increased expression during in vitro exposure of human leukemia cells  to atra , 2) the genes differentially regulated with a transgenic model of prostate neoplasia  following exposure to rad <dig>  3) the  <dig> genes used to predict outcome in localized node-negative breast cancer, and 4) genes differentially expressed between malignant epithelial cells in local v. metastatic prostate cancers.
the genes lists are included in additional files and a full description of the development of each gene set is in the supplemental methods .
western blots
protein from fresh ventral prostates were extracted in ripa buffer  separated by gel electrophoresis and transferred to nitrocellulose membrane  as described <cit> .
membranes were blotted with anti-hif-1α  and anti-tubulin   .
blots were scanned and intensities were measured.
immunohistochemistry
a prostate tissue microarray containing samples of benign prostate epithelium , locally invasive prostate adenocarcinoma , and metastatic prostate adenocarcinoma  were stained for fosb.
the fosb staining was performed as previously described  <cit> .
briefly,  <dig> micrometer sections were cut from the tma block, deparaffinized, rehydrated and subjected to microwaving in 10-mm citrate buffer  in a 750w oven, for  <dig> minutes.
the polyclonal anti-fosb primary antibody , was incubated  at room temperature in an automated stainer .
antigen-antibody reaction was revealed with standardized development times, using the streptavidin method with  <dig>   <dig> diaminobenzidine as substrate.
meyer hematoxylin was used as nuclear counterstaining.
fosb nuclear positivity was scored on a total of  <dig> nuclei per sample.
the statistical difference in fosb staining between the unpaired populations was assessed using a two-tailed mann whitney test .
application of established literature mining tools
mesher data for the gene lists was obtained from the website  <cit>  by supplying the affymetrix u133plus <dig> ids corresponding to the list of genes using a p-value threshold of  <dig>  and no p-value correction.
gominer data for the gene lists was obtained by following the instructions for downloading  and installing the gominer application and sql database from  <cit> .
a list of gene symbols for genes in the affymetrix u133plus <dig> probe set list was used for the 'total' file and lists of symbols for the experimental genes used for the 'changed' file.
other gominer settings used were the default values defined in the gominer application.
implementation
overview
to highlight and prioritize cellular physiology, metabolism, or pathways differentially active within a microarray experiment, literature lab uses an experimentally derived gene list, pre-defined sets of non-ambiguous key terms , inclusive gene nomenclature, comprehensive literature interrogation, and a comparison of the experimental gene set results with randomly generated gene sets.
for each experimentally derived gene set, literature lab performs an automated literature search to determine the number of abstracts listing any of the identified genes and each term within the term list .
specific domains include those for cell metabolism , cell physiology , and cellular pathways  .
terms are ranked with a measure of association between genes and terms known as the product of frequency  .
the strength of the relationship between an experimental gene set and a term is determined by comparing the log  of the sum of the pf values for the experimental gene set  with a distribution of the same values for  <dig> random gene sets of the same size as the experimental gene set.
a score representing the number of standard deviations above or below the mean value for the random gene sets is assigned to each term and terms are ranked in decreasing order of this score.
to highlight terms with particularly strong association with the gene list, we developed heuristic rules for labeling the relationships as "strong" or "moderate"  .
overall architecture of the software implementation
the software is implemented using sun's java programming language  <cit>  using the eclipse software development environment  <cit> .
the software consists of three major components:
• a series of programs which query pubmed for each term and each gene in the gene thesaurus using ncbi's entrez programming utilities  <cit>  and sun's jaxb xml binding classes  <cit> .
the result for each term or gene is a file containing a list of the pubmed ids for the term or gene.
in addition, summary files containing counts of the number of abstracts for each gene and for each term are prepared.
• a series of programs which identify the pubmed ids in common for each term/gene combination from the lists of ids for each term and each gene.
the results of these programs are files containing the pubmed ids for each term/gene combination and files containing summary counts of the number of abstracts for each term/gene combination.
• a series of programs for preparing the analysis of a specific gene list.
these programs compute the described statistics for the gene list and for one thousand random gene sets of the same size as the specific gene list being processed.
the final step of this process prepares a microsoft® excel spreadsheet containing the results of the analysis.
the apache software foundation's poi classes  were used to develop a set of rules required for more specific automated searching .
in subsequent literature searches, gene terms consisting entirely of numerals, of three characters or less, or that were identified as excessively ambiguous through manual review were excluded.
algorithm-based disambiguation was not used  <cit> .
while the database of gene terms  is frequently updated, for all experiments herein described, the gene thesaurus updated last on 12/30/ <dig> was used for all experiments presented .
<dig> of the gene terms were specifically excluded based upon the manual review .
all gene term curation was performed prior to testing for associations.
topic list curation
topic lists are sets of terms, each of which is to be tested for significant associations with the experimentally derived gene set.
topic lists used in the experiments were defined using terms from the mesh thesaurus provided by the national library of medicine.
the list of terms for each topic set consisted of all terms and all descendents  as listed in mesh.
the topic sets  used in these experiments were cell metabolism  and cell physiology  .
a few of the descendents of these mesh terms were ignored because very few abstracts were associated with the term.
when using mesh terms to search pubmed, the search syntax used "mesh subheading explosion" so that the resulting list of abstracts included abstracts coded with descendents  of the mesh term as well as abstracts coded with the mesh term itself.
in addition, a topic set entitled "pathways" was derived using the pathway descriptions from biocarta  <cit>  with some additional curation and testing .
all pathway term curation was performed prior to testing for associations.
literature search
a list of the pubmed abstract id's for each gene and each topic were obtained by searching pubmed using the appropriate terms and a constant date range.
the pubmed electronic date  was used to query a constant subset of the pubmed abstracts between 12/31/ <dig> and 12/31/ <dig>  whole text information was not used due to its incomplete and inconsistent availability.
to determine if results changed over time with a fixed chronological time frame and pre-set search parameters, three gene lists  and three curated term lists  were run repetitively every month for  <dig> months using the fixed dates above in order to assess if ongoing efforts at the ncbi to improve the search engine or annotation of the literature significantly influence our approach to literature mining.
term ranking
each topic list was analyzed independently.
within each topic set, specific terms were ranked according to how many times the term was associated with any of the genes in the gene list with respect to the total number of times the term is present in pubmed.
we applied two different methods to measure the degree of intersection between any term and the gene set.
first, we calculated the ratio of the number of abstracts containing any gene from the gene list and the term divided by all abstracts containing the term .
the second score  takes into account the number of abstract mentions against the entire target gene set .
in order to rank terms, we compared each terms score  to a distribution of scores between the same term and randomly selected gene sets.
in the case of the product of frequency, the log  more closely approximated a normal distribution thus justifying the mean and standard deviation statistics used .
testing against random gene sets
in order to provide a metric by which to interpret the rankings and determine the likelihood of finding a match given no association, we measure each topic score given the experimental set of genes against the distribution of scores from sets of genes chosen at random.
scoring  <dig>  such random sets against the topic set, we obtain estimates for the mean and standard deviation of the f score for each topic.
we tested if there was a significant difference in the statistics generated using or  <dig> random sets of genes containing the same number of genes as the experimental gene set.
RESULTS
we applied literature lab to lists of genes generated through microarray experiments to determine if such an approach provides biological insight.
we started with  <dig> lists of genes from previously published microarray experiments, one generated in vitro , and one generated from human tumors samples  to see if literature lab would correctly identify known biological processes and to how altering specific variables within literature lab impacts the results.
finally, we tested our method to a set of genes differentially expressed between local and metastatic prostate tumors and used immunohistochemistry to confirm the lead candidate.
literature lab associates "respiratory burst" with atra treatment of a leukaemia cell line
acute promyeloblastic leukemia  cells differentiate when stimulated with all-trans retinoic acid , a life-saving treatment for this disease  <cit> .
the physiological impact of atra is detected by an increase in nitro blue tetrazolium  reduction , an assay established to measure the production of oxygen intermediates associated with respiratory bursts in cells  <cit> .
we used microarray data available from a recent publication  <cit>  to identify the top genes with increased expression in hl <dig> cells  following atra exposure .
genes with increased expression were analyzed with literature lab using key term sets for cellular metabolism, physiology, and pathways.
within the top ranked key terms for both metabolism and physiology was "respiratory burst" .
this observation is confirmed by an increase in nitro blue tetrazolium  reduction, an established measure of the production of oxygen intermediates associated with respiratory bursts in cells  <cit> .
we tested the impact of gene set size and literature time frame on the successful association between "respiratory burst" and the genes with increased expression following atra.
while the specific literature time frame had relatively little impact on the association  , the number of differentially expressed genes included in the analysis did impact our results  specifically, the association between "respiratory burst" and the experimentally derived list of differentially expressed genes did not stabilize until greater than  <dig> genes were included.
while there are no obvious rules to guide the upper limit with respect to the number of genes to be used in literature lab, the variability of results observed with shorter gene lists suggests that gene lists numbering less than  <dig> will not provide robust results and for experiments comparing phenotypes with marked differences, gene lists greater than  <dig> are encouraged.
"hif and hypoxia" strongly associated with mtor inhibition
to test literature lab on a more challenging, in vivo derived dataset, we applied literature lab to a set of genes whose expression increased with the prostate-specific expression of myristoylated akt  and decreased when mice were treated with rad <dig>  an mtor inhibitor .
this experiment was well controlled, characterized in a published manuscript  <cit> , and has publicly available data .
we performed a 10-year literature query to identify pathway terms associated with the  <dig> genes previously found to have the expression pattern described above  <cit> .
literature lab identified "hif" and "hypoxia" terms in association with the list of genes thus highlighting the key biological insight discussed in the primary report .
while hif gene expression  was part of the original gene list and contributed to the strong association, literature lab found  <dig> and  <dig> additional genes associated with the key terms "hif" and "hypoxia", respectively, thus elevating these terms above others.
subsequent western blotting for protein expression of hif1α further supports the identified association  and demonstrates literature labs potential utility for data derived from animal models.
"matrix metalloproteinase 9" and "vegf" are associated with breast cancer prognosis
primary tumors perhaps provide the biggest challenge for analysis because of the additional associated technical and biological variation.
we applied literature lab to the set of  <dig> genes associated with outcome in two seminal papers predicting breast cancer recurrence using microarray data  <cit> .
interestingly, of the cellular pathways investigated, "matrix metalloproteinase" and "vegf" were strongly associated with the gene set .
this unbiased association is strongly supported by prior studies finding mmp  <cit>  and vegf  <cit>  activity strongly associated with the recurrent phenotype  <cit>  and supports literature lab's applicability to data from human samples.
literature drift
given the daily growth of biomedical literature, reproducible literature search results require investigators to designate a specific chronological interval within which they queried for associations between a gene list and key terms.
however, repeated queries using identical search criteria within a fixed interval demonstrated some lability hereto forward referred to as "literature drift".
in sequential runs over  <dig> months, we measured this drift using the absolute value of the percentage change in the lpf value.
literature drift resulted in a median difference between queries of  <dig> %.
this degree and rate of literature drift was dependent both on the gene list and key terms as three independent sets of genes queried for three sets of key terms experienced different degrees of drift .
by identifying the abstracts lost or gained between sequential runs, we identified a number of causal factors for literature drift:
• changes to the various components of the ncbi search engine that result in different results for the same query.
in particular, changes to the pubmed "phrase dictionary" are frequent and can yield different results for the same query at different points in time.
• the assignment of mesh terms to abstracts subsequent to the addition of abstracts to the pubmed database.
• the editing of pubmed abstracts so as to change the title or text.
• the occasional deletion of abstracts from the database.
many of these deletions appear to be the removal of duplicates added to the pubmed database in error.
importantly, while literature drift affected the numbers of abstracts linking a gene list with a term list, it had only minor effects on the final results of literature lab .
in addition, when the effects of literature drift were analyzed according to the strength of association between a key term and a set of genes; stronger associations were less likely to change when compared to associations with modest or no weight , 16% for moderate , and 27% for associations without weight .
thus while literature drift impacts the number of associations identified between the medial literature and a set of genes, our methodology minimizes its impact.
discovery of increased fosb in metastatic prostate cancer
while the literature lab results from the hl <dig> cell line, mpakt mouse, and breast cancer tumors are compelling, these sets had known biological associations and were used to help evaluate the methods herein described; they cannot be viewed as independent tests of our method.
in order to test literature lab's ability to identify valid gene-key term associations through automated literature interrogation, we applied literature lab to microarray data obtained after rna amplification of local and metastatic prostate cancer specimens.
we first identified the top  <dig> genes up-regulated in the metastatic samples in order to identify the top pathway associated with metastasis.
fosb was the top pathway and the only pathway term meeting heuristic criteria for a "strong" association .
expression of the fosb gene was associated with metastasis but the additional identification of associations between fosb and  <dig> other genes having increased expression in the metastatic tumors resulted in the top ranking of fosb and a "strong" heuristic label.
based on this result, we performed fosb immunohistochemistry on a tissue microarray containing benign, local malignant, and metastatic malignant prostate tissue.
nuclear fosb was significantly increased in the metastatic tumors compared to the locally invasive tumors  thus confirming the association highlighted by literature lab .
comparison with gominer, mesher and genecite
to determine how literature lab compares with existing, publicly available sample annotation and literature mining technologies, we imported the akt mouse and metastatic prostate cancer gene lists into gominer  <cit> , mesher  <cit> , and genecite  <cit> .
when the akt mouse gene set was imported into gominer,  <dig> go terms were found to be significantly associated  with the genes differentially expressed .
among the significant go terms were "response to hypoxia" and "glycolysis" thus demonstrating that gominer is able to identify similar underlying biology but these terms were among a large number of significant terms and not prominent.
here we note that gominer and literature lab represent different approaches  and, as such, the results are not directly comparable.
however, the purposes of literature lab and gominer are similar  and our results suggest that direct literature mining and our statistical approach provide insight that cannot be fully reproduced using gominer.
mesher did not find a significant association between "hypoxia", "hif", and "hypoxia p53" and the akt mouse gene list and there was very poor comparison between mesher and literature lab despite each method using mesh terms .
genecite identifies the abstracts associating the terms and genes but has no direct measure of significance other than the number of abstracts .
in addition, genecite has lower precision and recall when compared to literature lab due to the lack of a thesaurus for gene nomenclature .
for example, many of the abstracts for the cat gene refer to felines and cat scans and not to the cat gene.
when the genes differentially expressed between local and metastatic prostate cancers were imported into gominer and mesher, there was poor overlap with both.
fosb, the term significantly associated with metastatic prostate cancer and subsequently validated by immunohistochemistry, was either not present in the library  or not associated with the gene list .
results for genecite  exhibit the same limitations previously described for the akt mouse gene set.
discussion
full utilization of publicly available, data-rich resources remains a universal challenge in contemporary scientific investigation.
as technologies have diminished the cost and time associated with data collection, content within diverse repositories of data have increased exponentially.
the medical literature is one such data repository and a repository that continues to grow rapidly.
while investigators frequently use computational tools to interrogate genomic or gene expression data repositories, few use similar tools when reviewing the literature.
literature lab represents a method to comprehensively interrogate the literature for associations between a list of genes and a list of key terms in an unbiased manner in order to highlight potentially important biological processes implicated by the gene list.
while there are many methods by which to develop a gene list, we have designed literature lab to aid in the analysis of microarray experiments which frequently associate the expression of hundreds to thousands of seemingly unrelated genes with cellular behaviors, in vivo phenotypes, or disease outcomes.
we developed and refined our methodology using gene sets from previously published work and successfully tested literature lab on a novel dataset.
the pathway term fosb was ranked highest by literature lab and highlighted as having a "strong" association; an increase in nuclear fosb staining was subsequently confirmed with immunohistochemistry.
literature lab is complementary to the increasingly prevalent pathway oriented approaches to the analysis of microarray data .
as a general approach, these methods look for significant differential expression within a microarray experiment using pre-determined aggregations of genes  rather than individual genes  <cit> .
successful gene sets can identify underlying genetic abnormalities or signal transduction networks driving disease pathologies and help effectively bridge microarray data with biological significance  <cit> .
some pathway approaches methods use the literature and publicly available annotations  to develop gene sets and use these gene sets to interrogate expression data  <cit> .
literature lab offers the opportunity to use a gene set derived from microarray data to interrogate the biomedical literature without a priori classification or annotation.
as such, literature lab can appropriately interrogate the literature as it grows and evolves.
when compared to two publicly available methods of analysis , the results of literature lab were more comparable with gominer.
however, the statistical evaluation of associations identified by literature lab help improve the specificity of findings  while maintaining sensitivity .
it should be noted, however, that given the difference in the approaches, our results cannot be interpreted as demonstrating the superiority of literature lab over gominer
literature lab remains dependent on the strength of the term lists and while we have demonstrated the use of lists for metabolism, physiology, and pathways, further development is focused on creating lists to include disease, pharmacological agents, drug toxicities, and many additional classes.
we initially anticipated that fixing the chronological interval for a query would ensure exact reproduction of the results.
however, we identified literature drift within fixed retrospective intervals.
while the degree of literature drift seems to range from minimal to moderate depending on the specific gene list, literature lab successfully limits the effects of literature drift especially for associations identified as "strong" with the current heuristics.
thus, while literature drift is unlikely to have significant impact on the associations identified by literature lab, some variation in the specific weights and rankings of associations will change even when investigators define a fixed chronological interval within which they perform their query.
for this initial description, we focused on developing a robust measure of association, a relatively useful measure of significance, and heuristic rules to highlight the most important associations.
clearly, the specifics of our methods will be the subject of further investigation and refinement.
while we have identified some critical elements of success , work is ongoing to explore the effects of refining the genes based upon the statistical association between their expression and the phenotype, limiting literature lab to specific journals of high quality content, and increasing the number of sets of key terms with which to test the association between gene expression.
CONCLUSIONS
the methodology herein described for literature lab highlights the biomedical literature as a content-rich resource amenable to automated, comprehensive interrogation.
as with most complex data, successful comprehensive interrogation requires filtering out the noise and finding valuable information.
our current methods of gene annotation, key term curation, and literature interrogation, can find strong associations and are likely to benefit a diverse scientific community.
availability and requirement
the instructions, software, and data required to perform an analysis of a gene list using the techniques described herein can be obtained from .
sun's java runtime environment version  <dig>  or higher is required in order to run the software .
the software runs on both windows platforms  and linux platforms.
memory of  <dig> gb and  <dig> gb of available disk space are required .
in addition, the results of the analysis are presented in a microsoft excel spreadsheet for viewing on any system having a spreadsheet viewer capable of rendering the microsoft excel format.
• project name: literature lab
• project home page:
• operating system: windows  or linux
• programming language: e.g.
java
• other requirements: java runtime environment  <dig>  or higher
• license: not required for academic users
• any restrictions to use by non-academics: license needed from acumenta
authors' contributions
pgf oversaw all aspects of the research including development of the concept of literature lab, provision of microarray results for analysis by literature lab, evaluation of all results from literature lab, and drafting of the manuscript.
mgm helped develop and implement the software for literature lab.
das helped develop and implement literature lab.
ks oversaw all experiments with the leukemia cells, provided microarray data, provided the nbt data, and helped in the interpretation of literature lab results.
ddv performed, analyzed, and interpreted the fosb immunohistochemistry on prostate cancer specimens.
prm participated in the development, conceptualization, and implementation of literature lab.
ml oversaw the fosb immunohistochemistry and helped with the interpretation of literature lab results.
sct lead the development of literature lab from conceptualization to implementation, performed all literature lab analyses, and helped draft the manuscript.
supplementary material
additional file 1
supplemental methods.
click here for file
additional file 2
term lists.
excel spreadsheet of term lists used for physiology, metabolism, and pathway domains.
click here for file
additional file 3
gene thesaurus.
excel spreadsheet of all genes with all aliases used for all experiments in this manuscript.
click here for file
additional file 4
excluded terms.
excel spreadsheet of terms excluded from search due to ambiguity or limited associated information.
click here for file
additional file 5
effects of time frame.
standard literature lab interrogations performed on the hl <dig> gene set for key term lists for metabolism, pathways, and physiology.
the only difference between the two interrogations involved time frame: one search included citations from the first  <dig> years of the standard  <dig> year interval  and the other the latter  <dig> years .
log <dig>  was used to order key terms and standard heuristics were applied to identify associations as "strong" or "moderate".
click here for file
additional file 6
effects of gene set size.
standard literature lab interrogations performed on the hl <dig> gene set for key term lists for metabolism with increasing size of the gene set.
"metabolism-x" where x equals  <dig>   <dig>   <dig>   <dig>   <dig>   <dig>   <dig>  and  <dig> and represents the number of genes in the gene set.. log <dig>  was used to order key terms and standard heuristics were applied to identify associations as "strong" or "moderate".
click here for file
additional file 7
results of breast cancer gene set.
the 70-genes associated with outcome for localized breast cancer were used as a gene set and interrogated by literature lab for associations between members of the gene set and key terms included on the lists of physiology, metabolizm, and pathways.
log <dig>  was used to order key terms and standard heuristics were applied to identify associations as "strong" or "moderate".
click here for file
additional file 8
gominer.
excel spreadsheet of results from gominer using gene list from mpakt experiment annotated for overlap with literature lab.
click here for file
additional file 9
mesher.
excel spreadsheet of results from mesher using gene list from mpakt experiment annotated for overlap with literature lab.
click here for file
additional file 10
genecite mpakt.
excel spreadsheet of results from genecite using gene list from mpakt experiment annotated for overlap with literature lab.
click here for file
additional file 11
genecite metastatic prostate cancer.
excel spreadsheet of results from genecite using gene list from local v metastatic prostate cancer experiment annotated for overlap with literature lab.
click here for file
acknowledgements
p.g.f is a damon runyon-lily clinical investigator  and, in addition, was supported by nci grants ca <dig> and ca <dig> during the design, analysis, and preparation of this manuscript.
mgm, das, prm, and sct were supported by acumenta, inc during the design, analysis, and preparation of this manuscript.
all analysis of data by investigators supported by acumenta was performed in a blinded fashion; acumenta supported investigators were unaware of the experiments from which the gene lists uploaded into literature lab were derived.
ks was supported by the department of pediatric oncology during the design and analysis of this work.
ddv and ml were supported by the department of pathology at dana farber cancer institute during the design and analysis of this work.

 the accelerating expansion of biomedical research outpaces most individual attempts at comprehensive review even in relatively narrow fields. just as the vast sequence data available for the human and additional organisms require sophisticated genomic browsing tools, computational methods are required to thoroughly explore the corpus of biomedical literature. many computational methods for interrogating the scientific literature have been developed. these programs can be broadly defined as methods for information retrieval and those for information extraction. existing methods can identify significant association between individual genes and terms from the medical subject heading index and gene ontology databases, manually curated biological lists, or disease-specific lists. these prior methods demonstrate that genes with disease-specific differential expression can be strongly correlated with key terms within the medical literature. in addition, gene-gene associations within the literature have been combined with multiple available databases to extend associations beyond the literature alone. while these and similar approaches have underscored the potential of automated literature searching to facilitate discovery, few have provided both methods for assessing the statistical strength of identified associations and supported their methods with experimental validation. here, we describe and apply "literature lab", a method of automated data retrieval confined to publicly available citations and abstracts. literature lab statistically assesses identified associations within the corpus of medical literature between sets of experimentally derived genes and key terms derived from curated or mesh lists. we demonstrate that our methodology can identify previously reported relationships and can result in discovery. molecular methods gene expression sets literature lab was applied to three gene sets derived from previously published microarray data and one gene set from as yet, unpublished dataset to determine if literature mining identified important metabolic, physiologic, or pathway activity. these gene sets include, 1) the top genes with increased expression during in vitro exposure of human leukemia cells to atra, 2) the genes differentially regulated with a transgenic model of prostate neoplasia following exposure to rad 3) the genes used to predict outcome in localized node-negative breast cancer, and 4) genes differentially expressed between malignant epithelial cells in local v. metastatic prostate cancers. the genes lists are included in additional files and a full description of the development of each gene set is in the supplemental methods. western blots protein from fresh ventral prostates were extracted in ripa buffer separated by gel electrophoresis and transferred to nitrocellulose membrane as described. membranes were blotted with anti-hif-1α and anti-tubulin. blots were scanned and intensities were measured. immunohistochemistry a prostate tissue microarray containing samples of benign prostate epithelium, locally invasive prostate adenocarcinoma, and metastatic prostate adenocarcinoma were stained for fosb. the fosb staining was performed as previously described. briefly, micrometer sections were cut from the tma block, deparaffinized, rehydrated and subjected to microwaving in 10-mm citrate buffer in a 750w oven, for minutes. the polyclonal anti-fosb primary antibody, was incubated at room temperature in an automated stainer. antigen-antibody reaction was revealed with standardized development times, using the streptavidin method with diaminobenzidine as substrate. meyer hematoxylin was used as nuclear counterstaining. fosb nuclear positivity was scored on a total of nuclei per sample. the statistical difference in fosb staining between the unpaired populations was assessed using a two-tailed mann whitney test. application of established literature mining tools mesher data for the gene lists was obtained from the website by supplying the affymetrix u133plus ids corresponding to the list of genes using a p-value threshold of and no p-value correction. gominer data for the gene lists was obtained by following the instructions for downloading and installing the gominer application and sql database from. a list of gene symbols for genes in the affymetrix u133plus probe set list was used for the'total' file and lists of symbols for the experimental genes used for the'changed' file. other gominer settings used were the default values defined in the gominer application. implementation overview to highlight and prioritize cellular physiology, metabolism, or pathways differentially active within a microarray experiment, literature lab uses an experimentally derived gene list, pre-defined sets of non-ambiguous key terms, inclusive gene nomenclature, comprehensive literature interrogation, and a comparison of the experimental gene set results with randomly generated gene sets. for each experimentally derived gene set, literature lab performs an automated literature search to determine the number of abstracts listing any of the identified genes and each term within the term list. specific domains include those for cell metabolism, cell physiology, and cellular pathways. terms are ranked with a measure of association between genes and terms known as the product of frequency. the strength of the relationship between an experimental gene set and a term is determined by comparing the log of the sum of the pf values for the experimental gene set with a distribution of the same values for random gene sets of the same size as the experimental gene set. a score representing the number of standard deviations above or below the mean value for the random gene sets is assigned to each term and terms are ranked in decreasing order of this score. to highlight terms with particularly strong association with the gene list, we developed heuristic rules for labeling the relationships as "strong" or "moderate". overall architecture of the software implementation the software is implemented using sun's java programming language using the eclipse software development environment. the software consists of three major components: • a series of programs which query pubmed for each term and each gene in the gene thesaurus using ncbi's entrez programming utilities and sun's jaxb xml binding classes. the result for each term or gene is a file containing a list of the pubmed ids for the term or gene. in addition, summary files containing counts of the number of abstracts for each gene and for each term are prepared. • a series of programs which identify the pubmed ids in common for each term/gene combination from the lists of ids for each term and each gene. the results of these programs are files containing the pubmed ids for each term/gene combination and files containing summary counts of the number of abstracts for each term/gene combination. • a series of programs for preparing the analysis of a specific gene list. these programs compute the described statistics for the gene list and for one thousand random gene sets of the same size as the specific gene list being processed. the final step of this process prepares a microsoft® excel spreadsheet containing the results of the analysis. the apache software foundation's poi classes were used to develop a set of rules required for more specific automated searching. in subsequent literature searches, gene terms consisting entirely of numerals, of three characters or less, or that were identified as excessively ambiguous through manual review were excluded. algorithm-based disambiguation was not used. while the database of gene terms is frequently updated, for all experiments herein described, the gene thesaurus updated last on 12/30/ was used for all experiments presented. of the gene terms were specifically excluded based upon the manual review. all gene term curation was performed prior to testing for associations. topic list curation topic lists are sets of terms, each of which is to be tested for significant associations with the experimentally derived gene set. topic lists used in the experiments were defined using terms from the mesh thesaurus provided by the national library of medicine. the list of terms for each topic set consisted of all terms and all descendents as listed in mesh. the topic sets used in these experiments were cell metabolism and cell physiology. a few of the descendents of these mesh terms were ignored because very few abstracts were associated with the term. when using mesh terms to search pubmed, the search syntax used "mesh subheading explosion" so that the resulting list of abstracts included abstracts coded with descendents of the mesh term as well as abstracts coded with the mesh term itself. in addition, a topic set entitled "pathways" was derived using the pathway descriptions from biocarta with some additional curation and testing. all pathway term curation was performed prior to testing for associations. literature search a list of the pubmed abstract id's for each gene and each topic were obtained by searching pubmed using the appropriate terms and a constant date range. the pubmed electronic date was used to query a constant subset of the pubmed abstracts between 12/31/ and 12/31/ whole text information was not used due to its incomplete and inconsistent availability. to determine if results changed over time with a fixed chronological time frame and pre-set search parameters, three gene lists and three curated term lists were run repetitively every month for months using the fixed dates above in order to assess if ongoing efforts at the ncbi to improve the search engine or annotation of the literature significantly influence our approach to literature mining. term ranking each topic list was analyzed independently. within each topic set, specific terms were ranked according to how many times the term was associated with any of the genes in the gene list with respect to the total number of times the term is present in pubmed. we applied two different methods to measure the degree of intersection between any term and the gene set. first, we calculated the ratio of the number of abstracts containing any gene from the gene list and the term divided by all abstracts containing the term. the second score takes into account the number of abstract mentions against the entire target gene set. in order to rank terms, we compared each terms score to a distribution of scores between the same term and randomly selected gene sets. in the case of the product of frequency, the log more closely approximated a normal distribution thus justifying the mean and standard deviation statistics used. testing against random gene sets in order to provide a metric by which to interpret the rankings and determine the likelihood of finding a match given no association, we measure each topic score given the experimental set of genes against the distribution of scores from sets of genes chosen at random. scoring such random sets against the topic set, we obtain estimates for the mean and standard deviation of the f score for each topic. we tested if there was a significant difference in the statistics generated using or random sets of genes containing the same number of genes as the experimental gene set.  we applied literature lab to lists of genes generated through microarray experiments to determine if such an approach provides biological insight. we started with lists of genes from previously published microarray experiments, one generated in vitro, and one generated from human tumors samples to see if literature lab would correctly identify known biological processes and to how altering specific variables within literature lab impacts the results. finally, we tested our method to a set of genes differentially expressed between local and metastatic prostate tumors and used immunohistochemistry to confirm the lead candidate. literature lab associates "respiratory burst" with atra treatment of a leukaemia cell line acute promyeloblastic leukemia cells differentiate when stimulated with all-trans retinoic acid, a life-saving treatment for this disease. the physiological impact of atra is detected by an increase in nitro blue tetrazolium reduction, an assay established to measure the production of oxygen intermediates associated with respiratory bursts in cells. we used microarray data available from a recent publication to identify the top genes with increased expression in hl cells following atra exposure. genes with increased expression were analyzed with literature lab using key term sets for cellular metabolism, physiology, and pathways. within the top ranked key terms for both metabolism and physiology was "respiratory burst". this observation is confirmed by an increase in nitro blue tetrazolium reduction, an established measure of the production of oxygen intermediates associated with respiratory bursts in cells. we tested the impact of gene set size and literature time frame on the successful association between "respiratory burst" and the genes with increased expression following atra. while the specific literature time frame had relatively little impact on the association, the number of differentially expressed genes included in the analysis did impact our results specifically, the association between "respiratory burst" and the experimentally derived list of differentially expressed genes did not stabilize until greater than genes were included. while there are no obvious rules to guide the upper limit with respect to the number of genes to be used in literature lab, the variability of results observed with shorter gene lists suggests that gene lists numbering less than will not provide robust results and for experiments comparing phenotypes with marked differences, gene lists greater than are encouraged. "hif and hypoxia" strongly associated with mtor inhibition to test literature lab on a more challenging, in vivo derived dataset, we applied literature lab to a set of genes whose expression increased with the prostate-specific expression of myristoylated akt and decreased when mice were treated with rad an mtor inhibitor. this experiment was well controlled, characterized in a published manuscript, and has publicly available data. we performed a 10-year literature query to identify pathway terms associated with the genes previously found to have the expression pattern described above. literature lab identified "hif" and "hypoxia" terms in association with the list of genes thus highlighting the key biological insight discussed in the primary report. while hif gene expression was part of the original gene list and contributed to the strong association, literature lab found and additional genes associated with the key terms "hif" and "hypoxia", respectively, thus elevating these terms above others. subsequent western blotting for protein expression of hif1α further supports the identified association and demonstrates literature labs potential utility for data derived from animal models. "matrix metalloproteinase 9" and "vegf" are associated with breast cancer prognosis primary tumors perhaps provide the biggest challenge for analysis because of the additional associated technical and biological variation. we applied literature lab to the set of genes associated with outcome in two seminal papers predicting breast cancer recurrence using microarray data. interestingly, of the cellular pathways investigated, "matrix metalloproteinase" and "vegf" were strongly associated with the gene set. this unbiased association is strongly supported by prior studies finding mmp and vegf activity strongly associated with the recurrent phenotype and supports literature lab's applicability to data from human samples. literature drift given the daily growth of biomedical literature, reproducible literature search results require investigators to designate a specific chronological interval within which they queried for associations between a gene list and key terms. however, repeated queries using identical search criteria within a fixed interval demonstrated some lability hereto forward referred to as "literature drift". in sequential runs over months, we measured this drift using the absolute value of the percentage change in the lpf value. literature drift resulted in a median difference between queries of %. this degree and rate of literature drift was dependent both on the gene list and key terms as three independent sets of genes queried for three sets of key terms experienced different degrees of drift. by identifying the abstracts lost or gained between sequential runs, we identified a number of causal factors for literature drift: • changes to the various components of the ncbi search engine that result in different results for the same query. in particular, changes to the pubmed "phrase dictionary" are frequent and can yield different results for the same query at different points in time. • the assignment of mesh terms to abstracts subsequent to the addition of abstracts to the pubmed database. • the editing of pubmed abstracts so as to change the title or text. • the occasional deletion of abstracts from the database. many of these deletions appear to be the removal of duplicates added to the pubmed database in error. importantly, while literature drift affected the numbers of abstracts linking a gene list with a term list, it had only minor effects on the final results of literature lab. in addition, when the effects of literature drift were analyzed according to the strength of association between a key term and a set of genes; stronger associations were less likely to change when compared to associations with modest or no weight, 16% for moderate, and 27% for associations without weight. thus while literature drift impacts the number of associations identified between the medial literature and a set of genes, our methodology minimizes its impact. discovery of increased fosb in metastatic prostate cancer while the literature lab results from the hl cell line, mpakt mouse, and breast cancer tumors are compelling, these sets had known biological associations and were used to help evaluate the methods herein described; they cannot be viewed as independent tests of our method. in order to test literature lab's ability to identify valid gene-key term associations through automated literature interrogation, we applied literature lab to microarray data obtained after rna amplification of local and metastatic prostate cancer specimens. we first identified the top genes up-regulated in the metastatic samples in order to identify the top pathway associated with metastasis. fosb was the top pathway and the only pathway term meeting heuristic criteria for a "strong" association. expression of the fosb gene was associated with metastasis but the additional identification of associations between fosb and other genes having increased expression in the metastatic tumors resulted in the top ranking of fosb and a "strong" heuristic label. based on this result, we performed fosb immunohistochemistry on a tissue microarray containing benign, local malignant, and metastatic malignant prostate tissue. nuclear fosb was significantly increased in the metastatic tumors compared to the locally invasive tumors thus confirming the association highlighted by literature lab. comparison with gominer, mesher and genecite to determine how literature lab compares with existing, publicly available sample annotation and literature mining technologies, we imported the akt mouse and metastatic prostate cancer gene lists into gominer, mesher, and genecite. when the akt mouse gene set was imported into gominer, go terms were found to be significantly associated with the genes differentially expressed. among the significant go terms were "response to hypoxia" and "glycolysis" thus demonstrating that gominer is able to identify similar underlying biology but these terms were among a large number of significant terms and not prominent. here we note that gominer and literature lab represent different approaches and, as such, the results are not directly comparable. however, the purposes of literature lab and gominer are similar and our results suggest that direct literature mining and our statistical approach provide insight that cannot be fully reproduced using gominer. mesher did not find a significant association between "hypoxia", "hif", and "hypoxia p53" and the akt mouse gene list and there was very poor comparison between mesher and literature lab despite each method using mesh terms. genecite identifies the abstracts associating the terms and genes but has no direct measure of significance other than the number of abstracts. in addition, genecite has lower precision and recall when compared to literature lab due to the lack of a thesaurus for gene nomenclature. for example, many of the abstracts for the cat gene refer to felines and cat scans and not to the cat gene. when the genes differentially expressed between local and metastatic prostate cancers were imported into gominer and mesher, there was poor overlap with both. fosb, the term significantly associated with metastatic prostate cancer and subsequently validated by immunohistochemistry, was either not present in the library or not associated with the gene list. results for genecite exhibit the same limitations previously described for the akt mouse gene set. discussion full utilization of publicly available, data-rich resources remains a universal challenge in contemporary scientific investigation. as technologies have diminished the cost and time associated with data collection, content within diverse repositories of data have increased exponentially. the medical literature is one such data repository and a repository that continues to grow rapidly. while investigators frequently use computational tools to interrogate genomic or gene expression data repositories, few use similar tools when reviewing the literature. literature lab represents a method to comprehensively interrogate the literature for associations between a list of genes and a list of key terms in an unbiased manner in order to highlight potentially important biological processes implicated by the gene list. while there are many methods by which to develop a gene list, we have designed literature lab to aid in the analysis of microarray experiments which frequently associate the expression of hundreds to thousands of seemingly unrelated genes with cellular behaviors, in vivo phenotypes, or disease outcomes. we developed and refined our methodology using gene sets from previously published work and successfully tested literature lab on a novel dataset. the pathway term fosb was ranked highest by literature lab and highlighted as having a "strong" association; an increase in nuclear fosb staining was subsequently confirmed with immunohistochemistry. literature lab is complementary to the increasingly prevalent pathway oriented approaches to the analysis of microarray data. as a general approach, these methods look for significant differential expression within a microarray experiment using pre-determined aggregations of genes rather than individual genes. successful gene sets can identify underlying genetic abnormalities or signal transduction networks driving disease pathologies and help effectively bridge microarray data with biological significance. some pathway approaches methods use the literature and publicly available annotations to develop gene sets and use these gene sets to interrogate expression data. literature lab offers the opportunity to use a gene set derived from microarray data to interrogate the biomedical literature without a priori classification or annotation. as such, literature lab can appropriately interrogate the literature as it grows and evolves. when compared to two publicly available methods of analysis, the results of literature lab were more comparable with gominer. however, the statistical evaluation of associations identified by literature lab help improve the specificity of findings while maintaining sensitivity. it should be noted, however, that given the difference in the approaches, our results cannot be interpreted as demonstrating the superiority of literature lab over gominer literature lab remains dependent on the strength of the term lists and while we have demonstrated the use of lists for metabolism, physiology, and pathways, further development is focused on creating lists to include disease, pharmacological agents, drug toxicities, and many additional classes. we initially anticipated that fixing the chronological interval for a query would ensure exact reproduction of the results. however, we identified literature drift within fixed retrospective intervals. while the degree of literature drift seems to range from minimal to moderate depending on the specific gene list, literature lab successfully limits the effects of literature drift especially for associations identified as "strong" with the current heuristics. thus, while literature drift is unlikely to have significant impact on the associations identified by literature lab, some variation in the specific weights and rankings of associations will change even when investigators define a fixed chronological interval within which they perform their query. for this initial description, we focused on developing a robust measure of association, a relatively useful measure of significance, and heuristic rules to highlight the most important associations. clearly, the specifics of our methods will be the subject of further investigation and refinement. while we have identified some critical elements of success, work is ongoing to explore the effects of refining the genes based upon the statistical association between their expression and the phenotype, limiting literature lab to specific journals of high quality content, and increasing the number of sets of key terms with which to test the association between gene expression.  the methodology herein described for literature lab highlights the biomedical literature as a content-rich resource amenable to automated, comprehensive interrogation. as with most complex data, successful comprehensive interrogation requires filtering out the noise and finding valuable information. our current methods of gene annotation, key term curation, and literature interrogation, can find strong associations and are likely to benefit a diverse scientific community. availability and requirement the instructions, software, and data required to perform an analysis of a gene list using the techniques described herein can be obtained from. sun's java runtime environment version or higher is required in order to run the software. the software runs on both windows platforms and linux platforms. memory of gb and gb of available disk space are required. in addition, the results of the analysis are presented in a microsoft excel spreadsheet for viewing on any system having a spreadsheet viewer capable of rendering the microsoft excel format. • project name: literature lab • project home page: • operating system: windows or linux • programming language: e.g. java • other requirements: java runtime environment or higher • license: not required for academic users • any restrictions to use by non-academics: license needed from acumenta authors' contributions pgf oversaw all aspects of the research including development of the concept of literature lab, provision of microarray results for analysis by literature lab, evaluation of all results from literature lab, and drafting of the manuscript. mgm helped develop and implement the software for literature lab. das helped develop and implement literature lab. ks oversaw all experiments with the leukemia cells, provided microarray data, provided the nbt data, and helped in the interpretation of literature lab results. ddv performed, analyzed, and interpreted the fosb immunohistochemistry on prostate cancer specimens. prm participated in the development, conceptualization, and implementation of literature lab. ml oversaw the fosb immunohistochemistry and helped with the interpretation of literature lab results. sct lead the development of literature lab from conceptualization to implementation, performed all literature lab analyses, and helped draft the manuscript. supplementary material additional file 1 supplemental methods. click here for file additional file 2 term lists. excel spreadsheet of term lists used for physiology, metabolism, and pathway domains. click here for file additional file 3 gene thesaurus. excel spreadsheet of all genes with all aliases used for all experiments in this manuscript. click here for file additional file 4 excluded terms. excel spreadsheet of terms excluded from search due to ambiguity or limited associated information. click here for file additional file 5 effects of time frame. standard literature lab interrogations performed on the hl gene set for key term lists for metabolism, pathways, and physiology. the only difference between the two interrogations involved time frame: one search included citations from the first years of the standard year interval and the other the latter years. log was used to order key terms and standard heuristics were applied to identify associations as "strong" or "moderate". click here for file additional file 6 effects of gene set size. standard literature lab interrogations performed on the hl gene set for key term lists for metabolism with increasing size of the gene set. "metabolism-x" where x equals and and represents the number of genes in the gene set.. log was used to order key terms and standard heuristics were applied to identify associations as "strong" or "moderate". click here for file additional file 7 results of breast cancer gene set. the 70-genes associated with outcome for localized breast cancer were used as a gene set and interrogated by literature lab for associations between members of the gene set and key terms included on the lists of physiology, metabolizm, and pathways. log was used to order key terms and standard heuristics were applied to identify associations as "strong" or "moderate". click here for file additional file 8 gominer. excel spreadsheet of results from gominer using gene list from mpakt experiment annotated for overlap with literature lab. click here for file additional file 9 mesher. excel spreadsheet of results from mesher using gene list from mpakt experiment annotated for overlap with literature lab. click here for file additional file 10 genecite mpakt. excel spreadsheet of results from genecite using gene list from mpakt experiment annotated for overlap with literature lab. click here for file additional file 11 genecite metastatic prostate cancer. excel spreadsheet of results from genecite using gene list from local v metastatic prostate cancer experiment annotated for overlap with literature lab. click here for file acknowledgements p.g.f is a damon runyon-lily clinical investigator and, in addition, was supported by nci grants ca and ca during the design, analysis, and preparation of this manuscript. mgm, das, prm, and sct were supported by acumenta, inc during the design, analysis, and preparation of this manuscript. all analysis of data by investigators supported by acumenta was performed in a blinded fashion; acumenta supported investigators were unaware of the experiments from which the gene lists uploaded into literature lab were derived. ks was supported by the department of pediatric oncology during the design and analysis of this work. ddv and ml were supported by the department of pathology at dana farber cancer institute during the design and analysis of this work.
the biomedical literature is a rich source of associative information but too vast for complete manual review. this work demonstrates that literature lab can discover key biological processes by identifying meritorious associations between experimentally derived gene sets and key terms within the biomedical literature. we have developed an automated method of literature interrogation called "literature lab" that identifies and ranks associations existing in the literature between gene sets, such as those derived from microarray experiments, and curated sets of key terms terms, etc). when applied to a novel set of genes differentially expressed between locally invasive and metastatic prostate cancer, literature lab identified a strong association between the pathway term "fosb" and genes with increased expression in metastatic prostate cancer. when applied to the genesets from the published data including an in vitro experiment, an in vivo mouse experiment, and an experiment with human tumor samples, literature lab correctly identified known biological processes occurring within each experiment. literature lab was developed using differentially expressed gene sets from three previously published cancer experiments and tested on a fourth, novel gene set. immunohistochemistry subsequently confirmed increased nuclear fosb staining in metastatic compared to locally invasive prostate cancers.
 the biomedical literature is a rich source of associative information but too vast for complete manual review. we have developed an automated method of literature interrogation called "literature lab" that identifies and ranks associations existing in the literature between gene sets, such as those derived from microarray experiments, and curated sets of key terms terms, etc).  literature lab was developed using differentially expressed gene sets from three previously published cancer experiments and tested on a fourth, novel gene set. when applied to the genesets from the published data including an in vitro experiment, an in vivo mouse experiment, and an experiment with human tumor samples, literature lab correctly identified known biological processes occurring within each experiment. when applied to a novel set of genes differentially expressed between locally invasive and metastatic prostate cancer, literature lab identified a strong association between the pathway term "fosb" and genes with increased expression in metastatic prostate cancer. immunohistochemistry subsequently confirmed increased nuclear fosb staining in metastatic compared to locally invasive prostate cancers.  this work demonstrates that literature lab can discover key biological processes by identifying meritorious associations between experimentally derived gene sets and key terms within the biomedical literature. 