Functional gene clustering via gene annotation sentences , MeSH and GO keywords from biomedical literature Gene function annotation remains a key challenge in modern biology . This is especially true for high-throughput techniques such as gene expression experiments . Vital information about genes is available electronically from biomedical literature in the form of full texts and abstracts . In addition , various publicly available databases -LRB- such as GenBank , Gene Ontology and Entrez -RRB- provide access to gene-related information at different levels of biological organization , granularity and data format . This information is being used to assess and interpret the results from high-throughput experiments . To improve keyword extraction for annotational clustering and other types of analyses , we have developed a novel text mining approach , which is based on keywords identified at the level of gene annotation sentences -LRB- in particular sentences characterizing biological function -RRB- instead of entire abstracts . Further , to improve the expressiveness and usefulness of gene annotation terms , we investigated the combination of sentence-level keywords with terms from the Medical Subject Headings -LRB- MeSH -RRB- and Gene Ontology -LRB- GO -RRB- resources . We find that sentence-level keywords combined with MeSH terms outperforms the typical ` baseline ' set-up -LRB- term frequencies at the level of abstracts -RRB- by a significant margin , whereas the addition of GO terms improves matters only marginally . We validated our approach on the basis of a manually annotated corpus of 200 abstracts generated on the basis of 2 cancer categories and 10 genes per category . We applied the method in the context of three sets of differentially expressed genes obtained from pediatric brain tumor samples . This analysis suggests novel interpretations of discovered gene expression patterns . Background In recent years , increasing amounts of biological data have become available through techniques such as DNA microarrays and other high-throughput gene and protein assays . -LSB- 1,2 -RSB- As large numbers of genes can be included in such studies , the task of assigning meaningful biological function to gene patterns or gene clusters is a considerable challenge . Typical analyses using supervised -LRB- classification -RRB- or unsupervised -LRB- clustering -RRB- methods require the user to incorporate the necessary background knowledge . -LSB- 3 -RSB- This ability to incorporate background knowledge is fundamental to effective and efficient scientific discovery . A substantial amount of biomedical knowledge is captured in free-text form in abstracts and full-text articles and also in specialized biological information systems such as Gene Ontology -LRB- GO -RRB- -LSB- 4 -RSB- , Medical Subject Headings -LRB- MeSH -RRB- -LSB- 5 -RSB- , Database of Interacting Proteins -LRB- DIP -RRB- -LSB- 6 -RSB- etc. . Until only a few years ago , human reasoning was the primary method for the extracting , synthesizing and interpreting the information contained in the biomedical literature and supporting biological information systems . However , in recent years the number of online documents -LRB- and other biological information repositories -RRB- has grown tremendously . This is both an opportunity and a challenge . On one hand , such resources facilitate automated processing of the knowledge and information contained in these documents . On the other hand , such processing poses considerable algorithmic and computational challenges -LSB- 7 -RSB- . For example , the biomedical abstract database MEDLINE -LSB- 8 -RSB- currently contains about 15 million citations and about 40 000 citations are added monthly . Text mining is the application of techniques from machine learning , natural language processing -LRB- NLP -RRB- , information extraction and statistical/mathematical approaches to automated extraction of useful knowledge from text -LSB- 9 -RSB- . Text mining of biomedical literature has been applied successfully to various biological problems . Many studies focus on protein-protein -LSB- 10 -- 13 -RSB- and gene-protein interactions . -LSB- 14 -RSB- Other specific relationships between biological entities such as sub-cellular localization of proteins -LSB- 15,16 -RSB- , molecular binding relationships -LSB- 17 -RSB- and interaction between genes and drugs -LSB- 18 -RSB- are also explored . Text analysis of biomedical literature has also been applied successfully to incorporate functional information of gene expression data -LSB- 19 -- 23 -RSB- . For example , MedMOLE -LSB- 19 -RSB- identifies the functions among a group of genes by simple text clustering of entire MEDLINE documents associated with the genes . Blaschke et al. -LSB- 20 -RSB- extracted information about the common biological characteristics of gene clusters from MEDLINE using a statistical term weighting approach . This method returns an ordered set of keywords with a high probability of occurrence in abstracts . Liu et al. -LSB- 24 -RSB- extended this approach by clustering such keywords to find gene-to-gene relationships . Clustering genes by functional keyword association can provide direct information about the nature of genes and their functional association -LSB- 25 -RSB- . However , the quality of the keyword lists extracted from the biomedical literature for each gene significantly affects the clustering results . Commonly , these approaches represent genes by extracting keywords from entire abstracts -LSB- 25 -RSB- . These keywords may undergo transformations such as weighting or dimension reduction with the goal of improving clustering quality and efficiency . However , gene clustering using entire abstracts has the following main drawbacks . -LRB- a -RRB- Abstracts normally contain a large number of irrelevant sentences . These sentences may influence the clustering process and are likely to obscure information useful for gene annotation . -LRB- b -RRB- The number of unique terms in abstracts is typically very large . This requires the ability to deal with sparse data spaces or methods for dimensionality reduction . -LRB- c -RRB- Dimension reduction methods such as principal component analysis or signal-to-noise methods increase the computational complexity , may lead to the loss of important keywords and do not guarantee that the reduced dimensionality will yield better clustering/annotation information . Also , the composite features may be hard to interpret . To avoid the above drawbacks and improve the clustering process , we decided to use gene annotation sentences from abstracts instead of using full abstracts to extract the keywords . Current NLP techniques allow such sentence extraction from documents . Using clustering on the basis of sentence-extraction techniques has the advantage of avoiding complex dimensionality reduction and term weighting techniques . Further , this approach is likely to yield more specific terms which are easier to interpret . We first extracted the potential sentences describing gene annotation information from abstracts using a NLP method utilizing gene/protein name dictionaries and pattern-matching-based rules . In addition to this sentence-keywords approach , we carried out two further experiments involving MeSH terms and GO terms as supplementary keywords . Hence , in our method , each gene is represented by set of keywords extracted from sentences , MeSH terms and GO terms . To demonstrate the usefulness of the proposed text mining methods , we performed hierarchical clustering of a gene × keyword matrix to find functionally discrete sub-groups of genes . The overall experimental design and its components are illustrated in the Figure 1 . We validated the performance of keywords extracted by our method using a manually annotated corpus of 200 abstracts . We also evaluated the usefulness of our method by sorting differentially expressed genes from a microarray experiment into functional sub-groups . The objective of our gene clustering process using functional keywords is to identify and summarize potential functional gene groups and to complement the conventional gene expression data clustering tasks . Methodology Gene/Protein name and synonym dictionary creation One of the major obstacles in biomedical literature processing is the variety of names each gene or protein is known by . To address this problem in the present study , we developed a gene/protein name dictionary . Essentially , each entry of this dictionary consists of a preferred -LRB- or canonical -RRB- name for a gene/protein and a list of synonyms used for this gene/protein . The dictionary was created on the basis of the Entrez Gene -LSB- 26 -RSB- -LRB- previously LocusLink -RRB- database , one the most stable and complete sources of information on genes . Since our study is part of a wider investigation in the context of human brain tumor research , we focused specifically on human genes/proteins . We developed PERL scripts to extract and select from the Entrez Gene entries the official symbols as preferred name of the gene and other aliases as known synonyms . In addition , we augmented the dictionary with relevant synonyms from other publicly available databases including GeneCards -LSB- 27 -RSB- , SwissProt -LSB- 28 -RSB- , GoldenPath -LSB- 29 -RSB- and HUGO -LSB- 30 -RSB- . The final dictionary contains 26 731 unique human gene/protein names and 274 845 synonym names . Keywords extraction from biomedical literature In our study each gene is represented by a list of keywords extracted from MEDLINE abstract sentences , MeSH terms and GO terms . The procedure for extracting keywords from each data source is discussed below . MEDLINE abstracts keywords extraction To extract the keywords associated with each abstract , we decided to use gene annotation sentences from the abstracts instead of constructing a large keyword vector based on the entire abstract . The assumption is that the information given on sentence-level is much more specific and therefore useful to characterize the function of the genes . Only sentences that contain one or more genes reference from our gene lists will be considered as gene annotation sentences , all other sentences are discarded from the analysis . We applied the following three steps to extract sentence-level keywords -LRB- 1 -RRB- gene-name normalization , -LRB- 2 -RRB- sentence filtering , and -LRB- 3 -RRB- keyword extraction . Gene-name normalization This process replaces all the gene names in the abstract with its unique canonical identifier -LRB- Entrez gene ID -RRB- using the gene-synonym dictionary specially constructed for this study . Sentence filtering This process extracts all the gene annotation sentences from abstracts that contain one or more gene names from our gene lists using regular-expression pattern matching rules . We used different regular expressions -LRB- which rely on matching of pre-defined patterns or rules such as arrangement of gene/protein names with articles , prepositions and other keywords -RRB- to filter sentences containing one to three genes . We defined our regular expressions as nouns describing agents , passive verbs , active verbs and nouns describing actions . Table 1 -LRB- in supplementary material -RRB- depicts an example for each type of expression . For example , the regular expression -LRB- $ gene @ -LCB- 0,6 -RCB- $ action -LRB- of | with -RRB- @ -LCB- 0,2 -RCB- $ gene -RRB- extracts sentences that match the structure shown below the expression . The notational construct ` A → B → ... ' is interpreted as ` A followed by B followed by ... ' . gene name → 0-6 words → action verb → ` of ' or ` with ' → 0-2 words → gene name Sentence keyword extraction Sentences containing one or more gene names were parsed using the Brill part-of-speech tagger . -LSB- 31 -RSB- This program labels each word in a sentence with its part-of-speech information such as word category like noun , verb , adjective , preposition , etc. . This information plays a critical role in identifying corresponding noun and verb phrases . Then , with a simple PERL program , noun phrases containing gene names were filtered out and the remaining noun phrases and verb phrases were extracted as keywords . Initial tests showed that certain keywords were common for most of the genes in the list -LRB- e.g. , activates , associates , stimulates etc. -RRB- . We manually removed these common keyword words from the list . The following example illustrates this process : Sentence BRCA1 physically associates with p53 and stimulates its transcriptional activity . Brill-POS-tagged sentence BRCA1/NNP physically/RB associates/VBZ with/IN p53/NN and/CC stimulates/VBZ its/PRP $ transcriptional/JJ activity/NN . / . Sentence keywords associates , stimulates , transcription activity Sentence keywords after manual curation transcription activity MeSH keywords extraction To extract MeSH keywords , we searched for the gene names in our gene lists in the title and abstract of MEDLINE citations related to each gene and extracted the associated MeSH terms for each gene . The extracted gene-MeSH term list was represented by scores indicating the frequency of gene-MeSH term co-occurrence . Initial tests showed that certain MeSH keywords in the list were common biological terms and less useful from the point of view of gene annotation -LRB- e.g. , human , DNA , animal , Support U.S Govt etc. -RRB- . A collection of MeSH stop words was created manually and these terms were removed from the gene-MeSH term lists . Finally , from the thus filtered gene-MeSH lists , the 20 highest-frequency MeSH terms associated with each gene were taken as MeSH keywords associated with each gene . For example the MeSH keywords associated with a gene `` FOS '' in our gene list are oncogene , felypressin , transcription-factor , thermoreceptors , DNA-binding , antibiosis , inflammatory-response , zinc-fingers , gene-regulation , and neuronal-plasticity . GO keyword extraction We used the GO keywords information incorporated in Gene Ontology -LSB- Error ! Bookmark not defined . -RSB- to extract GO keywords associated with each gene . Out of the three GO annotation categories we included only molecular function and biological process as we believe that cellular component -LRB- e.g. nucleus , cell membrane etc. -RRB- is less important for characterizing genes in the context of this study . Further , due to the hierarchical nature of GO and multiple inheritance in the GO structure , we consider with every ancestor up to level 2 in the GO tree in assigning GO keywords . This enables us to use more generalized GO terms . For example the GO keywords associated with the gene `` FOS '' in our gene list are protein-dimerization , DNA binding , RNA polymerase , transcription factor , DNA methylation , inflammatory-response , and nucleus . Keyword representation and calculation of numeric vectors After the keyword extraction phase , each gene was described by a list of keywords extracted from MEDLINE abstract sentences , MeSH terms and GO terms . These keyword vectors then served as a basis for clustering -LRB- i.e. , unsupervised class discovery -RRB- . To do this , each term vector needed to be represented by a numeric vector representing the relative importance of keywords for each gene . This process is concerned with computing the numeric weight , wij , for each gene-term pair -LRB- gi , tj -RRB- -LRB- i = 1 , 2 , ... n and j = 1 , 2 , ... k -RRB- to represent the gene 's characteristics in terms of the associated keywords . Common techniques for such numeric encoding includes -LRB- 1 -RRB- Binary , the presence or absence of a keyword relative to a gene , -LRB- 2 -RRB- Term frequency , he frequency of occurrence of a keyword with a gene -LRB- 3 -RRB- Term frequency × inverse document frequency -LRB- TF * IDF -RRB- , the relative frequency of occurrence of a keyword with a gene compared to other genes . As we derived the keywords from gene annotation sentences but not from full abstracts , we found the number of keywords associated with each gene is small . We noticed also that absolute frequency of most keywords tended be one . Therefore , we adopted the binary encoding scheme as illustrated in Table 2 in supplementary material , in which each gene is represented by a vector of ` normalized ' absolute keywords frequencies . The ` normalized ' absolute frequency of each vector element -LRB- keyword -RRB- is either zero or one . Gene clustering Clustering is a data mining technique that groups or clusters data components -LRB- typically represented as numeric vectors -RRB- according to their similarity or dissimilarity . The goal is to maximize intra-cluster and minimize inter-cluster similarity among the components . -LSB- 32,33 -RSB- Clustering is typically used to identify sample groups in data . Unlike supervised learning methods that require explicit class label information , clustering is unsupervised and no information about target groups -LRB- classes -RRB- is used . Two basic approaches to clustering can be distinguished , hierarchical clustering -LRB- e.g. , agglomerative and divisive -RRB- and non-hierarchical clustering -LRB- e.g. , k-means/c-means clustering -RRB- . Agglomerative hierarchical clustering starts with each object representing a cluster and then merges the clusters in sequence . Divisive hierarchical clustering starts with all samples in one cluster and successively split clusters . In hierarchical clustering the distance -LRB- similarity -RRB- between clusters is measured using different techniques such as single linkage , average linkage or complete linkage -LSB- 32 -RSB- and basic distance and similarity metrics -LRB- e.g. , Euclidean , Minkowski , Hamming distance -RRB- . K-means clustering requires a priori specification of the desired number of clusters , k . This method clusters data into groups by iteratively optimizing the positions of cluster centers -LRB- means -RRB- so that the sum of within-cluster similarities -LRB- the similarity between data points and their cluster centers -RRB- is maximized . Essentially , the sentence-level binary coding scheme adopted in this study consists of numeric row vectors representing genes -LRB- via the associated biological function/process terms -RRB- , and numeric column vectors representing annotation terms -LRB- via the associated genes -RRB- . These two sets of vectors can be independently clustered using available clustering algorithms and tools . This approach can produce useful and specific information about the biological characteristics of sets of genes . In this study , we have used average linkage hierarchical clustering algorithm . -LSB- 33 -RSB- Using this algorithm has two advantages for this study . First , clustograms , a visualization of the substructures contained in a gene collection are produced , and second , individual clusters of genes are identified by clustogram splits at different levels . Clustering was performed using Cluto -LSB- 34 -RSB- and Cluster/Treeview -LSB- 35 -RSB- facilitates visualization of the clustograms . Results and discussion Evaluation To obtain a quantitative measure on the performance of the various keyword encoding schemes , we developed a text corpus of 200 manually annotated abstracts based on two cancer categories brain tumor and breast cancer of our interest -LRB- see Table 4 under supplementary material -RRB- . We used the following procedure to establish the corpus : -LRB- 1 -RRB- Determine randomly two cancer categories -LRB- brain tumor and breast cancer -RRB- , -LRB- 2 -RRB- For each cancer category , select randomly 10 genes from Entrez such that species = human and number of associated abstracts ≥ 50 , -LRB- 3 -RRB- For each gene identified in this way , select randomly 10 abstracts , resulting in a total of 200 abstracts ; 10 abstracts for each of the 10 genes associated with each of the two cancer categories , -LRB- 4 -RRB- For each of the 200 abstracts , identify manually the keywords characterizing biological function and processes from abstracts , MeSH terms and GO terms . With this text corpus we were able to construct a matrix containing all 20 genes and their associated keywords and keyword frequencies from abstracts , MeSH terms and Go terms . The manually annotated corpus of 200 abstracts and the matrix of 20 annotated genes served as gold standard for our evaluation experiments . We carried our four evaluation experiments : -LRB- 1 -RRB- Abstract keywords -LRB- baseline -RRB- . Extracts gene annotation terms based on term frequencies * inverse document frequencies -LRB- TF * IDF -RRB- within the entire abstract without regard to sentence structure , -LRB- 2 -RRB- Sentence keywords . Extracts gene annotation terms based sentence-level keywords , -LRB- 3 -RRB- Sentence + MeSH keywords . As in -LRB- 2 -RRB- above plus MeSH terms -LRB- see Section MeSH keywords extraction -RRB- , -LRB- 4 -RRB- Sentence + MeSH + GO keywords . As in -LRB- 2 -RRB- above plus MeSH terms -LRB- see Section MeSH keywords extraction -RRB- and GO terms -LRB- see Section GO keyword extraction -RRB- . Essentially , in each evaluation experiment the input is the text corpus of 200 abstracts and the output is a list of genes with its predicted annotation terms . Informally , the closer the predicted annotation terms match the manually established annotation terms , the better is the method . Performance is measured via commonly used criteria such a recall -LRB- analogous to sensitivity -RRB- , precision -LRB- analogous to positive predictive value -RRB- and the F-measure -LRB- a score that combines recall and precision -RRB- . The results we obtained are shown in Table 5 -LRB- below in supplementary material -RRB- . We notice that the baseline method comprising TF * IDF keywords fares worst among all four approaches . We interpret this as evidence for the validity of the methods involving sentence-level processing as this information is likely to carry most specific characterizing terms . The ` brute-force ' abstract-level processing will have difficulty in extracting these terms correctly and consistently . We further notice that the substantial improvements of precision and recall when we include MeSH terms and GO terms . This may be because these two categories are more specific and MeSH and GO annotations were done using full-papers and these biological functions and process are not described in all abstracts . Clustering of genes resulting from microarray experiment To demonstrate the usefulness of the presented keyword-extraction techniques to microarray data analysis , this method was applied to annotate and cluster gene lists that were found differentially expressed in a microarray experiment investigating the impact of two mitogenic proteins , Epidermal growth factor -LRB- EGF -RRB- and Sphingosine 1-phosphate -LRB- S1P -RRB- , on glioblastoma cell lines -LSB- 36 -RSB- . The microarray data set reveals three sets of differentially expressed genes -LRB- p < 0.05 -RRB- , namely , genes differentially expressed with response to EGF , G -LRB- EGF -RRB- , genes differentially expressed with respect to S1P , G -LRB- S1P -RRB- and genes differently expressed in response to both , G -LRB- COM -RRB- . Genes were considered differentially expressed if their p-value is smaller than 0.05 . We found that , when compared to the resting state , 19 genes were significantly differentially expressed as a response to EGF , 35 genes as a response to S1P and 30 genes as a response to COM , i.e. , combined stimuli of S1P and EGF . The three gene lists are referred to as G -LRB- EGF -RRB- , G -LRB- S1P -RRB- and G -LRB- COM -RRB- , respectively -LRB- see Table 6 in supplementary material -RRB- . Using these the three gene lists obtained from the microarray experiment -LRB- Table 6 shown in supplementary material -RRB- as query in MEDLINE returned the three corresponding sets of abstracts A -LRB- EGF -RRB- , A -LRB- S1P -RRB- and A -LRB- COM -RRB- , respectively . The abstracts were processed with the keyword extraction method involving sentence-level , MeSH and GO terms and the resulting representations were clustered using average linkage hierarchical clustering algorithm . Our gene clustering strategy and clustering algorithms are explained in the Methodology section . The resulting clustograms are presented in Figure 2 , Figure 3 , and Figure 4 , respectively . The clustograms depict associations between genes and biological function/process terms derived from the abstracts obtained with the various gene lists . For the investigating scientist , the clustograms fulfill the following main functions : -LRB- 1 -RRB- Squares highlighted in a horizontal line link a gene to one or more biological functions or processes . This is useful to see which genes are associated with which functions/processes and which genes have few or many associations . The interpretation of many and few is very much dependent on the associated biological function/process categories , the particular scientific question under investigation , and also on how extensively a particular gene has been researched and reported in the literature . -LRB- 2 -RRB- Users may visually delineate clusters , i.e. , rectangular areas with many highlighted squares in them and few highlighted squares around them . Any cluster , small or large , is potentially very useful to have discovered . Each cluster identified in this way relates a set of genes to a group of biological functions and processes . In a sense , each gene in the clustered is characterized by the same set of biological function and process concepts , a kind of ` guilt by association ' . This information is extremely useful as it provides clues as to the roles genes may play collectively in pathways and functions , processes , and possible phenotypes , that are associated with these pathways . Summary of analysis of EGF cluster , G -LRB- EGF -RRB- The clustograms in Figure 2 show the results obtained from extracting the sentence-level function/process keywords -LRB- plus MeSH and GO terms -RRB- from 28,913 abstracts -LRB- for the 19 genes detected in response to EGF stimulus -RRB- and the subsequent clustering . In Figure 2a several individual genes with very many -LRB- e.g. , CALD1 , CLU , FOS -RRB- and very few -LRB- e.g. , HRY , DUSP6 -RRB- associations stand out . Another interesting feature is the large cluster at the lower left corner of Figure 2a -LRB- reproduced in more detail in Figure 2b -RRB- containing the genes DUSP , ID1 , KLF2 , CALD1 , ABCA , CLU , FOS , JUN and SLC5A3 . Many genes in this cluster are associated with the same set of keywords -LRB- transcription factor , cell death and secretion -RRB- . Summary of analysis of S1P cluster , G -LRB- S1P -RRB- The clustograms in Figure 3 show the results obtained from extracting the sentence-level function/process keywords -LRB- plus MeSH and GO terms -RRB- from 19,705 abstracts -LRB- for the 30 genes detected in response to S1P stimulus -RRB- and the subsequent clustering . In Figure 3a several individual genes with very many -LRB- e.g. , CCL3 , IL6 , IL8 , F3 -RRB- and very few -LRB- e.g. , HERB2 , DOC1 -RRB- associations stand out . Another interesting feature is the large cluster at the upper left corner of Figure 3a -LRB- reproduced in more detail in Figure 3b -RRB- containing the genes TNAIP , KLF5 , BCL6 , NAB1 , BTG1 , NFKBIA , NR4A1 , SOCS5 , CITED2 , NRG1 , JAG1 , PLAU , CCL2 , IL8 , IL6 , GLIPR1 , F3 , MAP2K3 , and EHD1 . Many genes in this cluster are associated with the same set of keywords -LRB- atherogenesis , mitogenesis , assemble , inflammation , focal-contact , ... , and protein-binding -RRB- . Summary of analysis of the common gene cluster , G -LRB- COM -RRB- The clustograms in Figure 4 show the results obtained from extracting the sentence-level function/process keywords -LRB- plus MeSH and GO terms -RRB- from 39,890 abstracts -LRB- for the 30 genes detected in response to EFG and S1P stimuli -RRB- and the subsequent clustering . In Figure 4a several individual genes with very many -LRB- e.g. , MYC , MAFF , ATF3 -RRB- and very few -LRB- e.g. , DIPA , UGCG , SNARK -RRB- associations stand out . Another interesting feature is the large cluster at the upper left corner of Figure 4a -LRB- reproduced in more detail in Figure 4b -RRB- containing the genes SPRY2 , GEM , ZYX , NEDD9 , MYC , LIF , SERPINE1 , DTR , MUCL1 , C8FW , MAFF , ATF3 , RTP801 , EGR1 , JUNB , FOSL1 , CEPED , TIEG , EGR2 , EGR3 , and ZFP36 . Many genes in this cluster are associated with the same set of keywords -LRB- DNA binding , zinc fingers , repressor proteins , ... , and mitosis -RRB- . An important aim in microarray data mining is to bind transcriptionally modulated genes to functional pathways or to understand how transcriptional modulation can be associated with specific biological events such as genetic disease phenotype , molecular mechanism of drug action , cell differentiation etc. . However , the amount of functional annotation available with each transcriptionaly modulated genes is still a limiting factor because not all genes are well annotated . Our functional clustering/grouping will enable to select literally informative genes -LRB- Figure 2b , Figure 3b , and Figure 4b -RRB- for further investigations in the above data mining and knowledge discovery pipeline . Our evaluation suggests that this approach will provide more specific and useful information than typical approaches using abstract-level information . This is particularly the case when the sentence-level terms are augmented by MeSH and GO keywords . Conclusion The sequencing of whole genomes and the introduction high throughput analysis -LRB- e.g. , oligonucleotide and cDNA chips , MALDI/SELDI-TOF MS -RRB- provides biomedical research with a global perspective , which necessitates the development of novel mining tools to explore and interpret data in timely manner . This paper presents a novel approach to combine sentence-level keywords with GO and MeSH terms . In our evaluation experiment , this approach has shown promising results . The present evaluation suggests that this approach will provide more specific information than typical approaches using abstract-level information . This is particularly the case when the sentence-level terms are complemented by MeSH and GO terms . Further , clustering of genes into different functional groups based on literature keywords has the potential to help biologists identify and characterize literally informative genes of interest for further investigations . Future work Future enhancements of the system will include additional data resources -LRB- OMIM . DIP , KEGG -RRB- and the generation of association rules to identify correlations among genes in the same cluster . Association rules between the genes in the same cluster seem particularly interesting because it allows one to find the presence of regularities between gene groups . Finally , abstracts were used in this study as they are readily and easily available but they are limited in content . As full-text contains large number of irrelevent sentences compared to abstracts this approach may be useful for full-text analysis too , as it performs filtering of irrelevant sentences before clustering . The plan to perform the current study with full-text articles and compare the results with that of abstracts is on the way . Supplementary material Data 1

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Better Survival in Patients with Esophageal Cancer After Surgical Treatment in University Hospitals : A Plea for Performance by Surgical Oncologists Background In primary esophageal cancer , studies have frequently focused on surgical patients in an effort to link outcome to hospital - or surgeon-related experience , with operative mortality used as the main outcome measure . Many studies have found an inverse relationship between operative mortality and hospital volume and surgical expertise . This study aims to assess the influence of surgeon-related expertise and hospital volume on the relative survival of operated esophageal cancer patients . With an incidence of 6.3 per 100,000 -LRB- European standardized rate -RRB- in the period 1994 -- 1998 , esophageal cancer ranked 13th among malignancies in men and 12th in women in the Netherlands . Incidence increased in both male and female patients , with an estimated annual percentage change of 3.1 % and 2.0 % , respectively . The mortality for men and women was , respectively , 9.4 per 100,000 and 3.1 per 100,000 in 2004 , indicating that the prognosis of patients with esophageal cancer remained poor in this period -LRB- http://www.kankerregistratie.nl -RRB- . The only curative option for esophageal cancer is surgery , which implies that improving the outcome of surgery is the best means of reducing mortality .1 Esophageal cancer is one of the most challenging pathological conditions confronting the surgeon . It therefore seems reasonable to assume that concentration of esophageal surgery could improve outcome . Several studies have shown that various characteristics , including surgeon subspecialty certification , hospital setting , and the number of procedures performed , are associated with practice variation , complication rates , and even outcome .2 -- 4 Evidence of improved outcome associated with specialist care exists for breast cancer ,2,4 ovarian cancer ,3,5 and malignant teratoma .6 However , there is no evidence of comparable quality for esophageal cancers . Several studies on esophageal cancer have focused on surgical patients in an effort to link outcome to experience , either on the part of the institution or the surgeon , by using postoperative mortality as the main outcome measure .7 -- 9 A study on esophagectomies performed in England during the late 1980s found no independent association between operative mortality and hospital surgical volume ,10 whereas two North American studies did demonstrate a lower mortality rate for esophagectomy when high-volume and low-volume hospitals were compared .11,12 Various smaller studies also found lower operative mortality rates among high-volume surgeons .13,14 Few studies , if any , have attempted to relate esophageal cancer patient survival to surgeon expertise or hospital volume . To remedy this , in this study , we assessed the effect of surgeon-related expertise and hospital volume on the relative survival of operated esophageal cancer patients . We compared data from university , teaching nonuniversity , and nonteaching hospitals . PATIENTS AND METHODS Patients All patients diagnosed with a primary invasive esophageal cancer in the region of the Comprehensive Cancer Centre North-Netherlands -LRB- CCCN -RRB- between January 1994 and January 2002 were eligible for entry onto the study . Patients with a history of cancer other than nonmelanoma skin cancer were excluded . The patients were selected through the population-based Regional Cancer Registry of the CCCN , which covers the northern part of the Netherlands , a mainly rural area with a population of approximately 2.1 million . The area is served by 17 community hospitals , 3 of which are teaching hospitals and 1 of which is a university hospital ; the hospitals include four radiotherapy departments and seven pathology laboratories . Data Collection by the Regional Cancer Registry PALGA , a Dutch nationwide network and registry of histopathology and cytopathology , regularly submits reports of all diagnosed malignancies to the cancer registry . The national hospital discharge data bank , which receives discharge diagnoses of admitted patients from all hospitals , completes case ascertainment . The cancer registry has no access to death certificates . After notification , trained registry personnel collect data on diagnosis and staging from the medical records , including pathology and radiology reports , in the hospitals . The cancer registry collected all data regarding the diagnosis and staging , but collected no data on specific surgical treatment in this patient population before 2002 . The data collection occurs at least 4 months after diagnosis to comprehensively document all aspects of preoperative staging . All patients are staged according to the tumor , node , metastasis system -LRB- TNM -RRB- system for esophageal cancer in use during that period .15,16 In the Netherlands , the population registries of the municipality contain information on the vital status of their inhabitants . Vital status was established either through information derived from the patient 's medical records or through linkage of cancer registry data with information from the population registries of the municipality within the registry areas or through linkage with the national death registry of the Central Bureau of Genealogy . The regional cancer registry of the CCCN checked vital status by active record linkage with municipal population registries in 2002 -- 2003 and 2005 and with the national death registry of the Central Bureau of Genealogy in 2004 . Guidelines for Staging and Treatment By establishing multidisciplinary teams and cancer networks , the CCCN strives to improve the quality of cancer care . Within the CCCN area tumor working groups , comprising delegated specialists representing all regional hospitals , that have been developing and revising guidelines on diagnosis and treatment . The regional guidelines for esophageal cancer were based on the international TNM classification according to the International Union Against Cancer in use at that time . Statistics and Definitions of Variables The χ2 test was used to compare the distribution over the patient population for categorical variables . For continuous variables , analysis of variance was used . Relative survival analysis was performed to estimate the effect of university , teaching nonuniversity , and nonteaching hospitals on the prognosis of operated patients with esophageal cancer . The a priori hypothesis was that patient volume and hospital expertise would increase from nonteaching to university teaching hospitals . Survival time was calculated from the date of diagnosis and ended at the date of death , including perioperative death , or the date of most recent linkage with the municipal population registries and/or national death registry . The overall survival probability was estimated by the Kaplan-Meier method . The expected survival probability was calculated by using age - , sex - , and period-matched mortality rates that were based on life expectancy tables in the Netherlands -LRB- http://www.statline.cbs.nl/StatWeb/ -RRB- and was estimated by the Ederer 2 method .17 The cumulative relative survival -LRB- the ratio of the overall survival to the expected survival -RRB- was estimated by Stata -LRB- version 8.0 -RRB- software and the strs function . The relative survival , which estimates the net esophageal cancer survival in the hypothetical situation that esophageal cancer is the only possible cause of death , has been shown to be a good estimator of disease-specific survival in the absence of information on the cause of death or in case information on the cause of death is inaccurate . The excess mortality rate was calculated by subtracting the expected number of deaths , estimated from the expected survival probability , from the observed number of deaths in a subgroup or stratum and dividing the resulting excess number of deaths by the number of accumulated person-years , taking censoring into account . The relative excess risks -LRB- RER -RRB- of death were estimated as the ratio of excess mortality rates . RERs were estimated by a multivariate generalized linear model with a Poisson error structure , which was based on collapsed relative survival data , by using exact survival times .18 By use of this model , the effect of the type of hospital -LRB- university , teaching nonuniversity , and nonteaching -RRB- was studied , adjusting for the effect of various covariables on the excess mortality experienced by our cohort . Variables included in the final model were age -LRB- < 50 , 50 -- 59 , 60 -- 69 , > 70 -RRB- , stage based on collapsed TNM data -LRB- stage 1,2 a , 2b , 3/4 , unknown -RRB- , hospital volume -LRB- < 20 patients operated , ≥ 20 patients operated -RRB- , frequency of referral -LRB- high referral -LSB- > 33.3 % -RSB- , low referral -LSB- ≤ 33.3 % -RSB- -RRB- , and time since diagnosis -LRB- 1-year intervals -RRB- . The pathological stage was used whenever possible ; in the absence of information about the pathological stage , the clinical stage was used . RESULTS TABLE 1 . Characteristics of operated and nonoperated patients with esophageal cancer diagnosed 1994 -- 2002 Characteristic Total Operated Nonoperated P value N % n % n % Sex .465 Male 796 69.1 152 71.4 644 68.6 Female 353 30.9 61 28.6 292 31.2 Histology < .001 Squamous cell carcinoma 415 36.1 62 29.1 353 37.7 Adenocarcinoma 592 51.6 140 65.7 453 48.4 Other 141 12.3 11 5.2 130 13.9 Tumor location < .001 Upper thoracic 82 7.1 3 1.4 79 8.4 Middle thoracic 210 18.3 36 16.9 174 18.6 Lower thoracic 770 67.0 169 79.3 601 64.2 Overlapping and unspecified 87 7.6 5 2.3 82 8.8 Age at diagnosis -LRB- y -RRB- < .001 < 50 86 7.5 24 11.3 62 6.6 50 -- 59 221 19.2 67 31.5 154 16.5 60 -- 69 319 27.8 80 37.6 239 25.5 70 + 523 45.5 42 19.7 481 51.4 Stage < .001 1 52 4.5 32 15.0 20 2.1 2A 174 15.1 65 30.5 109 11.7 2B 69 6.0 26 12.2 43 4.6 3 207 18.0 74 34.7 133 14.2 4 316 27.5 8 3.8 308 32.9 Unknown 331 28.8 8 3.8 323 34.5 Total 1149 100.0 213 100.0 935 100.0 In the period 1994 -- 2002 , a total of 1149 patients were diagnosed with esophageal cancer , comprising 796 men and 353 -LRB- 69.3 % -RRB- women -LRB- 30.7 % -RRB- . The median age was 68 years -LRB- range , 17 -- 103 years -RRB- . Patient characteristics are described in Table 1 . A large proportion of the patients was diagnosed at an advanced stage of disease ; 45.5 % were stage III or higher . A further 28.8 % were insufficiently staged . Of the 1149 patients , 85 patients -LRB- 7.4 % -RRB- were initially diagnosed in the university hospital , 428 patients -LRB- 37.2 % -RRB- in teaching nonuniversity hospitals . The remaining 636 patients -LRB- 55.4 % -RRB- were diagnosed in the nonteaching hospitals . Patients who were referred from the hospital of initial diagnosis for treatment may have subsequently undergone further diagnostic testing , and previously performed tests determined to be insufficient were repeated . The 5-year relative survival rate for men with esophageal cancer was 12.8 % vs. 9.8 % for women -LRB- P = .496 -RRB- . The 5-year relative survival markedly decreased as the stage advanced . The 5-year relative survival was 71.5 % in stage I , 26.5 % in stage IIA , 13.3 % in stage IIB , 9.2 % in stage III , 1.3 % in stage IV , and 6.7 % for patients with unknown stage -LRB- P < .0001 -RRB- . As Table 1 shows , only 213 patients -LRB- 18.5 % -RRB- underwent surgery . Older age -LRB- P < .001 -RRB- , advanced or unknown stage -LRB- P < .001 -RRB- , and proximal tumor location -LRB- P < .001 -RRB- resulted in a lower probability of tumor resection . Squamous cell carcinoma was also associated with less surgery , but it was highly correlated with the tumor location . In all , 21.4 % of the patients diagnosed in nonteaching hospitals underwent surgery , compared with 15.9 % and 10.6 % for teaching nonuniversity and university hospitals , respectively -LRB- P = .011 -RRB- . Adjusted for age , stage , and tumor location , the odds of operation was 1.89 -LRB- 95 % confidence interval -LSB- 95 % CI -RSB- , 1.26 -- 2.82 -RRB- for patients diagnosed in a nonteaching hospital compared with patients diagnosed in a teaching nonuniversity hospital . Of all operated patients 45.1 % were referred for surgery after diagnosis in the hospital of presentation -LRB- Table 2 -RRB- . The nonteaching hospitals referred 57.4 % of patients diagnosed in their hospitals for an operation elsewhere . Of the 14 nonteaching hospitals in which esophageal cancer patients were diagnosed , 12 referred nearly all -LRB- 75 % -- 100 % -RRB- patients , and two rarely -LRB- .0 % -- 6.9 % -RRB- referred patients . The teaching nonuniversity hospitals referred 26.5 % of the patients diagnosed in their hospitals to a larger institution for therapy , with one hospital referring 63.2 % of their patients . TABLE 2 . Referral pattern for esophageal cancer surgery per hospital in the North-Netherlands , 1994 -- 2002HospitalOperated in hospital of diagnosis , n -LRB- % -RRB- Referred for surgery , n -LRB- % -RRB- Total -LRB- n -RRB- High-referral nonteaching hospitals 20 -LRB- 20.8 -RRB- 76 -LRB- 79.2 -RRB- 96Hospital A -- 2 -LRB- 100.0 -RRB- 2Hospital B -- 3 -LRB- 100.0 -RRB- 3Hospital C -- 3 -LRB- 100.0 -RRB- 3Hospital D -- 9 -LRB- 100.0 -RRB- 9Hospital E2 -LRB- 11.8 -RRB- 15 -LRB- 88.2 -RRB- 17Hospital F1 -LRB- 14.3 -RRB- 6 -LRB- 85.7 -RRB- 7Hospital G3 -LRB- 18.8 -RRB- 13 -LRB- 81.3 -RRB- 16Hospital H3 -LRB- 25.0 -RRB- 9 -LRB- 75.0 -RRB- 12Hospital I3 -LRB- 30.0 -RRB- 7 -LRB- 70.0 -RRB- 10Hospital J1 -LRB- 33.3 -RRB- 2 -LRB- 66.7 -RRB- 3Hospital K1 -LRB- 33.3 -RRB- 2 -LRB- 66.7 -RRB- 3Hospital L6 -LRB- 54.5 -RRB- 5 -LRB- 45.5 -RRB- 11Low-referral nonteaching hospitals 38 -LRB- 95.0 -RRB- 2 -LRB- 5.0 -RRB- 40Hospital M27 -LRB- 93.1 -RRB- 2 -LRB- 6.9 -RRB- 29Hospital N11 -LRB- 100.0 -RRB- -- 11High-referral teaching , nonuniversity hospitals7 -LRB- 36.8 -RRB- 12 -LRB- 63.2 -RRB- 19Hospital O7 -LRB- 36.8 -RRB- 12 -LRB- 63.2 -RRB- 19Low-referral teaching , nonuniversity hospitals43 -LRB- 87.8 -RRB- 6 -LRB- 12.2 -RRB- 49Hospital P24 -LRB- 82.8 -RRB- 5 -LRB- 17.2 -RRB- 29Hospital Q19 -LRB- 95.0 -RRB- 1 -LRB- 5.0 -RRB- 20Low-referral university hospital9 -LRB- 100.0 -RRB- -- 9Hospital R9 -LRB- 100.0 -RRB- -- 9117 -LRB- 54.9 -RRB- 96 -LRB- 45.1 -RRB- 213 Table 3 compares characteristics of the operated patients in the three hospital types . Of the 213 operated patients , 95 underwent surgery in the university hospital ; 86 of these patients were referrals . The three teaching nonuniversity hospitals provided surgery to 60 patients , including 10 referrals : one hospital performed surgery on 7 patients , with the two other hospitals operating on more than 20 . The remaining 58 patients underwent surgery at one of the 14 nonteaching hospital after all were initially diagnosed in the same hospital ; two low-referral hospitals operated on more than 10 patients , eight hospitals operated on 5 or fewer patients , and the remaining four high-referral hospitals did not perform any esophageal cancer surgery . There were no statistically significant differences in the distribution of age -LRB- P = .230 -RRB- , stage -LRB- P = .299 -RRB- , or tumor location -LRB- P = .130 -RRB- between the hospital types , again showing little evidence of selective referral . However , of all operated patients per hospital , a slightly larger proportion of stage III/IV tumors were operated on in the nonteaching -LRB- 46.6 % -RRB- and university hospitals -LRB- 41.1 % -RRB- compared with the teaching nonuniversity hospitals -LRB- 26.7 % -RRB- , and the portion of stage IIA tumors was somewhat higher in nonteaching -LRB- 31.0 % -RRB- and teaching nonuniversity hospitals -LRB- 38.3 % -RRB- compared with the university hospital -LRB- 25.3 % -RRB- . TABLE 3 . Characteristics for operated esophageal cancer patients diagnosed 1994 -- 2002 , according to hospital of surgeryCharacteristicTotalTeaching , nonuniversityUniversityNonteachingP valueN % n % n % n % StageStage 13215.51220.01414.7610.3.299 aStage 2A6526 .81838.32325.32431.0 Stage 2B2612 .2813.31212.6610.3 Stage 3 + 48238.51626.73941.12746.6 Unknown87 .011.766.311.7 Age at diagnosis -LRB- y -RRB- < 502411.3813.31313.735.2.23050 -- 596731.51626.73637.91525.960 -- 698037.62541.73031.62543.170 +4219.71118.31616.81525.9 HistologySquamous cell carcinoma6229 .12135.02728.41424.1.606 Adenocarcinoma14065 .73558.36366.34272.4 Other115 .246.755.323.4 Tumor locationUpper and middle thoracic3918 .31321.71717.9915.5.130 Lower thoracic16979 .34778.37376.84984.5 Not stated52 .3 -- -- 55.3 -- -- Total213100 .060100.095100.058100.0 a Excluding stage unknown . The cumulative relative survival for university , teaching nonuniversity , and nonteaching hospitals is shown in Fig. 1 . Surprisingly , relative survival was markedly better in the university hospital compared with teaching nonuniversity and nonteaching hospitals . The 5-year relative survival was 49.2 % for the university hospital versus 32.6 % and 27.3 % for teaching nonuniversity and nonteaching hospitals , respectively -LRB- P = .0039 , Table 4 -RRB- . In univariate analysis , the RER for university and teaching nonuniversity was , respectively , .48 -LRB- 95 % CI , .30 -- .74 -RRB- and .88 -LRB- 95 % CI , .55 -- 1.42 -RRB- compared with nonteaching hospitals . The proportion of operated patients who died within 3 months after diagnosis differed between university , teaching nonuniversity and nonteaching hospitals -LRB- 4.2 % , 13.3 % and 19.0 % respectively , P = .013 -RRB- . Excluding patients who died within the first 3 months , the RER for university and teaching nonuniversity was , respectively , .59 -LRB- 95 % CI , .35 -- 1.00 -RRB- and .97 -LRB- 95 % CI , .56 -- 1.69 -RRB- compared with nonteaching hospitals . FIG. 1 . Cumulative relative survival of patients operated for esophageal cancer diagnosed during 1994 -- 2002 according to hospital type.TABLE 4 . Overall and relative 5-year survival and estimated excess risk -LRB- RER -RRB- of death with 95 % confidence intervals -LRB- 95 % CI -RRB- for operated esophageal cancer patients diagnosed 1994 -- 2002CharacteristicNOS -LRB- 5 y -RRB- RS -LRB- 5 y -RRB- OD -LRB- 5 y -RRB- UnivariateMultivariate95 % CIP valueED -LRB- 5 y -RRB- RERa95 % CIRERaStage < .0001 Stage 1 3287.5 % 99.1 % 43.1.06.01 -- .32.05.01 -- .22 Stage 2A6546 .1 % 52.4 % 333.9.44.27 -- .70.39.24 -- .63 Stage 2B2621 .8 % 24.7 % 191.1.75.43 -- 1.29.72.40 -- 1.27 Stage 3 +4 -LRB- reference -RRB- 8214.2 % 14.5 % 651.61.001.00 Unknown816 .7 % 18.0 % 6.2.99.41 -- 2.351.62.65 -- 4.01 Hospital type .0126 Nonteaching -LRB- reference -RRB- 5824.9 % 27.3 % 402.31.001.00 Teaching nonuniversity6029 .7 % 32.6 % 391.9.89.55 -- 1.421.32.79 -- 2.22 University9544 .3 % 49.2 % 485.6.48.30 -- .77.57.29 -- 1.12 Age -LRB- y -RRB- .0467 < 50 -LRB- reference -RRB- 2447.6 % 48.6 % 11.21.001.0050 -- 596732.1 % 33.6 % 421.21.73.87 -- 3.441.51.74 -- 3.0460 -- 698030.9 % 35.1 % 523.52.121.07 -- 4.182.361.18 -- 4.7070 +4241.9 % 54.5 % 224.91.42.64 -- 3.142.05.94 -- 4.46 Hospital volume .1125 < 20 patients operated3819 .0 % 22.3 % 271.31.001.00 ≥ 20 patients operated17537 .8 % 41.7 % 1008.5.53.33 -- .83.62.34 -- 1.12 Referral rate .8080 High -LRB- > 33.3 % -RRB- 11535.4 % 39.2 % 665.21.001.00 Low -LRB- ≤ 33.3 % -RRB- 9834.1 % 37.8 % 614.61.16.79 -- 1.69.94.57 -- 1.54 OS , overall survival ; RS , relative survival ; OD , observed deaths ; ED , expected deaths.a Adjusted for time since diagnosis . In a multivariate analysis to adjust for the prognostic effect of patient age , tumor stage , tumor location , hospital volume , frequency of referral , and time since diagnosis , we found that stage , age , hospital type , and time since diagnosis were independently associated with the RER -LRB- Table 4 -RRB- . The RER increased with more advanced stage . Patients aged < 50 and patients aged ≥ 70 had a lower RER compared with patients aged 50 -- 69 years . Adjusted for age , stage , and time since diagnosis , the RER for the university hospital was still considerably lower , at .57 -LRB- 95 % CI , .29 -- 1.12 -RRB- , compared with nonteaching hospitals and .43 -LRB- 95 % CI , .24 -- .76 -RRB- compared with teaching nonuniversity hospitals -LRB- P = .0126 -RRB- . There was some evidence in the data for an independent effect of hospital volume , with a lower RER -LRB- .62 ; 95 % CI , .34 -- 1.12 -RRB- if a hospital operated on ≥ 20 patients during the study period . In our study 8.9 % of the operated patients received some form of adjuvant therapy -LRB- Table 5 -RRB- . Of these patients , 6.1 % received preoperative chemotherapy and 2.8 % received postoperative radiotherapy . Patients receiving chemotherapy were all treated in the university hospital . Comparing clinical and pathological stage , we found no evidence that chemotherapy led to a marked downstaging . TABLE 5 . Adjuvant therapies for patients operated for esophageal cancer diagnosed 1994 -- 2002 , according to hospital of treatmentTreatmentNonteachingTeaching , nonuniversityUniversityTotaln % n % N % N % Surgery5798 .35896.77983.219491.1 Surgery + radiotherapy11 .7 -- -- 55.362.8 Surgery + chemotherapy -- -- 23.31111.6136.1 Total58100 .060100.095100.0213100.0 DISCUSSION Our study shows that patients who underwent surgery for esophageal cancer in the university hospital had a markedly better relative survival , with a 50 % lower risk of death compared with patients treated in nonuniversity hospitals . The risk of death did not differ for patients operated in teaching nonuniversity or nonteaching hospitals . In our study , we found that a higher hospital volume was weakly associated with better survival . Although hospital volume seems to influence better survival , it is unlikely that the difference between the three hospital types can be completely explained by hospital volume ; other factors likely play a role . A recent British study found no effect of hospital volume on survival for the operated patients ; further , this study did not find teaching hospital status independently associated with survival , but it is unclear whether this reflects the operated patients or the population as a whole .10 Our study is one of the first studies to research the effect of a marker for hospital/surgeon experience on patient survival . Most studies to date have evaluated 30-day postoperative or in-hospital mortality , and several demonstrated an inverse relationship between volume or surgical experience and operative mortality after esophageal resection .8,11,19 A Dutch study found that hospitals performing 1 to 10 operations per year for esophageal cancer and cancer of the gastroesophageal junction had an operative mortality of 12.2 % , compared with 4.9 % for hospitals performing > 50 procedures per year .19 An American study had a similar result , with a mortality of 3.0 % among high-volume hospitals and 12.2 % among low-volume hospitals for both distal and proximal esophageal cancer ,12 but this study used a cutoff point of five procedures per year to differentiate between high - and low-volume hospitals . What threshold distinguishes high-volume from low-volume hospitals remains matter of discussion . The results of these studies do suggest that centralization of esophageal surgery , so that only a few hospitals per region operate on esophageal cancer patients , may improve survival . The results of our study support the recommendation for referral of esophageal cancer patients to a center where there is a specific focus on esophageal cancer treatment . Combined with the fact that the surgical literature is increasingly advocating the need for centralization , we think that further research into the advantages of centralization of esophageal cancer treatment is warranted . One of the possible pitfalls in our study remains selective referral . We showed that the likelihood of being operated on was 47 % higher for patients diagnosed in a nonteaching hospital than for those diagnosed in teaching nonuniversity hospitals . Adjusted for age , stage , and tumor location , this was even higher , 73 % , indicating that nonteaching hospitals considered patients with a worse prognosis possible candidates for surgery , or at least for referral to evaluate resectability . However , adding hospital referral frequency to our multivariate analyses showed no influence of referral on overall and relative survival or RER . The referral pattern showed little evidence for selective referral . Those nonteaching hospitals that referred patients referred almost all , and the teaching nonuniversity hospitals referred only few patients for surgery -LRB- Table 2 -RRB- . The referral pattern of the nonteaching hospitals implies that the university hospital operated on a priori prognostically worse patients . This is a likely explanation for the higher number of stage III patients in the university hospital . To minimize the eventual effect of any residual selection referral , the relative survival rate was adjusted for case mix , despite there being no statistically significant differences in the distribution of age , stage , and sex between the different types of hospital . One may suggest that patients who underwent esophagectomy in the university hospital were mostly referred for treatment , thus adding a delay before surgery . A consequence could be that these patients had thus a slightly longer preoperative survival time , estimated to be between 2 and 4 weeks in our study . This short delay could mean that prognostically worse patients scheduled for surgery eventually fall out of the surgery category through disease progression during the delay period . However , little is known in the literature about the effect of longer preoperative delays on surgical outcome or eligibility in esophageal cancer . Although patients with advanced disease may miss surgery through stage progression , patients who do end up having surgery also progress , meaning the university hospital operates on patients with more advanced disease . This should negatively influence the survival outcome and would not explain the better performance by the university hospital . Furthermore , even if we were very conservative and excluded all patients who died in the first 3 months of our study , the university hospital still performs far better than teaching nonuniversity and nonteaching hospitals . So although we can not discount early mortality as a factor in survival , we think that it is unlikely that the difference in performance can be fully explained by this . We had no information about the operative procedure that had been performed . Treatment guidelines indicated a curative surgical approach for tumors encompassing ≤ 5 cm of the length of the esophagus , as based on ultrasonographic or radiological examination . Surgical resection could be attempted for tumors 5 -- 8 cm in length . For this last group , neoadjuvant chemotherapy , which was provided in the university hospital after proof of locally advanced disease , could be attempted to improve resectability . For adenocarcinomas , a transthoracoabdominal approach with two-field lymphadenectomy was advised , combining a midline laparotomy and a right-sided thoracotomy . Alternatively a transhiatal blind esophagectomy could be performed with a cervical esophagogastrostomy . For distal adenocarcinomas without Barrett dysplasia , a left-sided thoracotomy with intrathoracic anastomosis was an alternative approach . In general , patients in our population who were operated on in the university hospital underwent a transthoracic esophagectomy , with the exception of superficial T1 tumors , whereas patients treated in regional hospitals frequently underwent surgery with a transhiatal approach . Therefore , the two main operative strategies we encountered were transhiatal resection and transthoracoabdominal resection with a two-field lymph node dissection . There is no evidence in literature that the outcome differs for these two procedures ,14,20 except for a tendency toward an improved long-term survival in the extended transthoracic group in the study of Hulscher et al. 21 So the clinical outcome in our population is likely uninfluenced by differences in surgical procedure . However , there is possibly a stage migration effect between the more thorough pathological staging in operations with lymphadenectomy , as in the university hospital , and understaging in patients undergoing a transhiatal esophageal resection . A few of the operated patients received neoadjuvant therapy in our study , mostly in the university setting , which might account for a small part of the better survival in the university hospital . Separate analysis , however , showed that patients in our study , who received neoadjuvant therapy with surgical removal of the tumor , did not perform better than patients who underwent surgical removal alone in the university hospital . In other studies , the preoperative effect of cisplatin-based chemotherapy on both adenocarcinoma and squamous cell carcinoma showed no increase in overall survival .22 -- 25 Preoperative chemotherapy or radiotherapy can result in downstaging and thus lead to a better resectability , but no clear downstaging was seen in our study when comparing pre - and postoperative clinical and pathological stage . In several randomized , controlled studies , postoperative radiotherapy demonstrated either no increase26 ,27 or a decrease28 in survival compared with resection alone . Postoperative chemotherapy has also been compared with surgical management alone in several randomized controlled trials , without demonstrating an improvement in survival .29 According to these results , the effect of perioperative treatment is not likely to influence our data . We found a tentative relationship between higher volume and a better relative survival . However , this issue still is a debatable problem in determining treatment guidelines . Therefore , we suggest that guidelines concerning specific referral of esophageal cancer patients should be based on hospital outcomes , preferably in experienced centers , rather than on annual numbers of procedures as long as the factor that is determining patient survival is still unknown . The individual surgeon could be an important parameter in determining the hospital outcome . Although the implications of the assertion that some surgeons have better outcomes than others make clinicians uncomfortable , there should be little doubt that it is true . Variation in performance has been shown to be related to surgeon characteristics , including surgical volume , subspecialization , and the hospital setting in which they operate .7,30 Individual surgical experience has been associated with the postoperative mortality of esophageal cancer . Sutton et al. 9 showed a reduction in mortality from 6 % to 3 % after 150 procedures . Miller et al. 14 published results in one center demonstrating an operative mortality of 22 % among esophageal resections performed by a surgeon who performed fewer than six procedures a year . These results are widely quoted in the surgical literature as proving that surgeons without the necessary expertise should not perform esophageal resections . There is some evidence that subspecialization improves outcomes . Herr et al. 31 found that patients who underwent radical cystectomy by urology oncologists had substantially lower rates of local tumor recurrence than those who were operated on by general urologists -LRB- 6 % vs. 23 % -LRB- P = .006 -RRB- . Dueck et al. 32 reported that patients who underwent surgery for a ruptured abdominal aortic aneurysm had markedly better outcomes when the surgery was performed by a vascular surgeon rather than a general surgeon . The effect of subspecialization of the surgeon on the outcome of esophageal cancer has not yet been studied , but it may be a promising factor for decisions with regard to centralization . It has been suggested in previous reports that the skill of the anesthesia and nursing staff affects morbidity and relative survival of esophagectomy patients and that it confounds the surgeon 's personal outcome .7,9 Better critical-care experience of the support staff may explain a higher relative survival in university hospitals ; staff may be more adept at caring for esophagectomy patients . Some authors have suggested that the expertise of the anesthesia and nurses in a hospital is directly correlated to the hospital and surgical load in that hospital .9 However , expertise can be acquired elsewhere , and expertise only develops through effective feedback , not only by number of patients . Although referral to dedicated centers possibly results in improved relative survival , the focus entails some disadvantages , which should be considered . Referral to centers means that many patients have to travel to distant sites , which can create hardship for the patient and his or her family . In-hospital family support and postoperative follow-up are more difficult when the hospital is farther away . Finlayson et al. 33 demonstrated that 45 % of the patients prefer to stay in their local area even if the projected operative mortality is doubled . However , that study represents the American situation , and it is questionable whether distance is perceived to be a problem in the Dutch situation . In our region , which has a relatively high density of hospitals , a recent patient survey showed that traveling distance was not considered a critical issue . In conclusion , we demonstrated that in our region , the relative survival for patients operated on for esophageal cancer is better in the university hospital compared with teaching nonuniversity and nonteaching hospitals , emphasizing the need for referral to centers focused on the treatment of esophageal cancer . The underlying parameter for the observed difference remains unclear . We suggest that centers at least periodically review the morbidity and mortality rates of esophageal resections to assess their outcome and the possibility of referral . Eligibility for centers focused on esophageal cancer treatment should therefore be based on patient outcomes rather than on patient numbers .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O B O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O