BACKGROUND
lung cancer is the leading cause of cancer-related death in the united states and worldwide  <cit> , more than 25% of cancer-related deaths in both men and women resulted from lung cancer.
in the past few decades, considerable research effort has been devoted to elucidating the etiology of lung cancer and identifying possible biomarkers for predicting survival outcomes  <cit> .
however, the overall five-year survival rate for lung cancer is still less than 20%.
although several studies have identified expression signatures associated with survival outcomes in their original discovery datasets, these expression signatures were often found to be irreproducible across studies.
a previous report has shown that the largest number of overlapping predictive genes between two independent studies was only four, and most often even zero  <cit> .
the lack of reproducibility in independent studies makes it very difficult to justify conducting prospective clinical validation studies and application to clinical practice.
therefore, how to effectively identify a set of predictive and reproducible biomarkers across different lung cancer studies poses a major challenge.
to address this issue, one possible approach for biomarker identification is to perform analysis based on biologically functional relevant gene sets rather than to examine the associations by traditional gene-by-gene approaches.
several reports have demonstrated that predictive genes that were identified by approaches considering biological pathway or cellular function information are more often reproduced and validated in independent studies  <cit> .
for example, kang et al.
selected  <dig> effective genes to predict patient survival outcomes and chemotherapy responses in ovarian cancer from a pool of  <dig> genes summarized by dna repair pathways  <cit> .
an approach based on functional gene sets not only provides better understanding of biological meaning of identified gene markers, but also greatly increases statistical power of identification by considering only the number of pathways rather than the number of genes in the multiple comparison adjustment.
the approach to performing gene set enrichment analyses for gene expression profiling has become a popular methodology even for genome-wide association studies  <cit> .
in a recent study of an analysis of  <dig> published gene expression signatures for breast cancer  <cit> , the authors showed that the predictive performance of several signatures was not superior to that obtained from random gene expression signatures, and sometimes even worse.
there was a tendency for significant marker sets to depend on the size of the set; and over 90% of random signatures with more than  <dig> gene members were significant predictors for survival outcome  <cit> .
this might be attributed to a lack of independence among the genes in the array, and a predictive model tending to become significant as more gene members were included.
in order to reduce the possibility of identifying false-positive signatures, resample techniques can be performed to demonstrate significant findings, as compared to a randomly selected signature set with a same number of genes.
genomic signatures can be classified into two major types: prognostic biomarkers and predictive biomarkers.
prognostic biomarkers aim to help physicians divide patients into distinct groups based on their risk factors and disease characteristics, which hints that different drug treatments including standard treatment, or advanced aggressive therapeutic procedures may be applied to groups of patients showing poor survival outcomes.
predictive biomarkers for a specific treatment are able to identify a subgroup of suitable patients for receiving the treatment, and those patients expected to have poor responses would be spared from the treatment.
compared with prognostic biomarkers, predictive biomarkers provide physicians direct suggestions for specific therapeutic methods for individual patients.
currently, most identified  signatures in lung cancer fall into the category of prognostic biomarkers; the lack of studies for predictive biomarkers is partly due to many considerations required before conducting a treatment-specific randomized clinical trial.
recently, barretina et al.
<cit>  used microarrays to investigate the gene expression profiles of  <dig> human cancer cell lines along with their responses after exposure to  <dig> anticancer drugs.
among the  <dig> drugs,  <dig> were designed to directly target egfr, whose association with lung cancer has been widely reported  <cit> .
this dataset provides a good template for evaluation of predictive biomarkers in lung cancer.
in the development of a predictive signature, there are two primary assessment considerations: 1) predictability  and 2) generalizability .
predictability is the ability of the signature to predict survival risk scores of patients who were not included in the signature development, but who are from a similar population.
to evaluate predictability is to determine whether the signature can provide a relative ranking of patient risk scores correctly.
a predictive signature that performed well in the development study might not be reproduced in performance when applied to other studies.
generalizability is to determine whether predictability is reproducible when applied to data generated independently in different circumstances, such as different locations or different times.
a predictive signature needs to establish its predictability and generalizability before a prospective clinical validation is conducted.
in this study, we propose an approach for identifying prognostic gene signatures using  <dig> lung adenocarcinoma patients from the study of shedden et al.
<cit> .
the approach consists of a two-stage assessment: predictability and generalizability.
the shedden dataset was used for predictability assessment, and  <dig> additional independent lung cancer datasets were used for generalizability assessment.
the shedden  dataset was tested using 2-fold cross-validation.
generalizability of signature sets was evaluated and compared with several published lung cancer prognostic signatures using a randomization test.
a set of  <dig> genes involved in the function of apoptotic execution phase was shown to have good predictability as well as generalizability in more than  <dig> lung adenocarcinoma patients, including the  <dig> patients from the shedden et al.
study.
this signature set was also found to associate with the efficacy of the anti-cancer drug zd- <dig> targeting egfr.
RESULTS
identification of significant survival predictors
among the  <dig>  genes investigated in the dataset of shedden et al., log-rank tests were performed to exclude those genes determined to have no association with survival .
a total of  <dig>  genes  were selected and their functions were annotated by four biological databases, including kegg, biocarta, pid and reactome.
we focused on  <dig> pathways that were closely related to the development of cancer cells, involving apoptosis, cell cycle and cell death regulations .
for each pathway, the risk score of each patient was computed, and patients were divided into two groups according to the procedure described in the method section.
the log-rank test was performed to determine statistical significance of the signature set associated with survival outcomes for the pathway.
table  <dig> shows the p-values of  <dig> pathways that were significant using the bonferroni correction .
a randomization test was performed and all  <dig> pathways were highly significant .
aonly genes showing association with survival  were analyzed.
bestimated by using log-rank tests.
cthe frequency showing p-values <  <dig>  in the  <dig> trials of the internal validation assessment.
dthe frequency showing p-values <  <dig>  in the  <dig> trials of the internal validation assessment.
predictability assessment
internal analytical validation was performed on the shedden study to assess predictability of the signature sets using 2-fold cross-validation.
the  <dig> samples were randomly divided into two  sets of equal size, a training and a test set.
in the training phase, gene filtering, cox model fitting, total risk score calculation, and log-rank test were performed according to the procedures described in the method section.
the median of the risk scores was computed for each pathway.
in the test phase, a score of  <dig> was initially assigned to each patient.
for each gene, a score of  <dig> was given to a patient if the cox coefficient from the training set was greater than  <dig> and the patient’s expression level was higher or equal to the median, or if the coefficient was less than  <dig> and the expression level was less than the median.
the total risk score of each patient was computed for each pathway .
next, patients were classified into two risk groups based on the median of the risk scores from the training set.
the log-rank test was performed to compare the two risk groups at the significance level of  <dig>  .
here we considered both the unadjusted p-value  and the bonferroni adjusted p-value .
the procedure was repeated, but the model was developed on the test set and applied to the training set.
the entire procedure was repeated  <dig> times to generate different 2-fold partitions.
the frequency of significant results out of the  <dig> repetitions was calculated for each pathway.
the last two columns of table  <dig> list the frequencies of the significant results for the  <dig> pathways.
we selected the top  <dig> pathways which showed at least  <dig> significant counts at α =  <dig> / <dig>  for the generalizability assessment.
these numbers can be regarded as the power for the predictability.
generalizability assessment
generalizability requires that predictive performance of a model developed from one study is reproducible across different studies.
five additional microarray studies for a total of  <dig> studies  were evaluated.
among the  <dig> studies considered,  <dig> studies contained adenocarcinoma samples and  <dig> studies contained squamous cell carcinoma samples .
evaluation of generalizability is similar to the evaluation of predictability in which all  <dig> samples from the shedden study were used as the training dataset and applied to each of  <dig> independent studies as a test dataset.
briefly, four microarray datasets were retrieved from the gene expression omnibus   <cit> , and the quantile normalization algorithm was utilized to remove systematic biases.
gene expression levels of multiple probes annotated with the same gene symbol were summarized into a single value by using their means.
each independent dataset was then evaluated using the procedures described in the assessment of predictability.
table  <dig> shows the p-values and the number of significant results  for the  <dig> most significant pathways  for the five adenocarcinoma datasets, including shedden and four independent datasets.
several pathways were significant for  <dig> or more datasets; the pathway “apoptotic execution phase” was significant for all  <dig> datasets, an excellent criterion for generalizability.
aonly adenocarcinoma patients were analyzed here.
in addition, a resampling test was performed to simulate chance significance for the five adenocarcinoma datasets.
for each pathway, a null signature set consisting of randomly selected genes from the shedden study was generated, where the numbers of genes were kept the same as the pathway.
the null signature set was applied to the five datasets; the total number of significant results was compared to the observed number of significant results in column  <dig> to determine if the selected signature set was superior to the null signature set.
the simulation was repeated  <dig>  times.
the last column shows the p-value for the  <dig> pathways, where the p-value was defined as the proportion of the number of significances from the null signature set was greater or equal to the observed number in column  <dig> out of the  <dig>  repetitions.
the  <dig> pathways were also applied to four squamous cell carcinoma datasets, gse <dig>  gse <dig>  gse <dig>  and gse <dig>  and no statistical significance was identified for any of  <dig> pathways and  <dig> datasets .
this suggested that these  <dig> pathways were specific to the lung adenocarcinoma.
the 16-gene signature
the  <dig> genes involved in the apoptotic execution phase pathway and statistical p-values of the log-rank test from step  <dig> are listed in table  <dig>  among them, the most significant gene was dnm1l, which concurred with high expression of pkp <dig> and dnm1l associated with poor survival in lung adenocarcinoma patients  <cit> .
figure  <dig> shows the kaplan–meier survival curves with the p-values from the log-rank test for the datasets.
the proposed calculation of risk scores scheme and classification of patients into “high” and “low” groups appeared to perform well in both training and test data.
for example, in the study of shedden et al., the median survival of the patients belonging to the “high” group is obviously shorter than that from the “low” group .
similarly, the median survival also showed dramatic differences between the “high” and “low” groups in gse <dig> .
the p-values from fitting the cox hazard regression model using the risk scores as an independent variable are shown in figure  <dig>  the p-values were consistent with the p-values from the log-rank test; a minor discrepancy is that the dataset gse <dig> has  <dig> ties at the median score resulting in lower discriminating ability.
we also performed an analysis dividing the patients into three groups , and an analysis of the squamous cell carcinoma patients across  <dig> independent datasets.
the results are consistent with the results present above.
a“+” denotes the fitted cox efficient >  <dig> whereas “-“denotes the fitted cox coefficient <  <dig>
the risk scores and four clinical variables were analyzed simultaneously using the multivariate cox regression.
the four clinical variables were age, gender, smoking, and grade .
additional file 1: table s <dig> shows the coefficients of the fitted cox regression.
the proposed risk score was the most significant predictor as compared with the other four clinical variables.
comparisons of the 16-gene signature with published lung cancer prognostic signatures
the predictive performances of the  <dig> gene signature were also compared with the published lung cancer prognostic signatures.
six signature sets from five studies were considered  <cit> .
we collected the gene symbols from their original literatures and followed the same evaluation procedures described previously to test their predictability.
the results are summarized in table 4; it appeared that the 16-gene signature identified in this study was superior to those identified in other studies.
specifically, among the published signatures, the best performance was observed in the  <dig> genes reported in the study of kratz et al.
<cit> .
these  <dig> genes were able to predict four datasets with one on the border line .
the other signatures were effective predictors for  <dig> or  <dig> datasets.
therefore, the results suggested that our analysis approach not only was able to reproduce published gene signatures from other studies but also identified a set of predictive genes for lung adenocarcinoma.
aonly adenocarcinoma patients were analyzed here.
bcpeb <dig> was missing in the probeset of affymetrix u133a platform.
cwnt3a was missing in the probeset of affymetrix u133a platform.
donly  <dig> genes remained after removing redundant probes.
emtx <dig> was missing in the probeset of affymetrix u133a platform.
exploration of association between the 16-gene signature and drug response
finally, we explored the ability of the 16-gene signature to serve as a predictive biomarker for drug response.
to address this issue, we considered the microarrays of gse <dig>  <cit> , which contained gene expression profiles and efficacy data of  <dig> drugs for  <dig> human lung cancer cell lines.
the drug irinotecan was not included in the analysis because the data were missing in more than 50% of the  <dig> cell lines.
the expression data were retrieved from the geo website and normalized by the quantile algorithm.
summarized pathway scores of identified genes were obtained by following the same procedures described previously.
the efficacies of drugs on each cell line were represented by the variable activity area, with a higher value of the activity area representing higher inhibition ability of the corresponding drug in the cell line.
the authors determined the activity area of a drug by measuring its proportion of relative growth inhibition compared to the untreated samples at different drug concentrations  <cit> .
for each drug, a linear regression model was utilized to evaluate the association between the value of the activity area and the derived risk score.
the results, summarized in additional file 1: table s <dig>  indicated that the risk score was associated with three drugs, nutlin- <dig>  raf <dig> and zd- <dig>  subsequently, the  <dig> cell lines were divided into three groups based on their scores and two non-parameter tests, kruskal-wallis test and wilcoxon rank sum test, were performed.
only the drug, zd- <dig>  targeting egfr, showed significant differences in both statistical tests , and better drug response was observed in the patients classified in the “high” group.
this suggested that high-risk lung adenocarcinoma patients with poor survival may benefit from receiving zd- <dig> treatment.
in summary, these results implied that our proposed pathway score is not only a prognostic biomarker for survival outcomes but also a predictive biomarker for selecting potentially useful drugs to treat lung adenocarcinoma patients.
discussion
in the development of the 16-gene signature, the gene filtering procedure used the log-rank test to exclude genes showing no associations with survival outcomes .
the p-value cut off was set at  <dig>  in order to prevent excessive removal of genes.
a similar strategy was utilized in a published report on ovarian cancer  <cit> , in which the authors used a slightly looser threshold  to retain the top quartile of associated genes.
in our data, about 22% of the genes  passed the threshold .
in other words, approximately 80% of the genes were removed by this step, which excluded the majority of genes examined by microarray.
a more stringent cut off of p-values may help decrease the false-positive rates.
however, two important questions may arise by using this approach.
first, the proposed pathway-based analyses required a sufficient number of input genes in order to preserve the biological meaning of functional-relevant gene sets.
next, it is well-known that a huge discrepancy was observed across independent datasets, which may be attributed to different experimental protocols, distinct ethnicity, and dissimilar clinical parameters of investigated patients.
although a stringent threshold may provide better results for a single dataset, it often simultaneously increases the chance of overfitting the training model.
since the goal of this study is to identify predictive and reproducible genes across different studies, we prefer a slightly looser p-value cut off to retain information and the generality of results.
four biological databases were utilized to provide functional annotations for the selected genes, and our results revealed that many cancer-related pathways were possible predictors for survival outcomes in lung adenocarcinoma .
among them, an internal validation test excluded  <dig> pathways that showed low predictability.
furthermore, in the external validation, the pathway that was shown to be reproducible across  <dig> datasets does not have the smallest p-value from the log-rank test results.
this implied that filtering survival predictors based purely on the p-value may not be the best strategy because such approach may cause overfitting of the model to the training dataset.
in addition, lung cancer is such a heterogeneous disease and independent datasets had different ethnic characteristics, various experimental platforms, and distinct clinical features of patients.
a pathway having the highest reproducibility across independent studies may be more applicable to clinic use than a pathway with the most significant p-value in a single dataset.
furthermore, for the  <dig> pathways shown in table  <dig>  we collected the most significant gene from each pathway into a gene set.
the same scoring and evaluation procedures described previously were performed in  <dig> lung adenocarcinoma datasets by using this gene set.
the result showed that its prediction performance and consistency were inferior to that obtained from the  <dig> genes, suggesting that using the union of the most significant gene across different pathways may be not the best choice.
several previous studies have reported that the  <dig> genes identified in this study were highly related to the development, progression and treatment of cancer cells  <cit> .
dnm1l was associated with poor survival in lung adenocarcinoma patients when co-expressing with pkp <dig> <cit> , and nuclear expression of dnm1l was correlated with not only poor prognosis for lung adenocarcinoma but also drug resistance during hypoxia  <cit> .
cleavage of dsg <dig> was mediated by junction opener  <dig> , and downstream signaling combined with a monoclonal antibody targeting egfr, erbitux, provide better therapeutic outcomes in egfr-positive lung cancer  <cit> .
dsp has been reported as an overexpressing gene in high-risk patients with stage one non-small cell lung cancer , and is included in the 64-gene signature to predict survival outcomes  <cit> .
the high mobility group  <dig> protein  was able to increase the drug sensitivity of cisplatin in lung cancer cells by inhibiting repair of the dna damage resulting from cisplatin  <cit> .
a previous study demonstrated that kpnb <dig> was overexpressed in cervical cancer cells, and decreased expression of kpnb <dig> greatly reduced cell proliferation and triggered cell death procedures  <cit> .
the expression of lmnb <dig> was significantly up-regulated in the plasma and tissue of hepatocellular carcinoma patients, and it was associated with tumor stages, tumor sizes, and number of nodules  <cit> .
low mrna and protein expression of mapt was correlated with high expression of tubb <dig>  which showed prognostic significance with disease-free survival and overall survival in patients with early breast cancer  <cit> .
cldn <dig> was a transcriptional activator of ocln, and was also reported as not only a metastasis suppressor but also a possible prognostic predictor for lung adenocarcinoma  <cit> .
pak <dig> was a necessary interaction partner of arhgdib, and knockdown of pak <dig> greatly reduced arhgdib-induced cell invasion and arhgdib-mediated chemoresistance in gastric cancer  <cit> , and loss of satb <dig> in lung cancer has been shown as a possible marker for poor survival  <cit> .
identification of possible prognostic biomarkers for various cancers has been widely investigated in the past two decades, and several successful studies have been reported  <cit> .
for instance, the mammaprint® test divided breast cancer patients into high and low risk groups for distant metastases within  <dig> years by examining the expression signature of  <dig> genes  <cit> , and it has been the only signature approved by the fda for clinical use.
regarding lung cancer, in addition to the irreproducibility of identified biomarkers across independent datasets  <cit> , gene expression profiles obtained from different microarray platforms pose another challenge.
a predicting model developed using one microarray platform usually cannot be directly utilized in another microarray platform due to differences in their dynamic ranges of intensity values and lack of a normalization algorithm to adjust them to the same baseline.
therefore, a predictive model will become more useful when it is not necessary to rebuild the model using different parameters in independent datasets.
in this study, the medians of proposed scores in the  <dig> lung adenocarcinoma datasets were  <dig>  except for the beer dataset.
the median score in the beer dataset was  <dig> based on only  <dig> genes studied, since two genes, dsp and stk <dig>  were not examined in the corresponding microarray platform.
in summary, our approach provides a consistent cutting threshold across different microarray platforms, and thus may enable application to independent studies.
to further investigate the proposed analysis procedures in this study, cox hazard regression models were used to evaluate the top  <dig> pathways shown in table  <dig>  generally, the risk scores obtained from the other  <dig> pathways were significantly associated with survival outcomes in at least  <dig> datasets , suggesting that our approach is effective in selection of predictive genes from the whole gene pool.
in addition, we followed the same procedures described previously and performed the three statistical tests to evaluate the drugs associated with the scores obtained from the second pathway, the biocarta cell cycle pathway.
the scores were significantly correlated with three drugs using a linear regression model , and two of them, erlotinib and lapatinib, also passed the kruskal-wallis and wilcoxon rank sum tests.
unexpectedly, these two drugs both target egfr, concurring that treating lung cancer patients using egfr inhibitors is a feasible approach.
therefore, our proposed methodology may facilitate classification of patients into different groups based on their drug responses.
identification of prognostic biomarkers for lung cancer is useful but it may become more valuable if the predicting signature can simultaneously reveal the drug response.
although zd- <dig>  also known as vandetanib, targeting egfr, has not been approved for treatment of nsclc, it has been used in medullary thyroid cancer  <cit> .
one randomized phase  <dig> clinical trial demonstrated the efficacy of vandetanib in locally advanced or metastatic differentiated thyroid cancer  <cit> ; another meta-analysis study including  <dig> trials indicated that nsclc patients treated by vandetanib showed better progression-free survival, especially the adenocarcinoma subtype  <cit> .
this suggests that vandetanib has high potential for approval and use in treatment of patients with lung cancer  <cit> .
therefore, since the efficacy of vandetanib was associated with the score proposed to summarize the  <dig> genes , this signature may serve as both prognostic and predictive biomarkers for lung adenocarcinoma patients in the future.
to further explore the potential of the 16-gene signature used as a predictive biomarker, we evaluated its performance by analyzing another nsclc dataset with treatment information, gse <dig>  <cit> .
the series matrix file of gse <dig> was retrieved from the geo database.
there were  <dig> lung cancer patients in the series matrix file but only  <dig> adenocarcinoma patients.
we applied the same procedure to derive the risk scores of each lung adenocarcinoma patient.
unfortunately, no significant difference between “high” and “low” groups was observed in the  <dig> lung adenocarcinoma patients using the 16-gene signature .
among the  <dig> lung adenocarcinoma patients, there are  <dig> treated patients and  <dig> controls.
again, no significant differences were observed in these two sub-groups.
these insignificant results might be attributed to the small sample size .
the  <dig> treated patients were insufficient for subgroup analysis of difference between high and low risk groups.
in addition, we evaluated the performance of the 16-genes on the  <dig> lung squamous cell carcinoma patients.
the result was similar to the previous analysis that the 16-gene signature can’t significantly predict the squamous cell carcinoma patients.
the 16-gene signature cannot serve as a predictive signature in gse <dig>
CONCLUSIONS
in this study, we identified  <dig> predictive genes for lung adenocarcinoma patients via pathways.
four widely used biological databases were utilized to provide the functional annotations of genes.
the risk scores derived from these  <dig> genes were reproduced in  <dig> independent studies, which account for a total of more than  <dig> lung adenocarcinoma patients.
however, the risk scores showed no predictive ability for lung squamous cell carcinoma, suggesting that these  <dig> genes were subtype-specific for lung cancer.
lastly, statistical tests showed that the 16-gene signature was associated with the efficacy of an anti-cancer drug, zd- <dig>  directly targeting egfr.
therefore, these  <dig> genes may serve as not only prognostic biomarkers but also predictive biomarkers for lung adenocarcinoma, and have potential for clinical use in the future.

 lung cancer is the leading cause of cancer-related death in the united states and worldwide, more than 25% of cancer-related deaths in both men and women resulted from lung cancer. in the past few decades, considerable research effort has been devoted to elucidating the etiology of lung cancer and identifying possible biomarkers for predicting survival outcomes. however, the overall five-year survival rate for lung cancer is still less than 20%. although several studies have identified expression signatures associated with survival outcomes in their original discovery datasets, these expression signatures were often found to be irreproducible across studies. a previous report has shown that the largest number of overlapping predictive genes between two independent studies was only four, and most often even zero. the lack of reproducibility in independent studies makes it very difficult to justify conducting prospective clinical validation studies and application to clinical practice. therefore, how to effectively identify a set of predictive and reproducible biomarkers across different lung cancer studies poses a major challenge. to address this issue, one possible approach for biomarker identification is to perform analysis based on biologically functional relevant gene sets rather than to examine the associations by traditional gene-by-gene approaches. several reports have demonstrated that predictive genes that were identified by approaches considering biological pathway or cellular function information are more often reproduced and validated in independent studies. for example, kang et al. selected effective genes to predict patient survival outcomes and chemotherapy responses in ovarian cancer from a pool of genes summarized by dna repair pathways. an approach based on functional gene sets not only provides better understanding of biological meaning of identified gene markers, but also greatly increases statistical power of identification by considering only the number of pathways rather than the number of genes in the multiple comparison adjustment. the approach to performing gene set enrichment analyses for gene expression profiling has become a popular methodology even for genome-wide association studies. in a recent study of an analysis of published gene expression signatures for breast cancer, the authors showed that the predictive performance of several signatures was not superior to that obtained from random gene expression signatures, and sometimes even worse. there was a tendency for significant marker sets to depend on the size of the set; and over 90% of random signatures with more than gene members were significant predictors for survival outcome. this might be attributed to a lack of independence among the genes in the array, and a predictive model tending to become significant as more gene members were included. in order to reduce the possibility of identifying false-positive signatures, resample techniques can be performed to demonstrate significant findings, as compared to a randomly selected signature set with a same number of genes. genomic signatures can be classified into two major types: prognostic biomarkers and predictive biomarkers. prognostic biomarkers aim to help physicians divide patients into distinct groups based on their risk factors and disease characteristics, which hints that different drug treatments including standard treatment, or advanced aggressive therapeutic procedures may be applied to groups of patients showing poor survival outcomes. predictive biomarkers for a specific treatment are able to identify a subgroup of suitable patients for receiving the treatment, and those patients expected to have poor responses would be spared from the treatment. compared with prognostic biomarkers, predictive biomarkers provide physicians direct suggestions for specific therapeutic methods for individual patients. currently, most identified signatures in lung cancer fall into the category of prognostic biomarkers; the lack of studies for predictive biomarkers is partly due to many considerations required before conducting a treatment-specific randomized clinical trial. recently, barretina et al. used microarrays to investigate the gene expression profiles of human cancer cell lines along with their responses after exposure to anticancer drugs. among the drugs, were designed to directly target egfr, whose association with lung cancer has been widely reported. this dataset provides a good template for evaluation of predictive biomarkers in lung cancer. in the development of a predictive signature, there are two primary assessment considerations: 1) predictability and 2) generalizability. predictability is the ability of the signature to predict survival risk scores of patients who were not included in the signature development, but who are from a similar population. to evaluate predictability is to determine whether the signature can provide a relative ranking of patient risk scores correctly. a predictive signature that performed well in the development study might not be reproduced in performance when applied to other studies. generalizability is to determine whether predictability is reproducible when applied to data generated independently in different circumstances, such as different locations or different times. a predictive signature needs to establish its predictability and generalizability before a prospective clinical validation is conducted. in this study, we propose an approach for identifying prognostic gene signatures using lung adenocarcinoma patients from the study of shedden et al.. the approach consists of a two-stage assessment: predictability and generalizability. the shedden dataset was used for predictability assessment, and additional independent lung cancer datasets were used for generalizability assessment. the shedden dataset was tested using 2-fold cross-validation. generalizability of signature sets was evaluated and compared with several published lung cancer prognostic signatures using a randomization test. a set of genes involved in the function of apoptotic execution phase was shown to have good predictability as well as generalizability in more than lung adenocarcinoma patients, including the patients from the shedden et al. study. this signature set was also found to associate with the efficacy of the anti-cancer drug zd- targeting egfr.  identification of significant survival predictors among the genes investigated in the dataset of shedden et al., log-rank tests were performed to exclude those genes determined to have no association with survival. a total of genes were selected and their functions were annotated by four biological databases, including kegg, biocarta, pid and reactome. we focused on pathways that were closely related to the development of cancer cells, involving apoptosis, cell cycle and cell death regulations. for each pathway, the risk score of each patient was computed, and patients were divided into two groups according to the procedure described in the method section. the log-rank test was performed to determine statistical significance of the signature set associated with survival outcomes for the pathway. table  shows the p-values of pathways that were significant using the bonferroni correction. a randomization test was performed and all pathways were highly significant. aonly genes showing association with survival were analyzed. bestimated by using log-rank tests. cthe frequency showing p-values <  in the trials of the internal validation assessment. dthe frequency showing p-values <  in the trials of the internal validation assessment. predictability assessment internal analytical validation was performed on the shedden study to assess predictability of the signature sets using 2-fold cross-validation. the samples were randomly divided into two sets of equal size, a training and a test set. in the training phase, gene filtering, cox model fitting, total risk score calculation, and log-rank test were performed according to the procedures described in the method section. the median of the risk scores was computed for each pathway. in the test phase, a score of was initially assigned to each patient. for each gene, a score of was given to a patient if the cox coefficient from the training set was greater than and the patient’s expression level was higher or equal to the median, or if the coefficient was less than and the expression level was less than the median. the total risk score of each patient was computed for each pathway. next, patients were classified into two risk groups based on the median of the risk scores from the training set. the log-rank test was performed to compare the two risk groups at the significance level of. here we considered both the unadjusted p-value and the bonferroni adjusted p-value. the procedure was repeated, but the model was developed on the test set and applied to the training set. the entire procedure was repeated times to generate different 2-fold partitions. the frequency of significant results out of the repetitions was calculated for each pathway. the last two columns of table  list the frequencies of the significant results for the pathways. we selected the top pathways which showed at least significant counts at α =  / for the generalizability assessment. these numbers can be regarded as the power for the predictability. generalizability assessment generalizability requires that predictive performance of a model developed from one study is reproducible across different studies. five additional microarray studies for a total of studies were evaluated. among the studies considered, studies contained adenocarcinoma samples and studies contained squamous cell carcinoma samples. evaluation of generalizability is similar to the evaluation of predictability in which all samples from the shedden study were used as the training dataset and applied to each of independent studies as a test dataset. briefly, four microarray datasets were retrieved from the gene expression omnibus, and the quantile normalization algorithm was utilized to remove systematic biases. gene expression levels of multiple probes annotated with the same gene symbol were summarized into a single value by using their means. each independent dataset was then evaluated using the procedures described in the assessment of predictability. table  shows the p-values and the number of significant results for the most significant pathways for the five adenocarcinoma datasets, including shedden and four independent datasets. several pathways were significant for or more datasets; the pathway “apoptotic execution phase” was significant for all datasets, an excellent criterion for generalizability. aonly adenocarcinoma patients were analyzed here. in addition, a resampling test was performed to simulate chance significance for the five adenocarcinoma datasets. for each pathway, a null signature set consisting of randomly selected genes from the shedden study was generated, where the numbers of genes were kept the same as the pathway. the null signature set was applied to the five datasets; the total number of significant results was compared to the observed number of significant results in column to determine if the selected signature set was superior to the null signature set. the simulation was repeated times. the last column shows the p-value for the pathways, where the p-value was defined as the proportion of the number of significances from the null signature set was greater or equal to the observed number in column out of the repetitions. the pathways were also applied to four squamous cell carcinoma datasets, gse gse gse and gse and no statistical significance was identified for any of pathways and datasets. this suggested that these pathways were specific to the lung adenocarcinoma. the 16-gene signature the genes involved in the apoptotic execution phase pathway and statistical p-values of the log-rank test from step are listed in table  among them, the most significant gene was dnm1l, which concurred with high expression of pkp and dnm1l associated with poor survival in lung adenocarcinoma patients. figure  shows the kaplan–meier survival curves with the p-values from the log-rank test for the datasets. the proposed calculation of risk scores scheme and classification of patients into “high” and “low” groups appeared to perform well in both training and test data. for example, in the study of shedden et al., the median survival of the patients belonging to the “high” group is obviously shorter than that from the “low” group. similarly, the median survival also showed dramatic differences between the “high” and “low” groups in gse. the p-values from fitting the cox hazard regression model using the risk scores as an independent variable are shown in figure  the p-values were consistent with the p-values from the log-rank test; a minor discrepancy is that the dataset gse has ties at the median score resulting in lower discriminating ability. we also performed an analysis dividing the patients into three groups, and an analysis of the squamous cell carcinoma patients across independent datasets. the results are consistent with the results present above. a“+” denotes the fitted cox efficient >  whereas “-“denotes the fitted cox coefficient <  the risk scores and four clinical variables were analyzed simultaneously using the multivariate cox regression. the four clinical variables were age, gender, smoking, and grade. additional file 1: table s shows the coefficients of the fitted cox regression. the proposed risk score was the most significant predictor as compared with the other four clinical variables. comparisons of the 16-gene signature with published lung cancer prognostic signatures the predictive performances of the gene signature were also compared with the published lung cancer prognostic signatures. six signature sets from five studies were considered. we collected the gene symbols from their original literatures and followed the same evaluation procedures described previously to test their predictability. the results are summarized in table 4; it appeared that the 16-gene signature identified in this study was superior to those identified in other studies. specifically, among the published signatures, the best performance was observed in the genes reported in the study of kratz et al.. these genes were able to predict four datasets with one on the border line. the other signatures were effective predictors for or datasets. therefore, the results suggested that our analysis approach not only was able to reproduce published gene signatures from other studies but also identified a set of predictive genes for lung adenocarcinoma. aonly adenocarcinoma patients were analyzed here. bcpeb was missing in the probeset of affymetrix u133a platform. cwnt3a was missing in the probeset of affymetrix u133a platform. donly genes remained after removing redundant probes. emtx was missing in the probeset of affymetrix u133a platform. exploration of association between the 16-gene signature and drug response finally, we explored the ability of the 16-gene signature to serve as a predictive biomarker for drug response. to address this issue, we considered the microarrays of gse, which contained gene expression profiles and efficacy data of drugs for human lung cancer cell lines. the drug irinotecan was not included in the analysis because the data were missing in more than 50% of the cell lines. the expression data were retrieved from the geo website and normalized by the quantile algorithm. summarized pathway scores of identified genes were obtained by following the same procedures described previously. the efficacies of drugs on each cell line were represented by the variable activity area, with a higher value of the activity area representing higher inhibition ability of the corresponding drug in the cell line. the authors determined the activity area of a drug by measuring its proportion of relative growth inhibition compared to the untreated samples at different drug concentrations. for each drug, a linear regression model was utilized to evaluate the association between the value of the activity area and the derived risk score. the results, summarized in additional file 1: table s indicated that the risk score was associated with three drugs, nutlin- raf and zd- subsequently, the cell lines were divided into three groups based on their scores and two non-parameter tests, kruskal-wallis test and wilcoxon rank sum test, were performed. only the drug, zd- targeting egfr, showed significant differences in both statistical tests, and better drug response was observed in the patients classified in the “high” group. this suggested that high-risk lung adenocarcinoma patients with poor survival may benefit from receiving zd- treatment. in summary, these results implied that our proposed pathway score is not only a prognostic biomarker for survival outcomes but also a predictive biomarker for selecting potentially useful drugs to treat lung adenocarcinoma patients. discussion in the development of the 16-gene signature, the gene filtering procedure used the log-rank test to exclude genes showing no associations with survival outcomes. the p-value cut off was set at in order to prevent excessive removal of genes. a similar strategy was utilized in a published report on ovarian cancer, in which the authors used a slightly looser threshold to retain the top quartile of associated genes. in our data, about 22% of the genes passed the threshold. in other words, approximately 80% of the genes were removed by this step, which excluded the majority of genes examined by microarray. a more stringent cut off of p-values may help decrease the false-positive rates. however, two important questions may arise by using this approach. first, the proposed pathway-based analyses required a sufficient number of input genes in order to preserve the biological meaning of functional-relevant gene sets. next, it is well-known that a huge discrepancy was observed across independent datasets, which may be attributed to different experimental protocols, distinct ethnicity, and dissimilar clinical parameters of investigated patients. although a stringent threshold may provide better results for a single dataset, it often simultaneously increases the chance of overfitting the training model. since the goal of this study is to identify predictive and reproducible genes across different studies, we prefer a slightly looser p-value cut off to retain information and the generality of results. four biological databases were utilized to provide functional annotations for the selected genes, and our results revealed that many cancer-related pathways were possible predictors for survival outcomes in lung adenocarcinoma. among them, an internal validation test excluded pathways that showed low predictability. furthermore, in the external validation, the pathway that was shown to be reproducible across datasets does not have the smallest p-value from the log-rank test results. this implied that filtering survival predictors based purely on the p-value may not be the best strategy because such approach may cause overfitting of the model to the training dataset. in addition, lung cancer is such a heterogeneous disease and independent datasets had different ethnic characteristics, various experimental platforms, and distinct clinical features of patients. a pathway having the highest reproducibility across independent studies may be more applicable to clinic use than a pathway with the most significant p-value in a single dataset. furthermore, for the pathways shown in table  we collected the most significant gene from each pathway into a gene set. the same scoring and evaluation procedures described previously were performed in lung adenocarcinoma datasets by using this gene set. the result showed that its prediction performance and consistency were inferior to that obtained from the genes, suggesting that using the union of the most significant gene across different pathways may be not the best choice. several previous studies have reported that the genes identified in this study were highly related to the development, progression and treatment of cancer cells. dnm1l was associated with poor survival in lung adenocarcinoma patients when co-expressing with pkp, and nuclear expression of dnm1l was correlated with not only poor prognosis for lung adenocarcinoma but also drug resistance during hypoxia. cleavage of dsg was mediated by junction opener, and downstream signaling combined with a monoclonal antibody targeting egfr, erbitux, provide better therapeutic outcomes in egfr-positive lung cancer. dsp has been reported as an overexpressing gene in high-risk patients with stage one non-small cell lung cancer, and is included in the 64-gene signature to predict survival outcomes. the high mobility group protein was able to increase the drug sensitivity of cisplatin in lung cancer cells by inhibiting repair of the dna damage resulting from cisplatin. a previous study demonstrated that kpnb was overexpressed in cervical cancer cells, and decreased expression of kpnb greatly reduced cell proliferation and triggered cell death procedures. the expression of lmnb was significantly up-regulated in the plasma and tissue of hepatocellular carcinoma patients, and it was associated with tumor stages, tumor sizes, and number of nodules. low mrna and protein expression of mapt was correlated with high expression of tubb which showed prognostic significance with disease-free survival and overall survival in patients with early breast cancer. cldn was a transcriptional activator of ocln, and was also reported as not only a metastasis suppressor but also a possible prognostic predictor for lung adenocarcinoma. pak was a necessary interaction partner of arhgdib, and knockdown of pak greatly reduced arhgdib-induced cell invasion and arhgdib-mediated chemoresistance in gastric cancer, and loss of satb in lung cancer has been shown as a possible marker for poor survival. identification of possible prognostic biomarkers for various cancers has been widely investigated in the past two decades, and several successful studies have been reported. for instance, the mammaprint® test divided breast cancer patients into high and low risk groups for distant metastases within years by examining the expression signature of genes, and it has been the only signature approved by the fda for clinical use. regarding lung cancer, in addition to the irreproducibility of identified biomarkers across independent datasets, gene expression profiles obtained from different microarray platforms pose another challenge. a predicting model developed using one microarray platform usually cannot be directly utilized in another microarray platform due to differences in their dynamic ranges of intensity values and lack of a normalization algorithm to adjust them to the same baseline. therefore, a predictive model will become more useful when it is not necessary to rebuild the model using different parameters in independent datasets. in this study, the medians of proposed scores in the lung adenocarcinoma datasets were except for the beer dataset. the median score in the beer dataset was based on only genes studied, since two genes, dsp and stk were not examined in the corresponding microarray platform. in summary, our approach provides a consistent cutting threshold across different microarray platforms, and thus may enable application to independent studies. to further investigate the proposed analysis procedures in this study, cox hazard regression models were used to evaluate the top pathways shown in table  generally, the risk scores obtained from the other pathways were significantly associated with survival outcomes in at least datasets, suggesting that our approach is effective in selection of predictive genes from the whole gene pool. in addition, we followed the same procedures described previously and performed the three statistical tests to evaluate the drugs associated with the scores obtained from the second pathway, the biocarta cell cycle pathway. the scores were significantly correlated with three drugs using a linear regression model, and two of them, erlotinib and lapatinib, also passed the kruskal-wallis and wilcoxon rank sum tests. unexpectedly, these two drugs both target egfr, concurring that treating lung cancer patients using egfr inhibitors is a feasible approach. therefore, our proposed methodology may facilitate classification of patients into different groups based on their drug responses. identification of prognostic biomarkers for lung cancer is useful but it may become more valuable if the predicting signature can simultaneously reveal the drug response. although zd- also known as vandetanib, targeting egfr, has not been approved for treatment of nsclc, it has been used in medullary thyroid cancer. one randomized phase clinical trial demonstrated the efficacy of vandetanib in locally advanced or metastatic differentiated thyroid cancer; another meta-analysis study including trials indicated that nsclc patients treated by vandetanib showed better progression-free survival, especially the adenocarcinoma subtype. this suggests that vandetanib has high potential for approval and use in treatment of patients with lung cancer. therefore, since the efficacy of vandetanib was associated with the score proposed to summarize the genes, this signature may serve as both prognostic and predictive biomarkers for lung adenocarcinoma patients in the future. to further explore the potential of the 16-gene signature used as a predictive biomarker, we evaluated its performance by analyzing another nsclc dataset with treatment information, gse. the series matrix file of gse was retrieved from the geo database. there were lung cancer patients in the series matrix file but only adenocarcinoma patients. we applied the same procedure to derive the risk scores of each lung adenocarcinoma patient. unfortunately, no significant difference between “high” and “low” groups was observed in the lung adenocarcinoma patients using the 16-gene signature. among the lung adenocarcinoma patients, there are treated patients and controls. again, no significant differences were observed in these two sub-groups. these insignificant results might be attributed to the small sample size. the treated patients were insufficient for subgroup analysis of difference between high and low risk groups. in addition, we evaluated the performance of the 16-genes on the lung squamous cell carcinoma patients. the result was similar to the previous analysis that the 16-gene signature can’t significantly predict the squamous cell carcinoma patients. the 16-gene signature cannot serve as a predictive signature in gse  in this study, we identified predictive genes for lung adenocarcinoma patients via pathways. four widely used biological databases were utilized to provide the functional annotations of genes. the risk scores derived from these genes were reproduced in independent studies, which account for a total of more than lung adenocarcinoma patients. however, the risk scores showed no predictive ability for lung squamous cell carcinoma, suggesting that these genes were subtype-specific for lung cancer. lastly, statistical tests showed that the 16-gene signature was associated with the efficacy of an anti-cancer drug, zd- directly targeting egfr. therefore, these genes may serve as not only prognostic biomarkers but also predictive biomarkers for lung adenocarcinoma, and have potential for clinical use in the future. 
subsequently, their predictability and generalizability were evaluated by the 2-fold cross-validation and a resampling test in independent datasets, respectively. were used as the training set, and the log-rank test was performed to select potential signature genes. lung cancer is the leading cause of cancer-related death worldwide. a scoring scheme was developed by the cox hazard regression model, and patients were divided into two groups based on the medians. a set of genes related to apoptosis execution was demonstrated to have good predictability as well as generalizability in more than lung adenocarcinoma patients and was reproducible in independent datasets. in summary, we presented a new approach to identify reproducible survival predictors for lung adenocarcinoma, and the identified genes may serve as both prognostic and predictive biomarkers in the future. in this study, we proposed an approach by identifying predictive genes through pathways. many biomarkers have been identified for predicting survival; challenges arise, however, in translating the findings into clinical practice due to their inconsistency and irreproducibility.
 lung cancer is the leading cause of cancer-related death worldwide. tremendous research efforts have been devoted to improving treatment procedures, but the average five-year overall survival rates are still less than 20%. many biomarkers have been identified for predicting survival; challenges arise, however, in translating the findings into clinical practice due to their inconsistency and irreproducibility. in this study, we proposed an approach by identifying predictive genes through pathways.  the microarrays from shedden et al. were used as the training set, and the log-rank test was performed to select potential signature genes. we focused on cancer-related pathways from biological databases. a scoring scheme was developed by the cox hazard regression model, and patients were divided into two groups based on the medians. subsequently, their predictability and generalizability were evaluated by the 2-fold cross-validation and a resampling test in independent datasets, respectively. a set of genes related to apoptosis execution was demonstrated to have good predictability as well as generalizability in more than lung adenocarcinoma patients and was reproducible in independent datasets. this signature set was shown to have superior performances compared to other published signatures. furthermore, the corresponding risk scores derived from the set were found to associate with the efficacy of the anti-cancer drug zd- targeting egfr.  in summary, we presented a new approach to identify reproducible survival predictors for lung adenocarcinoma, and the identified genes may serve as both prognostic and predictive biomarkers in the future. lung adenocarcinomamicroarraypathway analysisprognostic biomarkerpredictive biomarker 