Abstract ED02-03: Development of clinical trials incorporating genomic signatures: Lessons learned.
Authors: McShane Lisa M. 
Published: June 2012

Two types of genomic signatures commonly studied in clinical trials are prognostic and predictive signatures. A genomic signature can be viewed as a collection of genomic markers, for example gene expression levels or mutations, that are combined and assessed for patterns having biological or clinical significance, usually through development of mathematical models or predictors. The result generated by a predictor is generally a measurement such as a risk score or an indicator of membership in a biological subgroup. Prognostic refers to a measurement associated with clinical outcome in absence of therapy (natural course) or sometimes the definition is broadened to apply in the context of standard therapy all patients are likely to receive. Predictive refers to a measurement associated with response (benefit) or lack of response (benefit) to a particular therapy relative to other available therapies, so predictive signatures can help guide selection of treatment.

A first step in assessment of whether a genomic signature should be studied further is consideration of its potential clinical utility. Too many genomic predictors described in scientific literature result from exercises in mathematical and statistical model building without clear focus on clinical utility. A prognostic predictor is clinically useful if the prognostic information is sufficiently strong to influence clinical decisions and it provides information beyond standard prognostic factors. A predictive signature should have the ability to distinguish groups with sufficiently different treatment benefit such that, when weighed against potential risks and costs, the information is clinically actionable. Subgroups defined by a predictor can be both prognostic and predictive, and it is important to consider appropriate control groups to distinguish these properties. Specimens from randomized clinical trials permit the most reliable evidence for the value of predictive (treatment-selection) signatures.

Before a genomic signature is used in a clinical trial where it will influence patient care, there should be a review of the process followed to develop the signature. First, any available information about the specimens (single or multiple source, sample collection, handling and preservation) and laboratory settings where the “omics” assays were conducted (one or multiple laboratories, one or multiple assay batches) should be reviewed to examine for potential confounders that could bias the observed association between predictor result and outcome. A number of processing steps are involved in developing genomic predictors from raw data obtained by assay of specimen collections. These data pre-processing steps include normalization, calibration, standardization, and summary measure calculations. All of these steps must be documented, because changes in any of the steps could modify performance of the predictor. Some of these pre-processing steps may require modification for use in a prospective clinical trial setting where data will be processed from one specimen at a time, unlike in the signature development process where steps such as normalization might have been applied to data from a batch of specimens. Criteria to screen for and remove unsuitable data or specimens must be predetermined and not adjusted after a preliminary evaluation of predictor performance has been conducted using data from those specimens. These criteria should be in place, and automated to the extent possible, prior to initiating a prospective clinical trial.

Preliminary assessments of signature performance should be reviewed. When building omics predictors in the usual setting where the number of omics variables greatly exceeds the number of independent observations (study subjects), one must be particularly careful to guard against overfitting of models. Many model building procedures begin with selection of informative features, for example genes relevant to a clinical or pathological distinction of interest such as distinguishing patients who are likely to have a favorable outcome following surgery from patients who are likely to have an unfavorable outcome. Alternatively, or in addition, there might be a dimension reduction step such as calculation of principal components. A variety of statistical methods are available to accomplish these tasks. All selected features must then be combined into a mathematical model to define the predictor. It is not appropriate to evaluate the performance of the predictor by “plugging in” the same data that were were used to build it because that will lead to seriously over-optimistic estimates of performance (“re-substitution estimate”). The performance of the predictor must be evaluated using methods such as internal and external validation.

Internal validation procedures include approaches such as cross-validation, bootstrapping, and other statistical resampling methods. The basic idea underlying these approaches is to split the existing data into parts, to form “training” and “test” sets. The predictor model is built using the training set and then its performance is evaluated on the test set. This can be performed iteratively to consider multiple partitions of the data. For example, in leave-one-out cross-validation, one subject's data is removed to form the test set, the predictor model is built on the remaining data (training set), and then the left out subject's outcome is predicted from the model. The procedure iterates until the data corresponding to each patient has had a turn in the role of the test set. It is important that all steps in the model building procedure are repeated for each iteration of the cross-validation. A common error is that investigators neglect to repeat the feature selection step of the model building procedure for every iteration of the cross-validation procedure, and this can lead to substantial bias in the estimate of predictor performance in the direction of over-optimism. A limitation of internal validation is that it cannot guard against hidden biases in the data that result when there is an undetected factor that confounds the association between the molecular data and the outcome. For example, if the specimens for most patients who had a poor clinical outcome were processed in one laboratory and specimens for patients with more favorable outcome were processed in a different laboratory, a predictor built using that data could be spuriously modeling specimen processing conditions instead of predicting outcome. These types of hidden bias might not be detectable unless the predictor is tested on a completely independent data set.

External validation is the gold standard that is most protective against bias or overfitting of the predictor model. Much of the promise of high-dimensional genomic technologies lies in their ability to detect potentially subtle, but important biological characteristics; however, this also leaves them vulnerable to influence by extraneous factors, which can overshadow the important biological signals. Validation on a completely independent data set provides the best opportunity to distinguish true biological signal from artifact. A meaningful external validation of a predictor model should satisfy the following criteria:The intended patient population should be precisely defined and should constitute a group for which use of the predictor would lead to clinically relevant decisions.Criteria for specimen acceptability should be defined, and it should be established that the required types of specimens will be feasible to collect in typical clinical settings.The technical protocol for the genomic assay should be completely specified along with any assay quality monitoring and quality assurance procedures that will be applied.The predictor algorithm must be completely locked-down as a fixed mathematical model, including any data pre-processing steps or cut-points applied to the predictor for purposes of clinical decision making.Reproducibility of model predictions should be documented using a set of clinical specimens similar to those anticipated from the intended patient population.Rigorous blinding of outcome data must be enforced in the validation, generally by participation of an individual who acts as an “honest broker”.Adequate sample size is necessary to estimate predictor performance with sufficient precision.Criteria for acceptable predictor performance should be pre-specified.

Design and conduct of a prospective clinical is usually the final step in validating clinical utility of a genomic signature.

Occasionally there will be sufficient specimens available from already completed therapy trials to conduct a prospective-retrospective trial, particularly for validation of a prognostic signature. If a prospective trial is needed, as will often be the case for validation of predictive signatures, several design options are available. These include enrichment designs, stratified designs, strategy designs and variations on those types. Appropriate regulatory approvals such as investigational device exemptions must be secured as required. An approved clinical trial protocol is required and rigorous clinical trials procedures such as appropriately maintained clinical databases and monitoring oversight should be in place just as routinely expected for trials of new therapeutics.

Citation Format: {Authors}. {Abstract title} [abstract]. In: Proceedings of the AACR-NCI-EORTC International Conference: Molecular Targets and Cancer Therapeutics; 2011 Nov 12-16; San Francisco, CA. Philadelphia (PA): AACR; Mol Cancer Ther 2011;10(11 Suppl):Abstract nr ED02-03.