Many medical specialities have reviewed the statistical content of their journals. To our knowledge this has not been done in general practice. Given the main role of a general practitioner as a diagnostician we thought it would be of interest to see whether the statistical methods reported reflect the diagnostic process.British Medical Journal , British Journal of General Practice and Family Practice over a one-year period (1 January to 31 December 2000).Hand search of three UK journals of general practice namely the A wide variety of statistical techniques were used. The most common methods included t-tests and Chi-squared tests. There were few articles reporting likelihood ratios and other useful diagnostic methods. There was evidence that the journals with the more thorough statistical review process reported a more complex and wider variety of statistical techniques.BMJ had a wider range and greater diversity of statistical methods than the other two journals. However, in all three journals there was a dearth of papers reflecting the diagnostic process. Across all three journals there were relatively few papers describing randomised controlled trials thus recognising the difficulty of implementing this design in general practice.The  For example, the use of the naïve Bayes' discriminant function (and from it the derivation of likelihood ratios) is appropriate. Proponents of Bayes' argue for its simplicity and ease of interpretation,6. In coMany medical journals, both generalist,11 and sBritish Medical Journal (BMJ) , British Journal of General Practice (BJGP) and Family Practice. These three journals were chosen because they reflected the main primary care journals in the UK. The journals were hand searched for original research articles over a one-year period (1 January to 31 December 2000). Articles were classified for both their statistical content and methods of design according to criteria laid down elsewhere[Three statisticians   including one Professor, one Senior Lecturer and one Lecturer each reviewed one leading UK journal in general practice. The fourth author (NS) is a Primary Care Physician. The journals chosen were the elsewhere,20. TablThe main study was preceded by a pilot phase in which a random sample of 10 articles was classified both by statistical content and study design by the three statisticians. Where there were differences of opinion, consensus was reached by discussion. We met once to discuss our classification system, and to iron out differences of opinion. One problem lay in how we actually classified study design. For example, one of use used the phrase 'cross-sectional survey' while another used the phrase 'questionnaire survey' when both meant the same in terms of study design. Another problem was that we missed some of the statistical techniques (where there were many) and this required much more careful reading of the articles when we carried out the main survey. We did not carry out a formal reliability study of the pilot phase but instead relied on our experiences both as statisticians, and as journal reviewers. Similarly we chose not to carry out a formal reliability analysis in the main study.BMJ  (n = 79), BJGP (n = 145) and Family Practice (n = 81).The total number of articles reviewed over a one year period was as follows: BMJ, BJGP and Family Practice respectively (Table Family Practice (21.0% compared to an average of 11.8%). Qualitative studies included those encompassing terms such as 'focus groups' and 'semi-structured interviews' for example. Figure Helicobacter pylori for the detection of peptic ulcer). Examples of more unusual study designs include those based on video recordings, literature reviews and quasi-experimental designs.The most common design was that of a cross-sectional survey being found in 24.1%, 39.3% and 35.1% of articles in the BMJ shows a greater range and breadth of articles than Family Practice. More sophisticated techniques are reported more often in the BMJ than either of the other two journals. In the BMJ, the two most common statistical methods used were logistic regression  and the Chi-squared test . The two least common were the Mantel-Haenszel statistic  and Cronbach's alpha . Relatively new innovations such as random effects models were seen in both the BMJ and the BJGP. The least sophisticated statistical methods appeared in Family Practice. Methods based on likelihood ratios were seldom found in either the BMJ or BJGP and not at all in Family Practice. Nonparametric tests were often unspecified but where they were included Mann-Whitney U test, Spearman's correlation coefficient and the Wilcoxon matched-pairs signed ranks test. Multiple comparisons included Bonferonni techniques and Scheffe's contrasts. Survival analysis included Kaplan-Meier curves and Cox regression.The range of statistical methods reported can be seen in Table One-third of all articles reported no statistics or simple summaries . No journal article with a qualitative design had any statistical content.BJGP. This was due to a wide range of statistical methods being reported only once. Examples include time series, multilevel modelling and factor analysis. In others, we could not decipher which statistical techniques had been used.A large number of articles reported other statistical methods, in particular the Table BJGP and Family Practice. Papers in the BMJ and the BJGP used more sophisticated statistical methods than Family Practice . It is perhaps too easy for us to lay blame at the Editors door for this lack of sophistication. Statisticians are relatively rare, and review, for the most part, is unpaid.Two-thirds of all journal articles relied on some type of statistical analysis beyond descriptive statistics Table . The Chice Table . While byears ago. Despitelsewhere,,15,17,20lsewhere,. The BJGAlthough these three journals publish a large proportion of the research in general practice within the UK, they by no means represent 100% of it. To look at this further we examined the year 2000 and undertook a MEDLINE search using the key indexing phrase 'General Practice'. We found over 800 articles in a diversity of journals. Articles were published in the fields of rheumatology, medical ethics, obstetrics, public health, clinical pharmacology, clinical neurology and telemedicine to name but a few.We chose to look at the year 2000. Would our results be different had we selected a different year? The published literature suggests otherwise. In a 20 year old study, Emerson and Colditz found t-Now let us turn to study design. The gold standard research design is considered to be the randomised controlled trial (RCT). It has been acknowledged that carrying out RCTs in general practice are difficult,24. In oWhat are the issues here? Are they really that different from secondary care? A recent publication posed the question 'What do residents really need to know about statistics?'. The autFor all three journals there was a dearth of articles reflecting the diagnostic process. Why is this? It has already said that diagnosis is the Achilles Heel of GPs. If it iThe author(s) declare that they have no competing interests.Three authors  carried out the literature review while all four authors contributed to the writing.The pre-publication history for this paper can be accessed here:


This review article focuses on the various aspects of translational research, where research on human subjects can ultimately enhance the diagnosis and treatment of future patients. While we will use specific examples relating to the asbestos related cancer mesothelioma, it should be stressed that the general approach outlined throughout this review is readily applicable to other diseases with an underlying molecular basis. Through the integration of molecular-based technologies, systematic tissue procurement and medical informatics, we now have the ability to identify clinically applicable "genotype"-"phenotype" associations across cohorts of patients that can rapidly be translated into useful diagnostic and treatment strategies. This review will touch on the various steps in the translational pipeline, and highlight some of the most essential elements as well as possible roadblocks that can impact success of the program. Critical issues with regard to Institutional Review Board (IRB) and Health Insurance Portability and Accountability Act (HIPAA) compliance, data standardization, sample procurement, quality control (QC), quality assurance (QA), data analysis, preclinical models and clinical trials are addressed. The various facets of the translational pipeline have been incorporated into a fully integrated computational system, appropriately named Dx2Tx. This system readily allows for the identification of new diagnostic tests, the discovery of biomarkers and drugable targets, and prediction of optimal treatments based upon the underlying molecular basis of the disease. Our systematic approach to translational medicine has been designed to achieve a vision shared by numerous "ambitious" investigators who are focused on applying bench-side discoveries to the clinical setting Figure . We sharWhile a degree of project specificity must be incorporated into the design of any research effort, many of the components of translational research are shared across apparently disparate disease areas Figure . AlthougEffective translational research requires a multi-disciplinary and team approach. Our initial concept meetings include HIPAA/IRB advisors, research nurses, surgeons , pathologists and members of diagnostic service labs, information technologists  and statisticians in addition to the principal investigators. Each member of the translational research team is critical to the overall productivity and success of the pipeline. In acknowledgement of the effectiveness of this approach, we would advise other establishments to involve key personnel across the various disciplines early. We also believe that smaller dedicated teams can often be more efficient and less bureaucratic than oversized consortium.The development of research protocols on human subjects that involve specimen and/or data collection must be approved by the investigational review boards (IRB). The IRB governs patient safety and risk for the hospitals and/or universities/institutions. In accordance with federal regulations, one of the requirements states that each human subject must be thoroughly informed about the research to be undertaken on their sample and/or data during a consenting process. The specificity of the consent should be such that it outlines the research protocol, study procedures and risks and benefits and meets the federal, state and local requirements, so that the subject can make an informed decision regarding participation in the study. In retrospect, a well informed patient can help facilitate the procurement of clinical data and samples in an efficient and longitudinal manner..While the typical testing of a hypothesis requires the collection of only a subset of the available clinical data , the emerging field of medical informatics and electronic charts allows for the potential collection of vast quantities of standardized clinical data that potentially harbors invaluable information with regard to additional "traits" or "phenotypes". As such, we currently collect upward of 9000 potential data fields per patient within our specific IRB-approved protocols, all of which can be populated during the normal course of patient care and stored within either electronic or paper records for each subject. In addition, each patient receives a standardized self-reporting questionnaire addressing histories . Any protected fields are restricted through a security portal, which can be considered a suitable clinical gatekeeper  Figure . This seIn addition to IRB and HIPAA issues, patient recruitment can also be a major hurdle. The teamwork approach is critical here, together with patient/physician outreach and screening efforts as necessary. Our experience is the vast majority of patients are enthusiastic about inclusion in research studies so long as they are informed of the opportunity, particularly since we are careful to follow standard-of-care. For the most part, we are finding a willingness to participate for the hope of betterment for the whole, whether it is potentially helping their own families or others with their disease. At this level, the physician is the central figure and is uniquely positioned to introduce the protocol and support patient participation in the study. For studies where low participation is expected (predominantly due to low incidence), we typically include an IRB-approved public educational effort  that maximizes recruitment. Our clinical research nurses work closely with the physician(s) and their office scheduling staff for notification of potential candidates in advance wherever possible. This team approach with the physician advocating the effort allows the greatest accrual particularly when there are several geographic sites for meeting potential protocol volunteers. On site consenting of high risk or suspected individuals within the outpatient clinic and hospital settings where these patients are typically evaluated should to be considered to enhance accrual. For patients with known disease, the best environment for accrual occurs in a multidisciplinary setting in which the key physician works closely with the medical oncologist, radiation oncologist, nurses, physician assistants, residents and research staff. This is particularly important for longitudinal sampling of patients on treatment protocols to minimize the loss of follow-up data. The logistics of a coordinated approach may differ between a university medical school setting and a community based practice due to the competing obligations of a busy private practice. The multidisciplinary approach is optimal in a single setting in which patients can be seen by the various disciplines, thereby, reducing the extent of patient relocation between specialty offices and allowing for centralized specimen and data collection. Nonetheless, in community-based practice, we have introduced standardized mechanisms allowing us to consent patients and collect clinical samples and data from multiple sites.There are now abundant examples where clinical researchers are using the revolutionary "omics" technologies to demonstrate a clear association between the molecular make-up ("genotype") of clinical samples and well-defined clinical characteristics ("phenotype"). Near-complete genotypes can now be obtained through multiplex technologies such as sequencing, mutational screening (such as single nucleotide polymorphism (SNP) analysis) and gene/protein expression profiling. The absence of well annotated clinical information , however, is evident from most studies reported to date. Rather than taking the reductionist approach at the time of data collection, our group instead chooses to gather all information relating to an individual's medical history as well as follow-up data as it becomes available and as approved by the IRB. It is relatively trivial to reduce the clinical data pool retrospectively during analysis as deemed necessary. Our Dx2Tx system date/time stamps individual events for a given subject/sample, such that multiple events can be temporally associated. The first step is to gather the information in its rawest form from a standardized source. This could be as complicated as a centralized medical informatics database  housing rich longitudinal clinical data that can be expressed in Health Level 7 standards for data portability), or it could be as uncomplicated as a locally maintained excel spreadsheet or access database. Regardless of the source, we attempt to collect as much information regarding a subject's history, diagnosis, treatment, and response assessment as possible. As highlighted above, the critical elements are reliability of the source data and standardization. While we do perform statistical analysis on parsed text from open string comments, we attempt to force objective measurements  wherever feasible. In the absence of a centralized clinical database utilizing standardized clinical nomenclature and data, the responsibility falls on the clinical members of the team to interpret clinical data within isolated databases and/or paper charts. Clearly, the navigation from paper to electronic medical records, and the generation of middleware that can link disparate databases, will greatly alleviate this burden that rapidly becomes a major rate-limiting step in the translational pipeline. Coupled with voice recognition and electronic data recording with inbuilt QC check-points, standardized digital data entry should become the norm in the not so distant future.Clinical data permitted under protocol should be, where possible, prospectively accumulated. The accuracy of the data, particularly those parameters such as the pathologic characteristics, clinical staging, dates of intervention, dates of intermediate endpoints (such as disease progression or death), must be unquestionable in order to identify clinical and molecular correlations among diverse datasets. The accuracy of these data will depend to a large extent upon the frequency of follow-up for parameters such as disease progression, and the degree to which the patients' status is investigated longitudinally. Patients on clinical trials, where the procedures and follow-up are specified as part of the protocol, will have the most robust and standardized data since interval radiographic and physical examinations and/or other clinical procedures and methods of data collection must be adhered to in order to avoid protocol violation. Moreover, the approach of the physician(s) who follows the patient and the availability of an existing mechanism that readily allows the retrieval of prospective data (i.e. a database) will greatly influence the accuracy of the data. Within oncology, staging issues are some of the more difficult to resolve, especially if inadequate staging is performed in the operating room (OR) or if one must rely entirely on clinical or radiographic data. For the majority of organ systems, the oncological staging system is specified in the AJCC Staging Handbook, but it is the responsibility of the investigator to know whether the pathologic and intraoperative details necessary for accurate staging were adequately performed. These include but are not limited to accurate size measurement of the primary tumor, careful intraoperative description of abutment or invasion of nearby structures, verification of negative margins, and extent of lymph node involvement. The patients used in the examples below were all part of either Phase II or Phase III mesothelioma protocols, with computerized tomographic surveillance after surgery every 3–4 months until death. As such, the accuracy of the clinical data was optimized. One surgeon performed the operations, obtained aggressive intraoperative staging in all cases, and took the responsibility for collecting samples and data from all participants in the study. This approach led to the development of a consistent standard of care philosophy for the surgical management of mesothelioma.Our Dx2Tx system performs detailed statistical analysis on both clinical and molecular data, thus leaving little room for error when it comes to standardized data collection and entry. Physicians are uniquely positioned to organize the various aspects of data collection. This endeavor begins with the design of a clinical protocol for the procurement of specimens and the collection of data from a patient population. In addition to identifying the experimental group of patients (i.e. those with a condition or trait of interest), it is important to ensure the inclusion of the proper control populations wherever possible. With respect to our mesothelioma protocols, the correct controls for early diagnostic strategies include age-matched individuals exposed to similar doses of asbestos in order to compare with equivalent individuals diagnosed with the established disease. The recruitment of these control subjects ideally begins in parallel with the recruitment of the experimental group. In the absence of the correct control group, the investigator must identify cohorts of individuals within existing cooperative group mechanisms, SPORES, or the Early Detection Research Network (EDRN). In order to circumvent this problem, the Karmanos Cancer Institute has established a National Center for Vermiculite and Asbestos Related Cancers through a cooperative clinical trial with the Center for Environmental Medicine in Southeast Michigan. This center, under informed consent, evaluates individuals who have been exposed to asbestos in the workplace or at home, and depending on history and pulmonary function data, these individuals undergo computerized tomographic scanning to establish a baseline radiographic evaluation of asbestos exposure and risk. These individuals also give written permission for periodic sample collection including blood and urine for studies of marker assessment for asbestos related malignancies.A major focus of our research is to identify the molecular causes of differential therapeutic response across patient populations (pharmacogenetics). It is generally appreciated that patients and their disease show significant variations in response to a given treatment . It is tWe have introduced standard procurement procedures for various physiological samples including tissue, urine and blood, all of which are now routinely collected under our IRB-approved protocols. These have been optimized for both ease and reliability of clinical collection as well as maintaining the integrity of the sample for subsequent histopathological and molecular analysis. DNA from peripheral blood mononuclear lymphocytes (PBML) obtained from whole blood can be screened by SNP analysis, to identify possible genetic markers of a clinical event -4. Once The precise flow of the clinical sample from the patient to the researcher will depend largely on the disease in question and pre-existing departmental procedures. For example, in a study of pancreatic cancer, urine is provided by the patient prior to the procedure, since we have found that intra-procedural catheterization can result in blood contamination that effects subsequent proteomic analysis. Blood is drawn from an IV access for collection of plasma and PBML, while surgically resected tissue is snap frozen on site (see below). However, a multiple myeloma research protocol requires the collection of bone marrow aspirates from both inpatient and outpatient clinics, which are placed into tissue culture media and rapidly transported on ice to the Flow Cytometry Molecular Diagnostic Laboratory for immunophenotyping and cytometric sorting of plasma cells. The sorted fraction is then archived in freezing media for subsequent DNA, RNA and/or protein analysis. Irrespective of procedure, it is important to maintain a log book that tracks the flow of a sample from the subject to freeze. Procurement time is particularly important for RNA and some protein analysis, since biomolecular degradation can be a significant factor.Prior to collection, our procurement team (lab personnel and research nurses) pre-label collection tubes and containers with anonymous identifier tags that are freezer safe. Several kits that include equipment for blood draws, urine collection and tissue procurement are preassembled and made available to the research staff to ensure rapid response for obtaining specimens. We have an on-call list of several members for the research team who can reach the pick-up point within 30 minutes of receiving a collection call. Through a coordinated multidisciplinary approach, consenting and scheduling is typically obtained well in advance in either the outpatient or inpatient setting.At the Karmanos Institute, we have found that the easiest way to insure proper specimen harvest from patients with solid tumors is for the surgeon to divide the specimen directly in the OR and supervise its collection and distribution to the laboratory for archiving. However at the Van Andel Research Institute, we require oversight from the Hospital Department of Pathology to release surgically resected samples within the OR to the research team. The critical issue is to ensure suitable material is harvested in the correct fashion to allow accurate histopathological diagnosis. In no way should the research effort impede this quality-of-care. As a safety margin, we hold all frozen tissue for a period of time until an official diagnosis has been reported. In addition, wherever possible we embed samples in OCT freezing media, and generate hematoxylin/eosin (H/E)-stained histopathology slides for each research sample. This not only provides high quality histopathology slides for possible diagnostic back-up, but also allows us to correlate our molecular findings with the histology of the same working sample. In addition to entering pertinent data regarding the official diagnostic pathology report, we have also developed standardized report templates within Dx2Tx that accompany each research sample, in which the pathological image is associated with the corresponding data addressing critical variables such as relative amount of each cell type, stage, morphology etc. As with all data entered into Dx2Tx, these metadata are directly amenable to subsequent statistical analysis (see below).At the time of resection, solid tissue samples are immediately frozen in either liquid nitrogen or, for sites with no supply, a dry ice-chilled isopentane bath. Samples are transported in aluminum foil, and stored at -80°C until further processing. It should also be pointed out that many of our protocols involve the collection of tissue biopsies in addition to surgical resections. Since it is physically difficult to split a biopsy , we typically acquire additional samples beyond what are required for accurate pathological diagnosis. If these additional samples are required by the research team, it is important to disclose the supplementary procedures to the patient in the informed consent. While obviously smaller in size compared to surgical resections, we procure these specimens as fresh tissue as described above. It is important to note, some of our molecular technologies allow for the use of as few as 500 cells for complete multiplex analysis. Since biopsy material will doubtless be the predominant source of tissue for future molecular-based diagnostics, it is important to develop protocols to address smaller sample size and possible mis-sampling issues. In addition, as a result of not restricting samples to excess surgical specimens, biopsies can rapidly increase sample accrual from larger cohorts of patients.Urine and blood samples are transported on wet ice to the research labs, while frozen tissue is transferred on dry ice. Upon arrival in the laboratory, urine is typically divided into suitably sized aliquots and frozen at -80°C. Blood is fractionated into plasma or serum , while PBML are isolated by centrifugation and Ficoll/Hipaque density gradient centrifugation and cryogenically frozen. Fresh frozen tissue specimens are blocked in OCT if possible and stored at -80°C until further use. All specimens are associated with the donating subject within Dx2Tx, and archived in a simple grid system within dedicated and alarmed -80°C freezers. The location and use of each aliquot is tracked within Dx2Tx to readily allow for subsequent retrieval.). We enter these variables into Dx2Tx in association with the corresponding experimental procedure such that they can be analyzed together with the derived molecular and clinical data. In this fashion, important QC and QA parameters can be interrelated with molecular and clinical data. Experiments that do not meet certain QC/QA criteria can be excluded from the analysis if the data suggest that the variable(s) is a major confounder. In this fashion, Dx2Tx serves as an electronic notebook, the entries into which can be routinely recorded and statistically analyzed. This feature will be particularly important as potential applications advance towards clinical utility, for example through ensuring compliance with the FDA and accrediting agencies such as JACHO, CMS and CAP, and while maintaining the necessary state, federal, and insurance applications required to provide clinical diagnostic services.The tissue collection protocols outlined above should be evaluated for compatibility for the molecular analysis to be performed. We typically perform SNP analysis on DNA isolated from the PBML fraction of whole blood, gene expression (mRNA) analysis on tissue, and proteomic analysis of blood plasma/serum, urine and tissue. The detailed discussion of each of these molecular protocols is beyond the scope of this review  and detection (such as mass spectrometry) elements. Coupled with extensive clinical content, this collectively presents significant challenges for data storage and retrieval, as well as statistical analysis. The most frequent frustration of new users of the "omics" technologies remains "What are the data telling me?"Before proceeding with specific examples, it is important to grasp the concepts of normalization and data massaging or "filtering" . To compOne problem in dealing with vast amounts of clinical and/or multiplexed molecular data from a relatively small sample population is that many of the observed correlations could (not necessarily do) occur by random chance. One method investigators use to minimize the probability of this so called "false discovery" is to reduce the complexity of the data through successive rounds of filtering. Depending on the application, we do use some filtering under certain circumstances. However, as with normalization, this is done during data analysis and not prior to database entry. In this fashion, we can filter and analyze the data "on the fly", allowing the user to evaluate the effect of filtering the raw data. For example, if we were looking for a specific set of genes that are found highly expressed in mesothelioma relative to other tumor types, we may eliminate genes in mesothelioma samples below a certain expression (intensity) threshold, and those above a certain level in other tumor types. This would maximize the likelihood of identifying sensitive and specific mesothelioma genes, and minimize the gene pool and hence probability of false discovery. We also use fold-change filters depending on the application. For example, we may only be interested in diagnostic biomarkers that display at least a 4-fold increase in mesothelioma tumor tissue relative to other tumor types and normal mesothelium. In addition, genes with certain characteristics  to identify relationships between variables  within large datasets ,14. Thes), we also map results to molecular pathways using detailed pathway mapping software now available. Taking this approach, the coordinated expression of genes/proteins can be seen to map to specific molecular networks, therefore providing important information as to which pathways maybe activated or in-activated in association with a clinical event. For example, when all the genes differentially expressed in recurrent mesotheliomas relative to non-recurring tumors at a defined statistical significance (p < 0.001) are mapped using the MetaCore™ software (MetaCore™ ), a clear signaling pathway associated with cell proliferation is identified that appears to be hyper-activated in aggressive mesothelioma tumors  within the complex clinical and molecular datasets that could be translated into clinical diagnostic applications, we typically begin with unsupervised clustering techniques such as hierarchical clustering as shown above in Figures Once a hypothesis has been generated, Dx2Tx identifies samples against which the hypothesis can be tested. Certain inclusion and exclusion eligibility criteria can be considered and used to filter the content of the database to identify subjects/samples/experiments with certain characteristics. Dx2Tx also allows samples to be selected based upon the extent of any attribute(s) Figure . For exaOnce the sample population is selected, we test the hypothesis across the series of selected samples in a 2-step process. A subset of the samples  are selected  from each subgroup (for example disease versus control) to develop a discrimination algorithm that identifies statistical correlates of the feature in question. It is worthwhile to note that Dx2Tx identifies clinical, experimental and molecular correlates of the selected feature(s), thereby integrating both clinical and molecular data into the potential diagnostic algorithm. The user can exclude any attribute from the input to the training algorithm. In a second cross-validation test, the trained algorithm is applied to the remainder of the samples , to determine if the test could have accurately predicted the nature of the remaining samples. The outcome of the test is plotted using a receiver operator characteristic (ROC) curve to determine the accuracy of the test. The ROC curve is a way to visualize and quantify the effectiveness of a procedure by graphing the true positive rate (Y axis) against the false positive rate (X-axis). The area under the curve (AUC) provides an approximation of the accuracy of the test. A procedure with no effectiveness (AUC = 50%) would show a random 1:1 line, indicating that for every true positive, the procedure also generated a false positive. Generally, an AUC of 90–100% is considered excellent, while an AUC of 80–90% is good. The ideal diagnostic test would of course identify all true positives before encountering a false positive (AUC = 100%).In this working example, the hypothesis generated from analysis of unsupervised clustering of gene expression data from mesothelioma tumors is that survival time of patients can be predicted based upon the underlying genomic signatures of the tumor. Thus, patients with the shortest and longest survival time following surgery were placed into two groups. Each group was then randomly divided into 2 additional groups, the training set and the test set. The discriminating clinical and molecular features are first identified using a standard t-statistic for numerical data and chi squared for binary (including text) data. This test statistic is then used in a weighted voting metric . Data ar). These so called "companion diagnostics", which could accompany therapeutic agents and assist in treatment decisions, are also being developed for specific agents that display varying degrees of efficacy and toxicity across sample populations. With the accurate capture of longitudinal clinical data including toxicity and response assessment, associations between clinical response and molecular features of either the patient and/or the disease tissue should be readily identifiable within complex retrospective datasets. For example, we have identified a genomic signature within plasma cells isolated from multiple myeloma patients that correlates with tumor response to the drug melphalan. This genomic signature is currently being applied to additional patient samples using cross-validation statistics as outlined above, to determine the accuracy of this possible companion diagnostic application. As with most single agent treatment regimens, drug resistance in the area of oncology represents a significant problem in the treatment of the disease. Therefore, pre-treatment tests could conceivably identify the patients who would benefit the most and least from treatment.In addition to the identification of potential diagnostic applications, we have a major focus on identifying new treatment targets, and/or improving therapeutic strategies involving existing treatments. Patient variation, with respect to treatment response (efficacy and toxicity), is a well documented phenomenon . ThroughOur research also includes the discovery of possible early surrogate markers of therapeutic index. This ideally requires the collection of clinical specimens and data both pre and post treatment. In conjunction with the treatment of cell lines in culture and molecular analysis of livers and kidneys from treated mice, early biomarkers of efficacy and toxicity have been identified in association with several treatment regimens. If these biomarkers of response can be validated in retrospective patients, they could be written into future clinical studies to provide an early indication of therapeutic effect. These biomarkers may ultimately be used as surrogate markers to determine discontinuation or modification of protocols to maximize therapeutic index in prospective trials.There are currently a number of drugs in development, in clinical trials or that have recently received FDA approval that target specific molecular aberrations -23. UnliThe FDA approval process of the EGFR inhibitor Iressa has received a great deal of attention . RecentlIn addition to pursuing existing drug targets, we are also attempting to identify novel targets that may warrant future drug discovery efforts. We typically attempt to validate only candidate genes/proteins that have or are predicted to have "drugable" characteristics. To determine whether a potential target is drugable, we are currently using some relatively simple criteria based upon the classes of drug targets that have been actively pursued by pharmaceutical/biotech companies to date. For example, we have annotated the publicly available drug targets described above using gene ontology and literature mining tools within Dx2Tx , and ideThe majority of the preceding discussion has focused on the identification of molecular correlates of disease, and identifying those that may represent diagnostic biomarkers and/or treatment targets for intervention. A large proportion of our translational research effort is dedicated to functional validation, where through various means, the expression and/or activity of the target are modified and the functional consequences addressed. While determining the function of a diagnostic biomarker is not necessary, the functional consequence of target gene/protein disruption is essential when establishing the true therapeutic value of a potential target. An ideal therapeutic target would be one that is causative of the disease, and inhibitors against which thus cause disease regression. The following discussion will focus on a high-throughput means by which we assess the functional significance of target gene/protein disruption in murine models of various human malignancies.There are a variety of approaches one can take to interfere with gene/protein expression and/or function with the purpose of demonstrating a definitive role in a biological process . These iWe have developed an avian retroviral vector that can deliver siRNA to cells expressing the viral receptor (TVA) [35]. TVA expression can be directed to specific cells in vitro through exogenous transfection/infection, or in vivo through transgenic technology . This syin vivo significance of experimental findings in a relevant preclinical setting. We particularly focus on the development of orthotopic xenograft models, in which human tumor cells are implanted into their corresponding site of origin within an immune compromised mouse. In this fashion, human tumors form in the tissue site of origin that more closely resemble the human counterpart with respect to biological behavior. These orthotopic models better recapitulate the various stages of tumor progression and more accurately reflect responsiveness to various therapeutic intervention strategies [37]. Coupled with in vitro and in vivo delivery of siRNA against multiple target genes, these preclinical models readily allow us to evaluate the requirement of proposed target genes in tumor progression. Through these studies, we are beginning to identify the optimal combination targeting strategies that may lead to the eventual treatment of aggressive cancers.Improved murine models that depict specific features of the human disease in question are essential to validate the In the vast majority of cases in which molecular data is being correlated with clinical events, a hypothesis-driven inquiry is tested against archived clinically documented specimens. The majority of these trials are asking either a classification question (i.e. what profile sets this tumor apart from other tumors), a prognostication pattern (what group of genomic/proteomic patterns will predict time to progression or time to death), early detection , or response to therapy (is there a de novo set of genomic/proteomic parameters which predict either response or resistance to a given therapy). For classification phenomena, the investigator must currently rely on pathologic differences to stratify tumors into clusters which, upon genomic/proteomic analysis, have consistent and congruent distinguishing features. This is probably the easiest of analyses and relies mainly on established architectural and morphologic differences between tumors instead of clinical behavior and endpoints. Nevertheless, as with all discovery test-sets, validation prospectively must be performed in order to test the accuracy of the classification algorithms. This requires prospective application of the algorithm to blinded samples, and comparing the predicted outcome with the actual observed pathology.For the early detection, prognostication, and prediction of therapeutic response, one must link potential diagnostic tests to newly developing trials to allow validation of the established hypothesis. Before undertaking such an effort, however, the investigator must realize that the data were typically derived from a set of patients within a particular treatment regimen. For example, the patients presented in this article were characterized as having specimen procurement prior to definitive cytoreductive surgery followed by adjuvant therapy, and therefore a prospective trial which conforms to this treatment regimen should ideally be designed to avoid confounding variables. For the validation of diagnostics that predict time to recurrence or survival following treatment, a phase II trial of surgery and/or postoperative adjuvant therapy could be designed in which samples are harvested at the time of surgery. The diagnostic test would then be performed on the clinical specimen to obtain a prediction of patient outcome, and the patients' clinical course followed to see if there is prospective validation of the test. These analyses, however, must be performed in patient groups not weighted towards either high or low risk patients, and indeed, some stratification of clinical parameters should be specified at the outset in order to make reasonable comparisons. This is especially true if the molecular data is to be validated as part of a Phase III randomized trial comparing two treatment regimens after surgery. As the content of Dx2Tx expands to include clinical trial data, analysis can be performed on only those patients that conform to specified clinical parameters. Therefore, with the complete and standardized collection of clinical data from a large population of patients receiving various treatment regimens both on and off of protocol, established hypotheses can be further tested on additional retrospective subjects; in a sense a virtual trial that begins to more closely resemble the carefully controlled prospective clinical trial.in vitro discovery of genomic/proteomic patterns defined in cell lines treated with the corresponding agent(s).The ideal clinical trials addressing the validity of molecular data to predict a clinical parameter are those involving patients with no prior diagnosis or treatments, who subsequently receive a single therapeutic regimen. Preferably, both pre- and post-treatment specimens are obtained from these patients. Such a pure trial could, for example, test whether a predefined genomic/proteomic pattern could predict a poor outcome despite favorable clinical parameters. Trials of patients receiving initial chemotherapy could also help to define the fidelity of the omics technologies for the prediction of chemosensitivity. These hypotheses may have been derived from a retrospective set of patients receiving the same chemotherapy or targeted therapy for which outcomes were known, or from the Early detection clinical trials using genomic and/or proteomic technologies will typically take the longest to validate. After the discovery of proteins as candidate markers from analyses of serum, urine, or other fluids in patients with early stage malignancies compared to the appropriate high risk group, cohorts of high risk individuals (e.g. asbestos exposed or tobacco smokers) must be identified who are willing to have collections of the appropriate specimens longitudinally at given intervals. Moreover, in initial trials of these markers, the investigator must define at what time to disclose the value of the measured biomarker. One possibility would be to allow the prospective clinical trial to reach a certain endpoint  and then reveal the results of the marker to see whether there is predictive value. Another way would be to define the value of the marker at follow-up intervals during the course of the trial and, in the face of radiographic or physical findings, initiate an invasive workup to "find" the predicted cancer. It is generally agreed that, although a two stage validation effort takes longer , such a model is currently preferred.With respect to novel treatment strategies predicted from retrospective analysis and/or through the preclinical studies defined above, the next logical step is to attempt to validate the retrospective data which initially pointed the investigators to these potential treatment options. If the new therapy has never been used in humans before, it is important to define the maximum tolerated dose of the treatment. This is defined as the dose at which, in a Phase I clinical trial, a defined fraction  of the patient cohort develop dose limiting toxicity. Once the dose or treatment strategy is found to be acceptable, a Phase II trial is designed to provide a measure of the activity of the agent(s) and to begin to define whether there is any benefit of the regimen in certain patient cohorts. The use of tools such as Dx2Tx could be invaluable in the future tracking and analysis of such trials, especially if samples are prospectively harvested at various times pre and post treatment. For example, in the Phase I/II trial, genomic/proteomic correlates of clinical toxicity could be readily identified by melding the clinical data with "omic" data from the patients who have undue toxicity. Molecular pathways that intuitively result in toxicity could be identified which could possibly be abrogated with another agent. With respect to determination of therapeutic efficacy, our present means of defining "clinical correlates of response" is essentially a "best guess" or subjective prediction of what clinical markers may indicate that the new agent is yielding a therapeutic effect. A more objective measure of downstream events resulting from an efficacious agent would result from analysis of the molecular data in tandem with clinical information for the responders compared to the non-responders. Dx2Tx would be able to then define the most important markers of response, both clinical and molecular, which could then be validated in a Phase III trial assessing agent efficacy.It is unknown when an integrated clinical/molecular evaluation of the suspected or afflicted cancer patient will be of use to the end user, the practicing physician. Certainly, many of the same arguments that are used for and against "genetic testing" in other diseases may be used for such a global approach to oncology. There is no doubt however, the ability to define clinical behavior in a more efficacious and predictive manner, other than the archaic prognostic indicators which we use today, will help clinicians initiate and/or alter treatment course earlier and guide clinicians toward informed discussions with their patients in order to make treatment and/or surveillance decisions. If indeed the fidelity of combinatorial molecular and clinical medicine proves to be satisfactory, we may then be able to spare patients unnecessary treatment interventions which are currently doomed to failure. Moreover, the medical oncologist will be able to choose the correct cytotoxic and/or targeted therapy based on a global clinical-molecular snap-shot, which should translate into more favorable health economic policies and patient outcomes.Throughout this review, we have highlighted the importance of designing research protocols involving human subjects that permit the collection of not only clinical specimens, but also extensive standardized clinical data in a longitudinal fashion. Through the merging of clinical and molecular data, non-biased patterns can be discovered that could translate into novel diagnostic and/or treatment opportunities. We have introduced a methodical approach for archiving and mining these seemingly disparate data sources that we believe can accelerate the translational research discovery pipeline. While there are clearly improvements to be made in the systematic collection of accurate and nationally standardized clinical data, we believe the integration of medical informatics and the molecular technologies can convert the once visionary concept of molecular-based medicine into a present reality.CW – A provisional patent has been filed on the Dx2Tx system described in this article.HP – This author declares that he has no competing interests.


A mixed-effects model repeated measures approach (MMRM) was specified as the primary analysis in the Phase III clinical trials of duloxetine for the treatment of major depressive disorder (MDD). Analysis of covariance using the last observation carried forward approach to impute missing values (LOCF_ANCOVA) was specified as a secondary analysis. Previous research has shown that MMRM and LOCF_ANCOVA yield identical endpoint results when no data are missing, while MMRM is more robust to biases from missing data and thereby provides superior control of Type I and Type II error compared with LOCF_ANCOVA. We compared results from MMRM and LOCF_ANCOVA analyses across eight clinical trials of duloxetine in order to investigate how the choice of primary analysis may influence interpretations of efficacy.a priori in the various protocols were included in the comparisons.Results were obtained from the eight acute-phase clinical trials that formed the basis of duloxetine's New Drug Application for the treatment of MDD. All 202 mean change analyses from the 20 rating scale total scores and subscales specified rd decimal place . For the HAMD17 total score, the primary outcome in all studies, MMRM yielded 9/12 (75%) significant contrasts, compared with 6/12 (50%) for LOCF_ANCOVA. The expected success rate was 80%.In 166/202 comparisons (82.2%), MMRM and LOCF_ANCOVA agreed with regard to the statistical significance of the differences between duloxetine and placebo. In 25/202 cases (12.4%), MMRM yielded a significant difference when LOCF_ANCOVA did not, while in 11/202 cases (5.4%), LOCF_ANCOVA produced a significant difference when MMRM did not. In 110/202 comparisons (54.4%) the p-value from MMRM was lower than that from LOCF_ANCOVA, while in 69/202 comparisons (34.2%), the p-value from LOCF_ANCOVA was lower than that from MMRM. In the remaining 23 comparisons (11.4%), the p-values from LOCF_ANCOVA and MMRM were equal when rounded to the 3Important differences exist between MMRM and LOCF_ANCOVA. Empirical research has clearly demonstrated the theoretical advantages of MMRM over LOCF_ANCOVA. However, interpretations regarding the efficacy of duloxetine in MDD were unaffected by the choice of analytical technique. Treatment effects are often evaluated by comparing change over time in outcome measures. However, valid analyses of longitudinal data can be problematic, particularly if some data are missing for reasons related to the outcome measure ,2. SinceA common method of analyzing clinical trial data is to use analysis of variance or analysis of covariance (ANOVA or ANCOVA) with missing data imputed by the last observation carried forward approach (LOCF_ANCOVA). The popularity of LOCF_ANCOVA may be due to its simplicity, and also the belief that violations of the restrictive assumptions inherent to LOCF_ANCOVA lead to a conservative analysis . ConsideOne such method, termed MMRM (Mixed Model Repeated Measures ), has beThe MMRM approach was specified as the primary analysis in the Phase III clinical trials of duloxetine for the treatment of major depressive disorder (MDD), while LOCF_ANCOVA was specified as a secondary analysis. In the present investigation, we provide a comprehensive summary of results from MMRM and LOCF_ANCOVA in the eight acute-phase clinical trials that formed the basis for duloxetine's New Drug Application (NDA) for MDD. The primary objective of this investigation was to determine whether differences in results between MMRM and LOCF_ANCOVA influenced conclusions regarding the efficacy of duloxetine.The data source for this investigation was the eight acute-phase clinical trials in which duloxetine was compared with placebo in the treatment of MDD. Relevant details of these studies are highlighted in Table a priori in the various protocols to be analyzed for mean change from baseline to endpoint, and were collected at more than one postbaseline time point  With regard to statistical significance of the difference between duloxetine and placebo, the proportion of outcomes showing agreement between MMRM and LOCF_ANCOVA was compared with the proportion of outcomes for which MMRM and LOCF_ANCOVA yielded disparate results; 2) The proportion of outcomes for which MMRM yielded the lowest p-value was compared with the corresponding proportion for LOCF_ANCOVA; 3) The number of outcomes for which "substantial evidence of efficacy" was demonstrated. In regulatory settings, the criterion for substantial evidence of efficacy is frequently the demonstration of a statistically significant advantage over placebo in two or more studies. This criterion was utilized here to define substantial evidence of efficacy for a particular outcome.The frequency of lower p-values provides a "fine-tuned" measure of sensitivity of the two analytic methods. However, in certain cases such an assessment may actually be misleading. For example, to distinguish between p= .800 and p = .810, or between p =. 002 and p =. 003, implies that the methods yielded different results when in fact the similarities far outweigh the differences. Hence, it is equally appropriate to simply categorize based upon the presence or absence of a significant difference. Furthermore, given the large number of outcomes assessed across the eight studies, it would not be surprising to see the two methods disagree with regard to statistical significance on at least a small number of outcomes. Therefore, perhaps the most clinically meaningful summary measure is the number of outcomes for which substantial evidence of efficacy was demonstrated.17) total score, the HAMD17 Maier subscale, and the Visual Analog Scale (VAS) for overall pain severity. The HAMD17 total score was an obvious choice as it was the primary outcome in all studies. The other outcomes were selected since they are frequently focal points in manuscripts and presentations regarding duloxetine's efficacy. Finally, we provide case studies to help explain how and why results from MMRM and LOCF_ANCOVA may differ.Three outcomes were selected for more detailed presentation of results: the 17-item Hamilton Rating Scale for Depression , MMRM and LOCF_ANCOVA agreed with regard to the statistical significance of the difference between duloxetine and placebo. In 25 cases (12.4%) MMRM yielded a significant difference whereas LOCF_ANCOVA did not, while in 11 cases (5.4%) LOCF_ANCOVA yielded a significant difference when MMRM did not.17 total scores, leading to a three-fold greater standard error.Both methods tended to yield significance more frequently in depression rating scales and subscales than in outcomes related to somatic and painful physical symptoms. The studies were generally underpowered for these secondary somatic and pain outcomes owing to the greater variance in changes score for these outcomes. For example, the variance in VAS overall pain severity was approximately nine-fold greater than the variance in HAMDrd decimal place .In 110 of the 202 outcomes (54.4%) the p-value from MMRM was lower than that from LOCF_ANCOVA, while in 69 cases (34.2%) the p-value from LOCF_ANCOVA was lower than that from MMRM. In the remaining 23 cases (11.4%) the p-values from LOCF_ANCOVA and MMRM were equal when rounded to the 317 total score, HAMD17 Maier subscale, and VAS overall pain) are presented in Table 17 total score, the advantage of duloxetine over placebo in mean change from baseline to endpoint from MMRM analyses was greater than the corresponding advantage from LOCF_ANCOVA in 9/12 comparisons  are used to further illustrate MMRM and LOCF_ANCOVA  results. Results from these studies were originally reported by Detke et al [17 total score tended to increase over time whereas duloxetine's advantage in VAS overall pain was greatest at intermediate visits  from MMRM in Studies 1 and 2 were 4.86 (p < .001) and 2.17 (p = .024), respectively. The corresponding advantages from LOCF_ANCOVA were 3.80 (p < .001) and 1.73 (p = .048). Although the differences were significant for both methods in both studies, MMRM yielded treatment contrasts that were approximately 25% greater than LOCF_ANCOVA.In the case of the HAMDFor VAS overall pain, the advantage of duloxetine over placebo at endpoint from MMRM in Studies 1 and 2 were 5.88 (p = .055) and 4.40 (p = .135), respectively. The corresponding advantages from LOCF_ANCOVA were 6.91 (p = .019) and 5.17 (p = .037). In both studies, the endpoint differences were significant from LOCF_ANCOVA, but not from MMRM. The LOCF_ANCOVA treatment contrasts were approximately 15% greater than those from MMRM.17 total score and VAS overall pain.Standard errors from LOCF_ANCOVA were approximately 5% smaller than the Week 9 standard errors from MMRM for both the HAMDIn many areas of clinical research, the impact of missing data can be profound ,12-14. TConsiderable advances in statistical methodology, and in our ability to implement these methods, have been made in recent years. Methods such as MMRM, which require less restrictive assumptions regarding missing data, may now be easily implemented with standard software ,5,18,19.No universally superior approach to analysis of longitudinal data exists. However, a series of studies -9 demonsetc. to blind analysts to the treatment names, Molenberghs et al [Similar results regarding control of Type I and Type II error for LOCF_ANCOVA and mixed-effects model analyses have been obtained independently ,20-22. Fhs et al  concludeThe theoretical differences between MMRM and LOCF_ANCOVA have been summarized ,18, estaThe VAS pain results highlight a limitation of endpoint analyses of any type, namely that they provide only a snapshot view of the response profile. From LOCF_ANCOVA analysis, one can only conclude that drug was superior to placebo at endpoint. However, MMRM analysis reveals that drug had a significant effect early in the trials, but that advantage was somewhat transitory as the placebo group tended to "catch up" over time. In order to understand the response profile of a drug, the entire longitudinal profile should be considered . From MMIn the duloxetine database, results from MMRM and LOCF_ANCOVA were in general agreement regarding substantial evidence of efficacy and frequency of significant differences. However, MMRM tended to be more sensitive to drug-placebo differences for outcomes related to overall depressive symptoms and core emotional symptoms of depression, with mean advantages over placebo that were 10% to 20% greater than LOCF_ANCOVA. However, MMRM did not universally increase duloxetine's advantage over placebo in comparison to results from LOCF_ANCOVA. For example, in somatic and painful physical symptom outcomes, results from LOCF_ANCOVA showed mean advantages over placebo that were approximately 40% greater than that from MMRM.Therefore, while the overall conclusions regarding the efficacy of duloxetine were unaffected by the choice of analytic method, this should not mask the important differences between MMRM and LOCF_ANCOVA. The advantages of MMRM and similar methods over LOCF_ANCOVA have been conclusively demonstrated in many studies and are evident in the duloxetine data.17 total score), while LOCF_ANCOVA produced a 50% success rate, in comparison to the expected rate of 80%. While many factors may reduce the success rate in Phase III clinical trials, the use of a statistical method with known inflation of Type II error  is an obvious suspect.Khan et al  compiled17 total score) when LOCF did not was in the Phase II study (Study 3).An unduly high rate of false negative results could be especially problematic in early phases of drug development where only one or two chances exist to make the correct decision regarding the efficacy of a drug. It is noteworthy that one of the instances when MMMRM yielded a significant difference on the primary outcome  is an obvious suspect. Thus, the unexpectedly low success rate in Phase III is consistent with the conclusion that LOCF_ANCOVA inflates Type I error .Hence, results from the duloxetine NDA are consistent with research suggesting a move away from LOCF_ANCOVA and other simple analytic techniques to methods such as MMRM that are more robust to the biases from missing data.Important differences exist between MMRM and LOCF_ANCOVA. Research has clearly demonstrated the advantages of MMRM over LOCF_ANCOVA. However, interpretations regarding the efficacy of duloxetine in MDD were unaffected by the choice of analytical technique.Drs. Mallinckrodt, Raskin, Wohlreich, Watkin, and Detke are employees of Eli Lilly and Company.CHM designed the statistical analyses, and participated in interpretation of data and drafting of the manuscript. JR, MMW, JGW, and MJD participated in interpretation of data and drafting of the manuscript. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:


Traditional two dimensional journal publishing may no longer be enough, and may be hindering, the interpretation of the complex datasets and analyses generated by microarray experiments in medicine Statistical Applications in Genetics and Molecular Biology (3: article 37), discuss a paradigm for the presentation of complex data—in this case, from microarray analyses. The title of the article, “A Compendium to Ensure Reproducibility in High-Dimensional Classification Tasks,” may not lend itself easily to a clinical audience, but the underlying message to clinicians could not be more important: that, currently, studies involving large datasets, especially ones that have a clinical outcome, are so poorly reported (or possibly so poorly done) that many are not reproducible. This problem was also the topic of a recent meeting in Heidelberg, “Best Practice in Microarray Studies” (http://www.biometrie.uni-heidelberg.de/workshops/bestpractice/index.htm).“Publishing results in traditional paper based way in a journal hides too much information.” This is the verdict of Markus Ruschhaupt and colleagues who, in a paper in As microarrays have become mainstream research tools in biology and medicine, the large datasets and complex analyses from these studies have presented challenges: for authors in analyzing the data, for reviewers and editors in deciding on the suitability of papers for publication, for journals in determining how much data needs to be presented within the paper itself, for other researchers in reproducing the data, and, finally, for readers in deciding how to assess the data presented. The results from several high-profile papers have already proved difficult to reproduce, even by those with sufficient time and computing expertise.Where do such analyses leave the new science of molecular pathology? Ruschhaupt and colleagues comment that “the literature on the induction of prognostic profiles from microarray studies is a methodological wasteland.” Much the same could be said of other applications of molecular biology to clinical samples. A systematic review of molecular and biological tumor markers in neuroblastoma (Clin Cancer Res 10: 4–12) found that its conclusions were limited by “small sample sizes, poor statistical reporting, large heterogeneity across studies…and publication bias.” John Ioannidis and colleagues (Lancet 362: 1439–1444) did a similar analysis of 30 microarray studies with major clinical outcomes in cancer. They showed that the studies were small—median sample size was 25 patients, and validation was incomplete in most studies. They recommended that molecular prognostic studies be classified as phase 1 (early exploratory probing associations), phase 2 , or phase 3 (large confirmatory studies with pre-stated hypotheses and precise quantification of the magnitude of the effect), and that only studies that had undergone phase 3 testing should be considered robust enough for use in clinical practice. Most current studies should be considered as phase 1 or, at best, phase 2.So, despite considerable hype, the published studies are far from the level of evidence that would be accepted for virtually any other medical test. In a review in 2003 (Hematology [Am Soc Hematol Educ Program] 2003: 279–293), Rita Braziel and colleagues concluded, “rapid identification and neutralization of spurious results is essential to prevent them from becoming accepted facts.”But these problems are not new in medical research. In 1994 (BMJ 308: 283–284), Doug Altman, who was instrumental in developing the CONSORT guidelines for reporting of clinical trials, said that “huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation,” and “that quality control needs to be built in from the start rather than the failures being discarded.”PLoS Medicine is keen to work with authors towards making such reporting possible. But although the time might have gone when the two-dimensional journal article could suffice for complex papers, clinicians should nonetheless apply the same critical assessment that they would for any other clinical tool. If a result is too good to be true, it probably is.So how can we ensure that the wealth of data pouring out of microarray and other molecular diagnostic studies is turned into meaningful knowledge? The Microarray Gene Expression Data Society has proposed a set of guidelines (MIAME) for the reporting of microarray data, and that all such data should be deposited in public databases. But as Ruschhaupt and others have shown, disclosure of results and data is not enough, since there is little consensus on the appropriate statistical analyses and many are developed on a case by case basis, which may not be reproducible, even by the authors. Some researchers advocate the use of standard statistical packages, which allows the reader to repeat an entire analysis quickly and, hence, assess the robustness of the results. Some authors have produced a transcript of their statistical analyses as a supplement to their articles . At the very least authors should have a protocol with a prespecified plan for patient selection and statistical analysis—accepted practice for clinical trials, but not yet for other medical research. An ultimate aim for reporting would be the type of compendium discussed by Ruschhaupt and colleagues—“an interactive document that bundles primary data, statistical processing methods, figures, and derived data together with the textual documentation and conclusions.” One such compendium is illustrated in a paper by Robert Gentleman (Stat Appl Genet Mol Biol 4: article 2).


We describe a system of web applications designed to streamline the interdisciplinary collaboration in outcomes research.The outcomes research process can be described as a set of three interrelated phases: design and selection of data sources, analysis, and output. Each of these phases has inherent challenges that can be addressed by a group of five web applications developed by our group. QuestForm allows for the formulation of relevant and well-structured outcomes research questions; Research Manager facilitates the project management and electronic file exchange among researchers; Analysis Charts facilitate the communication of complex statistical techniques to clinicians with varying previous levels of statistical knowledge; Literature Matrices improve the efficiency of literature reviews. An outcomes research question is used to illustrate the use of the system.The system presents an alternative to streamline the interdisciplinary collaboration of clinicians, statisticians, programmers, and graduate students. In the last decade, the number of relevant data sources available for outcomes research has grown exponentially. In contrast, the number of individual researchers with clinical and statistical expertise required to explore these data sets increase at a much slower pace. As a result, an immense quantity of valuable clinical data are left untouched, never becoming clinical publications that could potentially improve health care.The disproportion between data volume and number of qualified researchers can be explained by the growing complexity involved in outcomes research projects using secondary data analyses. Researchers have to formulate of a clinically relevant and methodologically sound research question, find appropriate data sources, perform statistical analyses, and generate a final manuscript that will be submitted for peer-review. Frequently, individual researchers have the training and time to perform a few of these steps, but the integration of all tasks calls for an interdisciplinary systems approach ,2. This This article describes a suite of web applications developed to facilitate the process of converting outcome databases into clinical manuscripts, to streamline the interdisciplinary collaboration of researchers, and to connect all different steps of the outcomes research process. To illustrate its use, we will describe how a research project has been conducted using this system from its early phase of research question formulation to the completion of the final manuscript.. The following sections will describe each of the Web applications and their application in the answer of a real outcomes research question.The system of Web applications is composed by five different tools: QuestForm, Research Manager, Analysis Charts, and Literature Matrices. These tools were designed to assist researchers in each of the phases encountered in an outcomes research project involving secondary data analysis Figure . All too for an updated list). All databases have been de-identified and do not contain protected health information as specified by the Health Insurance Portability and Accountability Act . The application The application is built using Extensible HyperText Markup Language 1.0 (XHTML) [QuestForm, an acronym for "Question Formulation", is an application designed to assist researchers in the location of clinical databases and formulation of outcome research questions Figure . Clinica (XHTML) .QuestForm starts by presenting researchers with three main strategies to find research databases: use of pre-determined key words that describe the database as a unit Figure , user-deDr. Guller initiated the project searching for an existing database that would allow him to compare surgical outcomes between laparoscopic and open appendectomy procedures in the treatment of acute appendicitis. The outcomes were pre-specified as mortality and infection, although other existing outcomes would also be of interest. Although there are multiple single-institution studies attempting to answer this question ,7, few sAs a first step, Dr. Guller searched across previously formulated Question Diagrams to evaluate whether other studies could have used a similar research design. Since none was found in QuestForm, Dr. Guller searched across more than forty different databases for an existing database that would have the variables to answer his research question. After navigating through multiple data dictionaries, Dr. Guller found that the Nationwide Inpatient Sample (NIS) Release 6, 1997  presenteOnce the database was located, Dr. Guller selected the outcomes of interest , main predictors (laparoscopic versus open procedures), and confounders , inserting each of them into the research question fields in QuestForm. Using built-in search engines for ICD9 codes, Dr. Guller created the definition for each of the above-mentioned variables and defined the inclusion and exclusion criteria. The final research question was then saved as a Question Diagram and immediately submitted to Dr. Pietrobon for feasibility evaluation. Dr. Pietrobon judged that the project was feasible and could be completed using the database indicated by Dr. Guller. At this point, the project was initiated and a detailed project management plan was established using Research Manager.Research Manager is a Web application developed by our group designed to facilitate the project management of clinical research projects. Similar to QuestForm, Research Manager is licensed under the GNU Public License , which aResearch Manager provides multiple features to facilitate project management of clinical research projects. All projects are displayed by category  with a brief description. The internal content of all projects is password protected. All internal tasks within a project are assigned to individual researchers. Project administrators initially assign deadlines that can be modified by task leaders within three days. All participating members of the project receive weekly reports containing details about the activity and the latest electronic file within each task Figure . These fResearch Manager helps identifying such problems and enables their early elimination, thus avoiding delays in the project completion. Finally, since weekly reports are generated to all participating members, Research Manager also provides peer-incentive for project members to complete their tasks in a timely manner.Once the Question Diagram was evaluated by the clinical epidemiologist, this project was transferred to Research Manager. Contact information for each of the researchers involved and deadlines for completion of the main phases of the research process were set, including data extraction and cleaning, data analysis, literature review, and manuscript writing. Weekly reports were generated to update investigators on all tasks of the project, including different versions of the statistical analysis, modifications in the research question, and manuscript sections.With an established project plan, the statistician in charge selected the best methods for analysis. Since the database is a random sample of the United States and requires special survey analysis methods, it was necessary that all involved researchers understood the statistical approach by using Analysis Charts.Analysis Chart is a tool designed to enhance the understanding of statistical methods to a format that is understandable by clinical researchers with different previous levels of statistical knowledge. As such, it is important in the design as well as the analysis phases of a project Figure . The applayers of information. Layer 1 summarizes the general goal of the method. Layer 2 presents previous clinical applications so that the researcher can visualize situations in which the method may be realistically applied. Layer 3 describes the data requirements for the application of the statistical technique. Layer 4 describes the basic statistical underpinnings of the method, initially breaking down equations and then reassembling them. Finally, layer 5 presents a list of available software packages for the implementation of the technique as well as cases studies where all previous layers are applied to real data sets. Each layer ends with a section containing selected references that explain the topic in more detail.Analysis charts are composed by cascading links that display information about quantitative methods in progressive levels of complexity called "layers of information". Each layer explains the statistical method with an increasing level of complexity. In this manner, clinicians interested in simply understanding the method to evaluate whether it can be applied to a research project can simply read the first three layers. In contrast, researchers interested in a direct application of the method to available research data can follow all layers and their respective references. Most commonly, complex techniques are presented using five While deciding on the most appropriate analysis strategy for the Question Diagram, Drs. Pietrobon and Guller consulted the Analysis Chart searching for the most appropriate statistical methods of analysis. Given the nature of the research question and that the NIS database has a sample design, Drs. Guller and Pietrobon opted for an approach involving multiple and logistic regression models while adjusting for sampling weights, strata, and clusters. With a defined analysis protocol, the research question was then transferred to a statistical programmer trained in the translation of Question Diagrams into statistical code. This process was closely evaluated by Drs. Pietrobon and Guller, who scrutinized the statistical code and results from a clinical and statistical perspectives. Once the results were deemed to be accurate, Dr. Pietrobon started the literature review using a Literature Matrix.® or Open Office Calc [Literature matrices consist of a comprehensive but not necessarily exhaustive review of the literature focused on a narrow clinical topic Figure . Each arice Calc .The advantage of the Literature Matrix as implemented in the Web suite is its availability over the web to the whole outcomes research community. This allows Literature Matrices to be constantly updated, with authors receiving their due credit in a list of contributors. Literature Matrices also enable researchers to obtain a complete summary of the literature without going through the cumbersome process of copying and reading a manuscript for the first time. In contrast, researchers' time can be spent more efficiently in reviewing what has already been compiled and attempting to expand the Literature Matrix with other relevant bibliographic references.In order to evaluate the literature, a graduate student performed a thorough literature search. All relevant articles were copied, read, and the data extracted according to the established categories. Once the Literature Matrix had been completed, Drs. Guller compared the current project results with results published in the literature. The structured information in Literature Matrices also allowed Dr. Guller to compare the strengths and weaknesses of the current project in relation to previous publications. Once this phase was completed, Dr. Guller proceeded to the final writing of the manuscript using Output Templates.At the end of the project, Dr. Guller combined the Question Diagram, Analysis Chart, and Literature Matrices to write the final manuscript. While Analysis Charts provided the information concerning the statistical techniques used in this study, Literature Matrices provided the basis for comparison of the study results against previous publications. Although not included in the final manuscript, Dr. Guller also had access to multiple analysis files through Research Manager to orient him in each of the steps taken during the research question formulation, data analysis, and literature review.The use of web applications by individual clinical researchers can be summarized in Figure Since the concept of streamlining the interdisciplinary collaboration preceded the existence of the current suite of web applications, different versions of the central idea have been gradually applied since the second half of 2002. Although our usability, qualitative, and economic studies to evaluate this application are still ongoing, we have noticed a significant improvement in the number and quality of our publications as evidenced by the increasing acceptance rates of our manuscripts for publication in peer-reviewed journals. The Web applications are currently used in research projects involving Duke University and other universities in the United States and abroad, where they have been shown to facilitate inter-institutional collaborations.Improving the efficiency of an interdisciplinary approach to secondary data analyses has multiple potential benefits. These include an increase in the overall clinical significance of the final publication, decrease in the number of failed projects, decrease in the time for completion of individual projects, improvement in the education of future outcomes researchers, decrease in the cost-benefit ratio for individual outcomes research projects, and, perhaps most importantly, an automation of repetitive tasks. This last factor is crucial since eliminating repetitive tasks will allow researchers to concentrate on the design of innovative projects .Surprisingly, few systems and applications have been described to solve the problem of complex interdisciplinary collaboration between clinicians and statisticians. Isolated approaches have usually focused on specific portions of the outcomes research process, without attempting to integrate them into a cohesive system. For example, Marshall  has propAlthough this newly developed system of applications provides a significant improvement in the way secondary data analyses are conducted, it still has limitations. First, because of the lack of a formal evaluation of the effectiveness of this system, we are unable to quantify its real time and cost saving benefits. Second, the system is currently restricted to secondary analyses and does not allow for the planning of prospective data collection. Although one of the main advantages of formulating a research question based on existing data sets is the bounded nature of the process, future applications should attempt to create rules and algorithms that may guide prospective data collection.In summary, we have experienced that this system has significant advantages over the traditional manner of conducting outcomes research based on secondary data analyses. This tool may have important applications, not only resulting in an improvement in the overall efficiency of the outcomes research process, but also affecting the way new outcomes researchers are trained and introduced to a research environment.The Web application is available at XHTML Extensible HyperText Markup Language 1.0JAVA: by Sun MicrosystemsICD9-CM: International Classification of Diseases, Ninth Revision – Clinical ModificationNIS: Nationwide Inpatient SampleGNU: GNU's Not linux General Public LicenseWebMIRS: Web-based Medical Information Retrieval SystemNHANES: National Health and Examination SurveyUR: Uniform ResourceThe author(s) declare that they have no competing interestsRicardo Pietrobon – design, manuscript draftingUlrich Guller – design, manuscript revision for important intellectual contentHenrique Martins – design, manuscript revision for important intellectual content, software programmingAndreia P Menezes – design, manuscript revision for important intellectual contentLaurence D. Higgins – design, manuscript revision for important intellectual contentDanny O. Jacobs – design, manuscript revision for important intellectual contentThe pre-publication history for this paper can be accessed here:


There is confusion in the medical literature as to whether statistics should be reported in survey studies that query an entire population, as is often done in educational studies. Our objective was to determine how often statistical tests have been reported in such articles in two prominent journals that publish these types of studies.Academic Medicine and the Journal of General Internal Medicine in which an entire population was studied. We tallied whether inferential statistics were used and whether p-values were reported.For this observational study, we used electronic searching to identify all survey studies published in Academic Medicine and 22 in the Journal of General Internal Medicine. Overall, 38 (45%) of the articles reported or stated that they calculated statistics: 35% in Academic Medicine and 73% in the Journal of General Internal Medicine.Eighty-four articles were found: 62 in Educational enumeration surveys frequently report statistical tests. Until a better case can be made for doing so, a simple rule can be proffered to researchers. When studying an entire population  for factual information, do not perform statistical tests. Reporting percentages is sufficient and proper. The inclusion of statistical tests has been inculcated into investigators as a necessary step in the reporting of medical research. While descriptive statistics may always be used (unless the data are too sparse), the employment of inferential statistics – statistics calculated from a study sample in order to make generalizations about a larger study target population – is not always appropriate.One situation in which statistical testing must be used with caution is when tests are calculated from what may be a non-random sample . In the Another situation in which inferential statistics should not be used is when the study targets the entire population . This tyAcademic Medicine N = 1096) and a general medical journal  that frequently publish educational studies – using the search term "survey." From this search, we identified all studies in which surveys were sent to an entire population of responders and determined whether inferential statistics that generate p-values, such as t-, Wilcoxon, Fisher's exact, and chi-square tests were used.For this observational study, we used PubMed to identify all articles (from 1987 to 2006) from two high impact factor journals – an educational journal  of the articles reported or stated that they calculated statistics. Of these, 31 (86%) reported p-values, 19 (50%) reported that they used chi-square tests, 12 (32%) t-tests, 6 (16%) some type of multiple regression analysis that included p-values or confidence intervals, and 5 (13%) analysis of variance; 2 studies each calculated other statistics including Fisher's exact tests, Wilcoxon tests, Mann-Whitney tests, and confidence intervals; other statistics were used for 8 other studies.Academic Medicine articles reported statistics in 22 of 40 articles (35%) while Journal of General Internal Medicine articles reported statistics in 16 out of 22 articles (73%). Statistics were more prevalent in more recent articles in Academic Medicine. For example, only 4 out of 30 (13%) before 1996 reported statistics compared to 18 out of 32 (56%) articles published afterwards.This study demonstrates that statistics are frequently overused in articles from two prominent and highly-cited journals that report educational studies. In fact, since 1997 the majority of articles reporting enumeration studies have inappropriately included statistics.The authors of one article that did not include statistics wrote directly to the issue in their methods section: "All analyses are based on the entire population of interest. Therefore, tests of statistical significance are not provided" . In anotThe findings of our study are worrisome for a number of reasons. The two journals that were studied are among the most prominent in their field and are highly cited. Authors may use them as role models in the reporting of educational studies. In addition, the concept of sampling is central to the whole of inferential statistics and is usually discussed in the early chapters of statistical textbooks . If reseA reviewer of this article suggested that there may still be a role for statistics in finite populations by appealing to probability distributions that generated the scores in the population, and that statistical tests are appropriate to compare not the actual numbers in the population but the probability distributions that are imperfectly indicated by the values in the populations. One situation in which this might be the case is when a survey questions an entire population  about values, preferences or impressions such as is often done with Likert-type questions. In this case, the referent distribution might be envisioned to be the impressions of all persons who might hold the office of the person who is responding to the questionnaire. This consideration would not be germane for factual information often queried about in surveys. Dusoir has suggested that "statistics is a collection of warring factions, with deep disagreements over fundamentals" and differences in reporting statistics from finite probabilities may be one of these fundamental issues . On the In addition to confusion about fundamental issues in statistics, the increasing prevalence of statistics in these studies over time suggests that the inappropriate use of statistical packages may be partly to blame. Many of the studies included statements to the effect that data were entered in statistical packages, when for all of these studies a spreadsheet program would have been more than adequate. While statistical packages can generate tests quite readily, the proper interpretation of their output is the responsibility of the investigator. Anthony has suggested that the "use of such (packages) does, unfortunately, also allow you to perform meaningless statistics and incorrect statistical tests, and give misleading or wrong interpretations" .This study did not sample all articles in the medical literature that have reported enumeration studies. However, it reports on all such studies that have been published in two leading journals that report medical educational studies. We suspect that this problem is also rampant in other journals that report this type of study. The proband case that led to this study was published in another leading journal .The improper use of statistics in medical research has become a matter of much concern . In an aThe author declares that he has no competing interests.ND is the sole author.The pre-publication history for this paper can be accessed here:


Resistance to cytotoxic agents may be encountered during the treatment of acute myeloblastic leukaemia (AML). P-glycoprotein encoded by the MDR-1 gene has been implicated as a potential drug resistance mechanism in leukaemic cells. In recent years, many data have been accrued concerning the expression of P-glycoprotein in leukaemia, and several studies have been published which have related MDR status to outcome in AML. Conclusions as to the effect of P-glycoprotein expression on prognosis in AML have varied widely. The studies are not immediately comparable, since they differ in methodology, treatment regimens, demographic profile and, perhaps most importantly, criteria for positivity of MDR status. The technique of statistical overview  can be used to pool observational studies. Application of this statistical method to existing studies suggests an estimated relative risk of 0.68 for P-glycoprotein expression with respect to complete remission in AML. Further large studies are required to determine fully the role of P-glycoprotein in AML.