New Guidelines for Data Reporting and Statistical Analysis: Helping Authors With Transparency and Rigor in Research

Field: Medicine (Health Sciences)

The increasing concern about the lack of replicability of published research in the biomedical field1-3 has produced some initial consternation in the research community and in the public at large, but it has also fueled robust discussions among journal editors, funding agencies, academic institutions, and researchers on how to address the problem and improve the quality of scientific publications. The editorial leadership of the Journal of Bone and Mineral Research, in concert with our sister journal, JBMR Plus, and the ASBMR Publications Committee, has developed new guidelines for reporting and analyzing research data with the purpose of improving rigor and transparency of published material, thus minimizing reproducibility problems. The new guidelines have been developed by two task forces appointed by the Publications Committee, with input from the Editors. Members of the task forces were chosen among the JBMR/JBMR Plus Editorial Board membership, and their final recommendations were unanimously approved by the Publications Committee. The new guidelines are now incorporated in the instructions for authors for JBMR and JBMR Plus; reviewers and Editors will be asked to verify that submitted manuscripts comply to the new instructions. This editorial introduces the changes made, provides rationale for specific choices, and offers some advice, especially about statistical analysis. We focus on five key areas: 1) study design; 2) reporting of genetic studies; 3) presentation and analysis of preclinical studies; 4) fundamentals of statistical analysis; and 5) interpretation of p values. As with any changes, there will be some initial hurdles to overcome and details to settle down, but we are certain that we will be able to achieve the goal of having all published content compliant to the new guidelines by early 2020. Some may find such detailed instructions too prescriptive and cumbersome. However, ensuring transparency, integrity, and rigor in published research is paramount to the continued success and stature of our journals. We are confident that our readers and the musculoskeletal field in general will be appreciative of the effort, and our authors will quickly learn how to navigate the process. The thrust of a scientific study rests on its design, methodology, and execution. Clarity and rigor in the experimental approach represent key factors in acceptance decisions of a manuscript, aside from its intrinsic scientific merit. For a few years already, JBMR has required authors to provide details of study design and methods that are consistent with the CONSORT,4 STROBE,5 PRISMA,6 and ARRIVE7 guidelines for clinical trials, observational studies, systematic reviews and meta-analyses, and animal studies, respectively. In the revised guidelines, authors are now requested to include a detailed description of the study design and in particular power analysis. An adequate sample size is a key aspect of any scientific experiment, in both clinical studies and basic science experiments. A study with inadequate sample size has low sensitivity (eg, power) to determine a real effect or robustly claim the absence of an effect. Even if the results are positive, the effect size observed in underpowered studies is likely to be exaggerated (due to “winners' curse”), thus increasing the likelihood that the results may not be reproducible.8, 9 A clear description of the study design, sample size estimation, and assumptions underlying the calculation (eg, primary outcome, hypothesized effect size, type I and type II error) will greatly help reviewers and readers evaluate the strength of the results. The recently updated publication guidelines on “Genetic and Functional -Omics Studies in Cells, Animals, and Humans” were developed with the aims of enhancing transparency, clarifying data sharing policies, and increasing author compliance to reporting standards. One challenge for these -omic studies is that they commonly employ hypothesis-free approaches, which do not rely on existing knowledge and are carried out without prespecification of the factors to be tested. The guidelines are provided for studies investigating genetic variation, but they apply to other forms of -omic studies focusing on the transcriptome, epigenome, metabolome, and/or proteome. The crucial elements of the recommendations include: 1) a clear description of the approach (ideally part of a preconceived analysis plan); 2) adequate sample size and power considerations; 3) control for population stratification and design biases; and 4) validation (strengthening) of the findings through independent replication and/or through triangulation.10 Including functional data that support the biological and clinical significance of the reported findings is also highly desirable. Authors are now required to make the original data and summary results publicly available in data repositories, to improve transparency for readers and enable new research to stem from published results. Issues with transparency about study design and appropriate use of statistical methods are particularly acute with preclinical research. Although fundamental research methodologies do not typically require sophisticated statistical analysis, studies with animals in most cases are akin to clinical trials, particularly those where multiple groups of animals or different genotypes or backgrounds undergoing different treatments are included. As such, they should be approached in a similar way as a clinical study, and similar guidelines for study design, power analysis, sample size, and statistical approach should apply.11 For statistical analysis, such studies (typically drawn from quantitative data) may require the application of appropriate general linear models (eg, analysis of variance, analysis of covariance, linear regression) and if necessary, post hoc multiple t tests can be used, where the threshold for significance is adjusted for multiple comparisons (see below for further details). For repeated measures designs, linear mixed-effects models are preferable over repeated measures analysis of variance, although the latter is acceptable. The use of nonparametric tests is required when assumptions of normality, linearity, and constancy of variance do not hold, a scenario occurring frequently with small sample size data sets. A notable innovation in the new guidelines is the requirement for showing all data points for each experimental group (for those with n < 20) as box plots or whisker plots,12 with the indication of the estimated mean (or median) and confidence interval (or interquartile range). Simple bar charts are no longer acceptable because this type of graphical presentation does not show the full data set and its variability and may actually misrepresent data when distribution is not standard.13 Bar charts are perhaps the weakest link in the way data have been presented so far. We now ask authors to include a more transparent description of the statistical methods used and to provide a definition of dependent variables and covariates and justification for the method. The description should also include estimation of sample size, eg, the number of individual data points or animals per group, or the number of replicates for in vitro experiments. These details will increase transparency so that readers can better judge the strength of the conclusion (Table 1). {TBL 1} Statistical thinking is required at every stage of a research project: study design, sample selection, data management, and analysis. Statistical models are powerful methods to filter signals from noise in experimental data and help make accurate inferences. However, the application of statistics has been beset by misuse in biomedical research;14 following is a brief discussion of the most frequent problems.15, 16 Overfitting is a major threat to the validity and reproducibility of multivariable statistical models. In regression analyses, overfitting occurs when a model with too many parameters is fitted to a small data set, such that the number of parameters is greater than the number of events. It has been postulated that the number of events required for each risk factor is ≥10.17 However, recent studies have shown that such a rule of thumb is not adequate because the number of events is dependent on the number of risk factors/parameters, proportion of variance explained, shrinkage factor (ie, bias between sample statistics and population parameters), and incidence of disease.18 For example, a study on fracture risk with 10 risk factors, assuming 10% fracture incidence, shrinkage factor of 0.90, and 20% expected variance explained, would require approximately 400 individuals for a multivariable logistic regression model. However, if the proportion of variance explained is 10%, the required sample size would increase to 850 individuals. In observational studies, the inference of association between a certain exposure or treatment and an outcome can be distorted by confounders. Regression-based adjustment is a powerful method to control for confounders and help inference accuracy. In the presence of multiple covariates, some researchers are tempted to adjust for all covariates in the hope to derive the best possible unbiased association. However, appropriate adjustment requires substantive knowledge, and not all associations need adjustment. For example, adjustment for a covariate that does not affect the bias of the causal association between exposure and outcome is unnecessary.19 Another frequent problem is data-dependent or arbitrary categorization (or dichotomization) of continuous variables into distinct groups and using such categorization for further analysis. Such a categorization is statistically inefficient and can lead to misleading findings.20 It can also be a self-serving approach, to fit data within a preset model rather than testing a hypothesis. We recommend presenting a linear analysis of continuous data. When linear models are not satisfactory, other analyses, such as spline regression or nonparametric smoother, should be considered. Nonetheless, we recognize that categorization of data can sometimes be useful to illustrate clinical relationships that are not easily modeled by linear or spline models. There have been concerns that many discoveries have been missed because of the failure to use modern methods.21 For prognostic research, in addition to discrimination and calibration analyses, we recommend authors to adopt recent statistical developments such as bootstrap methods,22 reclassification analysis,23 and decision curve analysis.24 For selection of relevant predictors, the Bayesian model averaging25, 26 performs better than the stepwise regression. To “democratize” data analysis and improve replicability, we strongly invite authors to share their computational codes and raw data, if feasible. Testing statistical significance based upon estimation of a p value has been extremely helpful for scientific progress for the past 90 years. Let us remember that the p value represents the probability (p) of obtaining the data observed or more extreme data when the null hypothesis is true (eg, when there is no difference between two groups or no correlation between two variables). Unfortunately, p value is often misinterpreted or misused in questionable research practices, such as data dredging or “p hacking,” which occurs when data are being repeatedly analyzed without a preconceived plan until a “significant” finding is found, or, worse, when many outcome variables are analyzed and then only selected variables are reported.27 Journals have reacted to such practices. Notably, The New England Journal of Medicine has recently revised its guidelines for statistical reporting, strongly emphasizing interpretation of p values in the presence of multiple tests of hypothesis.28 The journal Basic and Applied Social Psychology has taken an extreme measure banning the p value altogether.29 Our position is that the concept of p value remains valid and is here to stay. In fact, we now ask that actual p values be presented as the result of null hypothesis significance testing; notations of inequality, eg, p < 0.05, or NS (not significant) will no longer be accepted in manuscripts submitted to JBMR or JBMR Plus. Authors are encouraged to more thoughtfully interpret the meaning of a p value within each experimental context. To help our readers and prospective authors, we would like to underscore a few key points for correct p value interpretation, based on recommendations by the American Statistical Association.30 The most frequent and problematic interpretation of p values is the dichotomization into “significant” versus “nonsignificant,” based upon a threshold of p = 0.05 that is considered “significant.”31 The inherent problem with this strict demarcation is that in some cases it may lead researchers to reaching conclusions that are not necessarily valid or sufficiently strong. For example, one can conclude that two groups “significantly” differ with p = 0.045 but readily dismiss a p = 0.06 as a null finding. Such conclusions may be fallacious if the sample size is not adequate. Even in simple probability terms, one could argue that a 6% probability is not “statistically significantly different” from a 4.5% probability. Furthermore, p values are highly variable between samples even when study power is high,32 and the rigid adoption of a threshold, such as 0.05, is arbitrary. In strictly statistical terms, a finding with p = 0.05 corresponds to a ~30% chance that there is in fact no real effect (eg, false positive).33, 34 This is a weak evidence against the null hypothesis. Accordingly, there have been calls to lower the threshold of p values to <0.00535 or even <0.00136 to reject the null hypothesis. With p < 0.001, the chance of false positive can be reduced to <5%, an acceptable risk to most researchers. It is important to keep in mind that p values do not reflect the effect size, nor do they represent the strength of a relationship; with a large enough sample size, even a very small difference can achieve very low p values. Likewise, a low p value does not by itself imply that the result has substantial clinical or biological interest. Ultimately, the conclusion of biological or clinical significance of an effect must be based on multiple lines of evidence and thoughtful assessment of the data as a whole, not on a single variable meeting an arbitrary probability threshold.30 Confidence intervals are better representation of evidence of effect as they provide the range of plausible values within which the point estimate may lie (eg, mean difference, odds ratio, relative risk). However, confidence intervals can sometimes be misinterpreted, leading to erroneous conclusions. For example, a study on the effect of supplementation of vitamins C and E during pregnancy concluded that the supplementation “does not reduce the risk of preeclampsia in nulliparous women, the risk of intrauterine growth restriction, or the risk of death or other serious outcomes in their infants.”37 However, a close inspection of the data reveals that although the p value was 0.20—therefore, “nonsignificant”—the relative risk of death or serious outcomes was 0.79, clearly favoring the supplementations group, but since the confidence interval included 1 (0.61 to 1.02) the estimated effect was “nonsignificant.”37 Although this is technically correct, it is also true that the likelihood that the estimated effect be lower than 1 (favoring supplements) is far greater than the chances of it being higher than 1. In fact, the effect can span a range from a clinically very relevant 39% risk reduction to a minuscule 2% risk increase. That the estimated risk falls well into the left side of 1 (21% risk reduction) strongly suggests that the study may just be underpowered. Authors must not forget that lack of statistical significance is by no means proof of no effect. Misinterpreting results of large-scale clinical studies may have disastrous consequences in clinical practice or future research. It is then very important to know what the probability of an effect is in its experimental context; p value from null hypothesis significance testing cannot answer that question. A good approach is Bayesian analysis, which combines prior knowledge of an effect (in the form of probability distribution) with existing data to estimate the probability of that effect. Consequently, Bayesian analysis allows the reporting of direct probability statements about any magnitude of difference that is of clinical interest. For instance, based on traditional p value and confidence intervals, the effect of alendronate on hip fracture may be deemed to be nonsignificant. However, a Bayesian analysis suggested that there is a 90% probability that alendronate reduced fracture risk by at least 20%.38 We encourage authors to adopt Bayesian analysis more often in their studies. Multiple tests of hypothesis It is well established that the more tests (group comparisons, associations) are conducted, the higher the likelihood of finding unexpected results, as the proportion of false-positive findings increases rapidly with the number of tests of hypothesis. For instance, at the alpha level of 5%, a study with 20 tests of association has 64% chance of finding at least one “significant” association, even if all of the associations are false. To minimize such spurious associations, the p value must be adjusted. This has been common practice with “modern” research involving “big data,” such as genomics and registry-based databases; such adjustments provide authors the opportunity of testing multiple hypotheses or run a complete “hypothesis-free” scan. Although a p value threshold of 5 × 10−8 has become a standard for assessing common variation in genomewide association studies, there is currently no such threshold for registry-based research. We thus recommend that authors take measures to adjust p values from multiple tests of hypothesis and discuss the impact on their findings and drawing of conclusions; we also recommend reporting the false-discovery rate in those scenarios.39 In summary, researchers rely on statistical models to extract meaning from complex data. Misapplication of statistical methods can result in misleading conclusions with negative consequences for the entire research enterprise. The revised guidelines for data reporting and statistical analysis have been developed to help authors submit rigorous and transparent research data and derive appropriate conclusions (Table 1). Improving transparency and rigor will require the efforts of authors, editors, reviewers, and readers. We are confident that implementation of the new guidelines represents a substantial step forward to ensure the robustness of the scientific findings published in JBMR and JBMR Plus.