Exploring Data Distribution Prior to Analysis: Benefits and Pitfalls

population of change scores and proceed to examine the diff erence between the mean baseline and follow-up salivary melatonin concentration with a parametric paired t-test. This test is statistically signifi cant (P = 0.044) for the data in ● ▶ Table 1 , indicating that there is a change in mean salivary melatonin concentration, the 95 % confi dence interval for this change being 0.10 – 6.34 pg / ml. The distribution of real data is often not Gaussian, and it is interesting to examine what happens when an ‘ outlier ’ is present in the sample. For example, let us assume that data have been collected from an additional participant ( ● ▶ Table 2 ). This tenth participant shows a very large change in melatonin concentration between baseline and follow-up measurements. Therefore, it would be tempting to think that this would make the mean change of the sample greater than before and the P-value for statistical signifi cance smaller (and the confi dence interval narrower). However, the fundamental signal-to-noise nature of statistical hypothesis tests means that this is not the case at all. When the data in ● ▶ Table 2 are analysed with a paired t-test as before, the P-value is now no longer statistically signifi cant (P = 0.09) and the confi dence interval of − 1.20 to13.84 pg / ml is now much wider than before and overlaps zero, even though the mean change has doubled to 6.3 pg / ml. Why is this? It can be seen in ● ▶ Table 2 that, although the mean change is larger than before, the standard deviation of the change scores is also much larger than before (10.5 vs. 4.1 pg / ml), i. e. the inclusion of the outlier subject has essentially increased the error variance in the statistical analysis so that statistical signifi cance is not reached and the confi dence interval is less precise However, it would be erroneous to conclude from the results of the paired t-test that the mean change is not statistically signifi cant in this sample of 10 participants. Essentially, the wrong test has been selected for the distribution of data that are being analysed. To explain, the Shapiro-Wilks test of normal distribution is now statistically signifi cant for the data in ● ▶ Table 2 (P = 0.001). Inclusion of the outlier subject has skewed the dataset, so much so that it is no longer likely to have come from a normally distributed population. One solution to this problem is to employ non-parametric analyses, a ‘ mirror ’ non-parametric test to a paired t-test being the Wilcoxon test. This test indicates that there has in fact been a signifi cant change in the data between baseline and followup measurements (P = 0.028). The choice of appropriate statistics is important for all research and should be based on the study design, the research question and whether assumptions about the data are upheld or not. Authors commonly select so-called ‘ parametric ’ tests, which are associated with several assumptions about the data, e. g. that data are normally distributed in the population of interest. Thankfully, many authors who submit manuscripts to IJSM check that their data meets the necessary assumptions for use of parametric statistical analyses. Nevertheless, some authors do not complete this important step, and are, therefore, asked by editors to justify their choice of statistics. Those authors who do check underlying statistical assumptions sometimes do so using an inappropriate aspect of the data and / or believe that the only solution to violation of parametric assumptions is the adoption of non-parametric ‘ ranked ’ tests. The aim of this statistical note is to illustrate the importance of checking assumptions with the correct aspect of the data and arriving at the correct choice of analysis, which may involve a mathematical transformation of the data prior to analysis. In ● ▶ Table 1 , some hypothetical data are presented to represent the salivary melatonin concentrations of 9 healthy people measured before and after a bright light intervention in the evening. It is hypothesised that melatonin concentration is reduced following bright light exposure [2] . Most readers will recognise that the most appropriate parametric test for this singlegroup pre / post design is a paired t-test. However, it is important to explore whether the underlying assumptions for this test are upheld or not. In my experience, it is often believed that the baseline and follow-up data-sets should be normally distributed for the paired t-test to be appropriate. This is not so. The necessary assumption for the paired t-test is that the change scores or ‘ residuals ’ are sampled from a normally distributed population [6] and this is the case for many statistical tests of the repeated measures type. These change scores (baseline minus follow-up data) are presented in the third column of ● ▶ Table 1 . The exploration of the underlying distribution of data is itself a thorny issue, especially when sample sizes are small [3, 4] . One approach is to examine, using the Shapiro-Wilks test for example, the null hypothesis that the change scores have been sampled from a normally distributed population. For the change scores presented in ● ▶ Table 1 , the Shapiro-Wilks test is not statistically signifi cant (P = 0.14). Therefore, it seems reasonable to assume a normally distributed G. Atkinson C. Pugh M. A. Scott