Method comparison – a practical approach based on error identification

In 1965 Roy Barnett, in consultation with William Youden, proposed a systematic scheme to compare methods that are still remarkably relevant after more than five decades [1]. In the interim, numerous papers and guidelines on the topic have appeared, and broadly speaking, there are two schools of thought, one advocating scatter plots, regression and correlation, with the second promoting difference plots and analysis of differences [2–8]. The CLSI recommends both as in fact Barnett did [9]. Our comments are directed towards the comparison of field methods, with the assumption that they have been validated and have known performance characteristics. We advocate a simple and consistent approach based on fundamental aspects and principles. The focus should primarily be that of common sense, “fit-for-purpose”, judicious use of statistical analysis and to some extent on “value judgements”. Results should be presented in a way that would enable a third party to critically evaluate the data and to draw their own conclusions. Wherever possible, the data and statistical parameters should be presented in the actual units of measurement. Imprecision is often presented in relative units (coefficient of variation), which has led to the misconception that precision of methods is worse at low concentrations when the opposite is in fact true when one considers the variance in actual units (standard deviation [SD]). Presenting information as ratios (percentages or logarithms) can potentially obscure the relationship, or the absence thereof, between methods. Transformation of data and elaborate statistical procedures should be used as a last resort as it creates difficult-to-grasp complexity. It is important to realise that in a chemical measurement procedure (method), a response signal is generated that is proportional to the concentration of the analyte of interest. This signal is used to estimate the concentration via a calibration procedure. The evaluated field methods are assumed to have no systematic bias but are always accompanied by imprecision (“noise”) [10]. Thus, the starting point is an expectation that bias is absent and that differences in responses can be explained by the imprecision of both methods. This can be assessed from a scatter plot, which is intuitively easy to interpret. The same information can be presented in a difference plot, which is simply a derivative of the former. The main advantage of difference plots is that they present the variances between the methods in a way that may be easier to relate to analytical imprecision than a plot of the residuals or the original scatter plot. Differences between methods are due to analytical errors and therefore an appreciation of the error components is essential. Errors can be classified as systematic or random. Systematic error includes constant and proportional bias and is the result of errors in calibration. Random error consists of imprecision and sample-method error or bias [10]. Sample-method bias, also called patient/method interaction, method-related factors, between-method analytical variation and aberrant sample bias (matrix effect), is often neglected and is the result of an error (bias) that is unique to a sample and is caused by non-specificity [3, 11–13]. This error may substantially contribute to the apparent random differences between methods, and it may also affect bias, especially if the interference is predominantly one-sided. Another type of error is outliers, which are large errors that are distinct from the distribution of the majority of results. IUPAC refers to these errors as blunders, whereas Barnett called these “large discrepancies” [1, 10]. Barnett recommended that these results be removed and investigated separately. Method comparison can be viewed as a process by which the error components are characterised. Errors within acceptable limits indicate adequate agreement and that the test method is fit-for-purpose. These limits should be determined a priori and based on what is deemed clinically acceptable [12]. Unexpectedly large differences indicate an error in one or both methods and this requires further scrutiny and may mandate more complicated statistical analysis. Hypothesis testing, where the null hypothesis states that there is no systematic difference and that the distribution of differences is due to imprecision, is straightforward. We recommend a stepwise approach to method comparison that will identify all components of error in the majority of cases (Figure 1). The acceptable error limits