Statistical Methods in Medical Research 1999; 8: 135±160

Measuring agreement in method comparison
studies
J Martin Bland Department of Public Health Sciences, St George's Hospital Medical School,
London, UK and Douglas G Altman ICRF Medical Statistics Group, Centre for Statistics in
Medicine, Institute of Health Sciences, Oxford, UK
Agreement between two methods of clinical measurement can be quanti®ed using the differences between
observations made using the two methods on the same subjects. The 95% limits of agreement, estimatedby mean difference  1.96 standard deviation of the differences, provide an interval within which 95% ofdifferences between measurements by the two methods are expected to lie. We describe how graphicalmethods can be used to investigate the assumptions of the method and we also give con®dence intervals.
We extend the basic approach to data where there is a relationship between difference and magnitude,
both with a simple logarithmic transformation approach and a new, more general, regression approach.
We discuss the importance of the repeatability of each method separately and compare an estimate of thisto the limits of agreement. We extend the limits of agreement approach to data with repeatedmeasurements, proposing new estimates for equal numbers of replicates by each method on each subject,
for unequal numbers of replicates, and for replicated data collected in pairs, where the underlying value ofthe quantity being measured is changing. Finally, we describe a nonparametric approach to comparing
methods.

1 The method comparison problem
In clinical medicine we often wish to measure quantities in the living body, such as
cardiac stroke volume or blood pressure. These can be extremely dif®cult or
impossible to measure directly without adverse effects on the subject and so theirtrue values remain unknown. Instead we have indirect methods of measurement, andwhen a new method is proposed we can assess its value by comparison only with otherestablished techniques rather than with the true quantity being measured. We cannot
be certain that either method gives us an unequivocally correct measurement and wetry to assess the degree of agreement between them. The standard method issometimes known as the `gold standard', but this does not ± or should not ± imply that
it is measured without error.
Some lack of agreement between different methods of measurement is inevitable.
What matters is the amount by which methods disagree. We want to know by how
much the new method is likely to differ from the old, so that if this is not enough tocause problems in clinical interpretation we can replace the old method by the new, oreven use the two interchangeably. For example, if a new machine for measuring bloodpressure were unlikely to give readings for a subject which differed by more than, say,
10 mmHg from those obtained using a sphygmomanometer, we could rely on measurements made by the new machine, as differences smaller than this would not materiallyAddress for correspondence: JM Bland, Department of Public Health Sciences, St George's Hospital Medical School,
Cranmer Terrace, London SW17 0RE, UK.

Ó Arnold 1999

0962-2802(99)SM181RA

136 JM Bland and DG Altman
affect decisions as to management. On the other hand, differences of 30 mmHg or
more would not be satisfactory as an error of this magnitude could easily lead to achange in patient management. How far apart measurements can be without leading
to problems will depend on the use to which the result is put, and is a question ofclinical judgement. Statistical methods cannot answer such a question. Methods which
agree well enough for one purpose may not agree well enough for another. Ideally, we
should de®ne satisfactory agreement in advance.
In this paper we describe an approach to analysing such data, using simple
graphical techniques and elementary statistical calculations.1;2 Our approach is basedon quantifying the variation in between-method differences for individual patients.
We provide a method which is simple for medical researchers to use, requiring only
basic statistical software. It gives estimates which are easy to interpret and in the sameunits as the original observations. We concentrate on the interpretation of theindividual measurement on the individual patient.
We extend the approach to the case where the between-method differences vary withthe size of the measurement, and we show how to compare methods when replicatedmeasurements are available. We also describe a nonparametric approach for use when
there may be occasional extreme deviations between the methods.

2 Limits of agreement
We want a measure of agreement which is easy to estimate and to interpret for ameasurement on an individual patient. An obvious starting point is the difference
between measurements by the two methods on the same subject. There may be aconsistent tendency for one method to exceed the other. We shall call this the bias andestimate it by the mean difference. There will also be variation about this mean, which
we can estimate by the standard deviation of the differences. These estimates aremeaningful only if we can assume bias and variability are uniform throughout therange of measurement, assumptions which can be checked graphically (Section 2.1).
Table 1 shows a set of systolic blood pressure data from a study in which simultaneous measurements were made by each of two experienced observers (denoted J andR) using a sphygmomanometer and by a semi-automatic blood pressure monitor(denoted S). Three sets of readings were made in quick succession. We shall start byconsidering only the ®rst measurement by observer J and the machine (i.e. J1 and S1)
to illustrate the analysis of unreplicated data (Figure 1).
The mean difference (observer minus machine) is d  ÿ16:29 mmHg and thestandard deviation of the differences is sd  19:61 mmHg. If the differences arenormally distributed, we would expect 95% of the differences to lie between d ÿ 1:96sd
and 
d  1:96sd (we can use the approximation d  2sd with minimal loss of accuracy).
We can then say that nearly all pairs of measurements by the two methods will becloser together than these extreme values, which we call 95% limits of agreement. These
values de®ne the range within which most differences between measurements
by the two methods will lie. For the blood pressure data these values areMeasuring agreement in method comparison studies

137

Table 1 Systolic blood pressure measurements made simultaneously by two observers (J and R) and an
automatic blood pressure measuring machine (S), each making three observations in quick succession (datasupplied by Dr E O'Brien, see Altman and Bland20)
Subject

J1

J2

J3

R1

R2

R3

S1

S2

S3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

100
108
76
108
124
122
116
114
100
108
100
108
112
104
106
122
100
118
140
150
166
148
174
174
140
128
146
146
220
208
94
114
126
124
110
90
106
218
130
136
100
100
124
164
100
136
114
148
160
84
156
110
100
100
86

106
110
84
104
112
140
108
110
108
92
106
112
112
108
108
122
102
118
134
148
154
156
172
166
144
134
138
152
218
200
84
124
120
124
120
90
106
202
128
136
96
98
116
168
102
126
108
120
150
92
162
98
106
102
74

107
108
82
104
112
124
102
112
112
100
104
122
110
104
102
114
102
120
138
144
154
134
166
150
144
130
140
148
220
192
86
116
122
132
128
94
110
208
130
130
88
88
122
154
100
122
122
132
148
98
152
98
106
94
76

98
108
76
110
128
124
118
112
100
108
102
108
114
104
104
118
102
116
138
148
164
136
170
174
140
128
146
146
220
204
94
112
124
126
110
88
106
218
128
136
100
100
126
164
100
136
114
146
160
86
156
108
100
100
88

98
112
88
100
112
140
110
108
106
98
108
116
112
108
106
122
102
118
136
146
154
154
170
166
144
134
138
152
218
200
84
126
120
126
122
88
108
200
126
138
96
98
116
168
104
124
108
130
152
92
158
100
108
102
76

111
110
82
106
114
126
102
112
112
100
106
120
110
104
102
114
100
118
134
144
148
132
164
154
144
130
138
148
220
190
88
118
120
120
126
94
110
206
128
128
86
88
122
154
102
122
122
132
146
98
152
98
108
96
76

122
121
95
127
140
139
122
130
119
126
107
123
131
123
127
142
104
117
139
143
181
149
173
160
158
139
153
138
228
190
103
131
131
126
121
97
116
215
141
153
113
109
145
192
112
152
141
206
151
112
162
117
119
136
112

128
127
94
127
131
142
112
129
122
113
113
125
129
126
119
133
116
113
127
155
170
156
170
155
152
144
150
144
228
183
99
131
123
129
114
94
121
201
133
143
107
105
102
178
116
144
141
188
147
125
165
118
131
116
115

124
128
98
135
124
136
112
135
122
111
111
125
122
114
126
137
115
112
113
133
166
140
154
170
154
141
154
131
226
184
106
124
124
125
125
96
127
207
146
138
102
97
137
171
116
147
137
166
136
124
189
109
124
113
104

138 JM Bland and DG Altman
Table 1 Continued
Subject

J1

J2

J3

R1

R2

R3

S1

S2

S3

56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

106
108
168
166
146
204
96
134
138
134
156
124
114
112
112
202
132
158
88
170
182
112
120
110
112
154
116
108
106
122

100
110
188
150
142
198
94
126
144
136
160
138
110
116
116
220
136
162
76
174
176
114
118
108
112
134
112
110
98
112

110
106
178
154
132
188
86
124
140
142
154
138
114
122
134
228
134
152
88
176
180
124
120
106
106
130
94
114
100
112

106
106
170
164
144
206
96
132
140
136
156
122
112
112
114
200
134
158
90
172
184
112
118
110
112
156
118
106
104
122

100
118
188
150
142
198
94
126
142
134
162
140
114
114
114
220
136
164
76
174
174
112
116
108
110
136
114
110
100
114

108
106
182
154
130
188
84
124
138
140
156
136
114
124
136
226
132
150
86
178
178
126
120
106
106
132
96
114
100
114

120
117
194
167
173
228
77
154
154
145
200
188
149
136
128
204
184
163
93
178
202
162
227
133
202
158
124
114
137
121

118
118
191
160
161
218
89
156
155
154
180
147
217
132
125
222
187
160
88
181
199
166
227
127
190
121
149
118
135
123

132
115
196
161
154
189
101
141
148
166
179
139
192
133
142
224
192
152
88
181
195
148
219
126
213
134
137
126
134
128

Figure 1 Systolic blood pressure measured by observer J using a sphygmomanometer and by machine S,
with the line of equality

Measuring agreement in method comparison studies139

ÿ16:29 ÿ 1:96  19:61 and ÿ16:29  1:96  19:61 mmHg or ÿ54:7 and +22.1 mmHg.
Provided differences within the observed limits of agreement would not be clinicallyimportant we could use the two measurement methods interchangeably.
Note that despite the super®cial similarity these are not the same thing as
con®dence limits, but like a reference interval. Of course, we could use somepercentage other than 95% for the limits of agreement, but we ®nd it convenient tostick with this customary choice.
As we indicated, the calculation of the 95% limits of agreement is based on theassumption that the differences are normally distributed. Such differences are, in fact,
quite likely to follow a normal distribution. We have removed a lot of the variation
between subjects and are left with the measurement error, which is likely to be normal
anyway. We then added two such errors together which will increase the tendency
towards normality. We can check the distribution of the differences by drawing ahistogram or a normal plot. If the distribution is skewed or has very long tails theassumption of normality may not be valid. This is perhaps most likely to happen when
the difference and mean are related, in which case corrective action can be taken as
described in Section 3. Further, a non-normal distribution of differences may not be as
serious here as in other statistical contexts. Non-normal distributions are still likely tohave about 5% of observations within about two standard deviations of the mean,although most of the values outside the limits may be differences in the same
direction. We note that is not necessary for the measurements themselves to follow anormal distribution. Indeed, they often will not do so as subjects are chosen to give awide and uniform distribution of the quantity measured rather than being a random
sample.
If there is a consistent bias it is a simple matter to adjust for it, should it benecessary, by subtracting the mean difference from the measurements by the new
method. In general, a large sd and hence widely spaced limits of agreement is a much
more serious problem.
In the blood pressure example we can see that there are some very large differenceswhere the machine gave readings considerably above the sphygmomanometer. There
are in fact still about 5% of values outside the limits of agreement (4/85) but they all
lie below the lower limit. We can evaluate the impact of the two largest, apparently
outlying values (from subjects 78 and 80) by recalculating the limits excluding them.
The mean difference becomes ÿ14:9 mmHg and the 95% limits of agreement areÿ43:6 to +15.0 mmHg. The span has reduced from 77 to 59 mmHg, a noticeable but
not particularly large reduction. We do not recommend excluding outliers fromanalyses, but it may be useful to assess their in¯uence on the results in this way. We
usually ®nd that this method of analysis is not too sensitive to one or two large
outlying differences.
From Table 1 we can see that these large discrepancies were not due to single odd
readings as the difference was present for all three readings by each method. In thecase of automatic blood pressure measuring machines this phenomenon is quite
common. For this reason a nonparametric approach was developed to handle suchdata ± we describe this method in Section 6.
As we remarked earlier (Section 1) the decision about what is acceptable agreement
is a clinical one; statistics alone cannot answer the question. From the above140 JM Bland and DG Altman
calculations we can see that the blood pressure machine may give values between
55 mmHg above the sphygmomanometer reading to 22 mmHg below it. Such differences would be unacceptable for clinical purposes.
2.1

Graphical presentation of agreement

Graphical techniques are especially useful in method comparison studies. Figure 1
shows the measurement by the observer using a sphygmomanometer plotted againstthat by the machine. The graph also shows the line of equality, the line all points
would lie on if the two meters always gave exactly the same reading. We do not
calculate or plot a regression line here as we are not concerned with the estimatedprediction of one method by another but with the theoretical relationship of equalityand deviations from it. It is helpful if the horizontal and vertical scales are the same sothat the line of equality will make an angle of 45 to both axes. This makes it easier toassess visually how well the methods agree. However, when the range of variation of
the measurements is large in comparison with the differences between the methods
this plot may obscure useful information.
A better way of displaying the data is to plot the difference between themeasurements by the two methods for each subject against their mean. This plot forthe blood pressure data (Figure 2) shows explicitly the lack of agreement that is lessobvious in Figure 1. The plot of difference against mean also allows us to investigate
any possible relationship between the discrepancies and the true value. We can
examine the possible relation formally by calculating the rank correlation between theabsolute differences and the average; here Spearman's rank correlation coef®cient is
rs  0:07. The plot will also show clearly any extreme or outlying observations. It is
often helpful to use the same scale for both axes when plotting differences against
mean values (as in Figure 2). This feature helps to show the discrepancies in relation

Figure 2 Systolic blood pressure: difference (JÿS) versus average of values measured by observer J andmachine S

Measuring agreement in method comparison studies

141

Figure 3 Systolic blood pressure: difference (JÿS) versus average of values measured by observer J andmachine S with 95% limits of agreement

to the size of the measurement. When methods agree quite closely, however, equal
scaling may be impracticable and useful information will again be obscured. We can
add the 95% limits of agreement to this plot (Figure 3) to provide a good summary
picture.
In such studies we do not know the true values of the quantity we are measuring, so
we use the mean of the measurements by the two methods as our best estimate. It is a
mistake to plot the difference against either value separately, as the difference will be
related to each, a well-known statistical phenomenon.3
2.2

Precision of the estimated limits of agreement

We can calculate standard errors and con®dence intervals for the limits ofagreement if we can assume that the differences follow a distribution which isapproximately normal. The variance of d is estimated by s2d =n, where n is the sample
size. Provided the differences are normally distributed, the variance of sd is2
2
approximately estimatedp by sd =2 n ÿ 1. (This because sd is distributed as222  d = n ÿ 1 and  has approximate variance 1/2.) The mean difference d andsd are independent. Hence the variance of the limits of agreement is estimated byVar d  1:96sd   Var d  1:962 Var sd 
s2s2d
 d  1:962n
2 n ÿ 1


1
1:962


s2
n 2 n ÿ 1 d

142 JM Bland and DG Altman
Unless n is small, this can be approximated closely by
1:962 s2d
Var d  1:96sd   1 2ns2
 2:92 dns2 1:712 dn ÿ 1:96sd and d  1:96sd are approximately
Hence,pthestandard errors of dd. Ninety-®ve per cent con®dence intervals can be calculated
1:71sd = n  1:71SE by ®nding the appropriate point of the t distribution with n ÿ 1 degrees of freedom.
The con®dence interval will be t standard errors either side of the observed value.
Fordata sd  19:61 mmHg, so the standard error of the bias d isp the bloodppressure
sd = n  19:61= 85  2:13 mmHg. For the 95% con®dence interval, we have 84
degrees of freedom and t=1.99. Hence the 95% con®dence interval for the bias isÿ16:29 ÿ 1:99  2:13 to ÿ16:29  1:99  2:13, giving ÿ20:5 to ÿ12:1 mmHg. The
standard error of the 95% limits of agreement is 1:71SE d  3:64 mmHg. The 95%
con®dence interval for the lower limit of agreement is ÿ54:7 ÿ 1:99  3:64 toÿ54:7  1:99  3:64, giving ÿ61:9 to ÿ47:5 mmHg. Similarly the 95% con®dence
interval for the upper limit of agreement is 22:1 ÿ 1:99  3:64 to 22:1  1:99  3:64,
giving 14.9 to 29.3 mmHg. These intervals are reasonably narrow, re¯ecting the quite
large sample size. They show, however, that even on the most optimistic interpretation
there can be considerable discrepancies between the two methods of measurement and
we would conclude that the degree of agreement was not acceptable.
These con®dence limits are based on considering only uncertainty due to sampling
error. There is the implicit assumption that the sample of subjects is a representative
one. Further, all readings with the sphygmomanometer were made by a single (skilled)
observer. The calculated uncertainty associated with the limits of agreement is thus
likely to be somewhat optimistic.

3 Relationship between difference and magnitude
In Section 2 we assumed that the mean and standard deviation of the differences are
the same throughout the range of measurement. The most common departure from theassumptions is an increase in variability of the differences as the magnitude of themeasurement increases. In such cases a plot of one method against the other shows aspreading out of the data for larger measurements. The mean difference (d) may also
be approximately proportional to the magnitude of the measurement. These effects areseen even more clearly in the difference versus mean plot. For example, Table 2 shows
measurements of plasma volume expressed as a percentage of the expected value fornormal individuals. The data are plotted in Figure 4(a). It can be seen immediatelythat the two methods give systematically different readings, and that all theMeasuring agreement in method comparison studies

143

Table 2 Measurements of plasma volume expressed as a percentage of normal in 99 subjects, using twoalternative sets of normal values due to Nadler and Hurley (data supplied by C DoreÂ, see Cotes et al.21)
Sub

NadlerHurleySub

NadlerHurleySub

NadlerHurley

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

56.9
63.2
65.5
73.6
74.1
77.1
77.3
77.5
77.8
78.9
79.5
80.8
81.2
81.9
82.2
83.1
84.4
84.9
86.0
86.3
86.3
86.6
86.6
86.6
87.1
87.5
87.8
88.6
89.3
89.6
90.3
91.1
92.1

52.9
59.2
63.0
66.2
64.8
69.0
67.1
70.1
69.2
73.8
71.8
73.3
73.1
74.7
74.1
74.1
76.0
75.4
74.6
79.2
77.8
80.8
77.6
77.5
78.6
78.7
81.5
79.3
78.9
85.9
80.7
80.6
82.8

34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

93.5
94.5
94.6
95.0
95.2
95.3
95.6
95.9
96.4
97.2
97.5
97.9
98.2
98.5
98.8
98.9
99.0
99.3
99.3
99.9
100.1
101.0
101.0
101.5
101.5
101.5
101.8
101.8
102.8
102.9
103.2
103.8
104.4

86.0
84.3
87.6
84.0
85.9
84.4
85.2
85.2
89.2
87.8
88.0
88.7
91.2
91.8
92.5
88.0
93.5
89.0
89.4
89.2
91.3
90.4
91.2
91.4
93.0
91.2
92.0
91.8
96.8
92.8
94.0
93.5
95.8

67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

104.8
105.1
105.5
105.7
106.1
106.8
107.2
107.4
107.5
107.5
108.0
108.2
108.6
109.1
110.1
111.2
111.7
111.7
112.0
113.1
116.0
116.7
118.8
119.7
120.7
122.8
124.7
126.4
127.6
128.2
129.6
130.4
133.2

97.1
97.3
95.1
95.8
95.5
95.9
95.4
97.3
97.7
93.0
97.6
96.1
96.2
99.5
99.8
105.3
103.6
100.2
100.0
98.8
110.0
103.5
109.4
112.1
111.3
108.6
112.4
113.8
115.6
118.1
116.8
121.6
115.8

observations lie above the line of equality. It is less easy to see that the differences
increase as the plasma volume rises, but a plot of difference versus mean shows suchan effect very clearly (Figure 4b).
We could ignore the relationship between difference and magnitude and proceed asin Section 2. The analysis will still give limits of agreement which will include mostdifferences, but they will be wider apart than necessary for small plasma volumes, andrather narrower than they should be for large plasma volumes. It is better to try to
remove this relationship, either by transformation of the measurements, or, if thisfails, by a more general method.
3.1

Logarithmic transformation

Under these circumstances, logarithmic (log) transformation of both measurements
before analysis will enable the standard approach to be used. The limits of agreement
derived from log transformed data can be back-transformed to give limits for the ratio144 JM Bland and DG Altman

Figure 4 (a) Measurements of plasma volume as listed in Table 2; (b) plot of differences versus average with
95% limits of agreementof the actual measurements.2 While other transformations could in principle be used(such as taking square roots or reciprocals), only the log transformation allows theresults to be interpreted in relation to the original data. As we think that there is aclear need in such studies to be able to express the results in relation to the actual
measurements we do not think that any other transformation should be used in thiscontext.
For the data of Table 2, log transformation is highly successful in producing
differences unrelated to the mean. Figure 5 shows the log transformed data and thedifference versus mean plot with superimposed 95% limits of agreement. The dataclearly meet the requirements of the statistical method very well. The mean difference(log Nadler ÿ log Hurley) is 0.099 with 95% limits of agreement 0.056 and 0.141.
Because of the high correlation and reasonably large sample size the con®dence
intervals for the limits of agreement are narrow. For example, the 95% con®dence
interval for the lower limit is from 0.049 to 0.064.
These results relate to differences between log percentages and are not easy tointerpret. As noted above, we can back-transform (antilog) the results to get values

Figure 5 (a) Measurements of plasma volume after log transformation; (b) Difference between plasma volume
measurements plotted against their average after loge transformation with 95% limits of agreement

Measuring agreement in method comparison studies

145

Figure 6 Ratio of plasma volume measurements plotted against their averagerelating to the ratios of measurements by the two methods. The geometric mean ratioof values by the Nadler and Hurley methods was 1.11 with 95% limits of agreement
1.06 to 1.15. Thus the Nadler method exceeds the Hurley method by between 1.06 and1.15 times, i.e. by between 6% and 15%, for most measurements.
This example illustrates the importance of both the bias (d) and the standard
deviation of between-method differences (sd ). Here there is a clear bias with allobservations lying to one side of the line of equality (Figure 4a). However, because thescatter around the average difference is rather small we could get excellent agreement
between the two methods if we ®rst applied a conversion factor, multiplying the
Hurley method or dividing the Nadler method by 1.11.
We can make the transformation process more transparent by working directly withthe ratios. Instead of taking logs and calculating differences we can simply calculate
the ratio of the two values for each subject and calculate limits of agreement based on
the mean and SD of these. Figure 6 shows the plot corresponding to this approach,which is almost identical to Figure 5(b).
The type of plot shown in Figure 6 was suggested as a general purpose approach formethod comparison studies,4 although without any suggestion for quantifying thedifferences between the methods. Another variation is to plot on the vertical axis thedifference between the methods as a percentage of their average.5
3.2

A regression approach for nonuniform differences

Sometimes the relationship between difference and mean is complicated and logtransformation may not solve the problem. For example, the differences may tend to
be in one direction for low values of the quantity being measured and in the otherdirection for high values. For data sets for which log transformation does not remove
the relationship between the differences and the size of the measurement, a plot in thestyle of Figure 5(b) is still enormously helpful in comparing the methods. In such

146 JM Bland and DG Altman
cases, formal analysis as described in Section 2 will tend to give limits of agreement
which are too far apart rather than too close, and so should not lead to the acceptanceof poor methods of measurement. Nevertheless, it is useful to have better approaches
to deal with such data.
We propose modelling the variability in the SD of the di directly as a function of thelevel of the measurement, using a method based on absolute residuals from a ®tted
regression line. If the mean also changes as a function of level we can ®rst model that
relation and then model the SD of the residuals. This approach is based on one used toderive age-related reference intervals.6
We ®rst consider the mean difference between the methods in relation to the size ofthe measurement. We suggested in our ®rst paper on method comparison studies1 thatwhen the agreement between the methods varies as the measurement varies, we canregress the difference between the methods (D) on the average of the two methods (A).
However, we did not give a worked example, and did not consider there the possibility
that the standard deviation of the differences, sd , might also vary with A. A similar idea
was put forward by Marshall et al.,7 who use a more complex method than that
proposed here.
Unless the plot of the data shows clear curvature (which is very unlikely in our
experience) simple linear regression is all that is needed, giving^  b0  b1 AD

3:1^  d, the mean difference. We have deliberately
If the slope b1 is not signi®cant then D
not de®ned what we mean by signi®cant here as we feel that this may require clinicaljudgement as well as statistical considerations. If b1 is signi®cantly different from zero
we obtain the estimated difference between the methods from equation (3.1) for any
true value of the measurement, estimated by A.
Table 3 shows the estimated fat content of human milk (g/100 ml) determined by
the measurement of glycerol released by enzymic hydrolysis of triglycerides andTable 3 Fat content of human milk determined by enzymic procedure for the determination of triglycerides
and measured by the Standard Gerber method (g/100 ml)8Trig.

GerberTrig.

GerberTrig.

Gerber

0.96
1.16
0.97
1.01
1.25
1.22
1.46
1.66
1.75
1.72
1.67
1.67
1.93
1.99
2.01

0.85
1.00
1.00
1.00
1.20
1.20
1.38
1.65
1.68
1.70
1.70
1.70
1.88
2.00
2.05

2.28
2.15
2.29
2.45
2.40
2.79
2.77
2.64
2.73
2.67
2.61
3.01
2.93
3.18
3.18

2.17
2.20
2.28
2.43
2.55
2.60
2.65
2.67
2.70
2.70
2.70
3.00
3.02
3.03
3.11

3.19
3.12
3.33
3.51
3.66
3.95
4.20
4.05
4.30
4.74
4.71
4.71
4.74
5.23
6.21

3.15
3.15
3.40
3.42
3.62
3.95
4.27
4.30
4.35
4.75
4.79
4.80
4.80
5.42
6.20

Measuring agreement in method comparison studies

147

measurements by the standard Gerber method.8 Figure 7(a) shows that the two
methods agree closely, but from Figure 7(b) we can see a tendency for the differences
to be in opposite directions for low and high fat content. The variation (SD) of thedifferences seems much the same for all levels of fat content. These impressions arecon®rmed by regression analyses. The regression of D on A gives^  0:079 ÿ 0:0283A g=100 mlD
as noted by Lucas et al.8 The SD of the residuals is 0.08033.
In the second stage of the analysis we consider variation around the line of best
agreement (equation (3.1)). We need to model the scatter of the residuals from model(3.1) as a function of the size of the measurement (estimated by A). Modelling is
considerably simpli®ed by the assumption that these residuals have a normal
distribution whatever the size of the measurement, which is a fairly natural extensionof the assumption we make already in such analyses. We then regress the absolutevalues of the residuals, which we will call R, on A to get^  c0  c1 AR

3:2If we take a normal distribution with mean zero and variance 2 , it is easy to showthatp
the mean of the absolute values, which follow a half-normal distribution, is  2=.
The standard
deviation of the residuals is thus obtained by multiplying the ®ttedp
values by
=2. The limits of agreement are obtained by combining the tworegression equations (see Altman6 ).
Although in principle any form of regression model might be ®tted here, it is very
likely that if the SD is not constant then linear regression will be adequate to describe
the relationship. If there is no `signi®cant' relation between R and A the estimatedstandard deviation is simply the standard deviation of the adjusted differences, theresiduals of equation (3.1).
In the general case where both models (3.1) and (3.2) are used, the expected value of^  b0  b1 A and the 95% limits ofthe difference between methods is given by D

Figure 7 (a) Fat content of human milk determined by enzymic procedure for the determination of triglycerides
and measured by the standard Gerber method (g/100 ml); (b) plot of difference (TriglycerideÿGerber) against
the average

148 JM Bland and DG Altman

Figure 8 Regression based limits of agreement for difference in fat content of human milk determined by
Triglyceride and Gerber methods (g/100 ml)

agreement are obtained asp
^  1:96 =2 R^ D^  2:46R
^
D

or as
b0  b1 A  2:46fc0  c1 Ag

3:3

Returning to the example, there was no relation between the residuals from the ®rst
regression model and A and so the SD of the adjusted differences is simply theresidual SD from the regression, so that sd  0:08033. We can thus calculate theregression based 95% limits of agreement as 0:079 ÿ 0:0283a  1:96  0:08033 g/100
ml, where a is the magnitude (average of methods) of the fat content. These values areshown in Figure 8. Of course, in clinical practice, when only one method is being used,
the observed value by that method provides the value of a.
4 The importance of repeatability
The comparison of the repeatability of each method is relevant to method comparison
because the repeatabilities of two methods of measurement limit the amount ofagreement which is possible. Curiously, replicate measurements are rarely made inmethod comparison studies, so that an important aspect of comparability is often
overlooked. If we have only one measurement using each method on each subject wecannot tell which method is more repeatable (precise). Lack of repeatability caninterfere with the comparison of two methods because if one method has poor
repeatability, in the sense that there is considerable variation in repeated measure-Measuring agreement in method comparison studies

149

ments on the same subject, the agreement between the two methods is bound to bepoor. Even if the measurements by the two methods agreed very closely on average,
poor repeatability of one method would lead to poor agreement between the methods
for individuals. When the old method has poor repeatability even a new method which
was perfect would not agree with it. Lack of agreement in unreplicated studies may
suggest that the new method cannot be used, but it might be caused by poor
repeatability of the standard method. If both methods have poor repeatability, thenpoor agreement is highly likely. For this reason we strongly recommend thesimultaneous estimation of repeatability and agreement by collecting replicated data.
It is important ®rst to clarify exactly what we mean when we refer to replicate
observations. By replicates we mean two or more measurements on the same
individual taken in identical conditions. In general this requirement means that themeasurements are taken in quick succession.
One important feature of replicate observations is that they should be independentof each other. In essence, this is achieved by ensuring that the observer makes eachmeasurement independent of knowledge of the previous value(s). This may be dif®cultto achieve in practice.
4.1

Estimating repeatability

A very similar analysis to the limits of agreement approach can be applied toquantify the repeatability of a method from replicated measurements obtained by thesame method. Using one-way analysis of variance, with subject as the factor, we canestimate the within-subject standard deviation, sw , from the square root of the residual
mean square. We can compare the standard deviations of different methods to see
which is more repeatable. Each standard deviation can also be used to calculate limits
within which we expect the differences between two measurements by the same
method to lie. As well as being informative in its own right, the repeatability indicatesa baseline against which to judge between-method variability.
The analysis is simple because we expect the mean difference between replicates tobe zero ± we do not usually expect second measurements of the same samples to differ
systematically from ®rst measurements. Indeed, such a systematic difference would
indicate that the values were not true replicates. A plot should show whether theassumption is reasonable, and also whether the differences are independent of themean. If repeatability gets worse as the measurements increase we may need ®rst to logtransform the data in the same way as for comparing methods.
Returning to the blood pressure data shown in Table 1, we can estimate therepeatability of each method. For observer J using the sphygmomanometer the withinsubject variance is 37.408. Likewise for observer R we have s2w  37:980 and for themachine (S) we have s2w  83:141: We can see that both observers have much better
repeatability than the machine and that their performancepis
 almost identical.
Two readings by the same method will be within 1:96 2 sw or 2:77sw for 95% ofsubjects. This value is called
the repeatability coef®cient. For observer J using thep
coef®cient issphygmomanometer sw  37:408  6:116 and so the repeatabilityp
2:77  6:116  16:95 mmHg. For the machine S, sw  83:141  9:118 and therepeatability coef®cient is 2:77  9:118  25:27 mmHg. Thus, the repeatability of themachine is 50% greater than that of the observer. We can compare these 95%

150 JM Bland and DG Altman
repeatability coef®cients to the 95% limits of agreement. The 95% limits of agreement
correspond to the interval ÿ2:77sw to 2:77sw . If these are similar, then the lack ofagreement between the methods is explained by lack of repeatability. If the limits ofagreement are considerably wider than the repeatability would indicate, then there
must be some other factor lowering the agreement between methods.
The use of the within-subject standard deviation does not imply that otherapproaches to repeatability, such as intraclass correlation, are not appropriate. The
use of sw , however, facilitates the comparison with the limits of agreement. It also
helps in the interpretation of the individual measurement, being in the same units.
5 Measuring agreement using repeated measurements
When we have repeated measurements by two methods on the same subjects it isclearly desirable to use all the data when comparing methods. A sensible ®rst step is tocalculate the mean of the replicate measurements by each method on each subject. We
can then use these pairs of means to compare the two methods using the limits ofagreement method. The estimate of bias will be unaffected by the averaging, but theestimate of the standard deviation of the differences will be too small, because some ofthe effect of measurement error has been removed. We want the standard deviation ofdifferences between single measurements, not between means of several repeats. Here
we describe some methods for handling such data, ®rst for the case with equal
replication and then allowing unequal numbers of replicates.
We assume that even though multiple readings are available the standard clinicalmeasurement is a single value. Where it is customary to use the average of two or moremeasurements in clinical practice (e.g. with peak expiratory ¯ow) the approachdescribed below would not be used. Rather the limits of agreement method would beapplied directly to the means.
5.1

Equal numbers of replicates

When we make repeated measurements of the same subject by each of two methods,
the measurements by each method will be distributed about the expected measurement by that method for that subject. These means will not necessarily be the same forthe two methods. The difference between method means may vary from subject tosubject. This variability constitutes method times subject interaction. Denote themeasurements on the two methods by X and Y. We are interested in the variance ofthe difference between single measurements by each method, D  X ÿ Y. If wepartition the variance for each method we get
Var X  2t  2xI  2xw
Var Y  2t  2yI  2yw

where 2t is the variance of the true values, 2xI and 2yI are method times subject
interaction terms, and 2xw and 2yw are the within-subject variances from measure-Measuring agreement in method comparison studies

151

ments by the same method, for X and Y, respectively. It follows that the variance ofthe between-subject differences for single measurements by each method is
Var X ÿ Y  2D  2xI  2yI  2xw  2yw

5:1We wish to estimate this variance from an analysis of the means of the measurement
 X ÿ Y,
 that is from Var X
 ÿ Y.
 With this model, the use of thefor each subject, Dmean of replicates will reduce the within-subject variance but it will not affect theinteraction terms, which represent patient-speci®c differences. We thus have  2  2 
Var X
t
xI

2xw
mx

where mx is the number of observations on each subject by method X, because only thewithin-subject within-method error is being averaged. Similarly  2  2 
Var Y
t
yI

2yw
my

2yw
2xw
2
2



Var X ÿ Y  Var D  xI  Iy mx
my
 depends only on the errors and interactions, because the true
The distribution of Dvalue is included in both X and Y, which are differenced. It follows from equation (5.1)
that




1
1
2
  1ÿ
Var X ÿ Y  Var D  1ÿ
2
5:2
mx xw
my yw
If s2d is the observed variance of the differences between the within-subject means,
Var X ÿ Y  2d is estimated by




1 2
1 2
2
2
5:3
s  1ÿs^d  sd  1 ÿ
mx xw
my yw
In the common case with two replicates of each method we haves2yw
s2^2d  s2d  xw 2
2
as given by Bland and Altman.2 It is easy to see that the method still works when one
method is replicated and the other is not.
For an example, we compare observer J and machine S from Table 1. From Section
4.1 the within-subject variances of the two methods of measuring blood pressure are
37.408 for the sphygmomanometer (observer J) and 83.141 for the machine (S). The
mean difference is ÿ15:62 mmHg. The variance of the differences between-subject
means is s2d  358:493 and we have mJ  mS  3 observations by each method on eachsubject. From equation (5.3) the adjusted variance of differences is given by152 JM Bland and DG Altman^2d  358:493 






1
1
 37:408  1 ÿ
 83:141  438:859
1ÿ
33p
so sd  438:859  20:95 mmHg and the 95% limits of agreement areÿ15:62 ÿ 1:96  20:95  ÿ56:68 and ÿ15:62  1:96  20:95  25:44. This estimate isvery similar to that obtained using a single replicate (Section 2).
An approximate standard error and con®dence interval for these limits of agreement
can be found as follows. Provided the measurement errors are normal and independent, for n subjects n mx ÿ 1s2xw =2xw follows a chi-squared distribution withn mx ÿ 1 degrees of freedom, and so has variance 2n mx ÿ 1. Henceÿ 
Var s2xw 24xw
n mx ÿ 1 
Var s2yw 

and

24yw
n my ÿ 1

and the variance of the correction term of equation (5.3) is given by



 
1 2
1 2
Var 1 ÿsxw  1 ÿ
s

mx
my yw




4
1 2 24xw
1 2 2yw
 1ÿ
1ÿ
mx n mx ÿ 1my n my ÿ 1

5:4

5:5

4



2 mx ÿ 14xw 2 my ÿ 1yw

nm2x
nm2y

5:6

Similarly, the variance of s2d is given by
Var s2d  

24d

5:7

nÿ1

Applying equations (5.6) and (5.7) to equation (5.3) we have
Var ^2d  

24d

nÿ1



2 mx ÿ 14xw 2 my ÿ 14yw

nm2x
nm2y

Using the well-known results that
Var f z and hence thatp
Var z 





df z
dz

2
zE z

Var z


1 2
1
p
Var z
Var z 
4E z
2 z zE z

5:8

Measuring agreement in method comparison studies
we have for Var ^d 
24d1
2 mx ÿ 14xw 2 my ÿ 14yw
Var ^d   2
nm2x
nm2y
4d n ÿ 1!
4dmy ÿ 14yw
1
mx ÿ 14xw


 2
nm2x
nm2y
2d n ÿ 1

153

!

5:9

The variance of the mean difference 
d is estimated by ^2d =n, and mean and standard
deviation of the differences are independent. Substituting the estimates of thevariances in equation (5.9), the variance of the limits of agreement d  1:96^
d isestimated by
!
4
4
2
2
4
s
m
ÿ
1s

^
1:96
m
ÿ
1s
y

x
yw
xw
d
5:10

Var 
d  1:96^
d   d 

n
nm2x
nm2y
2^
2d n ÿ 1
For mx  my  2 this equation becomess4d^2 1:962
s4xw s4yw
Var 
d  1:96^
d   d n2^
2d n ÿ 1 4n 4n!

and for unreplicated observations with mx  my  1, ^d is replaced by the direct
estimate sd and we haves2 1:962 s4dVar d  1:96sd   d n2s2d n ÿ 1


1:962
2 1

 sd
n 2 n ÿ 1as in Section 2.2. These values can be used to estimate 95% con®dence intervals for thelimits of agreement.
For the blood pressure data, the variance of the limits of agreement is  1:96sD  
Var D



438:859
1:962
358:4932 2  37:40782 2  83:14122

85
2  438:859
84
9  859  85

 11:9941
p
Hence the standard error is 11:9941  3:463 mmHg. The 95% con®dence interval forthe lower limit of agreement is ÿ56:68 ÿ 1:96  3:463 to ÿ56:68  1:96  3:463, giving
ÿ63:5 to ÿ49:9 and for the upper limit of agreement 25:44 ÿ 1:96  3:463 to
25:44  1:96  3:463, giving 18.70 to 32.2 mmHg.
The standard error here is very similar to that found for only one replicate(3.64 mmHg) in Section 2.2. The use of replicates only reduces that part of thevariation due each method's lack of precision, and the method times subject

154 JM Bland and DG Altman
interaction component remains. If this is large, e.g. if large discrepancies for a subject
exist in all replicates (as is often the case in our experience) replication does not
improve the precision of the limits of agreement much. We still advocate two
replicates, however, so that method repeatability can be investigated.
This method is different from that in our original paper1 which ignored the subject
times method interaction. We now think that approximation was unreasonable andthat the method given here is clearly superior.
5.2

Unequal numbers of replicates

We now consider the case where there are unequal numbers of observations per
subject, mxi and myi by methods X and Y on subject i. Such data can arise, for example,
if patients are measured at regular intervals during a procedure of variable length,
such as surgery. For example, Table 4 shows measurements of cardiac output by two
methods, impedance cardiography (IC) and radionuclide ventriculography (RV), on 12
subjects.9 The solution for equally replicated data (equation (5.2)) depends on the wellknown result that the variance of the mean of n independent random variables withthe same mean and variance 2 is 2 =n. If Wi is the mean of mi observations with mean

Table 4 Cardiac output by two methods, RV and IS, for 12 subjects (data provided by Dr LS Bowling9)
Sub

RV

IC

Sub

RV

IC

1

7.83

6.57

5

3.13

3.03

1

7.42

5.62

5

2.98

2.86

1

7.89

6.90

5

2.85

1

7.12

6.57

5

1

7.88

6.35

5

2

6.16

4.06

2

7.26

2

Sub

RV

IC

9

4.48

3.17

9

4.92

3.12

2.77

9

3.97

2.96

3.17

2.46

10

4.22

4.35

3.09

2.32

10

4.65

4.62

5

3.12

2.43

10

4.74

3.16

4.29

6

5.92

5.90

10

4.44

3.53

6.71

4.26

6

6.42

5.81

10

4.50

3.53

2

6.54

4.09

6

5.92

5.70

11

6.78

7.20

3

4.75

4.71

6

6.27

5.76

11

6.07

6.09

3

5.24

5.50

7

7.13

5.09

11

6.52

7.00

3

4.86

5.08

7

6.62

4.63

11

6.42

7.10

3

4.78

5.02

7

6.58

4.61

11

6.41

7.40

3

6.05

6.01

7

6.93

5.09

11

5.76

6.80

3

5.42

5.67

8

4.54

4.72

12

5.06

4.50

4

4.21

4.14

8

4.81

4.61

12

4.72

4.20

4

3.61

4.20

8

5.11

4.36

12

4.90

3.80

4

3.72

4.61

8

5.29

4.20

12

4.80

3.80

4

3.87

4.68

8

5.39

4.36

12

4.90

4.20

4

3.92

5.04

8

5.57

4.20

12

5.10

4.50

Measuring agreement in method comparison studies

155

 and variance 2 , so having variance 2 =mi , then the expected variance of the means
will be


1 X 1
2
5:11
Var Wi  nmi
For subject i we have mxi observations by method X and myi observations by method
Y. For each subject, we calculate the differences between means of measurements by
the two methods and then calculate the variance of these differences. The expectedvalue of this variance estimate is thus
X 


1
1 X 1
2
2
  2  1



5:12
Var D
2xIyInmxi xwn
myi yw
Thus we haveX  

X  
1
1
1
12
  1ÿ
xw  1 ÿ
2yw
Var D  Var D
n
mxi
n
myi

5:13

which reduces to equation (5.2) when mxi  mx and myi  my .
For the data of Table 4, we must ®rst check the assumption that the variances are
independent of the subject means. For each method separately, we can plot the withinsubject standard deviation against the subject mean. As Figure 9 shows, theassumption of independence is reasonable for these data. Then, for each subject, weplot the difference between the means for the two methods against their average(Figure 10). Again the assumption of independence is reasonable. We next estimate
2xw and 2yw by one-way analyses of variance for RV and IC separately, giving
s2xw  0:1072 and s2yw  0:1379.
In this case




1 X 1
1 X 1
and
n
mxi
n
myi
are the same, equal to 0.2097, because the data for each subject are balanced (this need

Figure 9 Subject standard deviation against subject mean for each method of measurement of cardiac output

156 JM Bland and DG Altman

Figure 10 Subject difference, RV ÿ IC, against mean

not be the case). The variance of the mean difference between methods for eachsubject is 0.9123. To calculate the variance of the difference between single
observations by the two methods, we use equation (5.13). We have
Var IC ÿ RV  0:9123  1 ÿ 0:2097  0:1072  1 ÿ 0:2097  0:1379  1:1060
The standard deviationof differences
between single observations by the two methodsp
is estimated by ^d  1:1060  1:0517. The mean difference was 0.7092, so the 95%
limits of agreement for RV ÿ IC are 0:7092 ÿ 1:96  1:0517  ÿ1:3521 and
0:7092  1:96  1:0517  2:7705. Thus, a measurement by RV is unlikely to exceed ameasurement by IC by more than 2.77, or be more than 1.35 below.
5.3

Replicated data in pairs

The methods of Section 5.1 and Section 5.2 assume that the subject's true value does
not change between repeated measurements. Sometimes, we are interested inmeasuring the instantaneous value of a continually changing quantity. We mightmake several pairs of measurements by two methods on each subject, where theunderlying true value changes from pair to pair. We can estimate the limits ofagreement by a components of variance technique. We use the differences for eachpair of measurements. The difference for pair of measurements j on subject i may bemodelled as
Dij  B  Ii  Eij
where B is the constant bias, Ii the subjects times methods interaction term, and Xij
the random error within the subject for that pair of observations. The variance of Dij isthusMeasuring agreement in method comparison studies157

2d  2dI  2dw
We can estimate 2dI and 2dw by components of variance estimation from one way
analysis of variance.10 Suppose that for subject i we have mi pairs of observations andthere are n subjects. We have the within-subject or error mean square MSw and the
between-subjects mean square MSb . Then the components of variance can be
estimated by ^2dw  MSw andPP
mi 2 ÿ m2i
2P
MSb ÿ MSw ^dI 
n ÿ 1 mi
Theprovides ^2d . The mean bias is estimated by
Pof these estimates
P sum


mi , where di is the mean difference for subject i. Hence we estimate
mi di =the 95% limits of agreement. Methods for the calculation of con®dence intervals for
combinations of components of variance are given by Burdick and Graybill.11

6 Nonparametric approach to comparing methods
The between-method differences do not always have a normal distribution. As wenoted in Section 2, in general this will not have a great impact on the limits ofagreement. Nevertheless, if there are one or more extreme discrepancies between themethods a nonparametric approach may be felt preferable. Such a situation arguably
arises in the evaluation of (semi-) automatic blood pressure recording machines, as
was illustrated by the blood pressure data shown in Figure 1. For this reason, theBritish Hypertension Society (BHS) protocol for evaluating these machines
recommended a simple nonparametric method.12
We can retain the basic approach outlined in Section 2.1 up to and including theplot of the differences versus mean values of the two methods. There are then two
similar ways of describing such data without assuming a normal distribution ofdifferences. First, we can calculate the proportion of differences greater than somereference values (such as 10 mmHg). The reference values can be indicated on the
scatter diagram showing the difference versus the mean. Second, we can calculate thevalues outside which a certain proportion (say 10%) of the observations fell. To do thiswe simply order the observations and take the range of values remaining after apercentage (say 5%) of the sample is removed from each end. The centiles can also besuperimposed on the scatter diagram. This second method is effectively a
nonparametric form of the limits of agreement method. The two nonparametricmethods are generally less reliable than those obtained using normal distribution
theory, especially in small samples. Con®dence intervals can be constructed using thestandard method for binomial proportions or the standard error of a centile.
The BHS protocol for evaluating blood pressure measuring devices suggested theuse of the ®rst of the above ideas, using the percentage of differences within certain
limits.12 Three such assessments are made, relating to the percentage of differences158 JM Bland and DG Altman
Table 5 Grading of blood pressure devices based on
differences between measurements (in mmHg) by device andthose by sphygmomanometer12

Grade

Difference (mmHg)
5
10

A
B
C
D

60
85
50
75
40
65
fails to achieve C

15
95
90
85

Figure 11 Data from Figure 2 showing differences between methods of 15 mmHg

within 5, 10 and 15 mmHg. Table 5 shows the conditions which the data must meet toreceive a grade of A, B, or C, which were based on what could be achieved using a
sphygmomanometer.
An example is shown in Figure 11, using the same data as are shown in Figure 1.
Only the values of 15 are shown as the spread of differences was so large. For thesedata the percentages of between-methods differences within 5, 10 and 15 mmHg were16%, 35% and 49%, so that the device clearly gets a grade D.
The nonparametric method is disarmingly simple yet provides readily interpreted
results. It has been used before13;14 but apparently only rarely. Perhaps its simplicity
has led to the belief that it is not a proper analysis of the data.

Measuring agreement in method comparison studies

159

7 Discussion
Previously, we have described the limits of agreement approach1;2 and thenonparametric variant.12 In this paper we have extended the method in several ways.
We have also described a powerful method for dealing with data where the agreement
varies in a complex way across the range of the measurement and we have describedseveral approaches for replicated data.
Our approach is based on the philosophy that the key to method comparison studies
is to quantify disagreements between individual measurements. It follows that we donot see a place for methods of analysis based on hypothesis testing. Agreement is not
something which is present or absent, but something which must be quanti®ed. Nor do
we see a r^ole for methods which lead to global indices, such as correlation coef®cients.
They do not help the clinician interpret a measurement, though they have a place inthe study of associated questions such as the validity of measurement methods. Widelyused statistical approaches which we think are misleading include correlation,1;17;18;19
regression,1 and the comparison of means.1 Other methods which we think
inappropriate are structural equations15 and intraclass correlation.16
We advocate the collection of replicated data in method comparison studies, because
this enables us to compare the agreement between the two methods with theagreement each method has to itself, its repeatability (Section 4.1). Such a study
should be designed to have equal numbers of measurements by each method for eachsubject and the methods of Section 5.1 can be used for its analysis. We are rather
disappointed that so few of the studies which cite our work have adopted the use of
replicates. Other studies using replicates have usually adopted this as a convenience
because subjects are hard to ®nd, often because the measurements are very invasive.
For these studies we offer two methods of analysis, one for use when the underlying
value is assumed to remain constant and the other for when it assumed to vary.
We think that any method for analysing such studies should produce numbers
which are useful to and easily understood by the users of measurement methods. For
rapid adoption by the research community they should be applicable using existing
software. The basic method which we have proposed and the extensions to cover themost frequent situations meets both these criteria. Only the more unusual designs andrelationships between agreement and magnitude described in this paper should
require the intervention of a statistician.

Acknowledgements
We thank the researchers who have provided the data included in this paper. The
paper is drawn largely from work in progress for our forthcoming book Statistical
approaches to medical measurement, Oxford University Press.

160 JM Bland and DG Altman
References
1 Altman DG, Bland JM. Measurement inmedicine: the analysis of method comparison
studies. Statistician 1983; 32: 307±17.
2 Bland JM, Altman DG. Statistical methods forassessing agreement between two methods ofclinical measurement. Lancet 1986; i: 307±10.
3 Bland JM, Altman DG. Comparing methods ofmeasurement: why plotting difference against
standard method is misleading. Lancet 1995;
346: 1085±87
4 Eksborg S. Evaluation of method-comparison
data. Clinical Chemistry 1981; 27: 1311±12.
5 Linnet K, Bruunshuus I. HPLC withenzymatic detection as a candidate reference
method for serum creatinine. Clinical
Chemistry 1991; 37: 1669±75.
6 Altman DG. Calculating age-related reference
centiles using absolute residuals. Statistics in
Medicine 1993; 12: 917±24.
7 Marshall GN, Hays RD, Nicholas R.
Evaluating agreement between clinicalassessment methods. International Journal of
Methods in Psychiatric Research 1994; 4: 249±57.
8 Lucas A, Hudson GJ, Simpson P, Cole TJ,
Baker BA. An automated enzymic
micromethod for the measurement of fat inhuman milk. Journal of Dairy Research 1987;
54: 487±92.
9 Bowling LS, Sageman WS, O'Connor SM,
Cole R, Amundson DE. Lack of agreement
between measurement of ejection fraction by
impedance cardiography versus radionuclide
ventriculography. Critical Care Medicine 1993;
21: 1523±27
10 Searle SR, Cassela G, McCulloch CE. Variancecomponents. New York: New York, 1992.
11 Burdick RK, Graybill FA. Con®dence intervals
on variance components. New York: Dekker,
1992.
12 O'Brien E, Petrie J, Littler W, de Swiet M,
Pad®eld PL, Altman DG, Bland M, Coats A,
Atkins N. The British Hypertension Society
protocol for the evaluation of blood pressure

13

14

15
16

17

18

19

20

21

measuring devices. Journal of Hypertension
1993; 11(suppl. 2): S43±S62.
Polk BF, Rosner B, Feudo F, Vandenburgh M.
An evaluation of the Vita-Stat automatic bloodpressure measuring device. Hypertension 1980;
2: 221±27.
Latis GO, Simionato L, Ferraris G. Clinicalassessment of gestational age in the newborn
infant. Early Human Development 1981; 5:29±37.
Altman DG, Bland JM. Comparing methods ofmeasurement. Applied Statistics 1987; 36:
224±25.
Bland JM, Altman DG. A note on the use of
the intraclass correlation coef®cient in theevaluation of agreement between two methodsof measurement. Computers in Biology andMedicine 1991; 20: 337±40.
Schoolman HM, Becktel JM, Best WR,
Johnson AF. Statistics in medical research:principles versus practices. Journal ofLaboratory and Clinical Medicine 1968; 71:
357±67.
Westgard JO, Hunt MR. Use andinterpretation of common statistical tests inmethod-comparison studies. Clinical Chemistry
1973; 19: 49±57.
Feinstein AR. Clinical biostatistics. XXXVII.
Demeaned errors, con®dence games,
nonplussed minuses, inef®cient coef®cients,and other statistical disruptions of scienti®ccommunication. Clinical Pharmacology &Therapeutics 1976; 20: 617±31.
Altman DG, Bland JM. The analysis of bloodpressure data. In O'Brien E, O'Malley K eds.
Blood pressure measurement. Amsterdam:
Elsevier, 1991: 287±314.
Cotes PM, DoreÂ CJ, Liu Yin JA, Lewis SM,
Messinezy M, Pearson TC, Reid C.
Determination of serum immunoreactive
erythropoietin in the investigation oferythrocytosis. New England Journal of
Medicine 1986; 315: 283±87.