Statistical Tests of Agreement 

Based on Non-Standard Data 

Elizabeth Stanwyck 

Bimal Sinha 

Department of Mathematics and Statistics 

University of Maryland, Baltimore County 

Barry Nussbaum 

Office of Environmental Information 

U.S. Environmental Protection Agency 
Proving equivalence is increasingly important 

• Testing is expensive & time consuming 

• Newer methods and procedures are being 

developed 

• Common goal: assess agreement between two 

methods of measurement 

FCSM Research Conference November 6, 2013 2 
Applications to EPA 

problems 

• Demonstrating equivalence between primary and 

secondary methods for measuring formaldehyde 

emissions from composite wood products 

o Large chamber test is expensive (single measurement) 

o Small chamber test is easier and less costly (multiple measurements) 

• Prediction of Dioxin-Furan Congener (TEQ) toxicity in

fresh-water fish based on fatty acid methyl ester 

(FAME) profiles 

o Equivalence between KVL and NERL labs for FAME 

o Equivalence between KVL & ECL labs for TEQ 

FCSM Research Conference November 6, 2013 3 
Common methods for assessing agreement 

• Hypothesis testing of the correlation coefficient 

• Regression analysis 

• Paired t-tests 

• Least-squares analysis for intercept and slope 

• Within-subject coefficient of variation 

FCSM Research Conference November 6, 2013 4 
Mean, variance, covariance approach 

• Some current tests are based only on the mean and 

standard deviation of the differences: 

• Does not guarantee equivalence!! 

• Even high correlation, by itself, does not guarantee 

agreement! 

FCSM Research Conference November 6, 2013 5 
Assessing agreement 

• Likelihood ratio test for combined hypothesis: 

(Yimprayoon et al., 2006) 

• Interval hypothesis test 

o Extremely difficult and complicated test 

• Equivalence is not the same as equality! 

FCSM Research Conference November 6, 2013 6 
Nonstandard data problem 

• Inference usually based on paired data X and Y

(bivariate normal assumption) 

• Yinprayoon, Tiensuwan, and Sinha, 2006 

• Generalize the LRT approach for nonstandard data 

o Balanced case: 

o Unbalanced case: 

FCSM Research Conference November 6, 2013 7 
Restricted dataset 

• Likelihood function is based on marginal likelihood 

of X and conditional likelihood of Y 

FCSM Research Conference November 6, 2013 8 
Likelihood function 

FCSM Research Conference November 6, 2013 9 
Unrestricted maximization 

• Maximum likelihood estimates 

• Maximized likelihood 

FCSM Research Conference November 6, 2013 10
Restricted maximization 

• Maximum likelihood estimates 

1^1⁄2 =

nx1(1 + 1⁄2) + M(y11 ¡ 1⁄2x11)

M(1 ¡ 1⁄2) + n(1 + 1⁄2)

2n3⁄4^

2

1⁄2

= Q1(1⁄2) = A +

D + C1⁄2

2 ¡ 2E1⁄2

1 ¡ 1⁄2

2

+

nM [y11 ¡ x1 + 1⁄2 (x1 ¡ x11)]2

(1 ¡ 1⁄2)[M (1 ¡ 1⁄2) + n (1 ¡ 1⁄2)]

• Likelihood function, maximized wrt μ and σ2 

• To maximize the likelihood, minimize wrt ρ

U1 (1⁄2) = h¡1 ¡ 1⁄2

2

¢

1

2 £ Q1 (1⁄2)

i

FCSM Research Conference November 6, 2013 11
Images of U1 

FCSM Research Conference November 6, 2013 12
Likelihood ratio test statistic 

• Test statistic 

• Reject H0 for large values of T1 

• Select cutoff d1 so that 

FCSM Research Conference November 6, 2013 13
Remarks 

• T1 is location and scale invariant 

• Composite null hypothesis: determine the cutoff 

value d1 under ρ = ρ0 and verify size is less than or 

equal to αlpha for ρ > ρ0 

• Simulations: different correlation, means, variances, 

and combinations thereof to get an idea of power 

FCSM Research Conference November 6, 2013 14
Unrestricted dataset 

• Likelihood function: 

FCSM Research Conference November 6, 2013 15
Unrestricted maximization 

• Maximum likelihood estimates 

• Maximized likelihood 

1

A

n

2 £ [D ¡

E2

C + Wy]

M

2

FCSM Research Conference November 6, 2013 16
Restricted maximization 

• Maximum likelihood estimates 

3⁄4^

2

1⁄2

=

1

n + M

Q2 (1⁄2)

• Likelihood maximized wrt μ and σ2 

• To maximize likelihood, minimize 

U2 (1⁄2) = h¡

1 ¡ 1⁄2

2

¢

£ Q2 (1⁄2)

1+ n

M

i

FCSM Research Conference November 6, 2013 17
Likelihood ratio test statistic 

• Test statistic 

• Reject H0 for large values of T2 

• Select cutoff d2 so that 

FCSM Research Conference November 6, 2013 18
Restricted dataset 

Simulations: Type I Error rates 

0.92 0.9 5 1 0.0439 

0.92 0.9 10 1 0.0396 

0.92 0.9 15 1 0.0371 

0.92 0.9 5 3 0.0452 

0.92 0.9 10 3 0.0409 

0.92 0.9 15 3 0.0335 

0.95 0.9 5 1 0.033 

0.95 0.9 10 1 0.0299 

0.95 0.9 15 1 0.0274 

0.95 0.9 5 3 0.0374 

0.95 0.9 10 3 0.0305 

0.95 0.9 15 3 0.0237 

0.99 0.9 5 1 0.0299 

0.99 0.9 10 1 0.0254 

0.99 0.9 15 1 0.0253 

0.99 0.9 5 3 0.0309 

0.99 0.9 10 3 0.0277 

0.99 0.9 15 3 0.0266 

FCSM Research Conference November 6, 2013 19
Type I Error rates 

FCSM Research Conference November 6, 2013 20
Simulations: Power 

0.5 0.9 0 1 5 1 0.05 0.2458 

0.5 0.9 0 1 10 1 0.05 0.642 

0.5 0.9 0 1 15 1 0.05 0.8527 

0.5 0.9 0 1 5 3 0.05 0.4265 

0.5 0.9 0 1 10 3 0.05 0.8875 

0.5 0.9 0 1 15 3 0.05 0.9723 

0.9 0.9 1 1 5 1 0.05 0.8815 

0.9 0.9 1 1 10 1 0.05 0.9999 

0.9 0.9 1 1 15 1 0.05 1 

0.9 0.9 1 1 5 3 0.05 0.9996 

0.9 0.9 1 1 10 3 0.05 1 

0.9 0.9 1 1 15 3 0.05 1 

FCSM Research Conference November 6, 2013 21
Simulations: Power 

0.9 0.9 0 4 5 1 0.05 0.5481 

0.9 0.9 0 4 10 1 0.05 0.961 

0.9 0.9 0 4 15 1 0.05 0.9984 

0.9 0.9 0 4 5 3 0.05 0.9096 

0.9 0.9 0 4 10 3 0.05 0.9996 

0.9 0.9 0 4 15 3 0.05 1 

0.9 0.9 1 4 5 1 0.05 0.8197 

0.9 0.9 1 4 10 1 0.05 0.9976 

0.9 0.9 1 4 15 1 0.05 1 

0.9 0.9 1 4 5 3 0.05 0.9885 

0.9 0.9 1 4 10 3 0.05 1 

0.9 0.9 1 4 15 3 0.05 1 

FCSM Research Conference November 6, 2013 22
Simulations: Power 

0.5 0.9 1 1 5 1 0.05 0.6795 

0.5 0.9 1 1 10 1 0.05 0.9836 

0.5 0.9 1 1 15 1 0.05 0.9988 

0.5 0.9 1 1 5 3 0.05 0.9515 

0.5 0.9 1 1 10 3 0.05 1 

0.5 0.9 1 1 15 3 0.05 1 

0.5 0.9 0 4 5 1 0.05 0.5043 

0.5 0.9 0 4 10 1 0.05 0.9442 

0.5 0.9 0 4 15 1 0.05 0.9955 

0.5 0.9 0 4 5 3 0.05 0.5077 

0.5 0.9 0 4 10 3 0.05 0.9486 

0.5 0.9 0 4 15 3 0.05 0.9888 

FCSM Research Conference November 6, 2013 23
Simulations 

0.5 0.9 1 4 5 1 0.05 0.6653 

0.5 0.9 1 4 10 1 0.05 0.9862 

0.5 0.9 1 4 15 1 0.05 0.9995 

0.5 0.9 1 4 5 3 0.05 0.8536 

0.5 0.9 1 4 10 3 0.05 0.9978 

0.5 0.9 1 4 15 3 0.05 0.9998 

• Test is most powerful when means are different 

• Least powerful when only variances are different

 

FCSM Research Conference November 6, 2013 24
Tests based on combinations of P-values 

• Consider the composite hypothesis test 

• We consider three separate tests for H01, H02, and 

H03, and combine the resulting P-values to derive an 

overall test. 

FCSM Research Conference November 6, 2013 25
Testing H01 

• Paired t-test: 

o Assumption: 

• Reject the null for large values of jtdj =

 ̄

 ̄

 ̄

 ̄

p

n 1d

sd

 ̄

 ̄

 ̄

 ̄

di = xi ¡ y1i;

1d =

Pn

i=1 di

n

;s2

d =

Pn

i=1(di ¡ 1d)

2

n ¡ 1

• P-value 

FCSM Research Conference November 6, 2013 26
Testing H02 

• Modified Pittman-Morgan 

• P-value 

FCSM Research Conference November 6, 2013 27
Testing H03 

o assume 

• P-value 

FCSM Research Conference November 6, 2013 28
Tests based on P-values 

1. Tippett’s test: 

2. Fisher’s test:

3. Stouffer’s test: 

FCSM Research Conference November 6, 2013 29
Tests based on P-values 

Simulations: Type I Error rates 

Tippett Fisher Stouffer

0.92 0.9 5 1 0.0498 0.0481 0.0358 

0.92 0.9 10 1 0.0468 0.0439 0.0327 

0.92 0.9 15 1 0.0409 0.0343 0.0248 

0.92 0.9 5 3 0.0484 0.0448 0.0349 

0.92 0.9 10 3 0.0416 0.0354 0.0271 

0.92 0.9 15 3 0.0412 0.0402 0.0271 

0.95 0.9 5 1 0.0457 0.0402 0.0183 

0.95 0.9 10 1 0.0388 0.0314 0.0092 

0.95 0.9 15 1 0.039 0.025 0.0053 

0.95 0.9 5 3 0.0474 0.0442 0.0172 

0.95 0.9 10 3 0.0473 0.0421 0.0116 

0.95 0.9 15 3 0.0551 0.0427 0.0088 

0.99 0.9 5 1 0.0399 0.0309 0.0007 

0.99 0.9 10 1 0.0386 0.0262 0 

0.99 0.9 15 1 0.0388 0.023 0 

0.99 0.9 5 3 0.1112 0.1067 0.0018 

0.99 0.9 10 3 0.3148 0.2344 0.0001 

0.99 0.9 15 3 0.5378 0.4211 0 

FCSM Research Conference November 6, 2013 30
Simulations: Power 

0.5 0.9 0 1 5 1 0.2151 0.2762 0.3224 

0.5 0.9 0 1 10 1 0.6453 0.6981 0.5593 

0.5 0.9 0 1 15 1 0.8661 0.8714 0.6836 

0.5 0.9 0 1 5 3 0.2984 0.3835 0.4372 

0.5 0.9 0 1 10 3 0.8323 0.8956 0.7832 

0.5 0.9 0 1 15 3 0.9764 0.9898 0.9391 

0.9 0.9 1 1 5 1 0.8507 0.8843 0.6941 

0.9 0.9 1 1 10 1 0.9998 0.9998 0.9243 

0.9 0.9 1 1 15 1 1 1 0.9796 

0.9 0.9 1 1 5 3 0.9981 0.9984 0.8461 

0.9 0.9 1 1 10 3 1 1 0.9781 

0.9 0.9 1 1 15 3 1 1 0.9987 

FCSM Research Conference November 6, 2013 31
Simulations: Power 

0.9 0.9 0 4 5 1 0.403 0.4249 0.3596 

0.9 0.9 0 4 10 1 0.9154 0.9615 0.754 

0.9 0.9 0 4 15 1 0.994 0.9984 0.9189 

0.9 0.9 0 4 5 3 0.6942 0.7543 0.5457 

0.9 0.9 0 4 10 3 0.9971 0.9994 0.916 

0.9 0.9 0 4 15 3 1 1 0.9925 

0.9 0.9 1 4 5 1 0.5252 0.7668 0.7903 

0.9 0.9 1 4 10 1 0.9759 0.9979 0.9904 

0.9 0.9 1 4 15 1 0.9991 0.9999 0.9993 

0.9 0.9 1 4 5 3 0.823 0.9734 0.9505 

0.9 0.9 1 4 10 3 1 1 0.9994 

0.9 0.9 1 4 15 3 1 1 1 

FCSM Research Conference November 6, 2013 32
Simulations: Power 

0.5 0.9 1 1 5 1 0.4099 0.6822 0.7232 

0.5 0.9 1 1 10 1 0.9163 0.9835 0.9622 

0.5 0.9 1 1 15 1 0.9957 0.9997 0.9963 

0.5 0.9 1 1 5 3 0.6486 0.9415 0.9381 

0.5 0.9 1 1 10 3 0.995 0.9999 0.9993 

0.5 0.9 1 1 15 3 1 1 1 

0.5 0.9 0 4 5 1 0.3051 0.5042 0.5782 

0.5 0.9 0 4 10 1 0.8448 0.9602 0.9209 

0.5 0.9 0 4 15 1 0.9982 0.9969 0.9846 

0.5 0.9 0 4 5 3 0.3223 0.458 0.5203 

0.5 0.9 0 4 10 3 0.8789 0.9489 0.8779 

0.5 0.9 0 4 15 3 0.9886 0.9962 0.9783 

FCSM Research Conference November 6, 2013 33
Simulations: Power 

0.5 0.9 1 4 5 1 0.3575 0.6788 0.7638 

0.5 0.9 1 4 10 1 0.8987 0.9887 0.979 

0.5 0.9 1 4 15 1 0.9929 0.9995 0.9984 

0.5 0.9 1 4 5 3 0.5109 0.852 0.8831 

0.5 0.9 1 4 10 3 0.9796 0.9987 0.9964 

0.5 0.9 1 4 15 3 0.9999 1 0.9998 

• Stouffer’s test has the lowest Type I Error rates (of all tests, 

including LRT) 

• LRT and Fisher’s tests have similar power

o Fisher’s test has the highest power of the combined P-value tests in almost 

every case 

o Stouffer’s has a higher power in some small sample size (n=5) cases

FCSM Research Conference November 6, 2013 34
Applications 

• Application to EPA data: measuring concentrations 

of pollutants in groundwater 

o Conventional purging methods i.e. low-flow sampling methods 

• A pump slowly collects groundwater so that the sample is not 

contaminated by water at different levels 

o New HydraSleeve method 

• A tube is lowered into the well and left there long enough for 

sediment etc. to settle, then water is collected as the tube is 

pulled upwards 

• Focus: specific pollutants 

FCSM Research Conference November 6, 2013 35
Results 

• TCE 

Test Cutoff Test Statistic Conclusion 

LRT 2.37547 2.206056 Do not reject 

Tippett 0.01803122 0.2217555 Do not reject 

Fisher 11.74769 5.849823 Do not reject 

Souffer -2.473122 0.4399887 Do not reject 

FCSM Research Conference November 6, 2013 36
Results 

• DCA 

Test Cutoff Test Statistic Conclusion 

LRT 2.462177 3.641468 Reject 

Tippett 0.01858661 0.0007817254 Reject 

Fisher 11.65932 20.72726 Reject 

Souffer -2.418705 -4.703667 Reject 

FCSM Research Conference November 6, 2013 37
Strong resemblance to 

bioequivalence testing 

• In an equivalence trial, the aim is to show that two 

treatments are not too different in characteristics 

• Not too different is defined in a clinical manner 

• Called bioequivalence testing 

• Nature of the data for bioequivalence testing 

o Same patients 

o Washout period 

o Crossover designs 

FCSM Research Conference November 6, 2013 38
Bioequivalence testing 

• Often data are collected from healthy volunteers 

• If two drug products perform the same in healthy 

volunteers, the assumption is made that they will 

perform the same in patients with the disease 

• Data obtained on three patient characteristics 

o Area under the curve (AUC) 

o Maximum blood concentration Cmax 

o Time to reach the maximum concentration Tmax 

FCSM Research Conference November 6, 2013 39
Bioequivalence testing 

• Two drug products are bioequivalent if they have 

similar rate and extent of absorption into the blood. 

• Two drug products are therapeutically equivalent if 

they provide similar therapeutic effects. 

• Fundamental bioequivalence assumption: If two 

drug products are bioequivalent, they are also 

therapeutically equivalent 

FCSM Research Conference November 6, 2013 40
Data for 

bioequivalence testing 

FCSM Research Conference November 6, 2013 41
Experimental designs 

• Reference drug (R) 

• Test drug (T) 

• Each subject receives both R and T, separated by a 

washout period 

• Crossover designs are used 

• A two sequence−two period crossover design:

FCSM Research Conference November 6, 2013 42
Average bioequivalence 

• Let μT, μR: average responses among the population 

of patients who will take the test drug, and the 

reference drug, respectively. 

• The response is usually AUC, after log-transformation

(could be Cmax or Tmax).

• Average bioequivalence holds if μT and μR are 

equivalent, i.e., they are “close“

FCSM Research Conference November 6, 2013 43
Average bioequivalence 

• μT and μR are considered equivalent if 

|μT − μR| < ln(1.25).

• Hypothesis to be tested: 

• Conclude average bioequivalence if H0 is rejected 

after a statistical test based on the log-transformed 

AUC data. 

FCSM Research Conference November 6, 2013 44
A canonical form 

• Under an appropriate model for the log-transformed 

data, a canonical form is

• Rewrite as 

• Average bioequivalence is concluded if both H01

and H02 are rejected. 

FCSM Research Conference November 6, 2013 45
Assessing bioequivalence 

• Carry out t-tests: conclude average bioequivalence at 

significance level α if

• Equivalently, if

• Two one-sided t-test (TOST) 

o Schuirmann (1981), Biometrics 

o Schuirmann (1987), Journal of Pharmacokinetics and Biopharmaceutics

• Main drawback: not scale invariant 

o Performance depends on unknown σ

FCSM Research Conference November 6, 2013 46
Type I Error rate: TOST 

FCSM Research Conference November 6, 2013 47
Improvements on TOST 

• The TOST can be quite conservative as σ gets large 

• Improved tests due to: 

o Anderson and Hauck (1983), Communications in Statistics 

o Munk (1993), Biometrics 

o Berger and Hsu (1996), Statistical Science 

o Brown, Hwang and Munk (1997), Annals of Statistics 

o Munk, Brown and Hwang (2000), Biometrical Journal 

o Cao and Mathew (2008), Biometrical Journal 

• Improvement in power at values of σ that are 

unlikely. 

FCSM Research Conference November 6, 2013 48
Criterion for equivalence 

• If the probability that Y/X is around 1 is large, conclude 

that the standard device and the alternative device are 

equivalent. 

• Let 

 for small δ. 

• If θ is large, conclude that the standard device and the

alternative device are equivalent.

FCSM Research Conference November 6, 2013 49
Criterion for equivalence 

• A usual choice is δ= 0.25 

• Use the data to test 

• Accept equivalence if H0 is rejected, i.e., if θ ≥ 0.90 

is concluded. 

FCSM Research Conference November 6, 2013 50
References 

Casella, George; Roger L. Berger. Statistical Inference. 2nd Edition. : Duxbury Press, California 2001. 

Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 

37-46. 

Cohen, J. (1968). Weighted kappa: nominal scale agreement with provision for scaled disagreement or 

partial credit. Psychological Bulletin, 70, 213-220. 

Lin, L.I.K. (1989) A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45, 255-268. 

Lin, L.I.K. (2000). Total deviation index for measuring individual agreement with application in laboratory 

performance and bioequivalence. Statistics in Medicine, 19, 255-270. 

Lin, L.I.K., Hedayat, A.S., Sinha, Bikas, and Yang, M. (2002). Statistical methods in assessing agreement: 

models, issues, and tools. Journal of American Statistical Association, 97,257-270.

Yinprayoon, P., Tiensuwan, M. and Sinha, Bimal (2006). Some statistical aspects of assessing agreement: 

theory and applications. Festschrift for Tarmo Pukkila on his 60th Birthday. Edited by Liski, Isotalo, Niemela, 

Puntanen, Styan. Department of Mathematics, Statistics, and Philosophy, University of Tampere, 327-346. 

FCSM Research Conference November 6, 2013 51
 

Bioequivalence references 

Anderson ,S. and Hauck, W.W. (1983). A new procedure for testing equivalence in comparative bioavailability 

and other clinical trials. Communications in Statistics: Theory and Methods 12: 2663 – 92

Berger, R.L.; Hsu, J.C. (1996), Bioequivalence trials, intersection-union tests and equivalence confidence sets. 

Statistical Science 11(4): 283-319 (with discussion).

Brown, L.D., Hwang, J. and Munk, A. (1997), An unbiased test for the bioequivalence problem. Annals of 

Statistics 25, 2345-2367

Cao, L. and Mathew, T. (2008), A Simple Numerical Approach Towards Improving the Two One-Sided Test for 

Average Bioequivalence. Biometrical Journal, 50: 205–211 

Munk, A. (1993),An Improvement on Commonly Used Tests in bioequivalence Assessment. Biometrics, 49(4): 

1225-1230 

Munk, A.; Hwang, J.T.; Brown, L. (2000), Testing Average Equivalence – Finding a Compromise between 

Theory and Practice. Biometrical Journal, 42(5): 531-552 

Schuirmann, D. L. (1981), On hypothesis testing to determine if the mean of a normal distribution is contained 

in a known interval. Biometrics 37(617): 137. 

Schuirmann, D.L. (1987), A comparison of the Two One-Sided Tests Procedure and the Power Approach for 

assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics 15(6): 657-

680. 

FCSM Research Conference November 6, 2013 52