Table I. Empirical type I error rates for statistical methods for paired vs independent samples. Covariate scenario Mean percent of treated subjects matched Pearson chi-squared test (independent sample method) McNemar’s test (paired sample method) 0.29 outcome probability, weak treatment-selection model Independent normal 90.4 Correlated normal 75.2 Mixture scenario 1 91.4 Mixture scenario 2 96.1 Independent Bernoulli 99.7 0.15 outcome probability, weak treatment-selection model Independent normal 90.4 Correlated normal 75.2 Mixture scenario 1 91.4 Mixture scenario 2 96.1 Independent Bernoulli 99.7 0.29 outcome probability, strong treatment-selection model Independent normal 93.5 Correlated normal 78.6 Mixture scenario 1 95.2 Mixture scenario 2 97.7 Independent Bernoulli 99 0.15 outcome probability, strong treatment-selection model Independent normal 93.5 Correlated normal 78.6 Mixture scenario 1 95.2 Mixture scenario 2 97.7 Independent Bernoulli 99 0.0427 0.0351 0.0296 0.0422 0.0416 0.0378 0.0345 0.0405 0.0427 0.051 0.0208 0.0208 0.029 0.0334 0.0395 0.0258 0.0214 0.0323 0.0312 0.0384 P. C. AUSTIN 0.0614 0.0537 0.0521 0.0499 0.0477 0.0493 0.0537 0.0499 0.0515 0.0537 0.0405 0.0526 0.0526 0.051 0.0532 0.0521 0.0466 0.0477 0.0433 0.0504 1 2 9 6 Note: Cells contain results averaged over 1825 Monte Carlo simulations. 20 scenarios, the empirical type I error rate exceeded 0.06 (the empirical type I error rate was 0.0614 in this scenario). Thus, in the majority of covariate scenarios, the use of a method for independent samples (i.e. the Chi-squared test) resulted in a type I error rate that was statistically signiﬁcantly different from the advertised rejection rate. When the results were stratiﬁed by the strength of the treatment- selection process, when methods for independent samples were used, the empirical type I error rate was statistically signiﬁcantly different from 0.05 in 40 per cent of the 10 scenarios when there was a weak treatment-selection process; however, the empirical type I error rate was statistically signiﬁcantly different from 0.05 in 100 per cent of the scenarios when there was a strong treatment-selection process. The empirical coverage rates of 95 per cent conﬁdence intervals, the mean length of 95 per cent conﬁdence intervals, and the ratio of the mean length of estimated 95 per cent conﬁdence intervals obtained using methods for independent samples to the mean length of estimated 95 per cent conﬁdence intervals obtained using methods for paired samples are reported in Tables II–V for the four different sets of ﬁve covariate scenarios. In 71 of the 100 scenarios, the empirical coverage rates of 95 per cent conﬁdence intervals obtained using a method for independent samples were statistically signiﬁcantly different from 0.95 (i.e. empirical coverage rates exceeded 0.96 or were less than 0.94). The median empirical coverage rate was 0.964 (25th and 75th percentiles: 0.955 and 0.971) across the 100 scenarios. Thus, in over half of the 100 scenarios, the empirical coverage rates were signiﬁcantly different from the advertised rate of 0.95. In contrast, in 15 of the 100 scenarios, the empirical coverage rates of 95 per cent conﬁdence intervals obtained using methods for paired samples were statistically signiﬁcantly different from 0.95. The median empirical coverage rate was 0.949 (25th and 75th percentiles: 0.944 and 0.951). As above, we examined the results for the independent method of analysis separately in scenarios with a weak treatment selection-process and in scenarios with a strong treatment selection- process. In 24 (48 per cent) of the 50 scenarios with a weak treatment-selection process, methods for independent samples resulted in 95 per cent conﬁdence intervals whose empirical coverage rates were not statistically signiﬁcantly different from 0.95. However, in only 5 (10 per cent) of the 50 scenarios with a strong treatment-selection process, did methods for independent samples result in 95 per cent conﬁdence intervals whose empirical coverage rates were not statistically signiﬁcantly different from 0.95. Copyright © 2011 John Wiley & Sons, Ltd. Statist. Med. 2011, 30 1292--1301