Table III. Coverage and width of empirical 95 per cent conﬁdence intervals and estimation of sampling variances of treatment effects – 0.15 treatment effect, weak treatment-selection model. Coverage of 95 per cent conﬁdence intervals True risk difference Independent Paired Lengths of 95 per cent conﬁdence intervals Independent Paired Ratio of length of independent CI to paired CI Ratio of mean estimated variance of treatment effect to variance of empirical sampling distribution Independent Paired Independent normal covariates 0 0.962 0.951 −0.02 0.953 0.942 −0.05 0.944 0.929 −0.1 0.873 0.856 −0.15 0.685 0.665 Correlated normal covariates 0 0.965 0.946 −0.02 0.956 0.939 −0.05 0.923 0.895 −0.1 0.722 0.676 −0.15 0.311 0.265 Mixture covariate scenario 1 0 0.959 0.95 −0.02 0.956 0.944 −0.05 0.953 0.944 −0.1 0.91 0.896 −0.15 0.744 0.723 Mixture covariate scenario 2 0 0.957 0.948 −0.02 0.952 0.944 −0.05 0.95 0.942 −0.1 0.925 0.919 −0.15 0.883 0.877 Independent Bernoulli covariates 0 0.949 0.946 −0.02 0.955 0.949 −0.05 0.952 0.95 −0.1 0.953 0.951 −0.15 0.946 0.945 0.047 0.047 0.045 0.043 0.04 0.053 0.053 0.051 0.049 0.047 0.047 0.046 0.045 0.042 0.039 0.045 0.044 0.042 0.04 0.036 0.042 0.041 0.04 0.036 0.032 0.045 0.044 0.043 0.041 0.039 0.049 0.049 0.048 0.046 0.044 0.045 0.044 0.043 0.041 0.038 0.043 0.042 0.041 0.039 0.036 0.042 0.041 0.039 0.036 0.032 1.044 1.068 1.047 1.049 1.026 1.082 1.082 1.063 1.065 1.068 1.044 1.045 1.047 1.024 1.026 1.047 1.048 1.024 1.026 1 1 1 1.026 1 1 1.113 1.097 1.07 1.074 1.073 1.193 1.207 1.192 1.153 1.138 1.105 1.106 1.126 1.099 1.058 1.039 1.027 0.994 0.955 0.95 1.038 1.03 1.031 0.983 0.972 P. C. AUSTIN 1.005 0.993 0.974 0.99 1.009 1.019 1.034 1.027 1.005 1.01 1.004 1.008 1.03 1.018 0.998 0.968 0.959 0.932 0.906 0.917 1.002 0.996 1.001 0.961 0.962 1 2 9 8 risk difference by at least 14.9 per cent. In 25 per cent of the scenarios, these methods overestimated the sampling variance of the estimated risk difference by at least 25.0 per cent. Furthermore, the estimated standard error overestimated the empirical standard deviation of the sampling distribution to a greater extent when there was a strong treatment-selection process compared with when there was a weak treatment-selection process. When methods for paired samples were used, the ratio ranged from a low of 0.906 to a high of 1.071; the median ratio was 1.003 (25th and 75th percentiles: 0.985 and 1.023, respectively) across the 100 scenarios. 4. Discussion We compared statistical inference when methods for independent samples were used compared with when methods for paired samples were used for signiﬁcance testing and for variance estimation when estimating risk differences in propensity-score matched samples. We found that compared with using methods for independent samples, the use of methods for paired samples resulted in: (i) empirical type I error rates that were closer to the advertised rate; (ii) empirical coverage rates of 95 per cent conﬁdence intervals that were closer to the advertised rate; (iii) narrower 95 per cent conﬁdence intervals; and (iv) estimated standard errors that were more closely reﬂected the sampling variability of the estimated risk difference. As noted in the Introduction, applied researchers using propensity-score matching frequently use statistical methods for independent samples when estimating the statistical signiﬁcance of estimated Copyright © 2011 John Wiley & Sons, Ltd. Statist. Med. 2011, 30 1292--1301