Measuring response errors in censuses and surveys through reinterview 

data having a different measurement error variance 

Piero Demetrio Falorsi, Alessandro Pallara 

National Institute of Statistics (ISTAT), Via Magenta 2, 00184, Rome, Italy 

e-mail: falorsi@istat.it; pallara@istat.it 

Aldo Russo 

Department of Political Institutions and Social Science, Università Roma Tre, Rome, Italy 

Keywords: Measurement errors, Reinterview, Response bias, Simple and correlated measurement variance. 

1. Introduction 

Reinterviews have been extensively used as a tool for estimating and reducing response errors in censuses and sample 

surveys. By response error it is meant any error occurring at the data collection stage for a variety of reasons. Errors may be 

due to the respondent, to the interviewer, or to both. Such errors can never be completely avoided and, therefore, in the 

practice of censuses and large surveys some different techniques have been proposed (Forsman and Schreiner, 1991) in order 

to measure response error components. One such method relies on replicated measurements obtained through reinterview of 

a subsample of units from the original survey on a set of questions from the original interview. When the responses given by 

the same units during the reinterview differ from those given in the original interview the differences can be evaluated 

through reconciliation, normally by asking the respondent to determine what is the correct information between the two 

interviews. 

When using the approach based on reinterviews to measure response error components, it is usually assumed (Hansen et al., 

1961; Särndal et al., 1992; pp. 614-616) that: (i) the two measures are modelled as random variables; (ii) the repeated 

measurements on the same unit are independent; (iii) the measurement conditions for the two occasions are identical or as 

close to identical as possible; this imply that the random variables which are associated to the two measurements are subject 

to the same measurement error variance. Under these assumptions, the standard response error model based on reinterview 

data, developed at the U.S. Bureau of Census (Hansen et al., 1964), yields unbiased estimates of simple and correlated 

measurement variances (Särndal et al., 1992; pp. 614-616). Furthermore, an estimate of the measurement bias can be 

obtained, assuming that the reconciled reinterview process yields the true value. While theoretically sound, the Bureau of 

Census survey error model seems to be not realistic in many practical situations, because some of the above assumptions are 

difficult to be met. Frequently, e.g., the reinterview program, because of budget and operational restrictions, are carried out 

using a different data collection technique with respect to the original interview. In this case, the two measurements, 

although still independent, cannot be considered as obtained under identical measurement conditions and, therefore, they can 

be deemed as having different measurement error variances. 

In this paper it is shown how to obtain an unbiased estimate of the response variance when the assumption of identical 

second-order moment of the two measurements fails to hold. Under the standard response error model the variance of the 

standard estimator of the population total of the true values is decomposed in the sum of the sampling variance and 

measurement variance and the measurement variance can be split in two terms, both depending from the (unknown) 

variances and covariance between measurements, but only one depending also from the sampling design. Indeed, in presence 

of measurement errors the estimator of the total variance is biased and it can be shown (Koch et al.; 1975) that the bias is 

equal to the component of the measurement variance not depending on the sampling design. A method of moments 

estimation of the simple and correlated measurement variance components under the standard response error model is then 

proposed, assuming different error variances of the repeated measurements in different surveys. 

The paper is organised as follows: in section 2 the parameter of interest and the statistical model for the response error are 

shortly introduced and then estimation error variance is decomposed in a sampling and a non-sampling component; in 

section 3 it is shown how to obtain an unbiased estimation of total error variance in presence of measurement errors. Finally, 

some empirical results are presented in section 4, concerning estimates of measurement variance components for some 

variables pertaining to land usage such as measured during the 5th Italian Census of Agriculture, and whose accuracy has 

been evaluated through a reinterview survey. In this case, while the original census responses have been obtained through 

face-to-face interview, the reinterview survey was based on Computer Assisted Telephone Interviewing (CATI): hence, the 

Page 30 
 

 

 

 

 

 

 

 

 

 

 

 

 

two survey measures have clearly not been collected under the same conditions, which may result in different measurement 

error variances. 

2. The measurement error model 

Consider the following conditions: (i) an original random sample s (survey I1) of size n is drawn from the population P, 

constituted by N units, with a sampling design p (

.

), with π k

 representing the probability that element k will be included in 

the sample and π kl representing the probability that both of the elements k and l will be included in the sample; (ii) for each 

element k ∈ s , yk ,1 denotes the observed value of the variable of interest y; (iii) a second phase sample r (r ⊆ s) (survey I 2

) 

of size nr is drawn according to the design p(

.

|s), such that p(r|s) is the conditional probability of choosing r. The inclusion 

probabilities under this design are denoted π and for elements k and l ∈ s; (iv) for each element k ∈r ,

k s π kl s

yk ,2 denotes the observed value of the variable y of interest. The measurement model m is specified as follows: 

yk ,t = μ k + ε k ,t ( t =1,2) (1) 

where μ k

 is the true value and ε k ,t is the erratic component. 

Given the general survey conditions of I1 and of I2 , the expectation values E (

.

m ) under the model (1) are: 

Em (yk ,t ) = μ k for k ∈s if t=1 or k ∈r ⊆ s if t=2 

2 2 Em (yk ,t − μ k ) = σ k ,t for k ∈s if t=1 or k ∈r ⊆ s if t=2 

Em ((yk ,t − μk ) (yl,t − μl )) = σ kl,t for k,l ∈s (with k ≠ l ) if t=1 or k,l ∈r ⊆ s (with k ≠ l ) if t=2 

Em ((yk ,1 − μk ) (yl,2 − μl )) = 0 for k,l ∈r ⊆ s (with k = l or k ≠ l ). 

2

It is important to note that in the above model both the values μ k

 and σ k ,t , are related to the specific unit k∈P, while σ kl,t 

is a value related to the specific couple (k,l) ∈P, independently by the selected sample in the It (t =1 or 2) survey. 

The objective of the survey I1 is to estimate the population total of the true values: t y =∑ μ k , where we denote with 

P 

the sum over the N units of the population P. An estimate of t may be obtained by the Horvitz-Thompson estimator ∑ y

P 

~ yk ,1

ty =∑ . (2) 

s π k 

~

As a measure of the accuracy of the estimator (2), we use the variance of ty defined with respect to design p(

.

) and model m 

jointly 

~ ~ V ( ) t = E ((t − t )

2 ) (3) pm y pm y y 

where the indexes p and m define the operators E and V with respect to the sample design and the model. 

~ The variance term Vpm (ty ) can be decomposed as follows 

~ ~ ~ V s)). pm (ty ) = Ep (Vm (ty s))+V (E (t p m y 

~ The measurement variance, given by E (V (t s)) = V +V , may be represented as a sum of two terms p m y A1 A2 

VA1 =VA1,1 +VA1,2 =∑ σ k 

2 

,1 +∑∑ P σ kl,1 (4) 

P k l ≠k 

(1−π ) 2 (π −π π ) k kl k l VA 2 = σ k ,1 + σ (5) ∑ kl,1 π ∑∑ P P k π πk l k l ≠k 

where VA1

, that denotes the measurement variance under complete enumeration, is the sum of the two components 

2 VA1,1 =∑ σ k ,1 , indicating the simple measurement variance and VA1,2 = ∑∑ σ , denoting the correlated P kl,1

P k l ≠k 

measurement variance. Usually in survey practice the main component in (4) is represented by the VA1,2 term, while the 

simple response variance is typically negligible. 

Page 31 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

~ The component V (E (ty s)) =V , denoting the sampling variance, is expressed by p m B 

~ π kl k l −π π

V (Em p y B (t s)) = V = μ k μl ∑∑ .

P π πk l k l 

3. Estimation of variance in presence of measurement errors 

~ As shown in Särndal et al. (1992), in presence of measurement errors, the standard estimator of variance of ty 

~ ~ π kl −π k π l yk ,1 yl,1

V ( ) t y =∑∑ π π π

s 

kl k l k l 

~ ~ ~ is biased, being valid the following result E pm (V (ty )) = Vpm (ty )−VA1

. 

Therefore, for obtaining an unbiased estimate of Vpm (

~

ty ), it is sufficient to yield an unbiased estimate of the VA1 

component. In order to find a solution to this problem, it will be introduced in model (1) the simplified assumption that 

2 2 2 2 σ k ,1 ≅ σ k ,2 ≅ σ k . This simplification may be justified by noting that the simple response variance σ k ,t (t=1,2) is directly 

related to the individual response mechanism which can be deemed as substantially not depending from the collection mode, 

while it is necessary to distinguish between the two correlated components ∑∑ P σ kl,1 and ∑∑ P σ kl,2 which are 

k l ≠k k l ≠k 

strictly related to the collection mode of the survey. Indeed since ∑∑ σ represents, as noted above, the leading P kl,1 

k l ≠k 

components of measurement variance, the assumed simplification will have a very small effect on the estimation of VA1

. 

2 2 2 Under the statistical model presented in section 2 and assuming that σ k ,1 ≅ σ k ,2 ≅ σ k , it can be shown (see Appendix) that 

' an unbiased estimate of the vector of four unknown terms X= (X1

,...,X4

) , where 

2

2 1   2 2 X1 =∑ μ − ∑ μ k  , X 2 = VA1,1 = ∑ σ = ∑ σ ,

k P k ,1 P k

P N  P 

X3 = VA1,2 = ∑ ∑ σ , X 4 = ∑ ∑ σ

P kl,1 P kl,2 

k l ≠k k l ≠k 

 may be found as 

Xˆ = A

−1

b ~ (6) 

in which A is a matrix of fixed coefficients 

1 (N −1)/ N −1/ N 0 

 

0 1 0 0 

  A =

 

0 1 0.5 0.5

 

 



1 0 0 0 

and 

~ ~ ~ ' b = (b ,...,b 4 ) is a vector of sampling estimates, where: 1 

2

y ~ k ,1 1  1 yk,1 yl,1 ~ 1 1 2

b1 = ∑



1−  − ∑∑ s 

, b 2 = ∑ (yk ,1 − yk ,2 ) , 

s π k  N  N π kl 2 r π k ,2 k l ≠k 

~ 1 1  2 2  1

b 3 = ∑  y + y − 2 y y  +∑ ∑ (yk ,1 yl,1 + yk ,2 yl,2 − 2yk,1 yl,2 )

r π 

k,1 k ,2 k,1 k,2 

r 2 k,2 π kl,2 k l ≠k 

Page 32 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

~ yk,1 yk,2 1  1 yk,1 yl,2

b 4 = ∑ 



1 −  − ∑∑ r

r π k ,2  N  N π kl,2 k l ≠k 

with π k ,2 = π k π k|s and π kl,2 = π kl π kl|s . 

Using expression (6) on the basis of the estimates ˆXi (i =1,...,4), it is then possible to obtain an estimate of the variance 

VA1 as 

Vˆ

A1 = Xˆ 2 + Xˆ 3 = Vˆ

A1,1 +VˆA1,2 , (7) 

~ ~ ~ ~

with ˆ ˆ X2 = b2 and X3 = (N −1)b2 + N (b4- b1) representing unbiased estimates of, respectively, the simple and correlated 

components of the measurement variance under complete enumeration in (4). 

~ An estimate of Vpm (ty ) may then be given by 

~ ~ ~ ~

( )=V t ( )+Vˆ V t (8) pm y y A1 

Finally, we note that in the standard approach, where it is assumed: (i)σ k 

2 

,1 = σ k 

2 

,2 (for k ∈r or k ∈s ); (ii) σ kl,1 = σ kl,2 (for 

k,l ∈r or k,l ∈s); (iii) Em ((yk ,1 − μk ) (yl,2 − μl )) = 0 (for k,l ∈r ⊆ s), an unbiased estimate of the components of 

measurement variance may be obtained as (Särndal et al., 1992; pp. 614-617): 

~ ~ 

stan dV

ˆ

A1 =b3

 ; stan dVˆ

A1,1 = b2 = VˆA1,1

 ; s tan dV

ˆ

A1,2 =s tan d V

ˆ

A1 −s tan d V

ˆ

A1,1

. (9) 

4. Empirical results 

The proposed methodology is illustrated with an application to the reinterview survey carried out in 2001 to assess accuracy 

of the 5th Italian Census of Agriculture. The census of agriculture, taken every ten years, collects data and publishes 

information on land usage, crops and livestock, operator as well as farm characteristics by farms in Italy. Data collection for 

the 5th Census of Agriculture began at the end of November 2000, through face-to-face interviewing of some 2,75 million 

names, from individual or family operations to very large corporations as well as publicly managed farms or woodlands 

included in the census farm list. Field operations were concluded by the end of March 2001, with a few exceptions that were 

made primarily for small towns that had been affected by floods during the data collection period. 

A reinterview survey was planned at the end of data collection to evaluate accuracy of the information recorded during 

census field work. The sample design is two-stage with farms selected within municipalities (primary sampling units) to 

control geographical spread of the sample. The choice of this sampling plan was motivated essentially from budget and time 

constraints, since survey design involved a number of preliminary operations related to special processing of census 

questionnaires of farms selected in the reinterview sample. The selected sample included some 8200 farms in about 200 

municipalities from all regions of the country. 

As mentioned previously, interviewing was conducted through CATI. Using CATI had a number of advantages with respect 

to field reinterview for obtaining uncontaminated estimates of response errors (cf. Forsman and Schreiner, 1991), although in 

this case it was the only option available due to limited budget allocated to the reinterview program. Primarily, for the 

measurement of response bias there were specific advantages concerning reconciliation of differences because: a) the 

reinterviewer had no access to the original interview data, unless the difference between responses to the original interview 

and the reinterview for a given question did not cross certain tolerance limits which had been programmed in the software 

for the reinterviews; b) it was not possible for the reinterviewer to alter the reinterview response once the reinterview had 

been completed. 

Because of the complexity of the original census questionnaire, it was decided that for successful realization of telephone 

interviewing only a subset of the original questions was included in the reinterview, pertaining to: i) major crops, flowers or 

tree-growing and vineyards, ii) cattle and poultry raising and iii) family and other personnel employed in the farm. The 

reinterview included some 50 questions, 30 of which involving reconciliation as part of the reinterview process. In order to 

replicate as much as possible the original interview scenario, reinterviewers were instructed to ask speaking to the person 

that completed census questionnaire, assuming that this was the most knowledgeable person in the farm. This was deemed to 

avoid spurious effects in estimating response error components. 

For the application here considered, since the first survey is a census, only the measurement variance under complete 

enumeration VA1 and its components VA1,1 andVA1,2 are different from zero. 

In table 1 estimates of the measurement variance obtained with the proposed methodology (expressions (6) and (7)), are 

compared with the estimates obtained with the standard approach (expression (9)), for some of the main items constituting 

land usage in census of agriculture. In order to have a better understanding, the results are presented as the percent ratio of 

Page 33 
 

 

 

 

 

 

 

 

the root square of the estimate of each component of the measurement variance over the estimated total surface for the 

selected census item 

~ Vc

RV = 100 (10) c ~ ty 

~ with Vc denoting alternatively the estimate of each component VA1

, VA1,1, VA1,2 , through (6)-(7) for the proposed 

methodology and (9) for the standard approach. 

Furthermore, the table shows the estimates of the correlation between response errors, ρ , (Cochran, 1977, pp.387) 

estimated by 

~ V

ˆ

A1 1 ~ s tan dV

ˆ

A1 1

ρ = − and s tan d ρ = − (11) (N −1)V

ˆ

A1,1 N −1 (N −1) s tan dV

ˆ

A1,1 N −1 

for the proposed and the standard approach, respectively. 

Table 1: Empirical comparison of the estimates of the components of measurement variance with 

different methodologies 

Item 

Simple 

Measurement 

Variance 

RVA1,1 

Proposed Methodology Standard Methodology 

RVA1,2 RVA1 

~ρ RVA1,2 RVA1 ~

s tan d ρ

Field crops (including greenhouse 

and fluriculture) 0.071 6.941 6.941 0.00762 1.734 1.735 0.00048 

of which: Wheat 

Corn for grains 

0.167 

0.091 

3.646 3.650 0.00047 2.846 2.851 0.00029 

5.578 5.578 0.00370 1.608 1.611 0.00031 

Fruit trees 0.057 1.375 1.376 0.00047 1.195 1.197 0.00035 

of which: Vineyards 

Olive trees 

Apple trees 

0.063 

0.031 

0.134 

2.009 2.010 

4.199 4.199 

4.027 4.029 

0.00082 

0.01619 

0.00074 

1.115 1.117 

0.764 0.764 

2.319 2.323 

0.00025 

0.00054 

0.00024 

Haylage, silage and permanent 

pasture and graceland 0.113 3.221 3.223 0.00065 1.251 1.256 0.00010 

All cropland 0.055 5.410 5.410 0.00784 1.297 1.299 0.00045 

Woodlands 0.043 3.865 3.865 0.00656 1.112 1.113 0.00054 

All other lands 0.153 2.972 2.976 0.00030 7.040 7.042 0.00170 

Total land use 0.074 5.886 5.886 0.00505 2.238 2.240 0.00073 

The results of the application suggest the following considerations: 

− the estimate of the correlated component of the measurement variance, VA1,2 , obtained with the proposed methodology 

is always larger than that resulting from the standard approach; since VA1,2 is the dominant component of the total 

measurement error, the estimate of the total measurement error VA1 resulting from the proposed methodology is larger 

with respect to the standard approach, which seems to underestimate the true values of the measurement errors, in the 

example herein considered; consequently, the resulting estimate of the correlation between response errors, ρ , which 

also depends from VA1

, is larger in the case of the proposed methodology; 

− the results confirm that the simple response variance is a small component of total response error and then the most 

relevant issue when analyzing measurement error is to yield accurate estimation of the correlated component of response 

error; 

− the differences in the results obtained with the two approaches suggest that when the reinterview uses a different 

collection mode, the standard approach - which assumes the two surveys having the same response variance - fails to 

hold, and one should consider an estimation which takes into account explicitly the difference between components of 

response errors in the two surveys; 

Page 34 
 

 

 

 

 

 

 

 

− the results with the proposed methodology indicates that the measurement error under complete enumeration may be a 

significant component of total error variance, that is important to account for when assessing the accuracy of the 

reported results of surveys and censuses. 

References 

Cochran W.G. (1977), Sampling Techniques, Wiley, New York. 

Forsman G., Schreiner I. (1991), The Design and Analysis of Reinterview: an Overview, in: Measurement Errors in Surveys, 

Biemer P., Groves, R.M., Lyberg L., Mathiowetz N. and Sudman S. (Eds.), Wiley, New York, 279-302. 

Hansen M., Hurwitz W., Bershad M. A. (1961), Measurements Errors in Census and Surveys, Bullettin of the International 

Statistical Institute, 38:2, 359-374. 

Hansen M. H., Hurwitz W.N., Pritzker L. (1964), The Estimation and Interpretation of Gross Differences and the Simple 

Response Variance, in: Contribution to Statistics, Rao C.R. (Eds.), Calcutta, Statistical Publishing Society, 111-136. 

Koch G. G., Freeman D. H., Jr., Freeman J. L. (1975), Strategies in the Multivariate Analysis of Data from Complex 

Surveys, International Statistical Review, 43: 59-78. 

Särndal C. E., Swensson B., Wretman J. (1992), Model Assisted Survey Sampling, Springer-Verlag, New York. 

APPENDIX 

It is trivial to demonstrate that 

~ ' 

p ( ) = b = ( 1 4 E b b ,...,b ) , (A.1) 

being 

2

2 1   1 2

b1 = ∑ y − ∑ yk,1

; b 2 = ∑ (yk,1 − yk ,2 )

P k ,1 N    P  2 P 

2

1   1

b3 =

∑ yk,1 −∑ yk,2 

; b 4 = ∑ P yk,1 yk,2 − ∑ yk ,1 ∑ yk ,2,

2  P P  N P P 

~ ~ ~ ~ indeed b1 is the Horvitz-Thompson estimate of the corresponding population parameter b1 ; while the estimates (b2 , b3 , b4 )

correspond to the π * estimators for two phase sample (Särndal et al., 1992; pp. 347-350) of the corresponding population 

parameters (b2 , b3 , b4 ). 

Consider now the following results. 

 2 

2 1   N −1 2 1

E ( ) = E 

m b y − y = E y ) − E y 1 m ∑ k,1 ∑ k ,1  ∑ m ( k,1 ∑ ∑ m ( k,1 yl,1

) =

P   N 

 P   N P N P 

k l ≠k  

N −1 2 2 1 

= ∑ P (σ k ,1 + μ

k ) − ∑ ∑ P (σ kl,1 + μ k μl ) =

N N k l ≠k 

2

2 1   N −1 2 1 N −1 1 

=∑P 

μ

k −



∑P 

μ k  +

N ∑ P σ k,1 − ∑∑ P σ kl,1 = X1 +

N

X 2 − X3 . (A.2) 

N  N N

k l ≠k 

1 2 2 1 2 2 2 2 2 Em (b 2 ) = ∑ Em (y )+ Em (y )− 2Em (yk, yk,2 ) = ∑ σ + μ + σ + μ − 2μ = X 2 . (A.3) k,1 k,2 1 k k k k k 2 P 2 P 

Page 35 
 

 

 



1 2 2 Em ( ) b3 = ∑ E m (y )+∑ ∑ E m k, (y y 1 l,1

)+∑ E m (y )+∑ ∑ E m k, (y y 2 l,2 )+

k,1 P k ,2 P 2  P P

 k l ≠k k l ≠k 



− 2∑∑ P Em (yk ,2 yl,2 ) =



kl 

 

1 2 2

= ∑ P σ k +∑ ∑ P σ kl,1 +∑ P σ k +∑ ∑ P σ kl,2  = X 2 + 0.5 X3 +0.5X 4 (A.4) 

2  

 k k l ≠k k k l ≠k 

2

1 2 1   Em ( ) b 4 = ∑ E (y y ) − y y = μ − μ =

P m k,1 k,2 ∑ k ,1 ∑ k ,2 ∑ P k ∑ k 

X1 . (A.5) 

N P P N  P 

From (A.1),..., (A.5) we derive the following result 

~ E m p [E (b)]= AX . (A.6) 

ˆ From the ab ove it is possi le demonstrate that b X , as defined by (6) is an unbiased estimator of the vector X, i.e. 

ˆ −1~ −1 ~ −1 E E m p m p m p [ ( ) X ] = E [E (A b)]= A E [E ( ) b ]= A AX = X .

Page 36