Inferential Methods to Identify Possible Interviewer Fraud Using

Leading Digit Preference Patterns and Design Effect Matrices

Moon J. Cho, John L. Eltinge, David Swanson,

U.S. Bureau of Labor Statistics

Moon J. Cho, 2 Massachusetts Avenue NE, Washington, DC 20212

(Cho M@bls.gov)

Key Words: Benford’s Law, Cluster sample, Consumer Expenditure Survey, Curbstoning, Misspecification effect, Rao-Scott adjusted goodness-of-fit test

.

1. Introduction

Interviewer fraud can damage the data quality

severely. How can we detect it? Turner et al. (2002)

used response patterns to detect falsification. They

reported that suspected falsifiers could be noticeable by an unexpectedly high yield of interviews per

assigned sample address, and/or unusual response

rates for specific reported variables on behaviors.

Turner et al. also discussed the systematic differences between suspected falsifiers and other interviewers in providing the verification means, such as

telephone numbers of the respondents. Biemer &

Stokes (1989) proposed a statistical model for describing dishonest interviewer behavior, which was

applied to a general quality control sample design

and several associated cost models. A 1982 U.S. Bureau of the Census study indicated a higher degree

of cheating in urban areas (Biemer & Stokes). The

study also shows a substantial and highly significant tendency for relatively inexperienced interviewers to cheat more frequently for the two largest demographic surveys, the Current Population Survey

and the National Crime Survey (Biemer & Stokes).

We used the leading digits to detect curbstoning in

this paper. The effect of the sampling design, such

as stratification and clustering, on standard Pearson chi-squared test statistics for goodness of fit is

investigated.

Statistical methods for analyzing cross-classified

categorical data has been extensively developed under the assumption of multinomial sampling. However, most of the commonly used survey designs

involve clustering and stratification and hence the

multinomial assumption is violated (Rao & Scott,

The views expressed in this paper are those of the authors

and do not necessarily reflect the policies of the U.S. Bureau

of Labor Statistics.

1981). Literature has shown that clustering can have

a substantial effect on the distribution of the standard Pearson chi-squared test statistic, χ2 and that

some adjustment to χ2 may be necessary, without

which one can get misleading results in practice (Rao

& Scott, 1981). Rao & Scott developed a simple correction to χ2 which requires only the knowledge of

deffs (or variance estimates) for individual cells in

the goodness of fit problem (Rao & Scott, 1981).

The original Rao & Scott papers considered inference for one vector of proportions, based on (essentially) one sample.

In this paper, we are considering inference for a

large number of proportion vectors pi, i = 1, ..., I,

where I is the total number of interviewers. Only

a small portion of an interviewer’s workload can be

verified because of the limited resources. Therefore,

we addressed the optimum allocation of resources

such as re-interview time using the optimal decision

rule.

2. Data Collection in Consumer Expenditure Survey

Consumer expenditure surveys are specialized studies in which the primary emphasis is on collecting

data relating to family expenditures for goods and

services used in day-to-day living (BLS Handbook,

1997, p.160). The current survey consists of two

separate surveys, each with a different data collection technique and sample: In the Interview survey, each consumer unit (CU) in the sample is interviewed every 3 months over five calendar quarters. The sample for each quarter is divided into

three panels, with CU’s being interviewed every 3

months in the sample panel of every quarter. The

interviewer uses a structured questionnaire to collect both demographic and expenditure data in the

Interview survey. In the Diary (or recordkeeping)

survey, demographic data is collected by the interviewer, whereas expenditure data is entered on the

diary form by the respondent family for two consecutive 1-week periods (BLS Handbook, 1997, p.161).
Both surveys are conducted by personal visits with telephone usage limited to appointment

scheduling and follow-up calls for information missed

at the time of the proposed interview. The current method for identification of possible interviewer

fraud or gross reporting error in the Consumer Expenditure Survey is a reinterview program. The

reinterview is conducted by a member of the supervisory staff. A subsample of approximately 6 percent

of households in the Interview survey and 17 percent

in the Diary survey are reinterviewed on an ongoing

basis (BLS Handbook, 1997, p.161).

3. CES Data Overview

Swanson et al.(2003) analyzed overall proportions

of leading digit, and the related covariance matrices. We studied the overall proportions in detail in

this paper. Our first question was whether we see

any distinct patterns in the overall proportions with

respect to quarters. Table 1 shows ratio estimates

from Quarter 2 to Quarter 5. The table also shows

the values of the standard errors from the balanced

repeated replication method (SEBRR) with 44 replicate weights, and the values of the standard errors

that one can obtain from multinomial assumption on

the underlying observations (SEχ2 ). Note that the

values of SEχ2 is consistently smaller than the ones

of SEBRR. The ratio of these two values, SEBRR

SEχ2 ,

can be considered as a deft (Kish, 1995), and is the

range of this ratio is between 0.996 and 1.601. We

did not observe any major change in the values of

proportions with respect to the quarters. When we

formally tested the quarterly effect on the leading

digit 2, the value of test statistic is 5.20 with the

critical value 9.01 at α = 0.05.

In the following sections, we will examine the test

statistics for the proportions collected by each interviewer.

4. Options for Quadratic-Form Test

Statistics

In our example data of the U.S. Consumer Expenditure Survey (CES), individual field representatives

are closely tied to a sample design. We will consider

a household as a cluster within an interviewer. Define xicj = (x1icj , ··· , x9icj ) a 9×1 vector such that

xdicj = 1 if the leading digit is d, and 0 otherwise,

for a given interviewer i, an interview c and an expenditure item j. Let Jic be the number of non-zero

reports obtained for an interview c by an interviewer

i, and xic = J

ic

j=1

xicj = (x1ic, ··· , xdic, ··· , x9ic) be

a 9 × 1 vector where xdic is the number of non-zero

reports with a leading digit d for an interview c by

an interviewer i. Let ˆpi be a mean vector of leading

digits for interviewer i.

pˆi = (ˆp1i, ··· , pˆ9i)



= 1

Ji

ni

c=1



Jic

j=1

xicj

= 1

Ji

ni

c=1

xic

where ni is the number of interviews conducted by

an interviewer i, and Ji = ni

c=1

Jic is the total number

of non-zero reports obtained by an interviewer i.

The general test statistic is:

Ti = (ˆpi − p0)



V −1

i (ˆpi − p0) .

where p0 is a 9 × 1 reference proportion vector. p0

can be a vector of proportions predicted by Bedford’s

Law, or a vector of empirical CES proportions.

The following are possible estimators of Vi.

4.1 Design-based variance estimator

A design-based variance estimator of ˆpi, Vˆ (ˆpi), is

Vˆ (ˆpi) = Vˆ



1

Ji

ni

c=1

xic

= Vˆ



1

ni

ni

c=1

ni

Ji

xic

= Vˆ



1

ni

ni

c=1

yic

= 1

ni

1

(ni − 1)

ni

c=1

	

(yic − pˆi) (yic − pˆi)






where yic = ni

Ji xic. A test statistic is:

Ti 1 = 

pˆi(8) − p0(8)




Vˆ (ˆpi(8))

−1



pˆi(8) − p0(8)

where ˆpi(8) is the first eight elements of ˆpi. Since

Vˆ (ˆpi)9×9 will be always singular because of the constraint on ˆpi in our case, we use ˆpi(8) to compute

Ti 1.

Vˆ (ˆpi(8)) can also be singular when ni is small. A

range of ni, the number of interviews covered by an

interviewer i, is 1 to 185 in our example, a mean of

25, a first quartile of 2, a median of 9, and a third

quartile of 37. Among total number of 1235 interviewers, there are 234 interviewers who conducted
Table 1: Quarterly Ratio Estimates and Associated Standard Errors

Quarter 2 Quarter 3 Quarter 4 Quarter 5

RˆQ2 SEBRR RˆQ3 SEBRR RˆQ4 SEBRR RˆQ5 SEBRR

SEχ2 SEχ2 SEχ2 SEχ2

0.3059 0.0012 0.3053 0.0014 0.3010 0.0013 0.3018 0.0015

0.0010 0.0011 0.0011 0.0011

0.1918 0.0011 0.1931 0.0014 0.1953 0.0012 0.1941 0.0013

0.0009 0.0009 0.0010 0.0009

0.1208 0.0010 0.1231 0.0008 0.1246 0.0009 0.1245 0.0010

0.0007 0.0008 0.0008 0.0008

0.0897 0.0008 0.0894 0.0007 0.0911 0.0010 0.0908 0.0008

0.0006 0.0007 0.0007 0.0007

0.1038 0.0007 0.1051 0.0008 0.1050 0.0011 0.1051 0.0011

0.0007 0.0007 0.0007 0.0007

0.0679 0.0006 0.0696 0.0007 0.0693 0.0009 0.0671 0.0006

0.0006 0.0006 0.0006 0.0006

0.0485 0.0007 0.0477 0.0006 0.0463 0.0005 0.0480 0.0006

0.0005 0.0005 0.0005 0.0005

0.0451 0.0006 0.0431 0.0006 0.0434 0.0007 0.0439 0.0005

0.0005 0.0005 0.0005 0.0005

0.0265 0.0005 0.0235 0.0006 0.0241 0.0006 0.0247 0.0004

0.0004 0.0004 0.0004 0.0004

only one interview. This indicates that we have quite

a few interviewers whose ni is very small. We found

that Vˆ (ˆpi(8)) is too unstable.

4.2 Variance estimator for ratio estimator

Consider the number of non-zero reports obtained

by an interviewer i, Ji, as a random variable. The

ratio estimator is

Rˆi =



ni

c=1



Jic

j=1

xicj



 /

ni

c=1



9

d=1

xdic

=

ni

c=1

xic

/Ji

.

Recall that Ji is the number of non-zero reports obtained by an interviewer i . Then the variance estimator for the ratio estimator is:

Vˆ (Rˆi)

= {ni(ni − 1)}

−1

×



(Ji/ni)

−2ni

c=1




(xic − Rˆi Jic)(xic − Rˆi Jic)





.

A test statistic is:

Ti 2 =



Rˆi − p0(8) 


Vˆ (Rˆi(8))

−1 

Rˆi − p0(8).

Note that Vˆ (Rˆi(8)) shares the same problems that

Vˆ (ˆpi(8)) has with small values of ni.

5. Misspecification Effect Matrices

A misspecification effect is one measure of the effect

of a complex sample design. It measures the bias of

a variance estimator which is computed under misspecified design or modeling assumptions. The eigen

structure of a misspecification effect matrix allows

comparison of some sensitivity properties of competing quadratic-form test methods. Define ∆i a 8 × 8

multivariate design effect matrix for an interviewer,

i. Then

∆i = Σ− 1

2

i ViΣ− 1

2

i

where Σ− 1

2

i is an inverse of a square root symmetric

matrix of Σi

Σi = 1

Ji

	

diag(p0(8)) − (p0(8))(p0(8))




 ,

and Vi is a corresponding covariance matrix of our

choice. The misspecification matrix for Vˆ (ˆpi) is:

∆i 1 = Σ− 1

2

i Vˆ (ˆpi)Σ− 1

2

i

The misspecification effect matrix for Vˆ (Rˆi) is:

∆i 2 = Σ− 1

2

i Vˆ (Rˆi)Σ− 1

2

i
The eigenvalues of the misspecification effect matrix are called generalized design effects. Rao &

Scott used the mean and coefficient of variation of

these eigenvalues to develop modifications of certain

chi-square goodness-of-fit test statistics.

5.1 Modified Chi-Squared Tests

Consider a naive χ2 statistic to test how a distribution of the leading digit proportions from non-zero

reports obtained by an interviewer differs from the

distribution of the reference proportions.

χ2

i = Ji



D

d=1

	

(ˆpdi − pd0)

2/pd0




where d is 1, ··· , D = 9.

Define λi the eigenvalues and λ ̄ the average eigenvalue from the estimated misspecification effect matrix. The first order Rao-Scott adjusted test is:

χ2

iM1 = χ2i / λ ̄

= (D − 1) χ2

i /



D

d=1

{(1 − pˆdi) deffdi}



where deffdi = {pˆdi(1 − pˆdi)/Ji}

−1 Vˆ (ˆpdi), and

Vˆ (ˆpdi) is the d-th diagonal element of Vˆ (ˆpi). The

performance of χ2

iM1 depends on the eigenstructure

of the misspecification effect matrix. When λˆi are

approximately equal to their average, λ ̄, χ2

iM1 is

approximately distributed as a χ2 random variable

with D − 1 degrees of freedom (Lee & Eltinge).

The second order approximation which is the better approximation can be used if there is more information available on the λˆi’s (Rao & Scott, 1981).

χ2

iM2 = χ2iM1 / (1 + ˆa2)

where ˆa2 = 	

(D − 1)λ ̄2
−1 (D

−1)

d=1

(λˆdi − λ ̄)2 .

For example, we have an interviewer who has

conducted 45 interviews with total of 725 non-zero

reports. The following are the computed statistics for the interviewer: Ti 2 = 37.01,χ2

i = 26.47,

χ2

iM1 = 14.74 with λ ̄ = 1.80, χ2iM2 = 14.45 with

aˆ2 = 0.02.

5.2 Other Related Tests

The examination of the expenditure data collected

by the CEQ survey in the year 2000 shows that

the leading digits of those expenditures follow Benford’s Law quite closely (Swanson, Cho, and Eltinge,

2003). However, digits such as 5’s or 9’s do not follow

Benford’s Law as closely as others. When comparing

the leading digits collected by an unusual field representative with overall CEQ’s leading digits distribution, the leading digit of 5’s or 9’s of an unusual field

representative do not follow the overall CEQ’s leading digits distribution (Swanson, Cho, and Eltinge,

2003). We may test deviations from null proportion only on those leading digits using an univariate

t-test or Bonferroni test statistics:

t = pˆdi − pd0

Vˆ (Rˆdi)

where Vˆ (Rˆdi) is the diagonal element of Vˆ (Rˆi) for

the leading digit d.

Korn & Graubard demonstrated that Wald statistics behave poorly when simultaneously testing a

large number of regression coefficients. They suggested that for some applications, the use of Bonferroni inequality on the individually tested regression

coefficients can have greater power than the Wald

procedure. The Bonferroni procedure is preferable

to the Wald procedure when the number of variables

is large compared with the number of sampled PSU’s

(Korn & Graubard).

6. Optimal Decision

Define πi to be the probability that a given item

recorded by interviewer i is erroneous. For simplicity, we will assume that a given interview records

data that can be classified entirely as erroneous or

not erroneous. In an interview that is not erroneous, all expenditure amounts are reported accurately. This ideal case would arise, for example, if

the interview subject had maintained comprehensive

and accurate records of all relevant expenditures by

the consumer unit, and based all interview responses

on these records. At the other extreme, we will define an interview to be erroneous if either: (a) the interviewer has fabricated all responses, i.e., the interview was “curbstoned” or (b) the interviewer accurately recorded all responses given by the interview

subject, but the interview subject’s responses were

based on uninformed guesses. Many practical cases

would fall between these two extremes. For example,

an interviewer might fabricate responses in one section for which the interview subject refused to provide responses; or an interview subject might provide uninformed guesses about expenditures within

groups for which another family member made the

purchase decisions. However, these more complex

cases are beyond the scope of the current work. In

general, an ideal procedure would be the one that

has a relatively high possibility of identifying interviewers who have digit-reporting patterns that deviate substantially from the overall pattern, while
also having a relatively low Type I error rate. For

the Consumer Expenditure Survey data, this issue

is complicated by the fact that the number of interviews covered by a given interviewer varies a great

deal as we mentioned in the Section 4.1.

A is the cost of conducting the one reinterview,

B is the cost of including an erroneous report, R

is the number of reinterviews, and U is the number of undetected erroneous reports. Let Ci be the

optimal cut off point which minimizes E(cost) =

A E(R) + B E(U). If ˆpi ≤ Ci, then we reinterview

all of the households which had previously been interviewed by interviewer i. Note that Ci depends on

the number of households (ni), the number of total

items (Ji) reported by interviewer i, and the design

effects.

E(R, interviewer i)

= ni P (ˆpi ≤ Ci)

= niP (ˆpi ≤ Ci|Err) πi

+ niP(ˆpi ≤ Ci|NotErr) (1 − πi)

= niπi P



pˆi − EGE(ˆpi)

VGE(ˆpi) ≤

Ci − EGE(ˆpi)

VGE(ˆpi) |Err

+ ni(1 − πi)

× P



pˆi − EGT (ˆpi)

VGT (ˆpi) ≤

Ci − EGT (ˆpi)

VGT (ˆpi) |Non

≈ ni πi Φ



Ci − EGE(ˆpi)

VGE(ˆpi)



+ ni (1 − πi) Φ 

Ci − EGT (ˆpi)

VGT (ˆpi)



= ni πi γE(Ci) + ni (1 − πi) γT (Ci)

where GE and GT are the distributions of the erroneous reports and the truthful reports respectively, VGE(ˆpi) = pi,GE(1−pi,GE )

Ji deff, pi,GE =

EGE(ˆpi), VGT (ˆpi) = pi,GT (1−pi,GT )

Ji deff, pi,GT =

EGT (ˆpi), γE(Ci)=Φ 

C

√i−EGE(ˆpi)

VGE(ˆpi)



, γT (Ci) =

Φ



C

√i−EGT (ˆpi)

VGT (ˆpi)



.

E(U, interviewer i)

= ni P(undetected erroneous report | Err)

= niπi P (ˆpi ≥ Ci| Err)

= niπiP



pˆi − EGE(ˆpi)

VGE(ˆpi) ≥

Ci − EGE(pi)

VGE(ˆpi) |Err

≈ niπi



1 − Φ



Ci − EGE(pi)

VGE(ˆpi)



= niπi {1 − γE(Ci)}

Option 2: Find Ci to minimize E(R) subject to

E(U) ≤ Ri = niR0. From the definitions of E(R)

and E(U), note that E(R) is monotone increasing

with respect to Ci, and E(U) is monotone decreasing

with respect to Ci. Hence the minimum of E(R)

can be obtained when E(U) = niR0, i.e. when Ci =

γ−1

E (1 − R0/πi) .

Option 3: Find Ci to minimize E(U) subject to

E(R) ≤ Ui = niU0. From the definitions of E(U)

and E(R), note that E(U) is monotone decreasing

with respect to Ci, and E(R) is monotone increasing

with respect to Ci.

Define γ(Ci) = niπi γE(Ci) + ni(1 − πi)γT (Ci)

Then the minimum of E(U) can be obtained when

E(R) = niU0, i.e. when Ci = γ−1 (niU0) .

7. Discussion

We tested in the section 3 whether the quarter in

which the data was collected matters. The same

test can be applied for the level of experience of interviewers, or other relevant factors.

We considered the univariate case in the Section 6,

but this can be extended to the more complex cases

such as the multivariate one. One can also apply the

mixture modeling method for parameters of GE and

GT. A sensitivity analysis for the optimal decision

would be of interest.

8. References

Biemer, P.P. and Stokes, S. L.(1989). The optimal design of quality control samples to detect interviewer cheating. Journal of Official Statistics 5,

23-39.

Bureau Of Labor Statistics (1997). BLS Handbook

of Methods. U.S. Department of Labor.

Hill, Theodore (1995). A Statistical Derivation of

the Significant-Digit Law. Statistical Science 10,

354-363.

Korn, E. and Graubard, B.(1990). Simultaneous

testing of regression coefficients with complex survey

data: use of Bonferroni t statistics. The American

Statistician 44, 270-276.

Lee, S. and Eltinge, J.( ). Exploratory analysis of estimated design effect matrices computed from complex survey data.
McLachlan, G.J. and Basford, K.E.(1988). Mixture

Models. Marcel Dekker, INC.

Rao, J.N.K. and Scott, A.J.(1981). The analysis of

categorical data from complex sample surveys: chisquared tests for goodness of fit and independence

of two-way table. J. Amer. Statist. Assoc. 76,

221-230.

Rao, J.N.K. and Scott, A.J.(1984). On chi-squared

tests for multi-way contingency tables with cell proportions estimated from survey data. Ann. Statist.

12, 46-60.

Swanson, D., Cho, M., Eltinge, J(2003). Detecting

possibly fraudulent or error-prone survey data using

Benford’s Law. Proceedings of the Section on Survey

Research Methods, American Statistical Association

to appear.

Turner, C., Gribble, J., Al-Tayyib, A., and Chromy,

J. (2002). Falsification in epidemiologic surveys: detection and remediation. Technical Paper on Health

and Behavior Measurement. 53. Washington DC:

Research Triangle Institute.