Office of Tax Analysis 

U.S. Treasury Department 

Washington, D.C. 20220 

Issued: July, 1976 

Statistical Problems of Merged 

Data Files 

Joseph B. Kadane 

OTA Paper 6 December 12, 1975 
TABLE OF CONTENTS 

page 

I. Introduction ...................... 1 

I1. A Statistical Model .................. 3 

A . Assumptions .................... 3 

B . Complications ................... 3 

C . The Model and Its Application to the TransportationProbleq .................... 4 

I11. Assumption of Conditional Independence . 7 

IV. Interpretation of Matched Sampling ........... 10 

V . unresolved Questions About Matching .......... 11 
-1-

I. INTRODUCTION 

Merging micro data files is a common occurrence. There are two 

aspects to the statistics of merged files that deserve attention: the 

proper procedure of merging, and the correct interpretation of the 

results. This report will concentrate on the first question, although 

some thoughts on the second are offered in Section IV. 

Suppose that two files are given with some overlapping variables 

and some variables unique to each of the two files. Notationally, let 

X represent the common variables, Y the variables unique to the first 

file, and 2 the variables unique to the second file. Thus, the basic 

data consists of a sample of pairs (X,Y) and a sample of pairs (X,Z). 

(Later the possibility of weighted samples, important to the Treasury 

application, will be considered. Weighted sampling does not radically 

complicate the analysis.) 

One important method, reported by Okner [1972a] sets up "equivalence 

classes" of X's, and makes a random assignment of an (X,Y) with an 

(X,Z) among "equivalent" (X,Z)'s which achieve a minimum closeness 

score. Sims, in his comment [1972a] and rejoinder [1972b], stresses the 

need for a theory of matching, and criticizes the Okner procedure for 

making the implicit assumption that Y and 2, given X, are independent. Peck [1972] defends the assumption, while Okner [1972b] discusses 

the validity of the assumption in various cases. 

In a second round of discussion, Okner [1974], Ruggles and Ruggles 

[1974], Alter [1974] and Budd [1974] improve the method, but continue to 
-2-

concentrate on equivalence classes. Sims' [1974] comment again stresses 

his belief that the methods proposed will not perform well in sparse 

X-regions. 

Section I1 of this report gives a model for matching and derives 

the maximum likelihood match, This leads to a distance function as 

sought by the Treasury Department. Section 111 gives some thoughts 

about Cyz and conditional independence. Section IV discusses the question of interpretation of calculations from a matched sample, and Section V 

concludes with some unanswered questions. 
-3-

11. A STATISTICAL MODEL 

A. Assumptions. We assume that originally there were true 

triples (X ,Y ,Z.) that had a normal distribution with means (ux ,uy ,uz) iil 

and some covariance matrix C. These were broken into two samples, 

(Xi,Yi) and (Xi,Zi), and then independent normal measurement errors 

(si) were added. 

Let 

1 x1 = xi + si i 

x2 = xi + 5,2 

i 

1 2 where (si’si) each has a normal distribution with zero mean. Suppose 

also that Si has covariance matrix Q

1 

and 5,2 has covariance matrix ,Q2, 

2 and that 5,1 and si are independent for all i. We may then observe 

a permutation of the paired observations (Xt,Yi) and (Xf,Zi). 

B. Complications. There are two aspects of unrealism in this model. 

First, we assume that the two samples represent the same individuals, 

and the X values have been distorted only by some measurement errors. 

We know that in general, this is not the case. Nonetheless, we use this 

assumption not only because it gives a reasonable answer, but also because 

it is the only assumption available. Second, we assume joint normality of 

X, Y, and Z. This is untrue of our data in at least two important respects, First, some of our data is binary or integer-valued. Second, 
-4-

joint normality implies that all the regressions are linear, which is 

not likely to be the case, as pointed out by Sims [1972a,b, 19741. One 

way around that problem might be to assume joint normality region-byregion in the X-space. This thought is not pursued further here. 

C. The Model and Its Application to the Transportation Problem. 

Let Ti = (Xi,Yi) 1 and Ui = (Xf,Zi) be vectors of length k and 1 

respectively, where without loss of generality we take k 1. Also 

without loss of generality, take 1-1~ = 0, py = 0, pZ = 0. The covariance 

matrix of T and U can be written as 

zxx+Q1 :

* cxx 

: cYx c .............;............. .;... r: cxy ;: %x+92 cxz 

I 

czx CZY : czx ~~~ ZZ i 1 

L J 

C12 = (ell - c12z;;c21)-1~12c;; . 

Note that all covariances above can be estimated easily except cYz* 

Treatment of 

cYz is deferred to Section 111. 

Now suppose that vl,....,v

n 

is the random permutation of 

T1,...,Tn which is observed, and wl,...,w is the random permutation n 

of U1,...,U n which is observed. Let $ = [$(l), ...,$( n)] be a permutation of the integers 1,...,n. 
-5-

According to DeGroot and Goel [ 1975a1, the likelihood function of 

$ is 

Thus the maximum likelihood $ minimizes 

c(+> = c v;c12w$(i).

i=1 

c

Let pij = viC12wj. 

Then minimizing C($) is equivalent to minimizing 

C = Zpijaij 

subject to the conditions 

Caij = 1 

i 

where aij = 0 or 1. 

This is a linear assignment problem (DeGroot and Goel, 1975a). 

In the case of observations with weights, suppose vi has weight 

xi(i=l, ...,n) and wj has weight y.(j=l, ...m) where we assume J 

n m 

c xi = c yi.

i=l i=l 

Then the natural generalization is to minimize 
-6-

subject to the conditions 

1 a 

i ij = y. for j=l,...,m J 

C aij = xi for i=l,...,n. 

j 

This is a transportation problem, which happens to be what the Treasury 

Department is now programing. Thus the matrix C12 above is my answer 

to the question of what distance function to use in the transportation 

prob1em. 
-7-

111. ASSLWTION OF CONDITIONAL INDEPENDENCE 

One of the difficulties of the preceding method is that it requires 

knowledge of Cyz. There are several possible sources of such information. First, from a coarse but perfectly matched sample, certain 

elements of Eyz may be known. If so, surely this information should 

be used. Second, the assumption may be made, as is customary in the 

matching literature, that Y and Z are conditionally independent, 

given the X's. That is 

1 2 1 2 1 2 f(Y,Z(X ,x ) = f(YIX ,x )f(Z(X ,x ). 

The covariance matrix of (Y,ZIX 1 ,X 2

) is (see Anderson pp. 28, 29) 

r 1 -1 

1 IZY Ezz 

L d 

Conditional independence occurs if and only if the upper-right partitioned 

submatrix is zero, i.e., if and only if 

r 1 -1 

-(C E 1 I xxlxl Ex1x2 2j 0 

=YZ YX1 Yx2 

xx xz 

Thus this assumption gives a condition which uniquely defines Cyz 

in terms of the other E's. Some simplification of this answer is possible. 
-8-

Using 

c =c =E and C =C 2-C , YX1 YX2 YX zxl zx zx 

we have 

r 

-1 

c =(C c ) 

YZ mYx 

L 

Suppose, without loss of generality, that 

Then 

= (ZyxR+CyxS ' CyxS+CyxV) I c,, I 

= CyX(R + S' + S + V.)Zxz 

A well-known fact about inverses of partitioned matrices (see 

Rao 1965, p. 29) is 1-1 r: [Fl;IF E+ 

1 B' D 

L 
-9-

where E = D - B'A-lB and F = A-1B. 

neri R + st + s + v = ~-1+ FE-'F~ - FE-~- E-~F?+ E-'­

= A-~+ (I-F)E-~(I-F)~ 

= ~-1 + (I - A-~B>E-~(I - BVA-~)

= A+A + (A-B) (D-B~A-~B)-~(A-B~ IA-~ 

Hence in our case, 

-1 -1 c + (rXIX1 - Cx1x2)(Cx2x2 

-e 2 lC1 lC1 2 cyz = c 1-l

yx xlxl(zxlxl 

Thus Cyz is given by this equation as a function of CyxyCxz, 

~x1x1'cx2x2 and zx2x1* All of these can be directly estimated except 

the last, Xx2x1. 

One way to obtain an estimate for is to formulate an 

opinion about the covariance matrix of the error process, that is, to 

take R1 and a2 as known. Knowing one implies an estimate for the 

other (under the assumption of independence between 5, and E:), so 

this can be used as a check on the procedure. 
- 10 -

IV. INTERPRETATION OF MATCHED SAMPLING 

If Cyz were known, the matching procedure of Section I1 could 

be used without further assumption. And yet it would not be necessary 

to do matching at all, since then the joint distribution of (X1,Y,X2,Z) 

would be known. Any probability desired could in principle be calculated 

from this distribution, or, if necessary, simulated directly. Therefore, 

the value of matching, at least in this jointly normal world, depends on 

a situation in which Cyz is not known. In this case, some assumption 

must be made about CYz in order to use the procedure of Section 11. 

It is my judgment that this line of criticism is not as damaging as 

it might first appear. If Y is well predicted by X1 and Z is well 

predicted by X2 , and if X1 and X2 are close, the conditional independence assumption is not bad because the conditional variability will 

be low. So while covariances might be well estimated, correlations might 

be poorly estimated. Yet recent work of DeGroot and Goel [1975b] suggests 

that even some information about the correlation can be squeezed out of 

a matched sample, although not very much. What is clear is that a matched 

sample cannot be treated uncritically as though it were a joint sample 

that had never been split and reunited. Thus the right question is not 

the quality of the match itself, but rather the correct use and interpretation of statistics derived from the matched sample. Our understanding of 

this question is in its infancy. 
- 11 -

V. UNRESOLVED QUESTIONS ABOUT MATCHING 

The foregoing discussion raises a number of unresolved questions. 

The last three questions listed below concern the relation of merging to 

file reduction (Turner and Gilliam, 1975). 

(i) Is there a simple form for C12? Under what conditions is it 

true that 

12 =Iil 

for some W? (In this case C12 would be a distance matrix just 

on X1 and X2.) 

(ii) The discussion of Section IV indicates that there might be a theorem 

characterizing those functions of the parameters that have consistent 

estimates. Possibly the conditions specified in Section IV are 

necessary for certain functions to be consistently estimated. 

(iii) Can a method be found for estimating X, the common value of X1 

and X2 ? When the time comes to use the matched sample, some values 

must be taken for the X variables. 

(iv) How should the theory be extended to deal with non-normal variables? 

This question is particularly important for binary variables. 

(v) Can a more realistic model for matching be found, one which does 

not assume the same people in both populations? What is the meaning 

of the estimates derived from such a model? 

(vi) Can a similar distance function be found for file reduction? 
- 12 ­

(vii) Can the common variables to which two or more files are reduced 

be estimated, perhaps similarly to (iii) above? 

(viii) Can a common model, theory, and procedure be found for simultaneous 

file reduction and merging? 
- 13 -

REFERENCES 

Alter, H. E., "Creation of a Synthetic Data Set by Linking Records of the 

Canadian Survey of Consumer Finances with the Family Expenditure

Survey 1970", Annals of Economic and Social Measurement 3, 1974, 

pp. 373-394. 

Anderson, T. W., Introduction to Multivariate Statistical Analysis,

J. Wiley and Sons, New York, 1958. 

Budd, E. C., "Comments", Annals of Economic and Social Measurement 1, 

1972, pp. 349-354. 

DeGroot, M. and P. Goel, "The Matching Problem for Multivariate Normal 

Data", Technical Report No. 94, Department of Statistics, CarnegieMellon University mimeo, 1975a. 

DeGroot, M. and P. Goel, "Estimation of the Correlation Coefficient from 

a Broken Random Sample", Technical Report No. 105, Department of 

Statistics, Carnegie-Mellon University mimeo, 1975b. 

Okner, B., "Constructing a New Data Base from Existing Microdata Sets: 

the 1966 Merge File", Annals of Economic and Social Measurement 1, 

1972a, pp. 325-342. 

Okner, B., "Reply and Comments", Annals of Economic and Social Measurement 1, 1972b, pp. 359-362. 

Okner, B., "Data Matching and Merging: An Overview", Annals of Economic 

and Social Measurement 3, 1974, pp. 347-352. 

Peck, J. K., "Comments", Annals of Economic and Social Measurement 1, 

1972, pp. 347-348. 

Rao, C. R., Linear Statistical Inference and Its Applicationa, (1st ed.)

J. Wiley and Sons, New York, 1965. 

Ruggles, N. and R. Ruggles, "A Strategy for Merging and Matching Microdata Sets", Annals of Economic and Social Measurement 3, pp. 353-371. 

Sims, C., "Comments", Annals of Economic and Social Measurement 1, 1972a, 

pp. 343-345. 

Sims, C. A., "Rejoinder", Annals of Economic and Social Measurement 1, 

1972b, pp. 355-357. 
- 14 -

Sims, C., "Comment", Annals of Economic and Social Measurement 3, 1974, 

pp. 395-397. 

Turner, J. S. and G. B. Gilliam, "A Network Model to Reduce the Size 

of Microdata Files", presented to the Las Vegas ORSA Conference, 

mimeo, 1975.