# Presentation: 322622

## Using the SIPP Synthetic Beta for Analysis

- John M. Abowd, Gary Benedetto, Martha Stinson
- Cornell University and U.S. Census Bureau
- Census Training Meeting, Oct. 26, 2007

## Outline

- Background
- Framework for Public Use Product
- Sources for the Gold Standard data
- Current System for data access
- Assessing analytical validity
- Creation of Synthetic Data

## Background

- In 2001, a new regulation authorized the Census Bureau and SSA to link SIPP and CPS data to SSA and IRS administrative data for research purposes
- Idea for a public use file was motivated by a desire to allow outside access to long administrative record histories of earnings and benefits linked to household demographic data
- These data allow detailed statistical and simulation study of retirement and disability programs
- Census Bureau, Social Security Administration, Internal Revenue Service, and Congressional Budget Office all participated in development

**Notes:**
Committee with representatives from CBO, IRS, SSA, and Census met for first time in June 2001

## Framework for SIPP/SSA/IRS Synthetic Beta

- Link 5 SIPP panels with lifetime earnings and benefit histories from IRS and SSA
- Keep a few key variables unchanged
- Choose and synthesize other SIPP, IRS, and SSA variables subject to the following requirements:
- List of variables must be long enough to be useful to some group of researchers
- Multivariate relationships across synthesized variables must be analytically valid:  shorter lists, less distortion
- disclosure avoidance challenge: users cannot re-identify source records in existing SIPP public use file:  shorter lists, more distortion

**Notes:**
Conceptual Idea
Administrative Earnings provide history
SIPP provides demographic and additional economic information
Challenge
Census would not allow release of additional SIPP variables
This public use data set must stand alone

## Important Design Decisions

- Four variables remain unsynthesized:  sex, marital status, benefit status (initial and 2000);  also link to spouse is not perturbed
- 616 variables would be synthesized
- Variables chosen for target users:  disability and retirement research communities
- Use SRMI and Bayesian Bootstrap methods for data synthesis
- Create “gold standard” data and compare to synthetic data to assess analytic validity
- Try to match back to public use SIPP to test disclosure problems

## Create Gold Standard Data

- Create a data extract from the SIPP panels conducted in the 1990s
- Five panels: 1990, 1991, 1992, 1993, 1996
- Data from core and topical module survey questions
- Standardize variables across panels
- Link to Earnings Records and SSA benefits data
- These data are the “truth.”  Any synthetic data must preserve the characteristics of and relationships among the variables on this file.

## SIPP Variables:  Demographic

- Variables that will not be synthesized
- Gender, marital status, link to spouse, and the same variables for the spouse, if present.
- Variables to be synthesized
- Five category education, black, Hispanic, birth month/year, death month/year, disability limits work, disability prevents work, total number of children in family, history of marital events up to 4 marriages, age at which marital events occur, foreign born, decade arrive in US

## SIPP Variables:  Economic

- Labor Force Participation
- Annual time series (1990-1999):  Weeks worked with pay, weeks worked part-time, total hours worked, total earnings
- Four category industry and occupation
- Income
- Annual time series (1990-1999) Family poverty threshold, total family income, total personal income, family welfare participation and income, disability participation and income (non-SSA)

## SIPP Variables:  Wealth and Benefits

- Wealth
- Total net worth, home ownership indicator, home equity, non-housing wealth
- Pension
- Defined benefit plan, defined contribution plan
- Health insurance
- Annual time series (1990-1999):  health insurance indicator, health insurance from employer indicator

## SSA/IRS Master Earnings File Variables

- Summary Earnings Record Extract 1951-2003
- Annual total FICA covered earnings (capped)
- Annual pattern of quarters worked 1951 – 1977
- Total earnings from 1937 to 1950
- Detailed Earnings Record Extract 1978-2003
- Wages, Tips, and other compensation (Box 1) (uncapped)
- Deferred Wages (Box 13) – for example 401(k) contributions
- FICA covered earnings (Box 3)
- Summed across all employers

## SSA Benefit Variables

- Master Beneficiary Record
- Initial Benefit
- Year first received benefits
- type of benefit
- amount of benefit
- Current Benefit
- year first began receiving current benefit
- type of benefit
- amount of benefit

## Decennial Weight

- Purpose
- Make the combined SIPP panels representative of U.S. population on Apr. 1, 2000
- Method
- Divide Decennial 2000 population into same groups from which SIPP sample was drawn
- Locate each SIPP respondent in the Decennial
- Weight = #Decennial persons in group / #SIPP persons in group
- Adjust the weight to match official U.S. population totals based on 1996 SIPP demographic subgroups
- Create a synthetic weight for each synthetic implicate

**Notes:**
We created a weight for the SIPP-SSA file which makes the five pooled SIPP panels representative of the U.S. civilian, non-institutionalized population ages 18 and over on April 1st, 2000.  To determine how many people in this U.S. population each SIPP respondent represented, we divided the Decennial population into the same groups from which the SIPP sample was drawn, according to the SIPP sampling plan.  We then located each SIPP respondent in the Decennial population.  The weight for each SIPP person was calculated separately for each group as the total number of Decennial persons in the group divided by the total number of SIPP persons in each group.  We then adjusted the weight so that population totals broken into subgroups by demographic variables (such as gender, age, race) matched the official U.S. population totals for those same subgroups.  The population subgroups used were the same as those used for adjusting the 1996 SIPP panel weights.  Next, we created a synthetic weight for each synthetic implicate.  For each person in the data, we calculated the probability of having each possible weight value.  Each person’s synthetic weight was then a draw from the distribution of possible weight values.  Finally, we tested the validity of the weight by calculating statistics for key SIPP and SSA variables and comparing them to the same statistics calculated from other independent sources.  

## Locating SIPP respondents in Decennial

- Match SIPP to Decennial by PIK (SSN)
- Locate remaining SIPP persons using probabilistic record linking
- Assign each SIPP person a set of Decennial candidates based on blocking variables (race, gender, etc)
- Choose a match from this set based on matching variables (birth month, # children, etc)

**Notes:**
To locate SIPP respondents in the Decennial, we linked the two files by PIK, a unique person identifier which replaces the SSN.  This link by PIK was successful for about 2/3 of the SIPP respondents.  For the remaining 1/3 of SIPP respondents, we chose a match from the Decennial for each SIPP person using a technique called probabilistic record linking.  In this technique, each SIPP person was first assigned a set of Decennial candidates (candidates for a match) whose values for certain variables like race and gender (called blocking variables) matched the SIPP person’s values exactly.  We then chose a match for each SIPP person from this set based on how closely the Decennial candidate's values for other variables like number of children (called matching variables) agreed with the SIPP person’s values for the same variables.  

## Synthetic Data Creation

- Purpose of synthetic data is to create micro data that can be used by researchers in the same manner as the original data while preserving the confidentiality of respondents’ identities
- Fundamental trade-off:  usefulness and analytic validity of data versus protection from disclosure
- Our goal:  not be able to re-identify anyone in the already released SIPP public use files while still preserving regression results

## Multiple Imputation Confidentiality Protection

- Denote confidential data by *Y * and disclosable data by *X. *
*Y *contains missing data so that *Y=*(*Y**obs **, Y**mis*) and *X *has no missing data.
- Use the posterior predictive distribution(PPD) p(*Y**mis**| Y**obs**,** **X*) to complete missing data and p(*Y| Y**m**,** **X*) to create synthetic data
- Data synthesis is same procedure as missing data imputation, just done for all observations
- Major emphasis is to find a good estimate of the PPD

## Testing Analytical Validity

- Run regressions on each synthetic implicate
- Average coefficients
- Combine standard errors using formulae that take account of average variance of estimates (within implicate variance) and differences in variance across estimates (between implicate variance).
- Run regressions on gold standard data
- Compare average synthetic coefficient and standard error to g.s. coefficient and s.e.
- Data are analytically valid if coefficient is unbiased and the same inferences are drawn

## Formulae:  Completed Data only

- Notation
- script *l* is index for missing data implicate
*m* is total number of missing data implicates
- Estimate from one completed implicate
- Average of statistic across implicates

## Formulae: Total VarianceBetween Variance – variation due to differences between implicates

- Total variance of average statistic
- Variance of the statistic across implicates:  between variance

## Formulae:  Within VarianceVariation due to differences within each implicate

- Variance of the statistic from each completed implicate
- Average variance of statistic:  within variance

## Formulae:  Synthetic and Completed Implicates

- Notation
- script *l* is index for missing data implicate
- script *k* is index for synthetic data implicate
*m* is total number of missing data implicates
*r* is total number of synthetic implicates per missing data implicate
- Estimate from one synthetic implicate
- Average of statistic across  synthetic implicates

## Formulae:  Grand Mean and Total Variance

- Average of statistic across all implicates
- Total variance of average statistic

## Formulae: Between VarianceVariation due to differences between implicates

- Variance of the statistic across missing data implicates:  between m implicate variance
- Variance of the statistic across synthetic data implicates:  between r implicate variance

## Formulae: Within VarianceVariation due to differences within each implicate

- Variance of the statistic on each implicate
- Average variance of statistic:  within variance
- Source: Reiter, *Survey Methodology* (2004): 235-42.

## Example:  Average AIME/AMW

- Estimate average on each of synthetic implicates
- AvgAIME(1,1) , AvgAIME(1,2) , AvgAIME(1,3) , AvgAIME(1,4) ,
- AvgAIME(2,1) , AvgAIME(2,2) , AvgAIME(2,3) , AvgAIME(2,4) ,
- AvgAIME(3,1) , AvgAIME(3,2) , AvgAIME(3,3) , AvgAIME(3,4) ,
- AvgAIME(4,1) , AvgAIME(4,2) , AvgAIME(4,3) , AvgAIME(4,4)
- Estimate mean for each set of synthetic implicates that correspond to one completed implicate
- AvgAIMEAVG(1) , AvgAIMEAVG(2) , AvgAIMEAVG(3) , AvgAIMEAVG(4)
- Estimate grand mean of all implicates
- AvgAIMEGRANDAVG

## Example (cont.)

- Between m implicate variance
- Between r implicate variance

## Example (cont.)

- Variance of mean from each implicate
- VAR[AvgAIME(1,1)] , VAR[AvgAIME(1,2)] , VAR[AvgAIME(1,3)] , VAR[AvgAIME(1,4)]
- VAR[AvgAIME(2,1)] , VAR[AvgAIME(2,2)] , VAR[AvgAIME(2,3)] , VAR[AvgAIME(2,4)]
- VAR[AvgAIME(3,1)] , VAR[AvgAIME(3,2)] , VAR[AvgAIME(3,3)] , VAR[AvgAIME(3,4)]
- VAR[AvgAIME(4,1)] , VAR[AvgAIME(4,2)] , VAR[AvgAIME(4,3)] , VAR[AvgAIME(4,4)]
- Within variance

## Example (cont.)

- Total Variance
- Use AvgAIMEGRANDAVG and Total Variance to calculate confidence intervals and compare to estimate from completed data

## SAS Programs

- Sample programs to calculate total variance and confidence intervals

## Results:  Average AIME

## Public Use of the SIPP Synthetic Beta

- Full version (16 implicates) released to the Cornell Virtual RDC
- Any researcher may use these data
- During the testing phase, all analyses must be performed on the Virtual RDC
- Census Bureau research team will run the same analysis on the completed confidential data
- Results of the comparison will be released to the researcher, Census Bureau, SSA, and IRS (after traditional disclosure avoidance analysis of the runs on the confidential data)

## Slide 31

## Slide 32

## Slide 33

## Methods for Estimating the PPD

- Sequential Regression Multivariate Imputation (SRMI) is a parametric method where PPD is defined as
- The BB is a non-parametric method of taking draws from the posterior predictive distribution of a group of variables that allows for uncertainty in the sample CDF
- We use BB for a few groups of variables with particularly complex relationships and use SRMI for all other variables

## SRMI Method Details

- Assume a joint density *p*(*Y,X,**θ*) that defines parametric relationships between all observed variables.
- Approximate the joint density by a sequence of conditional densities defined by generalized linear models.
- Same process for completing and synthesizing data
- Synthetic values of some             are draws from:where *Y**m**, X**m** *are completed data, and densities *p**k** *are  defined by an appropriate generalized linear model and prior.

## SRMI Details: KDE Transforms

- The SRMI models for continuous variables assume that they are conditionally normal
- This assumption is relaxed by performing a KDE-based transform of groups of related variables
- All variables in the group are transformed to normality, then the PPD is estimated
- The sampled values from PPD are inverse transformed back to the original distribution using the inverse cumulative distribution

## SRMI Example: Synthesizing Date of Birth 

- Divide individuals into homogeneous groups using stratification variables
- example:  male, black, age categories, education categories, marital status
- example:  decile of lifetime earnings distribution, decile of lifetime years worked distribution, worked previous year, worked current year
- For each group, estimate an independent linear regression of date of birth on other variables (not used for stratification) that are strongly related

## SRMI Example: Synthesizing Date of Birth

- Synthetic date of birth is a random variable
- Before analysis, it is transformed to normal using the KDE-based procedure
- Distribution has two sources of variation:
- variation in error term in regression model
- variation in estimated parameters: **’s and **2
- Synthetic values are draws from this distribution
- Synthetic values are inverse transformed back to the original distribution using the inverse cumulative distribution.

## Bayesian Bootstrap Method Details

- Divide data into homogeneous groups using similar stratification variables as in SRMI
- Within groups do a Bayesian bootstrap of all variables to be synthesized at the same time.
- n observations in a group, draw 1-n random variables from uniform (0,1) distribution
- let uo ... ui ... un define the ordering of the observations in the group
- ui – ui-1 is the probability of sampling observation i from the group to replace missing data or synthesize data in observation j
- conventional bootstrap, probability of sampling is 1/n

## Creating Synthetic Data

- Begin with base data set that contains only non-missing values
- Use BB to complete missing administrative data – i.e. find donor SSN based on non-missing SIPP variables
- Use SRMI to complete missing SIPP data
- Iterate 9 times – input for iteration 2 is completed data set from iteration 1
- On last iteration, run 4 separate processes to create 4 separate data sets or implicates

**Notes:**
Start with the variable with the least amount of missing data
Estimate the posterior predictive density using original non-missing observations for dependent variable and most up-to-date values for all independent variables.
take one draw for each missing observation based on most up-to-date independent variables.  
Using the updated values, proceed to the next variable in the list
Iterate this process until the regression estimates converge 

## Creating Synthetic Data, Cont.

- Synthesis is like one more iteration of data completion, except all observations are treated as missing
- Each completed implicate serves as a separate input file
- Run 16 separate processes to create 16 different synthetic data sets or implicates
- The separate processes to create implicates have different stratification variables
- Need enough implicates to produce enough variation to ensure that averages across the implicates will be close to “truth”

**Notes:**
At same time, don’t want to introduce too much between implicate variance which will produce very noisy estimates

## Features of our Synthesizing Routines

- Parent-child relationships
- foreign-born and decade arrive in US
- welfare participation and welfare amount
- presence of earnings, amount of earnings
- Restrictions on draws from PPD
- Some draws must be within a pre-specified range from the original value:  example MBA is +/- $50 of original value.
- impose maximum and minimum values on some variables