### DRAFT VERSION FOR 60 DAY NOTICE, WILL BE FINALIZED FOR 30 DAY

CONSUMER FINANCIAL PROTECTION BUREAU INFORMATION COLLECTION REQUEST

– SUPPORTING STATEMENT

Supporting Statement for Evaluations of Financial Capability Programs for Economically

Vulnerable Consumers : Two Randomized Control Trials



### (OMB CONTROL NUMBER: 3170

- XXXX)

### 1 Table of C ontents



### 1. Respondent Universe and Sampling Methods

................................ ................................ ........ 2

2. Information Collection Procedures

................................ ................................ ......................... 2 3. Methods to Maximize Response Rates and Address Issues of Non

- Response

...................... 5 4. Testing of Procedures or Methods

................................ ................................ ........................... 6

5 Contact Information for Statistical Aspects of the Design

................................ ....................... 7

Contact information:

................................ ................................ ................................ .................... 7

### 2 B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent U niverse and S ampling M ethods

We propose using randomized controlled trials to evaluate the effectiveness of financial capability interventions; specifically either bundled products or integrated service delivery programs aimed at helping economically

- vulnerable consumers build savings and credit. To date there are few rigorous randomized evaluations of financial capability programs in the United States. These evaluations have important implications for policymakers and practitioners alike.

Our samples will consist of low

- income consumers; with treatment randomized at the individual level after stratification along various demographic features

. 1 The individuals will come from the same geographic area in the United States or from a national sample of program participants (the location depending on which evaluations are selected), and will be selected from the pool of clients of the partner

financi al institution. Through random sampling among the target population (the economically vulnerable), the methods will ensure that the sample is representative of the target audience for the

treatments. Projected sample sizes depend on evaluation

- specific pow er calculations, as detailed below.



### We will obtain informed consent from each participant that lays out the voluntary nature of their participation and explains how their participation will aide in our research efforts. We will use

administrative data for our analysis for most if not all variables (also detailed below) ; some surveying may be required in some of the evaluations when administrative data is not available on outcomes of

interest . In some cases we will conduct online surveying, and in other cases surveying will be

administered in

- person. For example, for the valuation designs that look at integrated service delivery models, we will be able to take advantage of in

- person interviews at a social service provider to which study participants have built

- in incentive to regularly return (e

. g

. , their benefits check is distributed in this site). In the cases of online surveys, survey length will be less than 15 minutes and non

- monetary incentives may be offered. For example, those who consent to be in the study can be entered into a raffle to win a gift cer tificate to a local restaurant. Response rates will be dependent on the type of survey and point of offer. Further efforts to increase response rates are detailed below.

2. Information Collection P rocedures

Statistical method s for stratification and sample selecti on

The sample will ultimately be determined by the participating institutions . G iven that we are working

1 Stratification divides the sample into different groups and then randomly samples within each subgroup. Stratification increases precision of estimates. In order to increase precision, the sampling method should maximize variation across groups but minimize variation within groups. For that reason, we propose to use age, race, and income to stratify the groups. Serfling, R.J. (1968). "Approximately optimal

stratification.” Journal of th e American Statistical Association 63(324) provides a summary of the underlying theory. Recent work has gone into optimal stratification: see, for example, Baillargeon, Sophie

and Louis

- Paul Rivest (2009). “A General Algorithm for Univariate Stratification .” International

Statistical Review 77(3). Examples of stratification in the related research include Mandell, L. and L. Klein (2007). “Motivation and Financial Literacy.” Financial Services Review , 16(2007); Cole, S., T. Samplson, and B. Zia (2011). “Pric es or Knowledge? What Drives Demand for Financial Services in Emerging Markets?” Journal of Finance 66(6), and Fryer, R. (2011). “Financial Incentives and Student

Achievement: Evidence from Randomized Trials” Quarterly Journal of Economics, 126.

### 3 with existing products and services, we will randomly sample existing or potential customers or

beneficiaries of participating institutions . We will randomize into treatment and control groups within that sample after stratification

. For evalu ations where the sample is well

- defined prior to treatment (for example, if the sample is composed of all economically vulnerable members of the participation institution), we will assign treatment status randomly within strata. 2 For evaluations where the sample is conditional, such as conditional on visiting the participating institution, we will randomize either by previously assigning all potential entrants treatment status (similar to the non

- conditional stratified random assignment previously described) or using entry time across strata. 3

The sample will be stratified on one or more d emographic characteristics recorded through baseline

survey or administrative data , such as gender, race, and/or a continuous variable such as median income.

Randomization will occur at the individual level. In all regressions, we will control for survey l ocation, survey date, and surveyor, and report heteroskedasticity

- robust (Huber

- White) standard errors. 4 We will conduct orthogonality checks to test whether the treatment and control samples are balanced on observed

covariates at baseline in order to supp ort using the experimental results to compare outcome measures and interpret differences causally as impacts of the intervention. 5

Estimation procedure

The explicit research design will ultimately depend upon which evaluations are selected. However, many common estimation techniques will be employed, regardless of which evaluation is ultimately

2 For exam ple, we will assign each person a randomly generated number. We will then separate each person into different demographic strata, such as male and female, and then within each strata estimate the median random number. Persons whose random numbers are greater than the strata median will be assigned treatment group status, while persons below will be assigned control group status. This procedure is the same whether we randomize over tre atment, as is used here as an example, or over

incentives to receive treatment.

3 For example, each stratum will have a running switching of assignments to treatment or control groups. The first person in group 1 would be assigned treatment; the next perso n in group 1 would be assigned control, and so forth. Another option is to reweight the probability of assignments by the balance of treatment assignments within each stratum at time of entry. See Zhao, W., J. Ciolino, and Y. Palesch (2010), “Step

- Forward Randomization in Multicenter Emergency Treatment Clinical Trials.” Academic Emergency Medicine , 17(6) and Friedman LM

, Furberg CD

, DeMets DL

. (1998) Fundamentals of

Clinical Trials . New York: Springer Science Business Media, LLC.

4 Huber, P. J. (1967), “Th e Behavior of Maximum Likelihood Estimates Under Non

- standard Conditions,” in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability

, Berkeley: University of California Press; White, H. (1980), “A Heteroskedastic

- Consistent Covariance Matrix Estimator and a Direct Test of Heteroskedasticity,” Econometrica , 48; Eicker, F. (1963), “Asymptotic Normality and Consistency of the Least Squares Estimator for Families of Linear Regressions,” Annals of Mathematical Statistics , 34; and Eicker, F. (1967), “Limit Theorems for Regressions With Unequal and Dependent Errors,” in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability , Berkeley, CA: University of California Press. Heteroskedasticity, or non

- const ant variance in unobserved variables, can bias the estimated standard errors of regression coefficients. Using robust standard errors corrects for this potential bias, and is the more conservative approach for hypothesis testing.

5 We will compare means of treated and non

- treated individuals in the sample across several characteristics. Specifically, we will perform an Analysis of Variance, which tests multiple joint hypotheses of differences in means. Doing so will validate the ran dom sampling procedure, should we (as anticipated) fail to reject the joint null of no differences in the means between the treated and non

- treated.

### 4 implemented. We will use econometric regression techniques, s uch as difference

- in

- difference estimators using O rdinary Le ast S quares, quantile or median regressions , or fixed effects. Difference

- in

- difference estimators estimate the impact of an intervention by comparing pre

- and post

- treatment trends for those

trea ted (the treatment group) and those not treated (the control group). 6 Ordinary Least Squares approximates the conditional mean of the dependent (outcome) variable , whereas median or quantile regression estimates the conditional median (or other quantiles) of the outcome variable. 7 Fixed effects

models control for individua l level heterogeneity, reducing the threat of omitted variables bias. 8 Our analysis will account for circumstances that contaminate the initial randomization, such as attrition, spillover s and partial compliance

, through the use of appropriate econometric methods. These specifics will depend largely upon the program evaluation selected for experimentation.

Degree of accuracy needed for the purpose describe d in the justification

A suffic ient sample size is needed in order to precisely estimate whether interventions impact intended outcomes. We will estimate the number of individuals needed in our sample by conducting power

calculations. The power of a statistical test is the probability t hat the null hypothesis will be rejected conditional on it being incorrect. As the selection of a test statistic not only affects the power of the test, but the size (the likelihood of rejecting the null hypothesis when it is true), the test statistic must be selected in a way to minimize size while maximizing power. Both directly depend on the sample size, as larger sample sizes improve the ability to detect effects and choose between hypotheses. Unfortunately, larger sample sizes are also more costly to c ollect. To that end, power calculations help guide researchers to appropriately select sample sizes that will have the required size while also having sufficient power to detect some minimum detectable treatment effect (explained more below). 9 Specifically,

n ≥ � z α − Φ − 1 ( 1

− τ )

MDE σ β ⁄ � 2

z α is the test statistic associated with the selected size of the test. The research team will choose a statistical significance level for the evaluation, that is, the probability of detecting an effect when in fact the intervention has no effect. This is conventionally set at 5% or 10%, since this is the probability of a

type

- 1 (false positive) error generally accepted as significant in peer

- reviewed journal articles

. 10 The



### 6 Donald, S. and K. Lang (2007). “Inference with Difference

- in

- Differences and Other Panel Data.” The

Revi ew of Economics and Statistics , 89(2).

7 Ordinary Least Squares and the various methods that employ it (such as Difference

- in

- Difference and Fixed Effects estimators) estimate the conditional mean of the outcome given treatment and control variables. Quant ile regression allows for analysis of the expected effects of the treatment and control variables on the outcome for observations away from the mean (non

- average); for example, we might be interested in the impact of treatment on an individual in the 20 th percentile, or quantile, of the outcome variable. Quantile regression allows us to do that. See Buchinsky, M. (1998). “Recent Advances in Quantile Regression: A Practical Guideline for Empirical Research.” Journal of Human Resources , 33(1).

8 Hausman, J. ( 1978). “Specification Tests in Econometrics.” Econometrica , 46(6).

9 Bloom, Howard S. (2006). The Core Analytics of Randomized Experiments for Social Research. MDRC Working Paper. Bloom presents the results in terms of the Minimum Detectable Effect. Given our exposition purposes, it is here expressed in terms of the sample size

10 The equation is written for a one

- tailed test, because smaller sample sizes are needed for the same test size. In that case, α is the size, or significance level of the test. For a two

- tailed test, the equation would replace z α with z α / 2 to retain the same size.

### 5 power of the research design is the probability that, for a given effect size and a given statistical significance level, we will be able to reject the hypothesis of zero effect if the product or service provision

does indeed have a real effect . The power is represented by τ

. Φ − 1 is the i nverse cumulative density

function for the standard normal distribution.

The research team will then determine the minimum detectable treatment effect (MDE) , the smallest effect size that we would possibly want to detect. It is important to estimate this conservatively so that the analysis can detect relevant changes in outcomes of interest, even if lower than expected. Programs with a small effect size require a larger sample and if the sample size is underestimated there may be too much

noise in the data to detect a non

- zero effect (even if there is one). Determining the minimum detectable effect will be an important process that requires input from both partners and research team

. As a rule of thumb for policy

- relevant evaluation, this should be the smallest effect size that is large enough such that the intervention would be cost effective if it were to be scaled up. To arrive at the cost, we will enter into discussions with the participating institutions, and determine the minimum detectable effec t from this.

To estimate n, we also need an estimate of σ β , the standard erro r of the estimated program effect. To do
so, the research team will first establish a mean and variance of the outcome in the absence of the evaluation, after controlling for possible covariates and/or stratification. To estimate these values w e will use previously collected data from national samples , for instance from the Survey of Consumer Finances

or the RAND American Life Panel or
, where possible

, from market research or e xisting data from the programs themselves. When randomization is over incentives to participate and not treatment itself (as in this case), we also must estimate the correlation between treatment and incentives, or in other words, use guidance from externa l literature on how responsive individuals will be to incentives to participate.

Using these inputs and the equation above, t he research team will determine the level of power for which they are aiming ex

- ante and determine the necessary sample size

.

Unusual problems requiring specialized sampling procedures

We do not foresee any unusual problems requiring specialized sampling procedures.

Any use of periodic (less frequent than annual) data coll ection cycles to reduce burden

Some of the potential e valuation designs will rely on administrative data only

, in order to reduce data collection burdens. Soft credit report pulls will be conducted with the consent of consumers in order to learn about changes in credit scores. Account information will be repo rted from credit unions, banks and prepaid card providers where applicable. In other cases , surveys will be administered (in order to collect information on outstanding debt , self

- reported savings, and information on financial attitudes, financial knowledg e, and behavioral indicators ) . W here surveys are necessary in order to investigate research questions, w e will examine changes in outcomes for both treatment and control groups based on baseline and endline surveys, and midline surveys where necessary , but will make data collection as efficient and

un

- burdensome as possible, based in part on previous experience of both the RAND Corporation and IPA, and in part through the piloting process detailed below

.

3. Methods to Maximize Response Rates and Address Issues of Non

- R esponse

G iven that the mandate of this project is to evaluate existing products and services, for several of the potential evaluations, we may consider “encouragement designs,” wherein consumers are encouraged to take up a product or servic e that is already available to them. In order to experimentally estimate the impact of treatment offer, we will measure the average effect of “intent to treat” (ITT). This effect differs

### 6 from “treatment on the treated” (TOT). Those who are non

- compliant wi th the treatment of encouragement, i

. e

. do not take up the product, are accounted for under this estimate , as are “crossovers,” members of the control group (not encouraged) who take up the product or service anyways. This is a

relevant consideration for p olicymaking decisions as the products and services we evaluate are voluntary, and not mandated; furthermore, even mandated programs do not have complete compliance. 11 In the case of randomizing over incentives, we may estimate the effects of the program usi ng instrumental variable regression.

Those who repeatedly fail to respond to invitations for follow up surveys, on the other hand, will be

considered attrite rs and dropped from the sample, which is another reason that sample size calculations (as detailed above) must be conservative. 12 We will deal with systematic non

- response with state

- of the art statistical techniques that deal with non

- random sample attrition

. In order to maximize survey response rates, we will consider non

- monetary incentives where applicable , including small value gift cards , such as a gift card to a local restaurant . We will also consider additional outreach, e

. g

. , over the phone or through direct mailings, to increase response rates. With some of the potential evaluation designs that look at integrated service delivery models, we will be able to take advantage of in

- person interviews at the workplace or social service provider to which peop le have built

- in incentive to regularly return ( i.e., their benefits check is administered in this site). Where surveys would need to be administrated online, survey length will be minimal to maximize response and there may be a potential for non

- monetary incentive. Furthermore, as detailed above

, s ome of the potential evaluation designs rely exclusively on administrative data so as to eliminate the problem of non

- response. Administrative data includes information such as account balance and information pu lled from credit reports.

4. Test ing of Procedures or Methods

M any of the plans will incorporate survey (test) data, with identical questions asked of more than ten respondents ; however, given that at this juncture we have not selected evaluation plans, we are not yet able to provide specific questions that will be asked of survey participants

.

All surveys will be pre

- tested or piloted to refine and streamline the data collection process, making sure

that the most relevant information is collected from participants in the most efficient way possible. The internal validity of the results of any evaluation is of course dependent on the quality of the data collected; confirming that participants understand questions c orrectly and that the structure of survey questions leads participants to accurately self

- report to the highest degree possible is key. Piloting is also important in order to best integrate survey operations with implementing partner operations and make su re that the evaluation is minimizing any additional burden on practitioner partners. Additionally, piloting allows for early monitoring of quality data collection of surveyors.

During and a fter data collection, t he quality of the data will be systematical ly verified through “back

checking” (i.e.

, performing an audit by resurveying of a random subset of individuals who were already surveyed to ensure that responses are consistent), which is done by a second and independent team of

11 Bloom, Howard S. (2006). The Core Analytics of Randomized Experiments for Social Research. MDRC Working Paper. Bloom presents the results in terms of the Minimum Dete ctable Effect. Given our exposition purposes, it is here expressed in terms of the sample size

12 We will examine whether attriters are random across selection into treatment using Analysis of Variance. This is common practice; see for example Behman , J. et al. (2012) “Aligning Learning Incentives of Students and Teachers: Results from a Social Experiment in Mexican High Schools.” PIER Working Paper #13

- 004. See Bloom, Howard S. (200 3 ). “Sample Design for an Evaluation of the Reading First Program,” M DRC Working Paper , for a discussion on the implications of attrition.

### 7 surveyors.

Data entry i s typically undertaken using software such as EPI

- DATA, SQL or MS

- ACCESS which have the capability of conducting range and consistency checks. The data is double

- entered, where each element of data is entered twice by separate data entry operators. Both r ounds of entry are then compared, and all deviations between the two rounds must be re

- entered and scrutinized. Data entry should be conducted during the field work, i.e. while survey teams are still on the ground, in order to minimize delays and to allow simple quality checks to be undertaken while field teams are still in place. Any errors are resolved by referring to the original questionnaire or re

- surveying depending on the nature of the error. The entered data is then checked for completeness and corr ectness, by conducting a third entry on a sub

- sample (10% of the total sample) , which allows calculating the error rate of the final dataset. The data is typically re

- entered if the error rate is higher than 0.3%.

5 Contact Information for Statistical Aspects of the Design

This study is being conducted by two non

- profit contractor s , The RAND Corporation and Innovations for Poverty Action , under contract to the Consumer Financial Protection Bureau

.

Contact information:

CFPB:

Sarah Bainton Kahn Policy Analyst, Financial Empowerment Consumer Education & Engagement

Consumer Financial Protection Bureau

Office: 202

- 435

- 7030 sarah.bainton@cfpb.gov

The RAND Corporation: Principal Investigator: Kata Mihaly

, PhD

Economist RAND Corporation 1200 South Hayes Street Arlington, VA 22202

Office: 703

- 413

- 1100 x5393

Email: kmihaly@rand.org

Matthew Baird , PhD

Econometrician RAND Corporation 1200 South Hayes Street Arlington, VA 22202

Office: 703

- 413

- 1100 x 7376

Email: mbaird @rand.org

Innovations for Poverty Action: Principal Investigator: Dean Karlan, PhD

Professor of Economics Yale University

### 8 P.O. Box 208269 New Haven, CT 06520 Office: (203) 432

- 4479

Email: dean.karlan@yale.edu