DRAFT VERSION FOR 60 DAY NOTICE, WILL BE FINALIZED FOR 30 DAY

CONSUMER FINANCIAL PROTECTION BUREAU

INFORMATION COLLECTION REQUEST – SUPPORTING STATEMENT

Supporting Statement for Evaluations of Financial Capability Programs for Economically 

Vulnerable Consumers: Two Randomized Control Trials 

(OMB CONTROL NUMBER: 3170-XXXX)
1

Table of Contents

1. Respondent Universe and Sampling Methods........................................................................ 2

2. Information Collection Procedures......................................................................................... 2

3. Methods to Maximize Response Rates and Address Issues of Non-Response ...................... 5

4. Testing of Procedures or Methods........................................................................................... 6

5 Contact Information for Statistical Aspects of the Design ....................................................... 7

Contact information:.................................................................................................................... 7
2

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

We propose using randomized controlled trials to evaluate the effectiveness of financial capability 

interventions; specifically either bundled products or integrated service delivery programs aimed at 

helping economically-vulnerable consumers build savings and credit. To date there are few rigorous 

randomized evaluations of financial capability programs in the United States. These evaluations have 

important implications for policymakers and practitioners alike.

Our samples will consist of low-income consumers; with treatment randomized at the individual level

after stratification along various demographic features.

1 The individuals will come from the same 

geographic area in the United States or from a national sample of program participants (the location 

depending on which evaluations are selected), and will be selected from the pool of clients of the partner 

financial institution. Through random sampling among the target population (the economically 

vulnerable), the methods will ensure that the sample is representative of the target audience for the 

treatments. Projected sample sizes depend on evaluation-specific power calculations, as detailed below. 

We will obtain informed consent from each participant that lays out the voluntary nature of their 

participation and explains how their participation will aide in our research efforts. We will use 

administrative data for our analysis for most if not all variables (also detailed below); some surveying 

may be required in some of the evaluations when administrative data is not available on outcomes of 

interest. In some cases we will conduct online surveying, and in other cases surveying will be 

administered in-person. For example, for the valuation designs that look at integrated service delivery 

models, we will be able to take advantage of in-person interviews at a social service provider to which 

study participants have built-in incentive to regularly return (e.g., their benefits check is distributed in this 

site). In the cases of online surveys, survey length will be less than 15 minutes and non-monetary 

incentives may be offered. For example, those who consent to be in the study can be entered into a raffle 

to win a gift certificate to a local restaurant. Response rates will be dependent on the type of survey and 

point of offer. Further efforts to increase response rates are detailed below. 

2. Information Collection Procedures

Statistical methods for stratification and sample selection

The sample will ultimately be determined by the participating institutions. Given that we are working 

 

1 Stratification divides the sample into different groups and then randomly samples within each subgroup. 

Stratification increases precision of estimates. In order to increase precision, the sampling method should 

maximize variation across groups but minimize variation within groups. For that reason, we propose to 

use age, race, and income to stratify the groups. Serfling, R.J. (1968). "Approximately optimal 

stratification.” Journal of the American Statistical Association 63(324) provides a summary of the 

underlying theory. Recent work has gone into optimal stratification: see, for example, Baillargeon, Sophie 

and Louis-Paul Rivest (2009). “A General Algorithm for Univariate Stratification.” International 

Statistical Review 77(3). Examples of stratification in the related research include Mandell, L. and L. 

Klein (2007). “Motivation and Financial Literacy.” Financial Services Review, 16(2007); Cole, S., T. 

Samplson, and B. Zia (2011). “Prices or Knowledge? What Drives Demand for Financial Services in 

Emerging Markets?” Journal of Finance 66(6), and Fryer, R. (2011). “Financial Incentives and Student 

Achievement: Evidence from Randomized Trials” Quarterly Journal of Economics, 126.
3

with existing products and services, we will randomly sample existing or potential customers or 

beneficiaries of participating institutions. We will randomize into treatment and control groups within that 

sample after stratification. For evaluations where the sample is well-defined prior to treatment (for 

example, if the sample is composed of all economically vulnerable members of the participation 

institution), we will assign treatment status randomly within strata.2 For evaluations where the sample is 

conditional, such as conditional on visiting the participating institution, we will randomize either by 

previously assigning all potential entrants treatment status (similar to the non-conditional stratified 

random assignment previously described) or using entry time across strata. 3

The sample will be stratified on one or more demographic characteristics recorded through baseline 

survey or administrative data, such as gender, race, and/or a continuous variable such as median income. 

Randomization will occur at the individual level. In all regressions, we will control for survey location, 

survey date, and surveyor, and report heteroskedasticity-robust (Huber-White) standard errors.4 We will 

conduct orthogonality checks to test whether the treatment and control samples are balanced on observed 

covariates at baseline in order to support using the experimental results to compare outcome measures and 

interpret differences causally as impacts of the intervention.5

Estimation procedure

The explicit research design will ultimately depend upon which evaluations are selected. However, many 

common estimation techniques will be employed, regardless of which evaluation is ultimately 

 

2 For example, we will assign each person a randomly generated number. We will then separate each 

person into different demographic strata, such as male and female, and then within each strata estimate 

the median random number. Persons whose random numbers are greater than the strata median will be 

assigned treatment group status, while persons below will be assigned control group status. This 

procedure is the same whether we randomize over treatment, as is used here as an example, or over 

incentives to receive treatment.

3 For example, each stratum will have a running switching of assignments to treatment or control groups. 

The first person in group 1 would be assigned treatment; the next person in group 1 would be assigned 

control, and so forth. Another option is to reweight the probability of assignments by the balance of 

treatment assignments within each stratum at time of entry. See Zhao, W., J. Ciolino, and Y. Palesch 

(2010), “Step-Forward Randomization in Multicenter Emergency Treatment Clinical Trials.” Academic 

Emergency Medicine, 17(6) and Friedman LM, Furberg CD, DeMets DL. (1998) Fundamentals of

Clinical Trials. New York: Springer Science Business Media, LLC.

4 Huber, P. J. (1967), “The Behavior of Maximum Likelihood Estimates Under Non-standard 

Conditions,” in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 

Berkeley: University of California Press; White, H. (1980), “A Heteroskedastic-Consistent Covariance 

Matrix Estimator and a Direct Test of Heteroskedasticity,” Econometrica, 48; Eicker, F. (1963), 

“Asymptotic Normality and Consistency of the Least Squares Estimator for Families of Linear 

Regressions,” Annals of Mathematical Statistics, 34; and Eicker, F. (1967), “Limit Theorems for 

Regressions With Unequal and Dependent Errors,” in Proceedings of the Fifth Berkeley Symposium on 

Mathematical Statistics and Probability, Berkeley, CA: University of California Press. 

Heteroskedasticity, or non-constant variance in unobserved variables, can bias the estimated standard 

errors of regression coefficients. Using robust standard errors corrects for this potential bias, and is the 

more conservative approach for hypothesis testing.

5 We will compare means of treated and non-treated individuals in the sample across several 

characteristics. Specifically, we will perform an Analysis of Variance, which tests multiple joint 

hypotheses of differences in means. Doing so will validate the random sampling procedure, should we (as 

anticipated) fail to reject the joint null of no differences in the means between the treated and non-treated.
4

implemented. We will use econometric regression techniques, such as difference-in-difference estimators 

using Ordinary Least Squares, quantile or median regressions, or fixed effects. Difference-in-difference 

estimators estimate the impact of an intervention by comparing pre- and post-treatment trends for those 

treated (the treatment group) and those not treated (the control group).6 Ordinary Least Squares 

approximates the conditional mean of the dependent (outcome) variable, whereas median or quantile 

regression estimates the conditional median (or other quantiles) of the outcome variable.7 Fixed effects 

models control for individual level heterogeneity, reducing the threat of omitted variables bias.8

 Our 

analysis will account for circumstances that contaminate the initial randomization, such as attrition, 

spillovers and partial compliance, through the use of appropriate econometric methods. These specifics 

will depend largely upon the program evaluation selected for experimentation.

Degree of accuracy needed for the purpose described in the justification

A sufficient sample size is needed in order to precisely estimate whether interventions impact intended 

outcomes. We will estimate the number of individuals needed in our sample by conducting power 

calculations. The power of a statistical test is the probability that the null hypothesis will be rejected 

conditional on it being incorrect. As the selection of a test statistic not only affects the power of the test, 

but the size (the likelihood of rejecting the null hypothesis when it is true), the test statistic must be 

selected in a way to minimize size while maximizing power. Both directly depend on the sample size, as 

larger sample sizes improve the ability to detect effects and choose between hypotheses. Unfortunately, 

larger sample sizes are also more costly to collect. To that end, power calculations help guide researchers 

to appropriately select sample sizes that will have the required size while also having sufficient power to 

detect some minimum detectable treatment effect (explained more below).9 Specifically, 

n ≥ �

zα − Φ−1(1 − τ)

MDE⁄σβ

�

2

zα is the test statistic associated with the selected size of the test. The research team will choose a 

statistical significance level for the evaluation, that is, the probability of detecting an effect when in fact 

the intervention has no effect. This is conventionally set at 5% or 10%, since this is the probability of a 

type-1 (false positive) error generally accepted as significant in peer-reviewed journal articles.

10 The 

 

6 Donald, S. and K. Lang (2007). “Inference with Difference-in-Differences and Other Panel Data.” The 

Review of Economics and Statistics, 89(2).

7 Ordinary Least Squares and the various methods that employ it (such as Difference-in-Difference and 

Fixed Effects estimators) estimate the conditional mean of the outcome given treatment and control 

variables. Quantile regression allows for analysis of the expected effects of the treatment and control 

variables on the outcome for observations away from the mean (non-average); for example, we might be 

interested in the impact of treatment on an individual in the 20th percentile, or quantile, of the outcome 

variable. Quantile regression allows us to do that. See Buchinsky, M. (1998). “Recent Advances in 

Quantile Regression: A Practical Guideline for Empirical Research.” Journal of Human Resources, 33(1).

8 Hausman, J. (1978). “Specification Tests in Econometrics.” Econometrica, 46(6).

9 Bloom, Howard S. (2006). The Core Analytics of Randomized Experiments for Social Research. 

MDRC Working Paper. Bloom presents the results in terms of the Minimum Detectable Effect. Given our 

exposition purposes, it is here expressed in terms of the sample size

10 The equation is written for a one-tailed test, because smaller sample sizes are needed for the same test 

size. In that case, α is the size, or significance level of the test. For a two-tailed test, the equation would 

replace zα with zα/2 to retain the same size.
5

power of the research design is the probability that, for a given effect size and a given statistical 

significance level, we will be able to reject the hypothesis of zero effect if the product or service provision

does indeed have a real effect. The power is represented by τ. Φ−1 is the inverse cumulative density 

function for the standard normal distribution. 

The research team will then determine the minimum detectable treatment effect (MDE), the smallest 

effect size that we would possibly want to detect. It is important to estimate this conservatively so that the 

analysis can detect relevant changes in outcomes of interest, even if lower than expected. Programs with a 

small effect size require a larger sample and if the sample size is underestimated there may be too much 

noise in the data to detect a non-zero effect (even if there is one). Determining the minimum detectable 

effect will be an important process that requires input from both partners and research team. As a rule of 

thumb for policy-relevant evaluation, this should be the smallest effect size that is large enough such that 

the intervention would be cost effective if it were to be scaled up. To arrive at the cost, we will enter into 

discussions with the participating institutions, and determine the minimum detectable effect from this.

To estimate n, we also need an estimate of σβ , the standard error of the estimated program effect. To do 

so, the research team will first establish a mean and variance of the outcome in the absence of the 

evaluation, after controlling for possible covariates and/or stratification. To estimate these values we will 

use previously collected data from national samples, for instance from the Survey of Consumer Finances

or the RAND American Life Panel or, where possible, from market research or existing data from the 

programs themselves. When randomization is over incentives to participate and not treatment itself (as in 

this case), we also must estimate the correlation between treatment and incentives, or in other words, use 

guidance from external literature on how responsive individuals will be to incentives to participate.

Using these inputs and the equation above, the research team will determine the level of power for which 

they are aiming ex-ante and determine the necessary sample size.

Unusual problems requiring specialized sampling procedures

We do not foresee any unusual problems requiring specialized sampling procedures.

Any use of periodic (less frequent than annual) data collection cycles to reduce burden

Some of the potential evaluation designs will rely on administrative data only, in order to reduce data 

collection burdens. Soft credit report pulls will be conducted with the consent of consumers in order to 

learn about changes in credit scores. Account information will be reported from credit unions, banks and 

prepaid card providers where applicable. In other cases, surveys will be administered (in order to collect

information on outstanding debt, self-reported savings, and information on financial attitudes, financial 

knowledge, and behavioral indicators). Where surveys are necessary in order to investigate research 

questions, we will examine changes in outcomes for both treatment and control groups based on baseline 

and endline surveys, and midline surveys where necessary, but will make data collection as efficient and 

un-burdensome as possible, based in part on previous experience of both the RAND Corporation and IPA, 

and in part through the piloting process detailed below.

3. Methods to Maximize Response Rates and Address Issues of Non-Response

Given that the mandate of this project is to evaluate existing products and services, for several of the 

potential evaluations, we may consider “encouragement designs,” wherein consumers are encouraged to 

take up a product or service that is already available to them. In order to experimentally estimate the 

impact of treatment offer, we will measure the average effect of “intent to treat” (ITT). This effect differs 
6

from “treatment on the treated” (TOT). Those who are non-compliant with the treatment of 

encouragement, i.e. do not take up the product, are accounted for under this estimate, as are “crossovers,” 

members of the control group (not encouraged) who take up the product or service anyways. This is a 

relevant consideration for policymaking decisions as the products and services we evaluate are voluntary, 

and not mandated; furthermore, even mandated programs do not have complete compliance.11 In the case 

of randomizing over incentives, we may estimate the effects of the program using instrumental variable 

regression.

Those who repeatedly fail to respond to invitations for follow up surveys, on the other hand, will be 

considered attriters and dropped from the sample, which is another reason that sample size calculations 

(as detailed above) must be conservative.12 We will deal with systematic non-response with state-of the 

art statistical techniques that deal with non-random sample attrition. In order to maximize survey response 

rates, we will consider non-monetary incentives where applicable, including small value gift cards, such 

as a gift card to a local restaurant. We will also consider additional outreach, e.g., over the phone or 

through direct mailings, to increase response rates. With some of the potential evaluation designs that 

look at integrated service delivery models, we will be able to take advantage of in-person interviews at the 

workplace or social service provider to which people have built-in incentive to regularly return (i.e., their 

benefits check is administered in this site). Where surveys would need to be administrated online, survey 

length will be minimal to maximize response and there may be a potential for non-monetary incentive.

Furthermore, as detailed above, some of the potential evaluation designs rely exclusively on 

administrative data so as to eliminate the problem of non-response. Administrative data includes 

information such as account balance and information pulled from credit reports.

4. Testing of Procedures or Methods

Many of the plans will incorporate survey (test) data, with identical questions asked of more than ten 

respondents; however, given that at this juncture we have not selected evaluation plans, we are not yet 

able to provide specific questions that will be asked of survey participants.

All surveys will be pre-tested or piloted to refine and streamline the data collection process, making sure 

that the most relevant information is collected from participants in the most efficient way possible. The 

internal validity of the results of any evaluation is of course dependent on the quality of the data collected; 

confirming that participants understand questions correctly and that the structure of survey questions 

leads participants to accurately self-report to the highest degree possible is key. Piloting is also important 

in order to best integrate survey operations with implementing partner operations and make sure that the 

evaluation is minimizing any additional burden on practitioner partners. Additionally, piloting allows for 

early monitoring of quality data collection of surveyors.

During and after data collection, the quality of the data will be systematically verified through “backchecking” (i.e., performing an audit by resurveying of a random subset of individuals who were already 

surveyed to ensure that responses are consistent), which is done by a second and independent team of 

 11 Bloom, Howard S. (2006). The Core Analytics of Randomized Experiments for Social Research. 

MDRC Working Paper. Bloom presents the results in terms of the Minimum Detectable Effect. Given our 

exposition purposes, it is here expressed in terms of the sample size

12 We will examine whether attriters are random across selection into treatment using Analysis of 

Variance. This is common practice; see for example Behman, J. et al. (2012) “Aligning Learning 

Incentives of Students and Teachers: Results from a Social Experiment in Mexican High Schools.” PIER 

Working Paper #13-004. See Bloom, Howard S. (2003). “Sample Design for an Evaluation of the 

Reading First Program,” MDRC Working Paper, for a discussion on the implications of attrition.
7

surveyors. 

Data entry is typically undertaken using software such as EPI-DATA, SQL or MS-ACCESS which have 

the capability of conducting range and consistency checks. The data is double-entered, where each 

element of data is entered twice by separate data entry operators. Both rounds of entry are then 

compared, and all deviations between the two rounds must be re-entered and scrutinized. Data entry 

should be conducted during the field work, i.e. while survey teams are still on the ground, in order to 

minimize delays and to allow simple quality checks to be undertaken while field teams are still in place. 

Any errors are resolved by referring to the original questionnaire or re-surveying depending on the nature 

of the error. The entered data is then checked for completeness and correctness, by conducting a third 

entry on a sub-sample (10% of the total sample), which allows calculating the error rate of the final 

dataset. The data is typically re-entered if the error rate is higher than 0.3%. 

5 Contact Information for Statistical Aspects of the Design

This study is being conducted by two non-profit contractors, The RAND Corporation and Innovations for 

Poverty Action, under contract to the Consumer Financial Protection Bureau. 

Contact information: 

CFPB:

Sarah Bainton Kahn

Policy Analyst, Financial Empowerment

Consumer Education & Engagement

Consumer Financial Protection Bureau

Office: 202-435-7030

sarah.bainton@cfpb.gov

The RAND Corporation:

Principal Investigator: Kata Mihaly, PhD

Economist

RAND Corporation

1200 South Hayes Street

Arlington, VA 22202

Office: 703-413-1100 x5393

Email: kmihaly@rand.org

Matthew Baird, PhD

Econometrician

RAND Corporation

1200 South Hayes Street

Arlington, VA 22202

Office: 703-413-1100 x7376

Email: mbaird@rand.org

Innovations for Poverty Action:

Principal Investigator: Dean Karlan, PhD

Professor of Economics

Yale University
8

P.O. Box 208269

New Haven, CT 06520

Office: (203) 432-4479

Email: dean.karlan@yale.edu