Page 1 of 9

CONSUMER FINANCIAL PROTECTION BUREAU

INFORMATION COLLECTION REQUEST – SUPPORTING STATEMENT

PART B

DEVELOPMENT OF METRICS TO MEASURE FINANCIAL WELL-BEING OF 

WORKING-AGE AND OLDER AMERICAN CONSUMERS

(OMB CONTROL NUMBER: 3170-XXXX)

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Selection Methods

The Consumer Financial Protection Bureau has contracted with CFED to collect survey data to 

support the development of metrics to measure financial well-being. CFED has, in turn, 

subcontracted with Vector Psychometric Group, LLC (VPG) to conduct the psychometric 

analysis in support of the development of these metrics. VPG uses methods based on item 

response theory (IRT) in its metric development process. IRT-based methods are not 

probabilistic—they do not assume that the survey data being analyzed are from a random 

sample. IRT-based analysis does however require a large, socio-demographically diverse sample

that resembles the US population.

1 Large, diverse samples help ensure stable parameters around 

a wide-range of possible survey responses and enables analysts to test the performance of the 

metrics in different subpopulations.

CFED and its subcontractors will work with Survey Sampling International (SSI) to survey 

sample populations of working-age and older Americans that are sufficiently large and sociodemographically diverse to meet the needs of the project. 2 Specifically, CFED and its 

subcontractors will provide SSI with socio-demographic recruitment targets drawn from the 

2010 Census with respect to income, education, employment status, marital status, age, gender, 

race/ethnicity, presence/ages of children and geography. SSI will recruit until the specified sociodemographic targets are met and then they will close the survey to those subpopulations. SSI is a 

global company and one of the largest providers of sampling, data collection and data analytics 

services. SSI adheres to World Association for Social, Opinion and Market Research 

(ESOMAR) standards and is Grand Mean audited through Sample Source Auditors and annually 

audited by Ernst & Young. In 2012, SSI successfully completed 29 million surveys in 78 

countries. 

SSI will collect for this project completed, online surveys from 16,500 Americans (11,500 

working-age and 5,000 older Americans) who resemble the larger U.S. population. The sampling 

 1 Here we use ‘resemble’ to describe a sample population has the same socio-demographic profile as the US 

population at large, but was not drawn at random from the US population at large. The SSI sample will be 

representative of the US population on the observable socio-economic characteristics we use to recruit the sample, 

but it may not be representative of the US population with respect to unobservable characteristics that are not part of 

this data collection. 

2 ‘Working-age Americans’ include Americans ages 18-61 and ‘older Americans’ includes Americans ages 62 and 

older.
Page 2 of 9

universe for this study includes the approximately 163 million Americans who are age 18 and 

older and have access to the internet from some location.

3 In the future, the CFPB may field the 

survey tool developed via this requested information collection to a randomly selected, 

representative sample of the US population in order to assess the financial well-being of all U.S. 

residents.4 Americans ages 18 and older who have access to the internet from some location is 

therefore the appropriate potential respondent universe for the metrics testing associated with this 

project.

SSI’s sampling frame (the persons who have the potential to be part of the sample that SSI will 

collect for this study), includes “the unique audience across a set of SSI web “properties” 

[generally online advertisements] who notice the invitation to join and are motivated to act on 

it.”5 SSI asks persons who respond to their invitations to complete a brief online survey that 

includes socio-demographic questions. After receiving the responses to this survey, SSI utilizes a 

suite of quality control procedures including digital fingerprinting, to avoid 

duplication. Additionally, SSI samples incoming participants with its blending questionnaire, 

which asks behavior and psychographic questions in order to monitor the consistency of the 

sample coming into the system.

The sampling frames, for the purposes of this study, include working-age Americans (US 

residents ages 18 to 61) and older Americans (ages 62 and older) who are actively participating 

in an SSI panel. From these sample frames, SSI will develop “out go” or contact samples. 

Persons in the “out go” samples will receive invitations to take the surveys being fielded as part 

of this project. The contact samples will be sufficiently diverse to ensure that the eventual survey 

respondents resemble the US population at large. Because some respondents are more likely 

than others to respond to the survey participation invitation, the contact population will likely 

resemble the larger US population to a lesser degree than the population completing the surveys 

because SSI takes into account the fact that the survey take-up rate varies by subpopulation. For 

example, the contact sample is likely to contain a much larger number of males ages 18 to 24 

than females ages 45 to 54 given the differences in each group’s likelihood of responding (the 

older females being more likely to respond than the younger males). As the sample is fielded, 

SSI monitors response rates within the target demographics (age, education, income, gender, 

race/ethnicity, geography) and adjusts the “out go” as necessary to ensure the resulting sample 

resembles the larger US population.

Following the psychometric analysis of the SSI samples, Vector Psychometric Group will test 

online, paper-and-pencil as well as phone versions of the survey tool using a third sample from 

SSI. The sample that will be used for this testing can be relatively small and does not need to 

resemble the US population at large since the focus of the analysis will be on the stability of the 

item parameters across these different modes. 

 3 US Census Bureau 2012 estimates

4 The CFPB would seek appropriate OMB approval for such a collection, as needed.

5 See “Mixing the Right Sample Ingredients: A New Source Recipe.” Pete Cape, Kristin Cavallaro and Jackie Lorch. 

Survey Sampling International Memo.
Page 3 of 9

Sample Population 

universe

SSI sampling 

frame

Persons receiving 

an invitation from 

SSI to participate 

in the survey

Number of SSIfielded 

completed 

surveys

Working-age 

Americans

~140 million 

individuals in the 

US ages 18 and 

older who access 

the internet from 

some location

1.02 million 

individuals ages 

18 and older 

participating in 

SSI panels

Approximately 

280,000 SSI panel 

participants

11,500 SSI panel 

participants 

resembling the 

US population 

ages 18-61

Older Americans ~23 million 

individuals in the 

US ages 62 and 

older who access 

the internet from 

some location

180,000 

individuals ages 

62 and older 

participating in 

SSI panels

Approximately 

120,000 SSI panel 

participants

5,000 SSI panel 

participants 

resembling the 

US population 

ages 62 and 

older

2. Information Collection Procedures

Vector Psychometric Group relies on IRT-based methods in its metrics development process. 

IRT-based methods are not probabilistic—they do not assume that the survey data being 

analyzed are from a random sample. IRT-based analysis does however require a large, sociodemographically diverse sample that resembles the US population. Large, diverse samples help

ensure stable parameters around a wide-range of possible survey responses and enables analysts 

to test the performance of the metrics in different subpopulations (for example, low-, middle- and 

high-income populations). 

The contractor will work with SSI to obtain data from 11,500 working-age Americans and 5,000 

older Americans whose socio-demographic profiles resemble that of the larger US population of 

their respective age ranges per the procedures described in section 1. 

No population inferences will be made using this data.

3. Methods to Maximize Response Rates and Address Issues of Non-Response

Survey respondents will come from existing panels managed by Survey Sampling International 

(SSI). Agreements regarding incentive payments were established between SSI and the panelist 

at the time s/he was recruited. For example, when signing up for OpinionWorld, a panel 

administered by SSI, respondents are able to choose from a variety of incentives including 

donations to a preferred charity, entries into a drawing for a monetary prize etc. The incentives 

respondents will receive are not specific to this survey. SSI offers a wide variety of incentives in 

order to increase diversity of its sample frames because different types of rewards will motivate 

different respondents to participate in a survey.

For a survey of this length and type, a typical response rate, defined as opening the email and 

completing the survey, is roughly 3 to 5 percent. This response rate is typical of online panels. 

The participation survey rate or the percent of panelists entering survey (i.e., clicked on the link 
Page 4 of 9

and made it to the survey welcome and introduction) who completed the survey is roughly 80 

percent for SSI panels. 

High response and participation rates for this study will be encouraged through several means, 

including carefully worded and market research tested invitation emails, thoughtful planning of 

the participant experience that capitalizes on the latest survey research (for example, ensuring 

that surveys conducted as part of the panel are short enough to completed before respondent 

fatigue sets in), and follow-up/reminder emails to non-respondents. In addition, SSI has a strong 

set of incentives designed to appeal to respondents across demographic and behavioral groups to 

ensure participation without biasing results. Ensuring a quality participant experience is central 

to SSI’s survey recruitment strategy.

SSI will also try to maximize survey response rates by emphasizing the importance of this study 

in its participation invitation emails. SSI will also send a reminder email to non-respondents 

offering them second and third opportunity to participate. 

Issues of non-response will also be addressed through a comparison of responders to nonresponders to determine whether there were any particular types of consumers who elected to 

respond or not. Consumer characteristics available for non-response analysis will include age, 

gender, income, education, geography, and race/ethnicity. Analyses of drop-out point will also 

be conducted to determine email invitation open rate, survey entry from the invitation, survey 

initiation, and high exit points in the survey itself. CFED and its subcontractors will use these 

data to improve the quality of the survey.

4. Testing of Procedures or Methods

The entire purpose and planned process for this information collection is to test and refine data 

collection procedures and methods such as wording of survey questions and answer sets, and 

method of fielding the survey (online vs. telephone vs. paper and pencil). Using responses from 

3 distinct phases of data collection, the contractor will develop metrics to assess three constructs 

of interest (i.e., Financial Well-being, Financial Knowledge, and Financial Behavior) using IRTbased methods (e.g., Thissen & Wainer, 2001; Wirth & Edwards, 2007). A large pool of 

candidate items (questions and response sets) based on the finding from qualitative research, the

project literature review, and a review of other financial well-being related surveys will be 

developed (see Survey Question Item Bank, attached as Appendix A). A panel of psychometric 

and content experts will review and refine the pool of candidate items so that the remaining items 

are targeted and well-worded. No more than 100 items will be included in the refined item pool. 

This number of items is specified to ensure that respondents are not overly burdened when 

responding to the item pool.

In phase I, SSI will field the candidate items online to approximately 7,000 working-age and 

2,000 older Americans. When data is first received, it will be thoroughly investigated for each 

age-group individually and any necessary recoding or flagging of questionable response patterns 

(typically called “data cleaning”) will be recorded and performed in SAS 9.2. Data cleaning 

efforts include, but are not limited to, identifying suspect response patterns (e.g., participants 

provides the same response for every item), detailing any missing data occurrences which may 

be problematic at later stages, checking for non-logical values (e.g., logically inconsistent 
Page 5 of 9

responses across items , for example, respondent indicates they are unable to walk a mile and 

also indicate they are able to walk 3 miles easily), and reverse coding items as needed to ensure 

responses to all items are in the same direction (i.e., more agreement indicates a higher level of 

the construct of interest). With data cleaning complete, preliminary data analysis will begin. 

Preliminary data analysis, by age group, will also be conducted in SAS 9.2, unless otherwise 

noted. Plots will be produced using SAS or, if publication-quality graphics are needed, Excel or 

R. For each item, a frequency table of responses (and possibly a histogram which depicts the 

same information graphically) will be obtained and examined. For the analyses using the full 

samples of each group, any response category with less than 5 observations will be collapsed, 

towards the center, into the next adjacent category. Following any necessary collapsing 2-way 

frequency tables (comparing all items to all other items by pairs) will be obtained. These will be 

examined to identify any item pairs in which responses are not related as expected. With a 

similar goal in mind, a matrix of polychoric correlation values among the items will be obtained, 

using MPLUS 6.11 (Muthén & Muthén, 1998-2010). Polychoric correlations, rather than 

Pearson product-moment correlations, are specified due to the expected categorical nature of the 

item responses. Research has found that the use of correlation methods intended for continuous 

data, when applied to categorical data, will tend to under-estimate the true magnitude of the 

correlation (Carroll, 1961). 

Classical test theory analyses (Cronbach’s alpha, item total correlations, etc.) will then be 

conducted. Although these analyses will not be used in making final determinations of item 

inclusion on the final version of the developing scales, such analyses provide recognizable 

results for those less familiar with modern scale development techniques and will hopefully 

provide a bridge to understanding the results of the modern scale development techniques. To 

that end, for each construct, we will produce tables with the Cronbach’s alpha (internal 

consistency) value for each scale, as well as the item-total correlation and “alpha if removed” 

value for each item. These analyses will be conducted in SAS 9.2.

Structural assessments will be conducted to ensure that an appropriate number of dimensions are 

employed to model the relationships among the items for each construct. Due to the expected 

ordinal nature of the data, the use of methods developed for continuous data would be 

unadvisable (e.g., Wirth & Edwards, 2007). Exploratory and confirmatory factor analysis (EFA 

and CFA, respectively) model parameters will be obtained based on polychoric correlations and 

diagonally weighted least-squares estimation, as implemented in Mplus 6.11 (Muthén & Muthén, 

1998-2010). The fit of each model will be evaluated using the comparative fit index (CFI; 

Bentler, 1990), the Tucker-Lewis index (TLI: Tucker & Lewis, 1973), and the root mean squared 

error of approximation (RMSEA; Steiger & Lind, 1980), using customary cut-offs for adequate 

fit of 0.95 or greater for the CFI and TLI and less than 0.08 for the RMSEA (e.g., Hu & Bentler, 

1999; Browne & Cudeck, 1993). To ensure that preferred models are not over-fit to the data, 

each sample will randomly be divided into two sub-samples: one for exploratory modeling and 

the other for confirmatory modeling. 

For EFA models, the eigenvalues of the polychoric correlation matrix of items for each construct 

will be extracted from the MPLUS output and plotted in Excel (i.e., we will produce a scree plot 

for each construct). Such plots are useful in determining the optimal number of factors to 
Page 6 of 9

estimate. EFA solutions for each construct will be requested with the number of factors ranging 

from 1 to the number of factors suggested by the scree plot and results from a parallel analysis 

(e.g., Hayton, Allen & Scarpello, 2004). However, for each construct, no more than 6 factors (a 

reasonable maximum considering there will only be approximately 30-50 items per construct) 

will be allowed. For models with more than one factor, oblique CF-quartimax rotation (e.g., 

Browne, 2001) will be used to rotate initial factor extraction values to more readily interpretable 

solutions.

For each construct, several categorical confirmatory factor analysis (CCFA) models will be 

tested. The planned models for each construct are: a unidimensional model, a bifactor-type 

model in which all items load onto a single general factor and no more than one specific factor 

(with specific factors defined by the theoretical core characteristics/sub-domains of the items), 

and a multi-dimensional model in which several correlated factors are present (defined by the 

theoretical core characteristics/sub-domains of the items). Additionally, models suggested by the 

EFA results (if not covered by the description of the a priori models) will also be tested. Among 

the tested models, the preferred model will be the model with acceptable fit indices and a 

structure that possesses substantive interpretability. Preference will be given to structurally 

simpler models.

Item response theory (IRT) calibration analyses will be conducted within each age group for 

each construct individually in flexMIRT 2.0 (Cai, 2013). The factor structure of the model for 

each construct will be that of the preferred model based on the structural assessment results. 

Assuming Likert-type response data for all items, the IRT model used will be the graded 

response model (Samejima, 1969); for an item with only two categories, this model simplifies to 

the 2-parameter logistic model. For estimating IRT parameters, flexMIRT, by default, uses 

maximum marginal likelihood or modal Bayes via Bock-Aitkin EM (Bock & Aitkin, 1981). For 

high-dimensional models, the more efficient Metropolis Hasting-Robbins Monro (MH-RM) 

estimation algorithm (Cai, 2010) is also available. Regardless of estimation method, the standard 

errors for parameter estimates will be calculated via the supplemented EM algorithm (Cai, 2008). 

Item parameter results from the IRT calibration of each construct will be presented in tables and 

in the form of trace line plots/item characteristic curves (ICCs) for each item. An ICC for a given 

item represents the probability of endorsing an item as a function of an individual’s level on the 

underlying construct. 

Using the IRT calibration results and expert opinion, the candidate items used in phase I will be 

re-evaluated. Items that are performing poorly will be eliminated from further use. An 

examination of the remaining items will be conducted to determine any content areas that are not 

well-represented or areas on the construct continuum (e.g., ranges along the spectrum from low 

financial well-being to high financial well-being) that are not well-measured. Item development 

will be conducted to create items intended to fill any identified gaps.

Using the items that performed well in phase I and the newly developed items, new iterations of 

the 3 financially- related metrics will be created. The second round of online surveys (phase II)

will be fielded to 3,000 working-age and 3,000 older Americans. Analyses will follow the same 

structure as outlined previously. The IRT calibration results of the Phase II data and expert 

opinion will inform the selection of items to the final versions of the metrics for each construct.
Page 7 of 9

As the final step of analyzing the phase II data, it is currently planned that analyses will be 

conducted to assess if the performance of items differs across the two age groups, working-age 

and older Americans. However, for these differential item functioning (DIF) analyses to be 

conducted, it is necessary that there is a sufficient amount of overlap across age-groups, with 

respect to the items selected for the final metrics. A feasibility review will be conducted to 

determine, for each construct, if the overlap of the final scales for each age group is sufficient to 

allow for comprehensive DIF testing across the age groups.

Assuming favorable results from the feasibility review and using the factor model(s) supported 

by the structural assessments, DIF analyses will be conducted across the working and older 

Americans age groups in flexMIRT 2.0. The estimation method used for the DIF analyses will 

depend on the complexity of the preferred structural model, as in the calibration. If the model is 

unidimensional or bifactor-like in structure maximum marginal likelihood or modal Bayes via 

Bock-Aitkin EM (Bock & Aitkin, 1981) will be employed. If the preferred model is highdimensional, MH-RM will be used. For assessing DIF, flexMIRT reports Wald tests (e.g., 

Woods, Cai, & Wang, 2013) - DIF analyses calculate these test statistics for each item and each 

set of parameters for that item (i.e., slopes, intercepts) to allow the user to test whether item 

parameters differ across groups. The Benjamini-Hochberg (Benjamini & Hochberg, 1995) false 

discovery rate procedure will be applied to the resulting Wald test values to control Type I error 

(unwarranted rejection of the null hypothesis; a false positive). If the statistical test is significant, 

DIF is suspected to exist between the age groups. The analysis of DIF with bifactor models is a 

developing area - if a bifactor model is the preferred model for any of the constructs, the 

procedure outlined in Cai, Yang, and Hansen (2011) will be followed to obtain accurate 

estimates of any DIF between the groups. The end result of such DIF analyses, if feasible, is that

working-age and older American item parameters will be on the same metric, which will allow 

scores across the age groups to be directly comparable to each other.

The final set of analyses is planned for the third phase of data collection, which will contain 

responses from 1500 individuals and use three different modes of item presentation: Online, 

Phone, and Paper-and-Pencil. For data from the phone and paper-and-pencil collection modes, 

CCFA models will be fit using the factor structure mirroring that of the preferred model for each 

construct found using Phase II data, which was collected via online item presentation only. Using 

the models supported by the CCFA analyses, IRT calibrations for these new presentation modes 

will also be conducted, individually for each combination of construct and presentation mode. 

Next, DIF analyses will be conducted to identify any item performance changes across the three 

presentation modes. Items with poor or inconsistent performance across modes are candidates for 

removal. These analyses will be conducted using the same procedures outlined in the previous 

paragraph. Finally, targeted correlational and regression analyses will be conducted to examine 

the relationships among the 3 financial constructs and other theoretical drivers; the analyses 

conducted at this point are pre-determined by the hypotheses regarding the drivers of financial 

well-being that were developed during the qualitative phase of scale development.
Page 8 of 9

5. Contact Information for Statistical Aspects of the Design

Missy Amish, Survey Sampling International, 203-246-6051, 

Darci.Patrick@SurveySampling.com

Anita Drever, Corporation for Enterprise Development, 202-027-0142, adrever@cfed.org

Dee Warmath, Center for Financial Security, University of Wisconsin Madison, 

262-312-0606, warmath@wisc.edu

R.J. Wirth, Vector Psychometric Group, LLC, 919-768-7142, RJWirth@VPGCentral.com

Bibliography

Benjamini.Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and 

powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57, 

289-300.

Bentler, P. M. (1990). Comparative Fit Indexes in Structural Models. Psychological Bulletin, 

107, 238–246.

Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: 

Application of an EM algorithm. Psychometrika, 46 , 443–459.

Browne, M. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate 

Behavioral Research, 36, 111-150.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen, 

& J. Scott Lang (Eds.), Testing structural models (pp.136-162). Newbury Park, CA: Sage 

Publications.

Cai, L. (2008). SEM of another flavour: Two new applications of the supplemented EM 

algorithm. British Journal of Mathematical and Statistical Psychology, 61, 309–329

Cai, L. (2010). High-dimensional exploratory item factor analyses by a Metropolis-Hastings 

Robbins-Monro algorithm, Psychometrika, 75, 33-57.

Cai, L. , Yang, J.S., & Hansen, M. (2011). Full-information item bifactor analysis. Psychological 

Methods, 16, 221-247. 

Carroll, J. B. (1961). The nature of data, or how to choose a correlation coefficient. 

Psychometrika, 26, 347-372.

Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory 

factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 292-

305.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: 

Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

Muthén, L.K. and Muthén, B.O. (1998-2010). Mplus User’s Guide. Sixth Edition. Los Angeles, 

CA: Muthén & Muthén.

Samejima, F. (1969). Estimation of Latent Ability Using a Response Pattern of Graded Scores 

Psychometric Monograph No. 17. Richmond, VA: Psychometric Society

Steiger, J. H., & Lind, J. M. (1980, June). Statistically based tests for the number of common 
Page 9 of 9

factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, 

IA.

Thissen, D., & Wainer, H. (Eds.) (2001). Test Scoring. Mahwah, NJ: Lawrence Erlbaum 

Associates.

Tucker, L. R, & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor 

analysis. Psychometrika, 38, 1–10.

Wirth, R. J., & Edwards, M.C. (2007). Item factor analysis: Current approaches and future 

directions. Psychological Methods, 12, 58-79.

Woods, C., Cai, L., & Wang, M. (2013). The Langer-improved Wald test for DIF testing with 

multiple groups: Evaluation and comparison to two group IRT. Educational and 

Psychological Measurement, 73, 532–547.