Page 1 of 9 CONSUMER FINANCIAL PROTECTION BUREAU INFORMATION COLLECTION REQUEST

– SUPPORTING STATEMENT PART B DEVELOPMENT OF METRICS TO MEASURE FINANCIAL WELL

- BEING OF WORKING

- AGE AND OLDER AMERICAN CONSUMERS (OMB CONTROL NUMBER: 3170

- XXXX)

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Selection Methods

The Consumer Financial Protection Bureau has contracted with CFED to collect survey data to support the development of metrics to measure financial well

- being. CFED has, in turn, subcontracted with Vector Psychometric Group , LLC (VPG) to conduct the psych ometric analysis in support of the development of these metrics. VPG uses methods based on item response theory (IRT) in its metric development process. I RT

- based methods are not probabilistic

— they do not assume that the survey data being analyzed are from a random sample. I RT

- base d analysis does however require a large, socio

- demographically diverse sample that resembles the US population

. 1 L arge, diverse samples help ensure stable parameters around a wide

- range of possible survey responses and enabl es analysts to test the performance of the metrics in different subpopulations

.

CFED and its sub contractors will work with Survey Sampling International (SSI) to survey sample populations of working

- age and older Americans that are sufficiently large and so cio

demographically diverse to meet the needs of the project

. 2 Specifically, CFED and its subcontractors will provide SSI with socio

- demographic recruitment targets drawn from the 2010 Census with respect to income, education, employment status, marital status, age, gender, race/ethnicity, presence/ages of children and geography. S S I will recruit until the specified socio

demographic targets are met and then they will close the survey to those subpopulations. SSI is a global company and one of th e largest providers of sampling, data collection and data analytics services. SSI adheres to World Association for Social, Opinion and Market Research (ESOMAR) standards and is Grand Mean audited through Sample Source Auditors and annually audited by Ernst & Young. In 2012, SSI successfully completed 29 million surveys in 78 countries.

SSI will collect for this project completed , online surveys from 1 6
, 5 00 Americans (1 1

, 5 00 working

- age and 5,000 older Americans) who resemble the larger U.S. population. The sampling

1 Here we use ‘resemble’ to describe a sample population has the same socio

- demographic profile as the US population at large, but was not drawn at random from the US population at large. The SSI sample will be representative of the US population on the obs ervable socio

- economic characteristics we use to recruit the sample, but it may not be representative of the US population with respect to unobservable characteristics that are not part of this data collection.
2 ‘Working

- age Americans’ include Americans ages 18

- 61 and ‘older Americans’ includes Americans ages 62 and older.Page 2 of 9 universe for this study includes the approximately 1 63 million Americans who are age 18 and older and have access to the internet from some location

. 3 In the future, t he CFPB may field the survey tool developed via this requested information c ollection to a randomly selected, representative sample of the US population in order to assess the financial well

- being of all U.S. residents. 4 Americans ages 18 and older who have access to the internet from some location is therefore the appropriate potential respondent universe for the metrics testing associated with this project.

SSI’s sampling frame (the persons who have the potential to be part of the s ample that SSI will collect for this study ), includes “the unique audience across a set of SSI web “properties” [generally online advertisements] who notice the invitation to join and are motivated to act on it

. ” 5 SSI a sks persons who respond to their invitations to complete a brief online survey that includes socio

- demographic questions. After receiving the responses to this survey

, SSI utilizes a suite of quality control procedures including digital fingerprinting, to avoid duplication. Additionally, SSI samples incoming participants with its blending questi onnaire, which asks behavior and psychographic questions in order to monitor the consistency of the sample coming into the system.

The sampling frame s , for the purposes of this study, include working

- age Americans (US residents ages 18 to 61 ) and older Americans (ages 62 and older) who are actively participating in an SSI panel

. From these sample frames, SSI will develop “out go” or contact samples. Persons in the “out go ” samples will receive invitations to take the surveys being fielded as part of this project. The contact samples will be sufficiently diverse to en sure that the eventual survey respondents resemble the US population at large. Because some respondents are more likely than others to respond to the survey participation invitati on , the contact population will likely resemble the larger US population to a lesser degree than the population completing the surveys because SSI takes into account the fact that the survey take

- up rate varies by subpopulation. For example, the contact sample is likely to contain a much larger number of males ages 18 to 24 than females ages 45 to 54 given the differences in each group’s likelihood of responding (the older females being more likely to respond than the younger males). As the sample is fie lded, SSI monitors response rates within the target demographics (age, education, income, gender, race/ethnicity, geography) and adjusts the “ out go ” as necessary to ensure the resulting sample resembles the larger US population.

Following the psychometri c analysis of the SSI samples

, Vector Psychometric Group will test online, paper

- and

- pencil as well as phone versions of the survey tool using a third sample from SSI . The sample that will be used for this testing can be relatively small and does not need to resemble the US population at large since the focus of the analysis will be on the stability of the item parameters across these different modes

.

3 US Census Bureau 2012 estimates
4 The CFPB would seek appropriate OMB approval for such a collection, as needed.
5 See “Mixing the Right Sample Ingredients: A New Source Recipe.” Pete Cape, Kristin Cavallaro and Jackie Lorch. Survey Sampling International Memo.Page 3 of 9 Sample Population universe SSI sampling frame Persons receiving an invitation from SSI to participate in the survey Number of SSI

fielded completed
surveys
Working

- age Americans ~140 million individuals in the US ages 18 and older who access the internet from some location 1. 0 2 million individuals ages 18 and older participating in SSI panels Approximately 28 0,000 SSI panel participants 1 1 ,500 SSI panel participants resembling the US populati o n age s 18

- 61

Older Americans ~23 million individuals in the US ages 62 and older who access the internet from some location 180,000 individuals ages 6 2 and older participating in SSI panels Approximately 120,000 SSI panel participants 5,000 SSI panel participants resembling the US population ages 62 and older

2. Information Collection Procedures

Vector Psychometric Group relies on IRT

- based methods in its metrics development process. I RT

- based methods are not probabilistic

— they do not assume that the survey data being analyzed are from a random sample. I RT

- based analysis does however require a large, socio

demographically diverse sample that resemble s the US population. L arge, diverse samples help ensure stable parameters around a wide

- range of possible survey responses and enables analysts to test the performance of the metrics in different subpopulations (for example, low

- , middle

- and high

- income populations).

The contractor will work with SSI t o obtain data from 11,500 working

- age Americans and 5,000 older Americans whose socio

- demographic profiles resemble that of the la rger US population of their respective age ranges per the procedures desc ribed in section 1.

No population inferences will be made using this data.

3. Methods to Maximize Response Rates and Address Issues of Non

- Response

Survey respondents will come from existing panels managed by Survey Sampling International (SSI). Agreements regarding incentive payments were established between SSI and the panelist at the time s/he was recruited. For example, when signing up for Opini onWorld , a panel administered by SSI, respondents are able to choose from a variety of incentives including donations to a preferred charity, entries into a drawing for a monetary prize etc. The incentives respondents will receive are not specific to this survey. SSI offers a wide variety of incentives in order to increase diversity of its sample frames because different types of rewards will motivate different respondents to participate in a survey.

For a survey of this length and type, a typical response rate, defined as opening the email and completing the survey, is roughly 3 to 5 percent. This response rate is typical of online panels. The participation survey rate or the percent of panelists en tering survey (i.e., clicked on the linkPage 4 of 9 and made it to the survey welcome and introduction) who completed the survey is roughly 80 percent for SSI panels.

High response and participation rates for this study will be encouraged through several means, inc luding carefully worded and market research tested invitation emails, thoughtful planning of the participant experience that capitalizes on the latest survey research (for example, ensuring that surveys conducted as part of the panel are short enough to co mpleted before respondent fatigue sets in), and follow

- up/reminder emails to non

- respondents. In addition, SSI has a strong set of incentives designed to appeal to respondents across demographic and behavioral groups to ensure participation without biasin g results. Ensuring a quality participant experience is central to SSI’s survey recruitment strategy.

SSI will also try to maximize survey response rates by emphasizing the importance of this study in its participation invitation emails. SSI will also se nd a reminder email to non

- respondents offering them second and third opportunity to participate.

Issues of non

- response will also be addressed through a comparison of responders to non

responders to determine whether there were any particular types of c onsumers who elected to respond or not. Consumer characteristics available for non

- response analysis will include age, gender, income, education, geography, and race/ethnicity. Analyses of drop

- out point will also be conducted to determine email invitati on open rate, survey entry from the invitation, survey initiation, and high exit points in the survey itself. CFED and its subcontractors will use these data to improve the quality of the survey.

4. Testing of Procedures or Methods

The entire purpose and planned process for this information collection is to test and refine data collection procedures and methods such as wording of survey questions and answer sets, and method of fielding the survey (online vs. telephone vs. paper and pencil). Us ing responses from 3 distinct phases of data collection, t he contractor will develop metrics to assess three constructs of interest (i.e., Financial Well

- being, Financial Knowledge, and Financial Behavior) using IRT

based methods ( e.g., Thissen & Wainer

, 2001; Wirth & Edwards, 20 07 ). A large pool of candidate items (question s and response sets) based on the finding from qualitative research, the project literature review, and a review of other financial well

- being related surveys will be developed ( see Su rvey Question Item Bank , attached as Appendix A ) . A panel of psychometric and content experts will review and refine the pool of candidate items so that the remaining items are targeted and well

- worded. No more than 100 items will be included in the refine d item pool. This number of items is specif i ed to ensure that respondents are not overly burdened when responding to the item pool.

In phase I

, SSI will field the candidate items online to approximately 7,000 working

- age and 2,000 older Americans

. When data is first received, it will be thoroughly investigated for each
age

- group individually and any necessary recoding or flagging of questionable response patterns (typically called “data cleaning”) will be recorded and performed in SAS 9.2. Data clea ning efforts include, but are not limited to, identifying suspect response patterns (e.g., participants provides the same response for every item), detailing any missing data occurrences which may be problematic at later stages, checking for non

- logical va lues (e.g., logically inconsistentPage 5 of 9 responses across items , for example, respondent indicates they are unable to walk a mile and also indicate they are able to walk 3 miles easily), and reverse coding items as needed to ensure responses to all items are in the same direction (i.e., more agreement indicates a higher level of the construct of interest). With data cleaning complete, preliminary data analysis will begin.

Preliminary data analysis, by age group, will also be conducted in SAS 9.2, unless otherw ise noted. Plots will be produced using SAS or, if publication

- quality graphics are needed, Excel or R. For each item, a frequency table of responses (and possibly a histogram which depicts the same information graphically) will be obtained and examined. F or the analyses using the full samples of each group, any response category with less than 5 observations will be collapsed, towards the center, into the next adjacent category. Following any necessary collapsing 2

- way frequency tables (comparing all items to all other items by pairs) will be obtained. These will be examined to identify any item pairs in which responses are not related as expected. With a similar goal in mind, a matrix of polychoric correlation values among the items will be obtained, using MPLUS 6.11 (Muthén & Muthén, 1998

- 2010). Polychoric correlations, rather than Pearson product

- moment correlations, are specified due to the expected categorical nature of the item responses. Research has found that the use of correlation methods intended for continuous data, when applied to categorical data, will tend to under

- estimate the true magnitude of the correlation (Carroll, 1961).

Classical test theory analyses (Cronbach’s alpha, item total correlations, etc.) will then be conducted. Although th ese analyses will not be used in making final determinations of item inclusion on the final version of the developing scales, such analyses provide recognizable results for those less familiar with modern scale development techniques and will hopefully
pro vide a bridge to understanding the results of the modern scale development techniques. To that end, for each construct, we will produce tables with the Cronbach’s alpha (internal consistency) value for each scale, as well as the item

- total correlation and “alpha if removed” value for each item. These analyses will be conducted in SAS 9.2.

Structural assessments will be conducted to ensure that an appropriate number of dimensions are employed to model the relationships among the items for each construct. Du e to the expected ordinal nature of the data, the use of methods developed for continuous data would be unadvisable (e.g., Wirth & Edwards, 2007). Exploratory and confirmatory factor analysis (EFA and CFA, respectively) model parameters will be obtained ba sed on polychoric correlations and diagonally weighted least

- squares estimation, as implemented in Mplus 6.11 (Muthén & Muthén, 1998

- 2010). The fit of each model will be evaluated using the comparative fit index (CFI; Bentler, 1990), the Tucker

- Lewis index (TLI: Tucker & Lewis, 1973), and the root mean squared

error of approximation (RMSEA; Stei ger & Lind, 1980), using customary cut

- offs for adequate fit of 0.95 or greater for the CFI and TLI and less than 0.08 for the RMSEA (e.g., Hu & Bentler, 1999; Brown e & Cudeck, 1993). To ensure that preferred models are not over

- fit to the data, each sample will randomly be divided into two sub

- samples: one for exploratory modeling and the other for confirmatory modeling.

For EFA models, the eigenvalues of the polychoric correlation matrix of items for each construct will be extracted from the MPLUS output and plotted in Excel (i.e., we will produce a scree plot for each construct). Such plots are useful in determining the optimal number of factors toPage 6 of 9 estimate. EFA solutions for each construct will be requested with the number of factors ranging from 1 to the number of factors suggested by the scree plot and results from a parallel analysis (e.g., Hayton, Allen & Scarpello, 2004). However, for each construct, no more than 6 factors (a reasonable maximum considering there will only be approximately 30

- 50 items per construct) will be allowed. For models with more than one factor, oblique CF

- quartimax rotation (e.g., Browne, 200 1) will be used to rotate initial factor extraction values to more readily interpretable solutions.

For each construct, several categorical confirmatory factor analysis (CCFA) models will be tested. The planned models for each construct are: a unidimensio nal model, a bifactor

- type model in which all items load onto a single general factor and no more than one specific factor (with specific factors defined by the theoretical c ore characteristics/sub

- domains of the items), and a multi

- dimensional model in wh ich several correlated factors are present (defined by the theoretical core characteristics/sub

- domains of the items). Additionally, models suggested by the EFA results (if not covered by the description of the a priori models) will also be tested. Among
t he tested models, the preferred model will be the model with acceptable fit indices and a structure that possesses substantive interpretability. Preference will be given to structurally simpler models.

Item response theory (IRT) calibration analyses will be conducted within each age group for each construct individually in flexMIRT 2.0 (Cai, 2013). The factor structure of the model for each construct will be that of the preferred model based on the structural assessment results. Assuming Likert

- type respon se data for all items, the IRT model used will be the graded response model (Samejima, 1969); for an item with only two categories, this model simplifies to the 2

- parameter logistic model. For estimating IRT parameters, flexMIRT, by default, uses maximum m arginal likelihood or modal Bayes via Bock

- Aitkin EM (Bock & Aitkin, 1981). For high

- dimensional models, the more efficient Metropolis Hasting

- Robbins Monro (MH

- RM) estimation algorithm (Cai, 2010) is also available. Regardless of estimation method, the st andard errors for parameter estimates will be calculated via the supplemented EM algorithm (Cai, 2008). Item parameter results from the IRT calibration of each construct will be presented in tables and in the form of trace line plots/item characteristic cu rves (ICCs) for each item. An ICC for a given item represents the probability of endorsing an item as a function of an individual’s level on the underlying construct.

Using the IRT calibration results and expert opinion, the candidate items used in phase I will be

re

- evaluated. Items that are performing poorly will be eliminated from further use. An examination of the remaining items will be conducted to determine any content areas that are not well

- represented or areas on the construct continuum (e.g., ran ges along the spectrum from low financial well

- being to high financial well

- being) that are not well

- measured. Item development will be conducted to create items intended to fill any identified gaps

. Using the items that performed well in phase I and the newly developed items, new iterations of the 3 financially

- related metrics will be created. The second round of online surveys (phase II ) will be fielded to 3,000 working

- age and 3,000 older Americans. Analyses will follow the same structure as outline d previously. The IRT calibration results of the Phase II data and expert opinion will inform the selection of items to the final versions of the metrics for each construct.Page 7 of 9 As the final step of analyzing the phase II data, it is currently planned that anal yses will be conducted to assess if the performance of items differs across the two age groups, working

- age and older Americans. However, for these differential item functioning (DIF) analyses to be conducted, it is necessary that there is a sufficient amo unt of overlap across age

- groups, with respect to the items selected for the final met r ic s

. A feasibility review will be conducted to determine, for each construct, if the overlap of the final scales for each age group is sufficient to allow for comprehens ive DIF testing across the age groups.

Assuming favorable results from the feasibility review and using the factor model(s) supported by the structural assessments, DIF analyses will be conducted across the working and older Americans age groups in flexMIRT 2.0. The estimation method used for the DIF analyses will depend on the complexity of the preferred structural model, as in the calibration. If the model is unidimensional or bifactor

- like in structure maximum marginal likelihood or modal Bayes vi a Bock

- Aitkin EM (Bock & Aitkin, 1981) will be employed. If the preferred model is high

dimensional, MH

- RM will be used. For assessing DIF, flexMIRT reports Wald tests (e.g., Woods, Cai, & Wang, 2013)

- DIF analyses calculate these test statistics for each item and each set of parameters for that item (i.e., slopes, intercepts) to allow the user to test whether item parameters differ across groups. The Benjamini

- Hochberg (Benjamini & Hochberg, 1995) false discovery rate procedure will be applied to the res ulting Wald test values to control Type I error (unwarranted rejection of the null hypothesis; a false positive). If the statistical test is significant, DIF is suspected to exist between the age groups. The analysis of DIF with bifactor models is a develo ping area

- if a bifactor model is the preferred model for any of the constructs, the procedure outlined in Cai, Yang, and Hansen (2011) will be followed to obtain accurate estimates of any DIF between the groups. The end result of such DIF analyses, if fe asible, is that working

- age and older American item parameters will be on the same metric , which will allow scores across the age groups to be directly comparable to each other.

The final set of analyses is planned for the third phase of data collection, which will contain responses from 1500 individuals and use three different modes of item presentation: Online, Phone, and Paper

- and

- Pencil. For data from the phone and paper

- and

- pencil collection modes, CCFA models will be fit using the factor structure m irroring that of the preferred model for each construct found using Phase II data, which was collected via online item presentation only. Using the models supported by the CCFA analyses, IRT calibrations for these new presentation modes will also be conduc ted, individually for each combination of construct and presentation mode. Next
, DIF analyses will be conducted to identify any item performance changes across the three presentation modes. Items with poor or inconsistent performance across modes are candi dates for removal. These analyses will be conducted using the same procedures outlined in the previous paragraph. Finally, targeted correlational and regression analyses will be conducted to examine the relationships among the 3 financial constructs and ot her theoretical drivers; the analyses conducted at this point are pre

- determined by the hypotheses regarding the drivers of financial well

- being that were developed during the qualitative phase of scale development.Page 8 of 9 5. Contact Information for Statistical Aspects of the Design

Missy Amish, Survey Sampling International, 203

- 246

- 6051, Darci.Patrick@SurveySampling.com

Anita Drever, Corporation for Enterprise Development , 202

- 027

- 0142, adrever@cfed.org

Dee Warmath, Center for Financial Security, University of Wisconsin Madison, 262

- 312

- 0606, warmath@wisc.edu

R.J. Wirth, Vector Psychometric Group , LLC , 919

- 768

- 7142, RJWirth@VPGCentral.com

Bibliography

Benjamini.Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B

, 57

, 289

- 300. Bentler, P. M. (1990). Comparative Fit Indexes in Structural Models. Psychological Bulletin, 107, 238

– 246

. Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46 , 443

– 459

. Browne, M. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research

, 36 , 111

- 150. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen, & J. Scott Lang (Eds.), Te sting structural models (pp.136

- 162). Newbury Park, CA: Sage Publications. Cai, L. (2008). SEM of another flavour: Two new applications of the supplemented EM algorithm. British Journal of Mathematical and Statistical Psychology

, 61 , 309

– 329 Cai, L. (2010). High

- dimensional exploratory item factor analyses by a Metropolis

- Hastings Robbins

- Monro algorithm , Psychometrika, 75 , 33

- 57. Cai, L. , Yang, J.S., & Hansen, M. (2011). Full

- information item bifactor analysis. Psychological Methods, 16 , 221

- 247. C arroll, J. B. (1961). The nature of data, or how to choose a correlation coefficient. Psychometrika, 26 , 347

- 372. Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysi s. Organizational Research Methods

, 7 , 292

- 305. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling

, 6 , 1

- 55. Muthén, L.K. and Muthé n, B.O. (1998

- 2010). Mplus User’s Guide. Sixth Edition . Los Angeles, CA: Muthén & Muthén. Samejima, F. (1969). Estimation of Latent Ability Using a Response Pattern of Graded Scores Psychometric Monograph No. 17 . Richmond, VA: Psychometric Society Steiger, J. H., & Lind, J. M. (1980, June). Statistically based tests for the number of commonPage 9 of 9 factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA. Thissen, D ., & Wainer, H. (Eds.) (2001). Test Scoring . Mahwah, NJ: Lawrence Er lbaum Associates. Tucker, L. R, & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1

– 10. Wirth, R. J., & Edwards, M

. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12 , 58

- 79. Woods, C., Cai, L., & Wang, M. (2013). The Langer

- improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two group IRT. Educational and Psychological Measurement

, 73 , 532

– 547.