Visualizing and Interpreting Multi-Group Confirmatory Factor Analysis
Creators: name: Buchanan, Erin Michelle
Publisher: name: Open Science Framework
Year: 2023
Subjects: subject: Statistical Models
subject: Social and Behavioral Sciences
subject: Statistics and Probability
subject: Social Statistics
subject: Physical Sciences and Mathematics
subject: assessment
subject: multigroup cfa
subject: mgcfa
subject: visualization

description: Title: Visualizing and Interpreting Multi-Group Confirmatory Factor Analysis Author: Erin M. Buchanan Harrisburg University of Science and Technology ebuchanan@harrisburgu.edu Abstract: Latent variable modeling as a lens for psychometric theory is a popular tool for social scientists to examine measurement of constructs (Beaujean, 2014). Journals such as Assessment regularly publish articles supporting new or previously established measures of latent constructs (e.g., depression, anxiety) wherein a measurement model is established for the scale in question. These measurement models designate the relationship between the measured, observed variables, and the underlying construct, with the assumption that these relations hold in many samples. Confirmatory factor analysis can be used to investigate the replicability and generalizability of the measurement model in new samples, while multi-group confirmatory factor analysis is used to examine the measurement model across groups within samples (Brown, 2015). With the rise of the replication crisis and “psychology’s renaissance” (Nelson et al., 2018), interest in divergence in measurement has increased, often focused on small parameter differences within the latent model. While the statistical procedure for examining measurement invariance is moderately well established, it is clear that the toolkit for inspecting these results is lacking. This manuscript will outline ways to visualize potential non-invariance, to supplement large numbers of tables that often overwhelm a reader within these published reports. Further, given these visualizations, readers will learn how to interpret the impact and size of the proposed non-invariance in models. While it is tempting to suggest that problems with replication and generalizability are simply issues with measurement, it is crucial to remember that all models have variability and error, even those models estimating the differences between item functioning, such as multi-group confirmatory factor analysis. This manuscript will help provide a framework for researchers interested in registered reports in this area. Research Questions: By the end of this proposed manuscript, readers will: 1. Be able to create visualizations for common steps to multi-group confirmatory factor analysis. 2. Be able to interpret the impact and size of potential non-invariance on measurement. 3. Understand the impact of measurement variability on replication and generalizability. Design: Data will be simulated based on a multivariate normal distribution using R. The data will include two groups of individuals for multi-group confirmatory factor analysis (ns = 250; N = 500). The latent variables will be assumed to be continuous normal, and the measured items will be simulated using a traditional 1-7 Likert-type scale, matching many of the types of scales published in psychology and Assessment. Each item will be assumed to be related to the latent variable with loadings approximately equal to .60 to .80 (minus scenarios in which configural non-invariance is simulated). Variables: • Number of factors: one and two latent variable models will be simulated with five measured items for each latent variable. • Area of invariance: a model with non-invariance at each stage will be simulated including configural, metric, scalar, and strict non-invariance. One model with complete invariance will be used for comparison purposes. • Size of invariance: the size of the invariance will be simulated at three levels. Analysis: Multi-group confirmatory factor analysis will be examined across the simulated two groups of individuals using lavaan (Rosseel, 2012) focusing on the most common procedure outlined in Brown (2015). Each stage of measurement invariance (i.e., configural, metric, scalar, and strict) will be examined in the simulated data demonstrating how to visualize results using common tools, such as ggplot2 (Wickham et al., 2021). These visualizations will be compared across differences in non-invariance size to illustrate how to interpret the size of the non-invariance. Last, these effects will be examined on total score measurement (i.e., latent means) to explore the potential impacts on replication. Finally, suggestions will be provided for framing registered reports using multi-group confirmatory factor analysis.