Preregistration of the data analysis for the validation of the first wave data
Creators: name: Wischnewski, Magdalena
Publisher: name: OSF Registries
Year: 2024
Subjects: subject: Social Psychology
subject: Social and Behavioral Sciences
subject: Psychology
subject: FOS: Psychology

description: Development and validation of a new scale to measure trust in AI-powered systems


Second Assessment Wave Data Collection and Analysis

Three vignettes will be designed, based in part on the pre-trial vignettes but more elaborate. We aim for a sample of N = 600 participants per vignette. We determined the sample size we aim for by using rules of thumb to estimate what sample size is required for the most complex model we plan to fit to the data. 

Our goal is to avoid missing values via strict no-skipping rules in the survey implementation. We plan to carry out a complete case analysis.

The first step is to investigate whether we can assume measurement invariance (MI). If we can, then we are going to fit one model for item scores pooled across three vignettes, if we cannot, we are going to fit three separate models, one per vignette.

We are going to assess MI sequentially, starting with configural, then moving from weak over strong to strict. We are not going to investigate partial MI. We are going to use the single items as indicators.

We plan to conduct the following analyses. We are going to perform them in R with the packages psych, lavaan, and semTools. We are going to use the wlsmv estimator for ordered items in lavaan. For any test statistics and fit measures, we are correspondingly going to use robust options as well.

Item-wise description (per vignette):
• Item-wise distributions with bar plots
• Item-specific means and standard deviations
• Compare with description from first-wave assessment

Correlation table (per vignette):
• Compute correlations between all items
• Compare with description from first-wave assessment

Per vignette, for the trust scales, we are going to compute reliability estimates, using Cronbach’s coefficient alpha and hierarchical omega, using 6 factors. 

Per vignette, for the additionally administered items measuring situational factors (i.e., riskiness of the situation, affective responses, control agency, perceived anthropomorphism), we plan to compute internal consistency (using Cronbach’s coefficient alpha) per scale/construct. Compare with reliability estimates from first-wave assessment. 

After this scale assessment, we plan to compute appropriate sum scores and carry out correlational analyses together with (appropriate) sum scores from developed trustworthiness scales. The correlation matrices are going to be computed per vignette and compared between vignettes.

We are further going to carry out multiple-group CFA analyses (using the package lavaan and semTools) to assess measurement invariance (MI) between vignettes. 

For the possible multiple-group CFA models, we consider and are going to compute the following options:
Correlated-Factors CFA  compute but assume to discard
2-level CFA  compute but assume to discard
Bifactor (S-1) CFA (Eid et al., 2017)  most aligned with our test construction
- Orthogonal variance of all specific factors to GT
- Specific factors amongst each other allowed to correlate

Depending on the MI results, we are going to proceed differently. In case we found evidence for MI, we are going to fit a CFA model to the data from all vignettes together and estimate factor loadings. We are going to correlate the corresponding extracted factor scores with simple sum scores to allow for a recommendation regarding scoring rules.

For the context factors, we are going to fit SEMs to assess their relation to the trust scores. Using all manifest context variables and the factor model for trust scales, we are going to use the regsem (Jacobucci et al., 2016) package to select context variables with notable effects on any of the trust factors, using stability selection (Li &amp; Jacobucci, 2022). 

Hypothesis for context factors

• We will explore the relationship to the Big 5 exploratively
• Users’ agency/control correlates positively with the system’s transparency.
• Machine’s agency correlates positively with ability.
• Higher machine agency increases the variance of the factor unbiasedness 
• The more anthropomorphic a system is perceived, the more individuals trust it.
• Anthropomorphism correlates the most with the factor integrity. 
• The more risky the situation, the less people trust the machine. 
• The higher individuals’ propensity to trust (PPT), the higher trust.
• PPT correlates more with general trust factor than with specific trust factors.
• The more participants trust, the more positive (and less negative) affect they experience.

References

Eid, M., Geiser, C., Koch, T., &amp; Heene, M. (2017). Anomalous results in G-factor models: Explanations and alternatives. Psychological Methods, 22(3), 541.

Jacobucci, R., Grimm, K. J., &amp; McArdle, J. J. (2016). Regularized structural equation modeling. Structural equation modeling: A Multidisciplinary Journal, 23(4), 555-566.

Li, X., &amp; Jacobucci, R. (2022). Regularized structural equation modeling with stability selection. Psychological Methods, 27(4), 497.