DOI: 10.5061/dryad.573n5tbh8
title: Data for: How do we measure and increase systems thinking? Comparing self-reported and performative metrics in response to building causal loop models

description: Systems thinking is a mindset and skill essential to understanding and
 taking effective action to address complex challenges. However, increasing
 and measuring systems thinking is difficult. This data set includes
 multiple approaches to measuring systems thinking. It includes 3
 self-reported measures of a systems thinking mindset, and 2 performative
 measures of systems thinking skills (metrics derived from causal model
 diagrams of a system that participants created, and accuracy of responses
 to simple trophic cascade scenarios). The data allow for a comparison
 between different measurement approaches. They also allow for the
 evaluation of whether systems thinking is specific to disciplinary domains
 (ecological vs economic) or generalizes across domains.  Finally,
 the data provides a preliminary assessment of whether building causal
 models increased systems thinking.
description: Participants (N = 185) were workers from Amazon Mechanical Turk
 (MTurk) and located in the United States. Participants were paid $5.00
 (roughly $10/hour). Our survey used a mixed-methods
 counterbalanced design, resulting in both within-and between-subjects
 measures described in the table below.  The survey was
 administered using Qualtrics and was entirely online. After providing
 informed consent, all participants completed pre-measures (the exact
 measures varied based on random assignment). All participants then watched
 a 5-minute instructional video created by the researchers and available at
 https://vimeo.com/479005460. The video explained and provided an example of how to use <em>Mental Modeler</em>. It did not, however, use the term model or system or discuss or explain anything about systems thinking. Participants then read a written description of the ecological system. This narrative described how the foxes were dependent on the rabbits as a food source, and that the foxes reduced the number of rabbits by eating them, how the rabbits were dependent on the grass, how the grass was fertilized by the droppings of both species, etc. In addition to including the causal relationships just described, the narrative also included several non-causal factors. For example, it was stated that the landscape contained rocks. The description was written to define an unambiguously correct set of causal relationships that would be present in the model. A link then took participants to the <em>Mental Modeler</em> software, where they were asked to create a mental model of the scenario described to them. Participants were instructed to download the file produced by the software, as well as take a screenshot. Both were uploaded into the online survey. Participants then continued with the survey to complete the appropriate post-measures, given their random assignment. The measures are described below. <strong>Measure</strong> <strong>Description</strong> <strong>Role in Study</strong> <strong>Sample item</strong> Systems Thinking Scale (Randal & Stroink, 2018) Measures self-reported dispositional systems thinking Measured pre-MM exercise Q: “The Earth, including all its inhabitants, is a living system.” A: 1 = “Strongly Disagree” to 7 = “Strongly Agree” Ecological Knowledge (developed for this study) Measures self-reported familiarity with ecological concepts Measured pre-MM exercise, for use as covariate Q: “I have taken a biology or environmental science course in college or high school that provided me with an understanding of basic ecological concepts.” A: “Yes” or “No” Local Systems Perception Scale (Petersen et al., 2018) Self-report measure of the extent to which participants see themselves as part of local ecological, economic, and social systems Measured pre- and post-MM exercise, to evaluate whether MM exercise changed answers Q: “I think of the place I live as a social system composed of interrelated parts.” A: 1 = “Strongly Disagree” to 5 = “Strongly Agree” Perceptions of Causal Connection (Petersen et al., 2018) Measures participants’ self-reported tendency to see causal connections across different spatial and temporal scales Measured pre- and post-MM exercise, to evaluate whether MM exercise changed answers Q: “My local actions affect things that happen at a global scale.” A: 1 = “Strongly Disagree” to 5 = “Strongly Agree” Ripple Effect scenario (based on Maddux & Yuki, 2006) • Two ecological scenarios (goldfish, cat) •One economic scenario (bookstore)   Description of a trophic cascade scenario that presents both relevant and irrelevant factors   Seen either before or after MM, counter-balanced “A neighborhood borders a nature preserve …[that] contains a food web composed of sedge plants, insects, mice, lizards and hawks.  … [a free-roaming cat] catches and kills mice but does not hunt either lizards or birds.” Ripple Effect impact questions Performative assessment of the tendency to accurately identify chains of causality rippling out from a single event Performative measure of systems thinking. Measured either before or after MM, counter-balanced, to evaluate whether MM exercise changed answers Q: “Use the choices below to indicate the degree of impact of this introduction on the reservoir … 1. the goldfish” A: -2 = “Decrease substantially” to 2 = “Increase substantially” Ripple Effect responsibility questions Performative assessment of perceived responsibility for changes to a system   Performative measure of systems thinking. Measured either before or after MM, counter-balanced, to evaluate whether MM exercise changed answers   Q: “To what extent are the following responsible for the impacts of the goldfish on the reservoir? 1. Ted’s parents” A: 1 = “Not at all responsible” to 3 = “Very responsible” <em>Mental Modeler</em> metrics Numbers of each model feature Performative measure of model-building Descriptive metrics of participants’ models Number of positive arrows <em>Mental Modeler</em> accuracy metrics Performative measure of accuracy of each variable for each model   Performative measure of systems thinking. Completed between the two ecological ripple questions Number of correct positive arrows
description: # Data for How do we measure and increase systems thinking? Comparing
 self-reported and performative metrics in response to building causal loop
 models
 [https://doi.org/10.5061/dryad.573n5tbh8](https://doi.org/10.5061/dryad.573n5tbh8) ## Description of the data and file structure Data can be found in the file DataForHowDoWeMeasureIncreaseSystemsThinking.csv. Each row corresponds to one participant, and missing data is indicated with a blank field. Variable names and values can be found in the file MetaDataForHowDoWeMeasureIncreaseSystemsThinking.csv.Files and variables #### File: MetaDataForHowDoWeMeasureIncreaseSystemsThinking.csv **Description:** Metadata that describes the meaning of each variable and value in the data file #### File: DataForHowDoWeMeasureIncreaseSystemsThinking.csv **Description:** Data ## Supplemental materials (Zenodo) Detailed descriptions of aspects of the study can be found in the supplemental materials files.  **File**: SI1RippleEffectQuestionDetailsFinal.pdf **Description:** This file presents detailed analyses for the individual ripple effect questions (they are averaged together in the main analyses). **File**: SI2MentalModelerInstructionstoParticipants.pdf **Description:** This file details the instructions participants received before creating their model. **File:** SI3MentalModelerCodingInstructions.pdf **Description:** This file contains the full coding instructions used by researchers to code the Mental Models drawn by participants ## Code/software (Zenodo) The data file was created in SPSS but is stored in csv format and can be imported into open software such as R, as well as licensed software such as SPSS. #### Compressed File: SyntaxForFCMtoMeasureSystemsThinking **Description:** SPSS Syntax. SPSS software is needed to open these files. The following files are SPSS syntax files that were used for the analyses indicated in the file names. File: FCMtoMeasureSystemsThinkingSyntaxAnalysesinSupplement1.sps File: FCMtoMeasureSystemsThinkingSyntaxAnalysesReportedinPaper.sps File: FCMtoMeasureSystemsThinkingSyntaxChronbach'sReliability.sps File: FCMtoMeasureSystemsThinkingSyntaxCorrelationsEcologicalKnowledgewithOtherVariables.sps File: FCMtoMeasureSystemsThinkingSyntaxModelVariablesCoding.sps File: FCMtoMeasureSystemsThinkingSyntaxRecodeComputeRippleResponsibilityQuestions.sps File: FCMtoMeasureSystemsThinkingSyntaxRecodeComputeSelf-reportScales.sps File: FCMtoMeasureSystemsThinkingSyntaxRecodeRippleImpact.sps File: FCMtoMeasureSystemsThinkingSyntaxtoComputeRippleImpactAccuracy.sps ## Human subjects data We received written consent from participants to publish de-identified data. IP addresses, prolific IDs, and all demographic data has been removed from the file.