DOI: 10.5061/dryad.h18931zx2
title: Meta-synthesis and R analysis of stakeholder engagement in food, energy, and water systems literature

description: We conducted a literature review for manuscripts including food, energy,
 and water systems, and stakeholder engagement. Each manuscript was
 analyzed and the following data was entered into an Excel workbook in
 numerical, narrative or yes/no format: year of publication, citation of
 manuscript, location where research was conducted, country of residence of
 authors, author(s) affiliation, funding agency, whether a solution to the
 issue addressed in the paper was proposed or implemented, whether the
 authors employed a statistical or computational model, scale of solution,
 type of solution, types of stakeholders involved, description of when
 stakeholders were involved, how stakeholders were involved, and how
 stakeholders were identified. The variables that were analyzed
 using R were coded to a .csv format. Those variables include:
 solutions proposed, solutions implemented, type of solution, whether a
 computational or statistical model was used, researcher field, stakeholder
 types, level of stakeholder engagement measured by three scales
 (Ghodsvali, IAP2, and a scale developed by the authors), geographic scale
 of the issue addressed in the study, location of study and residence of
 researchers. R was used to analyze the data. Full
 details of the R analysis are included in a pdf file uploaded with the
 Excel and csv data.
description: <strong>Literature search and
 selection</strong>  We conducted two literature
 searches: an initial search in 2020 and a follow-up search ending in April
 2023 to capture literature published since the initial search. We
 conducted the searches in two different online databases, ScienceDirect
 and WorldCat, to ensure a comprehensive identification of relevant
 literature. We were interested in examining papers related to the nexus of
 food, energy, water systems (FEWS) so we used the following search string:
 [(“food energy water” OR “food water energy” OR “water energy food” OR
 “water food energy” OR “energy water food” OR “energy food water”) AND
 (stakeholder OR engagement OR community OR nexus)]. In ScienceDirect, we
 searched for this Boolean expression in “Title, Abstract, and Author
 specified keywords,” and in WorldCat, we searched in “Keywords.” We
 eliminated duplicates in the two databases. We limited our search to the
 English language and to articles and book chapters published since 2015.
 We reviewed the titles, abstracts, and keywords of each paper to determine
 whether they were relevant to FEWS. If the abstract was unclear, we
 conducted a word search within the body of the articles to examine the
 context to the nexus. We identified 177 publications from our initial
 search and 540 publications in our follow up search, resulting in a total
 of 717 publications. We then manually screened the papers and removed
 those in which there was no indication that the paper included at least
 one term from both groups of terms, those that related to fewer than two
 FEW systems (food and water, food and energy, water and energy, or food
 and energy), and review articles. Our database totaled 483 papers for
 analysis.  <strong>Literature
 assessment</strong> We analyzed each article to
 characterize the existence and extent of stakeholder involvement. In the
 beginning of the literature assessment, a sample of 50 articles were
 assessed by two team members to ensure agreement on the concepts of
 interest, in particular the level of engagement and types of solutions
 that were proposed in the article as those were more subject to
 differences of opinion. If we were unable to resolve differences of
 opinion, our working group met monthly and we brought questions for group
 discussion and resolution. Once sufficient agreement was reached, those
 team members individually analyzed the remainder of the papers and brought
 additional questions to the working group. We identified the citation
 year, the affiliation of the authors (department within their
 organization), whether stakeholders were involved, the extent of
 stakeholder involvement, how stakeholders were identified, and when they
 were involved. We also determined whether a solution was proposed or
 implemented. The data was entered into Excel in narrative or numeric
 format, as appropriate. Of the 483 papers, 90 engaged stakeholders at some
 level, but six did not describe stakeholder engagement sufficiently to
 characterize the interactions resulting in 84 that were analyzed for
 stakeholder engagement. We define ‘stakeholders’ as
 actors or organizations who have an interest in, have power or influence
 over, or are affected by the research or its implications. This may
 include individuals or groups, state or non-state actors, community
 members, or others. Although researchers and scientists can be
 stakeholders, for this research, they were not counted if they were the
 sole stakeholders, but were included if other stakeholders were involved
 in the research. We recorded all types of stakeholders described in each
 paper in an Excel database. The list was grouped into types through
 discussions by the working group. The typology developed included farmers,
 ranchers and landowners; governmental entities; coalitions, NGOs, and
 citizen groups; industry and local businesses; the general public;
 indigenous entities/tribal representatives; experts; youth; migrants; and
 academics. We defined ‘stakeholder engagement’ as the
 inclusion of people or organizations in any part of the project or
 research. We identified whether stakeholders were engaged and categorized
 the type of stakeholder engagement in each publication. To categorize
 stakeholder engagement, we used two frameworks (Table 1). The first is
 adapted from a categorization of transdisciplinary research for the FEWS
 nexus and includes engagement at four levels: Nominal, instrumental,
 representative, and transformative. The second framework, titled the
 Spectrum of Public Participation, was developed by the International
 Association for Public Participation (IAP2) to define the public’s role in
 a public participation process. The IAP2 framework has five levels:
 Inform, consult, involve, collaborate, and empower. In both frameworks,
 the levels denote increasingly greater roles, influence, or authority
 given to stakeholders in the engagement process. If stakeholders were
 engaged at multiple levels in a project, we coded that engagement at the
 highest level described in the paper.  The working
 group also developed its own scale with an emphasis on the interests of
 stakeholders and on finding solutions to problems at the community level.
 Those levels included: Data gathering; data gathering, but the study
 addresses a problem of importance to stakeholders; inform or educate
 stakeholders; help stakeholders view issues from different perspectives;
 engage stakeholders to plan for the future; help stakeholders identify
 solutions; help stakeholders envision how to put solutions into practice;
 and help stakeholders put solutions into practice. Each engagement was
 coded to the highest level described in the manuscript.
 In addition to analyzing whether and to what extent stakeholders
 were involved in the research, we analyzed manuscripts to determine
 whether researchers described how they had identified their stakeholders
 and when they had engaged them. We entered that information into the
 database in narrative form. We created an additional column in which we
 entered Y (Yes) or N (No) in Excel to count the number of papers that
 explicitly included these descriptions. We defined
 solutions and determined if they were proposed or implemented for each
 paper. Our working group developed the following definition and
 description of proposed solutions: A proposition to address or solve the
 problem identified in the manuscript. The manuscript must explicitly
 articulate the solution(s). Solutions should have intended outcome(s) to
 improve the previous state, must be implementable or achievable by the
 community experiencing the identified problem, must be related to the
 research conducted in that publication, and must inform decisions or
 behaviors in practice or be relevant to decision-making. Solutions must
 have sufficient specificity that they can be implemented by stakeholders
 without significant additional work. Solutions cannot be recommendations
 for further research or generic policy recommendations and cannot be
 hypothetical or theoretical. A list of additional research questions is
 not a solution, nor are ways in which research in FEWS networks can be
 improved if it only benefits researchers. We defined
 and operationalized implementation as “included methods that sought to be
 applied in practice by being accessible to decision-makers, addressing
 relevant questions, and offering a systematic process to employ the
 method(s) in practice” and creating a system to employ methods in practice
 (p. 5). Solution implementation should follow a plan of action (e.g.,
 process, workflow or framework with stages/phases) with details on the
 actions and resources needed to apply the solution. It can be monitored
 and evaluated over time. Providing the tools and methods to implement a
 solution to stakeholders, even if they choose not to implement it, is
 still considered implementation. For example, if a computer model is made
 available in a way that it is usable to the public, it is considered to be
 an implemented solution. Author affiliation was
 determined based on the listing in the manuscript. Affiliations were
 categorized into: NGO, engineering, math, computer science, physical
 science, interdisciplinary, social science, economics, agriculture, and
 other. If the affiliation was not clear from the manuscript, for example,
 if the affiliation was a center, one author searched for the listed
 affiliation and examined the stated purpose of the center to determine the
 affiliation. There were a number of affiliations that were unavailable and
 those were not coded. Many of the descriptions were unclear on websites,
 especially those located in China. The authors acknowledge that this
 dataset is not robust and attempts to reproduce this data are likely to
 have different results. The following data were coded
 and entered into a csv for analysis using R (Table 2).
 Table 2: Variable Descriptions 
 Variable
 Name 
 Description R
 Coding Year Year of
 Citation year 
 Solution proposed
 Was a solution proposed? solution
 proposed Solution implemented Was a
 solution implemented? solution implemented
 Stakeholder
 types Farmers, ranchers and landowners; governmental
 entities; coalitions, NGOs and citizen groups; industry and local
 businesses; the general public; experts; youth; indigenous entities/tribal
 representatives; migrants; academics. Farmers,
 combined government, combined coalition, combined industry, public,
 experts, youth, tribal nations, migrants, university.
 Ghodsvali
 stakeholder engagement scale (Table 1) Nominal,
 Instrumental, Representative, Transformative nominal,
 instrumental, representative, transformative 
 IAP2 stakeholder engagement
 scale (Table 1) NA, Inform, Consult, Involve,
 Collaborate, Empower data gathering, inform, consult,
 involve, collaborate, empower Williams et al. developed stakeholder engagement
 scale (Conceptual framework 3) Benefits researchers;
 problem of importance to stakeholders; informs stakeholders; different
 perspectives; plan for future, identify solutions; envision solutions in
 practice; put solutions into practice researcher,
 data gathering, inform, perspectives, plan, identify, envision,
 implement 
 Statistical Analysis Using the full sample
 size of the refined literature review (n = 483), several tests of
 independence (chi-squares test, Fishers exact test, Barnards unconditional
 test) were initially performed to determine if there was a significant
 association between whether a solution was proposed or not in comparison
 to whether stakeholder engagement was initiated. An initial generalized
 linear model (glm) was then constructed to evaluate the relationship of
 stakeholder engagement (all groups included), in comparison to whether a
 solution was proposed or not. To examine the diversity of stakeholder
 engagement and its impact on the likelihood of a proposed solution, a
 secondary glm was constructed which incorporated a diversity index, based
 on the number of stakeholder types involved in a particular research
 effort. The diversity index was calculated as follow:
 Diversity Index = total number of stakeholder types involved per
 research effort                          total
 potential stakeholder types (n=10) This index was then
 calculated for each observation and used as part of our secondary glm,
 using a binomial distribution, with whether a solution was proposed or not
 as the response variable (Figure x). Levels of stakeholder engagement are
 described in Table 1. 
description: # Meta-synthesis and R analysis of stakeholder engagement in food, energy,
 and water systems literature
 [https://doi.org/10.5061/dryad.h18931zx2](https://doi.org/10.5061/dryad.h18931zx2) ## Description of the data and file structure A meta-synthesis of food, energy, and water systems manuscripts was conducted to determine the extent of stakeholder engagement, how stakeholders were engaged, outcomes of engagement, and to identify best practices in stakeholder engagement. ### Files and variables #### Files: crc-engage.zip includes a PDF description of R analysis contained in Supplemental Information (Supplemental_Information_2025.pdf), an Excel workbook, and a .csv for R data analysis **Description:** Description of R analysis of the data #### File: csv_for_statistical_analysisAug2025.csv **Description:** csv for input into R for analysis ##### Variables * were stakeholders involved (Y, N) * level of stakeholder engagement (Ghodsvali Scale) - Whether stakeholders were involved in the research and details about their involvement if they were included. Stakeholder engagement is defined and operationalized as the inclusion of non-academic people or organizations in any part of the research process. Table 3 describes the SE scale used (Ghodsvali et al., 2019). * solution proposed and/or implemented - Whether a solution was proposed and details about the solution if it was proposed. A solution is defined and operationalized as a proposition to address or solve the identified problem. The article must explicitly articulate the solution, including a mechanism or plan of action for its implementation. (e.g., process, workflow, or framework with defined stages or phases) with details on the actions and resources needed to apply the solution. Providing the tools and methods for implementation of the solution to stakeholders is considered a solution, even if the stakeholders do not implement it. Solutions were required to be studied as a variable or directly related to the variables studied in the model, experiment, or methods. The solution must have intended outcomes to improve the previous state; must relate to the research conducted in the article; must inform decisions, decision-making, or behavior in practice; must be implementable and achievable; cannot be recommendations for future research; and cannot be hypothetical or theoretical. Whether the solution was implemented and details about the implementation. Implementation is defined and operationalized as the application in practice of methods specified in the article by making a tool, process, or other solution accessible to decision-makers, by creating a system to employ the methods in practice, or by putting the methods into practice within the community (Albrecht et al., 2018). If a computer or statistical model was accessible to policymakers through an interface allowing them to interact with the model without training and a url to use the model was provided we counted that as an implemented solution. * solution type - Technological Use of engineered or designed technology (civil, chemical, computer, electrical, and mechanical). Examples include sensors to monitor and adjust use (e.g., energy or water), anaerobic digesters, alternative or green energy generators, computational or statistical models and software, cyberinfrastructure, gamification, infrastructure additions, improvements, or changes to infrastructure (Carroll, 2017). Policy or Policy Change Use of bylaws, codes, regulations, statutes, strategies, or other formal policy mechanisms (Colebatch, 2009). Institutional Use of mechanisms that are directed at change at a governance level (city, county, state, nation, academic). This includes interventions by the institutions themselves, such as changes in mandate, mission, organization, roles, structure, members, functions, relationships, procedures, operations, or the creation of a new institution (Gräbner and Ghorbani, 2019). Social Any solution that requires awareness of, consideration of, or modification of attitudes, beliefs, preferences, behaviors, biases, mores, norms, or values. Includes outreach. Potential solutions could include capacity building (e.g., increases in the ability of groups to overcome a problem, providing money, social capacity). Economic, Financial, or Market Use of economic instruments including taxes and subsidies. Includes both economic incentives (tax credits, investments, subsidies, etc.) and economic disincentives (imposing a tax, fines, penalties, tariffs, etc.) Also includes economic analysis or economic models if proposed as part of another solution. Economic models do not count as economic solutions in and of themselves; the authors must provide an economic recommendation and a user interface and access (Backhouse and Medema, 2009). Ecological Use of biota, conservation practices related to ecosystems, ecological corridors, greenspaces, organisms, or pathogens to effect changes or maintenance of atmospheric, hydrological, oceanic, or terrestrial systems. Ecological outcomes or models do not count as ecological solutions on their own. Instead, the authors must provide an ecological recommendation (Wallace, 2007). Educational Use of instructional, formal or informal learning approaches to inform or transfer knowledge to citizens or stakeholders and improve their situational awareness and critical thinking. This includes games to educate and inform. This also includes service/outreach (e.g., service-learning, extension services, etc.). Education solutions transfer knowledge through tools; provide frameworks to address issues and self-sustaining methods to continue improved performance within communities; provide outreach to or from schools, and other educational institutions. The main goal of an educational solution is to improve awareness, cognition, and critical thinking (Rizvi and Lingard, 2009). 
 
 * researcher field of expertise - Disciplines of researchers were based
 on the affiliation or department(s) of the authors listed in the
 manuscript. These were categorized as engineering,
 mathematical/statistical, computer science, physical science (e.g.,
 chemistry, biology, ecology, hydrology, etc.), interdisciplinary
 (environmental science, sustainability or an institute that included both
 physical and social science components), social science, economics and
 agriculture. If the discipline was unclear from the name of the
 department, a Google search was conducted and a determination was made
 based upon the description on the departmental website. For those
 departments we were unable to find, we did not include the discipline in
 the analysis. * stakeholder type - Based on working group experience and
 discussions we included farmers; governmental; tribal/indigenous; combined
 coalitions, NGOS and citizen groups; business/industry; general public;
 youth; academics; experts; and migrants. "Experts" was added as
 a separate category because this was the word used in many manuscripts to
 describe their stakeholders. * Geographic scale of study - Local included
 households to metropolises. Regional included county to sub-national
 levels. National included nations. Multi-national included more than one
 nation and Global included worldwide studies. NA was coded if the scale of
 the study was not described (Leitner, 1997). * Location of study/regional
 scale - Country where the research or a case study took place. This could
 be the location of: data used in a model, experiment, community
 engagement, or other research. The countries were combined into the
 following geographic areas for analysis: China, US, Africa, Asia, Central
 America, Europe, Global, Middle East, North America, Oceania, and South
 America. If no region of study was provided, it was coded NA. China and
 the US were analyzed separately from their region because these two
 countries accounted for an outsize proportion of the articles (34%). * Was
 a computer or statistical model used - If a computer-generated or
 mathematical model was used, we input Y in the cell and entered a short
 description of the model in Excel. 
 
 #### File: complete_ENGAGE_analysisclean.xlsx **Description:** Original
 data entered from manuscripts into Excel in narrative or numerical form.
 Literature search and selection. Absent data are listed as "N/A"
 or are blank cells. To synthesize the literature on development and
 implementation of solutions to FEWS challenges, we conducted a systematic
 review, following PRISMA guidelines for systematic reviews and
 meta-analyses (Page et al., 2021). Systematic reviews can be designed to
 answer narrowly defined questions and critically appraise evidence
 (Atkinson and Cipriani, 2018)– we do this for the three research questions
 listed above. Eligibility criteria/Search strategy Our search string
 included the terms: [(“food energy water” OR “food water energy” OR “water
 energy food” OR “water food energy” OR “energy water food” OR “energy food
 water”) AND (stakeholder OR engagement OR community OR nexus)]. In
 ScienceDirect, we searched for this Boolean expression in “Title,
 Abstract, and Author specified keywords” and in WorldCat we searched in
 “Keywords.” We limited our search to the English language and to journal
 articles and book chapters. The initial searches returned 717 articles
 from which we removed 3 duplicate records. We reviewed the abstracts of
 the remaining 714 articles to verify whether the study included both SE
 and FEWS. If the abstract was unclear, we conducted a text search to
 examine the context around key terms within the body of the articles. If
 there was no indication that the paper included at least one term from
 both groups of terms (FEWS and SE) and that the paper related to at least
 two FEWS sectors, it was excluded (n=123). Any paper was also excluded if
 it was a review article or a purely conceptual paper, as we wanted to
 focus on solutions based on empirical investigation (n=106). We then
 manually screened the papers and removed papers that did not meet the
 criteria above upon reading the manuscript, yielding 483 papers for
 inclusion in the review (Figure 1). Information sources We searched the
 ScienceDirect and WorldCat online databases to identify relevant
 peer-reviewed research articles published between 2015 and April 2023.
 Selection process/Data collection process/Bias assessment Following the
 principles of systematic literature review (PRISMA) we analyzed each
 article to identify variables describing geographic scale, region, a
 stated proposed solution, a stated implemented solution, category of
 stakeholder, discipline of researcher(s), types of FEWS solutions, and
 types of stakeholder engagement levels (Tables 1 – 3). To start, a sample
 of 50 articles were assessed by two team members to ensure agreement on
 the concepts of interest, particularly regarding the level of engagement
 and the types of solutions proposed. Potential biases in study selection,
 types of solutions, and engagement levels were further minimized by
 assigning screening tasks to more than one person independently with
 regular reviews of the sample to address reviewer bias. Once sufficient
 agreement was reached, one team member individually analyzed the remainder
 of the papers with regular sampling and review by the coauthor group. Any
 questions were taken to monthly working group meetings (consisting of the
 authors and representatives of the research coordination network) for
 discussion and resolution. Table 1: Description of the variables that were
 examined in the selected research articles. **Research Variable**
 **Description** Proposed solution (*Dependent variable*) Whether a
 solution was proposed and details about the solution if it was proposed. A
 solution is defined and operationalized as a proposition to address or
 solve the identified problem. The article must explicitly articulate the
 solution, including a mechanism or plan of action for its implementation.
 (e.g., process, workflow, or framework with defined stages or phases) with
 details on the actions and resources needed to apply the solution.
 Providing the tools and methods for implementation of the solution to
 stakeholders is considered a solution, even if the stakeholders do not
 implement it. Solutions were required to be studied as a variable or
 directly related to the variables studied in the model, experiment, or
 methods. The solution must have intended outcomes to improve the previous
 state; must relate to the research conducted in the article; must inform
 decisions, decision-making, or behavior in practice; must be implementable
 and achievable; cannot be recommendations for future research; and cannot
 be hypothetical or theoretical. Implemented solution (*Dependent
 variable*) Whether the solution was implemented and details about the
 implementation. Implementation is defined and operationalized as the
 application in practice of methods specified in the article by making a
 tool, process, or other solution accessible to decision-makers, by
 creating a system to employ the methods in practice, or by putting the
 methods into practice within the community (Albrecht et al., 2018). If a
 computer or statistical model was accessible to policymakers through an
 interface allowing them to interact with the model without training and a
 url to use the model was provided we counted that as an implemented
 solution. Stakeholder type (*Independent variable*) Based on working group
 experience and discussions we included farmers; governmental;
 tribal/indigenous; combined coalitions, NGOS and citizen groups;
 business/industry; general public; youth; academics; experts; and
 migrants. "Experts" was added as a separate category because
 this was the word used in many manuscripts to describe their stakeholders.
 Discipline of researchers (*Independent variable*) Disciplines of
 researchers were based on the affiliation or department(s) of the authors
 listed in the manuscript. These were categorized as engineering,
 mathematical/statistical, computer science, physical science (e.g.,
 chemistry, biology, ecology, hydrology, etc.), interdisciplinary
 (environmental science, sustainability or an institute that included both
 physical and social science components), social science, economics and
 agriculture. If the discipline was unclear from the name of the
 department, a Google search was conducted and a determination was made
 based upon the description on the departmental website. For those
 departments we were unable to find, we did not include the discipline in
 the analysis. Geographic Scale (*Independent variable*) Local included
 households to metropolises. Regional included county to sub-national
 levels. National included nations. Multi-national included more than one
 nation and Global included worldwide studies. NA was coded if the scale of
 the study was not described (Leitner, 1997). Regional Scale (*Independent
 variable*) Country where the research or a case study took place. This
 could be the location of: data used in a model, experiment, community
 engagement, or other research. The countries were combined into the
 following geographic areas for analysis: China, US, Africa, Asia, Central
 America, Europe, Global, Middle East, North America, Oceania, and South
 America. If no region of study was provided, it was coded NA. China and
 the US were analyzed separately from their region because these two
 countries accounted for an outsize proportion of the articles (34%). Model
 (*Independent variable*) If a computer-generated or mathematical model was
 used, we input Y in the cell and entered a short description of the model
 in Excel. Stakeholder involvement (*Independent variable*) Whether
 stakeholders were involved in the research and details about their
 involvement if they were included. Stakeholder engagement is defined and
 operationalized as the inclusion of non-academic people or organizations
 in any part of the research process. Table 3 describes the SE scale used
 (Ghodsvali et al., 2019). Time of stakeholder involvement (*Independent
 variable*) If adequately described in the article, we characterized when
 stakeholders were involved as throughout the project, early in the
 project, midway in the project or late in the project.   **Table 2**:
 Description of the seven types of solutions that were identified in the
 review. 
 
 **Types of Solutions** **Description** Technological Use of engineered or
 designed technology (civil, chemical, computer, electrical, and
 mechanical). Examples include sensors to monitor and adjust use (e.g.,
 energy or water), anaerobic digesters, alternative or green energy
 generators, computational or statistical models and software,
 cyberinfrastructure, gamification, infrastructure additions, improvements,
 or changes to infrastructure (Carroll, 2017). Policy or Policy Change Use
 of bylaws, codes, regulations, statutes, strategies, or other formal
 policy mechanisms (Colebatch, 2009). Institutional Use of mechanisms that
 are directed at change at a governance level (city, county, state, nation,
 academic). This includes interventions by the institutions themselves,
 such as changes in mandate, mission, organization, roles, structure,
 members, functions, relationships, procedures, operations, or the creation
 of a new institution (Gräbner and Ghorbani, 2019). Social Any solution
 that requires awareness of, consideration of, or modification of
 attitudes, beliefs, preferences, behaviors, biases, mores, norms, or
 values. Includes outreach. Potential solutions could include capacity
 building (e.g., increases in the ability of groups to overcome a problem,
 providing money, social capacity). Economic, Financial, or Market Use of
 economic instruments including taxes and subsidies. Includes both economic
 incentives (tax credits, investments, subsidies, etc.) and economic
 disincentives (imposing a tax, fines, penalties, tariffs, etc.) Also
 includes economic analysis or economic models if proposed as part of
 another solution. Economic models do not count as economic solutions in
 and of themselves; the authors must provide an economic recommendation and
 a user interface and access (Backhouse and Medema, 2009). Ecological Use
 of biota, conservation practices related to ecosystems, ecological
 corridors, greenspaces, organisms, or pathogens to effect changes or
 maintenance of atmospheric, hydrological, oceanic, or terrestrial systems.
 Ecological outcomes or models do not count as ecological solutions on
 their own. Instead, the authors must provide an ecological recommendation
 (Wallace, 2007). Educational Use of instructional, formal or informal
 learning approaches to inform or transfer knowledge to citizens or
 stakeholders and improve their situational awareness and critical
 thinking. This includes games to educate and inform. This also includes
 service/outreach (e.g., service-learning, extension services, etc.).
 Education solutions transfer knowledge through tools; provide frameworks
 to address issues and self-sustaining methods to continue improved
 performance within communities; provide outreach to or from schools, and
 other educational institutions. The main goal of an educational solution
 is to improve awareness, cognition, and critical thinking (Rizvi and
 Lingard, 2009).   **Table 3**: Description of stakeholder engagement
 levels from the framework used in the analysis (Ghodsvali et al., 2019). 
 
 **Categorization of FEWS Engagement** Engagement Level Description
 Nominal Limited stakeholder involvement that does not lead to change in
 the outcomes (e.g., data gathering from stakeholders) Instrumental Use of
 skills and knowledge of stakeholders in the research process
 Representative Give stakeholders a voice in decision-making and in the
 implementation of solutions that may affect them Transformative Focus on
 empowering local stakeholders   Data items/Synthesis methods We identified
 solutions and determined if they were implemented using the guidelines set
 out in Table 1. We then categorized the identified solutions into seven
 types: technological, policy, institutional, social, economic, ecological,
 and educational (Table 2). The typology of solutions was an outcome of an
 expert panel assessment from the more than 100 members of the EngageINFEWS
 research coordination network (Kliskey et al., 2021). For technological
 solutions, only computational models made available in a manner usable by
 policymakers and/or the general public were included as solutions, which
 required providing the URL and a user interface. Models with the sole
 purpose of the advancement of computational or statistical science, or for
 the purpose of research, were identified but not counted as solutions. For
 ecological solutions, we differentiated between ecological actions (e.g.,
 preserving wetlands) which were included as solutions, versus ecological
 outcomes (e.g., improved water quality) or other activities that did not
 have explicit ecological influence (e.g., water recycling) which were not
 included. Finally, we included only solutions that were directly related
 to the research and were specifically described. For example, a study that
 said ‘policy solutions should be created for water management’ would not
 count as a solution, but a study that recommended ‘policies that provide
 economic subsidies to farmers to improve the efficiency of their
 irrigation practices’ with a plan or mechanism described for providing
 those subsidies would count. Solution types were coded in a CSV file for
 subsequent analysis in R using binary values for each of the seven
 separate types of solutions. Each solution type was entered into a
 separate column, so technological solutions, for example, in one column,
 was coded a ‘0’ if that type of solution was not present and coded as ‘1’
 if a technological solution was present. This binary coding was undertaken
 for each solution type across seven separate columns. If there was missing
 data it was simply excluded. Table 2: Description of the seven types of
 solutions that were identified in the review. We also identified whether
 stakeholders were engaged and if so, categorized the type of engagement
 and when they were engaged (Table 3). We define a ‘stakeholder’ as actors
 or organizations who have an interest in, have power or influence over, or
 are affected by the research or its implications. This may include
 individuals or groups, state or non-state actors, community members, or
 others. Although researchers and scientists can be stakeholders (Kliskey
 et al., 2021), for this analysis they were not counted if they were the
 only stakeholder but were included if there were other stakeholders. We
 defined ‘stakeholder engagement’ (SE) as the inclusion of people and/or
 organizations in any part of the project or research (Kliskey et al.
 2021). To categorize levels of SE, we used a framework adapted from a
 categorization of transdisciplinary research for the FEWS nexus that
 categorizes SE at four levels: Nominal, instrumental, representative, and
 transformative (Ghodsvali et al., 2019). The levels denote increasingly
 greater roles, influence, and/or authority given to stakeholders during
 the engagement process. Table 3 provides a description of the levels of
 engagement in the framework. Stakeholder engagement levels were coded in a
 CSV file for subsequent analysis in R using binary values for each of the
 four separate types of engagement level. Each engagement level was entered
 into a separate column, so nominal engagement, for example, in one column,
 was coded a ‘0’ if that type of engagement was not present and coded as
 ‘1’ if a nominal engagement was present. This binary coding was undertaken
 for each engagement level across four separate columns. If there was
 missing data it was simply excluded. Table 3: Description of stakeholder
 engagement levels from the framework used in the analysis (Ghodsvali et
 al., 2019). If described in the article, the following variables were
 entered into Excel in narrative form or entered as a count in a category
 as appropriate: discipline of researchers, stakeholder type, study
 location by region and spatial scale, time of stakeholder engagement, type
 of stakeholder and a narrative description of proposed and/or implemented
 solutions along with a count for each solution type. The following
 variables were coded into a csv file for analysis in R: solution proposed
 or implemented (Y, N), computational or statistical model (Y, N), solution
 type, researcher discipline (count for each discipline represented in the
 paper), stakeholder type (count for each type listed), geographic region,
 and spatial scale (1 if included, 0 if not). If there was missing data it
 was simply excluded. Statistical analysis All analyses were performed
 using R, version 4.1.2. Given our interest in determining which factors
 influence whether solutions are proposed and/or implemented, our approach
 was threefold (Figure 2): 1\. Descriptive statistics. Overall descriptive
 statistics and exploratory data analysis was performed to provide a broad
 assessment of manuscript characteristics. Frequencies and summary plots
 were used to examine trends over time and distributions across stakeholder
 types and engagement levels. Plotting and graphs were produced using the
 ggplot2 package. 2\. Generalized and logistic regression models. To assess
 multi-categorical and binary outcomes for both stakeholder engagement and
 whether a solution was proposed or not, several regression-based models
 were run across the entire 483 sample dataset. The primary dependent
 variable was whether a solution was proposed / developed while all other
 variables were used as independent variables (Table 1). Logistic
 regression models were used to estimate the odds ratios of proposing or
 implementing a solution based on specific stakeholder engagement
 categories. For categorical predictors, such as engagement level or
 stakeholder type, odds ratios (OR) compare each category to a reference
 group. Confidence intervals (95%) accompany each OR to indicate estimated
 precision, with significance evaluated using an alpha of .05. Regression
 models were produced using the caret package. 3\. Random forest
 classification. Random forest classifiers (ensembled decision trees) were
 employed to identify the most important predictors of solutions, as well
 as to visualize the conditional structure of these relationships. Again,
 the primary dependent variable was whether a solution was proposed /
 developed. Decision trees are a method of constructing a set of decision
 rules on a predictor variable (Brieman et al., 1984; Verbyla, 1987; Clark
 and Pregibon, 1992) that is categorical, using variance partitioning.
 These rules are constructed by recursively partitioning the data into
 successively smaller groups with binary splits based on a single predictor
 variable, with the goal of describing the relationship using the smallest
 number of splits (tree ‘branches’) (Prasad et al., 2006). Random forests,
 or ensemble decision trees, are a combination of many decision trees,
 where each tree depends on the values of a random vector, sampled
 independently and with the same distribution for all trees in the forest
 (Breiman, 2001). Random forest modeling reduces the potential for
 overfitting using bootstrap aggregation (averaging across many trees) and
 provides a level of feature importance for assessing predictor power. Each
 random forest model (1,000 trees) was evaluated with stratified 10-fold
 cross-validation (repeated 5x) in the caret package, with area under the
 curve (AUC) computed from the out-of-fold predictions. We also report the
 apparent AUC from the model refit to the full dataset with the selected
 hyperparameters. For single tree comparisons we used the recursive
 partitioning and regression trees package rpart (CART), while random
 forests were implemented with ranger.(Breiman et al, 1984; Wright and
 Ziegler, 2017; Atkinson and Therneau, 2000;).   ##### Variables * time of
 engagement of stakeholders - If adequately described in the article, we
 characterized when stakeholders were involved as throughout the project,
 early in the project, midway in the project or late in the project. *
 activities in which stakeholders were engaged - if described in the
 article, we copied and pasted that description into the Excel worksheet.
 These included, for example, questionnaires, focus groups, scenario
 development, brainstorming, visioning, identifying the research issue or
 problem, workshops, etc. We counted the number of articles that described
 how they engaged their stakeholders. * how were stakeholders identified -
 if described in the article we described how researchers identified their
 stakeholders. These included, for example, interviews, internet searches,
 snowball sampling, recommendations of key players, etc. We counted the
 number of articles that described how they identified stakeholders. ##
 Code/software Excel is needed to view the Excel file, Excel or a similar
 software is needed to view the csv. No software is needed to view the pdf.
 R 4.4.2 was used to run the csv. Workflow: Literature identification -&gt;
 literature assessment -&gt; coding of data from literature -&gt;
 independence testing -&gt; generation of stakeholder diversity index -&gt;
 generalized linear model