The Policy Impact of Complex Systems Research: A Quantitative Analysis
Creators: name: Rasuli, Behrooz
Publisher: name: Zenodo
Year: 2024
Subjects: Citation Analysis
Policy Impact
Altmetrics
Scholarly Communication
Bibliometrics
Complex Systems
Science Policy

description: 1          Introduction

Policy-making often deals with intricate issues that exhibit features of complex systems - interconnected components, non-linear dynamics, and emergent properties [1, 2]. Recognizing and approaching these issues through the lens of complex systems thinking holds the potential for more informed policy design [3]. In other words, understanding the intricate dynamics of complex systems is crucial for effective policymaking in different domains, from environmental management to public health and socioeconomic development. Complex systems theory offers a holistic framework that transcends traditional reductionist approaches [4], allowing policymakers to grasp the underlying mechanisms driving system behavior and anticipate unintended consequences of policy interventions. For instance, in environmental management, policies aimed at mitigating climate change must contend with the complex interplay of various factors [5] such as greenhouse gas emissions, land use changes, and socio-economic dynamics. By embracing complexity science, policymakers can develop more resilient and adaptive strategies that account for the interconnectedness and feedback loops inherent in environmental systems. Similarly, in the realm of public health, the spread of infectious diseases often exhibits complex dynamics influenced by factors ranging from human behavior and social networks to environmental conditions and healthcare infrastructure [6, 7]. Policies aimed at containing disease outbreaks or promoting public health must navigate this intricate web of interactions, leveraging insights from complex systems research to design targeted interventions and anticipate the ripple effects of policy decisions. Moreover, in the context of socioeconomic development, complex systems thinking can inform policies aimed at addressing systemic inequalities, fostering economic resilience, and promoting sustainable growth [8, 9]. By understanding the underlying dynamics of socio-economic systems, policymakers can identify leverage points for intervention and design policies that promote inclusive development while minimizing unintended consequences such as social unrest or ecological degradation. Incorporating insights from research on complex systems into policy-making processes holds the promise of more effective and adaptive governance in an increasingly interconnected and unpredictable world [10]. By embracing complexity science, policymakers can transcend reductionist approaches and develop policies that are better attuned to the multifaceted nature of the challenges facing society. Moreover, fostering interdisciplinary collaboration between researchers, policymakers, and stakeholders can enhance the co-production of knowledge and facilitate the translation of complex systems theory into actionable policy solutions. Ultimately, the desired outcomes and interventions that policymakers aim to implement are formally articulated and documented within the policy documents. Policy documents, encompassing government regulations, guidelines, think tank reports, and working papers, play a crucial role in shaping the implementation and direction of diverse initiatives [11]. They serve as official communication channels for outlining specific rules, procedures, and expectations intended to guide various stakeholders. By analyzing citations within these documents, we gain valuable insights into the sources of information and research that inform policy decisions. In this context, identifying citations of complex systems research in policy documents allows us to assess the extent to which this field of study contributes to the foundation of policy frameworks. Ultimately, understanding the types and sources of policy documents citing complex systems research sheds light on the areas of policy-making most receptive to this approach and the organizations driving its integration [12]. This research investigates the utilization of complex systems research in policy-making, specifically focusing on the extent, areas, types, and organizational sources of such usage. It aims to elucidate the extent to which publications on complex systems are cited in policy documents, the specific research areas within which these citations occur, the types of policy documents citing such publications, and the diverse array of organizations or institutions engaging with this scholarly literature in the realm of policy formulation.

2          Methodology

To address the research questions, the current study adopted an Altmetrics approach. Beyond traditional citation metrics, researchers and institutions are increasingly utilizing altmetrics to gauge the broader reach and potential influence of their scientific contributions. These alternative metrics capture various forms of online engagement, offering a more comprehensive picture of a research product's impact beyond the academic sphere [12, 13]. Initially, the Web of Science Core Collection (including Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation, Emerging Sources Citation Index, Conference Proceedings Citation Index – Science, Conference Proceedings Citation Index - Social Sciences & Humanities, Book Citation Index– Science, and Book Citation Index– Social Sciences & Humanities) was utilized to identify pertinent publications related to "complex systems." A targeted search query, encompassing titles, author keywords, and keyword plus fields, on 23 March, 2024 was employed: 

TI=("Complex system" OR “Multidimensional Scaling” OR “Correspondence Analysis”) OR AK=("Complex system" OR “Multidimensional Scaling” OR “Correspondence Analysis”) OR KP=("Complex system" OR “Multidimensional Scaling” OR “Correspondence Analysis”)

This query yielded a dataset comprising 12042 relevant records, serving as the foundation for subsequent analysis. Subsequently, publications within this dataset that possess a Digital Object Identifier (DOI) were subjected to further scrutiny using PlumX. PlumX facilitates the exploration of alternative metrics, particularly citations within policy documents. By cross-referencing publications with DOI identifiers, the study identified instances where scholarly works on complex systems are cited in policy documents. From the retrieved records, 9498 records had a DOI identifier and were selected for further analysis. Exploring DOIs on Altmetric.com platform and extracting data were the next steps.

Altmetric.com goes beyond traditional citation metrics by including a unique category called "policy citations." This metric tracks how many times a research article (or other scholarly work) is mentioned or referenced in official policy documents. The collected data on publication characteristics (e.g., publication year, journal) and policy citations were collected in an MS Excel file and analyzed using descriptive statistics to address the research questions. This analysis involved calculating measures like frequency distributions and percentages to understand the extent of policy citations for publications on complex systems. 

Following the identification of cited publications within policy documents, the analysis unfolds across several dimensions. Firstly, the number of publications cited in policy documents is quantified, providing quantitative insights into the prevalence and dissemination of complex systems research within policy contexts.

3          Limitations

The study relies on keyword-based search strategies, which may not capture all relevant publications or policy documents.

Analyzing citations alone might not fully capture the true impact of complex systems research on policy-making processes.

The study focuses on a single database and might not capture the full scope of policy documents across different regions or languages.

4          References

[1]         P. Cairney, "Complexity Theory in Political Science and Public Policy," Political Studies Review, vol. 10, no. 3, pp. 346-358, 2012/09/01 2012, doi: https://doi.org/10.1111/j.1478-9302.2012.00270.x.

[2]         M. M. Atkinson and W. D. Coleman, "Policy Networks, Policy Communities and the Problems of Governance," Governance, vol. 5, no. 2, pp. 154-180, 1992/04/01 1992, doi: https://doi.org/10.1111/j.1468-0491.1992.tb00034.x.

[3]         P. H. Jones, "Systemic Design Principles for Complex Social Systems," in Social Systems and Design, G. S. Metcalf Ed. Tokyo: Springer Japan, 2014, pp. 91-128.

[4]         F. Popa, M. Guillermin, and T. Dedeurwaerdere, "A pragmatist approach to transdisciplinarity in sustainability research: From complex systems theory to reflexive science," Futures, vol. 65, pp. 45-56, 2015/01/01/ 2015, doi: https://doi.org/10.1016/j.futures.2014.02.002.

[5]         X. Bai, R. R. J. McAllister, R. M. Beaty, and B. Taylor, "Urban policy and governance in a global environment: complex systems, scale mismatches and public participation," Current Opinion in Environmental Sustainability, vol. 2, no. 3, pp. 129-135, 2010/08/01/ 2010, doi: https://doi.org/10.1016/j.cosust.2010.05.008.

[6]         L. N. Susan et al., "Complex health interventions in complex systems: improving the process and methods for evidence-informed health decisions," BMJ Global Health, vol. 4, no. Suppl 1, p. e000963, 2019, doi: 10.1136/bmjgh-2018-000963.

[7]         R. L. Grant and R. Hood, "Complex systems, explanation and policy: implications of the crisis of replication for public health research," Critical Public Health, vol. 27, no. 5, pp. 525-532, 2017/10/20 2017, doi: 10.1080/09581596.2017.1282603.

[8]         A. L. Mudd, M. Bal, S. E. Verra, M. P. Poelman, J. de Wit, and C. B. M. Kamphuis, "The current state of complex systems research on socioeconomic inequalities in health and health behavior—a systematic scoping review," International Journal of Behavioral Nutrition and Physical Activity, vol. 21, no. 1, p. 13, 2024/02/05 2024, doi: 10.1186/s12966-024-01562-1.

[9]         D. Helbing, S. Balietti, S. Bishop, and P. Lukowicz, "Understanding, creating, and managing complex techno-socio-economic systems: Challenges and perspectives," The European Physical Journal Special Topics, vol. 195, no. 1, pp. 165-186, 2011/05/01 2011, doi: 10.1140/epjst/e2011-01410-7.

[10]       K. Akamani, E. J. Holzmueller, and J. W. Groninger, "Managing Wicked Environmental Problems as Complex Social-Ecological Systems: The Promise of Adaptive Governance," in Landscape Dynamics, Soils and Hydrological Processes in Varied Climates, A. M. Melesse and W. Abtew Eds. Cham: Springer International Publishing, 2016, pp. 741-762.

[11]       H. Pinheiro, E. Vignola-Gagné, and D. Campbell, "A large-scale validation of the relationship between cross-disciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database," Quantitative Science Studies, vol. 2, no. 2, pp. 616-642, 2021, doi: 10.1162/qss_a_00137.

[12]       Y. Houqiang, C. Xueting, X. Tingting, and Y. Zhenyi, "How accurate are policy document mentions? A first look at the role of altmetrics database," Scientometrics, vol. 125, no. 2, pp. 1517-1540, 2020, doi: 10.1007/s11192-020-03558-.

[13]       R. Haunschild and L. Bornmann, "How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data," Scientometrics, vol. 110, no. 3, pp. 1209-1216, 2017/03/01 2017, doi: 10.1007/s11192-016-2237-2.