PROLIFERATE: An adaptable framework with tools to evaluate different processes, outputs, and products via participatory research
Creators: name: Pinero de Plaza, Maria Alejandra
Publisher: name: figshare
Year: 2022
Subjects: subject: 160808 Sociology and Social Studies of Science and Technology
subject: FOS: Sociology
subject: 111799 Public Health and Health Services not elsewhere classified
subject: FOS: Health sciences
subject: 80702 Health Informatics
subject: FOS: Media and communications
subject: Health Care
subject: 111712 Health Promotion
subject: 111711 Health Information Systems (incl. Surveillance)
subject: 111708 Health and Community Services
subject: 160508 Health Policy
subject: FOS: Political science
subject: 111717 Primary Health Care
subject: 111705 Environmental and Occupational Health and Safety

description: This framework presents a constructivist evaluation procedure for assessing the personal construction of meaning (e.g., end-user experiences, understandings, beliefs, and perceptions)1-3. It considers the non-linear characteristics of complex and adaptive issues/products/problems4 from an end-user perspective. It is operationalized to evaluate and monitor real-world situations, research outcomes, and products. It assesses their fitness via person-centered parameters, which can be adaptable for evaluating different subjective experiences concerning any evaluated matter and associated behaviors. It can be utilized to evaluate "usability" and its impact (i.e., the consequence of the usability or a prediction of such effect) on products and processes. Usability can refer to a broad set of criteria (e.g., satisfaction, learnability, efficiency, memorability, robustness, effectiveness, and accessibility)2,3,5. These constructs fluctuate depending on each end-user context and their adaptive dynamics2,6-10. The versatile and variable nature of end-user perceptions/experiences can be captured, qualified, and quantified as 'feedback' by this approach: PROLIFERATE: An ada<strong>p</strong>table framewo<strong>r</strong>k with to<strong>o</strong>ls to eva<strong>l</strong>uate d<strong>i</strong>f<strong>f</strong>erent process<strong>e</strong>s, outputs, and p<strong>r</strong>oducts vi<strong>a</strong> par<strong>t</strong>icipatory res<strong>e</strong>arch. It is an emergent dialectical multimethod evaluation 11-15. The PROLIFERATE metaparadigm method (epistemological basis and procedure in Figure 1)11-14 accommodates a way to qualify and/or quantify each end-user's subjective experience using a dialectical pluralism approach16, which can be adaptable to the evaluation of AI (i.e., usability and impact) because, in PROLIFERATE, the evaluated matter is assessed considering these domains of end-users behaviors or constructs:17 1. Comprehension or understanding 2. Resonance or emotional responses 3. Uptake barriers 4. Motivation to use it and associated behavior change 5. Optimization suggestions PROLIFERATE design11-14 implies that, for instance, the usability of AI can be evaluated by adapting and analyzing the combination of its five constructs to obtain estimations of AI usability (constructs 1,2,3) and impact (constructs 4, 5). These constructs can be analyzed using different procedures, including Bayesian statistics and prediction modeling11-14. In this method, Knowledge Translation (KT) is understood as per the Knowledge Translation Complexity Network Model (KT-CNM)4. It refers to simultaneous networks interacting in non-linear ways in which different agents/people, entities, or nodes (i.e., researchers, community, clinicians, etc.), are linked, connected, or interact based on their social features, conditions, institutional structures, roles, and behaviors, represented in knowledge translation processes or stages: problem identification, evaluation, implementation, knowledge creation, etc.)4,8,18. This complex lens does not assume cause and effect but a continuum of interactions between people and knowledge. PROLIFERATE takes a complex system approach as an evaluation framework as it can be combined with quantitative tools while emphasizing collaboration with end-users (i.e., participatory research)11-14. This emergent evaluation framework is evolving within several KT projects: · It has been ethically approved to assess interdisciplinary learnings, e.g., (1) within a university allied health practice (Health2Go) and (2) pilot tested within the co-design of KT video resources.11,12 · It has been examined in knowledge actualization discussions concerning a science implementation project.13 · It is currently being adapted to evaluate research training modules, lab-training strategies, and AI.14,15 <strong>References</strong>: 1. Louder E, Wyborn C, Cvitanovic C, Bednarek AT. A synthesis of the frameworks available to guide evaluations of research impact at the interface of environmental science, policy and practice. <em>Environmental Science &amp; Policy</em> 2021; <strong>116</strong>: 258-65. 2. Dawood KA, Sharif KY, Ghani AA, Zulzalil H, Zaidan A, Zaidan B. Towards a unified criteria model for usability evaluation in the context of open source software based on a fuzzy Delphi method. <em>Information and Software Technology</em> 2021; <strong>130</strong>: 106453. 3. Schwartz JM, Moy AJ, Rossetti SC, Elhadad N, Cato KD. Clinician involvement in research on machine learning–based predictive clinical decision support for the hospital setting: A scoping review. <em>Journal of the American Medical Informatics Association</em> 2021; <strong>28</strong>(3): 653-63. 4. Kitson A, Brook A, Harvey G, et al. Using complexity and network concepts to inform healthcare knowledge translation. <em>International Journal of Health Policy and Management</em> 2018; <strong>7</strong>(3): 231. 5. Hartson HR, Andre TS, Williges RC. Criteria For Evaluating Usability Evaluation Methods. <em>International Journal of Human-Computer Interaction</em> 2001; <strong>13</strong>(4): 373-410. 6. May CK. Complex adaptive governance systems: a framework to understand institutions, organizations, and people in socio-ecological systems. <em>Socio-Ecological Practice Research</em> 2022. 7. Pinero De Plaza M. A transdisciplinary research program addressing complex health research problems. <em>JBI Evidence Implementation's Bulletin</em> 2022; <strong>2</strong>(Issue 1). 8. Pinero De Plaza MA, Conroy T, Mudd A, Kitson A. Using a Complex Network Methodology to Track, Evaluate, and Transform Fundamental Care. IOS Press, Nurses and Midwives in the Digital Age. IOS Press: IOS Press; 2021: 31-5. 9. Conroy T, Pinero de Plaza MA, Mudd A, Mitchell M, Kitson A. Measuring fundamental care using complexity science: A descriptive case study of a methodological innovation. <em>Journal of Clinical Nursing</em> 2021. 10. Sousa V, Lopez KD. Towards Usable E-Health. <em>Applied Clinical Informatics</em> 2017; <strong>08</strong>(02): 470-90. 11. Pinero de Plaza MA, Archibald M, Lawless M, et al. PROLIFERATE: An Adaptable Framework to Evaluate Participatory Research Products. https://www.researchsquare.com/article/rs-146129/v1: Research Square Platform LLC; 2021. 12. Archibald M, Ambagtsheer R, Lawless MT, et al. Co-Designing Evidence-Based Videos in Health Care: A Case Exemplar of Developing Creative Knowledge Translation “Evidence-Experience” Resources. <em>International Journal of Qualitative Methods</em> 2021; <strong>20</strong>: 160940692110196. 13. Pinero de Plaza M, Murray J, Hunter S, Kourbelis C. March KT Conversations [abstract]- Applying the PROLIFERATE Framework to the REDUCE Missed Oral Healthcare: It Takes a Team Project.22nd of March 2021, 2021. https://researchnow.flinders.edu.au/en/activities/march-kt-conversations-applying-the-proliferate-framework-to-the- (accessed 29.03.22). 14. Pinero de Plaza MA, Lambrakis K, Morton E, et al. PROLIFERATE: A Tool to Measure Impact and Usability of AI-Powered Technologies. Digital Health Institute Summit; 2022, Feb 21; https://researchnow.flinders.edu.au/en/publications/proliferate-a-tool-to-measure-impact-and-usability-of-ai-powered-: The Australasian Institute of Digital Health Summit; 2022, Feb 21. 15. Pinero-de-Plaza MA, Lambrakis K, Barrera-Causil CJ, et al. PROLIFERATE_AI: A novel approach to evaluate the implementation of Artificial Intelligence Clinical Decision Support. In preparation; 2022. 16. Johnson RB. Dialectical pluralism: A metaparadigm whose time has come. <em>Journal of Mixed Methods Research</em> 2017; <strong>11</strong>(2): 156-73. 17. Scotland ES. Evaluation methods and tools – resources- Body Map. http://www.evaluationsupportscotland.org.uk/media/uploads/resources/pdf_method_-_body_map.pdf (accessed 06/07/2019 2019). 18. Rycroft-Malone J. From Linear to Complicated to Complex Comment on "Using Complexity and Network Concepts to Inform Healthcare Knowledge Translation". <em>International Journal of Health Policy and Management</em> 2018; <strong>7</strong>(6): 566-8.