D5.2: Interim Evaluation and Impact Assessment Report
Creators: name: Schaefer, Teresa
name: Fabian, Claudia Magdalena
name: Kieslinger, Barbara
Publisher: name: Zenodo
Year: 2022
Subjects: impact assessment
citizen science
institutional changes

description: The current document, titled “Interim Evaluation and Impact Assessment Report”, has been developed within the framework of the TIME4CS project which is funded by the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No 101006201. Evaluation and impact assessment activities in TIME4CS aim to: jointly assess the progress of implemented institutional changes in the participating organizations and understand mutual learning effects between Front-Runners and Implementers of the project. develop a set of indicators to assess institutional change to promote citizens engagement in science along the four intervention areas of “Research”, “Education &amp; Awareness”, “Support Resources &amp; Infrastructure”, and “Policy &amp; Assessment”. provide research performing organizations (RPOs) and research funding organizations (RFOs) with a monitoring toolkit to self-assess and monitor their progress towards organizational change. learn about the implementation process of institutional changes in implementing organizations to understand the different pathways taken within the organizations and their intended and unintended consequences. Thus, evaluation and impact assessment comprises of: a summative evaluation of the project’s benefits in participating organizations a formative evaluation of the implementation of roadmaps and Grounding Actions that helps to iteratively shape the Grounding Actions, the training formats, encounters, and materials developed by the project In TIME4CS evaluation was integrated in all project activities from the very beginning - in a first step we elaborated an Evaluation and Impact Assessment plan. This plan defined key indicators for the summative and formative evaluation, as well as first evaluation instruments and a timeline. Also, a first stock-taking with the implementer organizations helped to understand the baseline before implementing the roadmaps and grounding actions. Based on this initial planning and stock taking, further activities in project year two where twofold: The stepwise implementation of grounding actions allowed a more detailed planning of evaluation instruments for each implementer organisation and the project training programme. 2 Implementer Journey Interviews and 2 Implementer Forums helped to keep track of Implementers’ challenges and lessons learned from following the roadmaps in their organizations. In addition, these encounters together with another stock-taking exercise at the end of project year two helped to understand the first outputs and outcomes of the project in each implementer organisation. What we learned from these first data collection activities is that the Implementers face several challenges related to the integration of citizen science in their research activities: Actively engaging citizens in research is perceived by many researchers as an “on-top” activity in their already very competitive work environment that requires specific skills regarding ethics, data quality, citizens’ engagement and facilitation etc. There are also research topics where the implementation of the citizen science methodology is not immediate (sometimes not possible) and requires considering very carefully how to involve citizens, e.g., when working with very expensive laboratory equipment or in sensible fields like biomedicine. For many researchers we observe a confusion between what is citizen science, science communication and other forms of participatory research. Operating in an environment with very scarce resources regarding time and budget it is perceived as a challenge for many scientists to engage in this new research approach. Given these challenges, the first activities of the TIME4CS team members in the different implementer organizations were dedicated to exploring the specific contexts in their organizations and start building citizen science “core teams” integrating representatives from different support organisations’ Open Science Offices, Science Communication teams, Training departments. These teams started developing targeted approaches of explaining and demonstrating the value of citizen science to researchers and the management and elaborating support structures within their organizations to lower the entry barrier of researchers to the new research approach that actively engages citizens in the scientific process. Thus, all Implementers succeeded in triggering first institutional changes to support citizen science due to their involvement in the TIME4CS project and the implementation of grounding actions. In every implementer organisation the core-teams started to plan and continually adapt the grounding actions. They themselves increased their knowledge on citizen science due to the mutual learning exercises with Front-Runners and the exchange with other Implementers. The members of these teams reached out to the European Citizen Science Association (ECSA), other European organizations which are actively involved in citizen science activities, and they reached out to potential interested researchers within their organizations. The core-team members collected material related to citizen science, started to plan and implement training and inspirational talks for researchers and students within their organizations and worked on first adaptations of institutional policy documents and guidelines to consider citizen science. Another important activity started was the implementation of a citizen science facilitator or contact point. The sustainable financial support of this person or persons is a key challenge in all implementing organisations. Also, Front-Runners benefited a lot from their involvement in the TIME4CS activities, getting a clearer picture of the activities they, themselves organise within their institutions and developing strategies on how to better communicate these activities to their audiences. They learned about all the challenges that Implementers face when driving citizen science top-down in RPOs and how the different contexts influence the paths and grounding actions that need to be taken for change to happen. As a result, they started thinking about their own new Grounding Actions and realised that despite different levels of institutional support for citizen science the challenges remain the same in all organizations: these are dedicated to a sustainable institutional funding for internal support structures and a wide-spread knowledge on citizen science amongst researchers and support staff in order to make citizen science an integral and widely applied research method. The final project year, when the implementation of grounding actions will be pursued in the different organisational contexts will bring on new outputs, outcomes and learnings for all partners involved, which will be documented in the D5.3. Final Evaluation and Impact Assessment Report.