Plan for experimental pilots' validation set-ups and evaluation concept
Creators: name: Österreichische Computer Gesellschaft
Publisher: name: Zenodo
Year: 2025

description: EU pending for approval

 

This deliverable establishes the piloting and validation plan for the Erasmus+ project ComeThinkAgain (COMputational and Entrepreneurship THINKing And Green Agenda INnovations). ComeThinkAgain develops cross-sectoral, standardised training and certification (so called ‘micro-Certification based Education Training System’ -ComeThinkAgainCETS) that improves the teaching of key cross-sectoral skills in the pillars Computational Thinking (CT), Entrepreneurship (EE) and Social Responsibility & Green Skills (GS). This deliverable is focused on project validation, but it should be noted that another related aspect is the design of a certification framework that will be delivered in project deliverable D4.3 - Concept for assessment and certification (M18).

Given the relation between the two frameworks, some aspects related to certification (like certification testing) are touched upon in this deliverable, but are not elaborated on beyond the scope needed to specify the piloting and validation. The plan presented in this deliverable has been prepared based on the knowledge available during the first phase of the project. However, new information may become available that requires changes to the piloting and evaluation approach. Monitoring for and communication of potential changes are part of the piloting and evaluation task and will be addressed accordingly.

Section 1.1 of this deliverable introduces the general piloting and validation plan, starting with the definition of the validation goals. The project has two core validation goals, to (1) validate the learning content and learning methods (didactical and pedagogical concepts) developed and applied in the project, and (2) to validate the certification scheme developed during the project. To achieve this, a set of validation criteria to validate against has been defined (Interactivity, Collaboration, Pedagogy, Content, Technology, Usability, Motivation, Assessment and Support for Learners). Furthermore, the key stakeholders to be addressed for validation have been identified to be (1) the learners participating in pilot interventions (Trainers/Teachers on primary, secondary and vocational levels) and (2) Project stakeholders (Project partners, Pilot organisations, Project advisory board, Community of practice).

As the core concept for validation, an iterative participatory evaluation approach has been chosen that utilises project stakeholders to iteratively assess and interpret the outcomes and results of validation activities, based on qualitative and quantitative data that is collected during validation events from learners. Participatory evaluation has been chosen over purely data-driven and statistical validation because of inherent characteristics that would limit the significance of statistical analysis. The type of research performed in the project, as well as the real-world characteristics in teacher education mandate that results can be piloted only with an indicative number of pilot studies, each performed on an indicative group of participants from the relevant target group. The number of data samples collected during piloting will allow experts to perform assessments and draw conclusions during validation, but will not be large enough to justify validation based on purely data-driven or statistics-based analysis methods.

However, data from piloting activities will be analysed using relevant quantitative, statistical, and qualitative methods to ensure a solid, data-driven foundation for a consensus-based outcome in participatory evaluation.

Section 1.2 lists the core validation events to test project developments and collect qualitative and quantitative feedback from the learners. The events are organised as interventions in which learning content and methods are used to train teachers/trainers.